Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
feat(telemetry): failure reasons and durations for connector events, plus server-side fixes and DB/bridge timing (#488)
Christian McDermottf4ada844c340
diff --git a/console/AGENTS.md b/console/AGENTS.mdindex 52c131bc3..5e6d415bb 100644--- a/console/AGENTS.md+++ b/console/AGENTS.md@@ -429,12 +429,19 @@ pnpm run lintenough — excess-property checking does not apply through a spread — so the runtimefilter is what makes "nothing free-text can reach a payload" true rather than intended.Permitted: which of the catalogued things happened, agent type, local-vs-remote,- success-vs-failure, app version, operating system, and the random install id. Never:- prompts, code, file paths, working directories, error messages or stack traces (use an- enumerated code), machine or user names, IP or MAC addresses, email or sign-in, and no- agent, room, project, location or server names or ids. Widening this is a consent- decision — the user-facing wording lives in- `src/renderer/features/telemetry/telemetry-copy.ts` and must be kept in step with it.+ success-vs-failure, how long an operation took, app version, operating system, and the+ random install id. A duration is a number rather than a value from a fixed set,+ so it is held to `TelemetryDurationMs`: measured on a monotonic clock, whole+ milliseconds, and never a span that could encode something else. That is a branded+ type, not an alias for `number`, and `startTimer()` is the only thing that mints one —+ so a call site cannot reach for `Date.now() - startedAt`, which is not monotonic and+ across a clock step yields a negative number nothing at the far end can tell from data.+ Never: prompts, code, file paths, working directories, error messages or stack traces+ (use an enumerated code), machine or user names, IP or MAC addresses, email or sign-in,+ and no agent, room, project, location or server names or ids. Widening this is a consent+ decision — the summary the user consents to lives in+ `src/renderer/features/telemetry/telemetry-copy.ts`, the field-by-field disclosure it+ links to is `docs/TELEMETRY.md`, and both must be kept in step with it.- **The interface may report only through the one gate.** Almost everything isreported from the main process, where a call site is type-checked against thecatalogue. The exceptions are moments that exist only in the UI — a screendiff --git a/console/apps/switch-console-desktop/src/main/core/providers/controller.ts b/console/apps/switch-console-desktop/src/main/core/providers/controller.tsindex 08d48662d..f10acd941 100644--- a/console/apps/switch-console-desktop/src/main/core/providers/controller.ts+++ b/console/apps/switch-console-desktop/src/main/core/providers/controller.ts@@ -5,11 +5,7 @@ import type {HostDependencySelection,InstallOverride,} from '@switch-console/core/deps/runtime';-import { agentTypeOf } from '@main/core/telemetry/agent-type';-import { cliFailureReason } from '@main/core/telemetry/cli-failure';-import type { TelemetryCliAction } from '@main/core/telemetry/events';-import { installMethodOf } from '@main/core/telemetry/narrow';-import { trackEvent } from '@main/core/telemetry/telemetry-service';+import { reportedCliAction } from '@main/core/telemetry/cli-action';import type { ProviderCustomConfig } from '@shared/core/app-settings';import type { AgentProviderId } from '@shared/core/providers/agent-provider-registry';import { createRPCController } from '@shared/lib/ipc/rpc';@@ -31,28 +27,13 @@ const enrichHostDep = () => agentUpdateService.enrichHostDependency(id, hostDep);/**- * Report installing, updating or removing an agent's CLI.+ * The target every CLI action reported from here carries.*- * `target` is always local here. This controller accepts a connection id and- * discards it — `getDependencyManager` returns the local manager whatever it is- * given — so nothing reached through it runs anywhere else. The genuinely remote- * equivalents live on the remote-hosts controller and report themselves.+ * Always local. This controller accepts a connection id and discards it —+ * `getDependencyManager` returns the local manager whatever it is given — so+ * nothing reached through it runs anywhere else.*/-function reportCliAction(- action: TelemetryCliAction,- id: string,- method: InstallMethod | undefined,- result: { success: boolean; error?: { type?: string } }-): void {- trackEvent('agent_cli_action', {- agent_type: agentTypeOf(id),- target: 'local',- install_method: installMethodOf(method),- action,- outcome: result.success ? 'success' : 'failure',- failure_reason: cliFailureReason(result),- });-}+const TARGET = 'local';export const providersController = createRPCController({// ── Metadata ────────────────────────────────────────────────────────────────@@ -86,8 +67,9 @@ export const providersController = createRPCController({install: async (id: AgentProviderId, connectionId?: string, method?: InstallMethod) => {const mgr = await getDependencyManager(connectionId);- const result = await mgr.install(id, method);- reportCliAction('install', id, method, result);+ const result = await reportedCliAction('install', TARGET, id, method, () =>+ mgr.install(id, method)+ );if (result.success) {// Persist the chosen method as an override, or clear to auto when no method was chosen.// Do NOT auto-promote the inferred method — that would freeze a heuristic guess.@@ -100,16 +82,12 @@ export const providersController = createRPCController({update: async (id: AgentProviderId, connectionId?: string, method?: InstallMethod) => {const mgr = await getDependencyManager(connectionId);- const result = await mgr.update(id, method);- reportCliAction('update', id, method, result);- return result;+ return reportedCliAction('update', TARGET, id, method, () => mgr.update(id, method));},uninstall: async (id: AgentProviderId, connectionId?: string, method?: InstallMethod) => {const mgr = await getDependencyManager(connectionId);- const result = await mgr.uninstall(id, method);- reportCliAction('uninstall', id, method, result);- return result;+ return reportedCliAction('uninstall', TARGET, id, method, () => mgr.uninstall(id, method));},// ── Settings ─────────────────────────────────────────────────────────────────diff --git a/console/apps/switch-console-desktop/src/main/core/remote-hosts/setup/host-setup-service.ts b/console/apps/switch-console-desktop/src/main/core/remote-hosts/setup/host-setup-service.tsindex 302b74eff..7da280c42 100644--- a/console/apps/switch-console-desktop/src/main/core/remote-hosts/setup/host-setup-service.ts+++ b/console/apps/switch-console-desktop/src/main/core/remote-hosts/setup/host-setup-service.ts@@ -341,7 +341,7 @@ export async function updateStep(): Promise<StepInstallResult> {if (step.kind === 'agent-plugin') {const service = await getRemoteSwitchSetupService(sshHost);- const result = await service.update(stepAgentId(step));+ const result = await service.update(stepAgentId(step), 'user');return result.success? { ok: true }: { ok: false, error: result.message ?? 'Could not update the Switch connector.' };diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.test.tsindex d42e20de1..d4d3b1d7c 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.test.ts@@ -159,7 +159,25 @@ describe('latching — the difference between a one-shot and a standing job', ()});await catchUpConnectorsToCurrentVersion();- expect(update).toHaveBeenCalledWith('codex');+ expect(update).toHaveBeenCalledWith('codex', 'catch_up');expect(kvSet).toHaveBeenCalledWith('state', { generation: 0, attempts: 1 });});++ /**+ * This is not somebody pressing Update.+ *+ * It goes through the same service method, unattended, at launch, across every+ * installed connector at once. Reported as `user` it would put its latencies+ * and its failures into the numbers that answer "how well does the Update+ * button work" — and it is the sweep, not the button, that runs on a machine+ * nobody is watching.+ */+ it('reports its updates as the catch-up rather than as the Update button', async () => {+ checkForUpdates.mockImplementation((id: string) => Promise.resolve(behind(id)));++ await catchUpConnectorsToCurrentVersion();++ expect(update.mock.calls.every(([, trigger]) => trigger === 'catch_up')).toBe(true);+ expect(update).not.toHaveBeenCalledWith(expect.anything(), 'user');+ });});diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.tsindex 23b700f1c..91b23b95c 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/catch-up-connectors.ts@@ -57,7 +57,7 @@ async function catchUpOne(agentId: string): Promise<Step> {if (!status.updateAvailable) return 'nothing-to-do';- const result = await switchSetupService.update(agentId);+ const result = await switchSetupService.update(agentId, 'catch_up');if (!result.success) {log.warn('catchUpConnectors: update failed', {event: 'connector_catch_up_failed',diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/connector-run.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/connector-run.tsnew file mode 100644index 000000000..5f92d3203--- /dev/null+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/connector-run.ts@@ -0,0 +1,204 @@+import type { TelemetryConnectorFailure } from '@main/core/telemetry/events';+import { log } from '@main/lib/logger';+import { getPlugin } from '../providers/plugin-registry';+import type { RegisteredMarketplace } from './switch-setup-cli-dialect';++/** Outcome of a mutating operation, mirroring the providers controller shape. */+export type SwitchSetupResult = { success: boolean; message?: string };++/**+ * A completed connector operation: what the caller gets back, and the+ * enumerated reason it failed.+ *+ * The two travel together because a `SwitchSetupResult` carries only a message,+ * and a message cannot be reported — so the code has to be named where the+ * failure is known rather than recovered from the text afterwards.+ *+ * This module is the leaf both drivers share. Everything here is either pure or+ * reads the plugin registry; nothing in it touches an execution context. That is+ * what lets the remote driver import it without pulling in the local one, whose+ * module body ends in `new SwitchSetupService()` over a `LocalExecutionContext`.+ * Anything added here has to keep that true. See the same lesson recorded on+ * `telemetry/agent-type.ts`.+ */+export type ConnectorRun = {+ result: SwitchSetupResult;+ failure: TelemetryConnectorFailure;+};++/** Status of an agent type's Switch connector plugin. */+export type SwitchSetupStatus = {+ agentId: string;+ /** False when the agent type declares no Switch setup (kind: 'none'). */+ supported: boolean;+ installed: boolean;+ installedVersion: string | null;+ latestVersion: string | null;+ updateAvailable: boolean;+ /**+ * Set when the returned versions are not known to be current: a checkForUpdates+ * marketplace refresh that failed, so they come from the stale local cache, or+ * a status read that could not be answered at all. Null when the status is+ * simply what it says.+ */+ refreshError: string | null;+};++/** Every command a connector driver runs is bounded by this. */+export const EXEC_TIMEOUT_MS = 120_000;++/**+ * What a driver's command came back with, and whether the shell could find the+ * binary at all.+ *+ * `notFound` is separate from a non-zero code because it is not a verdict on+ * what was asked: the host's CLI is not on the machine, so nothing was asked of+ * anything.+ */+export type ConnectorRunResult = {+ code: number;+ stdout: string;+ stderr: string;+ notFound: boolean;+};++/**+ * The host's own CLI is not installed.+ *+ * Its own error class because `ensureMarketplace` reports failure by throwing,+ * and without one the driver cannot tell "the marketplace would not register"+ * from "there is no CLI here to register it with".+ */+export class HostCliMissingError extends Error {+ constructor(binaryName: string) {+ super(`${binaryName} is not installed, so its Switch connector cannot be managed.`);+ this.name = 'HostCliMissingError';+ }+}++/**+ * The code for a command that did not succeed.+ *+ * Every command a marketplace-driven connector runs goes through the host's own+ * binary, so a shell reporting that binary absent says the same thing whichever+ * command it was refusing: the agent is not installed. Filing that under the+ * verb that happened to run first reports "the marketplace would not register"+ * against a machine that has no CLI to register one with — and sends whoever+ * reads it at the marketplace repository. It is the commonest way this fails and+ * the cheapest to act on, so it gets its own code rather than a share of three.+ */+export function commandFailureCode(+ res: { notFound: boolean },+ whenPresent: TelemetryConnectorFailure+): TelemetryConnectorFailure {+ return res.notFound ? 'host_cli_missing' : whenPresent;+}++/**+ * The failure a marketplace repair came back with.+ *+ * `ensureMarketplace` reports by throwing, so the error's class is the only+ * thing separating "the marketplace would not register" from "there is no CLI+ * here to register one with". The second answers with its own message —+ * prefixing it with "could not add marketplace" would name the wrong thing to+ * fix — which is why the caller's wording is passed in rather than built here.+ */+export function marketplaceFailed(err: unknown, message: string): ConnectorRun {+ return err instanceof HostCliMissingError+ ? connectorFailed(err.message, 'host_cli_missing')+ : connectorFailed(message, 'marketplace_failed');+}++/** Whether a registered marketplace entry points at the expected source. */+export function marketplaceMatchesSource(entry: RegisteredMarketplace, source: string): boolean {+ return entry.source === source;+}++/** A connector operation that did what was asked. */+export function connectorSucceeded(): ConnectorRun {+ return { result: { success: true }, failure: 'none' };+}++export function connectorFailed(message: string, failure: TelemetryConnectorFailure): ConnectorRun {+ return { result: { success: false, message }, failure };+}++/** The answer for an agent whose connector nothing here can manage. */+export function connectorUnsupported(): ConnectorRun {+ return connectorFailed('Switch setup is not supported for this agent.', 'unsupported');+}++/**+ * The status for an agent type that declares no Switch connector, or whose+ * connector nothing here can resolve.+ */+export function unsupportedStatus(agentId: string): SwitchSetupStatus {+ return {+ agentId,+ supported: false,+ installed: false,+ installedVersion: null,+ latestVersion: null,+ updateAvailable: false,+ refreshError: null,+ };+}++/**+ * Whether the agent type declares no Switch connector at all.+ *+ * Such an agent did not *fail* to install one, so the mutating entry points+ * return before their timer starts and report nothing. Without the distinction+ * every agent that has no connector would read as a failing install.+ */+export function declaresNoConnector(agentId: string): boolean {+ return getPlugin(agentId).capabilities.switchSetup.kind === 'none';+}++/** The result those entry points return, which is not reported. */+export const CONNECTOR_UNSUPPORTED_RESULT: SwitchSetupResult = connectorUnsupported().result;++/**+ * An agent that declares a file-based connector and ships no behavior for it.+ *+ * Its own class so the driver can tell it from every other way resolving can+ * fail. A dead SSH channel and a plugin that was authored wrong are different+ * walls, and reporting the first as the second puts a claim about the shipped+ * app on a host-side fault.+ */+export class FilesConnectorUnimplementedError extends Error {+ constructor(agentId: string) {+ super(+ `Agent '${agentId}' declares a file-based Switch connector but implements no behavior for it.`+ );+ this.name = 'FilesConnectorUnimplementedError';+ }+}++/**+ * Run a connector operation so that it always comes back as a result.+ *+ * A driver resolves the host binary before it reaches anything that returns a+ * `ConnectorRun`, and resolving rejects on a transport failure rather than+ * reporting absence — a dead SSH channel is not evidence the binary is missing.+ * Unguarded, that rejection escapes the entry point: no event for an attempt+ * the user definitely made, and a stack in the renderer instead of a message.+ * `error` is the residue with no better name; everything with one is classified+ * before it gets here.+ *+ * Only what happens before a driver can name an outcome. A file-based connector+ * builds its filesystem inside `runFiles`, so a channel dying there is the write+ * failing and is reported as that.+ */+export async function runReportedOperation(+ logPrefix: string,+ agentId: string,+ op: () => Promise<ConnectorRun>+): Promise<ConnectorRun> {+ try {+ return await op();+ } catch (err) {+ log.error(`${logPrefix}: connector operation threw`, { agentId, err });+ return connectorFailed(err instanceof Error ? err.message : String(err), 'error');+ }+}diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.test.tsindex 023ca26f9..df866fb81 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.test.ts@@ -1,5 +1,7 @@import { expect, it, vi } from 'vitest';-const mocks = vi.hoisted(() => ({ probe: vi.fn() }));+import type { SwitchSetupStatus } from './connector-run';++const mocks = vi.hoisted(() => ({ probe: vi.fn(), listAgentTypeStatuses: vi.fn() }));vi.mock('@main/core/dependencies/remote-dependency-manager', () => ({getRemoteDependencyManager: async () => ({ probe: mocks.probe }),}));@@ -13,11 +15,31 @@ vi.mock('@main/lib/logger', () => ({ log: { warn: vi.fn() } }));vi.mock('./switch-setup-service', () => ({ switchSetupService: {} }));vi.mock('./remote-switch-setup', () => ({getRemoteSwitchSetupService: async () => ({- listAgentTypeStatuses: async () => [{ agentId: 'claude', supported: true, installed: true }],+ listAgentTypeStatuses: mocks.listAgentTypeStatuses,}),}));const { switchSetupController } = await import('./controller');++/**+ * A whole status row, because the field that matters here is the one a partial+ * fixture leaves `undefined` — and a status that could not be read is exactly+ * what `refreshError` carries.+ */+function status(overrides: Partial<SwitchSetupStatus>): SwitchSetupStatus {+ return {+ agentId: 'claude',+ supported: true,+ installed: true,+ installedVersion: '0.1.0',+ latestVersion: '0.1.0',+ updateAvailable: false,+ refreshError: null,+ ...overrides,+ };+}+it('offers installed remote ACP providers without requiring connector plugins', async () => {+ mocks.listAgentTypeStatuses.mockResolvedValue([status({})]);mocks.probe.mockImplementation(async (id) => ({status: id === 'cursor' ? 'available' : 'missing',}));@@ -41,10 +63,12 @@ it('offers installed remote ACP providers without requiring connector plugins',blockedKind: null,});});+it('does not call a failed remote probe an installed CLI', async () => {+ mocks.listAgentTypeStatuses.mockResolvedValue([status({})]);mocks.probe.mockResolvedValue({ status: 'error' });const statuses = await switchSetupController.listAgentTypeAvailabilityRemote('example-host');- expect(statuses.find((status) => status.agentId === 'cursor')).toMatchObject({+ expect(statuses.find((s) => s.agentId === 'cursor')).toMatchObject({available: false,blockedReason: 'Could not verify this CLI on example-host. Recheck the host setup.',// Not 'not-installed': the host never answered, so nothing here says the@@ -52,3 +76,26 @@ it('does not call a failed remote probe an installed CLI', async () => {blockedKind: 'unknown',});});++/**+ * The driver keeps going when one agent type's status cannot be read, so the+ * row it produces reaches here with every field at its empty value and the+ * reason in `refreshError`. Branching on `installed` alone reads that as "the+ * connector is not installed", which is a statement of fact about the one thing+ * the read failed to establish — and it offers to install something that may+ * already be there.+ */+it('says a remote status could not be read rather than calling it not installed', async () => {+ mocks.listAgentTypeStatuses.mockResolvedValue([+ status({ installed: false, installedVersion: null, refreshError: 'ssh channel closed' }),+ ]);+ mocks.probe.mockResolvedValue({ status: 'missing' });+ const statuses = await switchSetupController.listAgentTypeAvailabilityRemote('example-host');+ expect(statuses.find((s) => s.agentId === 'claude')).toEqual({+ agentId: 'claude',+ available: false,+ blockedReason:+ 'Its Switch connector status could not be read on example-host: ssh channel closed',+ blockedKind: 'unknown',+ });+});diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.tsindex 1d179f3ba..388a68218 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/controller.ts@@ -64,6 +64,20 @@ export const switchSetupController = createRPCController({blockedKind: 'unsupported' as const,};}+ // A status that could not be read is not a status. `installed` is+ // false on such a row because it is false on every field of it, and+ // reporting that as "the connector is not installed" states as fact+ // the one thing the read failed to establish — then offers to install+ // something that may already be there. The local list says why; this+ // is the same answer for the same condition.+ if (status.refreshError !== null) {+ return {+ agentId: status.agentId,+ available: false,+ blockedReason: `Its Switch connector status could not be read on ${sshHost}: ${status.refreshError}`,+ blockedKind: 'unknown' as const,+ };+ }return status.installed? {agentId: status.agentId,@@ -83,6 +97,6 @@ export const switchSetupController = createRPCController({getStatus: (agentId: string) => switchSetupService.getStatus(agentId),checkForUpdates: (agentId: string) => switchSetupService.checkForUpdates(agentId),install: (agentId: string) => switchSetupService.install(agentId),- update: (agentId: string) => switchSetupService.update(agentId),+ update: (agentId: string) => switchSetupService.update(agentId, 'user'),uninstall: (agentId: string) => switchSetupService.uninstall(agentId),});diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.test.tsindex a3c26a433..da17bf6b3 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.test.ts@@ -35,6 +35,8 @@ vi.mock('@main/lib/logger', () => ({log: { warn: vi.fn(), info: vi.fn(), error: vi.fn() },}));+import { TransportError } from '@switch-console/core/exec';+import { aDurationMs } from '@tooling/utils/telemetry-duration';import { getRemoteSwitchSetupService } from './remote-switch-setup';const SSH_HOST = 'agent-host';@@ -291,9 +293,12 @@ describe('RemoteSwitchSetupService.getStatus', () => {});});- it('does not read manifests when the CLI already advertises versions', async () => {- // Claude Code reports them in `marketplace list --json`, so the fallback is- // dead weight there — two SSH round trips per status read.+ it('skips the marketplace manifests when the CLI already advertises versions', async () => {+ // Claude Code reports the advertised versions in `marketplace list --json`,+ // so that fallback is dead weight there — two SSH round trips per status+ // read. The installed version is still read from the plugin manifest, which+ // is the accurate source and the one the local driver uses: the CLI records+ // the version it installed and does not update it in place.mocks.getPlugin.mockReturnValue(CLAUDE_AGENT);mocks.resolveCommandPath.mockResolvedValue('/usr/bin/claude');mocks.exec.mockImplementation(claudeExecImpl('0.1.0', '0.2.0'));@@ -301,7 +306,11 @@ describe('RemoteSwitchSetupService.getStatus', () => {const service = await getRemoteSwitchSetupService(SSH_HOST);await service.getStatus('claude');- expect(calls()).toEqual(['plugin list --json', 'plugin marketplace list --json']);+ expect(calls()).toEqual([+ 'plugin list --json',+ '/home/dev/.claude/plugins/switch-connector/.claude-plugin/plugin.json',+ 'plugin marketplace list --json',+ ]);});});@@ -313,7 +322,7 @@ describe('RemoteSwitchSetupService.update', () => {mocks.exec.mockImplementation(codexExecImpl('sandbox-quantum/switch-legacy'));const service = await getRemoteSwitchSetupService(SSH_HOST);- const result = await service.update('codex');+ const result = await service.update('codex', 'user');expect(result.success).toBe(true);const seen = calls();@@ -334,7 +343,7 @@ describe('RemoteSwitchSetupService.update', () => {});const service = await getRemoteSwitchSetupService(SSH_HOST);- const result = await service.update('codex');+ const result = await service.update('codex', 'user');expect(result.success).toBe(false);expect(result.message).toMatch(/Could not add marketplace/);@@ -361,7 +370,7 @@ describe('RemoteSwitchSetupService.update', () => {mocks.exec.mockImplementation(codexExecImpl('sandbox-quantum/switch'));const service = await getRemoteSwitchSetupService(SSH_HOST);- const result = await service.update('codex');+ const result = await service.update('codex', 'user');expect(result.success).toBe(true);// The marketplace is repaired first: the re-add resolves against it, so a@@ -384,7 +393,7 @@ describe('RemoteSwitchSetupService.update', () => {});const service = await getRemoteSwitchSetupService(SSH_HOST);- const result = await service.update('codex');+ const result = await service.update('codex', 'user');expect(calls().slice(-2)).toEqual([`plugin remove ${CODEX_REF}`, `plugin add ${CODEX_REF}`]);expect(result).toEqual({@@ -393,6 +402,216 @@ describe('RemoteSwitchSetupService.update', () => {'Update failed: the plugin was removed but could not be reinstalled. Install it again for this host.',});});++ /**+ * The reinstall split, on the driver where it was not pinned.+ *+ * Both halves of a Codex update carry `was_reinstall: true`, so the flag alone+ * cannot tell "nothing changed" from "the host now has no connector". The+ * failure code is what separates them, and it has to be asserted on this+ * driver too — the two are maintained by hand and have drifted before.+ */+ it('reports the removal half of a failed reinstall', async () => {+ mocks.exec.mockImplementation((_bin: string, args: string[] = []) => {+ if (args.join(' ') === `plugin remove ${CODEX_REF}`) {+ return Promise.reject(Object.assign(new Error('exit 1'), { code: 1, stderr: 'locked' }));+ }+ return Promise.resolve({ stdout: '', stderr: '' });+ });++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ await service.update('codex', 'user');++ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_updated', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ was_reinstall: true,+ trigger: 'user',+ failure_reason: 'uninstall_command_failed',+ duration_ms: aDurationMs,+ });+ });++ it('reports the re-add half of a failed reinstall', async () => {+ mocks.exec.mockImplementation((_bin: string, args: string[] = []) => {+ if (args.join(' ') === `plugin add ${CODEX_REF}`) {+ return Promise.reject(Object.assign(new Error('exit 1'), { code: 1, stderr: '' }));+ }+ return Promise.resolve({ stdout: '', stderr: '' });+ });++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ await service.update('codex', 'user');++ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_updated', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ was_reinstall: true,+ trigger: 'user',+ failure_reason: 'install_command_failed',+ duration_ms: aDurationMs,+ });+ });++ it('reports a successful reinstall-style update', async () => {+ mocks.exec.mockImplementation(codexExecImpl('sandbox-quantum/switch'));++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ await service.update('codex', 'user');++ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_updated', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'success',+ was_reinstall: true,+ trigger: 'user',+ failure_reason: 'none',+ duration_ms: aDurationMs,+ });+ });+});++/**+ * An SSH channel that dies mid-operation is the most common way a remote+ * connector operation fails, and it arrives as a rejection rather than a result:+ * `resolveCommandPath` re-throws a transport failure by design, because it is+ * not evidence the binary is absent. Unreported, it is an attempt the user made+ * that no event records, and a stack in the renderer instead of a message.+ */+describe('RemoteSwitchSetupService transport failures', () => {+ beforeEach(() => {+ mocks.resolveCommandPath.mockRejectedValue(new Error('ssh channel closed'));+ });++ it('reports an install that threw rather than letting it escape', async () => {+ const service = await getRemoteSwitchSetupService(SSH_HOST);+ const result = await service.install('codex');++ expect(result).toEqual({ success: false, message: 'ssh channel closed' });+ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ failure_reason: 'error',+ duration_ms: aDurationMs,+ });+ });++ it('reports an update that threw, with nothing removed', async () => {+ const service = await getRemoteSwitchSetupService(SSH_HOST);+ const result = await service.update('codex', 'user');++ expect(result.success).toBe(false);+ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_updated', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ // Resolution throws before anything is removed, so the pair is not+ // counted as a half-finished reinstall.+ was_reinstall: false,+ trigger: 'user',+ failure_reason: 'error',+ duration_ms: aDurationMs,+ });+ });+});++describe('RemoteSwitchSetupService when the channel drops mid-operation', () => {+ function dropsOn(command: string) {+ return (_bin: string, args: string[] = []) =>+ args.join(' ') === command+ ? Promise.reject(new TransportError('SSH transport failure: channel closed'))+ : Promise.resolve({ stdout: '', stderr: '' });+ }++ it('does not blame the marketplace for a dead channel', async () => {+ mocks.exec.mockImplementation(dropsOn('plugin marketplace list --json'));++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ await service.install('codex');++ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ failure_reason: 'error',+ duration_ms: aDurationMs,+ });+ });++ it('says the plugin was removed when the channel drops before the re-add', async () => {+ mocks.exec.mockImplementation(dropsOn(`plugin add ${CODEX_REF}`));++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ const result = await service.update('codex', 'user');++ expect(calls().slice(-2)).toEqual([`plugin remove ${CODEX_REF}`, `plugin add ${CODEX_REF}`]);+ expect(result).toEqual({+ success: false,+ message:+ 'Update failed: the plugin was removed, then the connection to the host dropped before it could be reinstalled. Reconnect and install it again for this host.',+ });+ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_updated', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ was_reinstall: true,+ trigger: 'user',+ failure_reason: 'error',+ duration_ms: aDurationMs,+ });+ });+});++/**+ * A host that simply does not have the agent installed.+ *+ * Told apart from a transport failure by the shell's own answer: 127 is "no such+ * command", which the driver already recognises well enough to keep it out of+ * the warning log. Every command a marketplace-driven connector runs goes+ * through that binary, so without a code of its own the whole condition lands+ * under whichever verb ran first — `marketplace_failed` on an install, which+ * reads as a broken plugin source on a host whose only problem is that nobody+ * installed the agent.+ */+describe('RemoteSwitchSetupService when the agent CLI is not on the host', () => {+ beforeEach(() => {+ mocks.resolveCommandPath.mockResolvedValue(null);+ mocks.exec.mockImplementation(() =>+ Promise.reject(Object.assign(new Error('codex: command not found'), { code: 127 }))+ );+ });++ it('says so on install', async () => {+ const service = await getRemoteSwitchSetupService(SSH_HOST);+ await service.install('codex');++ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ failure_reason: 'host_cli_missing',+ duration_ms: aDurationMs,+ });+ });++ it('says so on update', async () => {+ const service = await getRemoteSwitchSetupService(SSH_HOST);+ await service.update('codex', 'user');++ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_updated', {+ agent_type: 'codex',+ target: 'remote',+ outcome: 'failure',+ // Nothing was removed, because nothing could run at all.+ was_reinstall: false,+ trigger: 'user',+ failure_reason: 'host_cli_missing',+ duration_ms: aDurationMs,+ });+ });});describe('RemoteSwitchSetupService.install', () => {@@ -408,6 +627,9 @@ describe('RemoteSwitchSetupService.install', () => {agent_type: 'codex',target: 'remote',outcome: 'success',+ failure_reason: 'none',+ // Elapsed wall time: a real number, but not one a test can pin.+ duration_ms: aDurationMs,});});@@ -430,6 +652,8 @@ describe('RemoteSwitchSetupService.install', () => {agent_type: 'codex',target: 'remote',outcome: 'failure',+ failure_reason: 'install_command_failed',+ duration_ms: aDurationMs,});});@@ -489,6 +713,8 @@ describe('RemoteSwitchSetupService.install', () => {agent_type: 'opencode',target: 'remote',outcome: 'success',+ failure_reason: 'none',+ duration_ms: aDurationMs,});});@@ -503,6 +729,8 @@ describe('RemoteSwitchSetupService.install', () => {agent_type: 'opencode',target: 'remote',outcome: 'failure',+ failure_reason: 'files_write_failed',+ duration_ms: aDurationMs,});});@@ -519,10 +747,14 @@ describe('RemoteSwitchSetupService.install', () => {const result = await service.install('opencode');expect(result.success).toBe(false);+ // Its own code, not `files_write_failed`: a fault in the plugin rather+ // than on the host, and the two would otherwise be one number.expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {agent_type: 'opencode',target: 'remote',outcome: 'failure',+ failure_reason: 'files_unimplemented',+ duration_ms: aDurationMs,});});});@@ -551,3 +783,57 @@ describe('RemoteSwitchSetupService.checkForUpdates', () => {expect(calls()).toContain('plugin marketplace add sandbox-quantum/switch');});});++/**+ * A broken pipe is not an answer about the connector.+ *+ * Every other failure in this driver folds into a `ConnectorRunResult`, and a+ * status read parses the empty stdout as "no plugin installed" — so a channel+ * dying mid-fan-out would render in the picker as "Its Switch connector is not+ * installed on <host>", with an Install button, for a read that never happened.+ * Reporting `refreshError` only helps if the driver raises rather than+ * inventing an empty plugin list.+ */+describe('RemoteSwitchSetupService when the transport dies', () => {+ function transportFailure() {+ return () => Promise.reject(new TransportError('SSH transport failure: not available'));+ }++ it('does not report a dead channel as a connector that is not installed', async () => {+ mocks.getPlugin.mockReturnValue(CLAUDE_AGENT);+ mocks.resolveCommandPath.mockResolvedValue('/usr/bin/claude');+ mocks.exec.mockImplementation(transportFailure());++ const service = await getRemoteSwitchSetupService(SSH_HOST);++ await expect(service.getStatus('claude')).rejects.toBeInstanceOf(TransportError);+ });++ it('still reads a shell that answered 127 as the connector being absent', async () => {+ // The distinction the re-raise has to preserve: a host without Codex is a+ // normal host, and exit 127 is that host answering — not the pipe failing.+ mocks.getPlugin.mockReturnValue(CODEX_AGENT);+ mocks.resolveCommandPath.mockResolvedValue('/usr/bin/codex');+ mocks.exec.mockImplementation(() =>+ Promise.reject(Object.assign(new Error('codex: command not found'), { code: 127 }))+ );++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ const status = await service.getStatus('codex');++ expect(status).toMatchObject({ supported: true, installed: false, refreshError: null });+ });++ it('turns the raise into a row that says why, rather than dropping it', async () => {+ mocks.listPlugins.mockReturnValue([CLAUDE_AGENT]);+ mocks.getPlugin.mockReturnValue(CLAUDE_AGENT);+ mocks.resolveCommandPath.mockResolvedValue('/usr/bin/claude');+ mocks.exec.mockImplementation(transportFailure());++ const service = await getRemoteSwitchSetupService(SSH_HOST);+ const [status] = await service.listAgentTypeStatuses();++ expect(status).toMatchObject({ agentId: 'claude', supported: true, installed: false });+ expect(status!.refreshError).toMatch(/transport failure/i);+ });+});diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.tsindex 8c1d2da1f..e0cb31388 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/remote-switch-setup.ts@@ -1,21 +1,45 @@import type { ISwitchSetupFilesBehavior, PluginFs } from '@switch-console/core/agents/plugins';import { resolveCommandPath } from '@switch-console/core/deps/runtime';+import { TransportError } from '@switch-console/core/exec';import { type ArtifactName, artifactVersion } from '@switch-console/shared';import { createRemoteHomePluginFs } from '@main/core/agent-runtime/impl/remote-home-plugin-fs';import { SshExecutionContext } from '@main/core/execution-context/ssh-execution-context';import { sshConnectionIdForHost } from '@main/core/locations/location-transport';import { ensureSshConnected } from '@main/core/ssh/connect/connect-agent-ssh';import { agentTypeOf } from '@main/core/telemetry/agent-type';+import { startTimer } from '@main/core/telemetry/duration';+import type {+ TelemetryConnectorFailure,+ TelemetryConnectorUpdateTrigger,+} from '@main/core/telemetry/events';import { trackEvent } from '@main/core/telemetry/telemetry-service';import { log } from '@main/lib/logger';import { isNewerVersion } from '@main/lib/semver';-import { isValidProviderId } from '@shared/core/providers/agent-provider-registry';import { getPlugin, listPlugins } from '../providers/plugin-registry';-import { cliRulesFor, type SwitchSetupCliRules } from './switch-setup-cli-dialect';-import type { SwitchSetupResult, SwitchSetupStatus } from './switch-setup-service';-import { marketplaceMatchesSource } from './switch-setup-service';--const EXEC_TIMEOUT_MS = 120_000;+import {+ commandFailureCode,+ CONNECTOR_UNSUPPORTED_RESULT,+ type ConnectorRun,+ connectorFailed,+ type ConnectorRunResult,+ connectorSucceeded,+ connectorUnsupported,+ declaresNoConnector,+ EXEC_TIMEOUT_MS,+ FilesConnectorUnimplementedError,+ HostCliMissingError,+ marketplaceFailed,+ marketplaceMatchesSource,+ runReportedOperation,+ type SwitchSetupResult,+ type SwitchSetupStatus,+ unsupportedStatus,+} from './connector-run';+import {+ cliRulesFor,+ type InstalledPlugin,+ type SwitchSetupCliRules,+} from './switch-setup-cli-dialect';/** POSIX shells use 127 for "command not found". */const COMMAND_NOT_FOUND = 127;@@ -24,17 +48,16 @@ const COMMAND_NOT_FOUND = 127;* The version stamped into a file-based connector install — the connector's own* artifact version, on a remote host exactly as locally, so the two report the* same number for the same connector.+ *+ * Takes the artifact rather than the agent id, matching the local helper of the+ * same name: two functions that share a name, both taking a `string`, and+ * disagreeing about which string is how a call site copied between the drivers+ * compiles and silently versions the wrong thing.*/-function connectorVersion(agentId: string): string {- const descriptor = getPlugin(agentId).capabilities.switchSetup;- if (descriptor.kind !== 'files') {- throw new Error(`Agent '${agentId}' has no file-based Switch connector to version.`);- }- return artifactVersion(descriptor.artifact as ArtifactName);+function connectorVersion(artifact: string): string {+ return artifactVersion(artifact as ArtifactName);}-type RunResult = { code: number; stdout: string; stderr: string };-/*** Join path segments for the remote host, which is POSIX regardless of what* Switch Console is running on. `node:path`'s `join` would emit backslashes on a@@ -49,18 +72,6 @@ function posixJoin(...segments: string[]): string {.join('/');}-function unsupported(agentId: string): SwitchSetupStatus {- return {- agentId,- supported: false,- installed: false,- installedVersion: null,- latestVersion: null,- updateAvailable: false,- refreshError: null,- };-}-/*** Parse JSON from remote CLI stdout that may be wrapped in login-shell noise* (MOTD, profile banners, tunnel warnings, trailing prompt text). The remote@@ -110,18 +121,26 @@ export class RemoteSwitchSetupService {/*** The `files` connector behavior for an agent whose connector Switch Console- * writes itself, rooted at the remote host's home. Null for any other agent.+ * writes itself. Mirrors the local driver: callers reach it only under a+ * `kind === 'files'` test, a plugin that declares files and implements none is+ * the one expected failure, and no filesystem is built here — on this side+ * that matters more, because building one can fail on a dead SSH channel and+ * that is not a fault in the plugin.*/- private resolveFiles(agentId: string) {+ private resolveFiles(agentId: string): { files: ISwitchSetupFilesBehavior; version: string } {const plugin = getPlugin(agentId);- if (plugin.capabilities.switchSetup.kind !== 'files') return null;- const files = plugin.behavior.switchSetup?.files;- if (!files) {- throw new Error(- `Agent '${agentId}' declares a file-based Switch connector but implements no behavior for it.`- );+ const descriptor = plugin.capabilities.switchSetup;+ if (descriptor.kind !== 'files') {+ throw new Error(`Agent '${agentId}' has no file-based Switch connector.`);}- return { files, homeFs: createRemoteHomePluginFs(this.ctx) };+ const files = plugin.behavior.switchSetup?.files;+ if (!files) throw new FilesConnectorUnimplementedError(agentId);+ return { files, version: connectorVersion(descriptor.artifact) };+ }++ /** The filesystem a file-based connector is written to on this host. */+ private homeFs(): PluginFs {+ return createRemoteHomePluginFs(this.ctx);}private async resolve(agentId: string) {@@ -141,16 +160,31 @@ export class RemoteSwitchSetupService {};}- private async run(bin: string, args: string[]): Promise<RunResult> {+ private async run(bin: string, args: string[]): Promise<ConnectorRunResult> {try {const { stdout, stderr } = await this.ctx.exec(bin, args, { timeout: EXEC_TIMEOUT_MS });- return { code: 0, stdout, stderr };+ return { code: 0, stdout, stderr, notFound: false };} catch (err: unknown) {+ // A broken pipe is not an answer about the connector. Everything below+ // turns a failure into a `ConnectorRunResult`, and a status read then+ // parses the empty stdout as "no plugin installed" — so a dead channel+ // renders as "its Switch connector is not installed on <host>", which is+ // a confident statement about the one thing the read failed to+ // establish. `TransportError` exists to draw exactly this line+ // ("probe-style callers can distinguish it from 'the command ran and+ // failed'"), so it is re-raised: `listAgentTypeStatuses` catches it and+ // the row says it could not be read.+ //+ // Only the transport. A shell exiting 127 IS an answer — a host without+ // Codex is a normal host — and still comes back as a result below.+ if (err instanceof TransportError) throw err;const e = err as { stdout?: string; stderr?: string; code?: number; message?: string };+ const code = e.code ?? 1;const result = {- code: e.code ?? 1,+ code,stdout: e.stdout ?? '',stderr: e.stderr ?? e.message ?? '',+ notFound: code === COMMAND_NOT_FOUND,};// A shell reports 127 when the binary is not on PATH. For "is this agent// type's connector installed?" that is the answer, not a fault: a host@@ -177,6 +211,30 @@ export class RemoteSwitchSetupService {return rules.parsePluginList(parseJsonLoose(stdout)).find((p) => p.ref === ref) ?? null;}+ /**+ * Read the true installed version from the plugin manifest on the host,+ * falling back to the CLI's.+ *+ * The same two sources the local driver reads, in the same order, and for the+ * same reason: the CLI reports the version it recorded when it installed the+ * plugin, which is not updated in place, so the manifest is the accurate one.+ * This feeds `updateAvailable`, so a driver preferring the CLI's number offers+ * updates the other would not.+ */+ private async installedVersion(+ entry: InstalledPlugin | null,+ rules: SwitchSetupCliRules+ ): Promise<string | null> {+ if (!entry) return null;+ if (entry.manifestPath) {+ const manifest = await this.readRemoteJson<{ version?: string }>(+ posixJoin(entry.manifestPath, rules.pluginManifestDir, 'plugin.json')+ );+ if (manifest?.version) return manifest.version;+ }+ return entry.version ?? null;+ }+/*** The version the marketplace advertises, or null when it genuinely cannot be* determined — which callers must not read as "up to date".@@ -239,9 +297,12 @@ export class RemoteSwitchSetupService {marketplaceSource: string,rules: SwitchSetupCliRules): Promise<void> {- const { stdout } = await this.run(bin, ['plugin', 'marketplace', 'list', '--json']);+ const listed = await this.run(bin, ['plugin', 'marketplace', 'list', '--json']);+ // No binary on the host means no marketplace to repair and no plugin to+ // install; every command after this one would fail the same way.+ if (listed.notFound) throw new HostCliMissingError(bin);const existing = rules- .parseMarketplaceList(parseJsonLoose(stdout))+ .parseMarketplaceList(parseJsonLoose(listed.stdout)).find((m) => m.name === marketplaceName);if (existing) {if (marketplaceMatchesSource(existing, marketplaceSource)) return;@@ -269,10 +330,9 @@ export class RemoteSwitchSetupService {* there is, and an install stamped with an older one is what "update* available" means.*/- private async filesStatus(agentId: string, version: string): Promise<SwitchSetupStatus> {- const resolved = this.resolveFiles(agentId);- if (!resolved) return unsupported(agentId);- const installedVersion = await resolved.files.installedVersion(resolved.homeFs);+ private async filesStatus(agentId: string): Promise<SwitchSetupStatus> {+ const { files, version } = this.resolveFiles(agentId);+ const installedVersion = await files.installedVersion(this.homeFs());return {agentId,supported: true,@@ -284,33 +344,46 @@ export class RemoteSwitchSetupService {};}- /** Install, update and uninstall for a file-based connector on this host. */+ /**+ * Install, update and uninstall for a file-based connector on this host.+ * Classifies exactly as the local driver does — see the note there.+ */private async runFiles(agentId: string,- action: (files: ISwitchSetupFilesBehavior, homeFs: PluginFs) => Promise<unknown>- ): Promise<SwitchSetupResult> {- const resolved = this.resolveFiles(agentId);- if (!resolved)- return { success: false, message: 'Switch setup is not supported for this agent.' };+ failure: TelemetryConnectorFailure,+ action: (+ files: ISwitchSetupFilesBehavior,+ homeFs: PluginFs,+ version: string+ ) => Promise<unknown>+ ): Promise<ConnectorRun> {+ let resolved: { files: ISwitchSetupFilesBehavior; version: string };try {- await action(resolved.files, resolved.homeFs);- return { success: true };+ resolved = this.resolveFiles(agentId);+ } catch (err) {+ if (!(err instanceof FilesConnectorUnimplementedError)) throw err;+ log.error('remote-switch-setup: file-based connector declares no behavior', { agentId, err });+ return connectorFailed(err.message, 'files_unimplemented');+ }+ try {+ await action(resolved.files, this.homeFs(), resolved.version);+ return connectorSucceeded();} catch (err) {log.error('remote-switch-setup: file-based connector operation failed', { agentId, err });- return { success: false, message: err instanceof Error ? err.message : String(err) };+ return connectorFailed(err instanceof Error ? err.message : String(err), failure);}}async getStatus(agentId: string): Promise<SwitchSetupStatus> {if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- return this.filesStatus(agentId, connectorVersion(agentId));+ return this.filesStatus(agentId);}const resolved = await this.resolve(agentId);- if (!resolved) return unsupported(agentId);+ if (!resolved) return unsupportedStatus(agentId);const { descriptor, bin, ref, rules } = resolved;const entry = await this.findInstalled(bin, ref, rules);- const installedVersion = entry?.version ?? null;+ const installedVersion = await this.installedVersion(entry, rules);const latestVersion = await this.advertisedVersion(bin,descriptor.marketplaceName,@@ -342,10 +415,10 @@ export class RemoteSwitchSetupService {async checkForUpdates(agentId: string): Promise<SwitchSetupStatus> {// A file-based connector ships inside the app: nothing to refresh.if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- return this.filesStatus(agentId, connectorVersion(agentId));+ return this.filesStatus(agentId);}const resolved = await this.resolve(agentId);- if (!resolved) return unsupported(agentId);+ if (!resolved) return unsupportedStatus(agentId);const { descriptor, bin, marketplaceSource, rules } = resolved;let refreshError: string | null = null;try {@@ -363,61 +436,51 @@ export class RemoteSwitchSetupService {return { ...(await this.getStatus(agentId)), refreshError };}+ /**+ * Install the connector on this host, reporting the outcome. The `none` guard+ * and the reporting of `unsupported` mirror the local driver — both now come+ * from the shared leaf rather than from two copies kept in step by hand.+ */async install(agentId: string): Promise<SwitchSetupResult> {- const { result, attempted } = await this.runInstall(agentId);- // An agent type with no connector to install did not fail to install one.- if (attempted) {- trackEvent('connector_installed', {- agent_type: isValidProviderId(agentId) ? agentId : 'unknown',- target: 'remote',- outcome: result.success ? 'success' : 'failure',- });- }- return result;+ if (declaresNoConnector(agentId)) return CONNECTOR_UNSUPPORTED_RESULT;+ const elapsed = startTimer();+ const run = await runReportedOperation('remote-switch-setup', agentId, () =>+ this.runInstall(agentId)+ );+ trackEvent('connector_installed', {+ agent_type: agentTypeOf(agentId),+ target: 'remote',+ outcome: run.result.success ? 'success' : 'failure',+ failure_reason: run.failure,+ duration_ms: elapsed(),+ });+ return run.result;}- private async runInstall(- agentId: string- ): Promise<{ result: SwitchSetupResult; attempted: boolean }> {+ private async runInstall(agentId: string): Promise<ConnectorRun> {if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- // `runFiles` resolves the behavior outside its own try, and that throws- // for a connector that declares files and implements none. An install the- // user asked for must come back as a failed result either way — a- // rejection here would skip the report and reach the UI as a stack.- try {- const version = connectorVersion(agentId);- const result = await this.runFiles(agentId, (files, fs) => files.install(fs, { version }));- return { result, attempted: true };- } catch (err) {- return {- result: { success: false, message: String(err) },- attempted: true,- };- }+ return this.runFiles(agentId, 'files_write_failed', (files, fs, version) =>+ files.install(fs, { version })+ );}const resolved = await this.resolve(agentId);- if (!resolved)- return {- result: { success: false, message: 'Switch setup is not supported for this agent.' },- attempted: false,- };+ if (!resolved) return connectorUnsupported();const { descriptor, bin, ref, marketplaceSource, rules } = resolved;try {await this.ensureMarketplace(bin, descriptor.marketplaceName, marketplaceSource, rules);} catch (err) {- return {- result: { success: false, message: `Could not add marketplace: ${String(err)}` },- attempted: true,- };+ // A dead channel says nothing about the marketplace; `runReportedOperation`+ // reports it as `error`.+ if (err instanceof TransportError) throw err;+ return marketplaceFailed(err, `Could not add marketplace: ${String(err)}`);}const res = await this.run(bin, rules.installArgs(ref, descriptor.scope));- return {- result:- res.code === 0- ? { success: true }- : { success: false, message: res.stderr.trim() || 'Install failed.' },- attempted: true,- };+ return res.code === 0+ ? connectorSucceeded()+ : connectorFailed(+ res.stderr.trim() || 'Install failed.',+ commandFailureCode(res, 'install_command_failed')+ );}/**@@ -425,44 +488,57 @@ export class RemoteSwitchSetupService {* below resolves against whatever marketplace is registered, so a stale source* would otherwise fail it after the uninstall has already succeeded.*/- async update(agentId: string): Promise<SwitchSetupResult> {- if (getPlugin(agentId).capabilities.switchSetup.kind === 'none') {- return { success: false, message: 'Switch setup is not supported for this agent.' };- }- const { result, wasReinstall } = await this.runUpdate(agentId);+ async update(+ agentId: string,+ trigger: TelemetryConnectorUpdateTrigger+ ): Promise<SwitchSetupResult> {+ if (declaresNoConnector(agentId)) return CONNECTOR_UNSUPPORTED_RESULT;+ const elapsed = startTimer();+ const { run, wasReinstall } = await this.runUpdateReported(agentId);trackEvent('connector_updated', {agent_type: agentTypeOf(agentId),target: 'remote',- outcome: result.success ? 'success' : 'failure',+ outcome: run.result.success ? 'success' : 'failure',was_reinstall: wasReinstall,+ trigger,+ failure_reason: run.failure,+ duration_ms: elapsed(),});- return result;+ return run.result;}- private async runUpdate(+ /** `runUpdate` that always comes back as a result — see the local driver. */+ private async runUpdateReported(agentId: string- ): Promise<{ result: SwitchSetupResult; wasReinstall: boolean }> {+ ): Promise<{ run: ConnectorRun; wasReinstall: boolean }> {+ let wasReinstall = false;+ const run = await runReportedOperation('remote-switch-setup', agentId, async () => {+ const outcome = await this.runUpdate(agentId);+ wasReinstall = outcome.wasReinstall;+ return outcome.run;+ });+ return { run, wasReinstall };+ }++ private async runUpdate(agentId: string): Promise<{ run: ConnectorRun; wasReinstall: boolean }> {// Installing overwrites in place, so update is the same operation — there// is no removed-but-not-reinstalled window to report on.if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- const version = connectorVersion(agentId);- const result = await this.runFiles(agentId, (files, fs) => files.install(fs, { version }));- return { result, wasReinstall: false };+ const run = await this.runFiles(agentId, 'files_write_failed', (files, fs, version) =>+ files.install(fs, { version })+ );+ return { run, wasReinstall: false };}const resolved = await this.resolve(agentId);- if (!resolved) {- return {- result: { success: false, message: 'Switch setup is not supported for this agent.' },- wasReinstall: false,- };- }+ if (!resolved) return { run: connectorUnsupported(), wasReinstall: false };const { descriptor, bin, ref, marketplaceSource, rules } = resolved;try {await this.ensureMarketplace(bin, descriptor.marketplaceName, marketplaceSource, rules);} catch (err) {+ if (err instanceof TransportError) throw err;return {- result: { success: false, message: `Could not add marketplace: ${String(err)}` },+ run: marketplaceFailed(err, `Could not add marketplace: ${String(err)}`),wasReinstall: false,};}@@ -471,10 +547,13 @@ export class RemoteSwitchSetupService {if (updateArgs) {const res = await this.run(bin, updateArgs);return {- result:+ run:res.code === 0- ? { success: true }- : { success: false, message: res.stderr.trim() || 'Update failed.' },+ ? connectorSucceeded()+ : connectorFailed(+ res.stderr.trim() || 'Update failed.',+ commandFailureCode(res, 'update_command_failed')+ ),wasReinstall: false,};}@@ -484,24 +563,41 @@ export class RemoteSwitchSetupService {const removed = await this.run(bin, rules.uninstallArgs(ref, descriptor.scope));if (removed.code !== 0) {return {- result: {- success: false,- message: removed.stderr.trim() || 'Update failed: could not remove the installed plugin.',- },+ run: connectorFailed(+ removed.stderr.trim() || 'Update failed: could not remove the installed plugin.',+ commandFailureCode(removed, 'uninstall_command_failed')+ ),+ wasReinstall: true,+ };+ }+ let added: ConnectorRunResult;+ try {+ added = await this.run(bin, rules.installArgs(ref, descriptor.scope));+ } catch (err) {+ // The remove already went through, so this cannot escape as a plain+ // `error`: the host may now have no connector, and the caller has to say so.+ if (!(err instanceof TransportError)) throw err;+ log.error('remote-switch-setup: connection dropped between remove and re-add', {+ agentId,+ err,+ });+ return {+ run: connectorFailed(+ 'Update failed: the plugin was removed, then the connection to the host dropped before it could be reinstalled. Reconnect and install it again for this host.',+ 'error'+ ),wasReinstall: true,};}- const added = await this.run(bin, rules.installArgs(ref, descriptor.scope));return {- result:+ run:added.code === 0- ? { success: true }- : {- success: false,- message:- added.stderr.trim() ||+ ? connectorSucceeded()+ : connectorFailed(+ added.stderr.trim() ||'Update failed: the plugin was removed but could not be reinstalled. Install it again for this host.',- },+ commandFailureCode(added, 'install_command_failed')+ ),wasReinstall: true,};}@@ -511,7 +607,22 @@ export class RemoteSwitchSetupService {const statuses: SwitchSetupStatus[] = [];for (const plugin of listPlugins()) {if (plugin.capabilities.switchSetup.kind === 'none') continue;- statuses.push(await this.getStatus(plugin.metadata.id));+ const agentId = plugin.metadata.id;+ // One agent type whose status cannot be read must not empty the panel.+ // Such a row has every field at its empty value, so a consumer branching+ // on `installed` alone reports "the connector is not installed" — a+ // statement about the one thing the read failed to establish. Check+ // `refreshError` first.+ try {+ statuses.push(await this.getStatus(agentId));+ } catch (err) {+ log.error('remote-switch-setup: could not read connector status', { agentId, err });+ statuses.push({+ ...unsupportedStatus(agentId),+ supported: true,+ refreshError: err instanceof Error ? err.message : String(err),+ });+ }}return statuses;}diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.test.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.test.tsindex ef80125ee..3fbc1fd81 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.test.ts@@ -7,8 +7,11 @@ const mocks = vi.hoisted(() => ({listPlugins: vi.fn(),readFile: vi.fn(),trackEvent: vi.fn(),+ createPluginFs: vi.fn(),}));+vi.mock('../providers/plugin-fs', () => ({ createPluginFs: mocks.createPluginFs }));+vi.mock('@main/core/execution-context/local-execution-context', () => ({LocalExecutionContext: class {exec = mocks.exec;@@ -36,6 +39,7 @@ vi.mock('@main/lib/logger', () => ({}));import { ARTIFACT_VERSIONS } from '@switch-console/shared';+import { aDurationMs } from '@tooling/utils/telemetry-duration';import { switchSetupService } from './switch-setup-service';const CLI_AGENT = {@@ -336,6 +340,43 @@ describe('switchSetupService.listAgentTypeAvailability', () => {expect(availability[0]!.blockedReason).toBeTruthy();});+ /**+ * One broken plugin must not empty the list.+ *+ * This fans out over every agent type, so a status read that throws took the+ * whole picker down with it — nothing rendered at all, which is a worse answer+ * than the one unavailable row it should have been. The same condition the+ * mutating paths report as `files_unimplemented` is the one that did it.+ */+ it('keeps the other types when one plugin cannot be read at all', async () => {+ const BROKEN_FILES_AGENT = {+ metadata: { id: 'opencode' },+ capabilities: {+ switchSetup: { kind: 'files', artifact: 'switch-connector-opencode' },+ hostDependency: { binaryNames: ['opencode'] },+ },+ behavior: { switchSetup: {} },+ };+ mocks.listPlugins.mockReturnValue([CLI_AGENT, BROKEN_FILES_AGENT]);+ mocks.getPlugin.mockImplementation((id: string) =>+ id === 'opencode' ? BROKEN_FILES_AGENT : CLI_AGENT+ );+ mocks.exec.mockImplementation(execImpl('0.1.0'));+ mocks.readFile.mockImplementation(readFileImpl('0.1.0', '0.1.0'));++ const availability = await switchSetupService.listAgentTypeAvailability();++ expect(availability).toEqual([+ { agentId: 'claude', available: true, blockedReason: null, blockedKind: null },+ {+ agentId: 'opencode',+ available: false,+ blockedReason: expect.stringContaining('implements no behavior'),+ blockedKind: 'unknown',+ },+ ]);+ });+it('never lists an agent type that declares no Switch setup', async () => {// `NONE_AGENT` cannot be onboarded at all, so it is not a thing the user// could fix — listing it greyed out would be noise, not information.@@ -449,6 +490,9 @@ describe('switchSetupService mutations', () => {agent_type: 'claude',target: 'local',outcome: 'success',+ failure_reason: 'none',+ // Elapsed wall time: a real number, but not one a test can pin.+ duration_ms: aDurationMs,});});@@ -490,14 +534,20 @@ describe('switchSetupService mutations', () => {expect(result.success).toBe(false);expect(result.message).toBe('no write access');+ // The host CLI refused the plugin — distinct from the marketplace failing+ // to register, which is the other way this same button fails.expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {agent_type: 'claude',target: 'local',outcome: 'failure',+ failure_reason: 'install_command_failed',+ duration_ms: aDurationMs,});});it('reports nothing when the agent type has no Switch setup to attempt', async () => {+ // Not a failed install — there was never one to attempt. Reporting it would+ // put every agent type in the app into the connector failure rate.mocks.getPlugin.mockReturnValue(NONE_AGENT);const result = await switchSetupService.install('no-switch-agent');@@ -508,6 +558,29 @@ describe('switchSetupService mutations', () => {});expect(mocks.trackEvent).not.toHaveBeenCalled();});++ it('reports `unsupported` when a declared connector has no binary to drive it', async () => {+ // The other way an install ends with nothing installed, and this one IS a+ // failure of what the user asked for: the agent type declares a CLI+ // connector and the host binary cannot be resolved. It used to be silenced+ // by the same flag as the case above while `update` and `uninstall`+ // reported it, so the condition looked like it only happened on update.+ mocks.getPlugin.mockReturnValue({+ ...CLI_AGENT,+ capabilities: { ...CLI_AGENT.capabilities, hostDependency: { binaryNames: [] } },+ });++ const result = await switchSetupService.install('claude');++ expect(result.success).toBe(false);+ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {+ agent_type: 'claude',+ target: 'local',+ outcome: 'failure',+ failure_reason: 'unsupported',+ duration_ms: aDurationMs,+ });+ });});describe('switchSetupService with the codex dialect', () => {@@ -630,7 +703,7 @@ describe('switchSetupService with the codex dialect', () => {return base(bin, args);});- const result = await switchSetupService.update('codex');+ const result = await switchSetupService.update('codex', 'user');expect(result.success).toBe(true);const seen = calls();@@ -652,7 +725,7 @@ describe('switchSetupService with the codex dialect', () => {return Promise.resolve({ stdout: '', stderr: '' });});- const result = await switchSetupService.update('codex');+ const result = await switchSetupService.update('codex', 'user');expect(result.success).toBe(false);expect(result.message).toMatch(/Could not add marketplace/);@@ -679,7 +752,7 @@ describe('switchSetupService with the codex dialect', () => {it('updates by removing then re-adding, in that order', async () => {mocks.exec.mockImplementation(codexExecImpl('0.1.0'));- const result = await switchSetupService.update('codex');+ const result = await switchSetupService.update('codex', 'user');expect(result.success).toBe(true);// The marketplace is repaired first: the re-add resolves against it, so a@@ -701,7 +774,7 @@ describe('switchSetupService with the codex dialect', () => {return Promise.resolve({ stdout: '', stderr: '' });});- const result = await switchSetupService.update('codex');+ const result = await switchSetupService.update('codex', 'user');expect(calls().slice(-2)).toEqual([`plugin remove ${CODEX_REF}`, `plugin add ${CODEX_REF}`]);expect(result).toEqual({@@ -715,6 +788,7 @@ describe('switchSetupService with the codex dialect', () => {describe('file-based connector version', () => {const installedVersion = vi.fn();const install = vi.fn();+ const uninstall = vi.fn();const FILES_AGENT = {metadata: { id: 'opencode' },capabilities: {@@ -725,12 +799,13 @@ describe('file-based connector version', () => {},hostDependency: { binaryNames: ['opencode'] },},- behavior: { switchSetup: { files: { installedVersion, install } } },+ behavior: { switchSetup: { files: { installedVersion, install, uninstall } } },};beforeEach(() => {vi.clearAllMocks();mocks.getPlugin.mockReturnValue(FILES_AGENT);+ mocks.createPluginFs.mockReturnValue({});});// The connector is versioned in its own directory and listed in the registry@@ -768,6 +843,8 @@ describe('file-based connector version', () => {agent_type: 'opencode',target: 'local',outcome: 'success',+ failure_reason: 'none',+ duration_ms: aDurationMs,});});@@ -777,10 +854,14 @@ describe('file-based connector version', () => {const result = await switchSetupService.install('opencode');expect(result.success).toBe(false);+ // The app's own write failed. Nothing about a marketplace or a host CLI is+ // involved in this connector, and the code has to say so.expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {agent_type: 'opencode',target: 'local',outcome: 'failure',+ failure_reason: 'files_write_failed',+ duration_ms: aDurationMs,});});@@ -793,10 +874,234 @@ describe('file-based connector version', () => {const result = await switchSetupService.install('opencode');expect(result.success).toBe(false);+ // Its own code, not `files_write_failed`: nothing was written because there+ // was nothing to write it with, which is a fault in the plugin rather than+ // on this machine — and the two would otherwise be one number.expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {agent_type: 'opencode',target: 'local',outcome: 'failure',+ failure_reason: 'files_unimplemented',+ duration_ms: aDurationMs,+ });+ });++ /**+ * A failed removal is not a failed write.+ *+ * Both go through the same helper, so without a code of its own a read-only+ * config directory on an uninstall is counted as the same thing as an install+ * that could not write — on the one connector kind these codes exist to+ * separate. The marketplace path has drawn this line all along.+ */+ it('separates a failed removal from a failed write', async () => {+ uninstall.mockRejectedValue(new Error('permission denied'));++ const result = await switchSetupService.uninstall('opencode');++ expect(result.success).toBe(false);+ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_uninstalled', {+ agent_type: 'opencode',+ target: 'local',+ outcome: 'failure',+ failure_reason: 'files_remove_failed',+ duration_ms: aDurationMs,+ });+ });++ /**+ * `files_unimplemented` is a claim about the shipped app — this plugin ships+ * no behavior — so only the plugin actually being unimplemented may produce+ * it. A filesystem that cannot be built is a fault on the machine, and it+ * counts as the write failing; reporting it as the plugin's fault would put a+ * plugin-authoring defect and a machine fault in one number, and on the remote+ * driver it would blame the shipped app for a dead SSH channel.+ */+ it('does not blame the plugin for a failure that is not the plugin', async () => {+ mocks.createPluginFs.mockImplementation(() => {+ throw new Error('home directory is not readable');+ });++ const result = await switchSetupService.install('opencode');++ expect(result).toEqual({ success: false, message: 'home directory is not readable' });+ expect(mocks.trackEvent).toHaveBeenCalledWith('connector_installed', {+ agent_type: 'opencode',+ target: 'local',+ outcome: 'failure',+ failure_reason: 'files_write_failed',+ duration_ms: aDurationMs,+ });+ });+});++/**+ * The codes exist to tell apart failures a user experiences identically.+ *+ * Every case below reaches the UI as some variant of "it did not work", and+ * each one needs a different fix — a marketplace source that will not resolve,+ * a host CLI that refuses the plugin, an update that got halfway. Counting them+ * as one number answers none of those questions, which is what this pins.+ */+describe('why a connector operation failed', () => {+ function reported(name: string): Record<string, unknown> {+ const call = mocks.trackEvent.mock.calls.find((c) => c[0] === name);+ if (!call) throw new Error(`nothing reported ${name}`);+ return call[1] as Record<string, unknown>;+ }++ /** Every exec fails, except the listings the operation reads first. */+ function execFailingAfterListings(stderr: string) {+ return (_bin: string, args: string[] = []) => {+ const a = args.join(' ');+ if (a === 'plugin list --json' || a === 'plugin marketplace list --json') {+ return Promise.resolve({ stdout: JSON.stringify([]), stderr: '' });+ }+ return Promise.reject(Object.assign(new Error('boom'), { code: 1, stderr }));+ };+ }++ it('blames the marketplace when it is the marketplace that would not register', async () => {+ // An empty listing sends `install` through `marketplace add`, which fails+ // here — before the plugin command is ever reached.+ mocks.exec.mockImplementation(execFailingAfterListings('could not resolve source'));++ await switchSetupService.install('claude');++ expect(reported('connector_installed')).toMatchObject({+ outcome: 'failure',+ failure_reason: 'marketplace_failed',+ });+ });++ it('blames the update verb when the host has one and it failed', async () => {+ const base = execImpl('0.1.0');+ mocks.exec.mockImplementation((bin: string, args: string[] = []) => {+ if (args.join(' ').startsWith('plugin update')) {+ return Promise.reject(Object.assign(new Error('boom'), { code: 1, stderr: 'locked' }));+ }+ return base(bin, args);+ });++ await switchSetupService.update('claude', 'user');++ expect(reported('connector_updated')).toMatchObject({+ outcome: 'failure',+ failure_reason: 'update_command_failed',+ was_reinstall: false,+ });+ });++ it('blames the uninstall when a reinstall-style update cannot remove the old plugin', async () => {+ // Codex has no update verb, so an update is remove-then-add. Failing at the+ // remove leaves the previous connector in place: nothing was lost.+ mocks.getPlugin.mockReturnValue(CODEX_AGENT);+ mocks.resolveCommandPath.mockResolvedValue('/usr/bin/codex');+ const base = codexExecImpl('0.1.0');+ mocks.exec.mockImplementation((bin: string, args: string[] = []) => {+ if (args.join(' ') === `plugin remove ${CODEX_REF}`) {+ return Promise.reject(Object.assign(new Error('boom'), { code: 1, stderr: 'in use' }));+ }+ return base(bin, args);+ });++ await switchSetupService.update('codex', 'user');++ expect(reported('connector_updated')).toMatchObject({+ outcome: 'failure',+ failure_reason: 'uninstall_command_failed',+ was_reinstall: true,+ });+ });++ it('blames the install when a reinstall-style update removed the plugin and could not put it back', async () => {+ // The same button, one step later, and a materially worse outcome: the+ // agent now has no connector at all. `was_reinstall` alone cannot separate+ // this from the case above — both are true — so the code has to.+ mocks.getPlugin.mockReturnValue(CODEX_AGENT);+ mocks.resolveCommandPath.mockResolvedValue('/usr/bin/codex');+ const base = codexExecImpl('0.1.0');+ mocks.exec.mockImplementation((bin: string, args: string[] = []) => {+ if (args.join(' ') === `plugin add ${CODEX_REF}`) {+ return Promise.reject(Object.assign(new Error('boom'), { code: 1, stderr: 'no network' }));+ }+ return base(bin, args);+ });++ await switchSetupService.update('codex', 'user');++ expect(reported('connector_updated')).toMatchObject({+ outcome: 'failure',+ failure_reason: 'install_command_failed',+ was_reinstall: true,+ });+ });++ it('blames the uninstall command when removing the connector failed', async () => {+ mocks.exec.mockImplementation(execFailingAfterListings('permission denied'));++ await switchSetupService.uninstall('claude');++ expect(reported('connector_uninstalled')).toMatchObject({+ outcome: 'failure',+ failure_reason: 'uninstall_command_failed',+ });+ });++ it('reports no reason at all when the operation worked', async () => {+ // `none` rather than an absent property: every connector_installed then+ // carries the same keys, so a gap in the data is a send that went wrong+ // rather than an outcome nobody thought about.+ mocks.exec.mockImplementation(execImpl(null));++ await switchSetupService.install('claude');++ expect(reported('connector_installed')).toMatchObject({+ outcome: 'success',+ failure_reason: 'none',+ });+ });++ /**+ * The agent's own CLI not being installed is not a marketplace fault.+ *+ * Every command goes through that binary, so the first one fails and the rest+ * are consequences — and the first one is the marketplace listing. Reported as+ * `marketplace_failed` the whole condition reads as a broken plugin source and+ * sends whoever acts on it at the wrong repository, when what is missing is the+ * agent.+ */+ describe('when the agent CLI is not on the machine', () => {+ /** What a spawn of a binary that does not exist rejects with. */+ function enoent() {+ return () =>+ Promise.reject(Object.assign(new Error('spawn claude ENOENT'), { code: 'ENOENT' }));+ }++ beforeEach(() => {+ mocks.resolveCommandPath.mockResolvedValue(null);+ mocks.exec.mockImplementation(enoent());+ });++ it.each([+ ['install', 'connector_installed'],+ ['update', 'connector_updated'],+ ['uninstall', 'connector_uninstalled'],+ ] as const)('says so on %s', async (verb, event) => {+ await (verb === 'update'+ ? switchSetupService.update('claude', 'user')+ : switchSetupService[verb]('claude'));++ expect(reported(event)).toMatchObject({+ outcome: 'failure',+ failure_reason: 'host_cli_missing',+ });+ });++ it('says which binary in the message the user reads', async () => {+ const result = await switchSetupService.install('claude');++ expect(result.message).toContain('claude');});});});diff --git a/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.ts b/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.tsindex 16b88b8ad..7253fa66b 100644--- a/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.ts+++ b/console/apps/switch-console-desktop/src/main/core/switch-setup/switch-setup-service.ts@@ -7,47 +7,42 @@ import { type ArtifactName, artifactVersion } from '@switch-console/shared';import { providerAdapterRegistry } from '@main/core/agent-runtime/impl/provider-adapter-registry';import { LocalExecutionContext } from '@main/core/execution-context/local-execution-context';import { agentTypeOf } from '@main/core/telemetry/agent-type';+import { startTimer } from '@main/core/telemetry/duration';+import type {+ TelemetryConnectorFailure,+ TelemetryConnectorUpdateTrigger,+} from '@main/core/telemetry/events';import { trackEvent } from '@main/core/telemetry/telemetry-service';import { log } from '@main/lib/logger';import { isNewerVersion } from '@main/lib/semver';-import { isValidProviderId } from '@shared/core/providers/agent-provider-registry';import type { AgentTypeAvailability } from '@shared/core/switch-setup/agent-type-availability';import { createPluginFs } from '../providers/plugin-fs';import { getPlugin, listPlugins } from '../providers/plugin-registry';+import {+ commandFailureCode,+ CONNECTOR_UNSUPPORTED_RESULT,+ type ConnectorRun,+ connectorFailed,+ type ConnectorRunResult,+ connectorSucceeded,+ connectorUnsupported,+ declaresNoConnector,+ EXEC_TIMEOUT_MS,+ FilesConnectorUnimplementedError,+ HostCliMissingError,+ marketplaceFailed,+ marketplaceMatchesSource,+ runReportedOperation,+ type SwitchSetupResult,+ type SwitchSetupStatus,+ unsupportedStatus,+} from './connector-run';import {cliRulesFor,type InstalledPlugin,- type RegisteredMarketplace,type SwitchSetupCliRules,} from './switch-setup-cli-dialect';-/** Status of an agent type's Switch connector plugin. */-export type SwitchSetupStatus = {- agentId: string;- /** False when the agent type declares no Switch setup (kind: 'none'). */- supported: boolean;- installed: boolean;- installedVersion: string | null;- latestVersion: string | null;- updateAvailable: boolean;- /**- * Set when a checkForUpdates marketplace refresh failed: the returned- * versions come from the stale local cache, not the source. Always null on- * plain getStatus reads.- */- refreshError: string | null;-};--/** Whether a registered marketplace entry points at the expected source. */-export function marketplaceMatchesSource(entry: RegisteredMarketplace, source: string): boolean {- return entry.source === source;-}--/** Outcome of a mutating operation, mirroring the providers controller shape. */-export type SwitchSetupResult = { success: boolean; message?: string };--const EXEC_TIMEOUT_MS = 120_000;-/*** The version stamped into a file-based connector install, and the one it is* compared against.@@ -68,7 +63,10 @@ function installFailureMessage(raw: string): string {return raw || 'Install failed.';}-type RunResult = { code: number; stdout: string; stderr: string };+/** A shell that cannot find the binary fails the spawn rather than the command. */+function isCommandNotFound(err: { code?: number | string }): boolean {+ return err.code === 'ENOENT';+}/** Parse CLI JSON, yielding null rather than throwing on unparseable output. */function parseJsonOrNull(stdout: string): unknown {@@ -79,18 +77,6 @@ function parseJsonOrNull(stdout: string): unknown {}}-function unsupported(agentId: string): SwitchSetupStatus {- return {- agentId,- supported: false,- installed: false,- installedVersion: null,- latestVersion: null,- updateAvailable: false,- refreshError: null,- };-}-/*** Drives an agent's plugin-marketplace CLI (`<bin> plugin ...`,* `<bin> plugin marketplace ...`) to manage that agent's Switch connector@@ -105,25 +91,32 @@ class SwitchSetupService {/*** The `files` connector behavior for an agent whose connector Switch Console- * writes itself, or null for any other agent.+ * writes itself.+ *+ * Every caller reaches this under a `kind === 'files'` test, so a descriptor of+ * any other kind is a programming error rather than an outcome. A declared+ * descriptor with no behavior is the one expected failure, and it gets its own+ * error class: reporting "not installed" forever would send the user to a+ * button that silently does nothing.+ *+ * Deliberately builds no filesystem. Creating one can fail for reasons that+ * have nothing to do with the plugin, and keeping it out leaves exactly one+ * thing this can throw.*/- private resolveFiles(agentId: string) {+ private resolveFiles(agentId: string): { files: ISwitchSetupFilesBehavior; version: string } {const plugin = getPlugin(agentId);- if (plugin.capabilities.switchSetup.kind !== 'files') return null;- const files = plugin.behavior.switchSetup?.files;- if (!files) {- // A declared descriptor with no behavior cannot install anything, and- // reporting "not installed" forever would send the user to a button that- // silently does nothing.- throw new Error(- `Agent '${agentId}' declares a file-based Switch connector but implements no behavior for it.`- );+ const descriptor = plugin.capabilities.switchSetup;+ if (descriptor.kind !== 'files') {+ throw new Error(`Agent '${agentId}' has no file-based Switch connector.`);}- return {- files,- homeFs: createPluginFs(homedir()),- version: connectorVersion(plugin.capabilities.switchSetup.artifact),- };+ const files = plugin.behavior.switchSetup?.files;+ if (!files) throw new FilesConnectorUnimplementedError(agentId);+ return { files, version: connectorVersion(descriptor.artifact) };+ }++ /** The filesystem a file-based connector is written to on this computer. */+ private homeFs(): PluginFs {+ return createPluginFs(homedir());}/** Returns the `cli` descriptor + agent binary, or null when unsupported/unresolvable. */@@ -139,13 +132,23 @@ class SwitchSetupService {}/** Run a CLI command, capturing output and exit code without throwing. */- private async run(bin: string, args: string[]): Promise<RunResult> {+ private async run(bin: string, args: string[]): Promise<ConnectorRunResult> {try {const { stdout, stderr } = await this.ctx.exec(bin, args, { timeout: EXEC_TIMEOUT_MS });- return { code: 0, stdout, stderr };+ return { code: 0, stdout, stderr, notFound: false };} catch (err: unknown) {- const e = err as { stdout?: string; stderr?: string; code?: number; message?: string };- return { code: e.code ?? 1, stdout: e.stdout ?? '', stderr: e.stderr ?? e.message ?? '' };+ const e = err as {+ stdout?: string;+ stderr?: string;+ code?: number | string;+ message?: string;+ };+ return {+ code: typeof e.code === 'number' ? e.code : 1,+ stdout: e.stdout ?? '',+ stderr: e.stderr ?? e.message ?? '',+ notFound: isCommandNotFound(e),+ };}}@@ -217,11 +220,15 @@ class SwitchSetupService {marketplaceSource: string,rules: SwitchSetupCliRules): Promise<void> {- const { stdout } = await this.run(bin, ['plugin', 'marketplace', 'list', '--json']);+ const listed = await this.run(bin, ['plugin', 'marketplace', 'list', '--json']);+ // No binary means no marketplace to repair and no plugin to install; every+ // command after this one would fail the same way. Said here so the caller+ // does not have to read it off whichever verb happened to run first.+ if (listed.notFound) throw new HostCliMissingError(bin);// An unreadable listing yields no entries; the add below is idempotent, so// attempting it is safer than treating an unparseable listing as fatal.const existing = rules- .parseMarketplaceList(parseJsonOrNull(stdout))+ .parseMarketplaceList(parseJsonOrNull(listed.stdout)).find((m) => m.name === marketplaceName);if (existing) {if (marketplaceMatchesSource(existing, marketplaceSource)) return;@@ -250,10 +257,8 @@ class SwitchSetupService {* "update available" means here.*/private async filesStatus(agentId: string): Promise<SwitchSetupStatus> {- const resolved = this.resolveFiles(agentId);- if (!resolved) return unsupported(agentId);- const installedVersion = await resolved.files.installedVersion(resolved.homeFs);- const latestVersion = resolved.version;+ const { files, version: latestVersion } = this.resolveFiles(agentId);+ const installedVersion = await files.installedVersion(this.homeFs());return {agentId,supported: true,@@ -271,7 +276,7 @@ class SwitchSetupService {return this.filesStatus(agentId);}const resolved = await this.resolve(agentId);- if (!resolved) return unsupported(agentId);+ if (!resolved) return unsupportedStatus(agentId);const { descriptor, bin, rules } = resolved;const entry = await this.findInstalled(bin, resolved.ref, rules);@@ -340,7 +345,24 @@ class SwitchSetupService {});continue;}- const status = await this.getStatus(agentId);+ // One plugin whose status cannot be read must not empty the list: this+ // fans out over every agent type, and the picker showing nothing at all+ // is a worse answer than one row saying why it is unavailable.+ let status: SwitchSetupStatus;+ try {+ status = await this.getStatus(agentId);+ } catch (err) {+ log.error('switch-setup: could not read connector status', { agentId, err });+ availability.push({+ agentId,+ available: false,+ blockedReason: `Its Switch connector status could not be read: ${+ err instanceof Error ? err.message : String(err)+ }`,+ blockedKind: 'unknown',+ });+ continue;+ }availability.push(status.installed? { agentId, available: true, blockedReason: null, blockedKind: null }@@ -367,7 +389,7 @@ class SwitchSetupService {return this.filesStatus(agentId);}const resolved = await this.resolve(agentId);- if (!resolved) return unsupported(agentId);+ if (!resolved) return unsupportedStatus(agentId);const { descriptor, bin, rules } = resolved;let refreshError: string | null = null;try {@@ -390,66 +412,75 @@ class SwitchSetupService {return { ...(await this.getStatus(agentId)), refreshError };}- /** Install, update and uninstall for a file-based connector. */+ /**+ * Install, update and uninstall for a file-based connector.+ *+ * `failure` is the caller's, because the two sides of the operation are+ * different walls: a write that could not land and a removal that could not+ * are as distinct here as `install_command_failed` and+ * `uninstall_command_failed` are on the marketplace path.+ *+ * Only a plugin that declares files and implements none is caught as+ * `files_unimplemented`. Anything else resolving throws is a fault somewhere+ * other than the plugin, and saying "this connector ships no behavior" about it+ * would be a claim about the app derived from, say, a dead SSH channel. Those+ * propagate to the entry point, which reports them as `error`.+ */private async runFiles(agentId: string,+ failure: TelemetryConnectorFailure,action: (files: ISwitchSetupFilesBehavior,homeFs: PluginFs,version: string) => Promise<unknown>- ): Promise<SwitchSetupResult> {- const resolved = this.resolveFiles(agentId);- if (!resolved)- return { success: false, message: 'Switch setup is not supported for this agent.' };+ ): Promise<ConnectorRun> {+ let resolved: { files: ISwitchSetupFilesBehavior; version: string };try {- await action(resolved.files, resolved.homeFs, resolved.version);- return { success: true };+ resolved = this.resolveFiles(agentId);+ } catch (err) {+ if (!(err instanceof FilesConnectorUnimplementedError)) throw err;+ log.error('switch-setup: file-based connector declares no behavior', { agentId, err });+ return connectorFailed(err.message, 'files_unimplemented');+ }+ try {+ await action(resolved.files, this.homeFs(), resolved.version);+ return connectorSucceeded();} catch (err) {log.error('switch-setup: file-based connector operation failed', { agentId, err });- return { success: false, message: err instanceof Error ? err.message : String(err) };+ return connectorFailed(err instanceof Error ? err.message : String(err), failure);}}+ /**+ * Install the connector, reporting the outcome.+ *+ * Everything past the `declaresNoConnector` guard is an attempt a person made+ * and is reported, `unsupported` included: a `cli` descriptor whose binary+ * cannot be resolved is a real failure of the thing the user asked for.+ */async install(agentId: string): Promise<SwitchSetupResult> {- const { result, attempted } = await this.runInstall(agentId);- // An agent type with no connector to install did not fail to install one.- if (attempted) {- trackEvent('connector_installed', {- agent_type: isValidProviderId(agentId) ? agentId : 'unknown',- target: 'local',- outcome: result.success ? 'success' : 'failure',- });- }- return result;+ if (declaresNoConnector(agentId)) return CONNECTOR_UNSUPPORTED_RESULT;+ const elapsed = startTimer();+ const run = await runReportedOperation('switch-setup', agentId, () => this.runInstall(agentId));+ trackEvent('connector_installed', {+ agent_type: agentTypeOf(agentId),+ target: 'local',+ outcome: run.result.success ? 'success' : 'failure',+ failure_reason: run.failure,+ duration_ms: elapsed(),+ });+ return run.result;}- private async runInstall(- agentId: string- ): Promise<{ result: SwitchSetupResult; attempted: boolean }> {+ private async runInstall(agentId: string): Promise<ConnectorRun> {if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- // `runFiles` resolves the behavior outside its own try, and that throws- // for a connector that declares files and implements none. An install the- // user asked for must come back as a failed result either way — a- // rejection here would skip the report and reach the UI as a stack.- try {- const result = await this.runFiles(agentId, (files, fs, version) =>- files.install(fs, { version })- );- return { result, attempted: true };- } catch (err) {- return {- result: { success: false, message: installFailureMessage(String(err)) },- attempted: true,- };- }+ return this.runFiles(agentId, 'files_write_failed', (files, fs, version) =>+ files.install(fs, { version })+ );}const resolved = await this.resolve(agentId);- if (!resolved)- return {- result: { success: false, message: 'Switch setup is not supported for this agent.' },- attempted: false,- };+ if (!resolved) return connectorUnsupported();const { descriptor, bin, ref, rules } = resolved;try {await this.ensureMarketplace(@@ -459,50 +490,71 @@ class SwitchSetupService {rules);} catch (err) {- return {- result: { success: false, message: installFailureMessage(String(err)) },- attempted: true,- };+ return marketplaceFailed(err, installFailureMessage(String(err)));}const res = await this.run(bin, rules.installArgs(ref, descriptor.scope));- return {- result:- res.code === 0- ? { success: true }- : { success: false, message: installFailureMessage(res.stderr.trim()) },- attempted: true,- };+ return res.code === 0+ ? connectorSucceeded()+ : connectorFailed(+ installFailureMessage(res.stderr.trim()),+ commandFailureCode(res, 'install_command_failed')+ );}/**- * Update the installed plugin. Dialects without a per-plugin update verb- * (Codex) are updated by uninstalling and reinstalling; a failed reinstall is- * reported as such rather than as a plain update failure, because it leaves- * the agent with no connector rather than with the previous version.+ * Update the connector, reporting the outcome.+ *+ * Dialects without a per-plugin update verb (Codex) are updated by+ * uninstalling and reinstalling; a failed reinstall is reported as such rather+ * than as a plain update failure, because it leaves the agent with no+ * connector rather than with the previous version.** The marketplace is repaired first, exactly as `install` does. The re-add* resolves against whatever marketplace is registered, so a stale source would* otherwise fail it — after the uninstall has already succeeded.- */- /**- * Update the connector, reporting the outcome.*- * The "not supported" answer is not an attempt and is not reported — the same- * distinction `install` draws with its `attempted` flag, without which every- * agent that has no connector would look like a failing update.+ * `trigger` is required rather than defaulted: the once-per-install catch-up+ * reaches this through the same door as the Update button, and a default would+ * quietly file it as whichever of the two the default happened to be.*/- async update(agentId: string): Promise<SwitchSetupResult> {- if (getPlugin(agentId).capabilities.switchSetup.kind === 'none') {- return { success: false, message: 'Switch setup is not supported for this agent.' };- }- const { result, wasReinstall } = await this.runUpdate(agentId);+ async update(+ agentId: string,+ trigger: TelemetryConnectorUpdateTrigger+ ): Promise<SwitchSetupResult> {+ if (declaresNoConnector(agentId)) return CONNECTOR_UNSUPPORTED_RESULT;+ const elapsed = startTimer();+ const { run, wasReinstall } = await this.runUpdateReported(agentId);trackEvent('connector_updated', {agent_type: agentTypeOf(agentId),target: 'local',- outcome: result.success ? 'success' : 'failure',+ outcome: run.result.success ? 'success' : 'failure',was_reinstall: wasReinstall,+ trigger,+ failure_reason: run.failure,+ duration_ms: elapsed(),+ });+ return run.result;+ }++ /**+ * `runUpdate` with the guarantee the other two entry points get from+ * `runReportedOperation`: it always comes back as a result rather than+ * rejecting.+ *+ * `wasReinstall` is false when it throws. Everything that can throw here runs+ * before the remove-then-add — resolving the binary, building a filesystem —+ * so a throw means nothing was removed, which is exactly what the flag says.+ */+ private async runUpdateReported(+ agentId: string+ ): Promise<{ run: ConnectorRun; wasReinstall: boolean }> {+ let wasReinstall = false;+ const run = await runReportedOperation('switch-setup', agentId, async () => {+ const outcome = await this.runUpdate(agentId);+ wasReinstall = outcome.wasReinstall;+ return outcome.run;});- return result;+ return { run, wasReinstall };}/**@@ -511,25 +563,18 @@ class SwitchSetupService {* verb, so for it every update is the second kind, with a window in between* where nothing is installed.*/- private async runUpdate(- agentId: string- ): Promise<{ result: SwitchSetupResult; wasReinstall: boolean }> {+ private async runUpdate(agentId: string): Promise<{ run: ConnectorRun; wasReinstall: boolean }> {// Installing a file-based connector overwrites in place, so update is the// same operation — there is no removed-but-not-reinstalled window.if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- const result = await this.runFiles(agentId, (files, fs, version) =>+ const run = await this.runFiles(agentId, 'files_write_failed', (files, fs, version) =>files.install(fs, { version }));// Overwritten in place: neither a verb update nor a remove-and-replace.- return { result, wasReinstall: false };+ return { run, wasReinstall: false };}const resolved = await this.resolve(agentId);- if (!resolved) {- return {- result: { success: false, message: 'Switch setup is not supported for this agent.' },- wasReinstall: false,- };- }+ if (!resolved) return { run: connectorUnsupported(), wasReinstall: false };const { descriptor, bin, ref, rules } = resolved;try {@@ -541,7 +586,7 @@ class SwitchSetupService {);} catch (err) {return {- result: { success: false, message: `Could not add marketplace: ${String(err)}` },+ run: marketplaceFailed(err, `Could not add marketplace: ${String(err)}`),wasReinstall: false,};}@@ -550,10 +595,13 @@ class SwitchSetupService {if (updateArgs) {const res = await this.run(bin, updateArgs);return {- result:+ run:res.code === 0- ? { success: true }- : { success: false, message: res.stderr.trim() || 'Update failed.' },+ ? connectorSucceeded()+ : connectorFailed(+ res.stderr.trim() || 'Update failed.',+ commandFailureCode(res, 'update_command_failed')+ ),wasReinstall: false,};}@@ -561,53 +609,57 @@ class SwitchSetupService {const removed = await this.run(bin, rules.uninstallArgs(ref, descriptor.scope));if (removed.code !== 0) {return {- result: {- success: false,- message: removed.stderr.trim() || 'Update failed: could not remove the installed plugin.',- },+ run: connectorFailed(+ removed.stderr.trim() || 'Update failed: could not remove the installed plugin.',+ commandFailureCode(removed, 'uninstall_command_failed')+ ),wasReinstall: true,};}const added = await this.run(bin, rules.installArgs(ref, descriptor.scope));return {- result:+ run:added.code === 0- ? { success: true }- : {- success: false,- message:- added.stderr.trim() ||+ ? connectorSucceeded()+ : connectorFailed(+ added.stderr.trim() ||'Update failed: the plugin was removed but could not be reinstalled. Install it again from Settings → Agents.',- },+ commandFailureCode(added, 'install_command_failed')+ ),wasReinstall: true,};}async uninstall(agentId: string): Promise<SwitchSetupResult> {- if (getPlugin(agentId).capabilities.switchSetup.kind === 'none') {- return { success: false, message: 'Switch setup is not supported for this agent.' };- }- const result = await this.runUninstall(agentId);+ if (declaresNoConnector(agentId)) return CONNECTOR_UNSUPPORTED_RESULT;+ const elapsed = startTimer();+ const run = await runReportedOperation('switch-setup', agentId, () =>+ this.runUninstall(agentId)+ );trackEvent('connector_uninstalled', {agent_type: agentTypeOf(agentId),target: 'local',- outcome: result.success ? 'success' : 'failure',+ outcome: run.result.success ? 'success' : 'failure',+ failure_reason: run.failure,+ duration_ms: elapsed(),});- return result;+ return run.result;}- private async runUninstall(agentId: string): Promise<SwitchSetupResult> {+ private async runUninstall(agentId: string): Promise<ConnectorRun> {if (getPlugin(agentId).capabilities.switchSetup.kind === 'files') {- return this.runFiles(agentId, (files, fs) => files.uninstall(fs));+ return this.runFiles(agentId, 'files_remove_failed', (files, fs) => files.uninstall(fs));}const resolved = await this.resolve(agentId);- if (!resolved)- return { success: false, message: 'Switch setup is not supported for this agent.' };+ if (!resolved) return connectorUnsupported();const { descriptor, bin, ref, rules } = resolved;const res = await this.run(bin, rules.uninstallArgs(ref, descriptor.scope));return res.code === 0- ? { success: true }- : { success: false, message: res.stderr.trim() || 'Uninstall failed.' };+ ? connectorSucceeded()+ : connectorFailed(+ res.stderr.trim() || 'Uninstall failed.',+ commandFailureCode(res, 'uninstall_command_failed')+ );}}diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/catalogue.test.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/catalogue.test.tsindex 43656fd71..9fd7600dd 100644--- a/console/apps/switch-console-desktop/src/main/core/telemetry/catalogue.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/catalogue.test.ts@@ -32,12 +32,13 @@ const CONTEXT: TelemetryContext = {/*** A value of the right kind for each property, chosen by name.*- * Counts are numbers, the yes/no properties are booleans, and everything else- * is a placeholder string — the builder does not check a string against the- * catalogue's unions, so any string exercises the same path.+ * Counts and durations are numbers, the yes/no properties are booleans, and+ * everything else is a placeholder string — the builder does not check a string+ * against the catalogue's unions, so any string exercises the same path.*/function sampleFor(property: string): string | number | boolean {if (property.endsWith('_count')) return 3;+ if (property.endsWith('_ms')) return 1234;if (property.startsWith('has_') ||property.startsWith('was_') ||diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/cli-action.test.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/cli-action.test.tsnew file mode 100644index 000000000..68b82ffc2--- /dev/null+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/cli-action.test.ts@@ -0,0 +1,103 @@+import { aDurationMs } from '@tooling/utils/telemetry-duration';+import { beforeEach, describe, expect, it, vi } from 'vitest';++const mocks = vi.hoisted(() => ({ trackEvent: vi.fn() }));+vi.mock('./telemetry-service', () => ({ trackEvent: mocks.trackEvent }));++import { reportedCliAction } from './cli-action';++function reported(): Record<string, unknown> {+ const call = mocks.trackEvent.mock.calls.find((c) => c[0] === 'agent_cli_action');+ if (!call) throw new Error('nothing reported agent_cli_action');+ return call[1] as Record<string, unknown>;+}++beforeEach(() => {+ vi.clearAllMocks();+});++describe('reporting a CLI install, update or removal', () => {+ it('carries the result of an operation that came back', async () => {+ const result = await reportedCliAction('install', 'local', 'claude', 'npm', async () => ({+ success: true,+ }));++ expect(result).toEqual({ success: true });+ expect(reported()).toEqual({+ agent_type: 'claude',+ target: 'local',+ install_method: 'npm',+ action: 'install',+ outcome: 'success',+ failure_reason: 'none',+ duration_ms: aDurationMs,+ });+ });++ it('names the code a failed operation returned', async () => {+ await reportedCliAction('update', 'remote', 'codex', undefined, async () => ({+ success: false,+ error: { type: 'permission-denied' },+ }));++ expect(reported()).toMatchObject({+ target: 'remote',+ // No method was supplied, and `unspecified` says that rather than picking+ // one — every one of these paths takes the method as optional.+ install_method: 'unspecified',+ outcome: 'failure',+ failure_reason: 'permission_denied',+ });+ });++ /**+ * The gap this exists to close.+ *+ * These operations return a `Result` rather than raising, so the reporting+ * used to sit on the line after the call — and a manager that raised took its+ * attempt out of the count altogether. Not a failure in the numerator: an+ * absence from both, which silently flatters the install success rate by+ * exactly the failures that went worst.+ */+ it('reports an operation that threw, and still lets the throw through', async () => {+ const boom = new Error('ssh: connection closed');++ await expect(+ reportedCliAction('uninstall', 'remote', 'opencode', undefined, () => Promise.reject(boom))+ ).rejects.toBe(boom);++ expect(reported()).toMatchObject({+ agent_type: 'opencode',+ action: 'uninstall',+ outcome: 'failure',+ failure_reason: 'error',+ duration_ms: aDurationMs,+ });+ });++ it('reports a dependency that names no agent as unknown rather than as itself', async () => {+ // These ids are dependency ids, and a core dependency is not an agent type.+ // Passing it through would put free text in a payload.+ await reportedCliAction('install', 'remote', 'docker', undefined, async () => ({+ success: true,+ }));++ expect(reported()).toMatchObject({ agent_type: 'unknown' });+ });++ it('times the operation and nothing around it', async () => {+ const SLEEP_MS = 20;+ await reportedCliAction(+ 'install',+ 'local',+ 'claude',+ undefined,+ () =>+ new Promise<{ success: boolean }>((resolve) =>+ setTimeout(() => resolve({ success: true }), SLEEP_MS)+ )+ );++ expect(reported().duration_ms).toBeGreaterThanOrEqual(SLEEP_MS - 5);+ });+});diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/cli-action.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/cli-action.tsnew file mode 100644index 000000000..832a60755--- /dev/null+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/cli-action.ts@@ -0,0 +1,70 @@+import type { InstallMethod } from '@switch-console/core/deps';+import { agentTypeOf } from './agent-type';+import { cliFailureReason } from './cli-failure';+import { startTimer } from './duration';+import type { TelemetryCliAction, TelemetryDurationMs } from './events';+import { installMethodOf } from './narrow';+import { trackEvent } from './telemetry-service';++/** As much of a dependency operation's result as reporting needs. */+type CliActionResult = { success: boolean; error?: { type?: string } };++/** Where the CLI is being installed. The host itself is never named. */+type CliActionTarget = 'local' | 'remote';++function report(+ action: TelemetryCliAction,+ target: CliActionTarget,+ id: string,+ method: InstallMethod | undefined,+ result: CliActionResult,+ durationMs: TelemetryDurationMs+): void {+ trackEvent('agent_cli_action', {+ // The id may be a core dependency rather than an agent, so it goes through+ // the same narrowing as anywhere else and reports `unknown` when it names+ // no provider.+ agent_type: agentTypeOf(id),+ target,+ install_method: installMethodOf(method),+ action,+ outcome: result.success ? 'success' : 'failure',+ failure_reason: cliFailureReason(result),+ duration_ms: durationMs,+ });+}++/**+ * Run a dependency operation — installing, updating or removing an agent's own+ * CLI — and report it whether or not it came back.+ *+ * The event's denominator is attempts. These operations return a `Result`+ * rather than raising, but that is a convention and not a type: one that raises+ * took its attempt out of the count altogether, which is not a failure showing+ * up in the numerator but an absence showing up in neither. `error` is already+ * the code for a throw the union says these paths "are not supposed to do but+ * can", so the event goes out carrying it and the exception is then let+ * through — the caller sees exactly what it saw before.+ *+ * The timer is started here, around the operation alone. Resolving the manager+ * is the caller's and stays outside it, since it is not part of how long an+ * install takes.+ */+export async function reportedCliAction<T extends CliActionResult>(+ action: TelemetryCliAction,+ target: CliActionTarget,+ id: string,+ method: InstallMethod | undefined,+ run: () => Promise<T>+): Promise<T> {+ const elapsed = startTimer();+ let result: T;+ try {+ result = await run();+ } catch (error) {+ report(action, target, id, method, { success: false }, elapsed());+ throw error;+ }+ report(action, target, id, method, result, elapsed());+ return result;+}diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/disclosure.test.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/disclosure.test.tsnew file mode 100644index 000000000..2953997cb--- /dev/null+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/disclosure.test.ts@@ -0,0 +1,163 @@+import { readFileSync } from 'node:fs';+import { dirname, join } from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { TELEMETRY_EVENT_PROPERTIES, type TelemetryEventName } from './events';++/**+ * The disclosure and the catalogue say the same thing.+ *+ * `console/AGENTS.md` makes widening what is sent a consent decision and names+ * two artefacts that must move with it: the summary in `telemetry-copy.ts` and+ * the field-by-field list in `docs/TELEMETRY.md`, which is what the link in the+ * consent dialog opens. Until this test there was nothing holding the second to+ * the code — and it had already drifted twice, naming a failure code that is+ * never sent and omitting one that is.+ *+ * Drift here is worse than an ordinary stale comment. The document is the+ * specific thing a user is pointed at to decide whether to turn telemetry on,+ * so a value missing from it is a value nobody agreed to send.+ *+ * What this can and cannot see: it compares names, not meanings. It cannot tell+ * you the prose around a table is still true, and it deliberately skips a union+ * that is built from another type (`InstallMethod | 'unspecified'`,+ * `Exclude<…>`) rather than written out as literals — resolving those needs the+ * type checker, which is gone by the time a test runs. Those are read by a+ * person; everything written as a literal is read by this.+ */++// …/src/main/core/telemetry → the repo root that holds `docs/`.+const REPO_ROOT = join(+ dirname(fileURLToPath(import.meta.url)),+ '..',+ '..',+ '..',+ '..',+ '..',+ '..',+ '..'+);++const DISCLOSURE = readFileSync(join(REPO_ROOT, 'docs', 'TELEMETRY.md'), 'utf8');+const CATALOGUE_SOURCE = readFileSync(+ join(dirname(fileURLToPath(import.meta.url)), 'events.ts'),+ 'utf8'+);++const EVENT_SECTION_HEADING = '### 3.3 Every event and its fields';++/**+ * Each event's row in the document, keyed by event name.+ *+ * Read from the per-event section alone. The section above it tabulates the+ * dimensions several events share (`outcome`, `agent_type`) in rows of the same+ * shape, and taking those for events would make the reverse check below —+ * "every row is an event" — unreadable.+ */+function eventRows(): Map<string, string> {+ const start = DISCLOSURE.indexOf(EVENT_SECTION_HEADING);+ if (start === -1) throw new Error(`docs/TELEMETRY.md no longer has "${EVENT_SECTION_HEADING}"`);+ const rest = DISCLOSURE.slice(start + EVENT_SECTION_HEADING.length);+ const end = rest.indexOf('\n### ');++ return new Map(+ (end === -1 ? rest : rest.slice(0, end))+ .split('\n')+ .map((line) => /^\|\s*`([a-z][a-z0-9_]*)`\s*\|(.*)\|\s*$/.exec(line))+ .filter((match): match is RegExpExecArray => match !== null)+ .map((match) => [match[1]!, match[2]!])+ );+}++const ROWS = eventRows();++const EVENT_NAMES = Object.keys(TELEMETRY_EVENT_PROPERTIES) as TelemetryEventName[];++/** A backticked token, so `error` does not match the word in a sentence. */+function mentions(text: string, token: string): boolean {+ return text.includes(`\`${token}\``);+}++/** Comments are stripped first: an apostrophe in prose reads as a string quote. */+function withoutComments(source: string): string {+ return source.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/[^\n]*/g, '');+}++/** True when a union is written out as string literals and nothing else. */+function isLiteralUnion(body: string): boolean {+ return body.length > 0 && /^\s*\|?\s*'[^']*'(\s*\|\s*'[^']*')*\s*$/.test(body);+}++function literalsOf(body: string): string[] {+ return [...body.matchAll(/'([^']*)'/g)].map((match) => match[1]!);+}++/**+ * Every set of values the catalogue spells out, as `label → values`.+ *+ * Both shapes it is written in: a named `export type Telemetry… = 'a' | 'b'`,+ * and a union written inline on a property of `TelemetryEventMap`.+ */+function enumeratedValueSets(): Array<{ label: string; values: string[] }> {+ const source = withoutComments(CATALOGUE_SOURCE);+ const sets: Array<{ label: string; values: string[] }> = [];++ for (const match of source.matchAll(/export type (\w+) =([^;]*);/g)) {+ const body = match[2]!.trim();+ if (!isLiteralUnion(body)) continue;+ sets.push({ label: match[1]!, values: literalsOf(body) });+ }++ const mapStart = source.indexOf('export type TelemetryEventMap');+ const mapEnd = source.indexOf('export const TELEMETRY_EVENT_PROPERTIES');+ const map = source.slice(mapStart, mapEnd);+ for (const match of map.matchAll(/^\s+(\w+):([^;\n]*(?:\n[^;\n]*)*?);/gm)) {+ const body = match[2]!.trim();+ if (!isLiteralUnion(body)) continue;+ sets.push({ label: `${match[1]!} (inline)`, values: literalsOf(body) });+ }++ return sets;+}++const VALUE_SETS = enumeratedValueSets();++describe('what the consent dialog links to', () => {+ it('parsed the catalogue rather than silently matching nothing', () => {+ // Every check below is "no mismatches found", which is also what a broken+ // parse returns. These floors are what tells the two apart.+ expect(ROWS.size).toBe(EVENT_NAMES.length);+ expect(VALUE_SETS.length).toBeGreaterThanOrEqual(10);+ });++ it.each(EVENT_NAMES)('discloses %s', (name) => {+ expect(ROWS.get(name)).toBeDefined();+ });++ it.each(EVENT_NAMES)('names every property %s carries', (name) => {+ const row = ROWS.get(name) ?? '';+ const undisclosed = TELEMETRY_EVENT_PROPERTIES[name].filter(+ (property) => !mentions(row, property)+ );++ expect(undisclosed).toEqual([]);+ });++ it.each(VALUE_SETS)('names every value of $label', ({ values }) => {+ // Somewhere in the document rather than in a particular row: a set shared by+ // several events is written out once and referred to from the others.+ const undisclosed = values.filter((value) => !mentions(DISCLOSURE, value));++ expect(undisclosed).toEqual([]);+ });++ it('describes no event the catalogue does not have', () => {+ // The reverse direction, and the one that catches a removed event: a row+ // left behind promises something is sent that no longer is.+ const unknown = [...ROWS.keys()].filter(+ (name) => !EVENT_NAMES.includes(name as TelemetryEventName)+ );++ expect(unknown).toEqual([]);+ });+});diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/duration.test.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/duration.test.tsnew file mode 100644index 000000000..0080f9e9e--- /dev/null+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/duration.test.ts@@ -0,0 +1,108 @@+import { aDurationMs } from '@tooling/utils/telemetry-duration';+import { describe, expect, it, vi } from 'vitest';+import { startTimer } from './duration';++const SLEEP_MS = 20;++function sleep(ms: number): Promise<void> {+ return new Promise((resolve) => setTimeout(resolve, ms));+}++describe('timing an operation', () => {+ it('reports whole milliseconds, never a fraction', async () => {+ // The emitter refuses a non-finite number and the far end averages what it+ // gets, so the contract worth pinning is that this is a plain integer.+ const elapsed = startTimer();+ await sleep(SLEEP_MS);++ const ms = elapsed();++ expect(Number.isInteger(ms)).toBe(true);+ expect(ms).toBeGreaterThan(0);+ });++ it('accumulates across reads rather than restarting at each one', async () => {+ // Asserting only that the second read is >= the first would pass against a+ // timer that resets on every read: both reads would return one sleep's+ // worth. The second read has to cover BOTH sleeps for that bug to show.+ const elapsed = startTimer();+ await sleep(SLEEP_MS);+ const first = elapsed();+ await sleep(SLEEP_MS);++ expect(first).toBeGreaterThanOrEqual(SLEEP_MS - 5);+ expect(elapsed()).toBeGreaterThanOrEqual(first + SLEEP_MS - 5);+ });++ it('is the span, not the time of day', ({ onTestFinished }) => {+ // The one property every assertion elsewhere is blind to. `aDurationMs`+ // accepts any non-negative integer, so `Math.round(performance.now())` —+ // milliseconds since the process started, which is a large number that only+ // grows — satisfies every other test in this file and every call-site+ // assertion in the suite. Only a clock held still says what is subtracted+ // from what.+ const now = vi.spyOn(performance, 'now');+ onTestFinished(() => now.mockRestore());++ now.mockReturnValue(1_000_000.4);+ const elapsed = startTimer();+ now.mockReturnValue(1_004_200.6);++ expect(elapsed()).toBe(4200);+ });+});++/**+ * The matcher the call-site assertions use, because `expect.any(Number)` is not+ * an assertion about a duration.+ *+ * It is satisfied by `NaN` and by `Infinity` — both of which the emitter throws+ * on, so an event carrying one is dropped with a log line and never arrives.+ * Written that way, every duration assertion in the suite would pass against a+ * payload that cannot be sent.+ */+describe('the duration matcher', () => {+ it.each([0, 1, 4200])('accepts a whole number of milliseconds: %s', (value) => {+ expect({ duration_ms: value }).toEqual({ duration_ms: aDurationMs });+ });++ it.each([Number.NaN, Number.POSITIVE_INFINITY, -1, 12.5, '12', null])(+ 'rejects what a timer cannot produce: %s',+ (value) => {+ expect({ duration_ms: value }).not.toEqual({ duration_ms: aDurationMs });+ }+ );++ it('rejects the values expect.any(Number) would have let through', () => {+ // The specific regression: these two pass `expect.any(Number)`.+ expect(aDurationMs.asymmetricMatch(Number.NaN)).toBe(false);+ expect(aDurationMs.asymmetricMatch(Number.POSITIVE_INFINITY)).toBe(false);+ expect(expect.any(Number).asymmetricMatch(Number.NaN)).toBe(true);+ });++ it('is what a real timer produces', async () => {+ const elapsed = startTimer();+ await sleep(1);++ expect({ duration_ms: elapsed() }).toEqual({ duration_ms: aDurationMs });+ });++ it('accepts a zero-length operation', ({ onTestFinished }) => {+ // A fast operation legitimately rounds to 0, so the matcher must not+ // require a positive number.+ //+ // Restored through the hook rather than after the assertion: a failing+ // expectation throws, and a frozen clock left behind would be inherited by+ // whatever ran next in this file.+ const now = vi.spyOn(performance, 'now').mockReturnValue(1000);+ onTestFinished(() => now.mockRestore());++ const elapsed = startTimer();++ // Both halves: that the timer really produces 0 for a span of no time, and+ // that the matcher takes it. Asserting only the second would pass against a+ // timer that returned the clock reading instead of the span.+ expect(elapsed()).toBe(0);+ expect({ duration_ms: elapsed() }).toEqual({ duration_ms: aDurationMs });+ });+});diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/duration.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/duration.tsnew file mode 100644index 000000000..673bceebc--- /dev/null+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/duration.ts@@ -0,0 +1,29 @@+import type { TelemetryDurationMs } from './events';++/**+ * Start timing an operation that reports how long it took.+ *+ * `performance.now()` rather than `Date.now()`: it is monotonic, so an NTP step+ * or a machine waking mid-install cannot yield a negative duration or an hour+ * that never passed. `Date.now()` can do both, and a negative number reaching a+ * payload is worse than no number — it is data nobody can tell from real data.+ * Because the clock only moves forwards, the result needs no clamping.+ *+ * Whole milliseconds. Sub-millisecond precision says nothing about an operation+ * that shells out to a package manager, and it keeps the value a plain integer+ * at the far end. See `TelemetryDurationMs` for what bounds the value and how to+ * read the result.+ *+ * Call it before the work and call the returned function at the point the+ * outcome is known — not after the event is built, and not around the send,+ * which is fire-and-forget and has nothing to do with what the user waited for.+ *+ * This is the only place a `TelemetryDurationMs` is made. The cast is the brand+ * being applied, and it is sound precisely here: the value came off a monotonic+ * clock and through `Math.round`, so it is a whole number and cannot be+ * negative.+ */+export function startTimer(): () => TelemetryDurationMs {+ const start = performance.now();+ return () => Math.round(performance.now() - start) as TelemetryDurationMs;+}diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/events.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/events.tsindex 2f7ec2199..c3c580189 100644--- a/console/apps/switch-console-desktop/src/main/core/telemetry/events.ts+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/events.ts@@ -136,6 +136,95 @@ export type TelemetryCliFailure =| 'still_present'| 'error';+/**+ * Why installing, updating or removing a **Switch connector** failed — not the+ * agent's own CLI, which `TelemetryCliFailure` covers.+ *+ * Named at the point of failure rather than mapped from an error type: these+ * paths report a message, and a message cannot be sent. The codes separate the+ * walls that need different fixes, because "connector install failed" tells+ * nobody which to look at.+ *+ * `uninstall_command_failed` and `install_command_failed` both occur on an+ * update — a host with no update verb removes the connector and puts it back,+ * and which half failed is the difference between "nothing changed" and "the+ * agent now has no connector at all". `files_write_failed` and+ * `files_remove_failed` draw that line for the connector the app writes itself.+ *+ * `error` is the only code not named at a branch, and deliberately a small+ * residue. One that starts filling up means a wall people are hitting has no+ * name yet — not a bucket to widen.+ */+export type TelemetryConnectorFailure =+ | 'none'+ /** The agent declares a connector but nothing could be resolved to manage it. */+ | 'unsupported'+ /**+ * The agent's own CLI is not on the machine, so there is nothing to drive.+ *+ * Every command a marketplace-driven connector runs goes through that binary,+ * so the first one fails and the rest are consequences. Without this the+ * whole condition landed under whichever verb ran first — `marketplace_failed`+ * for an install, `uninstall_command_failed` for a removal — which reads as a+ * fault in the marketplace rather than as an agent nobody installed.+ */+ | 'host_cli_missing'+ /** The plugin marketplace could not be registered or re-pointed at its source. */+ | 'marketplace_failed'+ | 'install_command_failed'+ | 'update_command_failed'+ | 'uninstall_command_failed'+ /** The app writes this connector itself, and the write failed. */+ | 'files_write_failed'+ /** The app writes this connector itself, and removing it failed. */+ | 'files_remove_failed'+ /** The agent declares a file-based connector and implements no behavior for it. */+ | 'files_unimplemented'+ /** The operation threw instead of returning a result. */+ | 'error';++/**+ * Who asked for a connector update.+ *+ * `catch_up` is the once-per-install sweep that brings every installed+ * connector to the version this build ships. It goes through the same service+ * method as the Update button, so without this the two are one number — and it+ * runs unattended at launch, several connectors at a time, so its latencies and+ * failures would be read as ones people sat through.+ *+ * The same reasoning as `TelemetryAgentRemoveTrigger`, which exists because a+ * server teardown deleting a dozen agents looked identical to a dozen people+ * giving up on them.+ */+export type TelemetryConnectorUpdateTrigger = 'user' | 'catch_up';++/**+ * How long an operation took, in whole milliseconds, on a monotonic clock.+ *+ * A number rather than a value from a fixed set, like `agent_count` and+ * `result_count`, so how to read it matters.+ * Nothing is bucketed and nothing is clamped: take percentiles, not means —+ * some of these legitimately include a password prompt left on screen.+ *+ * Two things shape the distribution:+ *+ * - Every connector command carries `EXEC_TIMEOUT_MS` (120 s) and an operation+ * runs several in sequence, so the wall is a multiple: ~480000 for an+ * install, ~600000 for a reinstall. Pile-ups there are the timeout, not+ * latency.+ * - The connector events time the whole operation, including registering the+ * marketplace — which clones a repository on a machine that has never had it+ * and is a no-op afterwards. The first install is legitimately much slower,+ * so a moving p50 can be a change in that mix rather than a regression.+ *+ * Branded so `startTimer()` is the only thing that can mint one. That is what+ * stops a call site reaching for `Date.now() - startedAt`, which is not+ * monotonic and across a clock step yields a negative number nothing at the far+ * end can tell from data.+ */+declare const durationMsBrand: unique symbol;+export type TelemetryDurationMs = number & { readonly [durationMsBrand]: true };+/*** Which messaging platform a room or bridge is on.*@@ -319,10 +408,23 @@ export type TelemetryEventMap = {server_kind: 'local' | 'remote_managed' | 'external';outcome: TelemetryOutcome;};+ /**+ * A Switch connector was installed.+ *+ * `agent_type` is also what says *which kind* of connector ran: a host with a+ * plugin marketplace is driven through its CLI, and a host without one has its+ * connector written by the app. The two fail in entirely different places,+ * which is what `failure_reason` separates.+ *+ * The denominator is every attempt: an operation that throws is caught at the+ * driver boundary and reported as `error` rather than escaping unreported.+ */connector_installed: {agent_type: TelemetryAgentType;target: TelemetryLocationKind;outcome: 'success' | 'failure';+ failure_reason: TelemetryConnectorFailure;+ duration_ms: TelemetryDurationMs;};/*** The app checked for an update. `trigger` separates a check someone asked for@@ -379,6 +481,9 @@ export type TelemetryEventMap = {target: 'local' | 'remote';outcome: TelemetryOutcome;was_reinstall: boolean;+ trigger: TelemetryConnectorUpdateTrigger;+ failure_reason: TelemetryConnectorFailure;+ duration_ms: TelemetryDurationMs;};/*** The connector was removed. The churn signal.@@ -390,6 +495,8 @@ export type TelemetryEventMap = {agent_type: TelemetryAgentType;target: 'local';outcome: TelemetryOutcome;+ failure_reason: TelemetryConnectorFailure;+ duration_ms: TelemetryDurationMs;};/*** An agent was removed. `delete_in_switch` says whether its identity on the@@ -453,6 +560,14 @@ export type TelemetryEventMap = {* connector, which `connector_installed` and friends report.** The single biggest wall a new user hits, and until now entirely uncounted.+ *+ * `duration_ms` matters most here: a CLI install that takes four minutes and+ * one that takes ten seconds are otherwise the same row, and a package manager+ * getting slower is the kind of regression nobody reports because it never+ * fails.+ *+ * A dependency manager that throws instead of returning a result is reported+ * as `error` by `reportedCliAction`, so every attempt is in the denominator.*/agent_cli_action: {agent_type: TelemetryAgentType;@@ -461,6 +576,7 @@ export type TelemetryEventMap = {action: TelemetryCliAction;outcome: TelemetryOutcome;failure_reason: TelemetryCliFailure;+ duration_ms: TelemetryDurationMs;};/*** A room was created on a server. `bridge_unavailable` is the failure worth@@ -605,15 +721,23 @@ export const TELEMETRY_EVENT_PROPERTIES = {],session_ended: ['agent_type', 'location', 'outcome'],server_added: ['server_kind', 'outcome'],- connector_installed: ['agent_type', 'target', 'outcome'],+ connector_installed: ['agent_type', 'target', 'outcome', 'failure_reason', 'duration_ms'],update_checked: ['trigger', 'result'],update_downloaded: ['outcome'],update_install_started: ['outcome'],bridge_connected: ['bridge_platform', 'outcome', 'failure_reason'],bridge_disconnected: ['bridge_platform', 'outcome'],bridge_identity_claimed: ['bridge_platform', 'outcome'],- connector_updated: ['agent_type', 'target', 'outcome', 'was_reinstall'],- connector_uninstalled: ['agent_type', 'target', 'outcome'],+ connector_updated: [+ 'agent_type',+ 'target',+ 'outcome',+ 'was_reinstall',+ 'trigger',+ 'failure_reason',+ 'duration_ms',+ ],+ connector_uninstalled: ['agent_type', 'target', 'outcome', 'failure_reason', 'duration_ms'],agent_removed: ['agent_type','location',@@ -634,6 +758,7 @@ export const TELEMETRY_EVENT_PROPERTIES = {'action','outcome','failure_reason',+ 'duration_ms',],room_created: ['server_kind',diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/relay-client.test.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/relay-client.test.tsindex 73286b1a7..8acda963f 100644--- a/console/apps/switch-console-desktop/src/main/core/telemetry/relay-client.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/relay-client.test.ts@@ -1,3 +1,4 @@+import { durationMs } from '@tooling/utils/telemetry-duration';import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';import type { TelemetryConfig } from './config';import {@@ -194,7 +195,13 @@ describe('the record that gets built', () => {// The guard drops any record with more than 128 attributes.const payload = buildOtlpPayload('connector_installed',- { agent_type: 'claude', target: 'remote', outcome: 'success' },+ {+ agent_type: 'claude',+ target: 'remote',+ outcome: 'success',+ failure_reason: 'none',+ duration_ms: durationMs(4200),+ },CONTEXT);diff --git a/console/apps/switch-console-desktop/src/main/core/telemetry/telemetry-service.test.ts b/console/apps/switch-console-desktop/src/main/core/telemetry/telemetry-service.test.tsindex b8deb3485..190acfca4 100644--- a/console/apps/switch-console-desktop/src/main/core/telemetry/telemetry-service.test.ts+++ b/console/apps/switch-console-desktop/src/main/core/telemetry/telemetry-service.test.ts@@ -8,6 +8,7 @@ vi.mock('@main/core/app/utils', () => ({ resolveAppVersion: vi.fn(async () => '1vi.mock('./consent', () => ({ isTelemetryAllowed: vi.fn() }));vi.mock('./install-id', () => ({ getInstallId: vi.fn(async () => 'install-abc') }));+import { durationMs } from '@tooling/utils/telemetry-duration';import { log } from '@main/lib/logger';import { isTelemetryAllowed } from './consent';import { telemetryService, trackEvent } from './telemetry-service';@@ -19,7 +20,21 @@ function sentBody(): Record<string, unknown> {return JSON.parse(init.body) as Record<string, unknown>;}-type Attribute = { key: string; value: { stringValue: string } };+type Attribute = {+ key: string;+ value: { stringValue?: string; doubleValue?: number; boolValue?: boolean };+};++/**+ * The value out of an OTLP attribute, whichever of the three shapes it took.+ *+ * Reading `stringValue` alone would make every number and boolean in a payload+ * read back as `undefined` — so an assertion that a numeric property is carried+ * would pass against a payload that dropped it.+ */+function attributeValue(attribute: Attribute): string | number | boolean | undefined {+ return attribute.value.stringValue ?? attribute.value.doubleValue ?? attribute.value.boolValue;+}function sentRecord(): Record<string, unknown> {const resourceLogs = sentBody().resourceLogs as [@@ -28,16 +43,17 @@ function sentRecord(): Record<string, unknown> {return resourceLogs[0].scopeLogs[0].logRecords[0];}-function sentResource(): Record<string, string> {+/** Resource attributes are strings by construction, unlike the record's own. */+function sentResource(): Record<string, string | undefined> {const resourceLogs = sentBody().resourceLogs as [{ resource: { attributes: Attribute[] } }];return Object.fromEntries(resourceLogs[0].resource.attributes.map((a) => [a.key, a.value.stringValue]));}-function sentAttributes(): Record<string, string> {+function sentAttributes(): Record<string, string | number | boolean | undefined> {return Object.fromEntries(- (sentRecord().attributes as Attribute[]).map((a) => [a.key, a.value.stringValue])+ (sentRecord().attributes as Attribute[]).map((a) => [a.key, attributeValue(a)]));}@@ -212,6 +228,8 @@ describe('the payload', () => {agent_type: 'claude',target: 'remote',outcome: 'failure',+ failure_reason: 'install_command_failed',+ duration_ms: durationMs(4200),});expect(sentResource()['service.version']).toBe('1.2.3');@@ -220,6 +238,10 @@ describe('the payload', () => {agent_type: 'claude',target: 'remote',outcome: 'failure',+ failure_reason: 'install_command_failed',+ // A duration goes as a number, not as text: `4200` and not `"4200"`, so+ // the far end can average it without parsing it back.+ duration_ms: durationMs(4200),build: 'dev',});});diff --git a/console/apps/switch-console-desktop/tooling/utils/telemetry-duration.ts b/console/apps/switch-console-desktop/tooling/utils/telemetry-duration.tsnew file mode 100644index 000000000..b159e44c9--- /dev/null+++ b/console/apps/switch-console-desktop/tooling/utils/telemetry-duration.ts@@ -0,0 +1,42 @@+import type { TelemetryDurationMs } from '@main/core/telemetry/events';++/**+ * Duration helpers for tests.+ *+ * Under `tooling/` rather than beside the emitter, because the brand on+ * `TelemetryDurationMs` is only worth having if nothing else can apply it:+ * `durationMs` below is exactly the cast `startTimer()` exists to be the one+ * holder of. Here, `switch-console/no-tooling-imports` makes importing it from+ * the main or preload process a lint error, and `@tooling` is not aliased in+ * `electron.vite.config.ts` at all — so a call site reaching for it fails twice+ * over, rather than compiling and minting a duration no clock produced.+ */++/**+ * A duration for a test that needs a specific number.+ *+ * `startTimer()` is the only thing that mints a `TelemetryDurationMs` in+ * production, which is the point of the brand. A test asserting on an exact+ * value cannot go through a real clock, so it says so here rather than+ * scattering casts through the suites.+ */+export function durationMs(value: number): TelemetryDurationMs {+ return value as TelemetryDurationMs;+}++/**+ * Matches a duration a real timer could have produced.+ *+ * `expect.any(Number)` is satisfied by `NaN` and `Infinity`, both of which the+ * emitter refuses at send time (`allowedProperties` throws on a non-finite+ * value) — so an assertion written with it passes against a payload that would+ * never reach the far end, which is the opposite of what the assertion is for.+ * This holds the value to what `startTimer()` guarantees: a whole number of+ * milliseconds that cannot be negative.+ */+export const aDurationMs = {+ asymmetricMatch: (value: unknown): boolean =>+ typeof value === 'number' && Number.isInteger(value) && value >= 0,+ toString: (): string => 'aDurationMs',+ getExpectedType: (): string => 'number',+};diff --git a/core/switch_core/bridges/agent/protocol/connections.py b/core/switch_core/bridges/agent/protocol/connections.pyindex 8e454a2b9..1362e3222 100644--- a/core/switch_core/bridges/agent/protocol/connections.py+++ b/core/switch_core/bridges/agent/protocol/connections.py@@ -30,6 +30,7 @@ from dataclasses import dataclass, fieldfrom typing import Literalfrom switch_core.artifacts import contract_range+from switch_core.logging_context import log_contextfrom switch_core.observability.catalogue import AGENT_CONNECTIONS_EXPIREDfrom switch_core.observability.metrics import metrics@@ -380,14 +381,21 @@ class ConnectionRegistry:# (beats=0 — it is not running the heartbeat, or cannot reach us) versus# one that beat and then stopped (beats>0 — it went away, or the server# was too busy to process ticks).- logger.info(- "[CONN] closed agent=%s connection=%s reason=%s beats=%d last_beat_age=%.1fs",- conn.agent_id,- connection_id,- reason,- conn.beats,- time.monotonic() - conn.last_beat,- )+ # `agent_id` as a field, not only in the message: a connection expiring+ # is swept by the registry's own reaper rather than by anything the+ # agent called, so there is no request context for it to inherit — and+ # this is the line somebody goes looking for when one agent will not+ # stay connected.+ with log_context(agent_id=conn.agent_id):+ logger.info(+ "[CONN] closed agent=%s connection=%s reason=%s beats=%d "+ "last_beat_age=%.1fs",+ conn.agent_id,+ connection_id,+ reason,+ conn.beats,+ time.monotonic() - conn.last_beat,+ )# After the bookkeeping and the log, so an observer sees the closed# state. Guarded because the registry's own contract — the connection# is closed and the caller gets it back — must not depend on whoeverdiff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.pyindex d37c8fdef..9b7242af4 100644--- a/core/switch_core/bridges/agent/protocol/event_buffer.py+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py@@ -29,6 +29,7 @@ from collections import dequefrom dataclasses import dataclassfrom switch_core.bridges.agent.protocol.types import AgentEvent+from switch_core.logging_context import log_contextfrom switch_core.observability.catalogue import AGENT_EVENTS_DROPPEDfrom switch_core.observability.metrics import metrics@@ -484,14 +485,20 @@ class EventBuffer:for _ in range(overflow):dropped_seq = events.popleft().seqmetrics().increment(AGENT_EVENTS_DROPPED, {"reason": "overflow"}, overflow)- logger.warning(- "[EVENT-BUF] agent=%s exceeded %s buffered events; dropped "- "through seq=%s — readers resuming from before this will be "- "told they missed events",- agent_id,- self._max_events,- dropped_seq,- )+ # The agent is a field as well as being in the message: this fires+ # from the buffer's own bookkeeping rather than from anything the+ # agent called, so there is no request context to inherit it from,+ # and "which agent is falling behind" is the only question this+ # line is ever read to answer.+ with log_context(agent_id=agent_id):+ logger.warning(+ "[EVENT-BUF] agent=%s exceeded %s buffered events; dropped "+ "through seq=%s — readers resuming from before this will be "+ "told they missed events",+ agent_id,+ self._max_events,+ dropped_seq,+ )if dropped_seq:self._dropped_through[agent_id] = max(diff --git a/core/switch_core/bridges/agent/protocol/service.py b/core/switch_core/bridges/agent/protocol/service.pyindex 71a9fd7ac..bc261c768 100644--- a/core/switch_core/bridges/agent/protocol/service.py+++ b/core/switch_core/bridges/agent/protocol/service.py@@ -101,6 +101,7 @@ from switch_core.events import (from switch_core.messages.recorded_types import MEMBERSHIP_EVENT_TYPEfrom switch_core.sessions.attachments import normalise_mime_typefrom switch_core.telemetry import TelemetryService, emit_safely+from switch_core.telemetry.ages import age_daysfrom switch_core.telemetry.snapshot import normalise_known_agent_typefrom switch_core.tenant_context import current_tenant_id, tenant_scopefrom switch_core.transport import (@@ -132,19 +133,6 @@ if TYPE_CHECKING:logger = logging.getLogger(__name__)-def _age_days(created_at: object) -> float:- """How old a row is, in days, for reporting. Zero if unknown.-- Takes `object` because the timestamp columns are annotated `Mapped[str]`- while carrying real `datetime`s, so the honest signature is "whatever the- column hands back", checked here rather than trusted.- """- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 86400.0, 0.0)--# \A and \Z rather than ^ and $: Python's $ also matches before a single# trailing newline, which would let an identifier carry a line break._VALID_NAME_RE = re.compile(r"\A[a-z0-9][a-z0-9._-]*\Z")@@ -944,7 +932,7 @@ class ProtocolService:# runtime or its age says almost nothing.removed: dict[str, str | int | float | bool] = {"known_agent_type": normalise_known_agent_type(agent.metadata_),- "age_days": _age_days(agent.created_at),+ "age_days": age_days(agent.created_at),"had_parent": agent.parent_agent_id is not None,}removed["room_count"] = await self._room_count_for(resolved_id)diff --git a/core/switch_core/bridges/agent/server_connectors/lifecycle.py b/core/switch_core/bridges/agent/server_connectors/lifecycle.pyindex 5b5200d4a..0f9d2a3e8 100644--- a/core/switch_core/bridges/agent/server_connectors/lifecycle.py+++ b/core/switch_core/bridges/agent/server_connectors/lifecycle.py@@ -146,14 +146,19 @@ class ServerSideConnectorLifecycleService:await self._connector_store.create(session, record)await session.commit()- await self.start(record.id)-+ # Before `start`, not after: the rows above are durably committed and+ # are the fact this event describes, while `start` is a post-commit+ # side effect (decrypting the token, validating the connector's config,+ # dialling out) that can raise without undoing the registration. A+ # failure there still propagates to the caller unchanged.emit_safely(self._telemetry,"server_connector_registered",{"connector_kind": _connector_kind(connector_type)},)+ await self.start(record.id)+logger.info("Registered server-side connector %s (%s): %s (owner: %s)",record.id,diff --git a/core/switch_core/bridges/collaboration/bridge_core.py b/core/switch_core/bridges/collaboration/bridge_core.pyindex f2dc7774e..3b25b3b27 100644--- a/core/switch_core/bridges/collaboration/bridge_core.py+++ b/core/switch_core/bridges/collaboration/bridge_core.py@@ -3,6 +3,7 @@ from __future__ import annotationsimport asyncioimport loggingimport re+import timeimport uuidfrom collections.abc import Awaitable, Callable, Iterator, Mappingfrom contextlib import contextmanager@@ -61,6 +62,7 @@ from switch_core.db.stores.session_request_post_store import SessionRequestPostSfrom switch_core.db.tenant_lookup import tenant_of_roomfrom switch_core.logging_context import log_contextfrom switch_core.observability.catalogue import (+ BRIDGE_CALL_DURATION,BRIDGE_ERRORS,BRIDGE_EVENTS_IN,BRIDGE_EVENTS_OUT,@@ -338,16 +340,27 @@ class BridgeCore:@contextmanagerdef _counted_outbound(self, kind: str) -> Iterator[None]:- """Count one relay out to the platform, and its failure if it fails.+ """Count one relay out to the platform, time it, and count its failure.Placed around the relay call rather than at the top of the handler: ahandler returns early for a puppet's own echo and for a room with nochannel mapping, and neither of those is a message anybody sentoutwards.++ The duration is the platform's round trip, and it is the only place in+ this process where an external API's latency is visible at all. A room+ that feels unresponsive is usually Slack taking two seconds rather than+ anything here being wrong, and without this the two are the same+ picture — a healthy server, relays being counted, and people waiting.++ Only a relay that succeeded is timed. A call that raised took however+ long its own failure took, which is a different distribution living+ under the same name; `switch.bridge.errors` is where that shows."""metrics().increment(BRIDGE_EVENTS_OUT, {"platform": self._bridge_type, "kind": kind})+ started = time.perf_counter()try:yieldexcept Exception:@@ -355,6 +368,11 @@ class BridgeCore:BRIDGE_ERRORS, {"platform": self._bridge_type, "direction": "outbound"})raise+ metrics().observe(+ BRIDGE_CALL_DURATION,+ {"platform": self._bridge_type, "kind": kind},+ (time.perf_counter() - started) * 1000.0,+ )def _traced(self, kind: str, handler: Callable[[_InboundEventT], Awaitable[None]]@@ -397,7 +415,12 @@ class BridgeCore:if room_ids is Noneelse await self._room_tenant(room_ids[0]))- with tenant_scope(tenant_id):+ # `room_ids` is (Switch room id, transport room id); the log+ # field is the first. None until the mapping exists, which is+ # the honest reading for a channel whose room the handler is+ # about to create.+ room_id = room_ids[0] if room_ids else None+ with tenant_scope(tenant_id), log_context(room_id=room_id):try:await handler(event)except Exception:diff --git a/core/switch_core/bridges/collaboration/lifecycle_service.py b/core/switch_core/bridges/collaboration/lifecycle_service.pyindex 9c4a50936..0ff041d0d 100644--- a/core/switch_core/bridges/collaboration/lifecycle_service.py+++ b/core/switch_core/bridges/collaboration/lifecycle_service.py@@ -3,16 +3,22 @@ from __future__ import annotationsimport asyncioimport loggingimport re-from datetime import UTC, datetime+import timefrom typing import TYPE_CHECKINGfrom uuid import uuid4+import aiohttp+import httpxfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmakerfrom switch_core.bridges.collaboration.adapter import CollaborationAdapterfrom switch_core.bridges.collaboration.bridge_core import BridgeCorefrom switch_core.bridges.collaboration.ingress import CallbackEndpoint, CallbackIngress-from switch_core.bridges.collaboration.models import BridgeConnectionConfig+from switch_core.bridges.collaboration.models import (+ BridgeConnectionConfig,+ BridgeCredentialError,+ BridgeOperationError,+)from switch_core.clients.bridge_client import BridgeClient, BridgeClientConfigfrom switch_core.clients.client_factory import ClientFactoryfrom switch_core.config import SwitchConfig@@ -29,54 +35,231 @@ from switch_core.db.tenant_lookup import all_tenant_ids, tenant_of_collaborationfrom switch_core.deeplinks import gateway_url_warningfrom switch_core.provisioning import Provisioningfrom switch_core.telemetry import TelemetryService, emit_safely+from switch_core.telemetry.ages import UNKNOWN_AGE, age_days, seconds_sincefrom switch_core.telemetry.deployment import (claim_milestone,milestone_claimed,seconds_since_install,)-from switch_core.telemetry.snapshot import normalise_platform+from switch_core.telemetry.snapshot import PLATFORMS, normalise_platformfrom switch_core.tenant_context import current_tenant_id, no_tenantif TYPE_CHECKING:from switch_core.clients.client_lifecycle_service import ClientLifecycleServicefrom switch_core.room_service import RoomService+# Every platform SDK below is registered dynamically (`register_adapter`), so a+# deployment that only wires up some of the five is plausible even though+# every one is a hard dependency of switch-core today — hence guarded rather+# than assumed. A dependency that is genuinely missing degrades classification+# to `unknown` instead of taking the whole classifier down with an ImportError,+# which would turn a bridge failure into a second, worse one.+try:+ from slack_sdk.errors import SlackApiError+except ImportError: # pragma: no cover - exercised only without slack-sdk installed+ SlackApiError = None # type: ignore[assignment, misc]++try:+ from discord.errors import DiscordException+ from discord.errors import LoginFailure as DiscordLoginFailure+except ImportError: # pragma: no cover+ DiscordException = None # type: ignore[assignment, misc]+ DiscordLoginFailure = None # type: ignore[assignment, misc]++try:+ from telegram.error import BadRequest as TelegramBadRequest+ from telegram.error import ChatMigrated as TelegramChatMigrated+ from telegram.error import Forbidden as TelegramForbidden+ from telegram.error import InvalidToken as TelegramInvalidToken+ from telegram.error import NetworkError as TelegramNetworkError+ from telegram.error import TelegramError+except ImportError: # pragma: no cover+ TelegramBadRequest = None # type: ignore[assignment, misc]+ TelegramChatMigrated = None # type: ignore[assignment, misc]+ TelegramForbidden = None # type: ignore[assignment, misc]+ TelegramInvalidToken = None # type: ignore[assignment, misc]+ TelegramNetworkError = None # type: ignore[assignment, misc]+ TelegramError = None # type: ignore[assignment, misc]++try:+ from mattermostdriver.exceptions import (+ NoAccessTokenProvided as MattermostNoAccessTokenProvided,+ )+ from mattermostdriver.exceptions import (+ NotEnoughPermissions as MattermostNotEnoughPermissions,+ )++ # `mattermostdriver` raises its named exceptions only for the status codes+ # it maps; anything else (including a bare connection failure) surfaces as+ # the underlying `requests` exception, since that is the client it wraps.+ from requests.exceptions import ConnectionError as RequestsConnectionError+ from requests.exceptions import HTTPError as RequestsHTTPError+ from requests.exceptions import InvalidJSONError as RequestsInvalidJSON+ from requests.exceptions import Timeout as RequestsTimeout+except ImportError: # pragma: no cover+ MattermostNoAccessTokenProvided = None # type: ignore[assignment, misc]+ MattermostNotEnoughPermissions = None # type: ignore[assignment, misc]+ RequestsConnectionError = None # type: ignore[assignment, misc]+ RequestsHTTPError = None # type: ignore[assignment, misc]+ RequestsInvalidJSON = None # type: ignore[assignment, misc]+ RequestsTimeout = None # type: ignore[assignment, misc]+logger = logging.getLogger(__name__)+# Slack error codes that mean the credentials themselves are rejected, as+# opposed to Slack refusing a call for some other reason (rate limited, a+# scope not granted, an internal error). `SlackApiError` carries no separate+# type for this — the distinction lives entirely in `response["error"]`.+_SLACK_AUTH_ERROR_CODES = frozenset(+ {+ "invalid_auth",+ "not_authed",+ "account_inactive",+ "token_revoked",+ "token_expired",+ }+)++# Telegram's own hierarchy already encodes "the server understood the request+# and refused it" for these three. `BadRequest` is a subclass of+# `NetworkError`, so they are checked ahead of `_NETWORK_EXCEPTIONS` below —+# the same ordering `telegram/adapter.py`'s `_as_rich_failure` uses, and for+# the same reason: checking `NetworkError` first would report a definite+# refusal as an unreachable network.+_TELEGRAM_DEFINITE_REFUSALS = tuple(+ exc_type+ for exc_type in (TelegramBadRequest, TelegramForbidden, TelegramChatMigrated)+ if exc_type is not None+)++# Reaching the platform failed outright, across every transport an adapter+# uses: aiohttp (Slack's Socket Mode, Teams' inbound listener), httpx (Teams'+# token exchange and Graph calls) and requests (Mattermost, via+# `mattermostdriver`, which raises its own named exceptions only for the+# status codes it maps and lets a connection failure surface as the+# underlying `requests` exception unchanged).+#+# Each library's own base class for "the transport failed", not the individual+# leaves. A timeout is the case that makes this worth stating: `ConnectTimeout`+# is a `TimeoutException` rather than a `ConnectError` in httpx, and+# `ServerTimeoutError` is not a `ClientOSError` in aiohttp — so a list of+# leaves classifies a refusal as `network` and the timeout beside it as+# `unknown`, which are the two outcomes an operator most needs to tell apart.+_NETWORK_EXCEPTIONS = tuple(+ exc_type+ for exc_type in (+ TelegramNetworkError,+ aiohttp.ClientConnectionError,+ httpx.TransportError,+ RequestsConnectionError,+ RequestsTimeout,+ # The two builtins, for a failure that reaches here without a library's+ # name on it: `asyncio.wait_for` raises the first, and a raw socket+ # connect the second.+ TimeoutError,+ ConnectionError,+ )+ if exc_type is not None+)++_MATTERMOST_AUTH_ERRORS = tuple(+ exc_type+ for exc_type in (MattermostNoAccessTokenProvided, MattermostNotEnoughPermissions)+ if exc_type is not None+)+++def _slack_failure_reason(exc: SlackApiError) -> str:+ """`auth_failed` for a rejected token, `platform_error` for everything else+ Slack refuses a call for (rate limits, a missing scope, an outage). Both+ arrive as the same `SlackApiError`, so the code inside the response — not+ the exception's type — is what tells them apart.++ A revoked or rotated bot token is the most common bridge failure in the+ field, and reporting it as `platform_error` would send an operator to+ Slack's status page instead of their own token.+ """+ # `.response` is not always a mapping. On the async client slack_sdk+ # raises `SlackApiError(message, res)` with the raw `aiohttp.ClientResponse`+ # whenever the body it was handed is not the JSON the content type claimed+ # — an empty 502, a proxy's error page. That object has no `.get`, and this+ # is evaluated inside the argument list of the `emit_safely` that reports+ # the failure, so an `AttributeError` here would replace the bridge's real+ # exception with a meaningless one *and* suppress the event. Nor does a+ # `.get` promise a mapping: `AsyncSlackResponse.get` itself raises when+ # the body it holds is text, as a proxy's plain-text 5xx is.+ response = getattr(exc, "response", None)+ reader = getattr(response, "get", None)+ try:+ code = reader("error") if callable(reader) else None+ except Exception:+ code = None+ if code in _SLACK_AUTH_ERROR_CODES:+ return "auth_failed"+ return "platform_error"++def _failure_reason(exc: BaseException) -> str:"""An enumerated reason for a bridge failure.Not the exception's message, which routinely carries a workspace name or a- token fragment. Matched on the class name so no adapter has to be imported.+ token fragment. Classified by type, and — where a library folds several+ outcomes into one exception class — by what the exception itself carries.+ Never by matching words in the class name: "SlackApiError" contains "api",+ which reads a revoked token as a platform fault.++ `KeyError` and `sqlalchemy.exc.DBAPIError` are deliberately left+ unclassified and fall through to `unknown`. A `KeyError` here is adapter+ code reading a platform payload whose shape changed — a bug in Switch, not+ a value the operator typed into their connection config. A `DBAPIError` is+ Switch's own database, not the messaging platform."""- name = type(exc).__name__.lower()- if any(word in name for word in ("auth", "unauthorized", "forbidden", "token")):+ if isinstance(exc, BridgeCredentialError):return "auth_failed"- if any(word in name for word in ("timeout", "connection", "socket", "dns")):- return "network"- if isinstance(exc, ValueError | KeyError):- return "config_invalid"- if any(word in name for word in ("api", "http", "server", "gateway")):++ if SlackApiError is not None and isinstance(exc, SlackApiError):+ return _slack_failure_reason(exc)++ if DiscordLoginFailure is not None and isinstance(exc, DiscordLoginFailure):+ return "auth_failed"++ if isinstance(exc, _MATTERMOST_AUTH_ERRORS):+ return "auth_failed"++ if TelegramInvalidToken is not None and isinstance(exc, TelegramInvalidToken):+ return "auth_failed"++ if isinstance(exc, _TELEGRAM_DEFINITE_REFUSALS):return "platform_error"- return "unknown"+ if isinstance(exc, _NETWORK_EXCEPTIONS):+ return "network"++ # These four are each a platform's own SDK saying it was reached and it+ # refused — never Switch's homeserver or database, which speak neither+ # vendor's exception language, so the label stays accurate even though the+ # check is broad.+ if isinstance(exc, BridgeOperationError):+ return "platform_error"+ if DiscordException is not None and isinstance(exc, DiscordException):+ return "platform_error"+ if RequestsHTTPError is not None and isinstance(exc, RequestsHTTPError):+ return "platform_error"+ if TelegramError is not None and isinstance(exc, TelegramError):+ return "platform_error"-def _age_days(moment: object) -> float:- """How old a row is, in days, for reporting. Zero if unknown."""- if not isinstance(moment, datetime):- return 0.0- anchored = moment if moment.tzinfo else moment.replace(tzinfo=UTC)- return max((datetime.now(UTC) - anchored).total_seconds() / 86400.0, 0.0)+ # Ahead of the `ValueError` below, which it is one of: `requests` folds a+ # body it could not parse into `InvalidJSONError`, and a Mattermost server+ # answering with a proxy error page is the platform misbehaving, not a+ # connection config somebody typed wrong.+ if RequestsInvalidJSON is not None and isinstance(exc, RequestsInvalidJSON):+ return "platform_error"+ if isinstance(exc, ValueError):+ return "config_invalid"-def _seconds_since(moment: object) -> float:- """Seconds since a timestamp column, or -1 when it is not one — so "could- not tell" is distinguishable from "just now"."""- if not isinstance(moment, datetime):- return -1.0- anchored = moment if moment.tzinfo else moment.replace(tzinfo=UTC)- return max((datetime.now(UTC) - anchored).total_seconds(), 0.0)+ return "unknown"def _bridge_client_localpart(bridge_type: str, display_name: str) -> str:@@ -145,6 +328,11 @@ class CollaborationBridgeLifecycleService:# How many times each bridge has failed to come up since this# process started. See `_note_connect_failure`.self._connect_failures: dict[str, int] = {}+ # bridge_id -> a monotonic reading taken when its connect attempt+ # began. Spans `start()` and the task it schedules, because those are+ # two halves of one attempt from the operator's side. See+ # `_connect_duration_ms`.+ self._connect_started: dict[str, float] = {}# Read off the row at start, so reporting never depends on what the# BridgeCore exposes.self._bridge_facts: dict[str, tuple[str, object]] = {}@@ -158,6 +346,11 @@ class CollaborationBridgeLifecycleService:# `_bridges` and leaves it here, which is what makes "configured but no# longer running" answerable.self._started: set[str] = set()+ # Every platform a bridge has been started for in this process, never+ # removed. `running_by_platform` reports zero for each, so a platform's+ # series survives its last bridge being stopped rather than ending —+ # see the note there on why an absent series is the worst answer.+ self._platforms_seen: set[str] = set()# The one listener every bridge that gets called back shares, and each# running bridge's place on it. Owned here rather than by an adapter# because the port is the process's, not a bridge's: two Mattermost@@ -519,122 +712,178 @@ class CollaborationBridgeLifecycleService:return bridgeasync def start(self, bridge_id: str) -> None:- # Two steps, because this is the point where the bridge's tenant is- # not yet known: the exemption answers which tenant the id is in- # (`db/tenant_lookup.py`), and the row itself is then read scoped to- # it. Reached both from boot, with nothing bound, and from an HTTP- # request, where what is bound is the caller's tenant and not- # necessarily the bridge's — so this deliberately does not inherit.- tenant_id = await tenant_of_collaboration_bridge(- self._session_factory, bridge_id- )- if tenant_id is None:- raise ValueError(f"Bridge not found: {bridge_id}")- async with tenant_session(self._session_factory, tenant_id) as session:- bridge = await self._bridge_store.get(session, bridge_id)- if bridge is None:- raise ValueError(f"Bridge not found: {bridge_id}")-- adapter_cls = self._adapter_registry.get(bridge.type)- config_cls = self._config_registry.get(bridge.type)- if adapter_cls is None or config_cls is None:- raise ValueError(f"Unknown bridge type: {bridge.type}")-- # Registration refuses a conflicting bridge, but rows predating that- # check still exist, and start_all would otherwise walk into the bind- # error one of them causes. Say which bridge holds it instead.- wanted = adapter_cls.exclusive_resource(bridge.connection_config or {})- if wanted is not None:- for other_id, held in self._held_resources.items():- if held == wanted and other_id != bridge_id:- raise ValueError(- f"Cannot start bridge {bridge_id} ({bridge.type}): "- f"{wanted} is already held by bridge {other_id}. Only "- "one of them can run; delete one, or give it a "- "different listen_port."- )-- typed_config = config_cls.model_validate(bridge.connection_config or {})- adapter = adapter_cls(config=typed_config) # type: ignore[call-arg]- adapter.set_service_url_persister(- lambda service_url: self._persist_service_url(- bridge_id, tenant_id, service_url+ """Start a bridge, reporting a failure even before one is ever run.++ Everything here runs before `_run_bridge` is scheduled, so nothing+ downstream reports a raise from this method. Unreported it would be an+ attempt counted in neither the numerator nor the denominator of the+ connect success rate — and the attempts that fail here are the ones+ that failed hardest. The `try` below reports and re-raises unchanged,+ so the caller (an HTTP handler, `start_all`, `restart`) is unaffected.++ Once `_run_bridge` is scheduled this method returns without waiting on+ it, so there is no window where both this method and that task's own+ handler could report the same attempt.+ """+ # `bridge_platform` is required on every `bridge_connected` event, but+ # a bridge whose row cannot even be read has no platform to name.+ # `unknown`, not `none`: every bridge here is on some platform, so+ # `none` would be a claim — and the one this catalogue reserves for a+ # room with no bridge at all.+ platform = "unknown"+ # Before the first statement that can fail, so a bridge that never gets+ # past reading its own row still reports how long that took.+ self._connect_started[bridge_id] = time.monotonic()+ try:+ # Two steps, because this is the point where the bridge's tenant+ # is not yet known: the exemption answers which tenant the id is+ # in (`db/tenant_lookup.py`), and the row itself is then read+ # scoped to it. Reached both from boot, with nothing bound, and+ # from an HTTP request, where what is bound is the caller's+ # tenant and not necessarily the bridge's — so this deliberately+ # does not inherit.+ tenant_id = await tenant_of_collaboration_bridge(+ self._session_factory, bridge_id)- )- adapter.set_channel_team_persister(- lambda channel_id, team_id: self._persist_channel_team(- bridge_id, tenant_id, channel_id, team_id+ if tenant_id is None:+ raise ValueError(f"Bridge not found: {bridge_id}")+ async with tenant_session(self._session_factory, tenant_id) as session:+ bridge = await self._bridge_store.get(session, bridge_id)+ if bridge is None:+ raise ValueError(f"Bridge not found: {bridge_id}")+ # A type the catalogue has no name for is still a bridge on some+ # platform, so it is `unknown` here, not the `none` the normaliser+ # gives it.+ platform = (+ normalise_platform(bridge.type)+ if bridge.type in PLATFORMS+ else "unknown")- )- adapter.set_max_attachment_bytes(self._config.agent_media_max_bytes)- callback_endpoint = self._callback_ingress.endpoint_for(bridge.type, bridge_id)- adapter.set_callback_endpoint(callback_endpoint)- self._callback_endpoints[bridge_id] = callback_endpoint+ adapter_cls = self._adapter_registry.get(bridge.type)+ config_cls = self._config_registry.get(bridge.type)+ if adapter_cls is None or config_cls is None:+ raise ValueError(f"Unknown bridge type: {bridge.type}")++ # Registration refuses a conflicting bridge, but rows predating+ # that check still exist, and start_all would otherwise walk into+ # the bind error one of them causes. Say which bridge holds it+ # instead.+ wanted = adapter_cls.exclusive_resource(bridge.connection_config or {})+ if wanted is not None:+ for other_id, held in self._held_resources.items():+ if held == wanted and other_id != bridge_id:+ raise ValueError(+ f"Cannot start bridge {bridge_id} ({bridge.type}): "+ f"{wanted} is already held by bridge {other_id}. "+ "Only one of them can run; delete one, or give it "+ "a different listen_port."+ )++ typed_config = config_cls.model_validate(bridge.connection_config or {})+ adapter = adapter_cls(config=typed_config) # type: ignore[call-arg]+ adapter.set_service_url_persister(+ lambda service_url: self._persist_service_url(+ bridge_id, tenant_id, service_url+ )+ )+ adapter.set_channel_team_persister(+ lambda channel_id, team_id: self._persist_channel_team(+ bridge_id, tenant_id, channel_id, team_id+ )+ )+ adapter.set_max_attachment_bytes(self._config.agent_media_max_bytes)- gateway_warning = gateway_url_warning(- self._config.gateway_public_url, adapter_cls.renders_custom_url_schemes- )- if gateway_warning:- logger.warning(- "%s (bridge %s, %s)", gateway_warning, bridge_id, bridge.type+ callback_endpoint = self._callback_ingress.endpoint_for(+ bridge.type, bridge_id)+ adapter.set_callback_endpoint(callback_endpoint)+ self._callback_endpoints[bridge_id] = callback_endpoint- async with tenant_session(self._session_factory, tenant_id) as session:- bridge_client_record = await self._client_store.get(- session, bridge.client_id+ gateway_warning = gateway_url_warning(+ self._config.gateway_public_url, adapter_cls.renders_custom_url_schemes)- if bridge_client_record is None:- raise ValueError(f"Bridge client not found: {bridge.client_id}")-- bridge_core = BridgeCore(- bridge_id=bridge_id,- bridge_tenant_id=tenant_id,- bridge_type=bridge.type,- bridge_display_name=bridge.display_name,- adapter=adapter,- room_store=self._room_store,- external_user_store=self._external_user_store,- bridge_message_map_store=self._bridge_message_map_store,- session_request_post_store=self._session_request_post_store,- agent_store=self._agent_store,- client_store=self._client_store,- room_service=self._room_service,- client_lifecycle=self._client_lifecycle,- matrix_admin=self._matrix_admin,- session_factory=self._session_factory,- matrix_server_name=self._config.matrix_server_name,- bridge_client_matrix_user_id=bridge_client_record.matrix_user_id,- max_attachment_bytes=self._config.agent_media_max_bytes,- session_demo_enabled=self._config.session_demo_enabled,- gateway_public_url=self._config.gateway_public_url,- )+ if gateway_warning:+ logger.warning(+ "%s (bridge %s, %s)", gateway_warning, bridge_id, bridge.type+ )- bridge_client = BridgeClient(- bridge_core=bridge_core,- client_id=bridge_client_record.id,- tenant_id=bridge_client_record.tenant_id,- matrix_user_id=bridge_client_record.matrix_user_id,- display_name=bridge_client_record.display_name,- session_factory=self._session_factory,- client_store=self._client_store,- config=BridgeClientConfig(bridge_id=bridge_id),- transport_factory=self._client_factory.transport_for,- )+ async with tenant_session(self._session_factory, tenant_id) as session:+ bridge_client_record = await self._client_store.get(+ session, bridge.client_id+ )+ if bridge_client_record is None:+ raise ValueError(f"Bridge client not found: {bridge.client_id}")++ bridge_core = BridgeCore(+ bridge_id=bridge_id,+ bridge_tenant_id=tenant_id,+ bridge_type=bridge.type,+ bridge_display_name=bridge.display_name,+ adapter=adapter,+ room_store=self._room_store,+ external_user_store=self._external_user_store,+ bridge_message_map_store=self._bridge_message_map_store,+ session_request_post_store=self._session_request_post_store,+ agent_store=self._agent_store,+ client_store=self._client_store,+ room_service=self._room_service,+ client_lifecycle=self._client_lifecycle,+ matrix_admin=self._matrix_admin,+ session_factory=self._session_factory,+ matrix_server_name=self._config.matrix_server_name,+ bridge_client_matrix_user_id=bridge_client_record.matrix_user_id,+ max_attachment_bytes=self._config.agent_media_max_bytes,+ session_demo_enabled=self._config.session_demo_enabled,+ gateway_public_url=self._config.gateway_public_url,+ )- # Stashed rather than passed: `_run_bridge`'s signature is what the- # tenant-binding tests patch.- self._bridge_facts[bridge_id] = (bridge.type, bridge.created_at)- task = asyncio.create_task(- self._run_bridge(bridge_id, tenant_id, bridge_core, bridge_client)- )- self._bridges[bridge_id] = bridge_core- self._tasks[bridge_id] = task- self._started.add(bridge_id)- if wanted is not None:- self._held_resources[bridge_id] = wanted+ bridge_client = BridgeClient(+ bridge_core=bridge_core,+ client_id=bridge_client_record.id,+ tenant_id=bridge_client_record.tenant_id,+ matrix_user_id=bridge_client_record.matrix_user_id,+ display_name=bridge_client_record.display_name,+ session_factory=self._session_factory,+ client_store=self._client_store,+ config=BridgeClientConfig(bridge_id=bridge_id),+ transport_factory=self._client_factory.transport_for,+ )- logger.info("Started collaboration bridge %s (%s)", bridge_id, bridge.type)+ # Stashed rather than passed: `_run_bridge`'s signature is what+ # the tenant-binding tests patch.+ self._bridge_facts[bridge_id] = (bridge.type, bridge.created_at)+ task = asyncio.create_task(+ self._run_bridge(bridge_id, tenant_id, bridge_core, bridge_client)+ )+ self._bridges[bridge_id] = bridge_core+ self._tasks[bridge_id] = task+ self._started.add(bridge_id)+ self._platforms_seen.add(normalise_platform(bridge.type))+ if wanted is not None:+ self._held_resources[bridge_id] = wanted++ logger.info("Started collaboration bridge %s (%s)", bridge_id, bridge.type)+ except Exception as exc:+ self._note_connect_failure(bridge_id)+ # Resolved before the call, never inside its argument list: an+ # expression there is evaluated before `emit_safely` is entered, so+ # anything it raised would escape the guard — replacing the+ # bridge's own exception with a meaningless one, and taking the+ # event this method exists to emit with it.+ reason = _failure_reason(exc)+ duration_ms = self._connect_duration_ms(bridge_id)+ emit_safely(+ self._telemetry,+ "bridge_connected",+ {+ "bridge_platform": platform,+ "outcome": "failure",+ "failure_reason": reason,+ "duration_ms": duration_ms,+ },+ )+ raiseasync def _persist_service_url(self, bridge_id: str, tenant_id: str, service_url: str@@ -737,24 +986,32 @@ class CollaborationBridgeLifecycleService:await endpoint.withdraw()# A failure before the adapter came up never connected at all.self._connected.discard(bridge_id)+ # Resolved here rather than inside the argument lists below,+ # for the reason `start` gives: an expression there runs+ # outside `emit_safely`'s guard, and this one is on the path of+ # a task nobody awaits, where anything it raised would surface+ # only as "Task exception was never retrieved" at collection.+ reason = _failure_reason(exc)if connected:emit_safely(self._telemetry,"bridge_disconnected",{"bridge_platform": normalise_platform(platform),- "reason": _failure_reason(exc),+ "reason": reason,},)else:self._note_connect_failure(bridge_id)+ duration_ms = self._connect_duration_ms(bridge_id)emit_safely(self._telemetry,"bridge_connected",{"bridge_platform": normalise_platform(platform),"outcome": "failure",- "failure_reason": _failure_reason(exc),+ "failure_reason": reason,+ "duration_ms": duration_ms,},)@@ -767,6 +1024,23 @@ class CollaborationBridgeLifecycleService:"""self._connect_failures[bridge_id] = self._connect_failures.get(bridge_id, 0) + 1+ def _connect_duration_ms(self, bridge_id: str) -> float:+ """How long this bridge's connect attempt took, in whole milliseconds.++ Spent on read: an attempt has exactly one outcome, and the next one+ starts its own clock in `start()`. Monotonic rather than the wall+ clock, so an NTP step mid-connect cannot produce a negative number or+ an hour that never passed.++ `-1` when no reading was taken, which should not happen — every path to+ an outcome goes through `start()` — but reporting `0` would assert the+ connect was instantaneous, which is the one thing it certainly was not.+ """+ started = self._connect_started.pop(bridge_id, None)+ if started is None:+ return UNKNOWN_AGE+ return round((time.monotonic() - started) * 1000)+async def _report_connector_up(self, bridge_id: str, platform: str, configured_at: object) -> None:@@ -782,6 +1056,7 @@ class CollaborationBridgeLifecycleService:"bridge_platform": normalise_platform(platform),"outcome": "success","failure_reason": "none",+ "duration_ms": self._connect_duration_ms(bridge_id),},)@@ -814,7 +1089,7 @@ class CollaborationBridgeLifecycleService:"seconds_since_install": (elapsed_since_install if elapsed_since_install is not None else -1.0),- "seconds_since_configured": _seconds_since(configured_at),+ "seconds_since_configured": seconds_since(configured_at),"is_first_connector": not self._any_connector_before(bridge_id),"failed_attempts_before_success": self._connect_failures.pop(bridge_id, 0@@ -848,6 +1123,12 @@ class CollaborationBridgeLifecycleService:self._bridges.pop(bridge_id, None)self._held_resources.pop(bridge_id, None)self._started.discard(bridge_id)+ # A bridge cancelled before it connected never reaches an outcome, and+ # `_run_bridge`'s handler does not catch `CancelledError`, so its+ # reading would otherwise sit here for the life of the process — and be+ # spent by the *next* attempt on the same id, which would then report a+ # duration measured from the one before it.+ self._connect_started.pop(bridge_id, None)logger.info("Stopped collaboration bridge %s", bridge_id)# `_bridges` membership is set before the connection is attempted, so@@ -947,7 +1228,7 @@ class CollaborationBridgeLifecycleService:"connector_removed",{"bridge_platform": normalise_platform(bridge.type),- "age_days": _age_days(bridge.created_at),+ "age_days": age_days(bridge.created_at),# A connector removed having never connected is a failed# setup; one removed after months of service is a# decision. Reporting both as "removed" would hide the@@ -956,7 +1237,12 @@ class CollaborationBridgeLifecycleService:"room_count": room_count,},)+ # Every per-bridge map keyed by an id that will never be seen again.+ # Bridge ids are fresh UUIDs, so a deployment that repeatedly connects+ # and removes connectors grows these without bound otherwise.self._bridge_facts.pop(bridge_id, None)+ self._connect_failures.pop(bridge_id, None)+ self._connect_started.pop(bridge_id, None)def get(self, bridge_id: str) -> BridgeCore | None:return self._bridges.get(bridge_id)@@ -978,6 +1264,37 @@ class CollaborationBridgeLifecycleService:running += 1return running+ def running_by_platform(self) -> dict[str, int]:+ """`running_count`, split by the platform each bridge talks to.++ The total answers "how many bridges died" and never "which", and which+ is the first thing anyone asks — a Slack outage and a misconfigured+ Teams app look identical in a single number. Read off `_bridge_facts`,+ which is written at start from the row, so a bridge whose `BridgeCore`+ has already been discarded by a crash is still attributable.++ Every platform this process has ever started a bridge for appears,+ zero included — which is why it is seeded from `_platforms_seen` rather+ than from `_started` alone. A gauge that stops being reported is+ indistinguishable on a dashboard from one nobody is looking at, and+ "Slack went from one to zero" is the whole signal; `_started` drops a+ bridge that was stopped deliberately, so reading only that would end+ the series at exactly the moment it has something to say.++ A process that has never started a bridge reports nothing at all, and+ that is the honest answer rather than a gap: there is no bridge here to+ be up or down, and five platforms sitting at zero would invite an alert+ on a platform nobody configured.+ """+ counts: dict[str, int] = dict.fromkeys(self._platforms_seen, 0)+ for bridge_id in self._started:+ platform, _ = self._bridge_facts.get(bridge_id, ("none", None))+ name = normalise_platform(platform)+ task = self._tasks.get(bridge_id)+ alive = bridge_id in self._bridges and task is not None and not task.done()+ counts[name] = counts.get(name, 0) + (1 if alive else 0)+ return counts+def bridges_for_tenant(self, tenant_id: str) -> list[BridgeCore]:"""Running bridges belonging to `tenant_id`, and none other.diff --git a/core/switch_core/bridges/resource/service.py b/core/switch_core/bridges/resource/service.pyindex f548c8416..e1eb8f2e2 100644--- a/core/switch_core/bridges/resource/service.py+++ b/core/switch_core/bridges/resource/service.py@@ -3,7 +3,6 @@ from __future__ import annotationsimport loggingimport refrom dataclasses import dataclass-from datetime import UTC, datetimefrom typing import Any, Literalfrom sqlalchemy import select@@ -37,6 +36,7 @@ from switch_core.db.stores.reference_store import ReferenceStorefrom switch_core.db.stores.reference_type_store import ReferenceTypeStorefrom switch_core.db.stores.room_link_store import RoomLinkStorefrom switch_core.telemetry import TelemetryService, emit_safely+from switch_core.telemetry.ages import age_dayslogger = logging.getLogger(__name__)@@ -55,14 +55,6 @@ def _visibility(value: str) -> str:return value if value in ("private", "public") else "private"-def _age_days(created_at: object) -> float:- """How old a row is, in days, for reporting. Zero if unknown."""- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 86400.0, 0.0)--ROOM_DOCUMENT_MAX_CONTENT_BYTES = 1_048_576REFERENCE_TYPE_SLUG_PATTERN = re.compile(r"^[a-z][a-z0-9_]{1,62}$")@@ -413,7 +405,7 @@ class ResourceService:f"Reference type '{type_}' is used by {in_use} reference(s) ""and cannot be deleted")- age = _age_days(reference_type.created_at)+ age = age_days(reference_type.created_at)await self._reference_types.delete(session, type_)emit_safely(self._telemetry, "reference_type_deleted", {"age_days": age})@@ -548,6 +540,12 @@ class ResourceService:value=normalised_value,)created = await self._references.create(session, ref)+ # Deliberate trade: `create` above flushes, so a validation failure+ # (a bad visibility pair, an unknown type) has already raised before+ # this line runs — what remains uncommitted is the gateway route's+ # own `session.commit()`, called on the same session right after this+ # returns. This service does not own that transaction boundary, so+ # the residual risk is a bare commit failure, not a business-rule one.emit_safely(self._telemetry,"reference_created",@@ -654,7 +652,7 @@ class ResourceService:if ref is None:raise ValueError(f"Reference not found: {reference_id}")require(Principal(user_id, is_admin), "delete", ref)- kind, age = _reference_type(ref.type), _age_days(ref.created_at)+ kind, age = _reference_type(ref.type), age_days(ref.created_at)detached = await self._references.delete(session, reference_id)emit_safely(self._telemetry,@@ -849,6 +847,8 @@ class ResourceService:content=content,)created = await self._documents.create(session, doc)+ # Same trade as `reference_created` above: flushed, not yet committed,+ # and the commit is the gateway route's to call.emit_safely(self._telemetry,"document_created",@@ -924,7 +924,7 @@ class ResourceService:if doc is None:raise ValueError(f"Document not found: {document_id}")require(Principal(user_id, is_admin), "delete", doc)- age = _age_days(doc.created_at)+ age = age_days(doc.created_at)detached = await self._documents.delete(session, document_id)emit_safely(self._telemetry,@@ -1012,6 +1012,8 @@ class ResourceService:content=content,)created = await self._documents.create(session, doc)+ # Same trade as `reference_created` above: flushed, not yet committed,+ # and the commit is the MCP handler's to call.emit_safely(self._telemetry,"document_created",@@ -1058,6 +1060,8 @@ class ResourceService:content=content,)created = await self._documents.create(session, doc)+ # Same trade as `reference_created` above: flushed, not yet committed,+ # and the commit is the caller's to call.emit_safely(self._telemetry,"document_created",@@ -1130,7 +1134,11 @@ class ResourceService:raise PermissionError(f"Agent {agent_id} did not create document {document_id}")+ age = age_days(doc.created_at)await self._documents.delete(session, document_id)+ emit_safely(+ self._telemetry, "document_deleted", {"scope": "room", "age_days": age}+ )async def delete_room_document_by_user(self,@@ -1146,7 +1154,11 @@ class ResourceService:raise ValueError(f"Room-scoped document {document_id} not found in room {room_id}")+ age = age_days(doc.created_at)await self._documents.delete(session, document_id)+ emit_safely(+ self._telemetry, "document_deleted", {"scope": "room", "age_days": age}+ )async def get_room_scoped_document(self, session: AsyncSession, room_id: str, document_id: str@@ -1208,6 +1220,8 @@ class ResourceService:instructions=instructions,)created = await self._packages.create(session, pkg)+ # Same trade as `reference_created` above: flushed, not yet committed,+ # and the commit is the gateway route's to call.emit_safely(self._telemetry, "package_created", {"created_by_kind": "user"})return created@@ -1273,7 +1287,7 @@ class ResourceService:if pkg is None:raise ValueError(f"Package not found: {package_id}")require(Principal(user_id, is_admin), "delete", pkg)- age = _age_days(pkg.created_at)+ age = age_days(pkg.created_at)detached = await self._packages.delete(session, package_id)emit_safely(self._telemetry, "package_deleted", {"age_days": age})return detacheddiff --git a/core/switch_core/clients/client_lifecycle_service.py b/core/switch_core/clients/client_lifecycle_service.pyindex d1d841066..9f58643bf 100644--- a/core/switch_core/clients/client_lifecycle_service.py+++ b/core/switch_core/clients/client_lifecycle_service.py@@ -15,6 +15,7 @@ from switch_core.db.session_scope import tenant_sessionfrom switch_core.db.stores.client_store import ClientStorefrom switch_core.db.stores.tenant_store import TenantStorefrom switch_core.db.tenant_lookup import all_tenant_ids+from switch_core.logging_context import log_contextfrom switch_core.provisioning import Provisioningfrom switch_core.tenant_context import no_tenant, tenant_scope@@ -354,8 +355,13 @@ class ClientLifecycleService:through an ordinary `tenant_session` under that tenant. An exemptioncall here would have been a question whose answer the caller wasalready holding.++ The room goes with it for the same reason: the inbound handler that+ mints a puppet has that room bound as log context, and the puppet's+ own log lines must not name it for the rest of its life. Each delivery+ binds its room."""- with no_tenant():+ with no_tenant(), log_context(room_id=None):try:await client.start()except Exception:diff --git a/core/switch_core/config.py b/core/switch_core/config.pyindex 5c731ce95..84e2069e8 100644--- a/core/switch_core/config.py+++ b/core/switch_core/config.py@@ -339,14 +339,16 @@ class SwitchConfig(BaseSettings):# Postgres terminates a connection that sits inside an open transaction# without executing anything for longer than this (a Postgres interval such# as "15s"), turning a slot that never comes back into a loud, attributable- # error. Disabled by default, and it must stay that way until Matrix I/O+ # error. Disabled by default, and it must stay that way until provisioning# moves out of the RoomService transactions: `add_agents_to_room`,- # `remove_agents_from_room` and `delete_room` currently hold a transaction- # across invite/kick round trips, so enabling this today would trade a- # latency problem for a consistency one — Matrix membership changed, the- # rows that record it rolled back. Applies to the application engine only;- # Alembic builds its own engine from `db_connect_args`, so a migration is- # never killed mid-transaction.+ # `remove_agents_from_room` and `delete_room` hold a transaction across+ # invite/kick calls, so enabling this today would trade a latency problem+ # for a consistency one — membership changed, the rows that record it+ # rolled back. `PostgresProvisioning` writes rows on its own session rather+ # than dialling a homeserver, which shortens that window without closing+ # it: the two still commit separately.+ # Applies to the application engine only; Alembic builds its own engine+ # from `db_connect_args`, so a migration is never killed mid-transaction.db_idle_in_transaction_session_timeout: str | None = None# How long a migration waits for a lock before giving up (a Postgresdiff --git a/core/switch_core/gateway/api_keys.py b/core/switch_core/gateway/api_keys.pyindex d95825f1e..11aa7d256 100644--- a/core/switch_core/gateway/api_keys.py+++ b/core/switch_core/gateway/api_keys.py@@ -3,7 +3,6 @@ from __future__ import annotationsimport hashlibimport loggingimport secrets-from datetime import UTC, datetimefrom typing import Annotatedfrom fastapi import APIRouter, Depends, HTTPException@@ -28,6 +27,7 @@ from switch_core.gateway.schemas import (RevealKeyResponse,)from switch_core.telemetry import emit_safely+from switch_core.telemetry.ages import age_dayslogger = logging.getLogger(__name__)@@ -41,13 +41,6 @@ def _key_type(value: str) -> str:return value if value in ("agent", "registration", "bootstrap") else "other"-def _age_days(created_at: object) -> float:- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 86400.0, 0.0)--router = APIRouter()@@ -142,6 +135,6 @@ async def delete_api_key(emit_safely(current_telemetry(),"api_key_revoked",- {"key_type": _key_type(key.type), "age_days": _age_days(key.created_at)},+ {"key_type": _key_type(key.type), "age_days": age_days(key.created_at)},)return {"ok": True}diff --git a/core/switch_core/gateway/room_groups.py b/core/switch_core/gateway/room_groups.pyindex 89ebd59e0..88b49c0d8 100644--- a/core/switch_core/gateway/room_groups.py+++ b/core/switch_core/gateway/room_groups.py@@ -1,6 +1,5 @@from __future__ import annotations-from datetime import UTC, datetimefrom typing import Annotatedfrom fastapi import APIRouter, Depends, HTTPException, Response@@ -25,14 +24,7 @@ from switch_core.gateway.schemas import (RoomGroupUpdateRequest,)from switch_core.telemetry import emit_safely---def _age_days(created_at: object) -> float:- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 86400.0, 0.0)-+from switch_core.telemetry.ages import UNKNOWN_AGE, age_daysrouter = APIRouter()@@ -169,7 +161,11 @@ async def delete_room_group(# group and dismantling a working one.group = await room_group_store.get(session, group_id)room_count = (await room_group_store.get_room_counts(session)).get(group_id, 0)- age = _age_days(group.created_at) if group else 0.0+ # `UNKNOWN_AGE`, not `0.0`: a group that could not be read has no age, and+ # zero is one a real group can have. Unreachable today — the delete below+ # 404s on a missing group before the event is built — but the sentinel is+ # what makes that stay true if the order ever changes.+ age = age_days(group.created_at) if group else UNKNOWN_AGEremoved = await room_group_store.delete(session, group_id)if not removed:diff --git a/core/switch_core/gateway/rooms.py b/core/switch_core/gateway/rooms.pyindex 446d06cd8..48ff8572b 100644--- a/core/switch_core/gateway/rooms.py+++ b/core/switch_core/gateway/rooms.py@@ -560,12 +560,16 @@ async def create_room_role() -> list[RoomRoleDetail]:await _require_room(session, room_store, room_id, user, "write", is_admin)try:- emit_safely(- current_telemetry(), "room_role_defined", {"exclusive": req.exclusive}- )await protocol.room_role_store.define_role(session, room_id, req.name, req.instructions, req.exclusive)+ # After the write succeeds but before the outer commit — the same+ # convention as room_role_deleted below, chosen so a request that+ # fails validation (e.g. a duplicate name) never reports an event for+ # a role that was never defined.+ emit_safely(+ current_telemetry(), "room_role_defined", {"exclusive": req.exclusive}+ )except ValueError as e:raise HTTPException(status_code=400, detail=str(e))await session.commit()diff --git a/core/switch_core/gateway/templates.py b/core/switch_core/gateway/templates.pyindex 967b62aff..5af5e995e 100644--- a/core/switch_core/gateway/templates.py+++ b/core/switch_core/gateway/templates.py@@ -14,7 +14,6 @@ already answered — agreeing right up until the day it did not.from __future__ import annotationsimport re-from datetime import UTC, datetimefrom typing import Annotatedfrom urllib.parse import quote@@ -57,6 +56,7 @@ from switch_core.gateway.schemas import (TemplateValidateResponse,)from switch_core.telemetry import emit_safely+from switch_core.telemetry.ages import age_daysfrom switch_core.template_lint import lint_template@@ -69,13 +69,6 @@ def _kind(value: str | None) -> str:return value if value in ("room", "group", "agent") else "other"-def _age_days(created_at: object) -> float:- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 86400.0, 0.0)--router = APIRouter()@@ -465,7 +458,7 @@ async def delete_template(template = await _load_for("delete", session, template_store, template_id, user, is_admin)- kind, age = _kind(template.kind), _age_days(template.created_at)+ kind, age = _kind(template.kind), age_days(template.created_at)try:await template_store.delete(session, template_id)except ValueError as e:diff --git a/core/switch_core/gateway/tenants.py b/core/switch_core/gateway/tenants.pyindex ed30787a2..30f6385c7 100644--- a/core/switch_core/gateway/tenants.py+++ b/core/switch_core/gateway/tenants.py@@ -60,24 +60,12 @@ from switch_core.gateway.schemas import (TenantMembershipResponse,)from switch_core.telemetry import emit_safely+from switch_core.telemetry.ages import age_hoursfrom switch_core.tenant_context import current_tenant_idlogger = logging.getLogger(__name__)-def _age_hours(created_at: object) -> float:- """How long an invitation sat before it was accepted. Zero if unknown.-- Hours rather than days: the interesting range is "immediately" to "a- couple of days", and a figure that reads 0.1 for most of it says less than- one that reads 2.4.- """- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 3600.0, 0.0)--router = APIRouter()TENANT_MEMBER_ROLES = ("owner", "admin", "member")@@ -440,7 +428,7 @@ async def accept_invitation(emit_safely(current_telemetry(),"invitation_accepted",- {"age_hours": _age_hours(invitation.created_at)},+ {"age_hours": age_hours(invitation.created_at)},)else:role = existing_rolediff --git a/core/switch_core/logging_context.py b/core/switch_core/logging_context.pyindex abef5180b..001816a3e 100644--- a/core/switch_core/logging_context.py+++ b/core/switch_core/logging_context.py@@ -40,7 +40,19 @@ from switch_core.tenant_context import current_tenant_id# The context fields, in the order they are rendered. Anything logged as# context must be listed here, so that a typo in a bind call is an error rather# than a field that silently never appears.-CONTEXT_FIELDS: tuple[str, ...] = ("tenant_id", "request_id", "agent_id", "user_id")+#+# `room_id` is here and not in a metric attribute, deliberately: a room id is+# unbounded and belongs to one tenant, which is disqualifying for a dashboard+# label (see `observability/catalogue.py`) and is exactly what a log line is+# for. A room is the unit almost every support question arrives in — "this room+# stopped relaying" — so it has to be something a log search can filter on.+CONTEXT_FIELDS: tuple[str, ...] = (+ "tenant_id",+ "request_id",+ "agent_id",+ "room_id",+ "user_id",+)@dataclass(frozen=True)@@ -48,6 +60,7 @@ class LogContext:tenant_id: str | None = Nonerequest_id: str | None = Noneagent_id: str | None = None+ room_id: str | None = Noneuser_id: str | None = None@@ -150,5 +163,6 @@ class LogContextFilter(logging.Filter):)record.request_id = context.request_idrecord.agent_id = context.agent_id+ record.room_id = context.room_idrecord.user_id = context.user_idreturn Truediff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex 8a24ae5ee..a838747da 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -133,6 +133,7 @@ from switch_core.observability.bootstrap import (start_observability,)from switch_core.observability.pool import pool_stats+from switch_core.observability.query import instrument_queriesfrom switch_core.observability.runtime import EventLoopLagfrom switch_core.provisioning import Provisioningfrom switch_core.provisioning.postgres import PostgresProvisioning@@ -331,9 +332,24 @@ async def run(config: SwitchConfig) -> None:engine = create_engine_from_config(config)await _check_tenant_isolation(config, engine)session_factory = create_session_factory(engine)+ # Wired here rather than inside the engine factory, so the database layer+ # keeps knowing nothing about observability. Unconditional: the listeners+ # record into whatever registry is installed, which is the null one until+ # `start_observability` runs and stays null when export is off — measured+ # at 0.3 µs per statement against a 130 µs statement, so a server that+ # reports nothing pays nothing worth counting, and turning export on does+ # not change how queries execute.+ instrument_queries(engine)+# Its connection is held rather than borrowed, so it builds its own outside# the pool. Nothing subscribes yet; it starts with the server so that the# subscription exists before the first consumer needs it.+ #+ # Deliberately NOT instrumented for query timing: `MessageListener` takes+ # the raw asyncpg connection off this engine and issues `LISTEN` and its+ # heartbeat through the driver, so SQLAlchemy's cursor events never fire+ # and the listeners would record nothing at all. That this connection is+ # alive is answered by the `message_listener` readiness check instead.message_listener = MessageListener(lambda: create_unpooled_engine(config))# Invitations for the Postgres transport, which has no durable one of its@@ -669,6 +685,7 @@ async def run(config: SwitchConfig) -> None:probes = RuntimeProbes(listener_connected=message_listener.connected.is_set,bridges_running=collab_lifecycle.running_count,+ bridges_running_by_platform=collab_lifecycle.running_by_platform,bridges_configured=collab_lifecycle.expected_count,clients_running=client_lifecycle.running_count,connectors_running=connector_lifecycle.running_count,diff --git a/core/switch_core/observability/bootstrap.py b/core/switch_core/observability/bootstrap.pyindex 24675c8c9..1d8a84717 100644--- a/core/switch_core/observability/bootstrap.py+++ b/core/switch_core/observability/bootstrap.py@@ -10,7 +10,7 @@ from __future__ import annotationsimport asyncioimport logging-from collections.abc import Callable, Iterator+from collections.abc import Callable, Iterator, Mappingfrom dataclasses import dataclassimport httpx@@ -76,6 +76,7 @@ class RuntimeProbes:listener_connected: Callable[[], bool]bridges_running: Callable[[], int]+ bridges_running_by_platform: Callable[[], Mapping[str, int]]bridges_configured: Callable[[], int]clients_running: Callable[[], int]connectors_running: Callable[[], int]@@ -119,7 +120,8 @@ def _state_readings(probes: RuntimeProbes) -> Callable[[], Iterator[GaugeReadingdef readings() -> Iterator[GaugeReading]:yield GaugeReading(AGENTS_CONNECTED, float(probes.agents_connected()), {})yield GaugeReading(CLIENTS_RUNNING, float(probes.clients_running()), {})- yield GaugeReading(BRIDGES_RUNNING, float(probes.bridges_running()), {})+ for platform, running in probes.bridges_running_by_platform().items():+ yield GaugeReading(BRIDGES_RUNNING, float(running), {"platform": platform})yield GaugeReading(CONNECTORS_RUNNING, float(probes.connectors_running()), {})stats = probes.pool_stats()diff --git a/core/switch_core/observability/catalogue.py b/core/switch_core/observability/catalogue.pyindex cd52063a2..55bcd67b5 100644--- a/core/switch_core/observability/catalogue.py+++ b/core/switch_core/observability/catalogue.py@@ -21,6 +21,8 @@ from __future__ import annotationsfrom dataclasses import dataclass, fieldfrom typing import Literal+from switch_core.observability.otlp import SUB_MILLISECOND_BOUNDS_MS+MetricKind = Literal["sum", "gauge", "histogram"]# A ceiling, not a target: crossing it means a call site is passing something@@ -35,6 +37,12 @@ class MetricSpec:unit: strdescription: strattributes: frozenset[str] = field(default_factory=frozenset)+ # Histograms only, and only where the default set cannot resolve the+ # measurement: bounds are the resolution, and a histogram whose readings+ # all land in one bucket reports that bucket for ever. None means the+ # shared latency bounds, which suit anything measured in tens of+ # milliseconds and upwards.+ bounds: tuple[float, ...] | None = Nonedef _spec(@@ -43,6 +51,7 @@ def _spec(unit: str,description: str,*attributes: str,+ bounds: tuple[float, ...] | None = None,) -> MetricSpec:return MetricSpec(name=name,@@ -50,6 +59,7 @@ def _spec(unit=unit,description=description,attributes=frozenset(attributes),+ bounds=bounds,)@@ -96,6 +106,22 @@ DB_POOL_OVERFLOW = _spec("{connection}","Connections open beyond the pool's nominal size.",)+# The pool gauges say whether connections are scarce; this says whether the+# database is slow, which is the other half and the one a slow request is+# usually about. `operation` is the statement's leading keyword mapped through+# a fixed table — never the statement, which is unbounded and carries literals.+DB_QUERY_DURATION = _spec(+ "switch.db.query.duration",+ "histogram",+ "ms",+ "Round trip for one statement, measured around the driver call. A "+ "statement that raised is not timed: a query that failed in four "+ "milliseconds is not evidence the database is fast. Not every statement "+ "the process runs — see `observability/query.py` for what is outside it, "+ "notably migrations and the message listener.",+ "operation",+ bounds=SUB_MILLISECOND_BOUNDS_MS,+)# ── Message transport ────────────────────────────────────────────────────────MESSAGES_SENT = _spec(@@ -169,8 +195,20 @@ BRIDGES_RUNNING = _spec("switch.bridges.running","gauge","{bridge}",- "Collaboration bridges with a live task. A configured bridge missing here "- "has crashed.",+ "Collaboration bridges with a live task, by platform. A configured bridge "+ "missing here has crashed — and without `platform` the total says how many "+ "died and never which, which is the first thing anyone asks.",+ "platform",+)+BRIDGE_CALL_DURATION = _spec(+ "switch.bridge.call.duration",+ "histogram",+ "ms",+ "Round trip for one outbound call to a collaboration platform. The bridge "+ "counters say whether relays are failing; this says whether they are "+ "arriving late, which is what a room that feels unresponsive actually is.",+ "platform",+ "kind",)# ── Agent protocol ───────────────────────────────────────────────────────────@@ -273,6 +311,7 @@ CATALOGUE: dict[str, MetricSpec] = {DB_POOL_IN_USE,DB_POOL_SIZE,DB_POOL_OVERFLOW,+ DB_QUERY_DURATION,MESSAGES_SENT,MESSAGES_DELIVERED,SEND_FAILURES,@@ -282,6 +321,7 @@ CATALOGUE: dict[str, MetricSpec] = {BRIDGE_EVENTS_OUT,BRIDGE_ERRORS,BRIDGES_RUNNING,+ BRIDGE_CALL_DURATION,AGENT_EVENTS_DROPPED,AGENT_CONNECTIONS_EXPIRED,AGENTS_CONNECTED,diff --git a/core/switch_core/observability/metrics.py b/core/switch_core/observability/metrics.pyindex e5a5f3007..891ed5b32 100644--- a/core/switch_core/observability/metrics.py+++ b/core/switch_core/observability/metrics.py@@ -152,7 +152,7 @@ class MetricsRegistry:returnhistogram = series.get(key)if histogram is None:- histogram = _Histogram(DEFAULT_LATENCY_BOUNDS_MS)+ histogram = _Histogram(spec.bounds or DEFAULT_LATENCY_BOUNDS_MS)series[key] = histogramhistogram.record(value)diff --git a/core/switch_core/observability/otlp.py b/core/switch_core/observability/otlp.pyindex 40523afd8..14a609a20 100644--- a/core/switch_core/observability/otlp.py+++ b/core/switch_core/observability/otlp.py@@ -68,6 +68,31 @@ DEFAULT_LATENCY_BOUNDS_MS: tuple[float, ...] = (10000.0,)+# For what a single database round trip costs, which is a different scale+# entirely: measured against Postgres 16 on this schema, an ordinary statement+# is 0.1–0.2 ms, so every one of it lands in the first bucket above and a p95+# reads "≤5 ms" until something is already badly wrong. Bounds that cannot+# resolve the normal case cannot show it degrading, which is the whole job.+# Extends to the same ceiling, because a statement that takes ten seconds is+# the thing you most want to see.+SUB_MILLISECOND_BOUNDS_MS: tuple[float, ...] = (+ 0.1,+ 0.25,+ 0.5,+ 1.0,+ 2.5,+ 5.0,+ 10.0,+ 25.0,+ 50.0,+ 100.0,+ 250.0,+ 500.0,+ 1000.0,+ 5000.0,+ 10000.0,+)+AttributeValue = str | bool | int | floatdiff --git a/core/switch_core/observability/query.py b/core/switch_core/observability/query.pynew file mode 100644index 000000000..28cbe8f65--- /dev/null+++ b/core/switch_core/observability/query.py@@ -0,0 +1,131 @@+"""How long the database takes, from this server's side of it.++The pool gauges answer "are we holding too many connections", not "is anything+slow". On this server a request is a handful of queries and little else, so a+slow endpoint is nearly always a slow query.++Timed with SQLAlchemy's cursor events rather than by wrapping call sites: they+fire around the driver call, so what is measured is the round trip rather than+the Python either side of it, and there is no call site to remember to wrap.++What that leaves out, so nobody reads the panel as "every statement": the+message listener, which takes the raw asyncpg connection and never executes+through SQLAlchemy; Alembic, which builds its own engine; `BEGIN`/`COMMIT`,+which the dialect issues directly; and pool pre-ping. An `executemany` is one+measurement for the whole batch, because it is one round trip.++**The statement text never leaves this module.** It is the obvious attribute+and the one that must not be a metric: unbounded, and it carries literals on+some paths. What goes on the wire is the leading keyword through a fixed table,+so the series count is the size of that table.+"""++from __future__ import annotations++import time+from typing import Any++from sqlalchemy import event+from sqlalchemy.ext.asyncio import AsyncEngine++from switch_core.observability.catalogue import DB_QUERY_DURATION+from switch_core.observability.metrics import metrics++# The verbs worth telling apart on a panel, and nothing else. `other` covers+# the rest — `SET`, a DDL statement, anything a later caller invents — which is+# not what anyone is looking for when a read is slow, and naming each one would+# put the shape of the schema on a dashboard.+_OPERATIONS = frozenset({"select", "insert", "update", "delete"})++# Every tenant-bound transaction opens with this, from `db/tenant_session.py`'s+# `after_begin`. It is a `SELECT`, it is always trivial, and it is roughly a+# third of all statements — so counted as one it drags the `select` percentiles+# toward bookkeeping and away from the queries the panel exists to show. Its+# own label rather than dropped: a transaction rate is worth having, and a+# `set_config` that started taking milliseconds would mean something.+_TENANT_BIND_MARKER = "set_config("+_TENANT_BIND = "tenant_bind"++_TIMER_KEY = "_switch_query_started"+++def _operation(statement: str) -> str:+ """The leading keyword, or `other`.++ Deliberately crude. A statement whose first word is not one of the four is+ not worth a series of its own, and anything cleverer would have to parse+ SQL to find out — on the hot path, for a label.+ """+ # Sliced before stripping: this runs on every statement, and a batched+ # insert can be long.+ stripped = statement[:128].lstrip()+ head = stripped[:16].split(None, 1)+ if not head:+ return "other"+ keyword = head[0].lower()+ if keyword not in _OPERATIONS:+ return "other"+ if keyword == "select" and _TENANT_BIND_MARKER in stripped[:64]:+ return _TENANT_BIND+ return keyword+++def _before(+ _conn: Any,+ _cursor: Any,+ _statement: str,+ _parameters: Any,+ context: Any,+ _executemany: bool,+) -> None:+ # On the execution context rather than in a module-level dict: a pool+ # serves many connections at once, and anything keyed by less than the+ # statement's own context would have one query's start time read by+ # another's finish.+ #+ # SQLAlchemy types the context optional on this event and one dialect does+ # pass None. A raise here lands *before* the driver call, so an+ # unguarded `setattr` would turn a metric into a failed query.+ if context is None:+ return+ setattr(context, _TIMER_KEY, time.perf_counter())+++def _after(+ _conn: Any,+ _cursor: Any,+ statement: str,+ _parameters: Any,+ context: Any,+ _executemany: bool,+) -> None:+ started = getattr(context, _TIMER_KEY, None)+ if started is None:+ return+ elapsed_ms = (time.perf_counter() - started) * 1000.0+ metrics().observe(+ DB_QUERY_DURATION, {"operation": _operation(statement)}, elapsed_ms+ )+++def instrument_queries(engine: AsyncEngine) -> None:+ """Time every statement this engine runs.++ Idempotent, but not by anything here: SQLAlchemy already refuses a+ `(target, identifier, fn)` it is holding, and these are module-level+ functions, so the same pair registered twice fires once. That is worth+ knowing rather than guarding — a guard would be dead code claiming to+ prevent something — and it is worth *testing*, because it stops being true+ the moment either listener becomes a closure or a bound method, at which+ point every query is timed twice and the only symptom is a p50 nobody can+ reconcile.++ A statement that raises is not recorded. `after_cursor_execute` does not+ fire on a failure, and that is the right way round — a query that failed in+ four milliseconds is not evidence the database is fast, and mixing the two+ would pull the percentiles down exactly when something is wrong. Failures+ are already visible as errors on the routes that provoked them.+ """+ sync_engine = engine.sync_engine+ event.listen(sync_engine, "before_cursor_execute", _before)+ event.listen(sync_engine, "after_cursor_execute", _after)diff --git a/core/switch_core/room_service.py b/core/switch_core/room_service.pyindex 11219bc3b..3d79de278 100644--- a/core/switch_core/room_service.py+++ b/core/switch_core/room_service.py@@ -16,7 +16,6 @@ room Switch has no record of it being in.from __future__ import annotationsimport logging-from datetime import UTC, datetimefrom typing import TYPE_CHECKING, Any, Literalfrom pydantic import BaseModel@@ -38,6 +37,7 @@ from switch_core.db.stores.room_store import RoomStorefrom switch_core.db.tenant_lookup import all_tenant_idsfrom switch_core.provisioning import Provisioningfrom switch_core.telemetry import TelemetryService, emit_safely+from switch_core.telemetry.ages import age_daysfrom switch_core.telemetry.snapshot import (normalise_actor_kind,normalise_channel_type,@@ -57,19 +57,6 @@ logger = logging.getLogger(__name__)SYSTEM_CLIENT_TYPES = ("observe", "admin")-def _age_days(created_at: object) -> float:- """How many days old a room is, for reporting. Zero if unknown.-- Takes `object` because the timestamp columns on the models are annotated- `Mapped[str]` while carrying real `datetime`s — so the honest signature is- "whatever the column hands back", checked here rather than trusted.- """- if not isinstance(created_at, datetime):- return 0.0- moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)- return max((datetime.now(UTC) - moment).total_seconds() / 86400.0, 0.0)--class LinkedRoomSpec(BaseModel):target_room_id: strlabel: str@@ -585,6 +572,50 @@ class RoomService:if bridge_core and external_channel_id:bridge_core.end_provisioning(external_channel_id)+ # The room row is durably committed above. Everything from here down —+ # channel capture, Matrix invites, adding agents/users on the external+ # platform — is best-effort against another system and can still fail+ # (and, on failure, still propagate out of this call, same as before).+ # Reported here rather than at the end of the function so a failure in+ # any of that does not also make the room's existence go unreported:+ # the row is the thing the event describes, and it is already true.+ #+ # Resolved on its own line, never inside the argument list below: an+ # `await` there is evaluated before `emit_safely` is entered, outside+ # the guard meant to contain it.+ platform = await self._bridge_platform(bridge_id)++ emit_safely(+ self._telemetry,+ "room_created",+ {+ "channel_type": normalise_channel_type(channel_type),+ "bridge_platform": platform,+ "agent_count": len(agent_ids),+ "human_count": len(config.user_names or []),+ "has_instructions": config.instructions is not None,+ "created_by_kind": config.created_by_kind,+ "from_template": config.from_template,+ },+ )++ # Only a room a person made counts as activation. Guarded like the+ # emit above it: this runs ahead of the invite, the agent adds and the+ # client joins, so an unguarded raise would leave a committed room+ # nobody is in.+ if self._telemetry is not None and config.created_by_kind == "user":+ try:+ await self._telemetry.emit_milestone(+ "first_room_created",+ channel_type=normalise_channel_type(channel_type),+ bridge_platform=platform,+ )+ except Exception:+ logger.exception(+ "Could not report the first_room_created milestone for room %s",+ room.id,+ )+if bridge_core and external_channel_id:await self._ensure_channel_capture(bridge_core, external_channel_id, channel_type@@ -641,32 +672,6 @@ class RoomService:len(system_clients),)- # Never inside the argument list: an `await` there is evaluated before- # `emit_safely` is entered, outside the guard meant to contain it.- platform = await self._bridge_platform(bridge_id)-- emit_safely(- self._telemetry,- "room_created",- {- "channel_type": normalise_channel_type(channel_type),- "bridge_platform": platform,- "agent_count": len(agent_ids),- "human_count": len(config.user_names or []),- "has_instructions": config.instructions is not None,- "created_by_kind": config.created_by_kind,- "from_template": config.from_template,- },- )-- # Only a room a person made counts as activation.- if self._telemetry is not None and config.created_by_kind == "user":- await self._telemetry.emit_milestone(- "first_room_created",- channel_type=normalise_channel_type(channel_type),- bridge_platform=platform,- )-failed_attachments = unreachable_users + await self._attach_after_creation(room.id, config)@@ -760,7 +765,7 @@ class RoomService:"created_by_kind": normalise_actor_kind((room.metadata_ or {}).get("created_by_kind")),- "age_days": _age_days(room.created_at),+ "age_days": age_days(room.created_at),"was_ever_active": was_ever_active,"agent_count": agent_count,},@@ -1009,7 +1014,7 @@ class RoomService:"room_archived",{"bridge_platform": await self._bridge_platform(room.bridge_id),- "age_days": _age_days(room.created_at),+ "age_days": age_days(room.created_at),"was_ever_active": await self._was_ever_active(room.tenant_id, room_id),},)@@ -1017,8 +1022,10 @@ class RoomService:async def _bridge_platform(self, bridge_id: str | None) -> str:"""The platform a bridge id names, as the telemetry catalogue spells it.- `none` for an internal-only room and for a lookup that failed. Never- raises: the operations this labels must not fail because it did.+ `none` for an internal-only room; `unknown` for a lookup that failed —+ the two must stay distinct, or a transient error reports a+ Slack-bridged room as internal-only. Never raises: the operations this+ labels must not fail because it did."""if bridge_id is None or self._telemetry is None:return "none"@@ -1032,26 +1039,30 @@ class RoomService:bridge_id,exc_info=True,)- return "none"+ return "unknown"return normalise_platform(bridge.type if bridge else None)- async def _was_ever_active(self, tenant_id: str, room_id: str) -> bool:- """Whether a human ever posted in this room.+ async def _was_ever_active(self, tenant_id: str, room_id: str) -> str:+ """Whether a human ever posted in this room: `"true"`, `"false"`, or+ `"unknown"` when the lookup itself failed.- Asked only on archive, so the cost lands on a rare operation. False on- failure — an archive must not fail because a count did.+ Asked only on archive/delete, so the cost lands on a rare operation.+ `"unknown"` rather than `"false"` on failure — a count that could not+ run is not evidence the room was never used, which is exactly the+ fact this property exists to carry."""try:async with tenant_session(self._session_factory, tenant_id) as session:- return await room_had_human_activity(session, tenant_id, room_id)+ active = await room_had_human_activity(session, tenant_id, room_id)except Exception:logger.warning("Could not determine whether room %s was ever active; "- "reporting it as inactive.",+ "reporting it as unknown.",room_id,exc_info=True,)- return False+ return "unknown"+ return "true" if active else "false"async def add_users_to_room(self, room_id: str, user_names: list[str]) -> list[str]:"""Add users to a bridged room; returns the names that did not make itdiff --git a/core/switch_core/telemetry/ages.py b/core/switch_core/telemetry/ages.pynew file mode 100644index 000000000..7a08c7ef5--- /dev/null+++ b/core/switch_core/telemetry/ages.py@@ -0,0 +1,68 @@+"""How old a row is, for the events that report it.++Every deletion event carries the lifespan of what was deleted — deleted after+an hour and after a year mean opposite things — and all of them use this one+helper, so they agree on what to report when the timestamp cannot be read.++That value is ``-1``, not ``0.0``. ``0.0`` is a real value a real row can+have, so it makes "we could not tell" indistinguishable from "created a moment+ago" — and the second is the far more common reading, which is how a lookup+failure gets counted as a burst of churn. ``-1`` is a number no age can take, so a reader+that does not know about it still cannot mistake it for data.++None of this is reachable today: every ``created_at`` in the schema is+``nullable=False`` with a server default, and the session factory sets+``expire_on_commit=False``, so a loaded row always hands back a real+``datetime``. The guard is for the day one of those stops being true.+"""++from __future__ import annotations++from datetime import UTC, datetime++# Not an age any row can have, so it cannot be read as one.+UNKNOWN_AGE = -1.0++_SECONDS_PER_DAY = 86400.0+_SECONDS_PER_HOUR = 3600.0+++def _elapsed_seconds(created_at: object) -> float | None:+ """Seconds since a timestamp column, or None when it is not one.++ Takes ``object`` because the timestamp columns on the models are annotated+ ``Mapped[str]`` while carrying real ``datetime``s — so the honest signature+ is "whatever the column hands back", checked here rather than trusted.++ Clamped at zero. The clock is the wall clock, because these span process+ restarts and there is no monotonic reading from before the row existed; a+ clock stepped backwards would otherwise produce a negative number, which is+ the one value that means something else here.+ """+ if not isinstance(created_at, datetime):+ return None+ moment = created_at if created_at.tzinfo else created_at.replace(tzinfo=UTC)+ return max((datetime.now(UTC) - moment).total_seconds(), 0.0)+++def age_days(created_at: object) -> float:+ """How old a row is, in days. :data:`UNKNOWN_AGE` when it cannot be told."""+ elapsed = _elapsed_seconds(created_at)+ return UNKNOWN_AGE if elapsed is None else elapsed / _SECONDS_PER_DAY+++def age_hours(created_at: object) -> float:+ """How old a row is, in hours. :data:`UNKNOWN_AGE` when it cannot be told.++ Hours rather than days where the interesting range is "immediately" to "a+ couple of days": a figure that reads 0.1 for most of it says less than one+ that reads 2.4.+ """+ elapsed = _elapsed_seconds(created_at)+ return UNKNOWN_AGE if elapsed is None else elapsed / _SECONDS_PER_HOUR+++def seconds_since(created_at: object) -> float:+ """Seconds since a timestamp column. :data:`UNKNOWN_AGE` when it is not one."""+ elapsed = _elapsed_seconds(created_at)+ return UNKNOWN_AGE if elapsed is None else elapseddiff --git a/core/switch_core/telemetry/catalogue.py b/core/switch_core/telemetry/catalogue.pyindex 40fc0df67..20f48a017 100644--- a/core/switch_core/telemetry/catalogue.py+++ b/core/switch_core/telemetry/catalogue.py@@ -83,7 +83,12 @@ class TelemetryCatalogueError(Exception):# The five collaboration platforms, plus the absence of one. `none` rather than# omitting the property: see the module docstring on exact property sets.-BRIDGE_PLATFORM = one_of("slack", "mattermost", "discord", "teams", "telegram", "none")+# `unknown` is a third, distinct case — not "no bridge" but "there is one and+# we could not resolve it" (e.g. the lookup itself failed) — so a transient+# error is never misreported as an internal-only room.+BRIDGE_PLATFORM = one_of(+ "slack", "mattermost", "discord", "teams", "telegram", "none", "unknown"+)CHANNEL_TYPE = one_of("channel_public", "channel_private", "direct", "none")@@ -127,11 +132,23 @@ BRIDGE_FAILURE_REASON = one_of("none", "auth_failed", "network", "platform_error", "config_invalid", "unknown")+# A yes/no fact the server sometimes cannot establish — a failed lookup reports+# `unknown` rather than collapsing "could not tell" into `false`, a value the+# code has no evidence for. Not BOOLEAN: a property that can be unknown is a+# three-valued fact, not a boolean with a spare bit smuggled into it.+TRISTATE = one_of("true", "false", "unknown")+# ── The catalogue ────────────────────────────────────────────────────────────_SNAPSHOT_COUNTS = (+ # How many tenants are actually behind every other number here, which is+ # not how many the deployment has: a tenant whose queries fail is stepped+ # over, and `tenant_failed_count` says how many. Without the pair a partial+ # pass is a drop in every count at once with nothing to attribute it to —+ # indistinguishable from a deployment losing its users."tenant_count",+ "tenant_failed_count","user_count","user_active_1d","user_active_7d",@@ -192,7 +209,15 @@ _SINCE_INSTALL: Mapping[str, PropertyType] = {"seconds_since_install": NUMBER}CATALOGUE: Mapping[str, Mapping[str, PropertyType]] = {# ── The daily snapshot ───────────────────────────────────────────────────# Counted locally, where the ids are known; only totals are reported.- "usage_snapshot": {name: NUMBER for name in _SNAPSHOT_COUNTS},+ # `duration_ms` is how long the pass took to collect. It is roughly fifteen+ # queries per tenant, on a schedule, against the same database serving+ # rooms — "what does this cost at scale" is an open question in+ # `docs/old/telemetry-events.md`, and this is the only place it can be+ # answered from.+ "usage_snapshot": {+ **{name: NUMBER for name in _SNAPSHOT_COUNTS},+ "duration_ms": NUMBER,+ },# ── Milestones ───────────────────────────────────────────────────────────# At most once per deployment, and only for one installed after this# shipped. Together they are the activation funnel.@@ -235,7 +260,7 @@ CATALOGUE: Mapping[str, Mapping[str, PropertyType]] = {"room_archived": {"bridge_platform": BRIDGE_PLATFORM,"age_days": NUMBER,- "was_ever_active": BOOLEAN,+ "was_ever_active": TRISTATE,},"room_agents_added": {"agent_count": NUMBER, "added_by_kind": ACTOR_KIND},"room_agents_removed": {"agent_count": NUMBER, "removed_by_kind": ACTOR_KIND},@@ -247,7 +272,7 @@ CATALOGUE: Mapping[str, Mapping[str, PropertyType]] = {"channel_type": CHANNEL_TYPE,"created_by_kind": ACTOR_KIND,"age_days": NUMBER,- "was_ever_active": BOOLEAN,+ "was_ever_active": TRISTATE,"agent_count": NUMBER,},"agent_deleted": {@@ -260,8 +285,10 @@ CATALOGUE: Mapping[str, Mapping[str, PropertyType]] = {"bridge_platform": BRIDGE_PLATFORM,"age_days": NUMBER,# Removed having never connected is a failed setup; removed after- # months is a decision.- "was_ever_connected": BOOLEAN,+ # months is a decision. `unknown` when the durable record itself+ # could not be read — that is not evidence either way, and reporting+ # it as `false` would misfile a lookup failure as a failed setup.+ "was_ever_connected": TRISTATE,"room_count": NUMBER,},"agent_registered": {@@ -295,6 +322,15 @@ CATALOGUE: Mapping[str, Mapping[str, PropertyType]] = {"outcome": OUTCOME,# `none` on success, so the property set stays exact either way."failure_reason": BRIDGE_FAILURE_REASON,+ # How long the attempt took. Nothing else times this: connecting a+ # bridge is a background task rather than a request the server serves,+ # so it is invisible to `switch.http.request.duration`, and a connect+ # that succeeds after ninety seconds and one that succeeds in two are+ # otherwise the same row. Measured on a monotonic clock, in whole+ # milliseconds, on both outcomes — a failure's latency is the more+ # interesting of the two, because a timeout and a refusal look identical+ # in the failure code and nothing alike in the time.+ "duration_ms": NUMBER,},# ── Resources, keys and groups ───────────────────────────────────────────# Creating is intent, attaching is use, and the gap between them is thediff --git a/core/switch_core/telemetry/deployment.py b/core/switch_core/telemetry/deployment.pyindex de3ed0392..3cd39c97f 100644--- a/core/switch_core/telemetry/deployment.py+++ b/core/switch_core/telemetry/deployment.py@@ -14,6 +14,7 @@ from __future__ import annotationsimport loggingfrom datetime import UTC, datetime+from typing import Literalfrom sqlalchemy import selectfrom sqlalchemy.dialects.postgresql import insert as pg_insert@@ -70,25 +71,35 @@ def seconds_since_install(installed_at: datetime | None) -> float | None:async def milestone_claimed(session_factory: async_sessionmaker[AsyncSession], name: str-) -> bool:+) -> Literal["true", "false", "unknown"]:"""Whether `name` has already been reported, without claiming it.- For the places that need to know a thing once happened — a connector's- removal asking whether it ever connected. False on failure.+ For the one caller that needs to know a thing once happened without+ taking the claim itself — a connector's removal asking whether it ever+ connected. Returns a tri-state rather than a bool: `"unknown"` on a failed+ lookup, since a lookup that could not run is not evidence the milestone+ was never claimed, and the caller reports it as a telemetry property that+ must say so rather than guess.++ Distinct from `claim_milestone`, whose `False` means "not yet claimed" and+ is always the safe answer for its callers (each takes the claim itself, so+ losing a race or a lookup only means reporting nothing rather than+ reporting wrongly) — that contract is unrelated to this one and is not+ changed here."""try:async with session_factory() as session:found = await session.execute(select(TelemetryMilestone.name).where(TelemetryMilestone.name == name))- return found.scalar_one_or_none() is not None+ return "true" if found.scalar_one_or_none() is not None else "false"except Exception:logger.warning(- "Could not read the %s telemetry milestone; treating it as unreported.",+ "Could not read the %s telemetry milestone; reporting it as unknown.",name,exc_info=True,)- return False+ return "unknown"async def claim_milestone(diff --git a/core/switch_core/telemetry/reporter.py b/core/switch_core/telemetry/reporter.pyindex 18d3e14a6..2dc09e9ca 100644--- a/core/switch_core/telemetry/reporter.py+++ b/core/switch_core/telemetry/reporter.py@@ -90,15 +90,9 @@ class SnapshotReporter:"""Collect and report one snapshot, plus any room that just went live."""counts = await collect_usage(self._session_factory, now=now)- # Skipped entirely on the first pass — see the module docstring.- active = (- await newly_active_rooms(self._session_factory, since=since, now=now)- if since is not None- else []- )-- logger.info("Usage snapshot: %s", summarise(counts, active))-+ # Emitted before the room half runs: per-tenant failures in either+ # call are contained inside `snapshot.py`, and a broader failure there+ # must not discard a snapshot that has already been collected.self._telemetry.emit("usage_snapshot",**counts.as_event_properties(@@ -106,17 +100,42 @@ class SnapshotReporter:),)- for room in active:- self._telemetry.emit(- "room_became_active",- seconds_since_room_created=room.seconds_since_room_created,- bridge_platform=room.bridge_platform,- channel_type=room.channel_type,- agent_count=room.agent_count,- created_by_kind=room.created_by_kind,+ # And nothing after that emit may raise out of this method. The+ # watermark advances only when `run_once` returns, and the loop retries+ # every poll interval — so a raise here would re-send the same snapshot+ # every five minutes for as long as the condition lasted, turning one+ # lost pass into hundreds of duplicates of a figure meant to be daily.+ # The rooms are the part that is given up instead: the window has moved+ # past them, which is the trade the docstring on `newly_active_rooms`+ # already describes, and it keeps "no room is reported twice" true.+ try:+ # Skipped entirely on the first pass — see the module docstring.+ active = (+ await newly_active_rooms(self._session_factory, since=since, now=now)+ if since is not None+ else [])- await self._report_first_room_active(active)+ logger.info("Usage snapshot: %s", summarise(counts, active))++ for room in active:+ self._telemetry.emit(+ "room_became_active",+ seconds_since_room_created=room.seconds_since_room_created,+ bridge_platform=room.bridge_platform,+ channel_type=room.channel_type,+ agent_count=room.agent_count,+ created_by_kind=room.created_by_kind,+ )++ await self._report_first_room_active(active)+ except Exception:+ logger.exception(+ "Usage snapshot: the snapshot was sent, but newly-active rooms "+ "could not be reported for this window. Any room that went "+ "active in it is not reported at all — the next window starts "+ "after it."+ )async def _report_first_room_active(self, active: list) -> None:"""The activation milestone: the first room a person actually used.diff --git a/core/switch_core/telemetry/snapshot.py b/core/switch_core/telemetry/snapshot.pyindex 54973f93a..446fc5aed 100644--- a/core/switch_core/telemetry/snapshot.py+++ b/core/switch_core/telemetry/snapshot.py@@ -16,13 +16,25 @@ something nobody needs within a day.from __future__ import annotationsimport logging+import timefrom collections.abc import Sequence-from dataclasses import dataclass, field+from dataclasses import dataclass, field, fieldsfrom datetime import UTC, datetime, timedeltafrom typing import Any-from sqlalchemy import Select, and_, case, distinct, exists, func, select+from sqlalchemy import (+ CompoundSelect,+ Select,+ and_,+ case,+ distinct,+ exists,+ func,+ select,+ union_all,+)from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker+from sqlalchemy.orm import aliasedfrom switch_core.db.models import (Agent,@@ -67,7 +79,24 @@ _WEEK = timedelta(days=7)class UsageCounts:"""The snapshot's numbers, accumulated across tenants."""+ # How many tenants actually made it into the sums below — not how many+ # tenants the deployment has. `collect_usage` collects one tenant at a+ # time into a scratch instance of this class and only folds it in once+ # that tenant's queries all succeed, so a tenant excluded after a failure+ # is excluded here too. That is what makes a partial pass a visibly+ # smaller number instead of the full tenant count sitting on top of an+ # undercounted everything else.tenant_count: int = 0+ # Tenants whose queries raised and were stepped over. Sits beside+ # `tenant_count` so a pass that excluded some is self-describing on the+ # wire: without it, a partial pass is every count dropping at once with+ # nothing to attribute it to.+ tenant_failed_count: int = 0+ # Wall time to collect the whole pass. Roughly fifteen queries per tenant+ # against the database that is also serving rooms, so what it costs at+ # scale is a real question — and a background task is invisible to the+ # HTTP request histogram that times everything else.+ duration_ms: int = 0user_count: int = 0user_active_1d: int = 0user_active_7d: int = 0@@ -109,6 +138,8 @@ class UsageCounts:"""Flatten to exactly the properties `usage_snapshot` declares."""properties: dict[str, float] = {"tenant_count": self.tenant_count,+ "tenant_failed_count": self.tenant_failed_count,+ "duration_ms": self.duration_ms,"user_count": self.user_count,"user_active_1d": self.user_active_1d,"user_active_7d": self.user_active_7d,@@ -156,7 +187,7 @@ class UsageCounts:@dataclass(frozen=True)class NewlyActiveRoom:- """A room whose first human interaction happened in the window just read."""+ """A room that became active in the window just read. See `newly_active_rooms`."""# The milestone is measured from this, not from when the pass ran: a pass# is up to a whole interval late, always in the same direction.@@ -168,53 +199,108 @@ class NewlyActiveRoom:created_by_kind: str-def _room_has_an_agent(tenant_id: str) -> Select[tuple[str]]:- """Correlated subquery: the message's room has an agent in it."""- return (+def _rooms_with_an_agent(tenant_id: str) -> CompoundSelect:+ """Room ids with an agent in them now, or that an agent has posted in.++ Uncorrelated, and matched with `IN` rather than as an `OR` of two+ `EXISTS`, so Postgres plans it as one semi-join: hashed across the tenant+ for the snapshot, or narrowed to the single room `room_had_human_activity`+ asks about. An `OR` of correlated probes cannot be planned that way and+ re-runs both for every human message, which on a large room with no agent+ is quadratic — and that check runs on every room archive and delete.++ Aliased so neither arm correlates with the human's `Client` and `Message`+ in the enclosing query.+ """+ member = aliased(Client)+ poster = aliased(Client)+ agent_message = aliased(Message)+ return union_all(select(ClientRoom.room_id)- .join(Client, Client.id == ClientRoom.client_id)+ .join(member, member.id == ClientRoom.client_id).where(- ClientRoom.room_id == Message.room_id,ClientRoom.tenant_id == tenant_id,- Client.type == AGENT_CLIENT_TYPE,- Client.tenant_id == tenant_id,- )+ member.type == AGENT_CLIENT_TYPE,+ member.tenant_id == tenant_id,+ ),+ select(agent_message.room_id)+ .join(poster, poster.id == agent_message.sender_client_id)+ .where(+ agent_message.tenant_id == tenant_id,+ agent_message.seq > 0,+ poster.type == AGENT_CLIENT_TYPE,+ poster.tenant_id == tenant_id,+ ),)-def _human_interaction(tenant_id: str, since: datetime) -> Select[tuple[str]]:- """Room ids a human spoke in since `since`.-- "Interaction" is a human posting in a room that has an agent in it. Both- halves matter: a message from a bridge relay or the admin client is not a- person, and a person talking in a room with no agent is not using the- product this telemetry is about. Two agents talking to each other is- likewise not activity, which is why this keys on the human side only.+def _human_activity_conditions(tenant_id: str) -> tuple[Any, ...]:+ """The one definition of "a human used this room", shared by all four paths+ that ask it: the two room-activity gauges, the once-per-room activation+ event, and the "was this room ever active" check on deletion and archival.+ They must agree, or the same room reads as active in one figure and+ never-active in another. Assumes the caller has already joined `Client` on+ `Message.sender_client_id`.++ Both conditions are kept:++ - **The room must have an agent in it, or have had one post in it.** A+ message from a bridge relay or the admin client is not a person, and a+ person talking in a room with no agent is not using the product this+ telemetry is about — two agents talking to each other is likewise not+ activity, which is why this keys on the human side only. Membership+ alone would make the lifetime questions depend on who is in the room at+ the moment they are asked: a busy room whose agents were removed before+ it was archived would read as never used. An agent's messages outlive+ its membership, so they answer for the room's history.+ - **`seq` must be positive.** `MessageStore.create_historical` backfills+ imported history with a negative `seq`, and a backfill is not someone+ using the product today. Nothing calls it yet, so this half is a latent+ guard rather than a live one — but it costs nothing to apply everywhere+ a message is read as activity, and every count in this module that+ already reads live traffic (`collect_tenant_counts`'s message and turn+ counts) applies the identical filter, so leaving it off here would be+ the inconsistent choice."""+ return (+ *_human_message_conditions(tenant_id),+ Message.room_id.in_(_rooms_with_an_agent(tenant_id)),+ )+++def _human_message_conditions(tenant_id: str) -> tuple[Any, ...]:+ """The message half of `_human_activity_conditions`: a person sent it, and+ it is live traffic rather than a backfill."""+ return (+ Client.type == HUMAN_CLIENT_TYPE,+ Client.tenant_id == tenant_id,+ Message.seq > 0,+ )+++def _human_interaction(tenant_id: str, since: datetime) -> Select[tuple[str]]:+ """Room ids a human used since `since`. See `_human_activity_conditions`."""return (select(distinct(Message.room_id)).join(Client, Client.id == Message.sender_client_id).where(Message.tenant_id == tenant_id,Message.sent_at >= since,- Client.type == HUMAN_CLIENT_TYPE,- Client.tenant_id == tenant_id,- exists(_room_has_an_agent(tenant_id)),+ *_human_activity_conditions(tenant_id),))def _active_humans(tenant_id: str, since: datetime) -> Select[tuple[str | None]]:- """Distinct human clients who interacted since `since`."""+ """Distinct human clients who used a room since `since`. See+ `_human_activity_conditions`."""return (select(distinct(Message.sender_client_id)).join(Client, Client.id == Message.sender_client_id).where(Message.tenant_id == tenant_id,Message.sent_at >= since,- Client.type == HUMAN_CLIENT_TYPE,- Client.tenant_id == tenant_id,- exists(_room_has_an_agent(tenant_id)),+ *_human_activity_conditions(tenant_id),))@@ -523,37 +609,106 @@ async def _collect_turns(async def room_had_human_activity(session: AsyncSession, tenant_id: str, room_id: str) -> bool:- """Whether a human has ever posted in this room.+ """Whether this room has ever seen human activity. See+ `_human_activity_conditions`.The session must already be bound to `tenant_id`; the predicate is namedanyway, for the reason `collect_tenant_counts` gives.++ The agent condition is asked of the one room rather than of each message.+ Uncorrelated, Postgres evaluates it once before reading anything, so a+ room with no agent is answered without scanning its history — this runs+ on every archive and delete."""+ agent_rooms = _rooms_with_an_agent(tenant_id).subquery()found = await session.execute(select(Message.id).join(Client, Client.id == Message.sender_client_id).where(Message.tenant_id == tenant_id,Message.room_id == room_id,- Message.seq > 0,- Client.type == HUMAN_CLIENT_TYPE,- Client.tenant_id == tenant_id,+ *_human_message_conditions(tenant_id),+ exists(+ select(agent_rooms.c.room_id).where(agent_rooms.c.room_id == room_id)+ ),).limit(1))return found.scalar_one_or_none() is not None+def _merge_tenant_counts(total: UsageCounts, tenant: UsageCounts) -> None:+ """Fold one tenant's collected numbers into the running deployment totals.++ Every field sums across tenants except `room_users_max` — the busiest room+ in the deployment is the largest of each tenant's busiest room, not their+ sum — and `connector_counts`, which sums per platform rather than as a+ dict. Neither `tenant_count` nor `user_count` is touched here: the first is+ the caller's to advance once a tenant's merge succeeds, and the second is+ never collected per tenant at all.+ """+ skip = {+ "tenant_count",+ "tenant_failed_count",+ "duration_ms",+ "user_count",+ "room_users_max",+ "connector_counts",+ }+ for f in fields(UsageCounts):+ if f.name in skip:+ continue+ setattr(total, f.name, getattr(total, f.name) + getattr(tenant, f.name))+ total.room_users_max = max(total.room_users_max, tenant.room_users_max)+ for platform, count in tenant.connector_counts.items():+ total.connector_counts[platform] += count++async def collect_usage(session_factory: async_sessionmaker[AsyncSession], *, now: datetime | None = None) -> UsageCounts:- """Every tenant's numbers, summed into one set of deployment totals."""+ """Every tenant's numbers, summed into one set of deployment totals.++ One tenant's failure does not take the whole pass down. Its queries run+ against a scratch `UsageCounts` rather than the running total, so a query+ that raises partway through a tenant leaves nothing from that tenant+ behind — not the fields collected before the failure, not any after —+ and the loop moves on to the next tenant with the total untouched. See+ `UsageCounts.tenant_count` and `tenant_failed_count` for how that failure+ is disclosed rather than swallowed.+ """moment = now or datetime.now(UTC)+ # Monotonic, not the wall clock: an NTP step mid-pass would otherwise+ # produce a duration the pass did not take, and a negative one is worse+ # than none at all — nothing at the far end can tell it from data.+ started = time.monotonic()counts = UsageCounts()tenant_ids = await all_tenant_ids(session_factory)- counts.tenant_count = len(tenant_ids)+ failed_tenants: list[str] = []for tenant_id in tenant_ids:- async with tenant_session(session_factory, tenant_id) as session:- await collect_tenant_counts(session, tenant_id, counts, moment)+ tenant_counts = UsageCounts()+ try:+ async with tenant_session(session_factory, tenant_id) as session:+ await collect_tenant_counts(session, tenant_id, tenant_counts, moment)+ except Exception:+ logger.exception(+ "Usage snapshot: failed to collect counts for tenant %s", tenant_id+ )+ failed_tenants.append(tenant_id)+ continue+ _merge_tenant_counts(counts, tenant_counts)+ counts.tenant_count += 1+ if failed_tenants:+ logger.error(+ "Usage snapshot: excluded %d of %d tenant(s) from this pass: %s. "+ "tenant_count on the emitted snapshot reports only what was "+ "counted, so it will read lower than the deployment's real "+ "tenant count until this is fixed.",+ len(failed_tenants),+ len(tenant_ids),+ ", ".join(failed_tenants),+ )+ counts.tenant_failed_count = len(failed_tenants)# `users` carries no tenant, so it is counted once for the deployment# rather than per tenant — summing a global table over tenants would# multiply it by however many there are.@@ -561,6 +716,7 @@ async def collect_usage(counts.user_count = await _scalar(session, select(func.count()).select_from(User))+ counts.duration_ms = round((time.monotonic() - started) * 1000)return counts@@ -570,83 +726,166 @@ async def newly_active_rooms(since: datetime,now: datetime | None = None,) -> list[NewlyActiveRoom]:- """Rooms whose *first* human interaction happened after `since`.+ """Rooms that became active after `since`.++ A room becomes active the moment it has both a human message and an agent:+ the later of its earliest human message and the earliest sign of an agent+ (the first agent to join among its current members, or the first agent+ message, whichever came first). Taking the later of the two is what lets a+ room where somebody spoke before any agent was added still be reported,+ once the agent arrives.The window is what makes this once-per-room without storing anything: a- room qualifies only if its earliest human message falls inside it, and each- window starts where the last one ended. A room that was already active- before `since` has an earlier first message and is skipped for good.+ room qualifies only if it became active inside it, and each window starts+ where the last one ended. A room that was already active before `since` is+ skipped for good.The cost of that is a room can be reported late — up to one window — andcannot be reported at all if the deployment was down when the window thatcovered it would have run. Both are acceptable for a figure nobody reads- inside a day, and neither can produce a duplicate.+ inside a day. One duplicate is possible: an agent that never posted,+ removed and added back, moves its room's join time later, and the room can+ be reported a second time.++ One tenant's failure is contained to that tenant, the same as+ `collect_usage`: its rooms are collected into a scratch list first and only+ folded into the result if every query for it succeeds, so a query that+ raises partway through never contributes a partial set of rooms. The+ caller advances its watermark regardless, so a contained tenant's rooms+ for this window are never reported — logged as an error rather than+ crashing the pass."""moment = now or datetime.now(UTC)found: list[NewlyActiveRoom] = []+ tenant_ids = await all_tenant_ids(session_factory)+ failed_tenants: list[str] = []- for tenant_id in await all_tenant_ids(session_factory):- async with tenant_session(session_factory, tenant_id) as session:- first_interaction = (- select(- Message.room_id.label("room_id"),- func.min(Message.sent_at).label("first_at"),+ for tenant_id in tenant_ids:+ tenant_found: list[NewlyActiveRoom] = []+ try:+ async with tenant_session(session_factory, tenant_id) as session:+ first_interaction = (+ select(+ Message.room_id.label("room_id"),+ func.min(Message.sent_at).label("first_at"),+ )+ .join(Client, Client.id == Message.sender_client_id)+ .where(+ Message.tenant_id == tenant_id,+ *_human_activity_conditions(tenant_id),+ )+ .group_by(Message.room_id)+ .subquery())- .join(Client, Client.id == Message.sender_client_id)- .where(- Message.tenant_id == tenant_id,- Client.tenant_id == tenant_id,- Client.type == HUMAN_CLIENT_TYPE,- Message.seq > 0,+ agent_joined = (+ select(+ ClientRoom.room_id.label("room_id"),+ func.min(ClientRoom.joined_at).label("at"),+ )+ .join(Client, Client.id == ClientRoom.client_id)+ .where(+ ClientRoom.tenant_id == tenant_id,+ Client.type == AGENT_CLIENT_TYPE,+ Client.tenant_id == tenant_id,+ )+ .group_by(ClientRoom.room_id)+ .subquery())- .group_by(Message.room_id)- .subquery()- )- agent_count = (- select(func.count())- .select_from(ClientRoom)- .join(Client, Client.id == ClientRoom.client_id)- .where(- ClientRoom.room_id == Room.id,- ClientRoom.tenant_id == tenant_id,- Client.type == AGENT_CLIENT_TYPE,- Client.tenant_id == tenant_id,+ agent_posted = (+ select(+ Message.room_id.label("room_id"),+ func.min(Message.sent_at).label("at"),+ )+ .join(Client, Client.id == Message.sender_client_id)+ .where(+ Message.tenant_id == tenant_id,+ Message.seq > 0,+ Client.type == AGENT_CLIENT_TYPE,+ Client.tenant_id == tenant_id,+ )+ .group_by(Message.room_id)+ .subquery())- .scalar_subquery()- )- rows = await session.execute(- select(- Room.created_at,- first_interaction.c.first_at,- Room.channel_type,- Room.bridge_id,- Room.metadata_["created_by_kind"].astext,- agent_count,+ # LEAST and GREATEST skip NULLs, so a room with no agent at all+ # would otherwise become active at its first human message.+ agent_since = func.least(agent_joined.c.at, agent_posted.c.at)+ activated_at = func.greatest(first_interaction.c.first_at, agent_since)+ agent_count = (+ select(func.count())+ .select_from(ClientRoom)+ .join(Client, Client.id == ClientRoom.client_id)+ .where(+ ClientRoom.room_id == Room.id,+ ClientRoom.tenant_id == tenant_id,+ Client.type == AGENT_CLIENT_TYPE,+ Client.tenant_id == tenant_id,+ )+ .scalar_subquery())- .join(first_interaction, first_interaction.c.room_id == Room.id)- .where(- and_(- Room.tenant_id == tenant_id,- first_interaction.c.first_at > since,- first_interaction.c.first_at <= moment,+ rows = await session.execute(+ select(+ Room.created_at,+ activated_at,+ Room.channel_type,+ Room.bridge_id,+ Room.metadata_["created_by_kind"].astext,+ agent_count,+ )+ .join(first_interaction, first_interaction.c.room_id == Room.id)+ .outerjoin(agent_joined, agent_joined.c.room_id == Room.id)+ .outerjoin(agent_posted, agent_posted.c.room_id == Room.id)+ .where(+ and_(+ Room.tenant_id == tenant_id,+ agent_since.is_not(None),+ activated_at > since,+ activated_at <= moment,+ )))- )- platforms = await _bridge_platforms(session)- for created_at, first_at, channel_type, bridge_id, kind, agents in rows:- found.append(- NewlyActiveRoom(- first_active_at=as_utc(first_at),- seconds_since_room_created=max(- (first_at - created_at).total_seconds(), 0.0- ),- bridge_platform=normalise_platform(platforms.get(bridge_id)),- channel_type=normalise_channel_type(channel_type),- agent_count=int(agents or 0),- created_by_kind=normalise_actor_kind(kind),+ platforms = await _bridge_platforms(session)+ for (+ created_at,+ active_at,+ channel_type,+ bridge_id,+ kind,+ agents,+ ) in rows:+ tenant_found.append(+ NewlyActiveRoom(+ first_active_at=as_utc(active_at),+ seconds_since_room_created=max(+ (active_at - created_at).total_seconds(), 0.0+ ),+ bridge_platform=normalise_platform(+ platforms.get(bridge_id)+ ),+ channel_type=normalise_channel_type(channel_type),+ agent_count=int(agents or 0),+ created_by_kind=normalise_actor_kind(kind),+ ))- )+ except Exception:+ logger.exception(+ "Usage snapshot: failed to collect newly-active rooms for tenant %s",+ tenant_id,+ )+ failed_tenants.append(tenant_id)+ continue+ found.extend(tenant_found)++ if failed_tenants:+ logger.error(+ "Usage snapshot: could not check %d of %d tenant(s) for "+ "newly-active rooms: %s. Any room of theirs that went active in "+ "this window will never be reported as room_became_active: the "+ "next window starts where this one ends.",+ len(failed_tenants),+ len(tenant_ids),+ ", ".join(failed_tenants),+ )return founddiff --git a/core/switch_core/transport/postgres.py b/core/switch_core/transport/postgres.pyindex 4faaae16c..281134861 100644--- a/core/switch_core/transport/postgres.py+++ b/core/switch_core/transport/postgres.py@@ -43,6 +43,7 @@ from typing import TYPE_CHECKING, Anyfrom switch_core.attachments import ATTACHMENT_GROUP_KEYfrom switch_core.db.models import ClientRoom, MediaBlob, Message, MessageAttachmentfrom switch_core.db.session_scope import tenant_session+from switch_core.logging_context import log_contextfrom switch_core.messages.recorded_types import EPHEMERALfrom switch_core.messages.row import attachments_in, text_field, thread_root_offrom switch_core.observability.catalogue import (@@ -287,22 +288,26 @@ class PostgresTransport:self._delivering = Truetry:for room_id in rooms:- try:- await self._drain_room(room_id)- except asyncio.CancelledError:- raise- except Exception:- # One room's failure is not the other rooms' problem,- # and this loop is the only delivery this client has.- # Counted as well as logged: swallowing it is what- # makes a stalled room invisible.- metrics().increment(DELIVERY_FAILURES, {})- logger.error(- "Delivery failed for client %s in room %s",- self.user_id,- room_id,- exc_info=True,- )+ # Bound around the drain rather than named in the message+ # below, so every line this room's delivery produces —+ # including the handlers' own — can be filtered to it.+ with log_context(room_id=room_id):+ try:+ await self._drain_room(room_id)+ except asyncio.CancelledError:+ raise+ except Exception:+ # One room's failure is not the other rooms'+ # problem, and this loop is the only delivery this+ # client has. Counted as well as logged: swallowing+ # it is what makes a stalled room invisible.+ metrics().increment(DELIVERY_FAILURES, {})+ logger.error(+ "Delivery failed for client %s in room %s",+ self.user_id,+ room_id,+ exc_info=True,+ )finally:self._delivering = Falsediff --git a/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py b/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.pyindex c41c3033e..1a023f03a 100644--- a/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py+++ b/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py@@ -26,6 +26,7 @@ from switch_core.clients.client_lifecycle_service import ClientLifecycleServicefrom switch_core.db.models import Client, Tenantfrom switch_core.db.stores.client_store import ClientStorefrom switch_core.db.stores.tenant_store import TenantStore+from switch_core.logging_context import current_log_context, log_contextfrom switch_core.tenant_context import current_tenant_id, tenant_scope@@ -39,6 +40,7 @@ class _RecordingClient:async def start(self) -> None:self._seen.append(current_tenant_id())+ self.room_seen = current_log_context().room_idasync def stop(self) -> None:return None@@ -77,6 +79,21 @@ async def test_a_client_task_runs_with_no_tenant_bound()+async def test_a_client_task_does_not_keep_the_room_it_was_created_in(+ session_factory: async_sessionmaker[AsyncSession],+) -> None:+ """The bridge's inbound handler binds its room as log context, and a+ puppet minted there must not log as that room for the rest of its life."""+ client = _RecordingClient([])+ service = _service(session_factory, MagicMock())++ with log_context(room_id="room-where-it-was-minted"):+ service._start_task("client-1", client) # type: ignore[arg-type]+ await asyncio.sleep(0)++ assert client.room_seen is None++@pytest.mark.no_ambient_tenantasync def test_start_all_binds_nothing_around_starting_each_client(session_factory: async_sessionmaker[AsyncSession],diff --git a/core/tests/switch_core/observability/test_bootstrap.py b/core/tests/switch_core/observability/test_bootstrap.pyindex 64014f659..b34efd6f9 100644--- a/core/tests/switch_core/observability/test_bootstrap.py+++ b/core/tests/switch_core/observability/test_bootstrap.py@@ -68,6 +68,7 @@ def _probes(**overrides) -> RuntimeProbes:defaults = dict(listener_connected=lambda: True,bridges_running=lambda: 2,+ bridges_running_by_platform=lambda: {"slack": 1, "mattermost": 1},bridges_configured=lambda: 2,clients_running=lambda: 5,connectors_running=lambda: 2,@@ -164,9 +165,17 @@ async def test_an_endpoint_installs_the_registry_and_reports_state(monkeypatch):assert values["switch.agents.connected"] == 3.0assert values["switch.clients.running"] == 5.0assert values["switch.connectors.running"] == 2.0- assert values["switch.bridges.running"] == 2.0assert values["switch.db.pool.in_use"] == 4.0assert values["switch.db.pool.size"] == 30.0++ # Bridges are reported per platform, so there is no unattributed+ # series to pick up above. Which platform is down is the first thing+ # anyone asks, and a bare total cannot answer it.+ running = {+ point.attributes["platform"]: point.value+ for point in payloads["switch.bridges.running"].numbers+ }+ assert running == {"slack": 1.0, "mattermost": 1.0}finally:await observability.aclose()diff --git a/core/tests/switch_core/observability/test_bridge_timing.py b/core/tests/switch_core/observability/test_bridge_timing.pynew file mode 100644index 000000000..c7f283772--- /dev/null+++ b/core/tests/switch_core/observability/test_bridge_timing.py@@ -0,0 +1,194 @@+"""Outbound platform calls are timed, and the running gauge says which.++Two gaps on the same signal. Bridges were counted in three ways and timed in+none, so a room that felt unresponsive — the commonest report there is — looked+from a dashboard like a healthy server relaying messages: the counters go up,+nothing errors, and people wait. An outbound relay is also the only external+API call this process makes, so its latency is the only place a slow platform+can show at all.++And `switch.bridges.running` was a bare total, which answers "how many bridges+died" and never "which" — a Slack outage and a misconfigured Teams app are the+same number.+"""++from __future__ import annotations++import pytest++from switch_core.bridges.collaboration.bridge_core import BridgeCore+from switch_core.bridges.collaboration.lifecycle_service import (+ CollaborationBridgeLifecycleService,+)+from switch_core.observability.catalogue import BRIDGE_CALL_DURATION+from switch_core.observability.metrics import MetricsRegistry, install, uninstall+++def registry():+ registry = MetricsRegistry()+ install(registry)+ yield registry+ uninstall()+++def _calls(registry: MetricsRegistry) -> dict[tuple[str, str], tuple[int, float]]:+ """(platform, kind) → (count, total ms)."""+ for payload in registry.collect():+ if payload.name != BRIDGE_CALL_DURATION.name:+ continue+ return {+ (point.attributes["platform"], point.attributes["kind"]): (+ point.count,+ point.total,+ )+ for point in payload.histograms+ }+ return {}+++class _Bridge:+ """The `_counted_outbound` contextmanager, off a real `BridgeCore`.++ Built by `__new__` rather than through the constructor: the manager reads+ one attribute, and standing up a whole `BridgeCore` would need a database,+ an adapter and six stores to test a `time.perf_counter()` pair.+ """++ def __new__(cls, bridge_type: str):+ core = object.__new__(BridgeCore)+ core._bridge_type = bridge_type # type: ignore[attr-defined]+ return core+++class TestAnOutboundRelayIsTimed:+ def test_a_successful_relay_records_its_duration(+ self, registry: MetricsRegistry+ ) -> None:+ bridge = _Bridge("slack")++ with bridge._counted_outbound("message"):+ pass++ assert _calls(registry)[("slack", "message")][0] == 1++ def test_the_platform_and_kind_are_carried(self, registry: MetricsRegistry) -> None:+ """Both, because "Slack is slow" and "attachments are slow" are+ different findings and a single series answers neither."""+ for platform, kind in [("slack", "message"), ("teams", "attachment")]:+ with _Bridge(platform)._counted_outbound(kind):+ pass++ assert set(_calls(registry)) == {("slack", "message"), ("teams", "attachment")}++ def test_a_failed_relay_is_not_timed(self, registry: MetricsRegistry) -> None:+ """A call that raised took however long its own failure took, which is+ a different distribution under the same name. `switch.bridge.errors` is+ where a failure shows."""+ bridge = _Bridge("mattermost")++ with pytest.raises(RuntimeError):+ with bridge._counted_outbound("message"):+ raise RuntimeError("platform refused")++ assert _calls(registry) == {}++ def test_the_exception_still_reaches_the_caller(+ self, registry: MetricsRegistry+ ) -> None:+ """Timing must not swallow: whoever handles the failure above still+ has to."""+ boom = RuntimeError("platform refused")++ with pytest.raises(RuntimeError) as raised:+ with _Bridge("discord")._counted_outbound("message"):+ raise boom++ assert raised.value is boom+++class TestTheRunningGaugeSaysWhichPlatform:+ def test_it_counts_each_platform_separately(self) -> None:+ service = object.__new__(CollaborationBridgeLifecycleService)+ service._started = {"a", "b", "c"} # type: ignore[attr-defined]+ service._platforms_seen = set() # type: ignore[attr-defined]+ service._bridges = {"a": object(), "b": object()} # type: ignore[attr-defined]+ service._tasks = { # type: ignore[attr-defined]+ "a": _DoneTask(False),+ "b": _DoneTask(False),+ "c": _DoneTask(True),+ }+ service._bridge_facts = { # type: ignore[attr-defined]+ "a": ("slack", None),+ "b": ("slack", None),+ "c": ("teams", None),+ }++ assert service.running_by_platform() == {"slack": 2, "teams": 0}++ def test_a_platform_that_is_fully_down_still_reports_zero(self) -> None:+ """The whole signal. A series that stops being reported looks on a+ dashboard exactly like one nobody is looking at — "Teams went from one+ to zero" is the alert, and it cannot fire on an absence."""+ service = object.__new__(CollaborationBridgeLifecycleService)+ service._started = {"c"} # type: ignore[attr-defined]+ service._platforms_seen = {"teams"} # type: ignore[attr-defined]+ service._bridges = {} # type: ignore[attr-defined]+ service._tasks = {"c": _DoneTask(True)} # type: ignore[attr-defined]+ service._bridge_facts = {"c": ("teams", None)} # type: ignore[attr-defined]++ assert service.running_by_platform() == {"teams": 0}++ def test_a_platform_whose_last_bridge_was_stopped_still_reports_zero(self) -> None:+ """`_started` drops a bridge that was stopped deliberately, so reading+ only that would end the series at the moment it has something to say —+ a dashboard cannot tell an ended series from one nobody is watching."""+ service = object.__new__(CollaborationBridgeLifecycleService)+ service._started = set() # type: ignore[attr-defined]+ service._platforms_seen = {"slack"} # type: ignore[attr-defined]+ service._bridges = {} # type: ignore[attr-defined]+ service._tasks = {} # type: ignore[attr-defined]+ service._bridge_facts = {} # type: ignore[attr-defined]++ assert service.running_by_platform() == {"slack": 0}++ def test_a_process_with_no_bridges_at_all_reports_nothing(self) -> None:+ """Not zero: there is no bridge here to be up or down, and five+ platforms sitting at zero would invite an alert on one nobody+ configured."""+ service = object.__new__(CollaborationBridgeLifecycleService)+ service._started = set() # type: ignore[attr-defined]+ service._platforms_seen = set() # type: ignore[attr-defined]+ service._bridges = {} # type: ignore[attr-defined]+ service._tasks = {} # type: ignore[attr-defined]+ service._bridge_facts = {} # type: ignore[attr-defined]++ assert service.running_by_platform() == {}++ def test_it_agrees_with_the_total_it_replaces(self) -> None:+ """The per-platform readings must sum to `running_count`, or two panels+ built from the same fact disagree."""+ service = object.__new__(CollaborationBridgeLifecycleService)+ service._started = {"a", "b", "c"} # type: ignore[attr-defined]+ service._platforms_seen = set() # type: ignore[attr-defined]+ service._bridges = {"a": object(), "b": object()} # type: ignore[attr-defined]+ service._tasks = { # type: ignore[attr-defined]+ "a": _DoneTask(False),+ "b": _DoneTask(False),+ "c": _DoneTask(True),+ }+ service._bridge_facts = { # type: ignore[attr-defined]+ "a": ("slack", None),+ "b": ("mattermost", None),+ "c": ("teams", None),+ }++ assert sum(service.running_by_platform().values()) == service.running_count()+++class _DoneTask:+ def __init__(self, done: bool) -> None:+ self._done = done++ def done(self) -> bool:+ return self._donediff --git a/core/tests/switch_core/observability/test_log_context_bindings.py b/core/tests/switch_core/observability/test_log_context_bindings.pynew file mode 100644index 000000000..7f470de5d--- /dev/null+++ b/core/tests/switch_core/observability/test_log_context_bindings.py@@ -0,0 +1,225 @@+"""The context fields are actually bound, not merely bindable.++Asserting a field *can* be bound proves it exists and nothing about whether+anything sets it, and a log field nobody binds is worse than no field: a column+of nulls that reads as "this never has a room" rather than "nobody wrote one".+So each test here drives a real code path and asserts the value reaches a+`LogRecord` — the filter is on the handler, so a bound value lands on every+record underneath it, including records from libraries.+"""++from __future__ import annotations++import asyncio+import logging+from typing import Any++import pytest++from switch_core.bridges.agent.protocol.connections import (+ PROTOCOL_VERSION,+ ClientDeclaration,+ ConnectionRegistry,+)+from switch_core.bridges.agent.protocol.event_buffer import EventBuffer+from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload+from switch_core.bridges.collaboration.bridge_core import BridgeCore+from switch_core.logging_context import LogContextFilter, current_log_context+from switch_core.transport.postgres import PostgresTransport+++class _Capture(logging.Handler):+ """Records with the context filter applied, as the real handler has it."""++ def __init__(self) -> None:+ super().__init__()+ self.addFilter(LogContextFilter(default_tenant_id="default"))+ self.records: list[logging.LogRecord] = []++ def emit(self, record: logging.LogRecord) -> None:+ self.records.append(record)++ def fields(self, name: str) -> list[Any]:+ return [getattr(record, name, None) for record in self.records]+++def captured():+ handler = _Capture()+ root = logging.getLogger()+ root.addHandler(handler)+ previous = root.level+ root.setLevel(logging.DEBUG)+ yield handler+ root.removeHandler(handler)+ root.setLevel(previous)+++class TestTheDeliveryLoopBindsItsRoom:+ """The loop drains one room at a time and its failure path names the room+ in the message. Everything *underneath* the drain — the handlers, the+ stores, whatever they log — had no room on it at all, which is most of what+ a stalled room produces."""++ async def test_a_failing_drain_carries_the_room(self, captured: _Capture) -> None:+ transport = object.__new__(PostgresTransport)+ transport.user_id = "@someone:test" # type: ignore[attr-defined]+ transport._wake = asyncio.Event() # type: ignore[attr-defined]+ transport._pending = {"room-7"} # type: ignore[attr-defined]+ transport._delivering = False # type: ignore[attr-defined]++ async def _boom(room_id: str) -> None:+ logging.getLogger("switch_core.test").info("inside the drain")+ raise RuntimeError("delivery blew up")++ transport._drain_room = _boom # type: ignore[attr-defined]+ transport._wake.set()++ loop = asyncio.create_task(transport._deliver_forever_unbound())+ await asyncio.sleep(0.05)+ loop.cancel()+ with pytest.raises(asyncio.CancelledError):+ await loop++ # Both the line the drain itself logged and the loop's own error line.+ assert "room-7" in captured.fields("room_id")+ assert captured.fields("room_id").count("room-7") >= 2++ async def test_the_binding_does_not_outlive_the_loop(+ self, captured: _Capture+ ) -> None:+ """A `ContextVar` set and never reset would leak one room's id onto+ every later line in the same task."""+ transport = object.__new__(PostgresTransport)+ transport.user_id = "@someone:test" # type: ignore[attr-defined]+ transport._wake = asyncio.Event() # type: ignore[attr-defined]+ transport._pending = {"room-7"} # type: ignore[attr-defined]+ transport._delivering = False # type: ignore[attr-defined]++ async def _ok(room_id: str) -> None:+ return None++ transport._drain_room = _ok # type: ignore[attr-defined]+ transport._wake.set()++ loop = asyncio.create_task(transport._deliver_forever_unbound())+ await asyncio.sleep(0.05)+ loop.cancel()+ with pytest.raises(asyncio.CancelledError):+ await loop++ assert current_log_context().room_id is None+++class TestTheProtocolBindsItsAgent:+ """Both of these are swept by the registry's own bookkeeping rather than+ called by an agent, so there is no request context to inherit — and both+ are exactly the line somebody goes looking for when one agent will not stay+ connected or keeps missing events."""++ def test_a_closing_connection_carries_the_agent(self, captured: _Capture) -> None:+ registry = ConnectionRegistry()+ registry.open(+ agent_id="agent-42",+ connection_id="c1",+ scope="single", # type: ignore[arg-type]+ delivery_filter="all", # type: ignore[arg-type]+ spawn_capable=False,+ cursor=0,+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),+ )++ registry.close("c1", "heartbeat lapsed")++ closed = [r for r in captured.records if "[CONN] closed" in r.getMessage()]+ assert closed, "the registry did not report the close at all"+ assert all(getattr(r, "agent_id", None) == "agent-42" for r in closed)++ def test_an_overflowing_buffer_carries_the_agent(self, captured: _Capture) -> None:+ buffer = EventBuffer(max_events_per_agent=2, retention_seconds=3600)+ for index in range(5):+ buffer.enqueue(+ "agent-99",+ "room-1",+ AgentEvent(+ type="message",+ room_id="room-1",+ payload=MessagePayload(+ addressed=True,+ sender="@u:s",+ sender_name="u",+ message_id=f"$m{index}",+ body="hi",+ timestamp=0,+ ),+ ),+ )++ overflow = [r for r in captured.records if "exceeded" in r.getMessage()]+ assert overflow, "the buffer did not report an overflow at all"+ assert all(getattr(r, "agent_id", None) == "agent-99" for r in overflow)+++class TestTheBridgeInboundPathBindsItsRoom:+ """`_traced` already resolves the channel's room to find the tenant, so the+ id is in hand — it just was not a field. An inbound platform event fans out+ across room lookup, identity provisioning and the transport, and none of+ those lines could be attributed to a room."""++ async def test_an_inbound_event_carries_the_room_it_is_for(+ self, captured: _Capture+ ) -> None:+ bridge = _bridge_core("slack", channel_to_room={"C1": ["room-5"]})++ async def handler(event) -> None:+ logging.getLogger("switch_core.test").info("handling the event")++ await bridge._traced("message", handler)(_InboundEvent("C1"))++ handled = [+ r for r in captured.records if "handling the event" in r.getMessage()+ ]+ assert handled, "the handler never ran"+ assert all(getattr(r, "room_id", None) == "room-5" for r in handled)++ async def test_a_channel_with_no_room_yet_binds_nothing(+ self, captured: _Capture+ ) -> None:+ """Auto-room-creation, still ahead of the handler. There is no room to+ name, and inventing one would be worse than the absence — `None` is the+ honest reading."""+ bridge = _bridge_core("slack", channel_to_room={})++ async def handler(event) -> None:+ logging.getLogger("switch_core.test").info("handling the event")++ await bridge._traced("message", handler)(_InboundEvent("C-unknown"))++ handled = [+ r for r in captured.records if "handling the event" in r.getMessage()+ ]+ assert handled+ assert all(getattr(r, "room_id", None) is None for r in handled)+++class _InboundEvent:+ def __init__(self, channel_id: str) -> None:+ self.channel_id = channel_id+++def _bridge_core(bridge_type: str, *, channel_to_room: dict[str, list[str]]):+ """A `BridgeCore` with only what `_traced` reads.++ Through `__new__` for the reason the outbound one is: the real constructor+ wants an adapter and six stores, none of which this path touches.+ """+ core = object.__new__(BridgeCore)+ core._bridge_type = bridge_type # type: ignore[attr-defined]+ core._bridge_tenant_id = "tenant-1" # type: ignore[attr-defined]+ core._channel_to_room = channel_to_room # type: ignore[attr-defined]++ async def _room_tenant(room_id: str) -> str:+ return "tenant-1"++ core._room_tenant = _room_tenant # type: ignore[attr-defined]+ return corediff --git a/core/tests/switch_core/observability/test_query_timing.py b/core/tests/switch_core/observability/test_query_timing.pynew file mode 100644index 000000000..81f3b241a--- /dev/null+++ b/core/tests/switch_core/observability/test_query_timing.py@@ -0,0 +1,181 @@+"""Database statements are timed, and the statement never becomes a label.++The pool gauges answer "are we holding too many connections". They cannot+answer "is anything slow", which is what a request taking two seconds actually+raises — and on this server a request is a handful of queries and very little+else, so a slow endpoint is nearly always a slow query. Until this existed+there was no query-duration signal of any kind: `grep '\\.observe('` over the+package returned two hits, neither of them the database.++The privacy half matters as much as the timing half. The statement text is the+most obviously useful attribute and the one that must never be one: it is+unbounded, and it carries literals on some paths.+"""++from __future__ import annotations++import pytest+from sqlalchemy import text+from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine++from switch_core.observability.catalogue import DB_QUERY_DURATION+from switch_core.observability.metrics import MetricsRegistry, install, uninstall+from switch_core.observability.query import _operation, instrument_queries+++def registry():+ registry = MetricsRegistry()+ install(registry)+ yield registry+ uninstall()+++def _durations(registry: MetricsRegistry) -> dict[str, tuple[int, float]]:+ """`operation` → (count, total ms), for whatever was recorded."""+ for payload in registry.collect():+ if payload.name != DB_QUERY_DURATION.name:+ continue+ return {+ point.attributes["operation"]: (point.count, point.total)+ for point in payload.histograms+ }+ return {}+++class TestTheOperationLabel:+ """A fixed table, so the series count is the size of that table however+ many distinct statements the server runs."""++ @pytest.mark.parametrize(+ ("statement", "expected"),+ [+ ("SELECT 1", "select"),+ (" select 1", "select"),+ ("\n\tSELECT\n 1", "select"),+ ("INSERT INTO rooms (id) VALUES ('r')", "insert"),+ ("UPDATE rooms SET name = 'x'", "update"),+ ("DELETE FROM rooms", "delete"),+ ("BEGIN", "other"),+ ("SET LOCAL app.tenant_id = 'acme'", "other"),+ ("CREATE TABLE t (a int)", "other"),+ ("", "other"),+ (" ", "other"),+ ],+ )+ def test_it_is_the_leading_keyword_or_other(+ self, statement: str, expected: str+ ) -> None:+ assert _operation(statement) == expected++ def test_a_statement_never_becomes_the_label(self) -> None:+ """The whole point. A room id or an email in a literal would otherwise+ become a metric series — unbounded, and on a dashboard read by people+ not entitled to that tenant's rows."""+ statement = "SELECT * FROM users WHERE email = '[email protected]'"++ assert _operation(statement) == "select"+ assert "example.test" not in _operation(statement)++ def test_every_label_it_can_produce_is_a_small_fixed_set(self) -> None:+ produced = {+ _operation(s)+ for s in [+ "SELECT",+ "INSERT",+ "UPDATE",+ "DELETE",+ "BEGIN",+ "COMMIT",+ "WITH x AS (SELECT 1) SELECT * FROM x",+ "EXPLAIN SELECT 1",+ "",+ ]+ }+ assert produced == {"select", "insert", "update", "delete", "other"}+++class TestTimingARealDatabase:+ async def test_it_times_the_round_trip(+ self, engine: AsyncEngine, registry: MetricsRegistry+ ) -> None:+ """`pg_sleep` is the only way to assert the measurement is the+ database's time rather than a constant: a bound below the sleep proves+ the clock is around the driver call, and one above it proves the+ reading is not the process uptime."""+ instrument_queries(engine)+ async with engine.connect() as conn:+ await conn.execute(text("SELECT pg_sleep(0.05)"))++ recorded = _durations(registry)+ count, total = recorded["select"]+ assert count == 1+ assert 50.0 <= total < 5_000.0++ async def test_each_operation_gets_its_own_series(+ self, engine: AsyncEngine, registry: MetricsRegistry+ ) -> None:+ instrument_queries(engine)+ async with engine.begin() as conn:+ await conn.execute(text("CREATE TEMP TABLE q (a int)"))+ await conn.execute(text("INSERT INTO q VALUES (1)"))+ await conn.execute(text("SELECT * FROM q"))+ await conn.execute(text("UPDATE q SET a = 2"))+ await conn.execute(text("DELETE FROM q"))++ recorded = _durations(registry)+ assert {"insert", "select", "update", "delete"} <= set(recorded)+ assert recorded["insert"][0] == 1++ async def test_a_failed_statement_is_not_timed(+ self, engine: AsyncEngine, registry: MetricsRegistry+ ) -> None:+ """A query that failed in four milliseconds is not evidence the+ database is fast. Mixing the two pulls the percentiles down exactly+ when something is wrong, which is when they are being read."""+ instrument_queries(engine)+ async with engine.connect() as conn:+ with pytest.raises(Exception):+ await conn.execute(text("SELECT * FROM a_table_that_is_not_there"))++ assert _durations(registry) == {}++ async def test_instrumenting_twice_does_not_double_count(+ self, engine: AsyncEngine, registry: MetricsRegistry+ ) -> None:+ """Both engines are built once, but a listener registered twice counts+ every query twice — invisible in a rate, and it doubles a p50.++ SQLAlchemy is what provides this: it refuses a `(target, identifier,+ fn)` it already holds, and both listeners are module-level functions so+ the identity matches. Pinned here rather than assumed, because it stops+ being true the moment either becomes a closure or a bound method — a+ refactor with no other visible effect."""+ instrument_queries(engine)+ instrument_queries(engine)++ async with engine.connect() as conn:+ await conn.execute(text("SELECT 1"))++ assert _durations(registry)["select"][0] == 1++ async def test_it_records_nothing_when_no_registry_is_installed(+ self, engine: AsyncEngine+ ) -> None:+ """The listeners are wired unconditionally at startup, so on a server+ with no `OTLP_ENDPOINT` every query runs through them into the null+ registry. That must be a no-op rather than an error, or observability+ being off would break the database."""+ uninstall()+ instrument_queries(engine)++ async with engine.connect() as conn:+ await conn.execute(text("SELECT 1"))+++async def engine(postgres_url: str):+ engine = create_async_engine(postgres_url)+ yield engine+ await engine.dispose()diff --git a/core/tests/switch_core/telemetry/test_bridge_connect_reporting.py b/core/tests/switch_core/telemetry/test_bridge_connect_reporting.pynew file mode 100644index 000000000..6250dc885--- /dev/null+++ b/core/tests/switch_core/telemetry/test_bridge_connect_reporting.py@@ -0,0 +1,577 @@+"""A bridge that fails to start is reported by something.++`bridge_connected{outcome:"failure"}` used to be emitted only from inside+`_run_bridge`'s own exception handler — but `_run_bridge` is scheduled as a+background task at the very end of `start()`, after every other way `start()`+can fail. An operator connecting a bridge on a host where its port (or its+stored config, or its adapter type) does not work never got a `start_all`-time+or request-time failure logged to telemetry at all: not a failure, because+nothing downstream reports one, and not a success, because none happened. The+connect success rate — `bridge_connected{outcome:"success"}` over every+`connector_configured` — was then computed over a denominator that silently+excluded the attempts that failed hardest.++These exercise `start()` itself (real Postgres, since the failures being+pinned are read failures — an unregistered adapter type, a resource already+held, a stored config that no longer validates) and confirm two things for+each: the `bridge_connected` failure fires with the right platform and+reason, and the original exception still reaches the caller unchanged, so+`start_all`'s existing catch-and-log and an HTTP handler's error response are+both unaffected by telemetry being added.+"""++from __future__ import annotations++import uuid+from typing import Any+from unittest.mock import AsyncMock, MagicMock++import pytest+from sqlalchemy import update+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++from switch_core.bridges.collaboration.lifecycle_service import (+ CollaborationBridgeLifecycleService,+)+from switch_core.bridges.collaboration.models import BridgeConnectionConfig+from switch_core.db.models import Client, CollaborationBridge+from switch_core.db.stores.client_store import ClientStore+from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore+from switch_core.db.stores.room_store import RoomStore+from switch_core.telemetry.service import TelemetryService+from switch_core.telemetry.sink import TelemetryRecord+from tests.switch_core.bridges.collaboration.test_lifecycle_callback_endpoint import (+ _FailingClient,+)+from tests.switch_core.bridges.collaboration.test_lifecycle_tenant_binding import (+ _make_bridge,+ _make_tenant,+ _StubAdapter,+ _StubConfig,+)+++class _RecordingSink:+ def __init__(self) -> None:+ self.sent: list[TelemetryRecord] = []++ async def send(self, record: TelemetryRecord) -> None:+ self.sent.append(record)++ async def aclose(self) -> None:+ return None+++def _service_with_telemetry(+ session_factory: async_sessionmaker[AsyncSession],+ *,+ callback_port: int = 0,+ client_store: Any = None,+) -> tuple[CollaborationBridgeLifecycleService, _RecordingSink]:+ sink = _RecordingSink()+ telemetry = TelemetryService(+ sink=sink, # type: ignore[arg-type]+ enabled=True,+ client_id="deployment-uuid",+ service_name="switch-core",+ version=None,+ environment=None,+ session_factory=session_factory,+ )+ config = MagicMock()+ config.gateway_public_url = "https://gw.example"+ config.collaboration_callback_host = "127.0.0.1"+ config.collaboration_callback_port = callback_port+ config.jwt_secret_key = "server-secret-for-tests"+ service = CollaborationBridgeLifecycleService(+ bridge_store=CollaborationBridgeStore(),+ external_user_store=MagicMock(),+ bridge_message_map_store=MagicMock(),+ session_request_post_store=MagicMock(),+ room_store=RoomStore(),+ agent_store=MagicMock(),+ client_store=client_store if client_store is not None else ClientStore(),+ client_lifecycle=MagicMock(),+ room_service=MagicMock(),+ matrix_admin=MagicMock(),+ session_factory=session_factory,+ config=config,+ client_factory=MagicMock(),+ telemetry=telemetry,+ )+ return service, sink+++async def _bridge_connected_events(sink: _RecordingSink) -> list[dict[str, Any]]:+ """Every `bridge_connected` the sink saw, in order, minus its duration.++ Sending is dispatched onto a background task by `TelemetryService.emit`, so+ the caller must have already closed (or otherwise awaited) the telemetry+ service producing `sink` before this is meaningful.++ `duration_ms` is elapsed wall time, so no test can pin its value — but+ dropping it silently would mean every assertion below passed against an+ event that had lost it, or that carried something no clock produces. It is+ asserted here to be a whole, non-negative number and then removed, so the+ comparisons stay exact.+ """+ events = []+ for record in sink.sent:+ if record.name != "switch_core.bridge_connected":+ continue+ properties = dict(record.properties)+ duration = properties.pop("duration_ms")+ assert isinstance(duration, int | float) and not isinstance(duration, bool)+ assert duration >= 0, "a connect cannot take less than no time"+ events.append(properties)+ return events+++class _StrictConfig(BridgeConnectionConfig):+ """A config schema with a required field nothing in these tests supplies,+ standing in for an adapter whose schema gained a new required setting+ after some bridges were already stored without it."""++ required_setting: str+++class _ExplodingAdapter(_StubAdapter):+ """An adapter whose construction itself fails — the same shape a real+ adapter takes if it does eager, fallible work in `__init__` (e.g.+ generating key material) rather than in `start`."""++ def __init__(self, *, config: Any) -> None:+ raise RuntimeError("adapter construction blew up")+++class _PortHoldingAdapter(_StubAdapter):+ @staticmethod+ def exclusive_resource(config: dict[str, Any]) -> str | None:+ port = config.get("listen_port")+ return f"port:{port}" if port else None+++class _FailingCore:+ """A `bridge_core.start()` that never reaches the platform at all -- the+ `_run_bridge` branch that still reports `bridge_connected{failure}`,+ as opposed to a `bridge_disconnected` for a bridge that connected and+ then dropped."""++ async def start(self) -> None:+ raise RuntimeError("adapter connect failed")+++async def _bridge_row(+ session: AsyncSession,+ *,+ tenant_id: str,+ bridge_type: str = "mattermost",+ connection_config: dict[str, Any] | None = None,+) -> str:+ """A stored bridge (and its client) with a caller-chosen type and config,+ for the scenarios `_make_bridge` (fixed to a plain "mattermost" row) does+ not cover."""+ client = Client(+ tenant_id=tenant_id,+ matrix_user_id=f"@bridge-{uuid.uuid4().hex[:8]}:test",+ display_name="bridge client",+ type="bridge",+ )+ session.add(client)+ await session.flush()+ bridge = CollaborationBridge(+ tenant_id=tenant_id,+ type=bridge_type,+ display_name="Test Bridge",+ client_id=client.id,+ status="active",+ connection_config=connection_config,+ )+ session.add(bridge)+ await session.flush()+ return bridge.id+++class TestEachStartFailurePointReportsAndReraises:+ async def test_a_bridge_id_nobody_has_heard_of(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`tenant_of_collaboration_bridge` finds nothing to bind, so there is+ no row and no platform to read off it.++ `unknown`, not `none`: every bridge is on some platform, so `none` is+ the value reserved for a room that has no bridge at all — and reporting+ an unreadable row as one would put a failure that cannot be attributed+ to any platform into the same bucket as the rooms that are on none.+ """+ service, sink = _service_with_telemetry(session_factory)++ with pytest.raises(ValueError, match="Bridge not found"):+ await service.start(f"no-such-bridge-{uuid.uuid4().hex[:8]}")+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "unknown",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]++ async def test_a_bridge_type_no_adapter_is_registered_for(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`lifecycle_service.py`'s `:541`. The platform is still nameable —+ the row says what type it is — even though nothing can run it."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ # Deliberately not calling register_adapter for "mattermost".++ with pytest.raises(ValueError, match="Unknown bridge type"):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "mattermost",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]++ async def test_a_bridge_type_the_catalogue_has_no_name_for(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`none` is reserved for a room with no bridge, so a bridge whose type+ cannot be named is `unknown`."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.execute(+ update(CollaborationBridge)+ .where(CollaborationBridge.id == bridge_id)+ .values(type="irc")+ )+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)++ with pytest.raises(ValueError, match="Unknown bridge type"):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "unknown",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]++ async def test_a_resource_another_bridge_already_holds(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`lifecycle_service.py`'s `:550-555`: the exact scenario an operator+ hits connecting Teams on a host whose listen port a bridge that+ predates the registration-time check already holds."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id = await _bridge_row(+ session,+ tenant_id=tenant,+ bridge_type="teams",+ connection_config={"listen_port": 3978},+ )+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ service.register_adapter("teams", _PortHoldingAdapter, _StubConfig)+ service._held_resources["some-other-bridge"] = "port:3978"++ with pytest.raises(ValueError, match="already held by bridge"):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "teams",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]++ async def test_a_stored_config_that_no_longer_validates(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`lifecycle_service.py`'s `:557`: an adapter's config schema gained a+ required field after this bridge was stored."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id = await _bridge_row(+ session, tenant_id=tenant, connection_config={}+ )+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ service.register_adapter("mattermost", _StubAdapter, _StrictConfig)++ with pytest.raises(Exception, match="required_setting"):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "mattermost",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]++ async def test_the_adapter_itself_fails_to_construct(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`lifecycle_service.py`'s `:558`."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ service.register_adapter("mattermost", _ExplodingAdapter, _StubConfig)++ with pytest.raises(RuntimeError, match="adapter construction blew up"):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "mattermost",+ "outcome": "failure",+ # RuntimeError carries no reason any rule recognises -- an+ # adapter constructor can fail for reasons that fit none of+ # `auth_failed`/`network`/`platform_error`/`config_invalid`.+ "failure_reason": "unknown",+ }+ ]++ async def test_the_bridges_own_client_row_is_gone(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`lifecycle_service.py`'s `:588`. The client row cannot actually be+ deleted out from under a live bridge (the foreign key refuses it), so+ this is exercised at the store boundary instead of with real bad data:+ `client_store.get` returning `None` is the only way `start` itself+ distinguishes this case from a client that is merely elsewhere."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ missing_client_store = AsyncMock()+ missing_client_store.get = AsyncMock(return_value=None)+ service, sink = _service_with_telemetry(+ session_factory, client_store=missing_client_store+ )+ service.register_adapter("mattermost", _StubAdapter, _StubConfig)++ with pytest.raises(ValueError, match="Bridge client not found"):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "mattermost",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]+++class TestStartAllsBootPathIsCoveredToo:+ async def test_a_bridge_that_fails_at_boot_is_reported_and_the_rest_still_start(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """`start_all` (`:309-313`) only ever logged and moved on past a+ bridge that failed to start — it still does, but the failure now+ reaches telemetry too, because `start` (which it calls unmodified)+ reports before raising back out to it."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ # No adapter is ever registered for "teams" below, so this one+ # fails at start()'s "Unknown bridge type" check.+ broken_bridge = await _bridge_row(+ session, tenant_id=tenant, bridge_type="teams"+ )+ healthy_bridge, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ service.register_adapter("mattermost", _StubAdapter, _StubConfig)+ # Isolates start_all's own boot loop from _run_bridge's separate+ # (already covered) success/failure reporting.+ service._run_bridge = AsyncMock() # type: ignore[method-assign]++ await service.start_all()+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert events == [+ {+ "bridge_platform": "teams",+ "outcome": "failure",+ "failure_reason": "config_invalid",+ }+ ]+ assert broken_bridge not in service._started+ assert healthy_bridge in service._started+++class TestStartingSuccessfullyDoesNotReportAnything:+ """`start` reports only on the paths that were previously silent. On the+ path that already worked -- scheduling `_run_bridge`, which reports its+ own success once the platform actually answers -- `start` itself must+ stay silent, or a working connect would now report twice."""++ async def test_start_alone_emits_nothing_when_it_schedules_the_task(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ service.register_adapter("mattermost", _StubAdapter, _StubConfig)+ # `_run_bridge` is where success (and a post-schedule failure) is+ # reported; stubbing it isolates exactly what `start` itself does.+ service._run_bridge = AsyncMock() # type: ignore[method-assign]++ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ assert await _bridge_connected_events(sink) == []+++class TestATaskFailureAfterStartSucceedsIsStillReportedExactlyOnce:+ """The other half of not double-reporting: `start` succeeding must not+ swallow a failure that only happens once `_run_bridge` actually runs, and+ that failure must not *also* be reported by `start`."""++ async def test_the_task_failing_is_the_only_event_and_start_reported_none(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ service.register_adapter("mattermost", _StubAdapter, _StubConfig)+ service._run_bridge = AsyncMock() # type: ignore[method-assign]++ await service.start(bridge_id)+ assert await _bridge_connected_events(sink) == []++ # What the scheduled task would have done, awaited directly rather+ # than raced, the same way `test_lifecycle_callback_endpoint.py`'s+ # crash test does -- except `bridge_core.start()` itself is what+ # fails here (`_FailingCore`), the branch that reports+ # `bridge_connected{failure}` from inside `_run_bridge`.+ # `bridge_client` never gets far enough to have `.start()` called, so+ # only its `client_id` attribute (read first, by+ # `_record_bridge_memberships`) matters.+ await type(service)._run_bridge(+ service, bridge_id, tenant, _FailingCore(), _FailingClient()+ )+ await service._telemetry.aclose() # type: ignore[union-attr]++ events = await _bridge_connected_events(sink)+ assert len(events) == 1+ assert events[0]["outcome"] == "failure"+++class TestHowLongTheConnectTook:+ """A connect that succeeds after ninety seconds and one that succeeds in+ two are otherwise the same row.++ Nothing else times this: a bridge comes up on a background task rather+ than inside a request the server serves, so it is invisible to+ `switch.http.request.duration`. The failure half matters more than the+ success half — a timeout and a refusal carry the same `failure_reason` and+ nothing alike in the time.+ """++ async def test_it_is_the_span_of_the_attempt_not_the_time_of_day(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A monotonic clock reads seconds since the process started, which is+ a large number that only grows — so an assertion that the duration is+ merely a non-negative number passes against a reading that was never+ subtracted from anything. Bounding it above is what catches that."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ # No adapter registered, so `start` fails immediately.+ with pytest.raises(ValueError):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ durations = [+ record.properties["duration_ms"]+ for record in sink.sent+ if record.name == "switch_core.bridge_connected"+ ]+ assert len(durations) == 1+ # Generous, because it is a real database read — but nothing like the+ # process uptime a bare clock reading would report.+ assert 0 <= durations[0] < 60_000++ async def test_a_second_attempt_is_timed_from_its_own_start(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """The reading is spent when the outcome is reported. Left behind, the+ next attempt on the same bridge would be timed from the first one, and+ a bridge that is retried reports a duration that only grows."""+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"+ async with session_factory() as session:+ await _make_tenant(session, tenant)+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)+ await session.commit()++ service, sink = _service_with_telemetry(session_factory)+ for _ in range(2):+ with pytest.raises(ValueError):+ await service.start(bridge_id)+ await service._telemetry.aclose() # type: ignore[union-attr]++ durations = [+ record.properties["duration_ms"]+ for record in sink.sent+ if record.name == "switch_core.bridge_connected"+ ]+ assert len(durations) == 2+ assert all(0 <= duration < 60_000 for duration in durations)diff --git a/core/tests/switch_core/telemetry/test_bridge_failure_reason.py b/core/tests/switch_core/telemetry/test_bridge_failure_reason.pynew file mode 100644index 000000000..1439d995e--- /dev/null+++ b/core/tests/switch_core/telemetry/test_bridge_failure_reason.py@@ -0,0 +1,315 @@+"""Which `BRIDGE_FAILURE_REASON` a bridge failure gets, and why.++`_failure_reason` used to match substrings in an exception's *class name*.+"api" is a substring of "SlackApiError", so a revoked or rotated Slack bot+token — the most common bridge failure in the field — reported as+`platform_error` ("Slack is broken") instead of `auth_failed` ("your+credentials are wrong"). These pin the real exception type each vendor SDK+actually raises (verified against the installed packages, not guessed) to the+value the classifier gives it, so a future rewrite cannot reintroduce that+silently.+"""++from __future__ import annotations++import aiohttp+import httpx+import pytest+from aiohttp.client_reqrep import ConnectionKey+from discord.errors import DiscordException, LoginFailure+from mattermostdriver.exceptions import (+ NoAccessTokenProvided,+ NotEnoughPermissions,+ ResourceNotFound,+)+from pydantic import BaseModel, ValidationError+from requests.exceptions import ConnectionError as RequestsConnectionError+from requests.exceptions import JSONDecodeError as RequestsJSONDecodeError+from requests.exceptions import Timeout as RequestsTimeout+from slack_sdk.errors import SlackApiError+from slack_sdk.web.async_slack_response import AsyncSlackResponse+from sqlalchemy.exc import DBAPIError+from telegram.error import (+ BadRequest,+ ChatMigrated,+ Conflict,+ Forbidden,+ InvalidToken,+ NetworkError,+)++from switch_core.bridges.collaboration.lifecycle_service import _failure_reason+from switch_core.bridges.collaboration.models import (+ BridgeCredentialError,+ BridgeOperationError,+)+from switch_core.telemetry.catalogue import BRIDGE_FAILURE_REASON+++def _slack_error(code: str) -> SlackApiError:+ """A `SlackApiError` carrying the response Slack actually sends: the whole+ distinction between a rejected token and every other refusal lives in+ `response["error"]`, not in the exception's type."""+ return SlackApiError(f"server responded: {code}", {"ok": False, "error": code})+++def _connector_error() -> aiohttp.ClientConnectorError:+ key = ConnectionKey("example.com", 443, True, False, None, None, None, None)+ return aiohttp.ClientConnectorError(+ connection_key=key, os_error=OSError(61, "Connection refused")+ )+++class _StoredBridgeConfig(BaseModel):+ """Stands in for a real `BridgeConnectionConfig` subclass — only its shape+ (a pydantic model) matters for producing a genuine `ValidationError`."""++ listen_port: int+++def _stale_stored_config_error() -> ValidationError:+ """What `config_cls.model_validate(bridge.connection_config)` raises for a+ row whose stored config no longer matches its adapter's schema."""+ try:+ _StoredBridgeConfig.model_validate({"listen_port": "not-a-port"})+ except ValidationError as exc:+ return exc+ raise AssertionError("expected model_validate to reject this payload")+++# (description, exception, expected BRIDGE_FAILURE_REASON)+CASES: list[tuple[str, BaseException, str]] = [+ (+ "an adapter's own signal that the platform rejected its credentials",+ BridgeCredentialError("Microsoft rejected these credentials"),+ "auth_failed",+ ),+ # The regression that matters: slack/adapter.py:442's `auth_test()` raises+ # this on a revoked or rotated bot token, and the substring classifier it+ # replaced read "SlackApiError" as carrying "api" and called it+ # `platform_error`.+ ("Slack invalid_auth", _slack_error("invalid_auth"), "auth_failed"),+ ("Slack not_authed", _slack_error("not_authed"), "auth_failed"),+ ("Slack account_inactive", _slack_error("account_inactive"), "auth_failed"),+ ("Slack token_revoked", _slack_error("token_revoked"), "auth_failed"),+ ("Slack token_expired", _slack_error("token_expired"), "auth_failed"),+ ("Slack rate limited", _slack_error("ratelimited"), "platform_error"),+ (+ "Slack refuses for an unrelated reason",+ _slack_error("internal_error"),+ "platform_error",+ ),+ (+ "Discord rejects the bot token at login",+ LoginFailure("Improper token"),+ "auth_failed",+ ),+ (+ "some other Discord SDK failure",+ DiscordException("the gateway connection dropped"),+ "platform_error",+ ),+ ("Mattermost 401 on admin login", NoAccessTokenProvided("401"), "auth_failed"),+ ("Mattermost 403 on admin login", NotEnoughPermissions("403"), "auth_failed"),+ (+ "Mattermost refuses for an unrelated reason",+ ResourceNotFound("404"),+ "platform_error",+ ),+ (+ "Mattermost's transport (requests) cannot connect",+ RequestsConnectionError("Connection refused"),+ "network",+ ),+ (+ "Mattermost's transport (requests) times out",+ RequestsTimeout("timed out"),+ "network",+ ),+ ("Telegram rejects the bot token", InvalidToken("Invalid token"), "auth_failed"),+ ("Telegram cannot be reached", NetworkError("timed out"), "network"),+ (+ "Telegram BadRequest — a NetworkError subclass that is a definite refusal",+ BadRequest("chat not found"),+ "platform_error",+ ),+ (+ "Telegram Forbidden (bot blocked)",+ Forbidden("bot was blocked by the user"),+ "platform_error",+ ),+ (+ "Telegram ChatMigrated",+ ChatMigrated(new_chat_id=-100123456789),+ "platform_error",+ ),+ (+ "Telegram refuses for an unrelated reason",+ Conflict("terminated by other getUpdates request"),+ "platform_error",+ ),+ ("aiohttp cannot connect", _connector_error(), "network"),+ (+ "aiohttp OS-level failure",+ aiohttp.ClientOSError(61, "Connection refused"),+ "network",+ ),+ (+ "aiohttp's server drops the connection",+ aiohttp.ServerDisconnectedError("Connection closed"),+ "network",+ ),+ (+ "httpx (Teams' token exchange and Graph calls) cannot connect",+ httpx.ConnectError("no route"),+ "network",+ ),+ (+ "Teams/Graph refuses an operation after credentials were accepted",+ BridgeOperationError("permission not granted"),+ "platform_error",+ ),+ (+ "a stored bridge config that no longer validates",+ _stale_stored_config_error(),+ "config_invalid",+ ),+ (+ "lifecycle_service's own invariant checks (ValueError)",+ ValueError("Bridge not found: some-id"),+ "config_invalid",+ ),+ (+ "adapter code reading a platform payload that changed shape",+ KeyError("channel"),+ "unknown",+ ),+ (+ "Switch's own database, not the messaging platform",+ DBAPIError("SELECT 1", {}, Exception("connection to server was lost")),+ "unknown",+ ),+ ("anything the classifier has no rule for", RuntimeError("a mystery"), "unknown"),+]+++class TestFailureReasonClassification:+ @pytest.mark.parametrize(+ "description,exc,expected", CASES, ids=[case[0] for case in CASES]+ )+ def test_classification(+ self, description: str, exc: BaseException, expected: str+ ) -> None:+ assert _failure_reason(exc) == expected, description++ def test_every_expected_value_is_one_the_catalogue_accepts(self) -> None:+ """`_failure_reason` and `bridge_connected.failure_reason` share one+ value set; a case above naming a value outside it would mean the+ classifier can produce something telemetry validation would reject."""+ for _, _, expected in CASES:+ assert BRIDGE_FAILURE_REASON.check(expected) is None+++class TestSlackAuthCodesAreExhaustiveOverNothingElse:+ """`response["error"]` carries free text from Slack; anything not on the+ known auth-code list must fall back to `platform_error` rather than being+ silently treated as a credentials problem."""++ @pytest.mark.parametrize(+ "code", ["missing_scope", "channel_not_found", "internal_error", "fatal_error"]+ )+ def test_a_non_auth_code_is_platform_error(self, code: str) -> None:+ assert _failure_reason(_slack_error(code)) == "platform_error"++ def test_a_response_with_no_error_key_does_not_crash(self) -> None:+ """Defensive: a malformed or unexpected response body must degrade to+ the safe default rather than raising out of the classifier itself."""+ assert _failure_reason(SlackApiError("boom", {"ok": False})) == "platform_error"++ def test_a_response_that_is_not_a_mapping_at_all_does_not_crash(self) -> None:+ """The case that actually bites.++ On the async client slack_sdk raises `SlackApiError(message, res)` with+ the raw `aiohttp.ClientResponse` — not a parsed body — whenever what it+ was handed is not the JSON the content type promised: an empty 502, a+ proxy's error page. That object has no `.get`.++ Reading it as a mapping raised `AttributeError` from inside the+ argument list of the very `emit_safely` call that reports the bridge+ failure. Arguments are evaluated before the call, so the guard did not+ cover it: the caller got an `AttributeError` in place of Slack's own+ exception, and the `bridge_connected` event — the one this whole area+ exists to emit — was never built.+ """++ class _NotAMapping:+ """Stands in for `aiohttp.ClientResponse`: no `.get`."""++ status = 502++ assert (+ _failure_reason(SlackApiError("boom", _NotAMapping())) # type: ignore[arg-type]+ == "platform_error"+ )++ def test_a_plain_text_response_does_not_crash(self) -> None:+ """A real `AsyncSlackResponse` holding a text body, as the async client+ builds for a proxy's plain-text 5xx. It has a `.get`, and that `.get`+ raises."""+ response = AsyncSlackResponse(+ client=None,+ http_verb="POST",+ api_url="https://slack.com/api/auth.test",+ req_args={},+ data="<html>502 Bad Gateway</html>", # type: ignore[arg-type]+ headers={},+ status_code=502,+ )++ assert _failure_reason(SlackApiError("boom", response)) == "platform_error"++ def test_a_response_of_none_does_not_crash(self) -> None:+ assert _failure_reason(SlackApiError("boom", None)) == "platform_error" # type: ignore[arg-type]+++class TestTheTransportFailuresThatHaveNoLibraryName:+ """A timeout and a refusal are the two outcomes an operator most needs to+ tell apart, and both must read `network`.++ Classifying on the individual leaf classes got this wrong in one direction+ per library: `httpx.ConnectTimeout` is a `TimeoutException` rather than a+ `ConnectError`, and `aiohttp.ServerTimeoutError` is not a `ClientOSError` —+ so the refusal classified and the timeout beside it came out `unknown`.+ Each library's own transport base class is what covers both.+ """++ @pytest.mark.parametrize(+ ("description", "exc"),+ [+ ("httpx connect refused", httpx.ConnectError("refused")),+ ("httpx connect timed out", httpx.ConnectTimeout("timed out")),+ ("httpx read timed out", httpx.ReadTimeout("timed out")),+ ("httpx pool exhausted", httpx.PoolTimeout("no connection")),+ ("httpx read failed", httpx.ReadError("reset")),+ ("aiohttp server hung up", aiohttp.ServerDisconnectedError()),+ ("aiohttp server timed out", aiohttp.ServerTimeoutError()),+ ("aiohttp connection reset", aiohttp.ClientConnectionResetError()),+ ("a bare asyncio timeout", TimeoutError()),+ ("a bare refused socket", ConnectionRefusedError()),+ ],+ )+ def test_it_reports_network(self, description: str, exc: Exception) -> None:+ assert _failure_reason(exc) == "network", description+++class TestABodyThatIsNotJsonIsNotTheOperatorsConfig:+ """`requests` folds an unparseable body into `InvalidJSONError`, which is a+ `ValueError` — so it fell through to `config_invalid` and reported a+ Mattermost server answering with a proxy error page as a connection config+ somebody typed wrong."""++ def test_an_unparseable_mattermost_body_is_the_platform(self) -> None:+ assert (+ _failure_reason(RequestsJSONDecodeError("not json", "<html>", 0))+ == "platform_error"+ )diff --git a/core/tests/switch_core/telemetry/test_deployment_and_reporter.py b/core/tests/switch_core/telemetry/test_deployment_and_reporter.pyindex 050c796c8..9de91b70a 100644--- a/core/tests/switch_core/telemetry/test_deployment_and_reporter.py+++ b/core/tests/switch_core/telemetry/test_deployment_and_reporter.py@@ -9,14 +9,22 @@ relies on a primary-key collision being the guard.from __future__ import annotations+import contextlib+import loggingimport uuid+from collections.abc import AsyncIteratorfrom datetime import UTC, datetime, timedeltaimport pytest-from sqlalchemy import delete+from sqlalchemy import delete, updatefrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker-from switch_core.db.models import DeploymentIdentity, TelemetrySnapshotWatermark+from switch_core.db.models import (+ TENANT_ZERO_ID,+ DeploymentIdentity,+ TelemetrySnapshotWatermark,+)+from switch_core.telemetry import snapshot as snapshot_modulefrom switch_core.telemetry.deployment import (DeploymentIdentityMissingError,claim_milestone,@@ -30,6 +38,20 @@ from switch_core.telemetry.sink import TelemetryRecordNOW = datetime(2026, 9, 16, 12, 0, tzinfo=UTC)+async def _always_broken_tenant_session(+ factory: async_sessionmaker[AsyncSession], tenant_id: str+) -> AsyncIterator[AsyncSession]:+ """A `tenant_session` replacement that fails for every tenant, every time.++ Stands in for a tenant whose queries never succeed — a permanently broken+ row, a statement that always times out — so a test can ask whether the+ schedule keeps moving regardless, rather than reproducing such a query.+ """+ raise RuntimeError("permanently broken tenant")+ yield # pragma: no cover - unreachable; keeps this an async generator++class _RecordingSink:def __init__(self) -> None:self.sent: list[TelemetryRecord] = []@@ -243,6 +265,56 @@ class TestTheSnapshotSchedule:names = [record.name for record in sink.sent]assert "switch_core.usage_snapshot" in names+ async def test_the_room_half_failing_does_not_resend_the_snapshot_forever(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """The snapshot is emitted before the newly-active-room half runs, so a+ failure there does not discard a pass already collected. That move has+ a trap: the watermark only advances when `run_once` returns, and+ `run_forever` retries every five minutes — so a raise *after* the emit+ would re-send the same daily figure hundreds of times a day, which is+ far worse than losing one pass. The room half is therefore contained+ too: the pass completes, the window moves on, and the rooms in it are+ given up rather than reported twice.+ """+ sink = _RecordingSink()+ reporter = self._reporter(sink, session_factory)++ # A watermark a day old: the pass below is due, and `since` is set —+ # which is what makes the room half run at all.+ async with session_factory() as session:+ await session.execute(delete(TelemetrySnapshotWatermark))+ session.add(+ TelemetrySnapshotWatermark(+ id=1, last_sent_at=datetime.now(UTC) - timedelta(days=2)+ )+ )+ await session.commit()++ # Only the room half: `collect_usage` runs before the emit, and a+ # failure there is the case the per-tenant containment already covers.+ def _explode(*_args: object, **_kwargs: object) -> list[object]:+ raise RuntimeError("the room query fell over")++ monkeypatch.setattr(+ "switch_core.telemetry.reporter.newly_active_rooms", _explode+ )++ with caplog.at_level(logging.ERROR, logger="switch_core.telemetry.reporter"):+ assert await reporter.run_once_if_due() is True++ # The pass completed, so the watermark advanced and the next one is not+ # due — which is what stops the duplicate.+ assert await reporter.run_once_if_due() is False+ # And it is disclosed rather than swallowed.+ assert any("newly-active rooms" in record.message for record in caplog.records)+ assert (+ len([r for r in sink.sent if r.name == "switch_core.usage_snapshot"]) == 1+ )+async def test_a_second_pass_inside_the_interval_does_nothing(self, session_factory: async_sessionmaker[AsyncSession]) -> None:@@ -320,3 +392,45 @@ class TestTheSnapshotSchedule:if record.name == "switch_core.usage_snapshot")assert snapshot.properties["session_live_count"] == 2++ async def test_a_permanently_broken_tenant_still_advances_the_watermark(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """The regression test for the defect a review found: `collect_usage`+ and `newly_active_rooms` used to let one tenant's exception escape the+ whole fan-out, and `_write_watermark` only runs after `run_once`+ returns — so a deterministically-failing tenant meant the watermark+ never advanced and no snapshot was ever sent again, for any tenant,+ for the life of the process. Contained per tenant, a snapshot still+ goes out and the schedule still moves, pass after pass, regardless."""+ await self._clear_watermark(session_factory)+ monkeypatch.setattr(+ snapshot_module, "tenant_session", _always_broken_tenant_session+ )+ sink = _RecordingSink()+ reporter = self._reporter(sink, session_factory)++ with caplog.at_level(logging.ERROR, logger="switch_core.telemetry.snapshot"):+ sent_first = await reporter.run_once_if_due()++ assert sent_first is True+ assert "switch_core.usage_snapshot" in [record.name for record in sink.sent]+ assert any(TENANT_ZERO_ID in record.message for record in caplog.records)++ # The watermark really did advance: called again immediately, the+ # schedule says not due rather than trying, and failing, again.+ assert await reporter.run_once_if_due() is False++ # And it keeps moving on a later pass, not only the first one.+ async with session_factory() as session:+ await session.execute(+ update(TelemetrySnapshotWatermark).values(+ last_sent_at=datetime.now(UTC) - timedelta(days=2)+ )+ )+ await session.commit()++ assert await reporter.run_once_if_due() is Truediff --git a/core/tests/switch_core/telemetry/test_milestone_claimed.py b/core/tests/switch_core/telemetry/test_milestone_claimed.pynew file mode 100644index 000000000..c17bc866a--- /dev/null+++ b/core/tests/switch_core/telemetry/test_milestone_claimed.py@@ -0,0 +1,70 @@+"""`milestone_claimed`: a tri-state read of a once-ever milestone.++Distinct from `claim_milestone` (`test_deployment_and_reporter.py`), which+takes the claim and is always safe to report `False` from: each of its+callers takes the claim itself, so losing a race or a lookup only means+reporting nothing rather than reporting wrongly.++`milestone_claimed` is read-only, for the one caller that needs to know a+thing once happened without claiming it — a connector's removal asking+whether it ever connected. A lookup that cannot run is not evidence either+way, so it reports `"unknown"` rather than collapsing into `"false"`, which+would misfile a lookup failure as "this connector never worked".+"""++from __future__ import annotations++from types import TracebackType++from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++from switch_core.telemetry.deployment import claim_milestone, milestone_claimed+++class _BrokenSession:+ """Fails on entry, the way a lost connection or a locked table would."""++ async def __aenter__(self) -> _BrokenSession:+ raise RuntimeError("database unreachable")++ async def __aexit__(+ self,+ exc_type: type[BaseException] | None,+ exc: BaseException | None,+ tb: TracebackType | None,+ ) -> bool:+ return False+++def _broken_session_factory() -> _BrokenSession:+ return _BrokenSession()+++class TestMilestoneClaimedIsTriState:+ async def test_an_unclaimed_milestone_reports_false(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ result = await milestone_claimed(session_factory, "first_room_created")++ assert result == "false"++ async def test_a_claimed_milestone_reports_true(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ await claim_milestone(session_factory, "first_room_created")++ result = await milestone_claimed(session_factory, "first_room_created")++ assert result == "true"++ async def test_a_failed_lookup_reports_unknown_rather_than_false(self) -> None:+ """The defect this guards: a connector that worked for months, removed+ while the milestone table cannot be read, must not report as though it+ never connected — `connector_removed.was_ever_connected` would then+ say "failed setup" about a deployment for which the opposite is true."""+ result = await milestone_claimed(+ _broken_session_factory, # type: ignore[arg-type]+ "connector_added:some-bridge",+ )++ assert result == "unknown"diff --git a/core/tests/switch_core/telemetry/test_removals.py b/core/tests/switch_core/telemetry/test_removals.pyindex 566c43e41..feae17727 100644--- a/core/tests/switch_core/telemetry/test_removals.py+++ b/core/tests/switch_core/telemetry/test_removals.py@@ -18,6 +18,7 @@ from datetime import UTC, datetime, timedeltaimport pytest+from switch_core.telemetry.ages import UNKNOWN_AGE, age_days, age_hoursfrom switch_core.telemetry.catalogue import CATALOGUE, TelemetryCatalogueError, validate@@ -37,11 +38,40 @@ class TestTheRemovalEventsExist:"channel_type": "channel_public","created_by_kind": "user","age_days": 12.5,- "was_ever_active": True,+ "was_ever_active": "true","agent_count": 3,},)+ def test_room_deletion_reports_unknown_rather_than_a_guess(self) -> None:+ """A failed activity lookup is not evidence the room was never used —+ it must say so rather than reporting `false`."""+ validate(+ "room_deleted",+ {+ "bridge_platform": "unknown",+ "channel_type": "channel_public",+ "created_by_kind": "user",+ "age_days": 12.5,+ "was_ever_active": "unknown",+ "agent_count": 3,+ },+ )++ def test_was_ever_active_is_no_longer_a_boolean(self) -> None:+ """Locks in the tri-state: a bare `True`/`False` must be refused so a+ caller cannot silently smuggle a boolean back in."""+ payload = {+ "bridge_platform": "slack",+ "channel_type": "channel_public",+ "created_by_kind": "user",+ "age_days": 12.5,+ "was_ever_active": True,+ "agent_count": 3,+ }+ with pytest.raises(TelemetryCatalogueError, match="expected one of"):+ validate("room_deleted", payload) # type: ignore[arg-type]+def test_agent_deletion_carries_what_it_was(self) -> None:validate("agent_deleted",@@ -61,7 +91,21 @@ class TestTheRemovalEventsExist:{"bridge_platform": "teams","age_days": 90.0,- "was_ever_connected": False,+ "was_ever_connected": "false",+ "room_count": 0,+ },+ )++ def test_connector_removal_reports_unknown_rather_than_a_guess(self) -> None:+ """A failed read of the durable milestone record is not evidence the+ connector never connected — collapsing it into `false` would misfile a+ lookup failure as a failed setup."""+ validate(+ "connector_removed",+ {+ "bridge_platform": "teams",+ "age_days": 90.0,+ "was_ever_connected": "unknown","room_count": 0,},)@@ -75,7 +119,7 @@ class TestTheRemovalEventsExist:"channel_type": "channel_public","created_by_kind": "user","age_days": 1.0,- "was_ever_active": True,+ "was_ever_active": "true","agent_count": 1,"room_name": "incident-response",},@@ -152,18 +196,37 @@ class TestAutoSessionIsAnAgentType:class TestAgeIsReportedInDays:- def test_a_fresh_row_is_near_zero(self) -> None:- from switch_core.room_service import _age_days-- assert _age_days(datetime.now(UTC)) < 0.01-- def test_an_old_row_reports_its_age(self) -> None:- from switch_core.room_service import _age_days+ """One implementation, tested once.- assert 9.9 < _age_days(datetime.now(UTC) - timedelta(days=10)) < 10.1+ There were eight copies of this arithmetic across the modules that emit a+ removal, and they disagreed on the only case that matters — half reported+ `0.0` for a timestamp they could not read and half reported `-1`. The+ duplication was the defect; a test per copy would only have pinned it.+ """- def test_an_unreadable_timestamp_is_zero_rather_than_a_crash(self) -> None:- from switch_core.room_service import _age_days+ def test_a_fresh_row_is_near_zero(self) -> None:+ assert 0 <= age_days(datetime.now(UTC)) < 0.01- assert _age_days(None) == 0.0- assert _age_days("yesterday") == 0.0+ def test_an_old_row_reports_its_age(self) -> None:+ assert 9.9 < age_days(datetime.now(UTC) - timedelta(days=10)) < 10.1++ def test_hours_are_the_same_elapsed_time_in_a_different_unit(self) -> None:+ moment = datetime.now(UTC) - timedelta(days=1)+ assert 23.9 < age_hours(moment) < 24.1++ @pytest.mark.parametrize("unreadable", [None, "yesterday", 0, object()])+ def test_an_unreadable_timestamp_is_negative_rather_than_a_crash(+ self, unreadable: object+ ) -> None:+ """`-1` rather than `0`: "could not tell" must stay distinguishable+ from "created just now", per docs/old/telemetry-events.md. Pinning `0`+ here would have locked in the opposite — a lookup failure silently+ reporting as a genuine, very fresh row."""+ assert age_days(unreadable) == UNKNOWN_AGE+ assert age_hours(unreadable) == UNKNOWN_AGE++ def test_a_clock_stepped_backwards_does_not_look_like_unknown(self) -> None:+ """The wall clock is unavoidable here — these ages span restarts — so a+ row stamped in the future must clamp to zero rather than going negative+ and colliding with the sentinel."""+ assert age_days(datetime.now(UTC) + timedelta(days=1)) == 0.0diff --git a/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py b/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.pynew file mode 100644index 000000000..024bfebbc--- /dev/null+++ b/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py@@ -0,0 +1,84 @@+"""`RoomService`'s telemetry helpers must not turn "the lookup failed" into a+value that looks like a real answer.++`_bridge_platform` used to return `"none"` — the same value an internal-only+room reports — for a lookup that raised, and `_was_ever_active` used to return+`False` — the same value a room with no activity reports — for a count that+raised. Both are read only when building `room_created`/`room_deleted`/+`room_archived`, so a transient database error at exactly the wrong moment+silently relabelled a Slack-bridged, actively-used room as an internal-only+room nobody ever spoke in. No real database is needed to exercise the failure+path: a session factory that fails on entry reproduces it directly.+"""++from __future__ import annotations++from types import TracebackType++from switch_core.room_service import RoomService+++class _BrokenSession:+ """Fails on entry, the way a lost connection or a locked table would."""++ async def __aenter__(self) -> _BrokenSession:+ raise RuntimeError("database unreachable")++ async def __aexit__(+ self,+ exc_type: type[BaseException] | None,+ exc: BaseException | None,+ tb: TracebackType | None,+ ) -> bool:+ return False+++def _broken_session_factory() -> _BrokenSession:+ return _BrokenSession()+++def _room_service() -> RoomService:+ """Only `_session_factory` and `_telemetry` are exercised by the methods+ under test; everything else is unreachable once the session fails."""+ return RoomService(+ matrix_admin=None, # type: ignore[arg-type]+ room_store=None, # type: ignore[arg-type]+ agent_store=None, # type: ignore[arg-type]+ client_lifecycle=None, # type: ignore[arg-type]+ collab_lifecycle=None, # type: ignore[arg-type]+ collab_bridge_store=None, # type: ignore[arg-type]+ resource_service=None, # type: ignore[arg-type]+ session_factory=_broken_session_factory, # type: ignore[arg-type]+ telemetry=object(), # type: ignore[arg-type]+ )+++class TestBridgePlatformReportsUnknownOnFailure:+ async def test_a_failed_lookup_is_unknown_not_none(self) -> None:+ """`none` means "no bridge"; a failed lookup must not say the same+ thing about a room that has one."""+ service = _room_service()++ result = await service._bridge_platform("bridge-1")++ assert result == "unknown"++ async def test_no_bridge_id_is_still_none(self) -> None:+ """The two cases must stay distinguishable in both directions: a+ genuinely internal-only room is not reported as unknown either."""+ service = _room_service()++ result = await service._bridge_platform(None)++ assert result == "none"+++class TestWasEverActiveReportsUnknownOnFailure:+ async def test_a_failed_lookup_is_unknown_not_false(self) -> None:+ """A count that could not run is not evidence the room was never+ used — the exact fact this property exists to carry."""+ service = _room_service()++ result = await service._was_ever_active("tenant-1", "room-1")++ assert result == "unknown"diff --git a/core/tests/switch_core/telemetry/test_snapshot.py b/core/tests/switch_core/telemetry/test_snapshot.pyindex 3cd0fee4f..5642be87e 100644--- a/core/tests/switch_core/telemetry/test_snapshot.py+++ b/core/tests/switch_core/telemetry/test_snapshot.py@@ -14,6 +14,7 @@ from __future__ import annotationsimport uuidfrom datetime import UTC, datetime, timedelta+from sqlalchemy import deletefrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmakerfrom switch_core.db.models import (@@ -111,8 +112,20 @@ async def _agent(session: AsyncSession, runtime: str | None) -> Agent:return agent-async def _join(session: AsyncSession, client: Client, room: Room) -> None:- session.add(ClientRoom(client_id=client.id, room_id=room.id))+async def _join(+ session: AsyncSession,+ client: Client,+ room: Room,+ *,+ when: datetime | None = None,+) -> None:+ """Pinned to the room's creation unless told otherwise: the column defaults+ to the database clock, which is later than `NOW`."""+ session.add(+ ClientRoom(+ client_id=client.id, room_id=room.id, joined_at=when or room.created_at+ )+ )await session.flush()@@ -234,6 +247,26 @@ class TestWhatCountsAsActive:assert counts.room_active_1d == 0assert counts.room_active_7d == 1+ async def test_backfilled_history_is_not_activity_either(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Same guard `message_count_1d` already applies, now shared with the+ activity gauges: a negative `seq` is reconstructed history, and a+ backfill is not a person using the room today even with an agent+ present."""+ async with session_factory() as session:+ room = await _room(session)+ human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, human, room)+ await _join(session, agent, room)+ await _say(session, room, human, seq=-1)++ counts = await _counts(session)++ assert counts.room_active_1d == 0+ assert counts.user_active_1d == 0+class TestRoomsAreCountedByWhoMadeThem:async def test_a_user_created_room_is_the_headline_figure(@@ -444,6 +477,8 @@ class TestNewlyActiveRooms:async with session_factory() as session:room = await _room(session, created_at=created)human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, agent, room)await _say(session, room, human, seq=1, when=NOW - timedelta(hours=2))await session.commit()@@ -463,6 +498,8 @@ class TestNewlyActiveRooms:async with session_factory() as session:room = await _room(session, created_at=NOW - timedelta(days=10))human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, agent, room)await _say(session, room, human, seq=1, when=NOW - timedelta(days=9))await _say(session, room, human, seq=2, when=NOW - timedelta(hours=1))await session.commit()@@ -488,6 +525,83 @@ class TestNewlyActiveRooms:assert found == []+ async def test_a_room_with_humans_and_no_agent_never_becomes_active(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """The defect this predicate now shares with `room_active_1d`: two+ people talking with no agent in the room is not "using the product",+ so it must not surface here either, even though nobody has ever+ called this room active by the room-count definition."""+ async with session_factory() as session:+ room = await _room(session)+ first = await _client(session, "user")+ second = await _client(session, "user")+ await _join(session, first, room)+ await _join(session, second, room)+ await _say(session, room, first, seq=1, when=NOW - timedelta(hours=2))+ await _say(session, room, second, seq=2, when=NOW - timedelta(hours=1))+ await session.commit()++ found = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=4), now=NOW+ )++ assert found == []++ async def test_backfilled_history_does_not_report_a_room_as_newly_active(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with session_factory() as session:+ room = await _room(session)+ human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, agent, room)+ await _say(session, room, human, seq=-1, when=NOW - timedelta(hours=1))+ await session.commit()++ found = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=4), now=NOW+ )++ assert found == []++ async def test_a_person_who_spoke_before_any_agent_counts_once_one_arrives(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A bridged channel often has people in it before anyone adds an agent.+ The room becomes active when the agent arrives, and is reported in the+ window that covers that moment rather than never."""+ async with session_factory() as session:+ room = await _room(session, created_at=NOW - timedelta(hours=10))+ human = await _client(session, "user")+ await _join(session, human, room)+ await _say(session, room, human, seq=1, when=NOW - timedelta(hours=6))+ await session.commit()++ before_the_agent = await newly_active_rooms(+ session_factory,+ since=NOW - timedelta(hours=8),+ now=NOW - timedelta(hours=5),+ )++ async with session_factory() as session:+ agent = await _client(session, "agent")+ await _join(session, agent, room, when=NOW - timedelta(hours=3))+ await session.commit()++ when_it_arrives = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=5), now=NOW+ )+ the_window_after = await newly_active_rooms(+ session_factory, since=NOW, now=NOW + timedelta(hours=5)+ )++ assert before_the_agent == []+ assert len(when_it_arrives) == 1+ assert when_it_arrives[0].first_active_at == NOW - timedelta(hours=3)+ assert when_it_arrives[0].seconds_since_room_created == 7 * 3600+ assert the_window_after == []+class TestRoomHadHumanActivity:async def test_it_reports_whether_a_person_ever_posted(@@ -497,6 +611,8 @@ class TestRoomHadHumanActivity:quiet = await _room(session)busy = await _room(session)human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, agent, busy)await _say(session, busy, human, seq=1)assert await room_had_human_activity(session, TENANT_ZERO, busy.id) is True@@ -504,6 +620,133 @@ class TestRoomHadHumanActivity:await room_had_human_activity(session, TENANT_ZERO, quiet.id) is False)+ async def test_a_room_with_no_agent_never_had_human_activity(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Same predicate as `room_active_1d`: a person posting in a room with+ no agent is not the activity this telemetry means, so it must not make+ `room_deleted.was_ever_active` or `room_archived.was_ever_active`+ `True` either."""+ async with session_factory() as session:+ room = await _room(session)+ human = await _client(session, "user")+ await _join(session, human, room)+ await _say(session, room, human, seq=1)++ assert await room_had_human_activity(session, TENANT_ZERO, room.id) is False++ async def test_backfilled_history_is_not_human_activity(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with session_factory() as session:+ room = await _room(session)+ human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, agent, room)+ await _say(session, room, human, seq=-1)++ assert await room_had_human_activity(session, TENANT_ZERO, room.id) is False+++class TestTheFourPathsAgreeOnActivity:+ """`_human_interaction`, `_active_humans`, `newly_active_rooms` and+ `room_had_human_activity` used to answer "did a human use this room"+ differently — two required an agent in the room and two did not — so the+ same room could be active in one figure and never-active in another. They+ now share `_human_activity_conditions`; this pins that all four agree on+ the same two rooms rather than exercising each in isolation."""++ async def test_all_four_agree_a_room_with_no_agent_is_not_active(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with session_factory() as session:+ room = await _room(session, created_at=NOW - timedelta(hours=5))+ first = await _client(session, "user")+ second = await _client(session, "user")+ await _join(session, first, room)+ await _join(session, second, room)+ await _say(session, room, first, seq=1, when=NOW - timedelta(hours=2))+ await _say(session, room, second, seq=2, when=NOW - timedelta(hours=1))+ await session.flush()++ counts = await _counts(session)+ was_ever_active = await room_had_human_activity(+ session, TENANT_ZERO, room.id+ )+ await session.commit()++ newly_active = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=4), now=NOW+ )++ assert counts.room_active_1d == 0+ assert counts.user_active_1d == 0+ assert was_ever_active is False+ assert newly_active == []++ async def test_all_four_agree_a_room_with_an_agent_is_active(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with session_factory() as session:+ room = await _room(session, created_at=NOW - timedelta(hours=5))+ human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, human, room)+ await _join(session, agent, room)+ await _say(session, room, human, seq=1, when=NOW - timedelta(hours=2))+ await session.flush()++ counts = await _counts(session)+ was_ever_active = await room_had_human_activity(+ session, TENANT_ZERO, room.id+ )+ await session.commit()++ newly_active = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=4), now=NOW+ )++ assert counts.room_active_1d == 1+ assert counts.user_active_1d == 1+ assert was_ever_active is True+ assert len(newly_active) == 1++ async def test_all_four_agree_a_room_keeps_its_history_when_its_agent_leaves(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Membership is only who is in the room now. A room archived after its+ agent was removed was still used, and the agent's replies say so."""+ async with session_factory() as session:+ room = await _room(session, created_at=NOW - timedelta(hours=5))+ human = await _client(session, "user")+ agent = await _client(session, "agent")+ await _join(session, human, room)+ await _join(session, agent, room)+ await _say(session, room, human, seq=1, when=NOW - timedelta(hours=2))+ await _say(session, room, agent, seq=2, when=NOW - timedelta(minutes=90))+ await session.execute(+ delete(ClientRoom).where(+ ClientRoom.room_id == room.id, ClientRoom.client_id == agent.id+ )+ )+ await session.flush()++ counts = await _counts(session)+ was_ever_active = await room_had_human_activity(+ session, TENANT_ZERO, room.id+ )+ await session.commit()++ newly_active = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=4), now=NOW+ )++ assert counts.room_active_1d == 1+ assert counts.user_active_1d == 1+ assert was_ever_active is True+ assert len(newly_active) == 1+ assert newly_active[0].first_active_at == NOW - timedelta(minutes=90)+class TestTheDeploymentTotal:async def test_users_are_counted_once_for_the_deployment(diff --git a/core/tests/switch_core/telemetry/test_snapshot_multi_tenant.py b/core/tests/switch_core/telemetry/test_snapshot_multi_tenant.pyindex 58fe16a26..f0f61e475 100644--- a/core/tests/switch_core/telemetry/test_snapshot_multi_tenant.py+++ b/core/tests/switch_core/telemetry/test_snapshot_multi_tenant.py@@ -18,9 +18,13 @@ harness to reproduce it. That is the point: CI runs the vulnerable shape.from __future__ import annotations+import contextlib+import loggingimport uuid+from collections.abc import AsyncIteratorfrom datetime import UTC, datetime, timedelta+import pytestfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmakerfrom switch_core.db.models import (@@ -32,12 +36,34 @@ from switch_core.db.models import (Tenant,)from switch_core.db.session_scope import tenant_session+from switch_core.telemetry import snapshot as snapshot_modulefrom switch_core.telemetry.snapshot import collect_usage, newly_active_roomsNOW = datetime(2026, 9, 16, 12, 0, tzinfo=UTC)OTHER_TENANT = "11111111-1111-1111-1111-111111111111"+def _tenant_session_that_fails_for(bad_tenant_id: str):+ """A drop-in `tenant_session` that raises while binding one tenant.++ Stands in for the real failure modes named in the review — a statement+ timeout, a binding error — without needing to reproduce one against the+ test database: whatever the cause, it surfaces the same way, as an+ exception out of the `async with tenant_session(...)` block.+ """++ @contextlib.asynccontextmanager+ async def flaky(+ factory: async_sessionmaker[AsyncSession], tenant_id: str+ ) -> AsyncIterator[AsyncSession]:+ if tenant_id == bad_tenant_id:+ raise RuntimeError("simulated per-tenant failure")+ async with tenant_session(factory, tenant_id) as session:+ yield session++ return flaky++async def _make_tenant(session_factory: async_sessionmaker[AsyncSession]) -> None:async with session_factory() as session:session.add(@@ -76,8 +102,12 @@ async def _room_with_a_conversation(await session.flush()session.add_all([- ClientRoom(client_id=human.id, room_id=room.id),- ClientRoom(client_id=agent.id, room_id=room.id),+ ClientRoom(+ client_id=human.id, room_id=room.id, joined_at=room.created_at+ ),+ ClientRoom(+ client_id=agent.id, room_id=room.id, joined_at=room.created_at+ ),])session.add(@@ -154,3 +184,125 @@ class TestRoomActivationIsNotReportedPerTenant:)assert len(found) == 1+++class TestOneTenantsFailureIsContained:+ """The regression test for the second defect a review found: nothing+ guarded the per-tenant fan-out, so one tenant raising — a statement+ timeout, a binding failure — took the whole pass down with it. A+ deterministically-failing tenant meant the watermark never advanced and no+ snapshot was ever sent again; see `TestTheSnapshotSchedule` in+ `test_deployment_and_reporter.py` for that half."""++ async def test_the_other_tenants_counts_still_arrive(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ ) -> None:+ await _make_tenant(session_factory)+ await _room_with_a_conversation(+ session_factory, TENANT_ZERO_ID, first_at=NOW - timedelta(hours=1)+ )+ await _room_with_a_conversation(+ session_factory, OTHER_TENANT, first_at=NOW - timedelta(hours=1)+ )+ monkeypatch.setattr(+ snapshot_module,+ "tenant_session",+ _tenant_session_that_fails_for(OTHER_TENANT),+ )++ counts = await collect_usage(session_factory, now=NOW)++ # The healthy tenant's room and message are still in the total.+ assert counts.room_count == 1+ assert counts.room_active_1d == 1+ assert counts.message_count_1d == 1++ async def test_a_partial_pass_is_disclosed_not_folded_into_the_full_count(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """Before this fix, `tenant_count` was set to the deployment's real+ tenant count up front — so a partial pass reported the full tenant+ count sitting on top of an undercounted everything else, with nothing+ on the wire to tell the two apart. It must now report only what was+ actually counted, and the failure must also reach the operator's log."""+ await _make_tenant(session_factory)+ await _room_with_a_conversation(+ session_factory, TENANT_ZERO_ID, first_at=NOW - timedelta(hours=1)+ )+ monkeypatch.setattr(+ snapshot_module,+ "tenant_session",+ _tenant_session_that_fails_for(OTHER_TENANT),+ )++ with caplog.at_level(logging.ERROR, logger="switch_core.telemetry.snapshot"):+ counts = await collect_usage(session_factory, now=NOW)++ assert counts.tenant_count == 1+ # And the pass says how many it lost, rather than leaving a reader to+ # work it out from a tenant count the event does not otherwise carry.+ assert counts.tenant_failed_count == 1+ assert any(OTHER_TENANT in record.message for record in caplog.records)++ async def test_a_complete_pass_reports_nothing_failed(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ ) -> None:+ """Zero rather than an absent property: every `usage_snapshot` carries+ the same keys, so a gap in the data is a send that went wrong rather+ than a pass nobody thought about."""+ await _make_tenant(session_factory)++ counts = await collect_usage(session_factory, now=NOW)++ assert counts.tenant_failed_count == 0++ async def test_the_pass_reports_how_long_it_took(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ ) -> None:+ """Roughly fifteen queries per tenant against the database that is also+ serving rooms. What that costs at scale is an open question in+ `docs/old/telemetry-events.md`, and a background task is invisible to+ the HTTP histogram that times everything else."""+ await _make_tenant(session_factory)++ counts = await collect_usage(session_factory, now=NOW)++ # A whole, non-negative number of milliseconds — the contract, not a+ # value a test can pin. Zero is legitimate for an empty deployment.+ assert isinstance(counts.duration_ms, int)+ assert counts.duration_ms >= 0+ assert "duration_ms" in counts.as_event_properties(session_live_count=0)++ async def test_newly_active_rooms_is_contained_the_same_way(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ await _make_tenant(session_factory)+ await _room_with_a_conversation(+ session_factory, TENANT_ZERO_ID, first_at=NOW - timedelta(hours=1)+ )+ await _room_with_a_conversation(+ session_factory, OTHER_TENANT, first_at=NOW - timedelta(hours=1)+ )+ monkeypatch.setattr(+ snapshot_module,+ "tenant_session",+ _tenant_session_that_fails_for(OTHER_TENANT),+ )++ with caplog.at_level(logging.ERROR, logger="switch_core.telemetry.snapshot"):+ found = await newly_active_rooms(+ session_factory, since=NOW - timedelta(hours=4), now=NOW+ )++ assert len(found) == 1+ assert any(OTHER_TENANT in record.message for record in caplog.records)diff --git a/core/tests/switch_core/test_logging_config.py b/core/tests/switch_core/test_logging_config.pyindex 0e2f89753..fd4b85962 100644--- a/core/tests/switch_core/test_logging_config.py+++ b/core/tests/switch_core/test_logging_config.py@@ -204,7 +204,23 @@ def test_a_third_party_logger_gets_the_same_fields(monkeypatch: pytest.MonkeyPatdef test_binding_an_unknown_field_is_an_error() -> None:with pytest.raises(ValueError, match="Unknown log context field"):- bind_log_context(room_id="room-1") # type: ignore[call-arg]+ bind_log_context(session_id="session-1") # type: ignore[call-arg]+++def test_a_room_can_be_bound_and_reaches_the_record() -> None:+ """A room is the unit almost every support question arrives in — "this room+ stopped relaying" — and it used to be rejected here, so the only way to+ follow one was to grep for an id that appeared in some messages' text and+ not others. It is a log field rather than a metric attribute on purpose:+ unbounded, and belonging to one tenant."""+ token = bind_log_context(room_id="room-1")+ try:+ assert current_log_context().room_id == "room-1"+ record = logging.LogRecord("t", logging.INFO, __file__, 1, "m", None, None)+ LogContextFilter(default_tenant_id="default").filter(record)+ assert record.room_id == "room-1" # type: ignore[attr-defined]+ finally:+ unbind_log_context(token)def test_unbind_restores_the_previous_context() -> None:diff --git a/deploy/observability/README.md b/deploy/observability/README.mdindex bf485f829..60dc1b961 100644--- a/deploy/observability/README.md+++ b/deploy/observability/README.md@@ -44,19 +44,26 @@ done## Check these before you trust the dashboard-**Histogram panels need two things, not one.** Request latency and delivery-lag are OTLP histograms, and the `p95:`/`p99:` queries here assume the-collector exports them to Datadog as *distributions*. Check the collector's+**Histogram panels need two things, not one.** Request latency, delivery lag,+database query duration and platform call duration are OTLP histograms, and the+`p95:`/`p99:` queries here assume the collector exports them to Datadog as+*distributions*. Check the collector'shistogram mode — in the older `histograms` mode they arrive as separate`.count`, `.sum`, `.min` and `.max` series and these panels stay empty.Then check the second thing, which is easy to miss because it is on Datadog'sside rather than the collector's: **percentile aggregations are off by defaulton a distribution metric and are billed separately.** Enable them per metric in-Metrics Summary, and add `route` to that metric's configured tag set, or-`p95: … by {route}` returns nothing on a fresh account. Empty is the honest-outcome either way; a panel is never silently switched to a different-statistic.+Metrics Summary, and add the tag each panel groups by to that metric's+configured tag set — `route` for requests, `operation` for database queries,+`platform` for bridge calls — or `p95: … by {…}` returns nothing on a fresh+account. Empty is the honest outcome either way; a panel is never silently+switched to a different statistic.++The two latency **monitors** depend on the same thing, and fail more quietly+than a panel does: an alert over a percentile that is not enabled evaluates+against no series and sits healthy for ever. Check them after enabling+percentiles, not before.**`env` is a filter, not a grouping.** The dashboard's `$env` variable defaultsto `*` and works whether or not anything sets it. The monitors deliberately do@@ -67,8 +74,8 @@ healthy for ever. If several deployments report into one Datadog org, set`ENVIRONMENT` on each and add `env:<name>` to the monitor scopes by hand.**`service` is the one tag everything depends on.** Every monitor filters on-`service:switch-core`. If a deployment changes `SERVICE_NAME`, all ten go-silent at once.+`service:switch-core`. If a deployment changes `SERVICE_NAME`, every one of+them goes silent at once.## The odd-looking first monitordiff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.jsonindex 54f52d0f4..5bd78d1b5 100644--- a/deploy/observability/dashboard.json+++ b/deploy/observability/dashboard.json@@ -91,7 +91,7 @@"title": "Bridges running","requests": [{- "q": "avg:switch.bridges.running{$env,$service}",+ "q": "sum:switch.bridges.running{$env,$service}","aggregator": "last"}],@@ -634,6 +634,67 @@"width": 6,"height": 4}+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Database query duration p95 / p99 (ms) by operation \u2014 needs distribution mapping, see README",+ "show_legend": true,+ "requests": [+ {+ "q": "p95:switch.db.query.duration{$env,$service} by {operation}",+ "display_type": "line"+ },+ {+ "q": "p99:switch.db.query.duration{$env,$service} by {operation}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 51,+ "width": 12,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Collaboration platform call duration p95 (ms) by platform \u2014 needs distribution mapping, see README",+ "show_legend": true,+ "requests": [+ {+ "q": "p95:switch.bridge.call.duration{$env,$service} by {platform}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 55,+ "width": 12,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Bridges running by platform",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.bridges.running{$env,$service} by {platform}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 59,+ "width": 12,+ "height": 4+ }}]}diff --git a/deploy/observability/monitors.json b/deploy/observability/monitors.jsonindex b8698a88c..5540d1308 100644--- a/deploy/observability/monitors.json+++ b/deploy/observability/monitors.json@@ -147,6 +147,44 @@"include_tags": true}},+ {+ "name": "[Switch] the database is slow",+ "type": "metric alert",+ "query": "avg(last_10m):p95:switch.db.query.duration{service:switch-core} > 500",+ "message": "The 95th percentile statement took more than half a second over ten minutes.\n\nDistinct from the pool alert beside it, and they fail in opposite directions: the pool one says this server is holding too many connections, this one says the database is taking too long to answer on the connections it has. A slow database exhausts the pool eventually, so this should fire first \u2014 which is the point of having it.\n\nBreak it down by operation on the dashboard. Statements that failed are not timed, so this is latency and never an error rate.\n\n@REPLACE-WITH-NOTIFICATION-HANDLE",+ "tags": [+ "service:switch-core",+ "team:switch"+ ],+ "options": {+ "notify_no_data": false,+ "renotify_interval": 60,+ "thresholds": {+ "critical": 500,+ "warning": 200+ },+ "include_tags": true+ }+ },+ {+ "name": "[Switch] the API is slow",+ "type": "metric alert",+ "query": "avg(last_10m):p95:switch.http.request.duration{service:switch-core} > 2000",+ "message": "The 95th percentile request took more than two seconds over ten minutes.\n\nThe one alert for the half of \"is it healthy\" that error rates cannot see: an endpoint answering 200 slowly is, from a user's side, an endpoint that is broken, and nothing else here fires on it.\n\np95 rather than an average, which one very slow request drags and a thousand fast ones hide. Across all routes rather than by route, because a per-route alert is thirteen alerts that each need their own threshold; break it down on the dashboard once this fires.\n\nThe agent event streams are excluded at the source \u2014 they are long polls held open for a duration the caller chose \u2014 so this is the routes where the number means something. Check the database query duration panel first: on this server a slow request is usually a slow query.\n\n@REPLACE-WITH-NOTIFICATION-HANDLE",+ "tags": [+ "service:switch-core",+ "team:switch"+ ],+ "options": {+ "notify_no_data": false,+ "renotify_interval": 60,+ "thresholds": {+ "critical": 2000,+ "warning": 1000+ },+ "include_tags": true+ }+ },{"name": "[Switch] server errors are elevated","type": "metric alert",diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.mdindex 29a8ee2eb..f40849424 100644--- a/docs/TELEMETRY.md+++ b/docs/TELEMETRY.md@@ -127,7 +127,7 @@ id, no IP field, no Switch identity.| `agent_created` | `agent_type`: `codex` · `location`: `remote` · `outcome`: `failure` · `failure_reason`: `none` / `unauthenticated` / `name_conflict` / `credentials_conflict` / `already_configured` / `invalid_name` / `directory_unusable` / `not_configured` / `agent_not_on_server` / `error` · `entry_point`: `sidebar` || `agent_removed` | `agent_type` · `location` · `delete_in_switch`: `true` · `trigger`: `user` / `server_teardown` · `outcome` · `failure_reason`: `none` / `not_linked_to_switch` / `gateway_unauthorized` / `gateway_http` / `gateway_network` / `error` || `agent_reset` | `agent_type` · `outcome` · `failure_reason`: `none` / `agent_not_found` / `not_remote` / `connect` / `error` |-| `agent_cli_action` | `agent_type` · `target`: `local` / `remote` · `install_method`: `homebrew` / `npm` / `winget` / `powershell` / `apt` / `curl` / `pip` / `cargo` / `installer-macos` / `installer-windows` / `installer-linux` / `other` / `unspecified` · `action`: `install` / `update` / `uninstall` · `outcome` · `failure_reason`: `none` / `unknown_dependency` / `no_install_command` / `no_update_strategy` / `no_uninstall_strategy` / `no_uninstall_command` / `permission_denied` / `command_failed` / `pty_open_failed` / `not_detected_after_install` / `not_detected_after_update` / `still_present` / `error` |+| `agent_cli_action` | `agent_type` · `target`: `local` / `remote` · `install_method`: `homebrew` / `npm` / `winget` / `powershell` / `apt` / `curl` / `pip` / `cargo` / `installer-macos` / `installer-windows` / `installer-linux` / `other` / `unspecified` · `action`: `install` / `update` / `uninstall` · `outcome` · `failure_reason`: `none` / `unknown_dependency` / `no_install_command` / `no_update_strategy` / `no_uninstall_strategy` / `no_uninstall_command` / `permission_denied` / `command_failed` / `process_open_failed` / `not_detected_after_install` / `not_detected_after_update` / `still_present` / `error` · `duration_ms`: `8421` || `session_started` | `agent_type`: `claude` · `location`: `local` · `outcome`: `success` · `failure_reason`: `none` / `agent_not_found` / `already_exists` / `spawn_failed` · `entry_point`: `command_palette` · `start_source`: `user` / `auto` / `adopted` / `unknown` · `has_initial_prompt`: `true` (**a boolean — never the prompt**) · `connected_to_room`: `false` || `session_ended` | `agent_type` · `location` · `outcome`: `normal` / `failed` || `session_attached` | `agent_type` · `outcome` |@@ -142,9 +142,21 @@ match a known code.| Event | Fields, with example values ||---|---|-| `connector_installed` | `agent_type`: `claude` · `target`: `local` · `outcome`: `success` |-| `connector_updated` | `agent_type` · `target`: `remote` · `outcome` · `was_reinstall`: `false` |-| `connector_uninstalled` | `agent_type` · `target`: `local` · `outcome` |+| `connector_installed` | `agent_type`: `claude` · `target`: `local` · `outcome`: `success` · `failure_reason`: `none` / `unsupported` / `host_cli_missing` / `marketplace_failed` / `install_command_failed` / `update_command_failed` / `uninstall_command_failed` / `files_write_failed` / `files_remove_failed` / `files_unimplemented` / `error` · `duration_ms`: `8421` |+| `connector_updated` | `agent_type` · `target`: `remote` · `outcome` · `was_reinstall`: `false` · `trigger`: `user` / `catch_up` · `failure_reason` (same set) · `duration_ms` |+| `connector_uninstalled` | `agent_type` · `target`: `local` · `outcome` · `failure_reason` (same set) · `duration_ms` |++`trigger` on `connector_updated` separates the Update button from the+once-per-install sweep that brings already-installed connectors up to the+version a new build ships. Both go through the same code, and the sweep runs+unattended at launch — without the field its latencies and its failures would be+read as ones somebody sat through.++`duration_ms` is how long the operation took, in whole milliseconds, measured on+a monotonic clock around the operation itself. Like `agent_count` and+`result_count` it is a number rather than a value from a fixed set, so to be+explicit: it is an elapsed time and nothing else. It names no path, host, command or repository,+and at this resolution it does not distinguish one machine from another.**Servers and sign-in**diff --git a/docs/old/observability.md b/docs/old/observability.mdindex 1ddaa5626..85f5373e1 100644--- a/docs/old/observability.md+++ b/docs/old/observability.md@@ -144,8 +144,12 @@ drop every live session to find that out.## The three things it emits**Logs** go to the container's output, with `tenant_id`, `request_id`,-`agent_id` and `user_id` stamped on every line by a filter on the handler — so-records from libraries carry them too. Set `LOG_FORMAT=json` to get them as+`agent_id`, `room_id` and `user_id` stamped on every line by a filter on the+handler — so records from libraries carry them too. A room id is a log field+and deliberately not a metric attribute: it is unbounded and belongs to one+tenant, which rules it out of a dashboard label for the reasons under "Why+there is a catalogue" — and following one room through a failure is the single+most common thing anyone asks these logs for. Set `LOG_FORMAT=json` to get them asfields rather than inside the message text; the default is `text`, which is forreading in a terminal, so anywhere the logs are actually collected wants theJSON form. With `OTLP_LOGS_ENABLED` the same records are *also* posted to the@@ -167,6 +171,26 @@ server did. In a latency histogram that is worse than useless: it would makethose routes' percentiles meaningless and, sharing an axis, flatten everyother route to the floor. They are counted like everything else.+**What is timed, as against counted.** Four things: HTTP requests, message+delivery lag, database statements and outbound calls to a collaboration+platform. That set is chosen to answer "what is slow" without guessing — on+this server a slow request is nearly always a slow query or a slow platform,+and those are the two the process cannot see from a count alone. A database+statement is timed around the driver call by SQLAlchemy's cursor events+(`observability/query.py`), so what is measured is the round trip rather than+the Python either side of it; the statement text never becomes an attribute,+only its leading keyword mapped through a fixed table — with the per-transaction+tenant `set_config` given a label of its own, because it is a third of all+statements and always trivial, so counted as a `select` it drags the+percentiles toward bookkeeping. A statement that raised is not timed, because a+query that failed in four milliseconds is not evidence the database is fast.++It is not every statement the process runs. Alembic builds its own engine, the+message listener talks to asyncpg directly, `BEGIN`/`COMMIT` come from the+dialect, and an `executemany` is one measurement for the whole batch because it+is one round trip. Read the panel as "the application's queries", not "all+database work".+**Not every HTTP surface is counted.** `switch.http.*` comes from middleware onthe FastAPI app, which is the agent bridge, the MCP mount and the gatewaybeneath it. Two listeners sit outside it — the Teams bridge and the@@ -316,6 +340,14 @@ and the relay Switch reports to does not serve `/v1/traces` — a POST therereturns 404. Two things have to happen: the collector must accept the signal,and the server must produce spans.+Tracing is the one observability item not built, and it was always scoped as+optional. The order matters: spans built against a collector+that 404s cannot be turned on, cannot be verified, and would be reviewed+against nothing. The collector side is tracked with the infrastructure work+(an agent that accepts OTLP traces directly is one of the two ways it closes);+the server side is a day's work once there is somewhere to send them, and the+log records already carry the fields to correlate against.+There is deliberately **no** `OTLP_TRACES_ENABLED` setting in the meantime. Aflag a deployment can turn on and see no difference from is a configurationsurface that lies about what it controls; it arrives when there is somethingdiff --git a/docs/old/telemetry-events.md b/docs/old/telemetry-events.mdindex e6b1da82a..814599e61 100644--- a/docs/old/telemetry-events.md+++ b/docs/old/telemetry-events.md@@ -241,7 +241,9 @@ never invisible.| Property | Type | Notes ||---|---|---|-| `tenant_count` | number | tenants on the deployment |+| `tenant_count` | number | tenants this pass actually counted — see `tenant_failed_count` |+| `tenant_failed_count` | number | tenants whose queries raised and were stepped over. One tenant's failure no longer takes the whole pass down, so the pair is what makes a partial pass self-describing: without it, every count dropping at once is indistinguishable from a deployment losing its users |+| `duration_ms` | number | wall time to collect the pass, on a monotonic clock. Roughly fifteen queries per tenant against the database that is also serving rooms, and a background task is invisible to `switch.http.request.duration` — so this is the only place "what does the snapshot cost at scale" can be answered from || `user_count` | number | user accounts that exist || `user_active_1d` | number | distinct humans who interacted in 24h || `user_active_7d` | number | same over 7 days |@@ -394,7 +396,8 @@ minutes and rooms created in week six never do, that is a different problem froma slow average.**`room_archived`** — `bridge_platform`, `age_days` (number), `was_ever_active`-(boolean).+(tri-state: `true` | `false` | `unknown` — `unknown` when the activity count+itself could not be run, which is not evidence the room was never used).**`room_agents_added`** — `agent_count` (number), `added_by_kind`.@@ -413,18 +416,21 @@ and "deleted after a year" are opposite signals: the first is a mistake or anexperiment, the second a deliberate clean-up.**`room_deleted`** — `bridge_platform`, `channel_type`, `created_by_kind`,-`age_days`, `was_ever_active`, `agent_count`. The activity flag is read before-the delete, because the cascade takes the room's messages with it and-afterwards every room looks like it was never used.+`age_days`, `was_ever_active` (tri-state, see `room_archived` above),+`agent_count`. The activity flag is read before the delete, because the+cascade takes the room's messages with it and afterwards every room looks like+it was never used.**`agent_deleted`** — `known_agent_type`, `age_days`, `room_count`,`had_parent`.-**`connector_removed`** — `bridge_platform`, `age_days`, `was_ever_connected`,-`room_count`. `was_ever_connected` is the one that matters: a connector removed-having never connected is a failed setup, and one removed after months of-service is a decision. Reporting both as "removed" would hide the first, which-is the one worth acting on.+**`connector_removed`** — `bridge_platform`, `age_days`, `was_ever_connected`+(tri-state: `true` | `false` | `unknown`), `room_count`. `was_ever_connected` is+the one that matters: a connector removed having never connected is a failed+setup, and one removed after months of service is a decision. Reporting both+as "removed" would hide the first, which is the one worth acting on.+`unknown` when the durable record backing it could not be read — reporting+that as `false` would misfile a lookup failure as a failed setup.**`agent_registered`**@@ -452,7 +458,22 @@ The connection registry already records a reason on every close, which is wherethese values come from; the set is closed here so a new reason string added inthe code does not silently become a new Amplitude value.-**`bridge_connected`** — `bridge_platform`, `outcome`, `failure_reason`.+**`bridge_connected`** — `bridge_platform`, `outcome`, `failure_reason`,+`duration_ms`.++Fires for every attempt, including the ones that fail before the bridge's task+is ever scheduled — an unregistered adapter type, a stored config that no+longer validates, a port another bridge already holds. Those used to emit+nothing at all, so the connect success rate was computed over a denominator+that excluded the attempts that failed hardest.++`duration_ms` is how long the attempt took, in whole milliseconds on a+monotonic clock, spanning both halves of it: the synchronous setup and the task+that actually reaches the platform. Nothing else times this — a bridge comes up+on a background task rather than inside a request the server serves — and the+failure half is the more interesting one, because a timeout and a refusal carry+the same `failure_reason` and nothing alike in the time. `-1` if no reading was+taken, following the convention below.**`bridge_disconnected`** — `bridge_platform`, `reason`(`shutdown` \| `restart` \| `auth_failed` \| `network` \| `platform_error` \|@@ -473,7 +494,10 @@ rather than something the product wants to know.### Closed value sets`bridge_platform`: `slack` | `mattermost` | `discord` | `teams` | `telegram` |-`none`.+`none` | `unknown`. `none` is "no bridge" (an internal-only room); `unknown` is+"there is one and the lookup that would have named it failed" — the two are+kept apart because collapsing them would misreport a Slack-bridged room as+internal-only whenever that lookup has a transient error.`outcome`: `success` | `failure`. `failure_reason` is an enumerated code perevent, `none` on success — never an exception message.@@ -483,15 +507,24 @@ set, because one classifier feeds both. A value that classifier can produce andonly one of the two declares is an event that fails validation at the moment abridge drops — which is precisely the event worth not losing.-Where a duration cannot be known — a deployment with no install clock, a bridge-whose configuration timestamp is unreadable — the property carries `-1` rather-than `0`, so "we could not tell" is distinguishable from "it happened-instantly".--Any property whose value is not in its set is a bug. It should raise where the-event is built rather than be coerced, dropped, or sent as `unknown` — a-silently-widening value set is how an analytics catalogue stops being-trustworthy.+**Tri-state facts**: `true` | `false` | `unknown`, for a yes/no property the+server can fail to establish (`room_archived`/`room_deleted`.`was_ever_active`,+`connector_removed`.`was_ever_connected`). Not a boolean, deliberately — a+property that can be unknown is a three-valued fact, and collapsing the failed+case into `false` reports a guess as a claim the code has no evidence for.++Where a duration or an age cannot be known — a deployment with no install+clock, a bridge whose configuration timestamp is unreadable, a row whose+`created_at` could not be read — the property carries `-1` rather than `0`, so+"we could not tell" is distinguishable from "it happened instantly" or+"created just now".++Any property whose value is not in its declared set is a bug. It should raise+where the event is built rather than be coerced, dropped, or silently widened+to a value nobody added to the catalogue. This is a different statement from+the tri-states above: `unknown` is fine *as a value the catalogue itself+declares* for the handful of properties that need it — what must never happen+is a value reaching the wire that the catalogue does not know about.## Consent