Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

feat(swarm): keep the Machines list fresh: report changes at once, pick starts up in 5 s, show reading age and an unreachable backend (#1528)

tony496s9313c44295ee
diff --git a/cli/src/superTabs/machines.spec.ts b/cli/src/superTabs/machines.spec.ts
index 0b518703a..bb8e9236f 100644
--- a/cli/src/superTabs/machines.spec.ts
+++ b/cli/src/superTabs/machines.spec.ts
@@ -19,7 +19,7 @@ function fixture() {
const start = vi.fn(async () => ({ ok: true }))
const prepare = vi.fn(async () => {})
const machine = new ContributedMachines(core, 'host', { backend, open, capabilities, start, prepare, now: () => now } as never)
- return { machine, pool, core, backend, open, close, capabilities, start, prepare, advance: () => { now += 16000 } }
+ return { machine, pool, core, backend, open, close, capabilities, start, prepare, advance: () => { now += 16000 }, step: (ms: number) => { now += ms } }
}
describe('contributed machine executor', () => {
@@ -83,4 +83,153 @@ describe('contributed machine executor', () => {
expect(reply).toMatchObject({ ok: true, queued: true, startId: 'start', machineId: 'host' })
expect(open).not.toHaveBeenCalled()
})
+
+ describe('how soon a change is seen', () => {
+ const posts = (f: ReturnType<typeof fixture>) => f.backend.mock.calls.filter(c => c[1] === 'POST' && String(c[2]).includes('/machines/'))
+ const reads = (f: ReturnType<typeof fixture>) => f.backend.mock.calls.filter(c => c[1] === 'GET' && String(c[2]).endsWith('/pools'))
+ const lastBody = (f: ReturnType<typeof fixture>) => posts(f).at(-1)![3] as { runningHarnessIds: string[]; stoppedHarnessIds: string[] }
+ /** The next host tick, `ms` later. */
+ const after = async (f: ReturnType<typeof fixture>, ms: number) => { f.step(ms); await f.machine.tick() }
+ const quiet = (f: ReturnType<typeof fixture>) => { f.pool.starts.length = 0 }
+
+ it('publishes at once when a harness starts or stops running', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ expect(posts(f)).toHaveLength(1)
+ expect(lastBody(f)).toMatchObject({ runningHarnessIds: ['running'], stoppedHarnessIds: ['stopped'] })
+
+ // A harness comes up on this machine: the next look says so, long before a beat has passed.
+ f.pool.harnessIds.push('newcomer')
+ vi.mocked(f.core.agents.byAgent).mockImplementation(id => ['running', 'newcomer'].includes(id) ? { agentId: id } as never : undefined)
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(2)
+ expect(lastBody(f).runningHarnessIds).toEqual(['running', 'newcomer'])
+
+ // And one stops.
+ vi.mocked(f.core.agents.stopped!).mockImplementation(async id => id === 'stopped' || id === 'running')
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(3)
+ expect(lastBody(f)).toMatchObject({ runningHarnessIds: ['newcomer'], stoppedHarnessIds: ['running', 'stopped'] })
+ })
+
+ it('publishes at once when new work is paused or allowed again', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ // Paused on the machine's own grant, then allowed; paused for the whole tab, then allowed.
+ for (const change of [() => { f.pool.machine.paused = true }, () => { f.pool.machine.paused = false },
+ () => { f.pool.paused = true }, () => { f.pool.paused = false }]) {
+ const before = posts(f).length
+ change()
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(before + 1)
+ }
+ })
+
+ it('publishes at once when a start is waiting, and picks it up in the same look', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ expect(f.start).not.toHaveBeenCalled()
+ f.pool.starts.push({ id: 'second', dispatched: false })
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(2)
+ expect(f.start).toHaveBeenCalledOnce()
+ expect((f.start.mock.calls as unknown[][])[0]![0]).toEqual({ tabId: 'tab', startId: 'second', hostDispatch: true })
+ // Settled: that too is a change, and is sent.
+ f.pool.starts.length = 0
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(3)
+ })
+
+ it('picks a waiting start up within five seconds, not fifteen', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ f.pool.starts.push({ id: 'late', dispatched: false })
+ await after(f, 4_000)
+ expect(f.start).not.toHaveBeenCalled()
+ await after(f, 1_000)
+ expect(f.start).toHaveBeenCalledOnce()
+ })
+
+ it('keeps the fifteen second heartbeat when nothing changes, and reads the machine only to send it', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ for (const ms of [5_000, 5_000]) await after(f, ms)
+ expect(reads(f)).toHaveLength(3)
+ expect(posts(f)).toHaveLength(1)
+ expect(f.capabilities).toHaveBeenCalledOnce()
+ expect(f.open).toHaveBeenCalledOnce()
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(2)
+ expect(f.capabilities).toHaveBeenCalledTimes(2)
+ // The beat counts from the last send.
+ await after(f, 10_000)
+ expect(posts(f)).toHaveLength(2)
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(3)
+ })
+
+ it('looks no more often than every five seconds', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ for (const ms of [1_000, 1_000, 1_000, 1_000]) await after(f, ms)
+ expect(reads(f)).toHaveLength(1)
+ await after(f, 1_000)
+ expect(reads(f)).toHaveLength(2)
+ })
+
+ it('never has more than one look in flight', async () => {
+ const f = fixture(); quiet(f)
+ let release!: (rows: unknown) => void
+ f.backend.mockImplementationOnce(() => new Promise(resolve => { release = resolve }))
+ const first = f.machine.tick()
+ // The host ticks every second whatever is in flight; none of those starts another look.
+ for (const ms of [1_000, 5_000, 10_000, 20_000]) { f.step(ms); void f.machine.tick() }
+ await Promise.resolve()
+ expect(reads(f)).toHaveLength(1)
+ release([f.pool])
+ await first
+ expect(reads(f)).toHaveLength(1)
+ expect(posts(f)).toHaveLength(1)
+ await after(f, 5_000)
+ expect(reads(f)).toHaveLength(2)
+ })
+
+ it('waits a whole beat after a look that failed, and after a send that failed', async () => {
+ const f = fixture(); quiet(f)
+ f.backend.mockRejectedValueOnce(new Error('backend unreachable'))
+ await expect(f.machine.tick()).rejects.toThrow('backend unreachable')
+ await after(f, 5_000); await after(f, 5_000)
+ expect(reads(f)).toHaveLength(1)
+ await after(f, 5_000)
+ expect(reads(f)).toHaveLength(2)
+
+ f.backend.mockImplementation(async (_core, method, path) => path.endsWith('/pools') ? [f.pool] : method === 'POST' ? Promise.reject(new Error('refused')) : { revision: 1 })
+ await after(f, 15_000)
+ expect(posts(f)).toHaveLength(2)
+ await after(f, 5_000); await after(f, 5_000)
+ expect(posts(f)).toHaveLength(2)
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(3)
+ })
+
+ it('does not reach the daemon in a look with nothing to send', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ f.open.mockClear(); f.close.mockClear()
+ await after(f, 5_000)
+ expect(f.open).not.toHaveBeenCalled()
+ expect(f.close).not.toHaveBeenCalled()
+ })
+
+ it('sends again once a contribution comes back after it was withdrawn', async () => {
+ const f = fixture(); quiet(f)
+ await f.machine.tick()
+ const pools = f.backend.getMockImplementation()!
+ f.backend.mockImplementation(async (core, method, path, body) => path.endsWith('/pools') ? [] : pools(core, method, path, body))
+ await after(f, 5_000)
+ f.backend.mockImplementation(pools)
+ await after(f, 5_000)
+ expect(posts(f)).toHaveLength(2)
+ })
+ })
})
diff --git a/cli/src/superTabs/machines.ts b/cli/src/superTabs/machines.ts
index e3766f074..d0eedcece 100644
--- a/cli/src/superTabs/machines.ts
+++ b/cli/src/superTabs/machines.ts
@@ -9,43 +9,74 @@ import { prepareSwarmProject } from './projectFolder.js'
const finite = (v: unknown) => typeof v === 'number' && Number.isFinite(v) ? v : null
const list = (v: unknown) => Array.isArray(v) ? v.map(object) : []
+/** How often the executor asks the backend what its tabs want of it (a start waiting, a pause) and compares what it
+ * would report with what it last did. Only this read is this frequent: the resource readers run when there is
+ * something to say. A start waits at most this long, plus the host's one-second tick, to be picked up. */
+export const MACHINE_POLL_MS = 5_000
+/** How long an observation stands before it is sent again unchanged, so the Desktop can tell a machine that is
+ * silent from one that has nothing new (its offline threshold is counted in these beats). */
+export const MACHINE_HEARTBEAT_MS = 15_000
+
/** Separate from chat delivery: slow resource readers and agent launches never
* hold up a delivery lease. Only a tab's explicit owner grant wakes this queue. */
export class ContributedMachines {
private stopped = false
private running: Promise<void> | null = null
private next = 0
+ /** A pass or one of its publishes failed: the next one waits a whole beat, as every pass once did. */
+ private failed = false
private readonly cursors = new Map<string, number>()
+ /** Per tab, when its observation was last sent and what it said that matters to a reader: the harnesses
+ * running and stopped, whether new work is paused, and which starts are waiting. A change in any of them is
+ * sent at once; otherwise the heartbeat keeps the reading fresh. */
+ private readonly published = new Map<string, { at: number; signature: string }>()
constructor(private readonly core: CoreApi, private readonly machineId: string,
private readonly deps: { backend: typeof backend; open: typeof openDaemon; capabilities: typeof readCapabilities;
start: typeof startAgent; now: () => number; prepare?: typeof prepareSwarmProject } =
{ backend, open: openDaemon, capabilities: readCapabilities, start: startAgent, now: Date.now }) {}
stop() { this.stopped = true }
+ /** One pass at a time, one [MACHINE_POLL_MS] after the last began (or a beat after one that failed). */
tick(): Promise<void> {
if (this.stopped || this.deps.now() < this.next) return Promise.resolve()
- return this.running ??= this.poll().finally(() => { this.running = null; this.next = this.deps.now() + 15_000 })
+ const began = this.deps.now()
+ return this.running ??= this.poll(began).catch(error => { this.failed = true; throw error }).finally(() => {
+ this.running = null
+ const done = this.deps.now()
+ this.next = this.failed ? done + MACHINE_HEARTBEAT_MS : Math.max(began + MACHINE_POLL_MS, done + 1_000)
+ this.failed = false
+ })
}
- private async poll() {
+ private async poll(began: number) {
const where = { dataDir: this.core.dataDir, port: this.core.daemon.port, machineId: this.machineId }
const pools = list(await this.deps.backend(this.core, 'GET', `/api/super-tabs/hosts/${this.machineId}/pools`)).filter(row => identifier(row.tabId) && object(row.machine).machineId === this.machineId && identifier(object(row.machine).generation))
- if (this.stopped || pools.length === 0) return
- const local = await this.deps.open(where)
+ if (this.stopped || pools.length === 0) { this.published.clear(); return }
+ for (const tab of this.published.keys()) if (!pools.some(row => row.tabId === tab)) this.published.delete(tab)
+ // The local daemon is reached only for a reading to send, never for a pass that has nothing to say.
+ let local: Awaited<ReturnType<typeof openDaemon>> | null = null
+ let usage: Record<string, unknown> | null = null
try {
- // The existing usage reader owns its provider cache. Raw responses and
- // account credentials never leave this machine.
- const usage = await local.request('usage_read', {}, 20_000).catch(() => ({}))
for (const row of pools) {
try {
if (this.stopped || !identifier(row.tabId)) return
const pool = object(row.machine)
if (!identifier(pool.generation) || pool.machineId !== this.machineId || typeof pool.cwd !== 'string') continue
+ const ids = Array.isArray(row.harnessIds) ? row.harnessIds.filter(identifier) : []
+ const stopped: string[] = []
+ for (const id of ids) if (await this.core.agents.stopped?.(id)) stopped.push(id)
+ const running = ids.filter(id => !!this.core.agents.byAgent(id) && !stopped.includes(id))
+ const pending = list(row.starts)
+ const signature = JSON.stringify([pool.generation, [...running].sort(), [...stopped].sort(), row.paused === true || pool.paused === true,
+ pending.map(start => start.id).sort()])
+ const last = this.published.get(String(row.tabId))
+ if (!last || last.signature !== signature || began - last.at >= MACHINE_HEARTBEAT_MS) {
+ local ??= await this.deps.open(where)
+ // The existing usage reader owns its provider cache. Raw responses and
+ // account credentials never leave this machine.
+ const readings = usage ??= await local.request('usage_read', {}, 20_000).catch(() => ({}))
const observed = await this.deps.capabilities({ cwd: pool.cwd, tools: ['git'], devices: [] })
if (this.stopped) return
const resources = object('resources' in observed ? observed.resources : null)
const gpus = list(resources.gpus).map(g => finite(g.utilizationPercent)).filter((n): n is number => n !== null)
- const ids = Array.isArray(row.harnessIds) ? row.harnessIds.filter(identifier) : []
- const stopped: string[] = []
- for (const id of ids) if (await this.core.agents.stopped?.(id)) stopped.push(id)
const observation = { generation: pool.generation, tools: 'tools' in observed ? observed.tools : {},
resources: { cpuPercent: finite(resources.cpuPercent), memoryUsedBytes: finite(resources.memoryUsedBytes), memoryTotalBytes: finite(resources.memoryTotalBytes),
diskFreeBytes: finite(resources.diskFreeBytes), diskTotalBytes: finite(resources.diskTotalBytes), projectDiskFreeBytes: finite(resources.projectDiskFreeBytes), projectDiskTotalBytes: finite(resources.projectDiskTotalBytes),
@@ -53,13 +84,17 @@ export class ContributedMachines {
chip: typeof resources.chip === 'string' ? resources.chip : null, os: typeof resources.os === 'string' ? resources.os : null,
gpuPercent: gpus.length ? Math.max(...gpus) : null },
quotas: (Array.isArray(pool.engines) ? pool.engines : []).filter((e): e is string => typeof e === 'string').map(engine => {
- const q = quotaOf(usage, engine, false)
+ const q = quotaOf(readings, engine, false)
return { engine, known: q.known, windows: q.windows }
- }), runningHarnessIds: ids.filter(id => !!this.core.agents.byAgent(id) && !stopped.includes(id)), stoppedHarnessIds: stopped }
- await this.deps.backend(this.core, 'POST', `/api/super-tabs/${row.tabId}/machines/${this.machineId}`, observation)
+ }), runningHarnessIds: running, stoppedHarnessIds: stopped }
+ try {
+ await this.deps.backend(this.core, 'POST', `/api/super-tabs/${row.tabId}/machines/${this.machineId}`, observation)
+ this.published.set(String(row.tabId), { at: began, signature })
+ } catch (error) { this.failed = true; throw error }
+ }
// Two launches per pass bounds expensive native work. Pending starts
// keep their original receipt across network failures and host restarts.
- const pending = list(row.starts), cursor = this.cursors.get(String(row.tabId)) ?? 0
+ const cursor = this.cursors.get(String(row.tabId)) ?? 0
this.cursors.set(String(row.tabId), cursor + 2)
const round = pending.length ? [...pending.slice(cursor % pending.length), ...pending.slice(0, cursor % pending.length)].slice(0, 2) : []
for (const start of round) {
@@ -87,6 +122,6 @@ export class ContributedMachines {
}
} catch { /* Revocation or one unavailable project cannot block another contribution. */ }
}
- } finally { local.close() }
+ } finally { local?.close() }
}
}
diff --git a/desktop/lib/super_tabs/controller.dart b/desktop/lib/super_tabs/controller.dart
index b1b889c53..a8f90e535 100644
--- a/desktop/lib/super_tabs/controller.dart
+++ b/desktop/lib/super_tabs/controller.dart
@@ -4,6 +4,7 @@ import 'dart:math';
import 'package:dio/dio.dart' show CancelToken, DioException, DioExceptionType;
import 'package:flutter/foundation.dart';
+import '../logging/app_log.dart';
import '../state/harness_attachments.dart';
import 'attachment_upload.dart';
import 'model.dart';
@@ -113,6 +114,19 @@ class SuperTabController extends ChangeNotifier {
bool _disposed = false;
Timer? _poll, _draftSave;
Future<void>? _refreshing;
+
+ /// The latest read of the Swarm's state failed (and the tab is still ours). While it is, what the
+ /// machines last reported cannot be told from what they report now: a machine is neither online nor
+ /// offline, and the Swarm says its backend is unreachable instead of calling machines offline.
+ bool get backendUnreachable => _refreshFailing && !accessRevoked;
+ bool _refreshFailing = false;
+
+ /// Since when the reads have been failing, while [backendUnreachable]: the first failure of this run of them.
+ DateTime? unreachableSince;
+
+ /// How many reads of the Swarm's state have been answered. A sidebar that is waiting for a new reading (back from
+ /// the background) knows it has one when this moves.
+ int stateReads = 0;
Future<void> _persisting = Future.value();
Map<String, dynamic>? _pending;
final _drafts = <String, String>{};
@@ -354,10 +368,20 @@ class SuperTabController extends ChangeNotifier {
Future<void> _read() async {
loading = snapshot == null;
_changed();
+ final took = Stopwatch()..start();
try {
final answer = await request('GET', _path);
if (_disposed || accessRevoked) return;
+ _refreshFailing = false;
+ unreachableSince = null;
+ stateReads++;
final nextRevision = (answer['revision'] as num).toInt();
+ // How long the Swarm took to answer, and nothing it said: the log shows a slow or failing
+ // backend, which the sidebar's ages and the machines' states otherwise only hint at.
+ appLog.debug(
+ 'swarm',
+ 'state read in ${took.elapsedMilliseconds} ms (revision $nextRevision)',
+ );
final next = SuperTabSnapshot.fromJson(superTabMap(answer['state']));
if (next.id != tabId) {
throw const FormatException('A different Swarm was returned.');
@@ -400,6 +424,16 @@ class SuperTabController extends ChangeNotifier {
await loadMessages(selectedId);
} catch (failure) {
_readFailure(failure);
+ if (!_disposed && !accessRevoked) {
+ _refreshFailing = true;
+ unreachableSince ??= DateTime.now();
+ }
+ // The kind and code of the failure, never its message: a refusal can quote what was sent.
+ appLog.warn(
+ 'swarm',
+ 'state read failed after ${took.elapsedMilliseconds} ms: '
+ '${failure is SuperTabFailure ? '${failure.code} (status ${failure.statusCode})' : failure.runtimeType}',
+ );
} finally {
loading = false;
_changed();
diff --git a/desktop/lib/super_tabs/machine_status.dart b/desktop/lib/super_tabs/machine_status.dart
new file mode 100644
index 000000000..73942e386
--- /dev/null
+++ b/desktop/lib/super_tabs/machine_status.dart
@@ -0,0 +1,352 @@
+import 'dart:math' as math;
+
+import 'package:flutter/material.dart';
+
+import '../shared/theme/app_theme.dart' as grid;
+import '../widgets/desktop_chrome.dart';
+
+/// A reading this old is late: the machine reports every 15 seconds
+/// (`MACHINE_HEARTBEAT_MS`, `cli/src/superTabs/machines.ts`), so a reading past a beat and a quarter has missed one.
+const kSuperTabMachineLateAfter = Duration(seconds: 20);
+
+/// How long a machine's reading may stand before the Swarm calls it offline: two missed beats, plus a margin for the
+/// Swarm to be read in turn. A machine reports at once when something changes, so a quiet one is still under this.
+const kSuperTabMachineOfflineAfter = Duration(seconds: 35);
+
+/// What the Machines list says of one machine, and draws as its dot.
+enum SuperTabMachineStatus {
+ /// Reported within [kSuperTabMachineLateAfter].
+ live,
+
+ /// Past that, not yet [kSuperTabMachineOfflineAfter].
+ late,
+
+ /// Silent for [kSuperTabMachineOfflineAfter] or more (or, for a computer read directly, not connected).
+ offline,
+
+ /// The Swarm cannot be read, so what a machine reports through it cannot be told from silence. Not offline.
+ unknown,
+
+ /// Back from the background: what is shown was read before, and the new reading has not arrived.
+ refreshing,
+}
+
+/// The status of a machine from how old its reading is.
+///
+/// [age]: null when it has no reading. [unreachable]: the Swarm cannot be read and the reading comes through it.
+/// [refreshing]: a new reading is awaited. [connected]: for a computer this Desktop reads itself, whether its
+/// connection is up; null for one whose reports come through the Swarm. A computer that is connected is never
+/// called offline for an old reading, only late: its connection says it is there.
+SuperTabMachineStatus superTabMachineStatus({
+ required Duration? age,
+ bool unreachable = false,
+ bool refreshing = false,
+ bool? connected,
+}) {
+ if (unreachable) return SuperTabMachineStatus.unknown;
+ if (refreshing) return SuperTabMachineStatus.refreshing;
+ if (connected == false || age == null) return SuperTabMachineStatus.offline;
+ if (age >= kSuperTabMachineOfflineAfter) {
+ return connected == true
+ ? SuperTabMachineStatus.late
+ : SuperTabMachineStatus.offline;
+ }
+ return age >= kSuperTabMachineLateAfter
+ ? SuperTabMachineStatus.late
+ : SuperTabMachineStatus.live;
+}
+
+/// How long ago, as the end of a machine's metrics line says it: "just now", "4s ago", "3 min ago", "2 h ago".
+String superTabAgo(Duration age) {
+ final seconds = age.inSeconds;
+ if (seconds < 1) return 'just now';
+ if (seconds < 60) return '${seconds}s ago';
+ if (age.inMinutes < 60) return '${age.inMinutes} min ago';
+ if (age.inHours < 24) return '${age.inHours} h ago';
+ return '${age.inDays} d ago';
+}
+
+/// The same, shortest: "6s", "1m", "2h", for the heading's "Reconnecting · 1m".
+String superTabAgoShort(Duration age) {
+ if (age.inSeconds < 60) return '${math.max(0, age.inSeconds)}s';
+ if (age.inMinutes < 60) return '${age.inMinutes}m';
+ if (age.inHours < 24) return '${age.inHours}h';
+ return '${age.inDays}d';
+}
+
+/// What hovering or focusing a machine's row says.
+String superTabMachineTip(SuperTabMachineStatus status, Duration? age) {
+ final ago = age == null ? null : superTabAgo(age);
+ return switch (status) {
+ SuperTabMachineStatus.refreshing => 'Refreshing…',
+ SuperTabMachineStatus.unknown =>
+ ago == null
+ ? "Can't reach Harness"
+ : "Can't reach Harness · data from $ago",
+ SuperTabMachineStatus.offline =>
+ ago == null ? 'Offline' : 'Offline · last seen $ago',
+ SuperTabMachineStatus.live ||
+ SuperTabMachineStatus.late => ago == null ? 'Updated' : 'Updated $ago',
+ };
+}
+
+/// The state in words, so that colour is never the only signal.
+String superTabMachineStateLabel(SuperTabMachineStatus status) =>
+ switch (status) {
+ SuperTabMachineStatus.live => 'Live',
+ SuperTabMachineStatus.late => 'Late',
+ SuperTabMachineStatus.offline => 'Offline',
+ SuperTabMachineStatus.unknown => 'Status unknown',
+ SuperTabMachineStatus.refreshing => 'Refreshing',
+ };
+
+/// The tabular figures the ages and the heading's pill count in.
+const _tabular = [FontFeature.tabularFigures()];
+
+TextStyle superTabAgeStyle(SuperTabMachineStatus status) {
+ final base = DesktopChrome.metadata().copyWith(fontFeatures: _tabular);
+ return switch (status) {
+ SuperTabMachineStatus.late => base.copyWith(
+ color: grid.AppPalette.warn,
+ fontWeight: FontWeight.w500,
+ ),
+ SuperTabMachineStatus.refreshing => base.copyWith(
+ fontStyle: FontStyle.italic,
+ ),
+ _ => base,
+ };
+}
+
+/// The 8 px mark before a machine's name: green when live (with one short ping as each new reading arrives),
+/// amber when late, a hollow ring when offline, a dashed ring when unknown, a breathing grey when refreshing.
+/// Motion stops when the platform asks for reduced motion.
+class SuperTabMachineDot extends StatefulWidget {
+ const SuperTabMachineDot({
+ super.key,
+ required this.status,
+ this.readingAt,
+ this.arrived = false,
+ });
+
+ final SuperTabMachineStatus status;
+
+ /// When the reading was taken; a newer one than before pings a live dot.
+ final DateTime? readingAt;
+
+ /// The reading is new as this is first built: the dot pings once.
+ final bool arrived;
+
+ @override
+ State<SuperTabMachineDot> createState() => _SuperTabMachineDotState();
+}
+
+class _SuperTabMachineDotState extends State<SuperTabMachineDot>
+ with TickerProviderStateMixin {
+ late final _ping = AnimationController(
+ vsync: this,
+ duration: const Duration(milliseconds: 1200),
+ );
+ late final _breath = AnimationController(
+ vsync: this,
+ duration: const Duration(seconds: 1),
+ );
+ bool _still = false;
+
+ @override
+ void initState() {
+ super.initState();
+ if (widget.status == SuperTabMachineStatus.live && widget.arrived) {
+ _ping.forward();
+ }
+ }
+
+ @override
+ void didChangeDependencies() {
+ super.didChangeDependencies();
+ _still = MediaQuery.maybeDisableAnimationsOf(context) ?? false;
+ _sync();
+ }
+
+ @override
+ void didUpdateWidget(SuperTabMachineDot old) {
+ super.didUpdateWidget(old);
+ if (widget.status == SuperTabMachineStatus.live &&
+ widget.readingAt != null &&
+ widget.readingAt != old.readingAt &&
+ !_still) {
+ _ping.forward(from: 0);
+ }
+ _sync();
+ }
+
+ void _sync() {
+ final breathing = widget.status == SuperTabMachineStatus.refreshing;
+ if (_still) {
+ _ping.stop();
+ _breath.stop();
+ } else if (breathing && !_breath.isAnimating) {
+ _breath.repeat();
+ } else if (!breathing && _breath.isAnimating) {
+ _breath.stop();
+ }
+ }
+
+ @override
+ void dispose() {
+ _ping.dispose();
+ _breath.dispose();
+ super.dispose();
+ }
+
+ @override
+ Widget build(BuildContext context) => ExcludeSemantics(
+ child: SizedBox.square(
+ dimension: 8,
+ child: AnimatedBuilder(
+ animation: Listenable.merge([_ping, _breath]),
+ builder: (context, _) => CustomPaint(
+ painter: _DotPainter(
+ status: widget.status,
+ ping: _still ? 0 : _ping.value,
+ breath: _still ? 0 : _breath.value,
+ online: grid.AppPalette.online,
+ warn: grid.AppPalette.warn,
+ quiet: grid.AppPalette.offline,
+ ),
+ ),
+ ),
+ ),
+ );
+}
+
+class _DotPainter extends CustomPainter {
+ _DotPainter({
+ required this.status,
+ required this.ping,
+ required this.breath,
+ required this.online,
+ required this.warn,
+ required this.quiet,
+ });
+
+ final SuperTabMachineStatus status;
+ final double ping, breath;
+ final Color online, warn, quiet;
+
+ @override
+ void paint(Canvas canvas, Size size) {
+ final center = size.center(Offset.zero);
+ final radius = size.width / 2;
+ switch (status) {
+ case SuperTabMachineStatus.live:
+ canvas.drawCircle(center, radius, Paint()..color = online);
+ // One short ping: a ring around the dot, from 0.6x to 1.6x of its 3 px halo, fading as it goes.
+ if (ping > 0 && ping < 1) {
+ canvas.drawCircle(
+ center,
+ (radius + 3) * (.6 + ping),
+ Paint()
+ ..style = PaintingStyle.stroke
+ ..strokeWidth = 1
+ ..color = online.withValues(alpha: .8 * (1 - ping)),
+ );
+ }
+ case SuperTabMachineStatus.late:
+ canvas.drawCircle(center, radius, Paint()..color = warn);
+ case SuperTabMachineStatus.offline:
+ canvas.drawCircle(
+ center,
+ radius - .75,
+ Paint()
+ ..style = PaintingStyle.stroke
+ ..strokeWidth = 1.5
+ ..color = quiet,
+ );
+ case SuperTabMachineStatus.unknown:
+ final paint = Paint()
+ ..style = PaintingStyle.stroke
+ ..strokeWidth = 1.5
+ ..color = quiet;
+ const dashes = 6;
+ final ring = Rect.fromCircle(center: center, radius: radius - .75);
+ for (var i = 0; i < dashes; i++) {
+ canvas.drawArc(
+ ring,
+ i * 2 * math.pi / dashes,
+ math.pi / dashes,
+ false,
+ paint,
+ );
+ }
+ case SuperTabMachineStatus.refreshing:
+ // The breath: full at rest, down to 35% half way round.
+ final dip = 1 - (breath < .5 ? breath : 1 - breath) * 2;
+ canvas.drawCircle(
+ center,
+ radius,
+ Paint()..color = quiet.withValues(alpha: .35 + .65 * dip),
+ );
+ }
+ }
+
+ @override
+ bool shouldRepaint(_DotPainter old) =>
+ old.status != status ||
+ old.ping != ping ||
+ old.breath != breath ||
+ old.online != online ||
+ old.warn != warn ||
+ old.quiet != quiet;
+}
+
+/// The pill beside the Machines heading: "Live", or "Reconnecting · 1m" while the Swarm cannot be read.
+class SuperTabMachinesPill extends StatelessWidget {
+ const SuperTabMachinesPill({super.key, this.reconnectingFor});
+
+ /// How long the Swarm has failed to be read; null while it reads.
+ final Duration? reconnectingFor;
+
+ @override
+ Widget build(BuildContext context) {
+ final failing = reconnectingFor != null;
+ final text = failing
+ ? 'Reconnecting · ${superTabAgoShort(reconnectingFor!)}'
+ : 'Live';
+ return Semantics(
+ label: failing
+ ? 'Machines: reconnecting, for ${superTabAgoShort(reconnectingFor!)}'
+ : 'Machines: live',
+ excludeSemantics: true,
+ child: DecoratedBox(
+ decoration: ShapeDecoration(
+ shape: StadiumBorder(side: BorderSide(color: DesktopChrome.rim)),
+ ),
+ child: Padding(
+ padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 2),
+ child: Row(
+ mainAxisSize: MainAxisSize.min,
+ children: [
+ DecoratedBox(
+ decoration: BoxDecoration(
+ shape: BoxShape.circle,
+ color: failing
+ ? grid.AppPalette.warn
+ : grid.AppPalette.online,
+ ),
+ child: const SizedBox.square(dimension: 6),
+ ),
+ const SizedBox(width: 6),
+ Text(
+ text,
+ style: DesktopChrome.metadata().copyWith(
+ fontSize: 11,
+ fontFeatures: _tabular,
+ ),
+ ),
+ ],
+ ),
+ ),
+ ),
+ );
+ }
+}
diff --git a/desktop/lib/super_tabs/machines.dart b/desktop/lib/super_tabs/machines.dart
index a2e1a042e..e0d6ad8d5 100644
--- a/desktop/lib/super_tabs/machines.dart
+++ b/desktop/lib/super_tabs/machines.dart
@@ -18,10 +18,13 @@ import '../widgets/workspace_subscription_usage.dart';
import 'controller.dart';
import 'dialog_scope.dart';
import 'identity.dart';
+import 'machine_status.dart';
import 'machine_details.dart';
import 'model.dart';
import 'section_heading.dart';
+export 'machine_status.dart';
+
class SuperTabMachines extends StatefulWidget {
const SuperTabMachines({
super.key,
@@ -29,11 +32,21 @@ class SuperTabMachines extends StatefulWidget {
required this.onAddAgent,
this.app,
this.embedded = false,
+ this.now = DateTime.now,
+ this.readLocal,
});
final SuperTabController controller;
final AppNotifier? app;
final bool embedded;
final ValueChanged<SuperTabConversation> onAddAgent;
+
+ /// The clock the readings' ages are counted against.
+ final DateTime Function() now;
+
+ /// Reads this computer's own view of a machine, in place of [AppNotifier.readMachineResources]. A test gives it
+ /// to have the readings taken (they are not, under `flutter test`, unless it does).
+ @visibleForTesting
+ final Future<MachineResources?> Function(String machineId)? readLocal;
@override
State<SuperTabMachines> createState() => _SuperTabMachinesState();
}
@@ -210,6 +223,18 @@ class _MachinePickerState extends State<_MachinePicker> {
class _SuperTabMachinesState extends State<SuperTabMachines> {
final _samples = <String, (MachineResources, DateTime)>{};
+
+ /// Machines whose sample in [_samples] was taken before the app last came back to the front: it is shown, dimmed,
+ /// as "refreshing…" until the read that follows replaces it.
+ final _awaiting = <String>{};
+
+ /// How many reads of the Swarm's state had been answered when the app came back to the front; the machines
+ /// reporting through it are refreshing until one more is. Null: the app has not been away.
+ int? _swarmReadsAtReturn;
+
+ /// Once a second, so that ages, states and the heading's count follow the clock between the Swarm's reads.
+ Timer? _tick;
+ final _tips = <String, GlobalKey<TooltipState>>{};
ModelsMenuController? _usage;
Timer? _timer;
bool _reading = false;
@@ -224,7 +249,10 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
@override
void initState() {
super.initState();
- if (!kUnderTest && widget.app != null) {
+ if (!kUnderTest) {
+ _tick = Timer.periodic(const Duration(seconds: 1), (_) => _changed());
+ }
+ if ((!kUnderTest || widget.readLocal != null) && widget.app != null) {
_updateUsage();
widget.app!.foreground.addListener(_foreground);
unawaited(_refresh());
@@ -252,9 +280,17 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
_updateUsage();
_revision++;
_timer?.cancel();
- _samples.clear();
- _changed();
- if (widget.app?.foreground.value == true) unawaited(_refresh());
+ if (widget.app?.foreground.value == true) {
+ // Back in front: what was read before stays on the screen, dimmed, as refreshing — it used to vanish and
+ // come back a second later. The Swarm is read again at once, and so are this computer's readings.
+ _awaiting.addAll(_samples.keys);
+ _swarmReadsAtReturn = c.stateReads;
+ unawaited(c.refresh());
+ _changed();
+ unawaited(_refresh());
+ } else {
+ _changed();
+ }
}
Future<void> _refresh() async {
@@ -266,18 +302,22 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
for (var i = 0; i < ids.length; i += 2) {
await Future.wait(
ids.skip(i).take(2).map((id) async {
- final value = await widget.app!.readMachineResources(id);
+ final value =
+ await (widget.readLocal ?? widget.app!.readMachineResources)(
+ id,
+ );
if (!mounted || revision != _revision || !_ids.contains(id)) return;
- if (value == null) {
- _samples.remove(id);
- } else {
- _samples[id] = (value, DateTime.now());
- }
+ // Read or not, the wait for it is over; with nothing read the last sample stands and ages.
+ _awaiting.remove(id);
+ if (value != null) _samples[id] = (value, widget.now());
+ // Each machine as its read arrives, not once all of them have been read.
+ _changed();
}),
);
if (!mounted || revision != _revision) return;
}
- _changed();
+ _samples.removeWhere((id, _) => !_ids.contains(id));
+ _awaiting.removeWhere((id) => !_samples.containsKey(id));
} finally {
_reading = false;
if (mounted && widget.app?.foreground.value == true) {
@@ -290,6 +330,7 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
void dispose() {
_revision++;
_timer?.cancel();
+ _tick?.cancel();
widget.app?.foreground.removeListener(_foreground);
_usage?.removeListener(_changed);
_usage?.dispose();
@@ -588,19 +629,43 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
final observation = superTabMap(
shared?['observation'] ?? const <String, dynamic>{},
);
+ final now = widget.now();
final received = observation['receivedAt'];
- final fresh =
- received is num &&
- DateTime.now().millisecondsSinceEpoch - received < 45000;
- final online = shared == null
- ? state?.connectionStatus == ConnectionStatus.connected &&
- state?.nodeOnline != false
- : fresh;
+ final observedAt = received is num
+ ? DateTime.fromMillisecondsSinceEpoch(received.toInt())
+ : null;
final sample = _samples[id];
+ final connected =
+ state?.connectionStatus == ConnectionStatus.connected &&
+ state?.nodeOnline != false;
+ // Where the reading comes from. A computer this Desktop reads itself is known by that reading — and so is
+ // this very computer when it contributes and its own sample is the newer, which is why it stays live while
+ // the Swarm cannot be read. Any other contributing machine is known by what it reports through the Swarm.
+ final direct =
+ shared == null ||
+ (state?.isLocalMachine == true &&
+ sample != null &&
+ (observedAt == null || sample.$2.isAfter(observedAt)));
+ final readingAt = direct ? sample?.$2 : observedAt;
+ final age = readingAt == null ? null : now.difference(readingAt);
+ final status = superTabMachineStatus(
+ age: age,
+ // The Swarm cannot be read: a machine's silence is not known to be its own, so it is not offline.
+ unreachable: !direct && c.backendUnreachable,
+ refreshing: direct
+ ? _awaiting.contains(id)
+ : _swarmReadsAtReturn != null && c.stateReads <= _swarmReadsAtReturn!,
+ connected: direct ? connected : null,
+ );
+ // Old numbers are kept, dimmed, for every state that is not live or late: never blanked.
+ final dim =
+ status == SuperTabMachineStatus.offline ||
+ status == SuperTabMachineStatus.unknown ||
+ status == SuperTabMachineStatus.refreshing;
final resource = superTabMap(
observation['resources'] ?? const <String, dynamic>{},
);
- final value = fresh
+ final observed = observedAt != null
? MachineResources.fromJson({
...resource,
if (resource['gpuPercent'] != null)
@@ -612,11 +677,8 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
},
],
})
- : online &&
- sample != null &&
- DateTime.now().difference(sample.$2) < const Duration(seconds: 45)
- ? sample.$1
: null;
+ final value = direct ? sample?.$1 : observed;
final name =
shared?['name'] as String? ??
state?.machine.displayName ??
@@ -635,7 +697,7 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
)
.toList() ??
const <Map<String, Object?>>[];
- if (accounts.isEmpty && fresh) {
+ if (accounts.isEmpty && observedAt != null) {
accounts = [
for (final raw in observation['quotas'] as List? ?? [])
// Routing reads quota for these two only; other engines drew "—".
@@ -643,13 +705,17 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
_quotaRow(superTabMap(raw), id),
];
}
- final usage = WorkspaceSubscriptionUsage.fromRows(online ? accounts : []);
+ final usage = WorkspaceSubscriptionUsage.fromRows(accounts);
final paused = shared?['paused'] == true;
- final status = !online
- ? 'Offline'
- : paused
- ? 'New work paused'
- : 'Online';
+ final tip = superTabMachineTip(status, age);
+ final details = switch (status) {
+ SuperTabMachineStatus.live ||
+ SuperTabMachineStatus.late => paused ? 'New work paused' : 'Online',
+ SuperTabMachineStatus.offline =>
+ paused ? 'Offline, new work paused' : 'Offline',
+ SuperTabMachineStatus.unknown => "Can't reach Harness",
+ SuperTabMachineStatus.refreshing => 'Refreshing…',
+ };
final diskFree = value?.projectDiskTotalBytes == null
? value?.diskFreeBytes
: value?.projectDiskFreeBytes;
@@ -672,14 +738,31 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
final visibleAccounts = usage.accounts
.where((a) => a.figure != '—')
.toList();
- return ListTile(
+ final tail = status == SuperTabMachineStatus.refreshing
+ ? 'refreshing…'
+ : age == null
+ ? null
+ : superTabAgo(age);
+ final tipKey = _tips.putIfAbsent(id, GlobalKey<TooltipState>.new);
+ final tile = ListTile(
key: ValueKey('super-machine-$id'),
dense: true,
minTileHeight: 48,
minVerticalPadding: 8,
contentPadding: const EdgeInsets.symmetric(horizontal: 16),
+ // Hover shows the row's tooltip; so does keyboard focus, which a Tooltip does not do by itself.
+ onFocusChange: (focused) {
+ if (focused &&
+ FocusManager.instance.highlightMode ==
+ FocusHighlightMode.traditional) {
+ tipKey.currentState?.ensureTooltipVisible();
+ } else if (!focused) {
+ Tooltip.dismissAllToolTips();
+ }
+ },
title: _MachineHeader(
key: ValueKey('super-machine-header-$id'),
+ id: id,
name: name,
owner: person,
agents: [
@@ -692,65 +775,97 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
: 'Engineer',
),
],
- online: online,
+ status: status,
+ readingAt: readingAt,
+ arrived: age != null && age < const Duration(milliseconds: 1200),
paused: paused,
),
- subtitle: metrics.isEmpty && visibleAccounts.isEmpty
+ subtitle: metrics.isEmpty && visibleAccounts.isEmpty && tail == null
? null
: Padding(
padding: const EdgeInsets.only(top: 6),
child: Column(
crossAxisAlignment: CrossAxisAlignment.stretch,
children: [
- if (metrics.isNotEmpty)
- Wrap(
- spacing: 8,
- runSpacing: 4,
+ if (metrics.isNotEmpty || tail != null)
+ Row(
+ crossAxisAlignment: CrossAxisAlignment.start,
children: [
- for (final metric in metrics)
- Tooltip(
- message: '${metric.$1} ${metric.$2} available',
- child: Text(
- '${metric.$1} ${metric.$2}',
- style: DesktopChrome.metadata(),
+ Expanded(
+ child: Opacity(
+ opacity: dim ? .55 : 1,
+ child: Wrap(
+ spacing: 8,
+ runSpacing: 4,
+ children: [
+ for (final metric in metrics)
+ Tooltip(
+ message:
+ '${metric.$1} ${metric.$2} available',
+ child: Text(
+ '${metric.$1} ${metric.$2}',
+ style: DesktopChrome.metadata().copyWith(
+ fontFeatures: const [
+ FontFeature.tabularFigures(),
+ ],
+ ),
+ ),
+ ),
+ ],
),
),
+ ),
+ // At the end of the metrics line: how old all of it is.
+ if (tail != null) ...[
+ const SizedBox(width: 8),
+ Text(
+ tail,
+ key: ValueKey('super-machine-age-$id'),
+ style: superTabAgeStyle(status),
+ ),
+ ],
],
),
if (metrics.isNotEmpty && visibleAccounts.isNotEmpty)
const SizedBox(height: 6),
if (visibleAccounts.isNotEmpty)
- Wrap(
- spacing: 16,
- runSpacing: 6,
- children: [
- for (final account in visibleAccounts)
- Tooltip(
- message: account.detail,
- child: Semantics(
- label:
- '${account.name} ${account.figure} remaining',
- excludeSemantics: true,
- child: Row(
- mainAxisSize: MainAxisSize.min,
- children: [
- EngineMark(engine: account.engine, size: 16),
- const SizedBox(width: 6),
- Text(
- account.figure,
- style: DesktopChrome.control(
- medium: true,
- color: account.ink(
- DesktopChrome.foreground,
- DesktopChrome.surface,
+ Opacity(
+ opacity: dim ? .55 : 1,
+ child: Wrap(
+ spacing: 16,
+ runSpacing: 6,
+ children: [
+ for (final account in visibleAccounts)
+ Tooltip(
+ message: account.detail,
+ child: Semantics(
+ label:
+ '${account.name} ${account.figure} remaining',
+ excludeSemantics: true,
+ child: Row(
+ mainAxisSize: MainAxisSize.min,
+ children: [
+ EngineMark(
+ engine: account.engine,
+ size: 16,
+ ),
+ const SizedBox(width: 6),
+ Text(
+ account.figure,
+ style: DesktopChrome.control(
+ medium: true,
+ color: account.ink(
+ DesktopChrome.foreground,
+ DesktopChrome.surface,
+ ),
),
),
- ),
- ],
+ ],
+ ),
),
),
- ),
- ],
+ ],
+ ),
),
],
),
@@ -765,11 +880,12 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
content: SuperTabMachineDetails(
name: name,
owner: person?.email,
- status: paused && !online ? 'Offline, new work paused' : status,
- resources: value,
+ status: details,
+ // The row keeps old numbers, dimmed; the details do not present them as capacity to use.
+ resources: dim ? null : value,
cwd: shared?['cwd'] as String?,
engines: allowed,
- accounts: online ? accounts : const [],
+ accounts: dim ? const [] : accounts,
),
actions: [
if (shared != null && contributor == c.accountId)
@@ -816,10 +932,23 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
),
),
);
+ // One label for the row, so that colour is never the only signal: the machine, its state in words, and
+ // what hovering it says.
+ return Semantics(
+ container: true,
+ label: '$name, ${superTabMachineStateLabel(status)}, $tip',
+ child: Tooltip(
+ key: tipKey,
+ message: tip,
+ excludeFromSemantics: true,
+ child: tile,
+ ),
+ );
}
@override
Widget build(BuildContext context) {
+ final failingSince = c.backendUnreachable ? c.unreachableSince : null;
final children = [
SuperTabSectionHeading(
'Machines',
@@ -827,6 +956,13 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
? () => unawaited(_contribute())
: null,
addTooltip: 'Add machine',
+ trailing: SuperTabMachinesPill(
+ reconnectingFor: c.backendUnreachable
+ ? (failingSince == null
+ ? Duration.zero
+ : widget.now().difference(failingSince))
+ : null,
+ ),
),
if (_ids.isEmpty)
Padding(
@@ -854,17 +990,24 @@ class _SuperTabMachinesState extends State<SuperTabMachines> {
class _MachineHeader extends StatelessWidget {
const _MachineHeader({
super.key,
+ required this.id,
required this.name,
required this.owner,
required this.agents,
- required this.online,
+ required this.status,
+ required this.readingAt,
+ required this.arrived,
required this.paused,
});
- final String name;
+ final String id, name;
final SuperTabPerson? owner;
final List<({String id, String name, String role})> agents;
- final bool online, paused;
+ final SuperTabMachineStatus status;
+
+ /// When the reading shown was taken, and whether that is just now (the dot pings once).
+ final DateTime? readingAt;
+ final bool arrived, paused;
Widget _avatar(String identity, String name, String label) => Tooltip(
message: label,
@@ -883,9 +1026,14 @@ class _MachineHeader extends StatelessWidget {
@override
Widget build(BuildContext context) => LayoutBuilder(
builder: (context, constraints) {
+ // Offline and unknown machines are named in the muted ink.
final style = DesktopChrome.control(
medium: true,
- color: online ? null : DesktopChrome.muted,
+ color:
+ status == SuperTabMachineStatus.offline ||
+ status == SuperTabMachineStatus.unknown
+ ? DesktopChrome.muted
+ : null,
);
double width(String text, TextStyle style) {
final painter = TextPainter(
@@ -905,9 +1053,10 @@ class _MachineHeader extends StatelessWidget {
).clamp(0.0, constraints.maxWidth * .55);
final room =
constraints.maxWidth -
+ 14 -
nameWidth -
(owner == null ? 0 : 32) -
- (!online || paused ? 24 : 0);
+ (paused ? 24 : 0);
var count = agents.length;
while (count > 0 &&
count * 28 +
@@ -923,6 +1072,18 @@ class _MachineHeader extends StatelessWidget {
}
return Row(
children: [
+ // The state, before the name: 8 px, and never colour alone (the row's label and tooltip say it too).
+ Padding(
+ padding: const EdgeInsets.only(right: 6),
+ child: KeyedSubtree(
+ key: ValueKey('super-machine-state-$id-${status.name}'),
+ child: SuperTabMachineDot(
+ status: status,
+ readingAt: readingAt,
+ arrived: arrived,
+ ),
+ ),
+ ),
Flexible(
child: Tooltip(
message: name,
@@ -963,17 +1124,11 @@ class _MachineHeader extends StatelessWidget {
),
),
],
- if (!online || paused) ...[
+ if (paused) ...[
const SizedBox(width: 8),
Tooltip(
- message: !online
- ? (paused ? 'Offline, new work paused' : 'Offline')
- : 'New work paused',
- child: Icon(
- !online ? AppIcons.monitorOff : AppIcons.pause,
- size: 16,
- color: DesktopChrome.muted,
- ),
+ message: 'New work paused',
+ child: Icon(AppIcons.pause, size: 16, color: DesktopChrome.muted),
),
],
],
diff --git a/desktop/lib/super_tabs/section_heading.dart b/desktop/lib/super_tabs/section_heading.dart
index c40f9b3c9..9dc870dfe 100644
--- a/desktop/lib/super_tabs/section_heading.dart
+++ b/desktop/lib/super_tabs/section_heading.dart
@@ -11,12 +11,16 @@ class SuperTabSectionHeading extends StatelessWidget {
super.key,
this.onAdd,
this.addTooltip,
+ this.trailing,
});
final String title;
final VoidCallback? onAdd;
final String? addTooltip;
+ /// Beside the title, before the add action: a status the heading carries, such as the Machines pill.
+ final Widget? trailing;
+
@override
Widget build(BuildContext context) => Padding(
padding: const EdgeInsets.fromLTRB(16, 0, 12, 4),
@@ -32,6 +36,7 @@ class SuperTabSectionHeading extends StatelessWidget {
),
),
),
+ if (trailing != null) ...[trailing!, const SizedBox(width: 4)],
if (onAdd != null)
AppIconButton(
icon: AppIcons.plus,
diff --git a/desktop/test/super_tab_machine_freshness_test.dart b/desktop/test/super_tab_machine_freshness_test.dart
new file mode 100644
index 000000000..a9dd44e45
--- /dev/null
+++ b/desktop/test/super_tab_machine_freshness_test.dart
@@ -0,0 +1,915 @@
+import 'dart:async';
+import 'dart:ui' show PointerDeviceKind;
+
+import 'package:flutter/material.dart';
+import 'package:flutter/services.dart';
+import 'package:flutter_test/flutter_test.dart';
+import 'package:harness/core/machine_resources.dart';
+import 'package:harness/core/models.dart';
+import 'package:harness/logging/app_log.dart';
+import 'package:harness/shared/theme/app_theme.dart' as grid;
+import 'package:harness/state/app_state.dart';
+import 'package:harness/super_tabs/controller.dart';
+import 'package:harness/super_tabs/machines.dart';
+import 'package:harness/widgets/desktop_chrome.dart';
+
+import 'super_tab_launch_test.dart' show SuperTabLaunchFixture, tabId;
+import 'support/super_tab_fixture.dart';
+
+/// The Machines list as the human approved it (https://claude.ai/artifact/3b4f1ZqbxwSDqL7QKq39S2): a status dot
+/// before each name, the age at the end of the metrics line, a tooltip and a label for the row, a pill by the
+/// heading, old numbers kept and dimmed — and the readings behind them: how old each is, what the Swarm not being
+/// readable does to them, and what this computer reads itself.
+void main() {
+ final base = DateTime.utc(2026, 10, 22, 12);
+ DateTime now = base;
+ setUp(() => now = base);
+
+ Map<String, dynamic> contributed(
+ String id,
+ Duration age, {
+ String name = 'Studio Mac',
+ DateTime? at,
+ }) => {
+ 'machineId': id,
+ 'name': name,
+ 'contributorId': 'owner',
+ 'observation': {
+ 'receivedAt': (at ?? base).subtract(age).millisecondsSinceEpoch,
+ 'resources': {
+ 'cpuPercent': 20,
+ 'memoryTotalBytes': 32000000000,
+ 'memoryUsedBytes': 16000000000,
+ 'diskTotalBytes': 1000000000000,
+ 'diskFreeBytes': 500000000000,
+ },
+ 'quotas': [
+ {
+ 'engine': 'codex',
+ 'known': true,
+ 'windows': [
+ {'name': '5-hour', 'usedPercent': 10},
+ ],
+ },
+ ],
+ },
+ };
+
+ Future<void> show(
+ WidgetTester tester,
+ SuperTabController controller, {
+ AppNotifier? app,
+ Future<MachineResources?> Function(String)? readLocal,
+ bool reduceMotion = false,
+ }) => tester.pumpWidget(
+ MaterialApp(
+ builder: (context, child) => MediaQuery(
+ data: MediaQuery.of(context).copyWith(disableAnimations: reduceMotion),
+ child: child!,
+ ),
+ home: Scaffold(
+ body: SizedBox(
+ width: 300,
+ height: 600,
+ child: SuperTabMachines(
+ controller: controller,
+ app: app,
+ now: () => now,
+ readLocal: readLocal,
+ onAddAgent: (_) {},
+ ),
+ ),
+ ),
+ ),
+ );
+
+ /// A read has finished: its continuation runs in the first pump and the frame it asked for is drawn by the second.
+ Future<void> landed(WidgetTester tester) async {
+ await tester.pump();
+ await tester.pump();
+ }
+
+ Finder rowOf(String id) => find.byKey(ValueKey('super-machine-$id'));
+ Finder inRow(String id, Finder finder) =>
+ find.descendant(of: rowOf(id), matching: finder);
+ SuperTabMachineStatus statusOf(WidgetTester tester, String id) => tester
+ .widget<SuperTabMachineDot>(inRow(id, find.byType(SuperTabMachineDot)))
+ .status;
+ String ageOf(WidgetTester tester, String id) =>
+ tester.widget<Text>(find.byKey(ValueKey('super-machine-age-$id'))).data!;
+ TextStyle ageStyleOf(WidgetTester tester, String id) =>
+ tester.widget<Text>(find.byKey(ValueKey('super-machine-age-$id'))).style!;
+ Color? nameInkOf(WidgetTester tester, String id, String name) =>
+ tester.widget<Text>(inRow(id, find.text(name))).style?.color;
+
+ /// What a screen reader is told of the row, first: the label, then the row's own words (owner, numbers, allowance).
+ String labelOf(WidgetTester tester, String id) =>
+ tester.getSemantics(rowOf(id)).label.split('\n').first;
+ double dimmest(WidgetTester tester, String id) => tester
+ .widgetList<Opacity>(inRow(id, find.byType(Opacity)))
+ .map((o) => o.opacity)
+ .fold(1.0, (a, b) => a < b ? a : b);
+ Finder pill(String text) => find.descendant(
+ of: find.byType(SuperTabMachinesPill),
+ matching: find.text(text),
+ );
+
+ group('the words and the rules', () {
+ test('how long ago, as the end of the metrics line says it', () {
+ String ago(Duration age) => superTabAgo(age);
+ expect(ago(Duration.zero), 'just now');
+ expect(ago(const Duration(milliseconds: 900)), 'just now');
+ expect(ago(const Duration(seconds: 1)), '1s ago');
+ expect(ago(const Duration(seconds: 4)), '4s ago');
+ expect(ago(const Duration(seconds: 28)), '28s ago');
+ expect(ago(const Duration(seconds: 59)), '59s ago');
+ expect(ago(const Duration(seconds: 60)), '1 min ago');
+ expect(ago(const Duration(minutes: 3)), '3 min ago');
+ expect(ago(const Duration(minutes: 59, seconds: 59)), '59 min ago');
+ expect(ago(const Duration(hours: 2)), '2 h ago');
+ expect(ago(const Duration(days: 3)), '3 d ago');
+ });
+
+ test('and shortest, for the heading pill', () {
+ expect(superTabAgoShort(const Duration(seconds: 6)), '6s');
+ expect(superTabAgoShort(const Duration(seconds: 70)), '1m');
+ expect(superTabAgoShort(const Duration(minutes: 59)), '59m');
+ expect(superTabAgoShort(const Duration(hours: 2)), '2h');
+ expect(superTabAgoShort(const Duration(days: 3)), '3d');
+ });
+
+ test(
+ 'live under 20 s, late to 35 s, then offline: two named thresholds',
+ () {
+ expect(kSuperTabMachineLateAfter, const Duration(seconds: 20));
+ expect(kSuperTabMachineOfflineAfter, const Duration(seconds: 35));
+ SuperTabMachineStatus of(int seconds) =>
+ superTabMachineStatus(age: Duration(seconds: seconds));
+ expect(of(0), SuperTabMachineStatus.live);
+ expect(of(19), SuperTabMachineStatus.live);
+ expect(of(20), SuperTabMachineStatus.late);
+ expect(of(34), SuperTabMachineStatus.late);
+ expect(of(35), SuperTabMachineStatus.offline);
+ expect(of(600), SuperTabMachineStatus.offline);
+ expect(superTabMachineStatus(age: null), SuperTabMachineStatus.offline);
+ },
+ );
+
+ test(
+ 'unknown is not offline, and refreshing waits for the new reading',
+ () {
+ const old = Duration(minutes: 3);
+ expect(
+ superTabMachineStatus(age: old, unreachable: true),
+ SuperTabMachineStatus.unknown,
+ );
+ expect(
+ superTabMachineStatus(age: null, unreachable: true),
+ SuperTabMachineStatus.unknown,
+ );
+ expect(
+ superTabMachineStatus(age: old, refreshing: true),
+ SuperTabMachineStatus.refreshing,
+ );
+ // The Swarm failing is the better news of the two: it answers what the refresh is waiting for.
+ expect(
+ superTabMachineStatus(age: old, refreshing: true, unreachable: true),
+ SuperTabMachineStatus.unknown,
+ );
+ },
+ );
+
+ test('a computer this Desktop is connected to is late, not offline, for an old reading', () {
+ const old = Duration(minutes: 3);
+ expect(
+ superTabMachineStatus(age: old, connected: true),
+ SuperTabMachineStatus.late,
+ );
+ expect(
+ superTabMachineStatus(age: Duration.zero, connected: false),
+ SuperTabMachineStatus.offline,
+ );
+ expect(
+ superTabMachineStatus(age: const Duration(seconds: 3), connected: true),
+ SuperTabMachineStatus.live,
+ );
+ });
+
+ test('hover and focus say each state in its own words', () {
+ String tip(SuperTabMachineStatus s, [Duration? age]) =>
+ superTabMachineTip(s, age);
+ expect(
+ tip(SuperTabMachineStatus.live, const Duration(seconds: 4)),
+ 'Updated 4s ago',
+ );
+ expect(
+ tip(SuperTabMachineStatus.late, const Duration(seconds: 28)),
+ 'Updated 28s ago',
+ );
+ expect(
+ tip(SuperTabMachineStatus.offline, const Duration(minutes: 3)),
+ 'Offline · last seen 3 min ago',
+ );
+ expect(
+ tip(SuperTabMachineStatus.unknown, const Duration(minutes: 1)),
+ "Can't reach Harness · data from 1 min ago",
+ );
+ expect(tip(SuperTabMachineStatus.refreshing), 'Refreshing…');
+ expect(tip(SuperTabMachineStatus.offline), 'Offline');
+ expect(
+ superTabMachineStateLabel(SuperTabMachineStatus.unknown),
+ 'Status unknown',
+ );
+ });
+ });
+
+ group('a machine that contributes', () {
+ testWidgets(
+ 'live: a green dot, "4s ago", and a tooltip and label that say so',
+ (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(seconds: 4)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ final semantics = tester.ensureSemantics();
+ await show(tester, c);
+
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.live);
+ expect(
+ find.byKey(const ValueKey('super-machine-state-m-live')),
+ findsOneWidget,
+ );
+ expect(ageOf(tester, 'm'), '4s ago');
+ expect(ageStyleOf(tester, 'm').color, isNot(grid.AppPalette.warn));
+ // The age is in the metrics line, at its end.
+ final age = tester.getRect(
+ find.byKey(const ValueKey('super-machine-age-m')),
+ );
+ final cpu = tester.getRect(inRow('m', find.text('CPU 80%')));
+ expect(age.left, greaterThan(cpu.right));
+ expect(age.right, tester.getRect(rowOf('m')).right - 16);
+ expect(
+ ageStyleOf(tester, 'm').fontFeatures,
+ contains(const FontFeature.tabularFigures()),
+ );
+ expect(find.byTooltip('Updated 4s ago'), findsOneWidget);
+ expect(labelOf(tester, 'm'), 'Studio Mac, Live, Updated 4s ago');
+ expect(dimmest(tester, 'm'), 1);
+ expect(
+ nameInkOf(tester, 'm', 'Studio Mac'),
+ isNot(DesktopChrome.muted),
+ );
+ expect(pill('Live'), findsOneWidget);
+
+ semantics.dispose();
+ await tester.pumpWidget(const SizedBox());
+ },
+ );
+
+ testWidgets('late: an amber dot and an amber "28s ago"', (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(seconds: 28)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ final semantics = tester.ensureSemantics();
+ await show(tester, c);
+
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.late);
+ expect(ageOf(tester, 'm'), '28s ago');
+ expect(ageStyleOf(tester, 'm').color, grid.AppPalette.warn);
+ expect(ageStyleOf(tester, 'm').fontWeight, FontWeight.w500);
+ expect(find.byTooltip('Updated 28s ago'), findsOneWidget);
+ expect(labelOf(tester, 'm'), 'Studio Mac, Late, Updated 28s ago');
+ expect(dimmest(tester, 'm'), 1);
+
+ semantics.dispose();
+ await tester.pumpWidget(const SizedBox());
+ });
+
+ testWidgets(
+ 'moves from live to late to offline at 20 s and at the named 35 s, not before',
+ (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [contributed('m', Duration.zero)];
+ final c = server.controller();
+ await c.refresh();
+ for (final (seconds, status) in [
+ (0, SuperTabMachineStatus.live),
+ (19, SuperTabMachineStatus.live),
+ (20, SuperTabMachineStatus.late),
+ (34, SuperTabMachineStatus.late),
+ (35, SuperTabMachineStatus.offline),
+ (44, SuperTabMachineStatus.offline),
+ ]) {
+ now = base.add(Duration(seconds: seconds));
+ await show(tester, c);
+ expect(statusOf(tester, 'm'), status, reason: '$seconds s');
+ }
+ await tester.pumpWidget(const SizedBox());
+ },
+ );
+
+ testWidgets(
+ 'offline: a hollow ring, a muted name, "3 min ago", and the old numbers kept, dimmed',
+ (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(minutes: 3)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ final semantics = tester.ensureSemantics();
+ await show(tester, c);
+
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.offline);
+ expect(ageOf(tester, 'm'), '3 min ago');
+ expect(nameInkOf(tester, 'm', 'Studio Mac'), DesktopChrome.muted);
+ expect(find.byTooltip('Offline · last seen 3 min ago'), findsOneWidget);
+ expect(
+ labelOf(tester, 'm'),
+ 'Studio Mac, Offline, Offline · last seen 3 min ago',
+ );
+ // Never blanked: the numbers and the allowance are there, at 55%, the age is not.
+ expect(inRow('m', find.text('CPU 80%')), findsOneWidget);
+ expect(inRow('m', find.text('RAM 16 GB')), findsOneWidget);
+ expect(dimmest(tester, 'm'), .55);
+ expect(
+ find.ancestor(
+ of: find.byKey(const ValueKey('super-machine-age-m')),
+ matching: find.byType(Opacity),
+ ),
+ findsNothing,
+ );
+ expect(pill('Live'), findsOneWidget);
+
+ semantics.dispose();
+ await tester.pumpWidget(const SizedBox());
+ },
+ );
+
+ testWidgets(
+ 'unknown while the Swarm cannot be read: not offline, its data from a minute ago kept, dimmed',
+ (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(minutes: 1)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ server.failReads = true;
+ await c.refresh();
+ expect(c.backendUnreachable, isTrue);
+ final semantics = tester.ensureSemantics();
+ await show(tester, c);
+
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.unknown);
+ expect(
+ find.byKey(const ValueKey('super-machine-state-m-unknown')),
+ findsOneWidget,
+ );
+ expect(ageOf(tester, 'm'), '1 min ago');
+ expect(nameInkOf(tester, 'm', 'Studio Mac'), DesktopChrome.muted);
+ expect(
+ find.byTooltip("Can't reach Harness · data from 1 min ago"),
+ findsOneWidget,
+ );
+ expect(find.byTooltip('Offline · last seen 1 min ago'), findsNothing);
+ expect(
+ labelOf(tester, 'm'),
+ "Studio Mac, Status unknown, Can't reach Harness · data from 1 min ago",
+ );
+ expect(inRow('m', find.text('CPU 80%')), findsOneWidget);
+ expect(dimmest(tester, 'm'), .55);
+
+ // Read again, what the machine reports decides again: this one was silent.
+ server.failReads = false;
+ await c.refresh();
+ await show(tester, c);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.offline);
+
+ semantics.dispose();
+ await tester.pumpWidget(const SizedBox());
+ },
+ );
+
+ testWidgets(
+ 'a machine that was fresh is unknown, not offline, however long the Swarm stays unreadable',
+ (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(seconds: 3)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ server.failReads = true;
+ await c.refresh();
+ now = base.add(const Duration(minutes: 10));
+ await show(tester, c);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.unknown);
+ expect(ageOf(tester, 'm'), '10 min ago');
+ await tester.pumpWidget(const SizedBox());
+ },
+ );
+
+ testWidgets(
+ 'refreshing back from the background: grey, "refreshing…", the old numbers dimmed — until the Swarm is read',
+ (tester) async {
+ final f = SuperTabLaunchFixture();
+ f.app.bindSuperTab(f.app.activeSwarmId, tabId, 'Pilot');
+ f.connection.server.state['machines'] = [
+ contributed('far', const Duration(seconds: 10), name: 'Far Mac'),
+ ];
+ Completer<void>? gate;
+ final c = SuperTabController(
+ tabId: tabId,
+ accountId: 'owner',
+ request: (method, path, {body}) async {
+ await gate?.future;
+ return f.connection.server.request(method, path, body: body);
+ },
+ save: f.connection.server.save,
+ pollInterval: Duration.zero,
+ );
+ await c.refresh();
+ final semantics = tester.ensureSemantics();
+ await show(tester, c, app: f.app, readLocal: (_) async => null);
+ expect(statusOf(tester, 'far'), SuperTabMachineStatus.live);
+
+ gate = Completer<void>();
+ f.app.foreground.value = false;
+ await tester.pump();
+ f.app.foreground.value = true;
+ await tester.pump();
+ await show(tester, c, app: f.app, readLocal: (_) async => null);
+ expect(statusOf(tester, 'far'), SuperTabMachineStatus.refreshing);
+ expect(ageOf(tester, 'far'), 'refreshing…');
+ expect(ageStyleOf(tester, 'far').fontStyle, FontStyle.italic);
+ expect(find.byTooltip('Refreshing…'), findsOneWidget);
+ expect(labelOf(tester, 'far'), 'Far Mac, Refreshing, Refreshing…');
+ expect(inRow('far', find.text('CPU 80%')), findsOneWidget);
+ expect(dimmest(tester, 'far'), .55);
+
+ gate.complete();
+ await landed(tester);
+ await show(tester, c, app: f.app, readLocal: (_) async => null);
+ expect(statusOf(tester, 'far'), SuperTabMachineStatus.live);
+ expect(ageOf(tester, 'far'), '10s ago');
+
+ semantics.dispose();
+ await tester.pumpWidget(const SizedBox());
+ await c.flush();
+ c.dispose();
+ f.app.superTabs.dispose();
+ await tester.pump();
+ },
+ );
+ });
+
+ group('the heading pill', () {
+ testWidgets(
+ 'says Live while the Swarm is read, and Reconnecting with how long when it is not',
+ (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', Duration.zero, at: DateTime.now()),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ await show(tester, c);
+ expect(pill('Live'), findsOneWidget);
+ expect(find.byType(SuperTabMachinesPill), findsOneWidget);
+
+ server.failReads = true;
+ await c.refresh();
+ expect(c.unreachableSince, isNotNull);
+ now = c.unreachableSince!.add(const Duration(seconds: 70));
+ await show(tester, c);
+ expect(pill('Reconnecting · 1m'), findsOneWidget);
+ expect(pill('Live'), findsNothing);
+ now = c.unreachableSince!.add(const Duration(seconds: 6));
+ await show(tester, c);
+ expect(pill('Reconnecting · 6s'), findsOneWidget);
+
+ server.failReads = false;
+ await c.refresh();
+ expect(c.unreachableSince, isNull);
+ await show(tester, c);
+ expect(pill('Live'), findsOneWidget);
+ await tester.pumpWidget(const SizedBox());
+ },
+ );
+ });
+
+ group('how the row asks to be read', () {
+ testWidgets('hovering it shows its tooltip', (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [contributed('m', const Duration(seconds: 4))];
+ final c = server.controller();
+ await c.refresh();
+ await show(tester, c);
+ expect(find.text('Updated 4s ago'), findsNothing);
+ final mouse = await tester.createGesture(kind: PointerDeviceKind.mouse);
+ await mouse.addPointer(location: Offset.zero);
+ await mouse.moveTo(tester.getCenter(rowOf('m')));
+ await tester.pump(const Duration(milliseconds: 800));
+ expect(find.text('Updated 4s ago'), findsOneWidget);
+ await mouse.removePointer();
+ await tester.pumpWidget(const SizedBox());
+ });
+
+ testWidgets('so does keyboard focus on it', (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [contributed('m', const Duration(minutes: 3))];
+ final c = server.controller();
+ await c.refresh();
+ await show(tester, c);
+ expect(find.text('Offline · last seen 3 min ago'), findsNothing);
+ await tester.sendKeyEvent(LogicalKeyboardKey.tab);
+ await tester.pump();
+ await tester.pump(const Duration(milliseconds: 800));
+ expect(find.text('Offline · last seen 3 min ago'), findsOneWidget);
+ await tester.pumpWidget(const SizedBox());
+ });
+ });
+
+ group('the dot moves, unless the platform asks it not to', () {
+ testWidgets('a live dot pings once when a newer reading arrives', (
+ tester,
+ ) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(seconds: 10)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ await show(tester, c);
+ await tester.pump();
+ expect(tester.hasRunningAnimations, isFalse);
+
+ server.state['machines'] = [contributed('m', const Duration(seconds: 2))];
+ await c.refresh();
+ await show(tester, c);
+ expect(tester.hasRunningAnimations, isTrue);
+ await tester.pump(const Duration(milliseconds: 1300));
+ expect(tester.hasRunningAnimations, isFalse);
+ await tester.pumpWidget(const SizedBox());
+ });
+
+ testWidgets('and not with reduced motion', (tester) async {
+ final server = ConversationServer();
+ server.state['machines'] = [
+ contributed('m', const Duration(seconds: 10)),
+ ];
+ final c = server.controller();
+ await c.refresh();
+ await show(tester, c, reduceMotion: true);
+ server.state['machines'] = [contributed('m', const Duration(seconds: 2))];
+ await c.refresh();
+ await show(tester, c, reduceMotion: true);
+ expect(tester.hasRunningAnimations, isFalse);
+ await tester.pumpWidget(const SizedBox());
+ });
+
+ for (final reduceMotion in [false, true]) {
+ testWidgets(
+ 'a refreshing dot ${reduceMotion ? 'stays still with reduced motion' : 'breathes'}',
+ (tester) async {
+ final f = SuperTabLaunchFixture();
+ f.app.bindSuperTab(f.app.activeSwarmId, tabId, 'Pilot');
+ f.connection.server.state['agents'] = [
+ {
+ 'id': 'agent-m',
+ 'name': 'Builder',
+ 'machineId': 'm',
+ 'harnessId': 'native-m',
+ 'contributorId': 'owner',
+ 'configuration': {'engine': 'codex'},
+ },
+ ];
+ final c = await f.app.superTabs.open(tabId);
+ await c.refresh();
+ final reads = <Completer<MachineResources?>>[];
+ Future<MachineResources?> read(String id) {
+ final answer = Completer<MachineResources?>();
+ reads.add(answer);
+ return answer.future;
+ }
+
+ await show(
+ tester,
+ c,
+ app: f.app,
+ readLocal: read,
+ reduceMotion: reduceMotion,
+ );
+ reads.single.complete(const MachineResources(cpuPercent: 30));
+ await landed(tester);
+ f.app.foreground.value = false;
+ await tester.pump();
+ f.app.foreground.value = true;
+ await tester.pump();
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.refreshing);
+ await tester.pump(const Duration(milliseconds: 100));
+ expect(tester.hasRunningAnimations, !reduceMotion);
+
+ reads.last.complete(const MachineResources(cpuPercent: 30));
+ await landed(tester);
+ await tester.pumpWidget(const SizedBox());
+ await c.flush();
+ f.app.superTabs.dispose();
+ await tester.pump();
+ },
+ );
+ }
+ });
+
+ group('the computer this Desktop reads itself', () {
+ MachineResources cpu(double used) => MachineResources(
+ cpuPercent: used,
+ memoryTotalBytes: 100,
+ memoryUsedBytes: 50,
+ );
+ Map<String, dynamic> agentOn(String machineId) => {
+ 'id': 'agent-$machineId',
+ 'name': 'Builder',
+ 'machineId': machineId,
+ 'harnessId': 'native-$machineId',
+ 'contributorId': 'owner',
+ 'configuration': {'engine': 'codex'},
+ };
+
+ /// A Swarm with an agent on each of [machines], none of them contributed: this computer reads them.
+ Future<(SuperTabLaunchFixture, SuperTabController)> local(
+ List<String> machines,
+ ) async {
+ final f = SuperTabLaunchFixture();
+ f.app.bindSuperTab(f.app.activeSwarmId, tabId, 'Pilot');
+ for (final id in machines.where((id) => id != 'm')) {
+ final machine = Machine(
+ machineId: id,
+ authMode: MachineAuthMode.remote,
+ name: 'Second Mac',
+ );
+ f.app.machines = [...f.app.machines, machine];
+ f.app.machineStates[id] = MachineState(machine)
+ ..nodeOnline = true
+ ..connectionStatus = ConnectionStatus.connected
+ ..agentLoadStatus = AgentLoadStatus.loaded;
+ }
+ f.connection.server.state['agents'] = [
+ for (final id in machines) agentOn(id),
+ ];
+ final c = await f.app.superTabs.open(tabId);
+ await c.refresh();
+ return (f, c);
+ }
+
+ Future<void> done(
+ WidgetTester tester,
+ SuperTabLaunchFixture f,
+ SuperTabController c,
+ ) async {
+ await tester.pumpWidget(const SizedBox());
+ await c.flush();
+ f.app.superTabs.dispose();
+ await tester.pump();
+ }
+
+ testWidgets(
+ 'keeps its last sample on coming back, as refreshing and dimmed, until the new read lands',
+ (tester) async {
+ final (f, c) = await local(['m']);
+ final reads = <Completer<MachineResources?>>[];
+ Future<MachineResources?> read(String id) {
+ final answer = Completer<MachineResources?>();
+ reads.add(answer);
+ return answer.future;
+ }
+
+ await show(tester, c, app: f.app, readLocal: read);
+ reads.single.complete(cpu(30));
+ await landed(tester);
+ expect(inRow('m', find.text('CPU 70%')), findsOneWidget);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.live);
+ expect(ageOf(tester, 'm'), 'just now');
+
+ // Away for a while, then back: the numbers stay, dimmed, and the row says it is refreshing.
+ now = base.add(const Duration(seconds: 50));
+ f.app.foreground.value = false;
+ await tester.pump();
+ f.app.foreground.value = true;
+ await tester.pump();
+ expect(reads, hasLength(2));
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.refreshing);
+ expect(ageOf(tester, 'm'), 'refreshing…');
+ expect(inRow('m', find.text('CPU 70%')), findsOneWidget);
+ expect(dimmest(tester, 'm'), .55);
+
+ reads.last.complete(cpu(90));
+ await landed(tester);
+ expect(inRow('m', find.text('CPU 10%')), findsOneWidget);
+ expect(inRow('m', find.text('CPU 70%')), findsNothing);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.live);
+ expect(ageOf(tester, 'm'), 'just now');
+ expect(dimmest(tester, 'm'), 1);
+
+ await done(tester, f, c);
+ },
+ );
+
+ testWidgets('keeps its last sample, ageing, when a read finds nothing', (
+ tester,
+ ) async {
+ final (f, c) = await local(['m']);
+ final reads = <Completer<MachineResources?>>[];
+ Future<MachineResources?> read(String id) {
+ final answer = Completer<MachineResources?>();
+ reads.add(answer);
+ return answer.future;
+ }
+
+ await show(tester, c, app: f.app, readLocal: read);
+ reads.single.complete(cpu(30));
+ await landed(tester);
+ now = base.add(const Duration(seconds: 25));
+ f.app.foreground.value = false;
+ f.app.foreground.value = true;
+ await tester.pump();
+ reads.last.complete(null);
+ await landed(tester);
+ // Connected, so late rather than offline; the old number stays, and so does the age.
+ expect(inRow('m', find.text('CPU 70%')), findsOneWidget);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.late);
+ expect(ageOf(tester, 'm'), '25s ago');
+ now = base.add(const Duration(minutes: 5));
+ await show(tester, c, app: f.app, readLocal: read);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.late);
+
+ await done(tester, f, c);
+ });
+
+ testWidgets(
+ 'is offline, with its numbers dimmed, when its connection is down',
+ (tester) async {
+ final (f, c) = await local(['m']);
+ Future<MachineResources?> read(String id) async => cpu(30);
+ await show(tester, c, app: f.app, readLocal: read);
+ await landed(tester);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.live);
+
+ f.app.stateOf('m')!.connectionStatus = ConnectionStatus.disconnected;
+ await show(tester, c, app: f.app, readLocal: read);
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.offline);
+ expect(inRow('m', find.text('CPU 70%')), findsOneWidget);
+ expect(dimmest(tester, 'm'), .55);
+
+ await done(tester, f, c);
+ },
+ );
+
+ testWidgets('shows each machine as its sample arrives, not when all have', (
+ tester,
+ ) async {
+ final (f, c) = await local(['m', 'second']);
+ final reads = <String, Completer<MachineResources?>>{};
+ Future<MachineResources?> read(String id) =>
+ (reads[id] = Completer<MachineResources?>()).future;
+
+ await show(tester, c, app: f.app, readLocal: read);
+ expect(reads.keys, unorderedEquals(['m', 'second']));
+ reads['m']!.complete(cpu(30));
+ await landed(tester);
+ expect(inRow('m', find.text('CPU 70%')), findsOneWidget);
+ expect(inRow('second', find.textContaining('CPU')), findsNothing);
+
+ reads['second']!.complete(cpu(60));
+ await landed(tester);
+ expect(inRow('second', find.text('CPU 40%')), findsOneWidget);
+
+ await done(tester, f, c);
+ });
+
+ testWidgets(
+ 'stays live from its own samples while the Swarm cannot be read, when it contributes too',
+ (tester) async {
+ final f = SuperTabLaunchFixture();
+ f.app.bindSuperTab(f.app.activeSwarmId, tabId, 'Pilot');
+ f.connection.server.state['machines'] = [
+ contributed('m', const Duration(seconds: 80), name: 'This one'),
+ contributed('far', const Duration(seconds: 80), name: 'Far Mac'),
+ ];
+ final c = await f.app.superTabs.open(tabId);
+ await c.refresh();
+ Future<MachineResources?> read(String id) async => cpu(30);
+ f.connection.server.failReads = true;
+ await c.refresh();
+ expect(c.backendUnreachable, isTrue);
+
+ await show(tester, c, app: f.app, readLocal: read);
+ await landed(tester);
+ // This computer is read directly: live, and not dimmed. The other is only known through the Swarm.
+ expect(statusOf(tester, 'm'), SuperTabMachineStatus.live);
+ expect(ageOf(tester, 'm'), 'just now');
+ expect(inRow('m', find.text('CPU 70%')), findsOneWidget);
+ expect(dimmest(tester, 'm'), 1);
+ expect(statusOf(tester, 'far'), SuperTabMachineStatus.unknown);
+ expect(find.byType(SuperTabMachinesPill), findsOneWidget);
+ expect(pill('Live'), findsNothing);
+
+ await done(tester, f, c);
+ },
+ );
+ });
+
+ group('the log of each Swarm state read', () {
+ late _Log log;
+ setUp(() {
+ log = _Log();
+ final previous = appLog;
+ appLog = log;
+ addTearDown(() => appLog = previous);
+ });
+
+ test(
+ 'records how long a read took and its revision, nothing it said',
+ () async {
+ final server = ConversationServer();
+ server.state['name'] = 'PRIVATE_SWARM_NAME';
+ final c = server.controller();
+ await c.refresh();
+ final read = log.lines.where((l) => l.startsWith('debug swarm')).single;
+ expect(
+ read,
+ matches(
+ RegExp(r'^debug swarm state read in \d+ ms \(revision \d+\)$'),
+ ),
+ );
+ expect(log.lines.join('\n'), isNot(contains('PRIVATE')));
+ },
+ );
+
+ test('records a failure with how long it took and what kind, not what was said', () async {
+ final server = ConversationServer();
+ final c = server.controller();
+ await c.refresh();
+ log.lines.clear();
+ server.failReads = true;
+ await c.refresh();
+ await c.refresh();
+ final failures = log.lines
+ .where((l) => l.startsWith('warn swarm'))
+ .toList();
+ expect(failures, hasLength(2));
+ expect(
+ failures.first,
+ matches(
+ RegExp(r'^warn swarm state read failed after \d+ ms: StateError$'),
+ ),
+ );
+ expect(failures.join('\n'), isNot(contains('Offline')));
+ expect(log.lines.where((l) => l.startsWith('debug swarm')), isEmpty);
+ });
+
+ test('names the code and status of a refusal, never its message', () async {
+ final c = SuperTabController(
+ tabId: tabId,
+ accountId: 'owner',
+ save: (_) async {},
+ request: (method, path, {body}) async => throw const SuperTabFailure(
+ 'BACKEND_TIMEOUT',
+ 'PRIVATE refusal text',
+ statusCode: 504,
+ ),
+ );
+ addTearDown(c.dispose);
+ await c.refresh();
+ expect(
+ log.lines.single,
+ matches(
+ RegExp(
+ r'^warn swarm state read failed after \d+ ms: BACKEND_TIMEOUT \(status 504\)$',
+ ),
+ ),
+ );
+ expect(c.backendUnreachable, isTrue);
+ expect(c.stateReads, 0);
+ });
+ });
+}
+
+class _Log implements AppLog {
+ final lines = <String>[];
+
+ @override
+ void record(
+ AppLogLevel level,
+ String category,
+ String message, {
+ Object? error,
+ StackTrace? stackTrace,
+ }) => lines.add('${level.name} $category $message');
+}
diff --git a/desktop/test/super_tab_machines_test.dart b/desktop/test/super_tab_machines_test.dart
index 1e412176c..c7ff7989f 100644
--- a/desktop/test/super_tab_machines_test.dart
+++ b/desktop/test/super_tab_machines_test.dart
@@ -83,7 +83,7 @@ void main() {
},
);
testWidgets(
- 'machine rows show known capacity and allowances while stale readings stay hidden',
+ 'machine rows show known capacity and allowances, and an offline machine keeps its old readings dimmed',
(tester) async {
await tester.binding.setSurfaceSize(const Size(280, 700));
addTearDown(() => tester.binding.setSurfaceSize(null));
@@ -129,27 +129,29 @@ void main() {
);
await tester.pumpAndSettle();
final offline = find.byKey(const ValueKey('super-machine-offline'));
+ final online = find.byKey(const ValueKey('super-machine-online'));
+ // Offline: said by the row's tooltip, and its old numbers stay, dimmed, instead of being blanked.
+ expect(find.byTooltip('Offline · last seen 1 min ago'), findsOneWidget);
expect(
- find.descendant(of: offline, matching: find.byTooltip('Offline')),
+ find.descendant(of: offline, matching: find.textContaining('CPU')),
findsOneWidget,
);
expect(
find.descendant(of: offline, matching: find.byType(EngineMark)),
- findsNothing,
- );
- expect(
- find.descendant(of: offline, matching: find.textContaining('CPU')),
- findsNothing,
- );
- final online = find.byKey(const ValueKey('super-machine-online'));
- expect(
- tester.getSize(online).height,
- greaterThan(tester.getSize(offline).height),
+ findsOneWidget,
);
- expect(find.text('CPU 100%'), findsOneWidget);
- expect(find.text('RAM 0 GB'), findsOneWidget);
- expect(find.text('SSD 141 GB'), findsOneWidget);
- expect(find.byType(EngineMark), findsOneWidget);
+ double dimmest(Finder row) => tester
+ .widgetList<Opacity>(
+ find.descendant(of: row, matching: find.byType(Opacity)),
+ )
+ .map((o) => o.opacity)
+ .fold(1.0, (a, b) => a < b ? a : b);
+ expect(dimmest(offline), lessThan(1));
+ expect(dimmest(online), 1);
+ expect(find.text('CPU 100%'), findsNWidgets(2));
+ expect(find.text('RAM 0 GB'), findsNWidgets(2));
+ expect(find.text('SSD 141 GB'), findsNWidgets(2));
+ expect(find.byType(EngineMark), findsNWidgets(2));
// The entire row opens its details, including keyboard activation.
Focus.of(tester.element(find.text('Studio Mac'))).requestFocus();
await tester.pump();
diff --git a/docs/swarms.md b/docs/swarms.md
index b136c75f5..58ecf1e04 100644
--- a/docs/swarms.md
+++ b/docs/swarms.md
@@ -80,6 +80,40 @@ refusal names each refused field (`cwd`, `engine`, or any of `codexHome`,
folder this project uses there and the engines it contributes. It does not
repeat the values the start sent.
+A contributing machine reports its state (processor and memory use, allowances, which
+harnesses run or have stopped) to the Swarm. It looks at what its Swarms ask of it every
+5 seconds, and sends a report at once when something it says changes: a harness starts or
+stops, new work is paused or allowed again, or a start is waiting. A start waiting on a
+machine is therefore picked up within about 5 seconds. When nothing changes it still sends
+a report every 15 seconds, so that a silent machine can be told from a quiet one; a pass
+that fails waits a whole 15 seconds before the next.
+
+Each machine in the sidebar has a status dot before its name, and the age of its reading
+at the end of its metrics line ("4s ago"):
+
+| State | When | Dot | Age |
+| --- | --- | --- | --- |
+| Live | reading under 20 s old | green, with one short ping as each new reading arrives | `4s ago` |
+| Late | 20 s to 35 s | amber | `28s ago`, in amber |
+| Offline | 35 s or more (one named constant) | hollow grey ring; the name is muted | `3 min ago` |
+| Unknown | the Desktop cannot read the Swarm (contributing machines only) | dashed grey ring; the name is muted; not offline | `1 min ago` |
+| Refreshing | back from the background, no new reading yet | grey, breathing | `refreshing…` |
+
+Hovering or focusing a row says it in words ("Updated 4s ago", "Offline · last seen 3 min
+ago", "Can't reach Harness · data from 1 min ago", "Refreshing…"), and the row's screen-reader
+label is the machine's name, its state and that text, so colour is never the only signal.
+Old numbers are always kept, dimmed, for offline, unknown and refreshing machines, never
+blanked. The Machines heading carries a pill, "Live" or "Reconnecting · 1m" while the Swarm
+cannot be read. The pings and the breathing stop when the platform asks for reduced motion.
+A machine reports every 15 s, so 35 s is two missed reports plus a margin.
+
+This computer is read directly by the Desktop, every 15 seconds, so it stays live while the
+Swarm cannot be read, even when it also contributes (its own newer sample is used); a
+computer that is connected is late, not offline, if its readings stop. The readings stay on
+screen when the app returns to the foreground, dimmed and refreshing, until the new ones
+arrive. The Desktop log (`app-*.log`, category `swarm`) records how long each read of the
+Swarm's state took and when one failed, without what it said.
+
Ordinary guest machine sharing does not authorize Swarm controls under the
owner's account. The owner contributes the machine or an agent. The lead's
routing policy still determines eligible engines, capacity and model choices.
@@ -133,7 +167,8 @@ Use the matching backend, CLI and Desktop versions together for this feature:
`SuperTabSessionScope`, before serving the new project commands.
2. Update participating hosts to the CLI with Swarm-scoped contributions and
project checkout preparation. A host on an older CLI ignores the request for
- the default branch and starts Builders from its checkout's current commit.
+ the default branch and starts Builders from its checkout's current commit. A host older than
+ the immediate reports still reports, and picks up a waiting start, only every 15 seconds.
3. Install the Desktop build with the project sidebar and Cmd-S entry point.
A signed internal Desktop artifact alone does not update the backend or host