Skip to content

an open-source alternative to the dots, bots and muses of the world, run an office of claude code/codex like agents on your laptop, sandboxes or anywhere, uses your existing subscriptions

16.0 KB
Raw Blame

v0.3.5 Feature Research — Voice Control · Settings · Markdown Preview · Git Visualization (+ QM survey)

Deep-research pass run 2026-08-06 against the v0.3.4 integration branch, by five parallel research agents (code-grounded + web-verified). No code changes were made. File:line references are against integrate/v0.3.4.


0. Cross-cutting findings

  • yc-software/qm is real and relevant: YC's open-sourced (MIT, 2026-07-31, ~11.6k★) "multiplayer agent harness for work" — TypeScript/Fastify/Postgres server + Lit web UI, pluggable harnesses (Claude Code, Codex, Pi, OpenCode), durable sandboxes, crons/watches, scoped skills, three-tier security postures. It has NO voice surface and NO git/diff UI — Munder Difflin is ahead of it on both. Adopt patterns, not modules (its stack doesn't transplant into Electron/React).
  • QM adoption shortlist (tiered):
    1. Security postures: one Strict/Auto/Dangerous dial + a non-negotiable command deny-floor (no recursive deletes / destructive SQL) enforced in every posture → maps to a Settings "Autonomy" preset + a hive-level command filter.
    2. Crons + watches: condition-triggered agent runs (CI red, file changed, queue depth) alongside schedules; steal the cron humanizer UX.
    3. Skills manager UI: per-agent/project skills viewable/editable/promotable to all agents.
    4. Durability by default: queues/tasks/audit must survive restart (SQLite, not memory).
    5. Streaming-markdown pipeline: streaming render + dedupe + lazy hljs/KaTeX (feeds feature 3).
    6. Deep-linkable UI state · conversation forking (pairs with worktrees) · per-scope resource sidebar · single-source versioned JSONC config with a doctor validator (fixes our 3-file HarnessConfig mirror pain) · untrusted-ingress prompt-injection screening (Browser/Slack).
  • Security finding (fix regardless): config:update IPC (src/main/index.ts:2475) accepts an arbitrary unvalidated Partial<HarnessConfig>; config carries secrets (groqApiKey, slackBotToken, slackSigningSecret, webhookSecret) and dangerous keys (harnessHome). Any voice/settings tooling must go through a main-side per-key allowlist — and the raw handler deserves validation on its own.

1. Realtime Michael — deep context + full control (est. 6.5–8.5 days, 7 PRs)

Current state

  • gpt-realtime-2 over WebRTC (@openai/agents-realtime 0.11.8), ephemeral token minted main-side (src/main/realtime.ts:50-106). Persona ~950 tokens (renderer session.ts:78-106).
  • Context = ONE connect-time snapshot (~150-250 tokens: agent counts, doing/blocked counts, last 6 log kinds) — never refreshed; only completion pushes + greeting inject mid-session.
  • 11 read tools (fleet/tasks/board/cost/schedules/config-allowlist/memory/activity/messages, via IPC; hive:agentDirectory at index.ts:2885 is already the consolidated per-agent join).
  • 14 action verbs with a solid main-side safety spine (realtimeActions.ts): soft vs destructive tiers, single pending slot + 120s TTL, distinct-token confirm, god/mass-target hard denylist, michael-voice attribution + god cross-notify.

Gaps

  • Snapshot goes stale; no push on status/task/spawn changes; live store state (Agent.action, queue depth) unused; every factual answer costs a tool round trip.
  • No voice path for: resume (pause is one-way by voice — sharpest gap), auto-delivery, /clear context, restart/model change, settings, archive, delete task, create schedule, tool gating, queue read/clear.
  • /clear has NO IPC — it's a renderer-store queued message (store.ts:220,677) delivered by the idle-gated flush loop (useHive.ts) which zeroes the gauge on delivery.
  • restartWithModel is component-local in CommandCenterPanel — not callable elsewhere.

Realtime API facts (verified 2026-08, developers.openai.com)

  • GA: gpt-realtime-2.1 / 2.1-mini (128k context, 60-min sessions, reasoning-capable — use reasoning.effort: low for voice). App pins gpt-realtime-2 (still fine; consider bump).
  • session.update CAN swap instructions/tools mid-session (voice locks after first audio; instructions+tools ≤ 16,384 tokens) — but it busts the prompt cache. Caching is automatic exact-prefix (≥1,024 tokens); cached audio input is ~99% off ($0.40 vs $32 /1M). Therefore: stable prefix + append-only conversation items, never mutate instructions.
  • conversation.item.create injects messages mid-stream; SDK: sendMessage(msg, {}, {triggerResponse:false}) silent injection, updateAgent(), out-of-band response.create {conversation:"none"}. Parallel tool calls: NOT supported (sequential). No hard tool-count cap — the token budget is the limit; ~30 crisp tools is fine. Mask tool latency with spoken preambles; return compact flagged JSON, not prose.

Recommended architecture

  1. Snapshot at connect + append-only deltas (never touch instructions): compact floor table (one line per agent: name|status|action|ctx%|breaker|inbox; ~400-600 tokens) injected as the FIRST conversation item; a main-side FloorDeltaWatcher (sibling of RealtimeCompletionWatcher) pushes coalesced deltas (status flips, task moves, spawn/archive, breaker trips, ctx 70/90%) via the existing sanitized sendMessage path — debounced ≥10-15s, ≤150 tokens each. Add get_floor_state reading useStore.getState() (zero IPC) returning flagged JSON.
  2. Extend the verb spine, keep narrow tools (no free-text query_hive in v1):
    • Soft: resume_agent, set_auto_delivery, gate_tool, delete_task, get/clear_queue.
    • Destructive (echo-back + distinct token): clear_context (main confirms → renderer enqueueMessage('/clear') so ALL existing delivery gates apply), restart_agent (extract restartWithModel into a shared renderer service first), archive_agent, create_schedule, update_setting.
    • update_setting policy table (main-side): forbidden = harnessHome + all secret keys + providerBaseUrls + integrations; soft = notifications/themes/freeflow/idle-disconnect/ strongKeepalive; confirm = autoMode/defaultModel/god engine/worker caps/breaker/slack/ webhook/maxTurns. Validate type+range; echo old→new in the confirm.
  3. Latency budget: persona+30 tools 5k tokens cached (free); snapshot once; deltas ~1-2k/hr; most "what's happening" answers need ZERO round trips; detail = 1 round trip masked by a preamble line added to the persona.

PR sequence

1 snapshot+get_floor_state (0.5-1d) · 2 FloorDeltaWatcher (1-1.5d) · 3 soft verbs (1-1.5d) · 4 clear_context + queue tools (0.5-1d) · 5 update_setting policy table (1d) · 6 restart_agent service extraction (1.5-2d, highest risk) · 7 polish/model bump (0.5d).


2. Settings redesign (est. 3–4 days, 4 phases)

Problems found (full inventory in agent report; SettingsModal is 1,428 lines)

  • "General" is six unrelated domains; dead read-only rows (Auto mode / Semantic memory / Command are displayed but not editable — autoMode has NO editor anywhere post-onboarding).
  • Onboarding-trapped settings: strongKeepalive, audience, open-at-login — set once, unreachable forever.
  • defaultModel — the most consequential setting — has no UI at all; ditto maxTurns, 4 of 6 circuit-breaker fields (incl. hardStop, a safety decision).
  • Duplication: auto-compact toggle in General AND Schedules; semanticMemory shown in General, toggled in MemoryPanel; theme forks into localStorage + terminalTheme.
  • Secrets scattered across 5 entry UIs; OpenAI key in "AI Engines" silently gates Talk in "Integrations".
  • Default-drift bugs in config.ts: knowledgeGraph enabled:true contradicts its own "Default OFF" comments; SettingsModal seeds freeflowEnabled ?? false while default is true (UI shows OFF when ON); multiWindow code=true vs comment="OFF by default"; legacy costCapUsd still enforced — delete it.

Proposed IA — 6 tabs, simple-by-default + per-tab Advanced disclosure + header search

  1. General — home folder + reset row (Danger Zone absorbed), appearance (ONE theme control; office-theme picker absorbs tvShowOffices), notifications, keep-awake, open-at-login, explain-simply (audience), auto-update. Adv: multiWindow.
  2. Agents & Models — default agent model (NEW UI), Michael's engine (shared component with Command Center), BYOK keys with "what each key unlocks" note. Adv: per-CLI base URLs + default models, defaultCommand, maxTurns.
  3. Autonomy & Budgets (QM-style) — Ask-first/Autonomous preset (autoMode), full circuit breaker as one unit (give the 4 hidden fields UI), floor budget + velocity. Adv: worker fleet knobs. Delete costCapUsd.
  4. Connections — MCP tiers, Slack, Webhook, REST registry; setup guides become collapsible flows; ports under Advanced.
  5. Voice — Free Flow (toggle + Groq key) + Realtime Michael (key status w/ jump-link, devices, idle disconnect, cost HUD).
  6. Memory & Knowledge — semantic memory + embedding model, Knowledge Graph. Adv: reflection enabled+interval (leave 4 threshold fields config-only).

Phases: extract SettingRow/Section primitives + split the modal (0.5d) → new nav + move blocks + missing editors, all via existing updateConfig (1d) → Voice tab + collapsible setup + cleanup (1d) → search + QA (0.5-1d). Watch the 3-file HarnessConfig mirror on every field change; the 15-state rehydration useEffect is the main regression risk.


3. Markdown preview — IDE + terminal click (est. 3–5 days core, 3 PRs + polish)

Current state

  • IDE (IdePanel.tsx): tabs {key, rel, mode: 'edit'|'diff'}; languageForPath already maps markdown; Monaco fully self-hosted (CSP precedent). No way to open the IDE "at a file".
  • Reusable overlay exists: fullscreenFilePath store field → FullscreenFileEditor (z 280 — already above fullscreen terminal 250, below IDE 290), resolves ABSOLUTE paths incl. outside-workspace, renders CodeMirror (lang-markdown already wired).
  • No terminal link plumbing at all — no WebLinksAddon, no registerLinkProvider; xterm 5.5's ILinkProvider API is available; agents in store carry cwd+ptyId for relative resolution.
  • CSP blocks remote images (a feature for untrusted md).

Recommendation: react-markdown@10 + remark-gfm@4 (~45-55 kB min+gz)

Safe by construction for untrusted agent output — renders to React elements, no HTML sink, no rehype-raw, default urlTransform drops javascript:. QM uses marked+DOMPurify with a fused postprocess-sanitize hook (plugins/web-ui/src/markdown-sanitize.ts) — validated fallback pattern, but react-markdown fits a React app better (link interception, tokens theming, position-based scroll sync). Phase 2: rehype-highlight; mermaid only as a lazy bundled chunk with securityLevel:'strict'. Reject @uiw/react-markdown-preview (enables rehype-raw, forces GitHub look, 3.6 MB).

Workflows

  • Shared: markdown/MarkdownPreview.tsx — CTH-token styled (light+dark), every anchor intercepted: http(s)/mailto → app:openExternal (add scheme allowlist), relative .md → open-in-context, all else inert.
  • A. IDE: per-markdown-tab view: 'code'|'split'|'preview' (default split, persisted); three-segment toggle in the EditorBar (only for md); Cmd+Shift+V cycles; preview reads the live edit buffer via useDeferredValue — save flow untouched; md links open sibling tabs.
  • B. Terminal: custom ILinkProvider registered once in acquireTerminal() (covers all three terminal surfaces automatically); regex for absolute/~/relative *.md (+ tolerant of quotes/backticks/:line); resolve vs owning agent's cwd; verify existence via ONE new fs:stat IPC (mirror in preload + index.d.ts — 3-file gotcha); Cmd/Ctrl+click → setFullscreenFile(abs, {view:'preview'}) → FullscreenFileEditor gains an edit|preview toggle (Obsidian-style; preview default from terminal); "open in IDE" escalation via a new ideInitialFile store field consumed on IDE mount. FilesTab's narrow editor gets the same toggle.
  • Security: no HTML sink; anchors never navigate the window; terminal-derived paths flow through safeJoin + 2 MB cap + binary sniff, opened only on explicit modifier-click.

PRs: 1 MarkdownPreview + IDE split (1-2d) · 2 overlay + FilesTab toggles (0.5-1d) · 3 link provider + fs:stat + tests (1-2d) · 4 polish: scroll sync, highlight, mermaid (1-2d).


4. Git visualization (est. 6–8 days, 4 PRs)

Striking finding: ~40% is already built and UNMOUNTED

  • GitTab.tsx — complete branch/status/log panel with commit graph — orphaned (nothing imports it since the tab slim-down).
  • git/CommitGraph.tsx — themed wrapper over the commit-graph npm dep already in package.json — display-only today (onCommitClick/getDiff props unused).
  • git/graph.ts — hand-written DAG lane layout — also unused (documented fallback).
  • git.ts already returns full DAG topology (getLog emits hash+parents+refs) and contains the exact dirty/ahead safety logic a safe checkout needs (worktreeHasUnintegratedWork).
  • Missing entirely: per-commit diffs, branch compare, checkout, graph interactivity.

Library verdict

[email protected] (DoltHub; active, React 18, permissive, 323 kB) is best-in-class and takes exactly our data shape; keep graph.ts as the custom-SVG fallback if its popups fight the theme. @gitgraph/* is dead; vscode Git Graph's license PROHIBITS reuse (approach only); isomorphic-git unnecessary (we shell out); no diff library needed — Monaco DiffEditor does it. QM has no git UI to borrow.

Architecture

  • Data (PR1): getLogGraph (+--topo-order, --skip pagination, run at mainRepoRoot so all agent/ worktree branches appear) · getCommitFiles (diff-tree -z -M --name-status, handle R/C renames) · getFileAtRev (generalize show HEAD: to any rev; keep 2 MB cap + binary sniff; per-commit diff = sha^ vs sha into MonacoDiff) · compareRefs (three-dot default w/ two-dot toggle + left-right counts + merge-base) · listWorktrees · guarded checkout. Strict rev regex + -- separation (renderer-supplied refs). New IPC beside index.ts:2570; update renderer type mirrors.
  • UI (PR2/3): IDE left rail becomes CHANGES · HISTORY · COMPARE sub-tabs. HISTORY: mount CommitGraph with onCommitClick → file list (reuse CHANGES row styling) → Monaco diff tabs keyed commit::sha::path; infinite scroll past 200. COMPARE: base/head dropdowns + swap + ahead/behind counts + dot-mode toggle → file list → diff tabs. Checkout via context menu on branch chips/commit rows; detached-HEAD affordance exists in GitTab. Re-mount slim GitTab as a 4th sidebar tab (one-line mount) for at-a-glance status.
  • Checkout safety (main-process, non-negotiable): clean tree only (no auto-stash v1); owning agent must be idle — and gate on ANY agent whose cwd resolves to the same root (the god-at-mainRepoRoot sneaky case); branch-held-by-worktree errors mapped to "checked out by agent X"; fail-safe on any query error.
  • Risks: big repos (--all --topo-order walk; 8s runGit timeout is the backstop; --skip re-walks — fine to ~50k commits); do NOT add graph queries to the 4s polling loops (refresh on tab activation/explicit/status change); commit-graph popup styling may need the global.css override block extended.

PRs: 1 data layer (1-1.5d) · 2 HISTORY (2-3d) · 3 COMPARE + checkout UX (2d) · 4 polish + GitTab remount (1d).


5. Suggested build order (if all four are greenlit)

  1. Markdown PR1-2 (fast, visible, zero risk) → Git PR1 + mount HISTORY (the unmounted 40% makes this the best value-per-day in the codebase).
  2. Voice PR1-2 (context wins — biggest UX delta for talk mode) → Voice PR3-5 (control).
  3. Settings phases 1-3 (independent; can interleave; fixes real config bugs).
  4. Long tail: Git COMPARE/checkout, Voice restart_agent, markdown polish, Settings search.
  5. Cross-cutting from QM (separate track): autonomy postures + command deny-floor, watches, skills manager, config single-sourcing.

Grand total if everything ships: ~19–26 dev-days.