Skip to content

Self-improving AI Employees for everyone

feat(mobile): improve Android reliability, tooling, and full-view controls

trohitgea94eb2f8e73
diff --git a/.gitignore b/.gitignore
index 30be989f..4da5fd3c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -150,6 +150,9 @@ bun.lockb
# launch by ``services.example_loader``), so it stays tracked.
/.opencompany/*
!/.opencompany/workflows/
+# Mobile runtime created by server-relative DATA_DIR resolution in existing
+# installations: SDK, engine environment, AVD disks, snapshots, and downloads.
+/server/.opencompany/
# Pre-rebrand repo-local state remains readable for upgrades but is never
# tracked. Do not remove this rule: local databases and deploy state may live
# here in existing checkouts.
diff --git a/CLAUDE.md b/CLAUDE.md
index 820d77af..e390bd37 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -13,7 +13,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[UI Migration Plan](./docs-internal/ARCHIVE/ui_migration_plan.md)** | *Archived.* antd → shadcn/ui migration completion log (Waves 1–10). Current state lives in [Frontend Architecture](./docs-internal/frontend_architecture.md) and [Theme System](./docs-internal/theme_system.md). |
| **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/ARCHIVE/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | *Archived — all four waves shipped.* 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory), 16 per-queue task routing (`TemporalWorkerPool`), 17 resilience hardening (cron `catchup_window`, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats; the one-shot LLM-step retry it records was later reverted to unlimited-with-backoff), 18 worker performance tuning. Current settings live in [Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md). |
| **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |
-| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee card, the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Android remains a placeholder), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` broadcasts. |
+| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee card, the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` broadcasts. |
| **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.opencompany-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (toggled by Dashboard's `visibilitychange` listener; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 inline SVG glyph strings across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |
| **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/opencompany/index.html`), 12-theme docs (`guidelines/THEMES.md`), **the merged handoff brief (`HANDOFF.md` — glow-ladder traps, step-edge contract, toolbar layout traps, canvas-layer asymmetries, plus the recorded product amendments)**, and **the panel×theme fidelity target (`reference-mockup/Panel Theme Matrix.dc.html` — 17 panels × 12 themes, open in a browser)**. `reference/themes/` no longer vendors CSS snapshots — it points at the authoritative [client/src/themes/](./client/src/themes/). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). |
| **[Schema Source of Truth RFC](./docs-internal/ARCHIVE/schema_source_of_truth_rfc.md)** | *Archived — shipped.* Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons; `VITE_NODESPEC_BACKEND` defaults on and the frontend node definitions are gone. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Current reference: [Plugin System](./docs-internal/plugin_system.md). |
@@ -29,7 +29,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Proxy Service](./docs-internal/proxy_service.md)** | Residential proxy provider management with template-based URL formatting, health scoring, and transparent HTTP node injection |
| **[Email Service](./docs-internal/email_service.md)** | IMAP/SMTP integration via Himalaya CLI with EmailService orchestrator, provider presets, custom credential fallback, and polling triggers |
| **[Browser](./docs-internal/browser.md)** | Current `browser` node: installed Chrome/Edge/Chromium in a dedicated profile rendered inside the workspace by default (`BROWSER_RUNTIME=system`, `BROWSER_HEADLESS=true`) and pinned browser-use CLI. Chrome for Testing requires explicit `BROWSER_RUNTIME=testing`. System mode retains the native user agent and OS password-store/keychain integration; basic/mock keychain flags are restricted to testing mode. Includes persistent employee/shared profiles (the Credentials / Connectors "Web browser" panel adds, deletes and imports logins from a session file; the node's profile dropdown lists only the caller's profiles, via `current_load_options_principal()`), session leases, agent/user handoff, network policy, screenshots as workspace FileRefs and authenticated `/ws/browser` live viewing. Plugin-owned installation is lazy (`_install_chrome.py`, `_install_bu.py`); importing or shutting down the plugin never installs it. Replaces the former agent-browser and separate `browserHarness` implementations. |
-| **[Browser Workspace](./docs-internal/browser_workspace.md)** | Home and Dev share `WorkspaceTabs` and `BrowserWorkspace`: Browser / Canvas / Android, capability-based discovery, binary JPEG frames, bounded frame credit and decoding, Take control / Hand back, visibility cleanup and payload-free opt-in diagnostics. The Browser node's parameter panel shows the same viewer, and the Dev dock opens on its Browser tab when an agent in the open workflow asks for help (`browser_updated`). Android remains a placeholder. |
+| **[Browser Workspace](./docs-internal/browser_workspace.md)** | Home and Dev share `WorkspaceTabs` and `BrowserWorkspace`: Browser / Canvas / Mobile, capability-based discovery, binary JPEG frames, bounded frame credit and decoding, Take control / Hand back, visibility cleanup and payload-free opt-in diagnostics. The Browser node's parameter panel shows the same viewer, and the Dev dock opens on its Browser tab when an agent in the open workflow asks for help (`browser_updated`). Mobile shows the managed local Android phone. |
| **[WhatsApp Business Service](./docs-internal/whatsapp_business_service.md)** | Official Meta Cloud API plugin in [`server/nodes/whatsapp_business/`](./server/nodes/whatsapp_business/) — **distinct from `nodes/whatsapp/`**, which is a personal account over an unofficial Go bridge; they share no node type, credential or palette group. Four nodes: `whatsappBusinessSend` (10 ops — Meta models message type as a *field* on one `/messages` endpoint, so text / media / template / interactive are operations, not nodes), `whatsappBusinessMedia` (the one genuinely separate endpoint family), and the `Receive` / `Status` triggers (kept separate because a canary trigger registers exactly one CloudEvents type). `phone_number_id` is deliberately **absent from Params** — on a dual-purpose ActionNode `execute_as_tool` merges `{**node_params, **tool_args}`, so a declared field is model-settable and a prompt injection could choose the sending identity; it comes from the credential only. Webhook via `register_webhook_source` on the shared catch-all, `X-Hub-Signature-256` HMAC failing closed, composite `wamid:status` dedup ids so `sent→delivered→read` don't collapse. |
| **[Stripe Service](./docs-internal/stripe_service.md)** | Stripe CLI integration — `stripeAction` (CLI pass-through) + `stripeReceive` (signed-webhook trigger). Reference plugin for the Wave 12 event framework AND the **CLI-managed-auth pattern**: browser OAuth via `stripe login --non-interactive` + `--complete <url>` (URL extracted from `next_step` shell command via `shlex.split`) + auto-installed binary (`ensure_stripe_cli` from GitHub releases) + marker-token reuse of `auth_service.store_oauth_tokens` + CloudEvents-shaped `broadcast_credential_event("credential.oauth.connected" \| ".disconnected", provider=...)` — zero per-provider hardcoding in the frontend. Daemon stdout/stderr ingested via `ProcessService`'s `line_handler` callback (no log-file tailing); credential gate routed through `await self.has_credential()` so non-api-key auth (`is_logged_in()`) plugs in via subclass override. |
| **[Vercel Service](./docs-internal/vercel_service.md)** | Vercel CLI integration — `vercelAction` (deploy / inspect / list + `custom` CLI passthrough; AI tool `vercel`). Second reference for the CLI-managed-auth pattern: the **device-flow variant** (`vercel login` is one blocking process — no two-step `--complete`; direct `create_subprocess_exec` + chunk-based banner read for the code-embedding URL, pumps drain pipes for the process lifetime, success gate = auth.json mtime advance + sniff). Dual auth: optional `vercel_token` field (injected as `VERCEL_TOKEN` env, never argv) OR CLI login with marker tokens. All invocations pin `--global-config <DATA_DIR>/vercel/` (the `CLAUDE_CONFIG_DIR` isolation idiom). `bun add` into the shared packages tree (pinned version, via `core/js_runtime.add_package`; the CLI runs on bun). First-deploy project guard raises `NodeUserError` instead of letting Vercel derive an invalid name from the workspace dir. Shipped with the generic OAuthConnect change: Login gates on **required** fields only, so optional fields never block CLI login. |
@@ -1719,6 +1719,14 @@ Toolkit aggregator nodes (the former `androidTool` and its `TOOLKIT_NODE_TYPES`
## Android Services Development Guide
+### Local emulator workspace (separate plugin)
+
+`server/nodes/mobile/` owns **Mobile Agent** (`mobile_use_agent`) and the **Android** tool (`android_tool`). Both declare `ui_hints.workspace.kind="mobile"`; Home and Dev render `MobileWorkspace`. Android's Tool output connects to an agent's Tools input and requires its own Model connection. Its AI-visible schema contains only `prompt`; saved configuration controls time/step limits. This plugin does not replace the relay services documented below.
+
+Startup must hold `driver_lock` while reading the initialization reply; regular commands use the same lock. `/status` must skip driver I/O during lifecycle transitions. A serial alone does not mean the phone is ready. Keep connection warnings separate from action errors and clear them after successful polling.
+
+Diagnostics live in `mobile/mobile.log` (1 MiB, three backups); recent events are exposed under **Help & diagnostics**. Never log prompts, credentials, screenshots or typed text. Relative mobile `DATA_DIR` currently resolves against backend CWD, so source-dev installations can live at `server/.opencompany/mobile`; that runtime directory is ignored by Git and must not be deleted as temporary test data. Full setup, lifecycle, storage caveats and iOS plan: [Mobile Workspace](docs/mobile-workspace.md).
+
### Architecture
Android services use a factory pattern with `createAndroidServiceNode()` for consistent node structure:
- **SquareNode Component**: Visual representation with configuration status indicators
diff --git a/README.md b/README.md
index 1543798a..134d7b10 100644
--- a/README.md
+++ b/README.md
@@ -133,6 +133,7 @@ Nothing here is a black box. What an employee learns lives in its memory, its no
- **Models** from OpenAI, Anthropic, Google, xAI, DeepSeek, Kimi, Mistral, Groq, Cerebras, Sarvam, and OpenRouter, or run local models for free with Ollama, LM Studio, or any OpenAI-compatible server such as llama.cpp or vLLM.
- **A live browser workspace** in Home and Dev mode: watch an employee browse, take control to sign in or help, then hand control back. It opens your installed Chrome, Edge or Chromium in a dedicated OpenCompany profile, keeping sessions between tasks without using your personal profile. [Browser runtime](docs-internal/browser.md) | [Workspace interface](docs-internal/browser_workspace.md).
- **Speech and translation**, so employees can listen, talk, and work in other languages.
+- **A local Android phone workspace** on Windows x64: set up a persistent emulator, watch it live, use it yourself, or let an AI agent use the **Android** tool. Optional setup requires SDK license acceptance; iOS is not yet supported. [Setup, controls and troubleshooting](docs/mobile-workspace.md).
- **78 skills** that ship ready to use, and a place to drop your own.
- **Built to keep running.** Employees survive restarts, pause and resume from Home or the canvas, and catch up on missed schedules. Your API keys are stored encrypted on your machine. Add a login for shared or cloud use; one command deploys to Google Cloud.
- **A canvas you will want to look at**, with 12 visual themes.
diff --git a/client/src/components/browser/BrowserWorkspace.tsx b/client/src/components/browser/BrowserWorkspace.tsx
index fd1f4f28..112732f5 100644
--- a/client/src/components/browser/BrowserWorkspace.tsx
+++ b/client/src/components/browser/BrowserWorkspace.tsx
@@ -1,3 +1,4 @@
+import { FullView } from '../workspace/FullView';
import { useCallback, useEffect, useRef, useState } from 'react';
import type { KeyboardEvent, PointerEvent } from 'react';
import { ArrowLeft, ArrowRight, Globe, RotateCw } from 'lucide-react';
@@ -312,7 +313,7 @@ function BrowserSessionView({ workflowId, nodeId, visible }: { workflowId: strin
const statusLabel = phase === 'connecting' ? 'Connecting…' : phase === 'idle' ? 'Browser is stopped' : phase === 'error' ? 'Disconnected' : control ? 'You have control' : state.state === 'agent' ? 'Employee is browsing' : state.state === 'awaiting_user' ? 'Your help is needed' : state.controller === 'other' ? 'Another viewer has control' : 'Live browser';
return (
- <div className="flex min-h-0 min-w-0 flex-1 flex-col gap-2">
+ <FullView label="Browser">
<div className="flex flex-wrap items-center justify-between gap-2 text-xs">
<span role="status" className="text-fg-muted">{installing ? 'Installing browser…' : starting ? 'Starting browser…' : statusLabel}</span>
{phase === 'live' && <Button size="sm" variant="outline" disabled={takingControl} onClick={() => {
@@ -352,6 +353,6 @@ function BrowserSessionView({ workflowId, nodeId, visible }: { workflowId: strin
<div className="flex gap-2">{[true, false].map((accept) => <Button key={String(accept)} size="sm" variant="outline" disabled={!control} onClick={() => { send({ type: 'dialog_reply', accept, prompt_text: promptText }); setDialog(null); }}>{accept ? 'OK' : 'Cancel'}</Button>)}</div>
{!control && <span className="text-xs text-fg-muted">Take control to respond.</span>}
</div>}
- </div>
+ </FullView>
);
}
diff --git a/client/src/components/mobile/MobileWorkspace.tsx b/client/src/components/mobile/MobileWorkspace.tsx
index e0482fdb..ed2efce8 100644
--- a/client/src/components/mobile/MobileWorkspace.tsx
+++ b/client/src/components/mobile/MobileWorkspace.tsx
@@ -1,3 +1,4 @@
+import { FullView } from '../workspace/FullView';
import { useCallback, useEffect, useRef, useState } from 'react';
import type { PointerEvent } from 'react';
import { ArrowLeft, Home, Play, RotateCw, Smartphone, Square } from 'lucide-react';
@@ -16,7 +17,7 @@ export default function MobileWorkspace({ workflowId, nodes, visible = true }: {
const [selected, setSelected] = useState('');
const nodeId = nodes.some((node) => node.node_id === selected) ? selected : nodes[0]?.node_id;
if (!workflowId || workflowId === 'new') return <Empty message="Save this workflow before opening its mobile workspace." />;
- if (!nodeId) return <Empty message="Add a Mobile Agent node to this workflow to use its phone here." />;
+ if (!nodeId) return <Empty message="Add a Mobile Agent or Android tool to this workflow to use its phone here." />;
return <div className="flex min-h-0 flex-1 flex-col gap-2">
{nodes.length > 1 && <select className={fieldClass} aria-label="Mobile agent" value={nodeId} onChange={(event) => setSelected(event.target.value)}>
{nodes.map((node) => <option key={node.node_id} value={node.node_id}>{node.label}</option>)}
@@ -35,6 +36,7 @@ function MobileSession({ workflowId, nodeId, visible }: { workflowId: string; no
const [status, setStatus] = useState<MobileStatus | null>(null);
const [doctor, setDoctor] = useState<Doctor | null>(null);
const [error, setError] = useState('');
+ const [connectionError, setConnectionError] = useState('');
const [busy, setBusy] = useState('');
const [accepted, setAccepted] = useState(false);
const [prompt, setPrompt] = useState('');
@@ -63,7 +65,7 @@ function MobileSession({ workflowId, nodeId, visible }: { workflowId: string; no
const running = status?.running === true;
const refresh = useCallback(async (signal?: AbortSignal) => {
const next = await mobileRequest<MobileStatus>(`${path}/status`, undefined, signal);
- if (active.current) setStatus(next);
+ if (active.current) { setStatus(next); setConnectionError(''); }
}, [path]);
useEffect(() => {
@@ -78,7 +80,7 @@ function MobileSession({ workflowId, nodeId, visible }: { workflowId: string; no
let timer: ReturnType<typeof setTimeout>;
const poll = async () => {
try { await refresh(controller.signal); }
- catch (cause) { if (!controller.signal.aborted && active.current) setError(describe(cause instanceof Error ? cause.message : cause)); }
+ catch { if (!controller.signal.aborted && active.current) setConnectionError('Connection to the phone was lost. Retrying automatically…'); }
if (!controller.signal.aborted) timer = setTimeout(() => void poll(), 2000);
};
void poll();
@@ -179,45 +181,47 @@ function MobileSession({ workflowId, nodeId, visible }: { workflowId: string; no
});
const installed = doctor?.adb && doctor?.emulator && doctor?.image && doctor?.engine && doctor?.video;
const setupActive = typeof status?.setup === 'string' && status.setup.startsWith('installing_');
- return <div className="flex min-h-0 flex-1 flex-col gap-2 overflow-y-auto">
+ const setupFailed = status?.setup === 'error' || status?.setup === 'interrupted' || !!status?.setup_error;
+ return <FullView label="Phone"><div className="flex min-h-0 flex-1 flex-col gap-2 overflow-y-auto group-data-[full-view=true]/fullview:overflow-hidden">
<div className="flex flex-wrap items-center gap-2">
<Smartphone aria-hidden className="size-4" />
- <span className="flex-1 text-sm font-medium">Shared phone</span>
- <span role="status" className="text-xs text-fg-muted">{busy ? `${busy}…` : running ? status.control_state : status ? 'Stopped' : 'Connecting…'}</span>
+ <span className="flex-1 text-sm font-medium">Android phone</span>
+ <span role="status" className="text-xs text-fg-muted">{setupActive ? 'Setting up' : status?.starting ? 'Starting phone…' : busy ? `${busy}…` : running ? (held ? 'You’re using the phone' : status.active?.status === 'running' ? 'AI is working' : status.control_state === 'recovering' ? 'Reconnecting…' : 'Ready') : status ? 'Phone is off' : 'Connecting…'}</span>
<Button size="icon-sm" variant="ghost" aria-label="Refresh mobile status" onClick={() => void perform('refresh', async () => { setDoctor(await mobileRequest(`${path}/doctor`)); })}><RotateCw /></Button>
</div>
- {error && <p role="alert" className="m-0 rounded border border-destructive/30 bg-destructive/10 p-2 text-sm text-destructive">{error}</p>}
- {status?.setup_error && <p role="alert" className="m-0 text-sm text-destructive">{status.setup_error}</p>}
+ {connectionError && <p role="status" className="m-0 text-sm text-fg-muted">{connectionError}</p>}
+ {error && <p role="alert" className="m-0 rounded border border-destructive/30 bg-destructive/10 p-2 text-sm text-destructive"><span>{error}</span><button type="button" aria-label="Dismiss message" className="ml-3 underline" onClick={() => setError('')}>Dismiss</button></p>}
+ {setupFailed && <p role="alert" className="m-0 text-sm text-destructive">{status?.setup_error || (status?.setup === 'interrupted' ? 'Setup was interrupted. Retry to finish preparing the phone.' : 'Setup failed. Review the details below and retry.')}</p>}
{doctor?.supported === false && <p className="m-0 text-sm text-fg-muted">This host cannot run the mobile runtime. Android setup currently requires Windows x64 with virtualization available.</p>}
- {doctor && !installed && !running && <section aria-label="Mobile setup" className="space-y-2 rounded border border-border-default bg-bg-panel p-3">
- <h3 className="m-0 text-sm font-semibold">Set up your shared Android phone</h3>
- <p className="m-0 text-xs text-fg-muted">Download the Android tools, emulator image and mobile agent runtime. Apps and sign-ins stay on this phone between tasks.</p>
- <ul className="m-0 list-none space-y-1 p-0 text-xs text-fg-muted">{(['adb', 'emulator', 'image', 'engine', 'video'] as const).map((key) => <li key={key}>{key === 'adb' ? 'Android tools' : key === 'engine' ? 'Agent runtime' : key === 'video' ? 'Live video' : key === 'image' ? 'Android image' : 'Emulator'}: {doctor[key] ? 'Ready' : 'Needed'}</li>)}</ul>
+ {((doctor && !installed && !running) || setupActive || setupFailed) && <section aria-label="Mobile setup" className="space-y-2 rounded border border-border-default bg-bg-panel p-3">
+ <h3 className="m-0 text-sm font-semibold">Set up your Android phone</h3>
+ <p className="m-0 text-xs text-fg-muted">One-time setup downloads everything your phone needs. This may take several minutes. Your apps and sign-ins are saved.</p>
+ <details><summary className="cursor-pointer text-xs text-fg-muted">Download details</summary><ul className="m-0 list-none space-y-1 p-0 text-xs text-fg-muted">{(['adb', 'emulator', 'image', 'engine', 'video'] as const).map((key) => <li key={key}>{key === 'adb' ? 'Android tools' : key === 'engine' ? 'Agent runtime' : key === 'video' ? 'Live video' : key === 'image' ? 'Android image' : 'Emulator'}: {doctor?.[key] ? 'Ready' : 'Needed'}</li>)}</ul></details>
<label className="flex items-start gap-2 text-xs"><input type="checkbox" checked={accepted} onChange={(event) => setAccepted(event.target.checked)} aria-label="Accept Android SDK license terms" />
<span>I accept the <a className="underline" href="https://developer.android.com/studio/terms" target="_blank" rel="noreferrer">Android SDK license terms</a> and want to download the required components.</span></label>
- <Button size="sm" disabled={!accepted || !!busy || setupActive || doctor.supported === false} onClick={() => void perform('setup', async () => {
+ <Button size="sm" disabled={!accepted || !!busy || setupActive || doctor?.supported === false} onClick={() => void perform('setup', async () => {
await mobileRequest(`${path}/setup`, { licenses_accepted: true }); setDoctor(await mobileRequest(`${path}/doctor`));
- })}>Set up phone</Button>
- {setupActive && <p role="status" className="m-0 break-words text-xs text-fg-muted">{status?.setup === 'installing_engine' ? 'Installing the agent runtime…' : 'Installing Android tools and preparing your phone…'}</p>}
+ })}>{setupFailed ? 'Retry setup' : 'Set up phone'}</Button>
+ {(setupActive || setupFailed || status?.setup_progress) && <SetupProgress status={status} ticking={visible && pageVisible} />}
</section>}
- {doctor?.acceleration && <details className="rounded border border-border-default px-3 py-2 text-xs text-fg-muted">
- <summary className="cursor-pointer font-medium">Host readiness: hardware acceleration</summary>
- <p className="mb-0 mt-2 whitespace-pre-wrap break-words">{doctor.acceleration}</p>
- <p className="mb-0 mt-2">If acceleration is unavailable on Windows, enable virtualization in your computer�s firmware and Windows Hypervisor Platform in Windows Features, then restart Windows and refresh mobile status.</p>
+ {status?.setup === 'ready' && installed && !setupFailed && status.setup_progress && <details aria-label="Completed mobile setup" className="group-data-[full-view=true]/fullview:hidden rounded border border-border-default p-3 text-xs text-fg-muted"><summary className="cursor-pointer">Setup finished · View details</summary>
+ <SetupProgress status={status} ticking={visible && pageVisible} />
</details>}
+ {!running && !status?.starting && installed && <p className="m-0 text-sm text-fg-muted">Your phone is ready to turn on. Click Start phone to use it.</p>}
<div className="flex flex-wrap gap-2">
- <Button size="sm" variant="outline" disabled={!!busy || setupActive || !status || doctor?.supported === false || (!running && !installed)} onClick={() => void perform(running ? 'stopping' : 'starting', async () => { await mobileRequest(`${path}/${running ? 'stop' : 'start'}`, {}); setEpoch(null); })}>
+ <Button size="sm" variant="outline" disabled={!!busy || setupActive || status?.starting || !status || doctor?.supported === false || (!running && !installed)} onClick={() => void perform(running ? 'stopping' : 'starting', async () => { await mobileRequest(`${path}/${running ? 'stop' : 'start'}`, {}); setEpoch(null); })}>
{running ? <Square className="size-3.5" /> : <Play className="size-3.5" />}{running ? 'Stop phone' : 'Start phone'}
</Button>
{running && (held ? <>
- <Button size="sm" onClick={() => void perform('releasing', async () => { await mobileRequest(`${path}/release`, { viewer_id: viewerId, epoch, resume: true }); leaseEpoch.current = null; setEpoch(null); })} disabled={!!busy}>Hand back & resume</Button>
- <Button size="sm" variant="outline" onClick={() => void perform('releasing', async () => { await mobileRequest(`${path}/release`, { viewer_id: viewerId, epoch, resume: false }); leaseEpoch.current = null; setEpoch(null); })} disabled={!!busy}>Release control</Button>
+ <Button size="sm" onClick={() => void perform('releasing', async () => { await mobileRequest(`${path}/release`, { viewer_id: viewerId, epoch, resume: true }); leaseEpoch.current = null; setEpoch(null); })} disabled={!!busy}>Let AI continue</Button>
+ <Button size="sm" variant="outline" onClick={() => void perform('releasing', async () => { await mobileRequest(`${path}/release`, { viewer_id: viewerId, epoch, resume: false }); leaseEpoch.current = null; setEpoch(null); })} disabled={!!busy}>Finish using phone</Button>
</> : <Button size="sm" variant="outline" disabled={!!busy} onClick={() => void perform('taking control', async () => { const claim = await mobileRequest<{ epoch: number; owner: string }>(`${path}/takeover`, { viewer_id: viewerId });
if (!active.current || !viewActive.current) { await mobileRequest(`${path}/release`, { viewer_id: viewerId, epoch: claim.epoch, resume: false }); return; }
- leaseEpoch.current = claim.epoch; setEpoch(claim.epoch); setLeaseOwner(claim.owner); })}>Take control</Button>)}
+ leaseEpoch.current = claim.epoch; setEpoch(claim.epoch); setLeaseOwner(claim.owner); })}>Use phone</Button>)}
</div>
{running && <>
- <div ref={surface} className={`relative flex min-h-64 flex-1 touch-none items-center justify-center overflow-hidden rounded border border-border-default bg-bg-canvas ${held ? 'cursor-crosshair' : ''}`} onPointerDown={down} onPointerUp={up} onPointerCancel={() => { pointer.current = null; }} aria-label={held ? 'Phone screen: tap or drag to interact' : 'Phone screen, view only'}>
+ <p className="m-0 text-xs text-fg-muted">{held ? 'Tap or swipe the screen, just like a phone. Finish when you want the AI to use it.' : 'Watch your phone here. Choose Use phone to tap and type yourself.'}</p>
+ <div ref={surface} className={`relative flex h-[min(60vh,680px)] min-h-80 shrink-0 group-data-[full-view=true]/fullview:h-auto group-data-[full-view=true]/fullview:min-h-0 group-data-[full-view=true]/fullview:flex-1 touch-none items-center justify-center overflow-hidden rounded border border-border-default bg-slate-950 ${held ? 'cursor-crosshair' : ''}`} onPointerDown={down} onPointerUp={up} onPointerCancel={() => { pointer.current = null; }} aria-label={held ? 'Phone screen: tap or drag to interact' : 'Phone screen, view only'}>
<canvas ref={canvas} className="absolute h-full w-full object-contain" />
{!live && !videoError && <p className="pointer-events-none z-10 text-sm text-fg-muted">Connecting live view…</p>}
</div>
@@ -228,7 +232,7 @@ function MobileSession({ workflowId, nodeId, visible }: { workflowId: string; no
<input className={`${fieldClass} flex-1`} aria-label="Text to type on phone" value={text} onChange={(event) => setText(event.target.value)} placeholder="Type on phone" />
<Button size="sm" variant="outline" disabled={!text || !!busy} onClick={() => void perform('input', async () => { await input('text', { text }); setText(''); })}>Type</Button>
</div>}
- {held && <label className="flex items-center gap-2 text-xs text-fg-muted">Install APK (up to 256 MB)
+ {held && <label className="flex items-center gap-2 text-xs text-fg-muted">Install an app file (.apk, up to 256 MB)
<input type="file" accept=".apk" aria-label="Install APK" disabled={!!busy} className="min-w-0 flex-1 text-xs" onChange={(event) => {
const file = event.target.files?.[0]; event.target.value = '';
if (!file) return;
@@ -241,16 +245,56 @@ function MobileSession({ workflowId, nodeId, visible }: { workflowId: string; no
}} />
</label>}
</>}
- <form className="mt-auto flex shrink-0 flex-col gap-2 border-t border-border-default pt-3" onSubmit={(event) => { event.preventDefault(); void submit(); }}>
- <label htmlFor={`task-${nodeId}`} className="text-sm font-medium">Give this mobile agent a task</label>
- <textarea id={`task-${nodeId}`} className={`${fieldClass} resize-y`} rows={2} maxLength={12000} value={prompt} onChange={(event) => setPrompt(event.target.value)} placeholder="What should it do on the phone?" />
- <p className="m-0 text-xs text-fg-muted">Runs in this workflow. The phone is shared; tasks wait while another employee or person controls it.</p>
- <div className="flex items-center gap-2"><Button size="sm" type="submit" disabled={!prompt.trim() || !!busy || hasTask}>Run task</Button>
+ <form className="mt-auto flex shrink-0 flex-col group-data-[full-view=true]/fullview:hidden gap-2 border-t border-border-default pt-3" onSubmit={(event) => { event.preventDefault(); void submit(); }}>
+ <label htmlFor={`task-${nodeId}`} className="text-sm font-medium">Ask AI to use the phone</label>
+ <textarea id={`task-${nodeId}`} className={`${fieldClass} resize-y`} rows={2} maxLength={12000} value={prompt} onChange={(event) => setPrompt(event.target.value)} placeholder="For example: Open Settings and turn on dark mode" />
+ <p className="m-0 text-xs text-fg-muted">Tell the AI what you want done. Only one person or AI can use this phone at a time.</p>
+ <div className="flex items-center gap-2"><Button size="sm" type="submit" disabled={!running || !prompt.trim() || !!busy || hasTask}>Run task</Button>
{hasTask && <Button size="sm" type="button" variant="outline" disabled={!!busy} onClick={() => void perform('cancelling', async () => { setTask(await mobileRequest(`${path}/tasks/${encodeURIComponent(submission!)}/cancel`, {})); })}>Cancel task</Button>}
- {task && <span role="status" className="text-xs text-fg-muted">{taskState(task) || 'Submitted'}</span>}
+ {task && <span role="status" className="text-xs text-fg-muted">{({queued: 'Waiting for the phone', running: 'Working on your request', completed: 'Done', failed: 'Could not finish', cancelled: 'Cancelled', awaiting_user: 'Waiting for you'})[taskState(task)] || 'Request received'}</span>}
</div>
{task?.error != null && <p role="alert" className="m-0 break-words text-sm text-destructive">{describe(task.error)}</p>}
{task?.result != null && <pre className="m-0 max-h-48 overflow-auto whitespace-pre-wrap rounded bg-bg-panel p-2 text-xs">{describe(task.result)}</pre>}
</form>
+ {doctor?.acceleration && <details className="group-data-[full-view=true]/fullview:hidden rounded border border-border-default px-3 py-2 text-xs text-fg-muted">
+ <summary className="cursor-pointer font-medium">Help &amp; diagnostics</summary>
+ <p className="mb-0 mt-2">If the phone will not start, share the details below when asking for help.</p><pre className="mb-0 mt-2 whitespace-pre-wrap break-words">{doctor.acceleration}</pre>
+ <p className="mb-0 mt-2">If acceleration is unavailable on Windows, enable virtualization in your computer’s firmware and Windows Hypervisor Platform in Windows Features, then restart Windows and refresh mobile status.</p>
+ {!!status?.diagnostics?.length && <details className="mt-2"><summary className="cursor-pointer">Recent phone activity</summary><pre className="max-h-40 overflow-auto whitespace-pre-wrap">{status.diagnostics.map((item) => `${item.at} ${item.level} ${item.event}${item.operation ? ` (${item.operation})` : ''}${item.error_type ? `: ${item.error_type}` : ''}${item.code ? ` [${item.code}]` : ''}`).join('\n')}</pre></details>}
+ </details>}
+ </div></FullView>;
+}
+
+function SetupProgress({ status, ticking }: { status: MobileStatus | null; ticking: boolean }) {
+ const [now, setNow] = useState(() => Date.now() / 1000);
+ useEffect(() => {
+ if (!ticking) return;
+ setNow(Date.now() / 1000);
+ const timer = setInterval(() => setNow(Date.now() / 1000), 1000);
+ return () => clearInterval(timer);
+ }, [ticking]);
+ const progress = status?.setup_progress;
+ const phase = status?.setup === 'installing_engine' ? 'Installing agent runtime'
+ : status?.setup === 'installing_device' ? 'Preparing Android phone'
+ : status?.setup === 'interrupted' ? 'Setup interrupted'
+ : status?.setup === 'error' ? 'Setup failed' : status?.setup === 'ready' ? 'Setup complete' : 'Setup progress';
+ const duration = (seconds: number) => {
+ const value = Math.max(0, Math.floor(seconds));
+ return value < 60 ? `${value}s` : `${Math.floor(value / 60)}m ${value % 60}s`;
+ };
+ const events = progress?.events?.slice(-20) ?? [];
+ return <div className="space-y-1 text-xs text-fg-muted">
+ <p role="status" className="m-0 font-medium">{phase}</p>
+ {progress?.message && <p className="m-0 break-words">{progress.message}</p>}
+ <p className="m-0">
+ {progress?.started_at != null && <span>Elapsed: {duration((progress.finished_at ?? now) - progress.started_at)}</span>}
+ {progress?.started_at != null && progress?.updated_at != null && ' · '}
+ {progress?.updated_at != null && <span>Last update: {duration(now - progress.updated_at)} ago</span>}
+ </p>
+ {events.length > 0 && <details><summary className="cursor-pointer">Recent setup activity</summary>
+ <ol className="my-2 max-h-40 list-none space-y-1 overflow-y-auto p-0">
+ {events.map((event, index) => <li key={`${event.at}:${index}`} className="break-words"><time>{new Date(event.at * 1000).toLocaleTimeString()}</time> — {event.message}</li>)}
+ </ol>
+ </details>}
</div>;
}
diff --git a/client/src/components/mobile/__tests__/MobileWorkspace.test.tsx b/client/src/components/mobile/__tests__/MobileWorkspace.test.tsx
index d52977b8..2bf81806 100644
--- a/client/src/components/mobile/__tests__/MobileWorkspace.test.tsx
+++ b/client/src/components/mobile/__tests__/MobileWorkspace.test.tsx
@@ -20,6 +20,26 @@ beforeEach(() => {
afterEach(() => { cleanup(); vi.unstubAllGlobals(); });
describe('Mobile Workspace', () => {
+ it('clears a failed status check when the phone reconnects', async () => {
+ let polls = 0;
+ snapshot.running = true;
+ fetchMock.mockImplementation(async (url: string) => {
+ if (url.endsWith('/status') && polls++ === 0) throw new Error('Start the device before using it');
+ return json(url.endsWith('/doctor') ? ready : snapshot);
+ });
+ render(<MobileWorkspace workflowId="flow" nodes={nodes} />);
+ await screen.findByText(/Retrying automatically/);
+ await waitFor(() => expect(screen.queryByText(/Retrying automatically/)).toBeNull(), { timeout: 4000 });
+ expect(screen.getByText('Ready')).toBeInTheDocument();
+ expect(screen.queryByRole('alert')).toBeNull();
+ });
+
+ it('explains how to start and disables AI tasks while the phone is off', async () => {
+ render(<MobileWorkspace workflowId="flow" nodes={nodes} />);
+ await screen.findByText('Phone is off');
+ await userEvent.type(screen.getByRole('textbox', { name: 'Ask AI to use the phone' }), 'Open Settings');
+ expect(screen.getByRole('button', { name: 'Run task' })).toBeDisabled();
+ });
it('requires a saved workflow and an explicitly discovered Mobile node', () => {
const view = render(<MobileWorkspace nodes={nodes} />);
expect(screen.getByText(/Save this workflow/)).toBeInTheDocument();
@@ -33,7 +53,7 @@ describe('Mobile Workspace', () => {
await screen.findByRole('button', { name: 'Start phone' });
await waitFor(() => expect(fetchMock).toHaveBeenCalledTimes(2));
expect(fetchMock.mock.calls.every(([, options]) => options.method === 'GET')).toBe(true);
- expect(screen.getByText('Host readiness: hardware acceleration')).toBeInTheDocument();
+ expect(screen.getByText('Help & diagnostics')).toBeInTheDocument();
expect(screen.getByText('WHPX is installed and usable.')).toBeInTheDocument();
});
@@ -47,7 +67,45 @@ describe('Mobile Workspace', () => {
await waitFor(() => expect(fetchMock.mock.calls.some(([url, options]) => url.endsWith('/setup') && JSON.parse(options.body).licenses_accepted === true)).toBe(true));
});
+ it('shows active setup and bounded activity even when all tools are ready', async () => {
+ const now = Date.now() / 1000;
+ snapshot = { ...snapshot, setup: 'installing_device', setup_progress: {
+ message: 'Downloading Android image', started_at: now - 90, updated_at: now - 10,
+ events: Array.from({ length: 25 }, (_, index) => ({ at: now - 25 + index, message: `Step ${index}` })),
+ } };
+ render(<MobileWorkspace workflowId="flow" nodes={nodes} />);
+ await screen.findByText('Setting up');
+ expect(screen.getByRole('region', { name: 'Mobile setup' })).toBeInTheDocument();
+ expect(screen.getByText('Downloading Android image')).toBeInTheDocument();
+ expect(screen.getByText(/Elapsed: 1m/)).toBeInTheDocument();
+ expect(screen.getByText(/Last update: \d+s/)).toBeInTheDocument();
+ expect(screen.queryByText(/— Step 0$/)).toBeNull();
+ expect(screen.getByText(/Step 24$/)).toBeInTheDocument();
+ expect(screen.getByRole('button', { name: 'Set up phone' })).toBeDisabled();
+ });
+
+ it('retains completed setup activity without install controls', async () => {
+ const now = Date.now() / 1000;
+ snapshot = { ...snapshot, setup: 'ready', setup_progress: { message: 'Phone is ready', started_at: now - 120, finished_at: now - 30, updated_at: now - 30, events: [{ at: now - 30, message: 'Installation finished' }] } };
+ render(<MobileWorkspace workflowId="flow" nodes={nodes} />);
+ await screen.findByText('Setup complete');
+ expect(screen.getByText('Elapsed: 1m 30s')).toBeInTheDocument();
+ expect(screen.getByText(/Installation finished/)).toBeInTheDocument();
+ expect(screen.queryByRole('checkbox', { name: 'Accept Android SDK license terms' })).toBeNull();
+ });
+
+ it('keeps interrupted setup visible after reopening and offers an explicit retry', async () => {
+ snapshot = { ...snapshot, setup: 'interrupted', setup_error: 'Setup stopped when the host restarted.' };
+ render(<MobileWorkspace workflowId="flow" nodes={nodes} />);
+ await screen.findByText('Setup stopped when the host restarted.');
+ const retry = screen.getByRole('button', { name: 'Retry setup' });
+ expect(retry).toBeDisabled();
+ await userEvent.click(screen.getByRole('checkbox', { name: 'Accept Android SDK license terms' }));
+ expect(retry).toBeEnabled();
+ });
+
it('submits only to the displayed workflow and retries a lost response with the same id', async () => {
+ snapshot.running = true;
let attempts = 0;
fetchMock.mockImplementation(async (url: string, options: RequestInit) => {
if (url.endsWith('/tasks') && options.method === 'POST') {
@@ -59,11 +117,11 @@ describe('Mobile Workspace', () => {
return json(url.endsWith('/doctor') ? ready : snapshot);
});
render(<MobileWorkspace workflowId="home-flow" nodes={nodes} />);
- await userEvent.type(screen.getByRole('textbox', { name: 'Give this mobile agent a task' }), 'Open settings');
+ await userEvent.type(screen.getByRole('textbox', { name: 'Ask AI to use the phone' }), 'Open settings');
await userEvent.click(screen.getByRole('button', { name: 'Run task' }));
await screen.findByText('Connection dropped');
await userEvent.click(screen.getByRole('button', { name: 'Run task' }));
- await screen.findByText('Done');
+ await screen.findByText('Done', { selector: 'pre' });
const requests = fetchMock.mock.calls.filter(([url, options]) => url.endsWith('/tasks') && options.method === 'POST');
expect(requests).toHaveLength(2);
expect(requests[0][0]).toBe('/api/mobile/home-flow/flow%3Amobile_agent%3A1/tasks');
@@ -82,7 +140,7 @@ describe('Mobile Workspace', () => {
});
const view = render(<MobileWorkspace workflowId="flow" nodes={nodes} />);
expect(screen.queryByRole('button', { name: 'Phone Home' })).toBeNull();
- await userEvent.click(await screen.findByRole('button', { name: 'Take control' }));
+ await userEvent.click(await screen.findByRole('button', { name: 'Use phone' }));
await userEvent.click(await screen.findByRole('button', { name: 'Phone Home' }));
await waitFor(() => expect(fetchMock.mock.calls.some(([url]) => url.endsWith('/input'))).toBe(true));
const request = fetchMock.mock.calls.find(([url]) => url.endsWith('/input'))!;
diff --git a/client/src/components/mobile/api.ts b/client/src/components/mobile/api.ts
index d700a337..24e46888 100644
--- a/client/src/components/mobile/api.ts
+++ b/client/src/components/mobile/api.ts
@@ -3,9 +3,12 @@ import { buildApiUrl } from '@/config/api';
export interface Geometry { width: number; height: number; rotation?: number; revision?: number; [key: string]: unknown }
export interface MobileStatus {
running: boolean;
+ starting?: boolean;
+ diagnostics?: { at: string; event: string; level: string; operation?: string; error_type?: string; code?: string; duration_ms?: number }[];
geometry?: Geometry | null;
setup?: string | Record<string, unknown> | null;
setup_error?: string | null;
+ setup_progress?: { message?: string; started_at?: number | null; updated_at?: number | null; finished_at?: number | null; events?: { at: number; message: string }[] } | null;
controller?: string | null;
epoch: number;
control_state: string;
diff --git a/client/src/components/workspace/FullView.tsx b/client/src/components/workspace/FullView.tsx
new file mode 100644
index 00000000..9aa72014
--- /dev/null
+++ b/client/src/components/workspace/FullView.tsx
@@ -0,0 +1,46 @@
+import { useEffect, useRef, useState, type ReactNode } from 'react';
+import { Maximize2, Minimize2 } from 'lucide-react';
+import { Button } from '@/components/ui/button';
+
+/** Fullscreen the existing viewer; never remount its canvas or connection. */
+export function FullView({ label, children }: { label: string; children: ReactNode }) {
+ const root = useRef<HTMLDivElement>(null);
+ const button = useRef<HTMLButtonElement>(null);
+ const [full, setFull] = useState(false);
+ const [error, setError] = useState('');
+ useEffect(() => {
+ const element = root.current;
+ let wasFull = false;
+ const changed = () => {
+ const next = document.fullscreenElement === element;
+ setFull(next);
+ if (wasFull && !next) button.current?.focus();
+ wasFull = next;
+ };
+ document.addEventListener('fullscreenchange', changed);
+ return () => {
+ document.removeEventListener('fullscreenchange', changed);
+ if (document.fullscreenElement === element) void document.exitFullscreen().catch(() => {});
+ };
+ }, []);
+ const toggle = async () => {
+ setError('');
+ try {
+ if (document.fullscreenElement === root.current) await document.exitFullscreen();
+ else if (root.current?.requestFullscreen) await root.current.requestFullscreen();
+ else setError('Full view is unavailable in this browser. Use the Workspace expand button instead.');
+ } catch {
+ setError('Could not open full view. Try again or use the Workspace expand button.');
+ }
+ };
+ return <div ref={root} data-full-view={full} className={`group/fullview flex min-h-0 min-w-0 flex-1 flex-col gap-2 bg-bg-app ${full ? 'h-screen w-screen overflow-hidden p-3' : ''}`}>
+ <div className="flex shrink-0 items-center justify-end gap-2">
+ {full && <span className="mr-auto text-sm font-medium">{label} <span className="text-xs font-normal text-fg-muted">· Esc to return</span></span>}
+ <Button ref={button} size="sm" variant="outline" aria-label={full ? `Exit ${label.toLowerCase()} full view` : `${label} full view`} aria-pressed={full} onClick={() => void toggle()}>
+ {full ? <Minimize2 className="size-4" /> : <Maximize2 className="size-4" />}{full ? 'Exit full view' : 'Full view'}
+ </Button>
+ </div>
+ {error && <p role="alert" className="m-0 text-xs text-destructive">{error}</p>}
+ {children}
+ </div>;
+}
diff --git a/client/src/components/workspace/__tests__/FullView.test.tsx b/client/src/components/workspace/__tests__/FullView.test.tsx
new file mode 100644
index 00000000..adf1b9e9
--- /dev/null
+++ b/client/src/components/workspace/__tests__/FullView.test.tsx
@@ -0,0 +1,42 @@
+import { useEffect } from 'react';
+import { afterEach, expect, it, vi } from 'vitest';
+import { act, cleanup, fireEvent, render, screen } from '@testing-library/react';
+import { FullView } from '../FullView';
+
+afterEach(() => {
+ cleanup(); vi.restoreAllMocks();
+ Reflect.deleteProperty(document, 'fullscreenElement');
+ Reflect.deleteProperty(document, 'exitFullscreen');
+ Reflect.deleteProperty(HTMLElement.prototype, 'requestFullscreen');
+});
+
+it('enters and exits full view without remounting the viewer', async () => {
+ let fullscreen: Element | null = null;
+ const mount = vi.fn();
+ function Viewer() { useEffect(() => { mount(); }, []); return <canvas data-testid="viewer" />; }
+ Object.defineProperty(document, 'fullscreenElement', { configurable: true, get: () => fullscreen });
+ const request = vi.fn(() => {
+ fullscreen = screen.getByTestId('viewer').parentElement;
+ document.dispatchEvent(new Event('fullscreenchange')); return Promise.resolve();
+ });
+ Object.defineProperty(HTMLElement.prototype, 'requestFullscreen', { configurable: true, value: request });
+ Object.defineProperty(document, 'exitFullscreen', { configurable: true, value: vi.fn(async () => {
+ fullscreen = null; document.dispatchEvent(new Event('fullscreenchange'));
+ }) });
+ render(<FullView label="Phone"><Viewer /></FullView>);
+ const canvas = screen.getByTestId('viewer');
+ await act(async () => { fireEvent.click(screen.getByRole('button', { name: 'Phone full view' })); });
+ expect(screen.getByRole('button', { name: 'Exit phone full view' })).toHaveAttribute('aria-pressed', 'true');
+ expect(screen.getByTestId('viewer')).toBe(canvas);
+ // Escape is owned by the browser; fullscreenchange synchronizes the UI.
+ await act(async () => { await document.exitFullscreen(); });
+ expect(screen.getByRole('button', { name: 'Phone full view' })).toHaveFocus();
+ expect(mount).toHaveBeenCalledTimes(1);
+});
+
+it('explains when fullscreen is unavailable without hiding the viewer', async () => {
+ render(<FullView label="Browser"><canvas data-testid="viewer" /></FullView>);
+ await act(async () => { fireEvent.click(screen.getByRole('button', { name: 'Browser full view' })); });
+ expect(screen.getByRole('alert')).toHaveTextContent('Full view is unavailable');
+ expect(screen.getByTestId('viewer')).toBeInTheDocument();
+});
diff --git a/docs-internal/SETUP.md b/docs-internal/SETUP.md
index 9b80f7b8..b793b6b1 100644
--- a/docs-internal/SETUP.md
+++ b/docs-internal/SETUP.md
@@ -2,6 +2,8 @@
## Project Structure
+For the optional local Android emulator, see [Mobile Workspace setup](../docs/mobile-workspace.md). It requires Windows x64, working virtualization, one backend process with embedded workers, and explicit SDK license acceptance. Add **Mobile Agent** or the **Android** tool, save the workflow, then open **Workspace → Mobile**. After backend/plugin changes, restart the backend and refresh the frontend. See the guide's runtime-file notes before moving or deleting an existing phone installation.
+
```
OpenCompany/
├── client/ # React frontend (Vite dev server on VITE_CLIENT_PORT, proxying the backend; production build served by uvicorn on PYTHON_BACKEND_PORT)
diff --git a/docs-internal/browser_workspace.md b/docs-internal/browser_workspace.md
index c96f3fe2..43781cfe 100644
--- a/docs-internal/browser_workspace.md
+++ b/docs-internal/browser_workspace.md
@@ -11,15 +11,28 @@ installed Chrome dynamically and keeps it selected even when another browser
has a newer version; no machine-specific executable path is needed.
Normal mode and Dev mode share `components/workspace/WorkspaceTabs.tsx`
-(Browser, Canvas, Android) and `components/browser/BrowserWorkspace.tsx`.
+(Browser, Canvas, Mobile) and `components/browser/BrowserWorkspace.tsx`.
The Dev toolbar's Workspace button opens the existing resizable dock;
Canvas boards, push notifications and temporary file previews still use
`canvasDockStore`. Its selected tab is persisted. A Canvas push does not
switch an already-open Browser tab; explicitly opening a file preview does
-switch to Canvas. Android currently has an explanatory panel, not a live mirror.
+switch to Canvas. Mobile renders the local Android phone through
+`MobileWorkspace`; see [Mobile Workspace](../docs/mobile-workspace.md).
The Browser node's parameter panel (`MiddleSection`, `isBrowserPanel`) shows
the same viewer above the node's settings.
+## Full view
+
+Click **Full view** above the browser to fill the screen. Navigation, address,
+control and exit buttons remain outside the page image. **Exit full view** or
+**Esc** returns to the previous layout. The same mounted canvas and stream stay
+active, so entering or leaving does not reopen the session. The shared
+`components/workspace/FullView.tsx` boundary handles native fullscreen changes
+and restores focus to its button on exit. Mobile uses the same boundary, hiding
+its task form and technical panels while enlarged. Full view does not grant
+manual control; use the existing control button separately. If fullscreen is
+unavailable, the UI explains how to use the Workspace expand button instead.
+
## Discovery and identity
Normal-mode employee summaries include `browser_nodes: [{node_id, label}]`.
diff --git a/docs-internal/frontend_architecture.md b/docs-internal/frontend_architecture.md
index 47f048d3..af0d8d27 100644
--- a/docs-internal/frontend_architecture.md
+++ b/docs-internal/frontend_architecture.md
@@ -502,12 +502,22 @@ See [media_transport.md](./media_transport.md).
## Browser workspace live view
-Home and Dev share [WorkspaceTabs](../client/src/components/workspace/WorkspaceTabs.tsx) and [BrowserWorkspace](../client/src/components/browser/BrowserWorkspace.tsx). Browser nodes are discovered through the backend `isBrowserPanel` capability; viewing attaches to a saved workflow/node without launching Chrome. The runtime defaults to installed Chrome/Edge/Chromium with a dedicated profile rendered headless inside the workspace; a separate desktop window and testing-browser mode are explicit server settings. The Browser tab supports server-authorized Take control / Hand back for navigation, mouse, keyboard, paste and login. The Browser node's parameter panel shows the same viewer above its settings. When an agent in the open workflow calls `request_user`, the `browser_updated` case in `WebSocketContext` opens the Dev dock on its Browser tab (`canvasDockStore.showBrowser`); Home shows Needs you from the employee summary's `browser_request`. Canvas retains its renderer; Android remains an explanatory placeholder.
+Home and Dev share [WorkspaceTabs](../client/src/components/workspace/WorkspaceTabs.tsx) and [BrowserWorkspace](../client/src/components/browser/BrowserWorkspace.tsx). Browser nodes are discovered through the backend `isBrowserPanel` capability; viewing attaches to a saved workflow/node without launching Chrome. The runtime defaults to installed Chrome/Edge/Chromium with a dedicated profile rendered headless inside the workspace; a separate desktop window and testing-browser mode are explicit server settings. The Browser tab supports server-authorized Take control / Hand back for navigation, mouse, keyboard, paste and login. The Browser node's parameter panel shows the same viewer above its settings. When an agent in the open workflow calls `request_user`, the `browser_updated` case in `WebSocketContext` opens the Dev dock on its Browser tab (`canvasDockStore.showBrowser`); Home shows Needs you from the employee summary's `browser_request`. Canvas retains its renderer. The **Mobile** tab renders the shared Android phone through `MobileWorkspace`; it is no longer a placeholder. See [Mobile Workspace](../docs/mobile-workspace.md).
The viewer uses a separate authenticated same-origin `/ws/browser` socket, not the general `WebSocketContext` frame stream. Binary JPEG envelopes are decoded sequentially with a two-frame local bound; consumed/skipped frames are acknowledged. Canvas dimensions change only when image dimensions change, and input coordinates use the metadata of the frame actually painted. Hidden, disposed or obsolete frames cannot restore a stale picture. Blur, pointer cancellation, hiding and unmount release user control; the server owns the release barrier and lease checks.
`?browserStreamDebug=1` enables numeric first-frame/decode/draw summaries without recording page or input payloads. The [Browser workspace contract](./browser_workspace.md) describes identity, visibility and interaction; [Browser](./browser.md) owns runtime installation, profiles, security and capture/control internals. Tests live under `components/browser/__tests__`, with Home and Dev dock integration tests beside their hosts.
+## Mobile workspace
+
+`components/workspace/FullView.tsx` gives both browser and phone sessions a native fullscreen boundary. It requests fullscreen on the existing DOM container and listens for `fullscreenchange`, preserving mounted canvases and stream connections. A separate toolbar provides exit; Escape uses the browser's native exit behavior and focus returns to the full-view button. Phone secondary panels hide through the `group/fullview` state, while the live surface flexes to remaining height. Browser navigation and dialogs remain outside the page canvas. Full view never acquires device control.
+
+Home and Dev use the same `components/mobile/MobileWorkspace.tsx`. Both `mobile_use_agent` and `android_tool` advertise `ui_hints.workspace.kind="mobile"` and are discovered through `workspace_nodes`. The persisted tab key remains `android` for compatibility; the visible label is **Mobile**.
+
+Status is polled every two seconds while visible. `starting` describes initialization; `running` requires the emulator, driver, and geometry to be ready. Status polling avoids driver I/O during lifecycle transitions. The driver initialization handshake and later commands share the same lock. The frontend clears recovered connection warnings separately from action errors, and keeps AI task submission disabled while the phone is off.
+
+Video uses an authenticated binary `/ws/mobile/{workflow_id}/{node_id}` scrcpy/H.264 stream; HTTP mutations carry the viewer identity and control epoch. Setup details and **Help & diagnostics** stay collapsed by default. Runtime lifecycle, storage, model wiring and limitations are documented in [Mobile Workspace](../docs/mobile-workspace.md).
+
## Ownership boundary: TanStack Query vs Zustand vs WebSocketContext
This is the rule that keeps the data layer schema-driven instead of imperatively glued together.
diff --git a/docs-internal/node-logic-flows/README.md b/docs-internal/node-logic-flows/README.md
index def4ed66..b72daed8 100644
--- a/docs-internal/node-logic-flows/README.md
+++ b/docs-internal/node-logic-flows/README.md
@@ -178,6 +178,11 @@ that a retired node remains registered.
- [Outlook Mail (`msMail`)](./microsoft/msMail.md)
- [Outlook Mail Receive (`msMailReceive`)](./microsoft/msMailReceive.md)
+### mobile
+
+- [Android (`android_tool`)](./mobile/android_tool.md)
+- [Mobile Agent (`mobile_use_agent`)](./mobile/mobile_use_agent.md)
+
### search
- [Brave Search (`braveSearch`)](./search/braveSearch.md)
diff --git a/docs-internal/node-logic-flows/mobile/android_tool.md b/docs-internal/node-logic-flows/mobile/android_tool.md
new file mode 100644
index 00000000..7f48e98d
--- /dev/null
+++ b/docs-internal/node-logic-flows/mobile/android_tool.md
@@ -0,0 +1,57 @@
+# Android (`android_tool`)
+
+| Field | Value |
+| --- | --- |
+| **Category** | tool |
+| **Backend handler** | [`AndroidTool.execute_op`](../../../server/nodes/mobile/_tool.py) via `ToolNode` |
+| **Tests** | [`test_android_tool.py`](../../../server/tests/test_android_tool.py) |
+| **Tool name** | `android` |
+
+## Purpose
+
+Give an existing AI agent a natural-language tool for the shared local Android phone. Setup and startup happen explicitly in Workspace → Mobile. This is separate from the legacy Android relay services.
+
+## Inputs (handles)
+
+| Handle | Required | Purpose |
+| --- | --- | --- |
+| `input-model` | Yes | Exactly one enabled OpenAI, Anthropic, or Gemini model connector for the embedded mobile-use engine |
+
+## Parameters
+
+The AI-visible schema accepts only `prompt`: a required string, 1–20,000 characters. Whitespace-only requests are rejected by the shared execution path. Extra tool arguments are forbidden.
+
+Saved configuration uses `MobileParams`: `prompt` defaults to empty, `max_steps` defaults to 40 (1–200), and `timeout_s` defaults to 900 (30–3600 seconds). In tool mode, the model supplies the request while the saved configuration controls the step/time limits. A parent agent's model connection does not replace `input-model` on this node.
+
+## Outputs (handles)
+
+`output-tool` connects to an AI agent's `input-tools`. The result contains `response`, `outcome`, `run_id`, and `artifacts` (currently empty). Standard tool dispatch returns a flat result or error; there are no main-flow handles.
+
+## Logic Flow
+
+1. Standard tool dispatch validates the model input separately from saved configuration.
+2. The operation combines the prompt with saved limits and calls the shared Mobile Agent execution path.
+3. That path verifies local owner access, saved workflow/execution identity, and the model connector.
+4. The shared runtime queues the task and grants a fenced device lease before starting the isolated mobile-use worker.
+5. The worker returns its result; cleanup revokes capabilities and drains pending device operations.
+
+## Decision Logic
+
+Missing setup or a stopped phone produces an actionable error; execution never accepts SDK licenses or starts installation. Human control pauses agent use. Tasks use the same queue, cancellation and manual-resume rules as Mobile Agent. Automatic activity retries are disabled to avoid replaying phone actions.
+
+## Side Effects
+
+Model API requests, device reads and mutations through the broker, a task worker subprocess, and bounded operational logs. The plugin reads saved model configuration and provider credentials server-side. Prompts and screenshots reach the selected model provider but are excluded from diagnostic logs.
+
+## External Dependencies
+
+The installed optional mobile-use environment, managed Android SDK/AVD, local embedded Temporal worker, and a supported model credential. Node import only registers capabilities; it starts no emulator or downloads.
+
+## Edge cases & known limits
+
+Windows x64 managed Android only; one shared phone and one video viewer. No iOS, remote device farm, arbitrary device attachment, or independent worker process. The task schema cannot override workflow scope, device serial, credentials, or saved execution limits.
+
+## Related
+
+- [Mobile Agent](mobile_use_agent.md)
+- [Mobile Workspace setup, controls, logging and limitations](../../../docs/mobile-workspace.md)
diff --git a/docs-internal/node-logic-flows/mobile/mobile_use_agent.md b/docs-internal/node-logic-flows/mobile/mobile_use_agent.md
new file mode 100644
index 00000000..31ebf543
--- /dev/null
+++ b/docs-internal/node-logic-flows/mobile/mobile_use_agent.md
@@ -0,0 +1,62 @@
+# Mobile Agent (`mobile_use_agent`)
+
+| Field | Value |
+| --- | --- |
+| **Category** | agent |
+| **Backend handler** | [`MobileUseAgent.execute_op`](../../../server/nodes/mobile/_node.py) via `ActionNode` |
+| **Tests** | [`test_mobile_integration.py`](../../../server/tests/test_mobile_integration.py), [`test_mobile_runtime.py`](../../../server/tests/test_mobile_runtime.py) |
+| **Delegation** | Explicit `supports_delegation=True` |
+
+## Purpose
+
+Carry out direct Workspace requests or delegated employee tasks on one persistent local Android phone, using the pinned mobile-use engine. The agent-style card is intentional. Use the separate [Android tool](android_tool.md) when connecting to an existing agent's Tools input.
+
+## Inputs (handles)
+
+| Handle | Required | Purpose |
+| --- | --- | --- |
+| `input-main` | No | Task/main workflow input |
+| `input-model` | Yes | Exactly one enabled OpenAI, Anthropic, or Gemini model connector |
+| `input-context` | No | Context connection |
+
+## Parameters
+
+| Name | Default | Limits |
+| --- | --- | --- |
+| `prompt` | Empty | Required at execution; maximum 20,000 characters |
+| `max_steps` | 40 | 1–200 |
+| `timeout_s` | 900 | 30–3600 seconds of active task time |
+
+## Outputs (handles)
+
+`output-main` returns the action result; `output-top` is the **Delegate** connection. The result payload contains `response`, `outcome`, `run_id`, and `artifacts` (currently empty), within the standard action envelope.
+
+## Logic Flow
+
+1. Require local installation owner access and a saved workflow/execution identity.
+2. Validate the request and resolve one model from the saved connector and provider credentials.
+3. Derive a scoped task identity and connect to the private loopback capability broker.
+4. Queue against the shared phone, claim a control lease, and start an isolated worker with the selected model and remaining budget.
+5. On manual takeover, revoke worker access and drain admitted device actions. Resume uses a fresh worker and the remaining budget.
+6. Return the result; revoke capability and clean up the worker on completion, cancellation or failure.
+
+## Decision Logic
+
+Empty prompts, missing models, non-owner access and stopped devices fail explicitly. Setup and SDK license acceptance remain UI actions. There is one shared runtime per backend process. Direct Workspace calls use durable Temporal invocation identities, and repeated submission IDs cannot change the prompt. Activity retries are disabled.
+
+## Side Effects
+
+Model-provider requests, screenshot/hierarchy reads, phone mutations through the broker, task worker processes and operational logs. Apps and sign-ins persist in the AVD. No hosted Minitap account is required; no hosted run reports are produced.
+
+## External Dependencies
+
+Optional installed mobile-use environment, managed Android SDK/AVD, scrcpy video, supported model credentials, and the backend's embedded worker. Model choice applies to every mobile-use stage.
+
+## Edge cases & known limits
+
+Windows x64 only; one shared device and video viewer. No current iOS implementation or arbitrary emulator attachment. Multiple API processes or a separate worker process cannot share the process-local broker. Device startup serializes the driver handshake with commands; status polling reports startup without reading the initialization pipe.
+
+## Related
+
+- [Android tool](android_tool.md)
+- [Mobile Workspace guide and iOS plan](../../../docs/mobile-workspace.md)
diff --git a/docs-internal/node-logic-flows/specialized_agents/android_agent.md b/docs-internal/node-logic-flows/specialized_agents/android_agent.md
index ffcf5fc3..7843c39c 100644
--- a/docs-internal/node-logic-flows/specialized_agents/android_agent.md
+++ b/docs-internal/node-logic-flows/specialized_agents/android_agent.md
@@ -9,6 +9,8 @@
## Purpose
+For the managed local emulator in Workspace → Mobile, use [Mobile Agent](../mobile/mobile_use_agent.md) or the [Android tool](../mobile/android_tool.md). The `android_agent` node below is the existing specialized agent for Android service/relay tools; adding it alone does not create the local emulator.
+
AI agent pre-configured for Android device control. Users typically connect
Android service nodes (`batteryMonitor`, `wifiAutomation`, `appLauncher`,
etc.) directly to `input-tools`, or attach
diff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.md
index 2a42d541..8593658d 100644
--- a/docs-internal/plugin_system.md
+++ b/docs-internal/plugin_system.md
@@ -42,6 +42,8 @@ plus one `@Operation` method, and nothing else.
## Architecture
+The [Mobile plugin](../server/nodes/mobile/__init__.py) is an optional runtime example: **Mobile Agent** (`ActionNode`) and **Android** (`ToolNode`) register together and advertise `ui_hints.workspace.kind="mobile"`. Import only registers the router, schemas and shutdown hook. The Android tool uses a locked, prompt-only `ToolInput`; saved `MobileParams` retain time/step limits. Both nodes share owner checks, model resolution, task queue and broker, without integrating through the legacy Android relay. See [Mobile Workspace](../docs/mobile-workspace.md) and the [Android node contract](node-logic-flows/mobile/android_tool.md).
+
### Class hierarchy
```
diff --git a/docs/mobile-workspace.md b/docs/mobile-workspace.md
index 88f08488..e9b779fc 100644
--- a/docs/mobile-workspace.md
+++ b/docs/mobile-workspace.md
@@ -1,6 +1,6 @@
# Mobile Workspace
-Mobile Workspace runs the open-source **mobile-use** agent against one persistent, local Android emulator. The Mobile Agent is a plugin node with a Workspace capability; its task execution uses the saved workflow, node configuration, and authenticated principal. The emulator belongs to the OpenCompany installation and is shared across Mobile nodes and workflows. Tasks queue for this device.
+Mobile Workspace runs the open-source **mobile-use** agent against one persistent, local Android emulator. Mobile Agent and the Android tool are plugin nodes with a Workspace capability; its task execution uses the saved workflow, node configuration, and authenticated principal. The emulator belongs to the OpenCompany installation and is shared across Mobile nodes and workflows. Tasks queue for this device.
## Supported deployment
@@ -13,16 +13,73 @@ The runtime singleton and its leases are process-local. Run one backend process
## Setup and use
1. Install `uv` for the optional Python environment and enable the host virtualization support required by Android Emulator. Setup downloads pinned Android command-line tools and a private Temurin Java runtime when needed. An existing SDK can be selected with `ANDROID_SDK_ROOT` or `ANDROID_HOME`; the standard Windows Android SDK location is also detected.
-2. Add a Mobile Agent node, save the workflow, and open its Mobile Workspace. The setup diagnostics inspect SDK tools, emulator acceleration, the system image, engine, and video server.
+2. Add **Mobile Agent** for direct tasks or employee delegation, or **Android** for an existing AI agent to call as a tool. Save the workflow and open **Workspace → Mobile**. The setup diagnostics inspect SDK tools, emulator acceleration, the system image, engine, and video server.
3. Review and accept the Android SDK license in setup. Setup installs the optional engine and Android packages and creates the persistent `OpenCompany` AVD using a Pixel 7 device profile. Network access and sufficient disk space are required for these downloads.
-4. Start the shared device. The emulator runs without a separate desktop window; the Workspace shows its video. Device storage persists across stopping and starting the emulator.
-5. Take manual control to sign in to the Play Store, install applications, or upload an APK. APK uploads are limited to 256 MB and require the current manual-control lease. Play Store availability comes from the selected Google Play system image; account sign-in is performed on the device.
-6. Connect a provider in OpenCompany Settings, then connect exactly one AI model node to the Mobile Agent's Model input and save the workflow. The model selected on that connector is used for every mobile-use stage. OpenAI, Anthropic, and Gemini credentials are supported. Choose a model with image input and tool use, then submit a task from Workspace or delegate from an employee.
+4. Click **Start phone** and wait for **Ready**. The emulator runs without a separate desktop window; the Workspace shows its video. Device storage persists across stopping and starting the emulator.
+5. Click **Use phone** to sign in to the Play Store, install applications, or upload an APK. APK uploads are limited to 256 MB and require the current manual-control lease. Play Store availability comes from the selected Google Play system image; account sign-in is performed on the device.
+6. Connect a provider in OpenCompany Settings, then connect exactly one AI model node to the phone node's **Model** input and save the workflow. The model selected on that connector is used for every mobile-use stage. OpenAI, Anthropic, and Gemini credentials are supported. Choose a model with image input and tool use, then enter a request under **Ask AI to use the phone** and click **Run task**, or call the node through its agent connection.
The setup is optional. Ordinary application startup does not install the mobile engine or boot an emulator. Runtime data lives under the configured OpenCompany data directory's `mobile` subdirectory, including the engine environment, AVD, and device resource metadata. Stopping the emulator does not reset its apps or account sessions.
+Setup shows the current component, download or SDK output, elapsed time, last update age, and recent activity. Complete compatible SDK packages are reused. Progress is saved in `mobile/setup-status.json`; reopening Workspace retains it. A backend shutdown or restart during setup is shown as interrupted with a retry action, rather than silently returning to idle. Retry reuses completed components; an incomplete download may need to restart. Setup does not continue while the backend is stopped. Timeout failures include the command and its latest output.
+
+## Choosing a node
+
+| Node | Use it for | Connection |
+| --- | --- | --- |
+| **Mobile Agent** (`mobile_use_agent`) | Direct Workspace requests or delegated phone tasks | Exactly one model into **Model**; **Delegate** output for employee delegation |
+| **Android** (`android_tool`) | Giving an existing AI agent a phone tool | **Tool** output into the agent’s **Tools** input, plus exactly one model into Android’s **Model** input |
+
+Both nodes use the same phone. Adding another node does not create another emulator. The existing `android_agent` and relay service nodes are separate integrations; they do not set up this local phone.
+
+## Everyday controls
+
+Choose **Full view** above the phone to fill the screen. The live phone expands to the available height; the task form, completed setup history and technical help are hidden. Phone controls remain outside the picture so they do not cover Android buttons. Use **Exit full view** or **Esc** to return. Entering full view keeps the current connection and does not restart the phone or grant manual control. Browser Workspace has the same full-view control, retaining its address bar and navigation outside the page. If the browser does not support fullscreen, use the existing Workspace expand button instead.
+
+| Label | Meaning |
+| --- | --- |
+| **Phone is off / Start phone** | The phone is installed but needs to be started before tasks can run. |
+| **Starting phone…** | Android and its automation connection are initializing. Wait for **Ready**. |
+| **Ready** | The phone can accept a task or manual control. |
+| **Use phone** | Pause AI control and use the screen yourself. |
+| **Finish using phone** | Release manual control without explicitly resuming a paused task. |
+| **Let AI continue** | Release manual control and resume a waiting task from the current screen. |
+| **AI is working** | An agent is using the shared phone. |
+| **Stop phone** | Shut down the phone while preserving apps and sign-ins. Cancel an active task first. |
+
+Connection warnings clear after a successful status check. Action errors have a **Dismiss** button. Task submission is disabled while the phone is off. Download details, completed setup history, and technical diagnostics are collapsed by default.
+
+## Runtime files and Git
+
+`mobile_root()` currently resolves a relative `DATA_DIR` against the backend process’s working directory. With source development launched from `server/` and `DATA_DIR=.opencompany`, the installed phone is therefore under `server/.opencompany/mobile/`. This differs from the application’s canonical repository-relative data resolver. An absolute `DATA_DIR` avoids this ambiguity. Do not move an existing installation or switch its data directory while the backend is running.
+
+The repository ignores `server/.opencompany/`. Emulator disks, snapshots, downloaded dependencies and logs are runtime data, not source changes; deleting them is not Git cleanup. Preserve `avd/` to retain the phone’s apps and sign-ins. Test fixtures and research checkouts under `.tmp/` can be removed once their checks finish.
+
## Control and task lifecycle
+### Android tool for AI agents
+
+Add the **Android** tool node (`android_tool`) and connect its **Tool** output to an AI agent's **Tools** input. Connect an OpenAI, Anthropic, or Gemini model node to Android's **Model** input; the same model node can also feed the parent agent. Save the workflow, then set up and start the phone in Workspace → Mobile. The agent calls `android` with one argument, `prompt`, such as “Open Settings and turn on dark mode.” The tool uses the same mobile-use engine, owner checks, task queue, manual takeover, and saved time/step limits as Mobile Agent. It does not install software or accept licenses automatically.
+
+### Troubleshooting and logs
+
+Workspace shows plain-language phone status and clears connection warnings after a successful status check. **Use phone** enables manual interaction; **Let AI continue** returns control to a waiting task. Technical checks and recent events are under **Help & diagnostics**.
+
+Operational events are written to `mobile/mobile.log` relative to the active data directory, rotating at 1 MiB with three backups. They include startup, driver/video failures, task lifecycle and command durations. Errors include exception type, error code where available, and stack locations. Prompts, typed text, screenshots, credentials, and command parameters are excluded. Existing emulator output remains in `mobile/emulator.log`.
+
+| Symptom | Check or recovery |
+| --- | --- |
+| Setup appears slow | Expand download details and recent setup activity; large system images take time. Complete compatible SDK packages are reused. |
+| Setup interrupted after a backend restart | Use **Retry setup** after accepting the SDK terms. Completed components are reused; partial downloads may restart. |
+| Phone will not start | Open **Help & diagnostics** for the acceleration check, then inspect `mobile/mobile.log` and `mobile/emulator.log`. |
+| `readuntil() called while another coroutine is already waiting` during startup | Older code let status polling read the driver’s initialization pipe concurrently. The fix shares `driver_lock` across initialization and commands and skips geometry polling during lifecycle transitions. Restart the backend after updating. |
+| Phone works but an old error remains | Current UI clears recovered connection warnings; dismiss action errors. Refresh the frontend after updating. |
+| New Android node is absent | Restart the backend to register the plugin and refresh the frontend. Search for **Android**, type `android_tool`. |
+| Task asks for a model | Connect exactly one supported model to the phone node’s **Model** input and save the workflow. The parent agent’s model connection alone is insufficient. |
+| Live view cannot connect | Check recent video events. Only one viewer is supported; close another preview and use **Reconnect**. |
+
+`setup-status.json` retains setup progress across backend restarts. Recent diagnostic events in status are bounded to 40 records in memory; `mobile.log` and its rotated backups retain operational history on disk. Logs exclude user content, but still include workflow/run identifiers: review them before sharing. The startup race explains a failed request and phone cleanup; it does not by itself prove a whole-backend crash.
+
Video is a read-only scrcpy transport. Mouse, touch, text, navigation, rotation, APK installation, and agent mutations go through the server's device-control broker rather than the video socket. The initial preview supports one browser tab at a time.
Taking control revokes the agent's lease and capability before stopping its worker, then waits for an already admitted device mutation to drain. A monotonically increasing epoch rejects delayed inputs from an old controller. Operation identifiers deduplicate successful mutations within the current lease. An uncertain device failure fences control and requires recovery rather than replaying the mutation automatically.
@@ -68,9 +125,11 @@ Minitap's [platform](https://www.minitap.ai/platform#slack-cli-mcp) and [miniTes
Automated tests exercise fencing, successful-operation deduplication, cancellation while draining, takeover and resume with fake subprocesses, queued cancellation, source-patch compatibility, and Python syntax compilation of the patched upstream package. Additional tests use the actual loopback HTTP broker with a fake driver, reject unsafe archives and checksum mismatches, and verify that SDK tools run through Java arguments without a command shell. Patch tests require `.tmp/mobile-use-upstream` or `MOBILE_USE_UPSTREAM_SOURCE` pointing to the pinned checkout; they skip when that fixture is absent.
-A local smoke test executed `worker.main` profile construction, SDK initialization, a fake task, and cleanup using the actual pinned dependency environment in `.tmp/engine-smoke` and a fake controller. It made no model request and connected to no emulator.
+An earlier local smoke test executed `worker.main` profile construction, SDK initialization, a fake task, and cleanup using the actual pinned dependency environment in `.tmp/engine-smoke` and a fake controller. It made no model request and connected to no emulator.
+
+A separate Windows host check confirmed usable WHPX acceleration, a real Android boot, a 1080 × 2400 automation connection, and screenshot/UI inspection, followed by clean shutdown. That sequential check did not cover concurrent Workspace polling; dedicated regression tests now cover the startup race. The temporary smoke-test fixtures were removed after use.
-These tests do not validate a real emulator, Play Store login, APK execution, model inference, hardware acceleration, or browser video decoding. Validate those on a supported Windows host before treating this as production device automation. Current task results contain a response and run identity; the artifact list is empty, and hosted-style videos or reports are not generated.
+Automated tests also cover the Android tool’s schema and saved execution limits, privacy-filtered bounded diagnostic logging, setup interruption recovery, and connection-warning recovery in the UI. These checks do not establish successful Play Store login, APK execution, live model inference, or a complete browser-video acceptance run. Validate those on a supported Windows host before treating this as production device automation. Current task results contain a response and run identity; the artifact list is empty, and hosted-style videos or reports are not generated.
## Planned iOS implementation
diff --git a/server/nodes/mobile/__init__.py b/server/nodes/mobile/__init__.py
index c617b766..223f49e2 100644
--- a/server/nodes/mobile/__init__.py
+++ b/server/nodes/mobile/__init__.py
@@ -4,9 +4,10 @@ from services.ws_handler_registry import register_router
from services.plugin.shutdown_hooks import register_shutdown_hook
from services.node_output_schemas import register_output_schema
from ._node import MobileUseAgent, MobileOutput
+from ._tool import AndroidTool
from ._router import router
-__all__ = ["MobileUseAgent", "MobileOutput"]
+__all__ = ["MobileUseAgent", "MobileOutput", "AndroidTool"]
async def shutdown() -> None:
@@ -25,3 +26,4 @@ async def shutdown() -> None:
register_router(router, name="mobile")
register_shutdown_hook("mobile", shutdown)
register_output_schema("mobile_use_agent", MobileOutput)
+register_output_schema("android_tool", MobileOutput)
diff --git a/server/nodes/mobile/_diagnostics.py b/server/nodes/mobile/_diagnostics.py
new file mode 100644
index 00000000..d947f8ac
--- /dev/null
+++ b/server/nodes/mobile/_diagnostics.py
@@ -0,0 +1,45 @@
+"""Bounded operational logs. Never record prompts, screenshots, keys or input text."""
+
+import json
+import logging
+import sys
+import traceback
+from pathlib import Path
+from logging.handlers import RotatingFileHandler
+from collections import deque
+from datetime import datetime, timezone
+from ._paths import mobile_root
+
+_recent: deque = deque(maxlen=40)
+_logger = logging.Logger("opencompany.mobile.diagnostics")
+_logger.setLevel(logging.INFO)
+_logger.propagate = False
+_fields = {"operation", "duration_ms", "error_type", "code", "serial", "run_id", "node_id", "workflow_id", "state"}
+
+
+def event(name: str, *, failed: bool = False, **fields) -> None:
+ record = {"at": datetime.now(timezone.utc).isoformat(), "event": name,
+ "level": "error" if failed else "info", **{k: v for k, v in fields.items() if k in _fields}}
+ if failed and sys.exc_info()[2] is not None:
+ # Include code locations for diagnosis, never exception text or locals.
+ record["trace"] = [f"{Path(frame.filename).name}:{frame.lineno}:{frame.name}"
+ for frame in traceback.extract_tb(sys.exc_info()[2])[-12:]]
+ _recent.append(record)
+ try:
+ path = mobile_root() / "mobile.log"
+ if not _logger.handlers or _logger.handlers[0].baseFilename != str(path.resolve()):
+ for handler in list(_logger.handlers):
+ _logger.removeHandler(handler)
+ handler.close()
+ path.parent.mkdir(parents=True, exist_ok=True)
+ handler = RotatingFileHandler(path, maxBytes=1024 * 1024, backupCount=3, encoding="utf-8")
+ handler.setFormatter(logging.Formatter("%(message)s"))
+ _logger.addHandler(handler)
+ _logger.log(logging.ERROR if failed else logging.INFO, json.dumps(record))
+ except OSError:
+ # Diagnostics must never prevent phone operation (e.g. a full disk).
+ pass
+
+
+def recent() -> list[dict]:
+ return list(_recent)
diff --git a/server/nodes/mobile/_install.py b/server/nodes/mobile/_install.py
index f24e550f..ba637e1b 100644
--- a/server/nodes/mobile/_install.py
+++ b/server/nodes/mobile/_install.py
@@ -6,11 +6,14 @@ import hashlib
import json
import os
import platform
+import re
import shutil
import sys
+import time
import urllib.request
import zipfile
from pathlib import Path
+from typing import Callable
from ._paths import mobile_root, runtime_python
from ._process import command
@@ -20,6 +23,11 @@ SCRCPY_VERSION = "4.1"
SCRCPY_SHA256 = "deacb991ed2509715160ffdc7907e47b4160eb30d1566217e9047fd5b8850cae"
IMAGE = "system-images;android-36;google_apis_playstore;x86_64"
AVD_NAME = "OpenCompany"
+# Minimum required by the pinned API 36 Google Play image. Newer installed
+# image revisions can raise this through source.properties Pkg.Dependencies.
+MIN_EMULATOR_VERSION = (35, 4, 9)
+SDK_INSTALL_TIMEOUT = 2 * 60 * 60
+Progress = Callable[[str], None]
COMMAND_TOOLS_REVISION = "15859902"
COMMAND_TOOLS_SHA256 = "90ae805d20434428bffcb699c290860f19bb5f66a67e6b330067e3de801fb04a"
JAVA_URL = (
@@ -74,26 +82,93 @@ def sdk_command(name: str, *args: str) -> list[str]:
return [str(path), *args]
-def download(url: str, target: Path, digest: str, *, limit: int = 512 * 1024 * 1024) -> None:
+def download(url: str, target: Path, digest: str, *, limit: int = 512 * 1024 * 1024, progress: Progress | None = None) -> None:
+ if target.is_file() and hashlib.sha256(target.read_bytes()).hexdigest() == digest:
+ if progress:
+ progress(f"Using verified download: {target.name}")
+ return
target.parent.mkdir(parents=True, exist_ok=True)
temp = target.with_suffix(target.suffix + ".part")
hasher = hashlib.sha256()
try:
with urllib.request.urlopen(url, timeout=60) as response, temp.open("wb") as output:
size = 0
+ total = int(getattr(response, "headers", {}).get("Content-Length", 0) or 0)
+ last_update = 0.0
+ if progress:
+ progress(f"Downloading {target.name}")
while data := response.read(256 * 1024):
size += len(data)
if size > limit:
raise ValueError("Download exceeds expected size")
hasher.update(data)
output.write(data)
+ if progress and time.monotonic() - last_update >= 0.5:
+ suffix = f" / {total / 1024**2:.1f} MiB ({min(100, size * 100 // total)}%)" if total else " MiB"
+ progress(f"Downloading {target.name}: {size / 1024**2:.1f}{suffix}")
+ last_update = time.monotonic()
if hasher.hexdigest() != digest:
raise ValueError("Runtime download checksum mismatch")
temp.replace(target)
+ if progress:
+ progress(f"Verified {target.name} ({size / 1024**2:.1f} MiB)")
finally:
temp.unlink(missing_ok=True)
+async def _download(url: str, target: Path, digest: str, progress: Progress | None) -> None:
+ # urllib runs off-loop. All observable installer/runtime state updates
+ # still happen on the owning asyncio loop, never the downloader thread.
+ loop = asyncio.get_running_loop()
+ callback = (lambda message: loop.call_soon_threadsafe(progress, message)) if progress is not None else None
+ await asyncio.to_thread(download, url, target, digest, progress=callback)
+
+
+def _properties(path: Path) -> dict[str, str]:
+ try:
+ return {
+ key.strip(): value.strip()
+ for line in path.read_text(encoding="utf-8").splitlines()
+ if "=" in line and not line.lstrip().startswith("#")
+ for key, value in [line.split("=", 1)]
+ }
+ except (OSError, UnicodeError):
+ return {}
+
+
+def _version(value: str) -> tuple[int, ...]:
+ match = re.match(r"^(\d+)(?:\.(\d+))?(?:\.(\d+))?", value)
+ return tuple(int(part or 0) for part in match.groups()) if match else ()
+
+
+def sdk_package_ready(package: str) -> bool:
+ """Check core package files and revision; never update a complete package blindly."""
+ root = sdk_root() / Path(*package.split(";"))
+ props = _properties(root / "source.properties")
+ if not _version(props.get("Pkg.Revision", "")):
+ return False
+ if package == "platform-tools":
+ files = ("adb.exe", "AdbWinApi.dll", "AdbWinUsbApi.dll")
+ elif package == "emulator":
+ image = _properties(sdk_root() / Path(*IMAGE.split(";")) / "source.properties")
+ dependency = re.search(r"(?:^|[,;\s])emulator#([\d.]+)", image.get("Pkg.Dependencies", ""))
+ minimum = max(MIN_EMULATOR_VERSION, _version(dependency.group(1)) if dependency else MIN_EMULATOR_VERSION)
+ if _version(props["Pkg.Revision"]) < minimum:
+ return False
+ files = ("emulator.exe", "qemu/windows-x86_64/qemu-system-x86_64.exe")
+ elif package == IMAGE:
+ if (
+ props.get("AndroidVersion.ApiLevel") != "36"
+ or props.get("SystemImage.Abi") != "x86_64"
+ or props.get("SystemImage.TagId") != "google_apis_playstore"
+ ):
+ return False
+ files = ("system.img", "vendor.img", "ramdisk.img", "kernel-ranchu")
+ else:
+ raise ValueError("Unsupported managed Android package")
+ return all((root / file).is_file() and (root / file).stat().st_size > 0 for file in files)
+
+
def extract_zip(archive: Path, destination: Path) -> None:
destination = destination.resolve()
with zipfile.ZipFile(archive) as source:
@@ -106,7 +181,7 @@ def extract_zip(archive: Path, destination: Path) -> None:
source.extractall(destination)
-async def install_engine() -> None:
+async def install_engine(progress: Progress | None = None) -> None:
from .runtime.patch_source import patch
root = mobile_root()
@@ -121,14 +196,18 @@ async def install_engine() -> None:
ready = engine_ready() and marker.is_file()
server = root / "scrcpy-server"
if ready and server.is_file() and hashlib.sha256(server.read_bytes()).hexdigest() == SCRCPY_SHA256:
+ if progress:
+ progress("Mobile engine and video server are already installed")
return
uv = os.environ.get("OPENCOMPANY_UV_BIN") or shutil.which("uv")
if not uv:
raise ValueError("uv is required for optional mobile runtime setup")
if not marker.is_file():
archive = root / "downloads" / f"mobile-use-{UPSTREAM_SHA}.zip"
- await asyncio.to_thread(download, f"https://codeload.github.com/minitap-ai/mobile-use/zip/{UPSTREAM_SHA}", archive, SOURCE_SHA256)
+ await _download(f"https://codeload.github.com/minitap-ai/mobile-use/zip/{UPSTREAM_SHA}", archive, SOURCE_SHA256, progress)
staging = root / "source"
+ if progress:
+ progress("Extracting and preparing the mobile engine")
await asyncio.to_thread(extract_zip, archive, staging)
extracted = staging / f"mobile-use-{UPSTREAM_SHA}"
patch(extracted)
@@ -138,15 +217,22 @@ async def install_engine() -> None:
if not ready:
env = {**os.environ, "UV_PROJECT_ENVIRONMENT": str(source / ".venv"), "UV_NO_CONFIG": "1"}
env.pop("VIRTUAL_ENV", None)
- await command([uv, "sync", "--frozen", "--no-dev", "--python", "3.12", "--project", str(source)], timeout=900, env=env)
+ if progress:
+ progress("Installing the isolated Python runtime and mobile engine dependencies")
+ await command(
+ [uv, "sync", "--frozen", "--no-dev", "--python", "3.12", "--project", str(source)],
+ timeout=SDK_INSTALL_TIMEOUT,
+ env=env,
+ progress=progress,
+ )
stamp = root / "engine-ready.json.tmp"
stamp.write_text(json.dumps({"source": UPSTREAM_SHA, "patch": PATCH_VERSION}), encoding="utf-8")
stamp.replace(root / "engine-ready.json")
- await asyncio.to_thread(
- download,
+ await _download(
f"https://github.com/Genymobile/scrcpy/releases/download/v{SCRCPY_VERSION}/scrcpy-server-v{SCRCPY_VERSION}",
server,
SCRCPY_SHA256,
+ progress,
)
@@ -175,50 +261,78 @@ async def doctor() -> dict:
}
-async def install_sdk_tools() -> None:
+async def install_sdk_tools(progress: Progress | None = None) -> None:
root = mobile_root()
if not next((root / "java").glob("*/bin/java.exe"), None):
archive = root / "downloads" / "temurin-jre.zip"
- await asyncio.to_thread(download, JAVA_URL, archive, JAVA_SHA256)
+ await _download(JAVA_URL, archive, JAVA_SHA256, progress)
+ if progress:
+ progress("Extracting the private Java runtime")
await asyncio.to_thread(extract_zip, archive, root / "java")
- if not sdk_tool("sdkmanager"):
+ manager = sdk_tool("sdkmanager")
+ tools_ready = (
+ manager is not None
+ and _properties(manager.parent.parent / "source.properties")
+ and all((manager.parent.parent / "lib" / name).is_file() for name in ("sdkmanager-classpath.jar", "avdmanager-classpath.jar"))
+ )
+ if not tools_ready:
archive = root / "downloads" / f"commandlinetools-win-{COMMAND_TOOLS_REVISION}.zip"
- await asyncio.to_thread(
- download,
+ await _download(
f"https://dl.google.com/android/repository/commandlinetools-win-{COMMAND_TOOLS_REVISION}_latest.zip",
archive,
COMMAND_TOOLS_SHA256,
+ progress,
)
staging = root / "sdk-staging"
+ if progress:
+ progress("Extracting Android SDK command-line tools")
await asyncio.to_thread(extract_zip, archive, staging)
target = sdk_root() / "cmdline-tools" / COMMAND_TOOLS_REVISION
target.parent.mkdir(parents=True, exist_ok=True)
if target.exists():
raise ValueError("Incomplete SDK tools exist; remove that incomplete revision before retrying setup")
(staging / "cmdline-tools").rename(target)
+ elif progress:
+ progress("Android SDK command-line tools are already installed")
-async def create_device(*, licenses_accepted: bool) -> None:
+async def create_device(*, licenses_accepted: bool, progress: Progress | None = None) -> None:
if sys.platform != "win32" or platform.machine().lower() not in {"amd64", "x86_64"}:
raise ValueError("This release supports managed Android setup on Windows x64 only")
if not licenses_accepted:
raise ValueError("Accept the Android SDK license in setup before installing packages")
- await install_sdk_tools()
+ await install_sdk_tools(progress=progress)
args = [f"--sdk_root={sdk_root()}"]
# Explicit user acceptance above is the only path allowed to answer these prompts.
# Accept only the packages selected below, not every unrelated pending SDK
# license in a shared Android Studio installation.
- await command(sdk_command("sdkmanager", *args, "platform-tools", "emulator", IMAGE), timeout=1800, input_text="y\n" * 100)
+ # Install the image before checking emulator compatibility: its package
+ # metadata can demand a newer emulator than the baseline minimum.
+ for package in ("platform-tools", IMAGE, "emulator"):
+ if sdk_package_ready(package):
+ if progress:
+ progress(f"Already installed: {package}")
+ continue
+ if progress:
+ progress(f"Installing Android package: {package}. Large images may take a while on slower connections.")
+ await command(sdk_command("sdkmanager", *args, package), timeout=SDK_INSTALL_TIMEOUT, input_text="y\n" * 100, progress=progress)
+ if not sdk_package_ready(package):
+ raise ValueError(f"Android package {package} did not finish installing. Retry setup to repair the incomplete package.")
avd_home = mobile_root() / "avd"
avd_home.mkdir(parents=True, exist_ok=True)
env = {**os.environ, "ANDROID_AVD_HOME": str(avd_home), "ANDROID_SDK_ROOT": str(sdk_root())}
if not (avd_home / f"{AVD_NAME}.ini").is_file():
+ if progress:
+ progress("Creating the persistent OpenCompany Android device")
await command(
sdk_command("avdmanager", "create", "avd", "--name", AVD_NAME, "--package", IMAGE, "--device", "pixel_7"),
- timeout=120,
+ timeout=600,
env=env,
input_text="no\n",
+ progress=progress,
)
+ elif progress:
+ progress("Reusing the existing Android device, apps, and sign-ins")
(mobile_root() / "resource.json").write_text(
json.dumps(
{
@@ -232,3 +346,5 @@ async def create_device(*, licenses_accepted: bool) -> None:
),
encoding="utf-8",
)
+ if progress:
+ progress("Android setup complete. The device is ready to start.")
diff --git a/server/nodes/mobile/_node.py b/server/nodes/mobile/_node.py
index c9cd8ff6..bfb9962e 100644
--- a/server/nodes/mobile/_node.py
+++ b/server/nodes/mobile/_node.py
@@ -58,11 +58,11 @@ async def resolve_model(ctx: NodeContext) -> dict:
connections = [edge for edge in ctx.edges if edge.get("target") == ctx.node_id and edge_target_handle(edge) == "input-model"]
if len(connections) != 1:
- raise NodeUserError("Connect exactly one AI model to Mobile Agent's Model input")
+ raise NodeUserError("Connect exactly one AI model to this phone node's Model input")
source = next((node for node in ctx.nodes if node.get("id") == connections[0].get("source")), None)
cls = get_node_class(source.get("type", "")) if source else None
if cls is None or cls.component_kind != "model" or (source.get("data") or {}).get("disabled"):
- raise NodeUserError("Connect an enabled AI model node to Mobile Agent")
+ raise NodeUserError("Connect an enabled AI model node to this phone node")
parameters = await get_database().get_node_parameters(source["id"]) or {}
provider_ref = detect_ai_provider(source["type"], parameters)
adapters = {
diff --git a/server/nodes/mobile/_process.py b/server/nodes/mobile/_process.py
index 06888be1..2de5f300 100644
--- a/server/nodes/mobile/_process.py
+++ b/server/nodes/mobile/_process.py
@@ -2,8 +2,14 @@
from __future__ import annotations
import asyncio
+import codecs
+import contextlib
import os
+import re
import subprocess
+from typing import Callable
+
+OUTPUT_TAIL_BYTES = 64 * 1024
def hidden_options() -> dict:
@@ -18,6 +24,7 @@ async def command(
cwd: str | None = None,
input_text: str | None = None,
check: bool = True,
+ progress: Callable[[str], None] | None = None,
) -> str:
proc = await asyncio.create_subprocess_exec(
*map(str, argv),
@@ -28,14 +35,67 @@ async def command(
cwd=cwd,
**hidden_options(),
)
+ tail = bytearray()
+
+ async def stream() -> None:
+ decoder = codecs.getincrementaldecoder("utf-8")("replace")
+ pending = ""
+
+ async def write_input() -> None:
+ if input_text is not None:
+ try:
+ proc.stdin.write(input_text.encode())
+ await proc.stdin.drain()
+ except (BrokenPipeError, ConnectionResetError):
+ pass
+ finally:
+ proc.stdin.close()
+
+ async def read_output() -> None:
+ nonlocal pending
+ while data := await proc.stdout.read(4096):
+ tail.extend(data)
+ del tail[:-OUTPUT_TAIL_BYTES]
+ if progress is not None:
+ pending += decoder.decode(data)
+ lines = re.split(r"[\r\n]", pending)
+ pending = lines.pop()
+ for line in lines:
+ if line.strip():
+ progress(line[-4096:])
+ # Some installers print long status fragments with no
+ # newline. Bound both memory and time-to-visible-output.
+ if len(pending) >= 4096:
+ progress(pending[-4096:])
+ pending = ""
+ if progress is not None:
+ pending += decoder.decode(b"", final=True)
+ if pending.strip():
+ progress(pending[-4096:])
+
+ await asyncio.gather(write_input(), read_output())
+ await proc.wait()
+
try:
- data, _ = await asyncio.wait_for(proc.communicate(input_text.encode() if input_text is not None else None), timeout)
+ await asyncio.wait_for(stream(), timeout)
+ except asyncio.TimeoutError:
+ if proc.returncode is None:
+ with contextlib.suppress(ProcessLookupError):
+ proc.kill()
+ await proc.wait()
+ last_output = bytes(tail).decode("utf-8", "replace")[-1500:].strip()
+ raise TimeoutError(
+ f"{os.path.basename(str(argv[0]))} timed out after {timeout:g} seconds. "
+ "Check your network connection and available disk space, then retry setup; completed packages are reused."
+ + (f" Last output: {last_output}" if last_output else " No output was received.")
+ ) from None
except BaseException:
if proc.returncode is None:
- proc.kill()
+ with contextlib.suppress(ProcessLookupError):
+ proc.kill()
await proc.wait()
raise
- text = data.decode("utf-8", "replace")
+ text = bytes(tail).decode("utf-8", "replace")
if check and proc.returncode:
raise RuntimeError(f"{os.path.basename(str(argv[0]))} failed ({proc.returncode}): {text[-1500:]}")
return text
diff --git a/server/nodes/mobile/_router.py b/server/nodes/mobile/_router.py
index 690279e4..a509e936 100644
--- a/server/nodes/mobile/_router.py
+++ b/server/nodes/mobile/_router.py
@@ -94,7 +94,9 @@ async def status(principal: str = Depends(authorize)):
from ._runtime import get_runtime
runtime = get_runtime()
- if runtime.serial:
+ # During startup/shutdown return lifecycle status without touching the
+ # driver. The serial is assigned before Android and automation are ready.
+ if runtime.serial and not runtime.lifecycle_lock.locked():
await call(runtime.driver_call("geometry"))
return runtime.snapshot()
diff --git a/server/nodes/mobile/_runtime.py b/server/nodes/mobile/_runtime.py
index d7d926cf..40ce77f7 100644
--- a/server/nodes/mobile/_runtime.py
+++ b/server/nodes/mobile/_runtime.py
@@ -13,6 +13,7 @@ from ._control import DeviceControl, Lease, MobileError
from ._install import AVD_NAME, install_engine, create_device, engine_ready, sdk_tool, sdk_root
from ._paths import mobile_root, runtime_python
from ._process import command, hidden_options
+from ._diagnostics import event, recent
READS = frozenset({"geometry", "observe", "date", "packages", "foreground"})
ACTIONS = frozenset({"tap", "swipe", "touch", "text", "erase", "key", "launch", "terminate", "url", "rotate"})
@@ -32,6 +33,9 @@ class MobileRuntime:
self.setup_task: asyncio.Task | None = None
self.setup_state = "idle"
self.setup_error = ""
+ self.setup_progress: dict = {}
+ self._setup_loaded = False
+ self._setup_saved = 0.0
self.active: dict | None = None
self.queue: list[dict] = []
self.capabilities: dict[str, tuple[str, Lease]] = {}
@@ -98,18 +102,77 @@ class MobileRuntime:
return self.broker_url
def snapshot(self) -> dict:
+ self._load_setup()
return {
- "running": self.process is not None and self.process.returncode is None,
+ "running": self.process is not None and self.process.returncode is None
+ and self.driver is not None and self.driver.returncode is None and self.geometry is not None,
+ "starting": self.lifecycle_lock.locked() and self.geometry is None and not self.stopping,
"serial": self.serial,
"geometry": self.geometry,
"device": "Shared Android device",
"setup": self.setup_state,
"setup_error": self.setup_error,
+ "setup_progress": self.setup_progress,
+ "diagnostics": recent(),
"active": {k: v for k, v in (self.active or {}).items() if k not in {"prompt"}},
"queue": [{k: v for k, v in q.items() if k != "prompt"} for q in self.queue],
**self.control.snapshot(),
}
+ def _load_setup(self) -> None:
+ if self._setup_loaded:
+ return
+ self._setup_loaded = True
+ try:
+ record = json.loads((mobile_root() / "setup-status.json").read_text(encoding="utf-8"))
+ if not isinstance(record, dict):
+ return
+ if record.get("state") not in {"installing_engine", "installing_device", "ready", "error", "interrupted"}:
+ return
+ if not isinstance(record.get("progress"), dict):
+ return
+ progress = record["progress"]
+ events = progress.get("events", [])
+ if not isinstance(events, list) or any(
+ not isinstance(event, dict) or not isinstance(event.get("message"), str)
+ or not isinstance(event.get("at"), (int, float)) for event in events
+ ):
+ return
+ progress["events"] = events[-80:]
+ self.setup_state = record["state"]
+ self.setup_error = str(record.get("error", ""))[:1500]
+ self.setup_progress = record["progress"]
+ if self.setup_state.startswith("installing_"):
+ self.setup_state = "interrupted"
+ self.setup_error = "Setup was interrupted by a backend restart. Retry setup to reuse completed components."
+ self.setup_progress["finished_at"] = time.time()
+ self._setup_update(self.setup_error, force=True)
+ except (OSError, ValueError, TypeError):
+ pass
+
+ def _setup_update(self, message: str, *, force: bool = False) -> None:
+ now = time.time()
+ message = message.strip()[-1500:]
+ if not message:
+ return
+ self.setup_progress.update(message=message, updated_at=now)
+ events = self.setup_progress.setdefault("events", [])
+ if not events or events[-1]["message"] != message:
+ events.append({"at": now, "message": message})
+ del events[:-80]
+ if not force and time.monotonic() - self._setup_saved < 1:
+ return
+ try:
+ root = mobile_root()
+ root.mkdir(parents=True, exist_ok=True)
+ temp = root / "setup-status.json.tmp"
+ temp.write_text(json.dumps({"state": self.setup_state, "error": self.setup_error, "progress": self.setup_progress}), encoding="utf-8")
+ temp.replace(root / "setup-status.json")
+ self._setup_saved = time.monotonic()
+ except OSError:
+ # Disk failures must not erase the live status or abort installation.
+ pass
+
def setup(self, accepted: bool) -> None:
if not accepted:
raise MobileError("license_required", "Read and accept the SDK license before setup")
@@ -118,16 +181,35 @@ class MobileRuntime:
if self.stopping or self.lifecycle_lock.locked() or self.active or self.serial:
raise MobileError("device_busy", "Stop the device before changing its runtime")
+ self._setup_loaded = True
+ self.setup_error = ""
+ self.setup_state = "installing_engine"
+ self.setup_progress = {"started_at": time.time(), "events": []}
+ self._setup_update("Checking the mobile agent runtime", force=True)
+ event("setup_started")
+
async def run():
- self.setup_error = ""
try:
- self.setup_state = "installing_engine"
- await install_engine()
+ await install_engine(progress=self._setup_update)
self.setup_state = "installing_device"
- await create_device(licenses_accepted=True)
+ self._setup_update("Preparing Android tools and the shared phone", force=True)
+ await create_device(licenses_accepted=True, progress=self._setup_update)
self.setup_state = "ready"
+ self.setup_progress["finished_at"] = time.time()
+ self._setup_update("Setup complete. Start the phone to connect it.", force=True)
+ event("setup_completed")
+ except asyncio.CancelledError:
+ self.setup_state = "interrupted"
+ self.setup_error = "Setup stopped because the backend shut down. Retry setup to reuse completed components."
+ self.setup_progress["finished_at"] = time.time()
+ self._setup_update(self.setup_error, force=True)
+ event("setup_interrupted", failed=True)
+ raise
except Exception as exc:
- self.setup_state, self.setup_error = "error", str(exc)[-1500:]
+ self.setup_state, self.setup_error = "error", str(exc)[-1500:] or f"Setup failed ({type(exc).__name__}). Retry setup."
+ self.setup_progress["finished_at"] = time.time()
+ self._setup_update(self.setup_error, force=True)
+ event("setup_failed", failed=True, error_type=type(exc).__name__)
self.setup_task = asyncio.create_task(run())
@@ -162,6 +244,7 @@ class MobileRuntime:
if port is None:
raise MobileError("ports_busy", "No emulator port pair is free")
root = mobile_root()
+ event("phone_starting")
root.mkdir(parents=True, exist_ok=True)
log = (root / "emulator.log").open("ab")
env = {**os.environ, "ANDROID_AVD_HOME": str(root / "avd"), "ANDROID_SDK_ROOT": str(sdk_root())}
@@ -194,9 +277,11 @@ class MobileRuntime:
else:
raise MobileError("boot_timeout", "Android did not finish booting within three minutes")
await self._start_driver()
+ event("phone_ready", serial=self.serial)
self.control.revoke()
self.control.state = "idle"
- except BaseException:
+ except BaseException as exc:
+ event("phone_start_failed", failed=True, error_type=type(exc).__name__, code=getattr(exc, "code", "startup_failed"))
await self._stop_owned()
raise
return self.snapshot()
@@ -240,6 +325,18 @@ class MobileRuntime:
return env
async def _start_driver(self):
+ event("driver_starting")
+ # Initialization consumes the first protocol reply. It must own the
+ # same lock as commands so polling cannot read that reply concurrently.
+ async with self.driver_lock:
+ try:
+ await self._initialize_driver()
+ event("driver_ready")
+ except BaseException:
+ await self._stop_driver()
+ raise
+
+ async def _initialize_driver(self):
entry = Path(__file__).with_name("runtime") / "device_server.py"
self.driver = await asyncio.create_subprocess_exec(
str(runtime_python()),
@@ -267,19 +364,24 @@ class MobileRuntime:
async with self.driver_lock:
if not self.driver or self.driver.returncode is not None:
raise MobileError("device_offline", "Start the device before using it")
- self.driver.stdin.write((json.dumps({"operation": operation, "parameters": parameters or {}}) + "\n").encode())
- await self.driver.stdin.drain()
+ started = time.monotonic()
try:
+ self.driver.stdin.write((json.dumps({"operation": operation, "parameters": parameters or {}}) + "\n").encode())
+ await self.driver.stdin.drain()
line = await asyncio.wait_for(self.driver.stdout.readline(), 30)
if not line:
raise RuntimeError("Device driver disconnected")
result = json.loads(line)
- except BaseException:
+ except BaseException as exc:
+ event("driver_connection_failed", failed=True, operation=operation, error_type=type(exc).__name__)
self.control.state = "recovering"
await self._stop_driver()
raise
if not result.get("success"):
+ event("device_action_failed", failed=True, operation=operation, code=result.get("code", "device_action_failed"))
raise MobileError(result.get("code", "device_action_failed"), result.get("error", "Device command failed"))
+ if operation not in READS:
+ event("device_action_completed", operation=operation, duration_ms=round((time.monotonic() - started) * 1000))
if operation == "geometry":
self.geometry = result["result"]
elif operation == "observe":
@@ -318,6 +420,7 @@ class MobileRuntime:
await self._stop_worker()
lease = await self.control.claim("viewer:" + viewer, takeover=True)
self.viewer = viewer
+ event("manual_control_started")
return lease
def release(self, viewer: str, *, resume: bool = False):
@@ -349,6 +452,7 @@ class MobileRuntime:
raise MobileError("duplicate_run", "This task is already admitted")
entry = {"run_id": run_id, "workflow_id": workflow_id, "node_id": node_id, "status": "queued"}
self.queue.append(entry)
+ event("task_queued", run_id=run_id, workflow_id=workflow_id, node_id=node_id)
try:
async with self.task_lock:
self.queue.remove(entry)
@@ -386,6 +490,7 @@ class MobileRuntime:
entry["status"] = "awaiting_user"
continue
entry["status"] = "running"
+ event("task_running", run_id=run_id)
config = {
**model,
"source": str(mobile_root() / "engine"),
@@ -428,6 +533,7 @@ class MobileRuntime:
continue
if not result.get("success"):
raise MobileError("agent_failed", result.get("error", "Mobile agent failed"))
+ event("task_completed", run_id=run_id)
return {"response": result.get("result"), "outcome": "completed", "run_id": run_id, "artifacts": []}
finally:
self.capabilities.clear()
@@ -437,6 +543,12 @@ class MobileRuntime:
async with self.control.lock:
pass
self.active = None
+ except asyncio.CancelledError:
+ event("task_cancelled", run_id=run_id)
+ raise
+ except Exception as exc:
+ event("task_failed", failed=True, run_id=run_id, error_type=type(exc).__name__, code=getattr(exc, "code", "task_failed"))
+ raise
finally:
if entry in self.queue:
self.queue.remove(entry)
@@ -498,6 +610,7 @@ class MobileRuntime:
await proc.wait()
async def _stop_owned(self):
+ event("phone_stopping", serial=self.serial)
self.control.revoke()
self.capabilities.clear()
await self._stop_worker()
@@ -518,6 +631,7 @@ class MobileRuntime:
self.control.results.clear()
self.viewer = None
self.resume_event.clear()
+ event("phone_stopped")
async def stop(self):
if self.active:
diff --git a/server/nodes/mobile/_tool.py b/server/nodes/mobile/_tool.py
new file mode 100644
index 00000000..90fdeb7f
--- /dev/null
+++ b/server/nodes/mobile/_tool.py
@@ -0,0 +1,48 @@
+"""Natural-language Android tasks exposed through the standard agent tool port."""
+
+from pydantic import BaseModel, ConfigDict, Field
+from services.plugin import NodeContext, Operation, ToolNode
+from ._node import MobileUseAgent, MobileParams, MobileOutput
+
+
+class AndroidTaskInput(BaseModel):
+ model_config = ConfigDict(extra="forbid")
+ prompt: str = Field(min_length=1, max_length=20000, description="What to do on the Android phone, in plain language.")
+
+
+class AndroidTool(ToolNode):
+ type = "android_tool"
+ display_name = "Android"
+ description = "Let an AI agent carry out a task on your Android phone."
+ group = ("tool",)
+ tool_name = "android"
+ tool_description = (
+ "Carry out a task on the shared Android phone. Supply a clear plain-language prompt. "
+ "The phone must be set up and started in Workspace > Mobile. Tasks share the phone and wait "
+ "when a person is using it. Do not request passwords in chat; let the person sign in on the phone."
+ )
+ tool_schema_locked = True
+ requires_context = True
+ needs_canvas = True
+ workspace_task = True
+ ui_hints = {"workspace": {"kind": "mobile"}}
+ handles = (
+ {"name": "input-model", "kind": "input", "position": "bottom", "label": "Model", "role": "model"},
+ {"name": "output-tool", "kind": "output", "position": "top", "label": "Tool", "role": "tools"},
+ )
+ annotations = {"destructive": True, "readonly": False, "open_world": True}
+ task_queue = MobileUseAgent.task_queue
+ retry_policy = MobileUseAgent.retry_policy
+ start_to_close_timeout = MobileUseAgent.start_to_close_timeout
+ Params = MobileParams
+ ToolInput = AndroidTaskInput
+ Output = MobileOutput
+
+ @Operation("execute")
+ async def execute_op(self, ctx: NodeContext, params: AndroidTaskInput | MobileParams) -> dict:
+ config = ctx.raw.get("_tool_config")
+ if isinstance(config, MobileParams):
+ params = config.model_copy(update={"prompt": params.prompt})
+ elif not isinstance(params, MobileParams):
+ params = MobileParams(prompt=params.prompt)
+ return await MobileUseAgent().execute_op(ctx, params)
diff --git a/server/nodes/mobile/_video.py b/server/nodes/mobile/_video.py
index 1fdda856..55a8e90e 100644
--- a/server/nodes/mobile/_video.py
+++ b/server/nodes/mobile/_video.py
@@ -55,6 +55,7 @@ def server_arguments(serial: str, scid: str) -> list[str]:
async def stream_video(websocket: WebSocket, viewer: str) -> None:
+ from ._diagnostics import event
from ._runtime import get_runtime
runtime = get_runtime()
@@ -110,6 +111,7 @@ async def stream_video(websocket: WebSocket, viewer: str) -> None:
if reader is None:
raise MobileError("video_failed", "The device video stream did not start")
await websocket.accept()
+ event("video_connected", serial=serial)
await websocket.send_json({"type": "video", "version": SCRCPY_VERSION, "codec": "h264"})
async def relay():
@@ -134,6 +136,8 @@ async def stream_video(websocket: WebSocket, viewer: str) -> None:
for job in done:
job.result()
except (MobileError, OSError, ValueError, asyncio.TimeoutError, WebSocketDisconnect) as exc:
+ event("video_disconnected" if isinstance(exc, WebSocketDisconnect) else "video_failed",
+ failed=not isinstance(exc, WebSocketDisconnect), error_type=type(exc).__name__)
with contextlib.suppress(Exception):
await websocket.close(code=1011, reason=str(exc)[:110])
finally:
@@ -159,3 +163,4 @@ async def stream_video(websocket: WebSocket, viewer: str) -> None:
# its exact epoch through the broker; an old stream must never revoke
# a newer lease held by the same tab.
_streams.discard(current)
+ event("video_closed")
diff --git a/server/tests/test_android_tool.py b/server/tests/test_android_tool.py
new file mode 100644
index 00000000..9767b217
--- /dev/null
+++ b/server/tests/test_android_tool.py
@@ -0,0 +1,58 @@
+"""Android uses the standard tool schema and the same controlled runtime."""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+import json
+import pytest
+from pydantic import ValidationError
+from services.plugin import NodeContext, ToolNode
+from nodes.mobile._tool import AndroidTool, AndroidTaskInput
+from nodes.mobile import _node, _runtime, _diagnostics
+
+
+def test_android_is_discoverable_as_a_tool():
+ from services.node_registry import get_node_class
+ assert get_node_class("android_tool") is AndroidTool
+ assert issubclass(AndroidTool, ToolNode)
+ assert AndroidTool.component_kind == "tool"
+ assert set(AndroidTool.ToolInput.model_fields) == {"prompt"}
+ assert any(handle["role"] == "tools" for handle in AndroidTool.handles)
+ with pytest.raises(ValidationError):
+ AndroidTaskInput(prompt="Open settings", workflow_id="other")
+
+
+async def test_tool_uses_saved_limits_and_existing_mobile_pipeline(monkeypatch):
+ runtime = SimpleNamespace(run=AsyncMock(return_value={"response": "Done", "run_id": "test", "outcome": "completed"}),
+ ensure_broker=AsyncMock(return_value="http://127.0.0.1/private"))
+ monkeypatch.setattr(_runtime, "get_runtime", lambda: runtime)
+ owner = AsyncMock()
+ model = AsyncMock(return_value={"provider": "openai"})
+ monkeypatch.setattr(_node, "require_mobile_owner", owner)
+ monkeypatch.setattr(_node, "resolve_model", model)
+ ctx = NodeContext(node_id="phone-tool", node_type="android_tool", workflow_id="wf", execution_id="run", user_id="owner")
+ result = await AndroidTool().execute_as_tool({"prompt": "Open Settings"}, {"max_steps": 7, "timeout_s": 60}, ctx)
+ assert result.get("error") is None, result
+ args = runtime.run.await_args.kwargs
+ assert args["params"] == {"prompt": "Open Settings", "max_steps": 7, "timeout_s": 60}
+ assert args["node_id"] == "phone-tool"
+ owner.assert_awaited_once_with("owner")
+ model.assert_awaited_once_with(ctx)
+
+
+def test_diagnostics_are_bounded_and_exclude_content(monkeypatch, tmp_path):
+ monkeypatch.setattr(_diagnostics, "mobile_root", lambda: tmp_path)
+ try:
+ for index in range(45):
+ _diagnostics.event("device_action_completed", operation="text", duration_ms=index,
+ prompt="secret prompt", parameters={"text": "password"}, api_key="secret-key")
+ records = [json.loads(line) for line in (tmp_path / "mobile.log").read_text().splitlines()]
+ assert len(records) == 45
+ assert len(_diagnostics.recent()) == 40
+ assert "secret" not in json.dumps(records)
+ assert "password" not in json.dumps(records)
+ assert records[-1]["duration_ms"] == 44
+ finally:
+ for handler in list(_diagnostics._logger.handlers):
+ handler.close()
+ _diagnostics._logger.removeHandler(handler)
+ _diagnostics._recent.clear()
diff --git a/server/tests/test_mobile_install.py b/server/tests/test_mobile_install.py
index cbf5cd9d..6c5fd772 100644
--- a/server/tests/test_mobile_install.py
+++ b/server/tests/test_mobile_install.py
@@ -81,3 +81,99 @@ def test_incomplete_environment_is_not_ready(monkeypatch, tmp_path):
assert _install.engine_ready()
(tmp_path / "engine-ready.json").write_text(json.dumps({"source": "old", "patch": PATCH_VERSION}))
assert not _install.engine_ready()
+
+
+def _package(root, package, revision="37.1.11", dependency="35.4.9"):
+ path = root.joinpath(*package.split(";"))
+ path.mkdir(parents=True, exist_ok=True)
+ props = f"Pkg.Revision={revision}\n"
+ if package == install.IMAGE:
+ props += f"AndroidVersion.ApiLevel=36\nSystemImage.Abi=x86_64\nSystemImage.TagId=google_apis_playstore\nPkg.Dependencies=emulator#{dependency}\n"
+ files = ("system.img", "vendor.img", "ramdisk.img", "kernel-ranchu")
+ elif package == "emulator":
+ files = ("emulator.exe", "qemu/windows-x86_64/qemu-system-x86_64.exe")
+ else:
+ files = ("adb.exe", "AdbWinApi.dll", "AdbWinUsbApi.dll")
+ (path / "source.properties").write_text(props)
+ for name in files:
+ target = path / name
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_bytes(b"installed")
+ return path
+
+
+def test_complete_packages_are_reused_but_incomplete_files_are_not(monkeypatch, tmp_path):
+ monkeypatch.setattr(install, "sdk_root", lambda: tmp_path)
+ for package in ("platform-tools", install.IMAGE, "emulator"):
+ _package(tmp_path, package)
+ assert install.sdk_package_ready(package)
+ (tmp_path / "platform-tools" / "AdbWinApi.dll").unlink()
+ assert not install.sdk_package_ready("platform-tools")
+ image = tmp_path.joinpath(*install.IMAGE.split(";"))
+ (image / "system.img").write_bytes(b"")
+ assert not install.sdk_package_ready(install.IMAGE)
+
+
+def test_emulator_minimum_tracks_installed_image_dependency(monkeypatch, tmp_path):
+ monkeypatch.setattr(install, "sdk_root", lambda: tmp_path)
+ _package(tmp_path, "emulator", revision="35.4.8")
+ assert not install.sdk_package_ready("emulator")
+ _package(tmp_path, "emulator", revision="37.1.11")
+ assert install.sdk_package_ready("emulator")
+ _package(tmp_path, install.IMAGE, dependency="38.0.1")
+ assert not install.sdk_package_ready("emulator")
+
+
[email protected]("missing", [None, install.IMAGE])
+async def test_setup_only_installs_missing_packages_and_preserves_avd(monkeypatch, tmp_path, missing):
+ monkeypatch.setattr(install.sys, "platform", "win32")
+ monkeypatch.setattr(install.platform, "machine", lambda: "AMD64")
+ monkeypatch.setattr(install, "mobile_root", lambda: tmp_path)
+ monkeypatch.setattr(install, "sdk_root", lambda: tmp_path / "sdk")
+ tools = AsyncMock()
+ monkeypatch.setattr(install, "install_sdk_tools", tools)
+ ready = {p for p in ("platform-tools", install.IMAGE, "emulator") if p != missing}
+ monkeypatch.setattr(install, "sdk_package_ready", lambda p: p in ready)
+ monkeypatch.setattr(install, "sdk_command", lambda name, *args: [name, *args])
+ avd = tmp_path / "avd"
+ avd.mkdir()
+ (avd / f"{install.AVD_NAME}.ini").write_text("persistent-device")
+
+ async def execute(argv, **kwargs):
+ assert argv[0] == "sdkmanager"
+ assert kwargs["timeout"] == 7200
+ ready.add(argv[-1])
+ kwargs["progress"]("Installing 50%")
+
+ command = AsyncMock(side_effect=execute)
+ monkeypatch.setattr(install, "command", command)
+ events = []
+ await install.create_device(licenses_accepted=True, progress=events.append)
+ assert command.await_count == int(missing is not None)
+ if missing:
+ assert command.await_args.args[0][-1] == missing
+ assert "Installing 50%" in events
+ assert (avd / f"{install.AVD_NAME}.ini").read_text() == "persistent-device"
+ assert events[-1].startswith("Android setup complete")
+ tools.assert_awaited_once_with(progress=events.append)
+
+
+async def test_download_progress_callback_runs_on_event_loop_thread(monkeypatch, tmp_path):
+ import asyncio
+ import threading
+
+ loop_thread = threading.get_ident()
+ worker_threads = []
+ observed = []
+
+ def download(*_, progress, **kwargs):
+ worker_threads.append(threading.get_ident())
+ progress("Downloading 50%")
+
+ monkeypatch.setattr(install, "download", download)
+ await install._download(
+ "https://example.invalid", tmp_path / "artifact", "digest", lambda message: observed.append((threading.get_ident(), message))
+ )
+ await asyncio.sleep(0)
+ assert worker_threads[0] != loop_thread
+ assert observed == [(loop_thread, "Downloading 50%")]
diff --git a/server/tests/test_mobile_process.py b/server/tests/test_mobile_process.py
new file mode 100644
index 00000000..70d3e499
--- /dev/null
+++ b/server/tests/test_mobile_process.py
@@ -0,0 +1,87 @@
+"""Streaming installer subprocess contracts, without launching an installer."""
+
+import asyncio
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from nodes.mobile import _process as module
+
+
+class Process:
+ def __init__(self):
+ self.stdout = asyncio.StreamReader()
+ self.stdin = SimpleNamespace(write=lambda _: None, drain=AsyncMock(), close=lambda: None)
+ self.returncode = None
+ self.done = asyncio.Event()
+ self.killed = False
+
+ async def wait(self):
+ await self.done.wait()
+ self.returncode = 0
+ return 0
+
+ def kill(self):
+ self.killed = True
+ self.stdout.feed_eof()
+ self.done.set()
+
+
+async def test_reports_carriage_return_progress_before_process_exits(monkeypatch):
+ process = Process()
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", AsyncMock(return_value=process))
+ events = []
+ received = asyncio.Event()
+
+ def progress(line):
+ events.append(line)
+ received.set()
+
+ task = asyncio.create_task(module.command(["sdkmanager"], progress=progress))
+ process.stdout.feed_data(b"Downloading 10%\rDownloading 20%\r\n")
+ await asyncio.wait_for(received.wait(), 1)
+ assert not task.done()
+ process.stdout.feed_data("Extracting café".encode()[:-1])
+ await asyncio.sleep(0)
+ process.stdout.feed_data("Extracting café".encode()[-1:])
+ process.stdout.feed_eof()
+ process.done.set()
+ output = await task
+ assert events == ["Downloading 10%", "Downloading 20%", "Extracting café"]
+ assert "Downloading 10%" in output
+
+
+async def test_output_and_unterminated_progress_are_bounded(monkeypatch):
+ process = Process()
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", AsyncMock(return_value=process))
+ process.stdout.feed_data(b"old-output\n" + b"x" * (module.OUTPUT_TAIL_BYTES * 3) + b"last-output")
+ process.stdout.feed_eof()
+ process.done.set()
+ lines = []
+ output = await module.command(["sdkmanager"], progress=lines.append)
+ assert len(output.encode()) == module.OUTPUT_TAIL_BYTES
+ assert output.endswith("last-output") and "old-output" not in output
+ assert max(map(len, lines)) <= 4096
+
+
+async def test_timeout_kills_waits_and_explains_last_progress(monkeypatch):
+ process = Process()
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", AsyncMock(return_value=process))
+ process.stdout.feed_data(b"Downloading image: 42%\r")
+ with pytest.raises(TimeoutError, match="timed out after.*retry setup.*42%"):
+ await module.command(["sdkmanager"], timeout=0.02)
+ assert process.killed and process.returncode == 0
+
+
+async def test_cancelled_command_reaps_process(monkeypatch):
+ process = Process()
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", AsyncMock(return_value=process))
+ entered = asyncio.Event()
+ task = asyncio.create_task(module.command(["sdkmanager"], progress=lambda _: entered.set()))
+ process.stdout.feed_data(b"working\r")
+ await entered.wait()
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert process.killed and process.returncode == 0
diff --git a/server/tests/test_mobile_runtime.py b/server/tests/test_mobile_runtime.py
index 6ff11118..5827beec 100644
--- a/server/tests/test_mobile_runtime.py
+++ b/server/tests/test_mobile_runtime.py
@@ -8,6 +8,58 @@ import pytest
from nodes.mobile._runtime import MobileRuntime
+async def test_driver_initialization_serializes_concurrent_command(monkeypatch, tmp_path):
+ import nodes.mobile._runtime as module
+
+ value = MobileRuntime()
+ value.serial = "emulator-test"
+ proc = FakeProcess()
+ waiting = asyncio.Event()
+ original_readline = proc.stdout.readline
+
+ async def readline():
+ waiting.set()
+ return await original_readline()
+
+ async def spawn(*_args, **_kwargs):
+ return proc
+
+ monkeypatch.setattr(module, "mobile_root", lambda: tmp_path)
+ monkeypatch.setattr(value, "driver_env", lambda: {})
+ monkeypatch.setattr(proc.stdout, "readline", readline)
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)
+ starting = asyncio.create_task(value._start_driver())
+ await waiting.wait()
+ command = asyncio.create_task(value.driver_call("geometry"))
+ await asyncio.sleep(0)
+ assert not command.done()
+ assert b'"operation"' not in proc.stdin.data
+ proc.stdout.feed_data(b'{"success":true,"result":{"width":1080,"height":2400}}\n')
+ await starting
+ proc.stdout.feed_data(b'{"success":true,"result":{"width":2400,"height":1080}}\n')
+ assert await command == {"width": 2400, "height": 1080}
+ assert not proc.terminated.is_set()
+
+
+async def test_status_during_startup_does_not_query_driver(monkeypatch, tmp_path):
+ from unittest.mock import AsyncMock
+ import nodes.mobile._runtime as module
+ from nodes.mobile._router import status
+
+ value = MobileRuntime()
+ value.serial = "emulator-test"
+ value.process = FakeProcess()
+ monkeypatch.setattr(module, "mobile_root", lambda: tmp_path)
+ monkeypatch.setattr(module, "get_runtime", lambda: value)
+ query = AsyncMock()
+ monkeypatch.setattr(value, "driver_call", query)
+ async with value.lifecycle_lock:
+ result = await status(principal="owner")
+ assert result["starting"] is True
+ assert result["running"] is False
+ query.assert_not_awaited()
+
+
class FakeStdin:
def __init__(self):
self.data = b""
diff --git a/server/tests/test_mobile_setup_progress.py b/server/tests/test_mobile_setup_progress.py
new file mode 100644
index 00000000..4c8183c6
--- /dev/null
+++ b/server/tests/test_mobile_setup_progress.py
@@ -0,0 +1,75 @@
+"""Setup survives page reloads and reports backend interruption explicitly."""
+
+import asyncio
+import json
+from unittest.mock import AsyncMock
+
+import pytest
+
+from nodes.mobile import _runtime as module
+
+
+def isolated(monkeypatch, tmp_path):
+ monkeypatch.setattr(module, "mobile_root", lambda: tmp_path)
+ return tmp_path
+
+
+async def test_success_has_durable_progress_and_reuses_state(monkeypatch, isolated):
+ async def engine(*, progress):
+ progress("Downloading engine: 50%")
+
+ monkeypatch.setattr(module, "install_engine", engine)
+ monkeypatch.setattr(module, "create_device", AsyncMock())
+ runtime = module.MobileRuntime()
+ runtime.setup(True)
+ assert runtime.snapshot()["setup"] == "installing_engine"
+ await runtime.setup_task
+ restored = module.MobileRuntime().snapshot()
+ assert restored["setup"] == "ready"
+ assert restored["setup_progress"]["finished_at"] >= restored["setup_progress"]["started_at"]
+ assert any("50%" in event["message"] for event in restored["setup_progress"]["events"])
+
+
+async def test_empty_timeout_still_has_visible_error(monkeypatch, isolated):
+ monkeypatch.setattr(module, "install_engine", AsyncMock(side_effect=TimeoutError()))
+ runtime = module.MobileRuntime()
+ runtime.setup(True)
+ await runtime.setup_task
+ assert runtime.snapshot()["setup"] == "error"
+ assert "TimeoutError" in runtime.snapshot()["setup_error"]
+
+
+async def test_shutdown_records_interruption(monkeypatch, isolated):
+ started = asyncio.Event()
+
+ async def engine(**_kwargs):
+ started.set()
+ await asyncio.Event().wait()
+
+ monkeypatch.setattr(module, "install_engine", engine)
+ runtime = module.MobileRuntime()
+ runtime.setup(True)
+ await started.wait()
+ await runtime.shutdown()
+ restored = module.MobileRuntime().snapshot()
+ assert restored["setup"] == "interrupted"
+ assert "Retry setup" in restored["setup_error"]
+
+
+def test_abrupt_exit_detected_on_next_status(isolated):
+ (isolated / "setup-status.json").write_text(json.dumps({
+ "state": "installing_device", "error": "", "progress": {"started_at": 1, "events": []},
+ }))
+ status = module.MobileRuntime().snapshot()
+ assert status["setup"] == "interrupted"
+ assert "backend restart" in status["setup_error"]
+ assert json.loads((isolated / "setup-status.json").read_text())["state"] == "interrupted"
+
+
+def test_history_is_bounded(isolated):
+ runtime = module.MobileRuntime()
+ for index in range(100):
+ runtime._setup_update(f"Progress {index}")
+ assert len(runtime.setup_progress["events"]) == 80
+ assert runtime.setup_progress["events"][-1]["message"] == "Progress 99"