Skip to content

Doubao-powered voice input for Linux, Wayland and Omarchy

10.3 KB
Raw Blame

Midscene desktop E2E

The native macOS suite is documented in macOS Midscene settings E2E. Run it with npm run test:macos -- --project macos-settings; it uses a separate config and isolated native fixture, not the Linux desktop setup.

GitHub Actions exposes two Linux distribution-named suites: Ubuntu 22.04 and Omarchy 4.0.3. The Omarchy suite also checks the real desktop shell visually.

Each platform runs in one Actions job. Product cases and the platform's auxiliary project run serially, without matrix shards. Ubuntu remains manual-only; Omarchy runs on relevant pushes and same-repository pull requests. The shared Linux concurrency group queues workflows that use the same model quota. The job budget is 240 minutes, with 220 minutes reserved for visual execution.

The image-builder CI proves that a GitHub-hosted runner can install the exact official Omarchy ISO in a headless QEMU/KVM VM. It reuses Omarchy's own ISO acceptance harness, saves installer evidence as short-lived artifacts, and publishes the resulting test-only VM bundle to this repository's private GHCR namespace.

The full Omarchy install is intentionally not triggered by ordinary pushes. Build Omarchy VM image installs the verified official ISO only when its version, checksum, harness, or rebuild marker changes, then publishes the installed base disk as a private, versioned OCI artifact in GHCR. The Omarchy E2E restores that base and creates a throwaway overlay, so normal manual runs no longer download or install the 6 GB ISO.

The GHCR artifact also contains the harness-created UEFI state and SSH key. That key is valid only for the disposable test VM, whose SSH port is bound to the runner's loopback interface; it is never used for GitHub or production.

Omarchy 4.0.3 was installed successfully on a GitHub-hosted runner in Actions run 34866563731. The run took 13 minutes 41 seconds, including about 55 seconds to transfer the ISO.

The versioned 4.0.3 base bundle is 3.8 GB and was published successfully in Actions run 34926824941. The first restore-backed end-to-end run then passed in Actions run 34927901882: 3 minutes 55 seconds to prepare and restore the VM, 3 minutes for the real Omarchy Midscene flow, and 7 minutes 31 seconds for the whole job. The prior install-per-run workflow took 17 minutes 43 seconds, so the restored path saves 10 minutes 12 seconds (about 58%).

The Omarchy workflow continues from that installed base image:

  1. Boot a throwaway overlay from the installed base image.
  2. Copy this checkout to md.lifeos.doubao-say in the guest and run Omarchy's plugin validator.
  3. Run the complete Midscene product suite through VNC in the real Omarchy/Hyprland guest session. The fixture explicitly uses the Wayland backend, loads layer-shell before GTK (matching the production entrypoint), and checks layer-shell support before declaring readiness. The installer runs with the same Wayland session variables and its layer-shell package and typelib are checked before starting the tests. A missing native backend fails setup instead of placing a fallback overlay over the onboarding controls.

The Ubuntu workflow runs the deterministic GTK fixture through @midscene/test and @midscene/computer. Pull requests from forks are skipped because GitHub does not expose the required model secret to untrusted workflow code.

Both distributions execute the same 14 declarative product cases from cases/onboarding.yaml, cases/onboarding-regressions.yaml, and cases/runtime.yaml, including synthetic microphone, voice, regression, and post-setup dictation checks. midscene.config.ts passes each case identity to a fresh fixture. ubuntu-product and omarchy-product each include all product case files without tag filtering. Projects and cases run serially so window focus cannot leak between tests. The config prepares the GTK fixture or Omarchy VNC viewer, and resets that state for every retry. Custom shell.* Nodes prepare the real Omarchy menu and bar; cases/omarchy-shell.yaml keeps the three pixel-level assertions explicit. Run npm run nodes in this directory to regenerate the Node reference, or npm test -- --project ubuntu-product on a prepared Linux desktop. Projects use Midscene Test's case-level retry setting, so a failed model attempt is rerun with a fresh fixture while successful cases are kept. Every attempt remains visible in the generated report. Each project writes a Midscene Test HTML report under midscene_run/report/. CI keeps the native reports separate, then aggregates their cases into one evidence table. Every row links to its exact native report node. A failed project captures its first failing screenshot, so the summary shows the relevant failure detail.

The Ubuntu stage maps the real GTK onboarding window inside the headless Midscene desktop. Its synthetic fixture starts signed out, opens an explicitly labelled CI-only sign-in window, and simulates a successful return without a network request or real credentials. It also performs no recording, system clipboard access, global paste, or user-settings read. Midscene verifies the automatic transition to the microphone step, visually navigates to the shortcut page, runs the fake endpoint check, verifies its visible feedback, continues to the voice page, checks that navigation reset the scroll position, completes a synthetic microphone check and voice test, and finishes setup. Focused cases cover cancellation, device changes, official API configuration, custom shortcut capture, and a synthetic F8/Escape dictation target. The runtime target uses the real Delivery state machine but writes only to its own GTK field, so CI never records audio or reads or changes the runner clipboard. The HTML replay is uploaded as a private workflow artifact for every run.

The AI stage needs a multimodal model credential. Its MIDSCENE_MODEL_* configuration is stored only as GitHub Actions Secrets; tests contain only synthetic data.

Omarchy shell visual PoC

After onboarding, the Omarchy job opens the real system menu through Omarchy's shell IPC and uses Midscene's VNC view to assert that Shutdown is readable and one menu row has a clean focus highlight. It then sets the bar to the left, asserts that the rendered bar is vertical and docked to the left edge, and restores the original bar configuration. The HTML replay is included in the omarchy-midscene-e2e-report artifact. The Actions run summary shows the three visual verdicts directly. Successful runs on main publish a Pages entrance image that links directly to the Midscene Test shell report; the onboarding report is published alongside it.

Omarchy acceptance test today Midscene visual check
screen_contains "Shutdown": enlarge a grim image, run Tesseract, and grep for text YAML aiAssert checks that Shutdown is legible in the rendered system menu
hyprctl -j layers and bar_is_vertical: compare layer dimensions YAML aiAssert checks the visible bar's shape and left-edge placement
Standalone grim PNGs Step-by-step HTML replay with assertions and screenshots

The SSH and hyprctl calls in this PoC set up the scene and wait for the menu; the three user-visible conclusions come from Midscene's image assertions. The guest runs Hyprland/Wayland; Midscene operates an X11 VNC viewer on the host. This demonstrates visual testing of a Wayland desktop through a VM bridge.

Ubuntu regression cases

cases/onboarding-regressions.yaml covers ten focused cases: sign-in cancellation and persistence, disabled and custom triggers, polishing state, endpoint recovery, microphone checks and device changes, voice-test cancellation, Volcengine API configuration, and Deepgram English API configuration. cases/runtime.yaml adds successful and cancelled F8 dictation delivery. The Ubuntu project runs these alongside the successful onboarding case.

Each case receives a fresh computer agent, GTK fixture, and temporary config directory. Fixture cleanup completes before the next case starts. Shortcut and polishing changes stay in memory; ci-fail-model produces a synthetic endpoint error, while ci-ok-model succeeds without network traffic. Navigation and trigger controls are located from visible labels rather than fixed screen coordinates. Microphone readiness cases visit the intervening Trigger key page before Voice test, and endpoint cases scroll the result into view before checking the full feedback. The assertions still verify the original readiness gates, saved values, and complete endpoint responses.

These cases exercise production GTK widgets and the Delivery state machine with synthetic callbacks. They do not establish live ASR, recording, durable credential persistence, system-wide shortcut capture, clipboard paste, or delivery into an unrelated application.

Run npm test -- --project ubuntu-product with the system dependencies and model configuration from the Ubuntu workflow. The complete Ubuntu job also runs ubuntu-polishing. Omarchy runs omarchy-product and omarchy-shell sequentially inside one VM session. Each invocation keeps its own native report; a failed product invocation does not suppress the auxiliary project. Focused manual runs can still select a product, auxiliary, or provider project.

Polishing overlay regression

The Ubuntu workflow also runs npm test -- --project ubuntu-polishing. polish_fixture.py exposes test controls around the real Overlay component, using synthetic transcripts without microphone, network, or input injection. cases/polishing.yaml visually checks English and Chinese status/transcript separation, stars, absence of a divider, the delayed hint, streaming text updates, reduced-motion presentation, completion, and hiding. Timing and static-star behavior are additionally checked by tests/manual/polish_overlay.py; a still image alone does not prove animation or reduced-motion timing.

The test shares the existing isolated desktop setup and model credentials. Inspect the HTML replay under midscene_run/report/. This component flow does not establish live ASR, polishing service, or paste acceptance, and does not add Wayland coverage for the overlay.