A personal agent with a browser, terminal, files, and work that keeps going built with CopilotKit and AG-UI.
TypeScript
docs: add a 38-second widescreen mobile demo to the README hero
Jerel John Velarde16fcc378a6c3
diff --git a/README.md b/README.mdindex 184ac86..154480f 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@# OpenMuse-**A personal agent with a browser, files, and work that keeps going.**+**A personal agent with a browser, terminal, files, and work that keeps going.**Ask for an outcome. Follow the plan, review actions, and come back to the result.Built with CopilotKit React Native for iOS, Android, and web.@@ -12,45 +12,46 @@ Built with CopilotKit React Native for iOS, Android, and web.[](https://github.com/jerelvelarde/openmuse/actions/workflows/ci.yml)[](LICENSE)+[](assets/openmuse-mobile-demo.mp4)++**[Watch the mobile demo · 38 seconds](assets/openmuse-mobile-demo.mp4)**+</div>> **Alpha, for self-hosting and building on.** The default runs locally with fictional data and no API keys. Open-ended reasoning, live Google accounts, and CopilotKit Rich Threads require their own configuration. See [what is verified](docs/VERIFICATION.md) and the [roadmap](ROADMAP.md).## Demo-[**Watch the native iPhone walkthrough**](https://github.com/jerelvelarde/openmuse/releases/download/v0.1.0-alpha/openmuse-demo.mp4)--<a href="https://github.com/jerelvelarde/openmuse/releases/download/v0.1.0-alpha/openmuse-demo.mp4"><img src="assets/computer.png" alt="OpenMuse's agent computer showing a real persistent Chromium session" width="280"></a>-<a href="https://github.com/jerelvelarde/openmuse/releases/download/v0.1.0-alpha/openmuse-demo.mp4"><img src="assets/document.png" alt="The native PDF reader displaying a saved filled permission form" width="280"></a>+A 38-second native iPhone walkthrough, framed in 16:9: the agent's browser, Linux terminal, saved files, PDF reader, rich conversation, and goals. Captured from the working app with fictional personal data and real public websites; cut and accelerated for pace.-Actual app capture with fictional mail, a generated PDF, saved tasks, a finance artifact, goals, and a real Chromium browser. Edited for pace; no simulated provider connections. [Recording details and reproduction](docs/DEMO.md).+[Watch the MP4](assets/openmuse-mobile-demo.mp4) · [Recording details and reproduction](docs/DEMO.md)## What it isOpenMuse is a personal-agent application inspired by Meta Muse and patterned after [OpenBot](https://github.com/CopilotKit/OpenBot)'s emphasis on an agent's computer, visible work, and rich results. It runs its own server, task worker, and browser worker. You can inspect and change the source under the MIT license.-The current computer is **persistent Chromium plus documents**. The agent can read public pages and collect PDFs; you can open the same browser session and interact with it. Full desktop VMs and autonomous checkout are future work.+The computer combines **persistent Chromium and an optional Linux workspace**. The agent can browse public pages, run commands in its own container, work with files, and move PDFs between the computer and the app. You can open its browser or terminal and continue the work. Graphical desktops and autonomous checkout remain future work.## Features| Surface | What runs in this alpha || --- | --- |-| **Chat** | CopilotKit headless chat with streamed AG-UI events, delegated tasks, and inline browser, PDF, plan, and finance cards. |-| **Agent computer** | Persistent browser profiles, real screenshots, interactive console, public-page reads, PDF downloads, and files. |+| **Chat** | CopilotKit headless chat with streamed AG-UI events, a visible follow-up queue, retained drafts, delegated tasks, and inline browser, PDF, plan, and finance cards. |+| **Agent computer** | Persistent browser profiles and takeover console; optional isolated Linux terminal, saved command receipts, editable workspace files, and PDF transfer. || **Activity** | Durable task plans, progress, input requests, pause/resume/cancel/retry, approvals, and saved receipts. SQL leases recover interrupted work. || **Ideas** | Suggestions with source evidence; edit, accept, or dismiss. Sent replies and completed matching work are excluded. || **Goals & Tracking** | Goals and milestones; recurring public-page checks for changes, text availability, or USD price thresholds, with deduplicated alerts and failure backoff. || **Documents** | Email attachment → PDF → requested form values → filled copy → reviewed reply → receipt. Native/web PDF viewing, paging, zoom, supported fields, and sharing. || **Finance** | Import transaction CSV to create a spending summary with categories, transactions, and a savings-goal action. || **Gmail & Calendar** | Google OAuth adapters, complete mail threads, drafts/attachments, calendar discovery, and reviewed event creation/update/deletion. Live credentials required. |-| **Personal context** | Editable agent identity and memories you can inspect, update, and forget. Durable in-app notifications. |-| **Rich Threads** | Optional CopilotKit Intelligence persistence with native thread listing, switching, renaming, archiving, restoring, and replay. A project key is required; live acceptance is pending. |+| **Personal context** | Editable name, tone, avatar, and memories. Background-update preferences and durable in-app notifications. |+| **Rich Threads** | Optional CopilotKit Intelligence persistence with a stable main conversation, side chats, renaming, archiving, restoring, and replay. A project key is required; live acceptance is pending. |The [feature inventory](docs/FEATURES.md) maps the Muse references to the implementation. Health/bank/social connectors, device push, voice, generated executable tools, and automatic reservations/payments are on the [roadmap](ROADMAP.md).## Quick start-**Requirements:** Node 24 LTS and pnpm 11.19.0. The sample app needs no model, Google account, Docker, or Intelligence subscription.+**Requirements:** Node 24 LTS and pnpm 11.19.0. The local app needs no model, Google account, Docker, or Intelligence subscription.```shgit clone https://github.com/jerelvelarde/openmuse.git@@ -70,8 +71,8 @@ Open [localhost:8081](http://localhost:8081). The API runs at [localhost:8787/ap### Try it-1. In Chat, send **“Complete the permission slip”**. Open the task, supply fictional form values, inspect the saved PDF, and review the sample reply. This writes only to the local sample mailbox.-2. In **Goals → Track**, create a built-in availability watch, then change the sample page to trigger an alert.+1. In Chat, send **“Complete the permission slip”**. Open the task, supply fictional form values, inspect the saved PDF, and review the prepared reply. This writes only to the local mailbox.+2. In **Goals → Track**, create a built-in availability watch, then change the built-in test page to trigger an alert.3. In **Menu → Delegate task → Finance**, use **Try example transactions** to create an interactive spending tracker.4. Start the [browser worker](#browser-worker), then open **Computer** and navigate to `https://example.com`.@@ -81,12 +82,12 @@ For iOS or Android, use `pnpm --dir apps/mobile ios` or `pnpm --dir apps/mobileCopy the commented settings in [.env.example](.env.example) into your private `.env`:-1. Set `AGENT_BACKEND=model`, `MODEL=provider/model-id`, and the matching provider key. CopilotKit supports the configured OpenAI, Anthropic or Google provider. Sample data can still be used with a real model. Provider keys stay on the server.+1. Set `AGENT_BACKEND=model`, `MODEL=provider/model-id`, and the matching provider key. CopilotKit supports the configured OpenAI, Anthropic or Google provider. Fictional data can still be used with a real model. Provider keys stay on the server.2. For personal mail/calendar, set `WORKSPACE_MODE=live`, a random `OPENMUSE_ACCESS_KEY` of at least 24 characters, and `TOKEN_ENCRYPTION_KEY` containing 32 random bytes encoded as base64. Restart the API.3. Configure a Google OAuth web client with Gmail and Calendar APIs enabled. Set `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET`; register `${PUBLIC_API_URL}/api/google/callback` as its redirect URI. Configure consent/test-user access in your Google project.4. Open **Apps → Gmail** (or **Google Calendar**), connect read access, and grant write access when needed. Every send or calendar change still requires its own stored review. Changing/disconnecting the account invalidates pending connection-bound work.-Google credentials are encrypted at rest. File URLs and browser consoles use short-lived signatures. This deployment uses one owner protected by a shared access key; it is not a multi-tenant authentication system. Use HTTPS and restricted network access for a remote host. Do not expose sample mode beyond loopback.+Google credentials are encrypted at rest. File URLs and browser consoles use short-lived signatures. This deployment uses one owner protected by a shared access key; it is not a multi-tenant authentication system. Use HTTPS and restricted network access for a remote host. Keep the default local-data mode on loopback.## Browser worker@@ -101,6 +102,19 @@ Or use `docker compose --env-file .env -f infra/compose.yaml up --build -d`. The## Persistence and operation+### Linux terminal and workspace++Build the computer image, enable it on the API, then open **Computer → Terminal → Start computer**:++```sh+docker build -t openmuse-computer:local apps/computer+COMPUTER_ENABLED=true pnpm dev+```++The API needs access to the Docker CLI and engine. Commands run in a nonroot container with no host-directory mounts or credentials. A named `/workspace` volume retains files when stopped. Terminal networking is disabled; public web access uses the browser worker. Commands have a 30-second limit and saved output/exit receipts. **Files** supports folders, text editing, and PDF transfer to/from Documents. This is a Linux container, not Meta Secure VM. [Setup, Colima option, and boundaries](docs/COMPUTER.md).++### Application storage+By default, embedded PGlite, documents and the signing key live in `.openmuse/`; browser profiles live in `.openmuse/browser-profiles/`. Keep that directory private and back it up. The API hosts the task worker. The host must remain running for background work.For a separate task worker, configure the same `DATABASE_URL`, secrets and shared `DATA_DIR` for both processes, then set `TASK_WORKER_ENABLED=false` on the API and run `pnpm dev:worker`. PGlite cannot be opened by separate processes. Production commands are `pnpm build:server`, `pnpm start` and `pnpm start:worker`. Run one API instance; task workers coordinate through SQL leases.@@ -126,6 +140,9 @@ flowchart TDReview --> Google[Gmail / Calendar adapters]Tasks --> Browser[Chromium worker + persistent profiles]API --> Browser+ API --> Computer[Optional Docker Linux computer]+ Tasks --> Computer+ Computer --> Volume[(Persistent workspace volume)]Tasks --> Files[PDF files + structured artifacts]API -. future adapter .-> OpenBot[OpenBot]```@@ -135,6 +152,7 @@ flowchart TD| `apps/mobile` | Shared iOS, Android, and web UI with CopilotKit headless hooks. || `apps/server` | API, CopilotKit runtime, identity boundary, task engine, reviews, files, and persistence. || `apps/worker` | Token-protected Playwright browser service with persistent profiles. |+| `apps/computer` | Nonroot Linux image, bounded filesystem helper, and real container verification. || `packages/domain` | Shared types and request validation. || `packages/integrations` | Google and browser protocol adapters. || `packages/backends` | Optional OpenBot HTTP adapter and its identity boundary. |@@ -156,12 +174,13 @@ pnpm build:iospnpm build:androidpnpm --dir apps/worker typecheckpnpm test:browser+pnpm test:computer```-Platform build scripts export JavaScript/Hermes bundles; they do not produce signed app binaries. Browser checks require installed Chromium and public fixture access. CI also exercises the browser container. See [contribution guidance](CONTRIBUTING.md) and [verification results](docs/VERIFICATION.md).+Platform build scripts export JavaScript/Hermes bundles; they do not produce signed app binaries. Browser checks require installed Chromium and public fixture access. CI also exercises the browser and Linux computer containers. See [contribution guidance](CONTRIBUTING.md) and [verification results](docs/VERIFICATION.md).## Contributing and licenseIssues and pull requests are welcome. Start with [CONTRIBUTING.md](CONTRIBUTING.md), [ROADMAP.md](ROADMAP.md), and the [security policy](SECURITY.md).-MIT licensed. OpenMuse is independent of Meta and is not an official CopilotKit product. Its original interface and sample assets are included; Meta's screenshots and mascot are not redistributed. Website, email, and document content supplies evidence, not permission to act.+MIT licensed. OpenMuse is independent of Meta and is not an official CopilotKit product. Its original interface and fictional assets are included; Meta's screenshots and mascot are not redistributed. Website, email, and document content supplies evidence, not permission to act.diff --git a/assets/openmuse-demo.gif b/assets/openmuse-demo.gifnew file mode 100644index 0000000..c7b4a1fBinary files /dev/null and b/assets/openmuse-demo.gif differdiff --git a/assets/openmuse-mobile-demo.mp4 b/assets/openmuse-mobile-demo.mp4new file mode 100644index 0000000..35532a9Binary files /dev/null and b/assets/openmuse-mobile-demo.mp4 differdiff --git a/assets/openmuse-preview.png b/assets/openmuse-preview.pngnew file mode 100644index 0000000..15a4e9eBinary files /dev/null and b/assets/openmuse-preview.png differdiff --git a/docs/DEMO.md b/docs/DEMO.mdindex fcea387..6608e3a 100644--- a/docs/DEMO.md+++ b/docs/DEMO.md@@ -1,41 +1,50 @@-# Native walkthrough+# Mobile demo-[Watch the MP4](https://github.com/jerelvelarde/openmuse/releases/download/v0.1.0-alpha/openmuse-demo.mp4).+[Watch the 38-second MP4](../assets/openmuse-mobile-demo.mp4) · [Animated hero](../assets/openmuse-demo.gif) · [Cover image](../assets/openmuse-preview.png).-Recorded from the actual OpenMuse development build on an iPhone 17 simulator. The capture is edited into short chapters and played at 2× speed. It runs for 75 seconds and has no narration. All mail, goals, transactions, and form details are fictional local data; the browser visits the real public `example.com` page.+Recorded from the working OpenMuse development build on an iPhone 17 simulator. The video is **1920 × 1080 (16:9), 30 fps, and 38 seconds**, with a cream, blue, and lilac background and captions designed for sound-off viewing. The opening hook is “Your agent has a computer.” Actual app captures are cut and accelerated up to 2×; no provider connections or execution results are simulated in the edit.## What the recording shows-| Time | Chapter |+| Time | Scene || --- | --- |-| 0:00 | CopilotKit task-linked PDF result and the polished message composer |-| 0:07 | The agent's persistent browser computer |-| 0:12 | The interactive Chromium console |-| 0:21 | A real filled PDF in the native reader |-| 0:35 | A saved finance task, plan, and interactive spending artifact |-| 0:53 | Goals and tracking |-| 1:00 | Source-backed Ideas |-| 1:06 | Gmail/Calendar local-data status and computer connection |+| 0:00–0:03 | The agent computer and its saved browser session |+| 0:03–0:10 | Opening the real CopilotKit website and Hacker News in persistent Chromium |+| 0:10–0:15 | A Linux command receipt with successful output and a saved `today.md` file |+| 0:15–0:21 | Opening and saving that same file in the workspace editor |+| 0:21–0:27 | A PDF exported from the computer into the native reader, including paging |+| 0:27–0:32 | A saved document result and action update in the CopilotKit conversation |+| 0:32–0:38 | Goals and the open-source repository link |-The walkthrough opens persisted results. It is not a claim that live Google, a paid model provider, CopilotKit Intelligence, or OpenBot is connected. Rich Threads has runtime integration tests; live persistence/replay still requires a project key. See [verification](VERIFICATION.md).+The walkthrough shows manual browser/computer interaction and previously saved task results. It does not present these edits as an uninterrupted autonomous agent run. Personal mail, goals, and form details are fictional. CopilotKit and Hacker News are real public pages. The Linux workspace runs in the project's isolated Docker container, using a dedicated Colima VM on the recording machine; it is not Meta Secure VM or a graphical desktop.++Live Google, model-provider quality, CopilotKit Intelligence persistence/replay, and OpenBot still require separately configured acceptance runs. See [verification](VERIFICATION.md).## Reproduce the workspace-1. Follow the [quick start](../README.md#quick-start). Keep `WORKSPACE_MODE=sample` and `AGENT_BACKEND=sample` for fictional local data.-2. In Chat, ask **“Complete the permission slip”**. Open the task and enter fictional details. Review the new PDF, then approve the local action. The task saves its receipt and its CopilotKit tool card stays in the conversation.-3. In **Menu → Delegate task → Finance**, enter a request, choose **Try example transactions**, and delegate. The saved artifact shows income 4,200.00, spending 110.99, and remaining 4,089.01. These are imported amounts, not a bank balance.-4. Open that artifact to create a savings goal, then visit Goals.-5. Start the [browser worker](../apps/worker/README.md#local-development), open Computer, and create a session at `https://example.com`.-6. In Ideas, use **Find ideas** and inspect a source. Completed matching document work and sent replies should not be suggested again.-7. Import or fill a PDF in Files to exercise the reader, pagination, and sharing.+1. Follow the [quick start](../README.md#quick-start). Keep the default local-data mode for fictional personal information.+2. In Chat, ask **“Complete the permission slip”**. Open the task, enter fictional form values, inspect the PDF, and review the local reply. Its saved tool result remains in the conversation.+3. Start the [browser worker](../apps/worker/README.md#local-development), open Computer, and navigate to `copilotkit.ai` or `news.ycombinator.com`. Bare domains become HTTPS addresses.+4. Enable the [Linux computer](COMPUTER.md) and open **Computer → Terminal → Start computer**. Run:++ ```sh+ printf '# Today\n\n- Read Hacker News\n- Review my calendar\n- Plan the week\n' > today.md+ cat today.md+ ```++5. Open **Files → today.md** and save it. The file is in the same persistent `/workspace` volume used by the terminal.+6. Choose **Copy a document here** to import an app PDF. Open the copied file from the workspace list to export it into Documents and display it in the native reader. Use **Next** to inspect the second page.+7. In **Menu → Delegate task → Finance**, use **Try example transactions**. Open the saved artifact to create a savings goal and visit Goals.## Record your own demo-Use a simulator containing only fictional test data. With the native app open:+Use a simulator containing fictional personal data. With the native app open:```shxcrun simctl io booted recordVideo --codec=h264 openmuse-recording.mp4-# Interact with the app. Press Control-C in this terminal to finish the video.+# Interact with the app. Press Control-C to finish the video.```-The MP4 in the release is an actual screen capture, cut and encoded for size. The repository contains its original OpenMuse screenshot assets; Meta reference screenshots are not redistributed.+Capture at native resolution, trim idle time, and place the portrait capture inside a 16:9 canvas. Keep captions short and show the first working screen immediately. Check the final video for development reload banners, credentials, and accidental private content before publishing.++The original [75-second alpha walkthrough](https://github.com/jerelvelarde/openmuse/releases/download/v0.1.0-alpha/openmuse-demo.mp4) remains available as a release archive. The README hero uses the newer 38-second cut. OpenMuse's own artwork is included; Meta reference screenshots and mascot are not redistributed.