Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

feat(drive): put three.ws Drive on Android Auto and reach a connected home from the car

nirholascc8087646966
diff --git a/docs/carplay.md b/docs/carplay.md
index 2a3c3be10..eb3cbc431 100644
--- a/docs/carplay.md
+++ b/docs/carplay.md
@@ -1,8 +1,9 @@
# three.ws Drive: the agent in the car
**Status:** research complete, architecture decided, the shared car surface (`/drive`) is
-built and verified in a browser, and the native CarPlay scene is written and wired into the
-iOS app. The one step that cannot be finished from here is Apple's entitlement grant.
+built and verified in a browser, the native CarPlay scene is written and wired into the iOS
+app, and the Android Auto car app is written, wired into the Android overlay, and compiled.
+The steps that cannot be finished from here are Apple's entitlement grant and a car.
**Date:** 2026-09-03.
**Question asked:** can a three.ws 3D agent be the assistant on an Apple CarPlay screen, and
how much of it already exists in open source?
@@ -70,7 +71,7 @@ another template instead of taking the whole screen, and the template was opened
category. An app holding `carplay-maps` can now keep a map rendering underneath a live
assistant. That is a door worth walking through later; it is not the first shipment.
-### Android's side, for symmetry
+### Android's side
Android draws the same line in a different place. The Car App Library
(`androidx.car.app`, Apache-2.0) is template-based too, and the host renders the UI. But its
@@ -81,6 +82,20 @@ technically reachable on Android Auto through the navigation category, and block
CarPlay by the exclusive-map rule. Android Automotive OS, which runs the app on the head
unit itself, is looser still.
+What Android does **not** have is a conversational category. The manifest categories are
+`NAVIGATION`, `POI`, `IOT`, `WEATHER`, `MEDIA`, `MESSAGING` and `CALLING`, and an app
+declares exactly one. The accurate one here is **`androidx.car.app.category.IOT`**, for apps
+that let people "take relevant actions on connected devices from within the car". That is
+not a workaround: a three.ws agent already reaches a real house through the home tools wired
+into `/api/chat` ([smart-home.md](smart-home.md)), with a safety gate that freezes anything
+physical until a person approves it. "Turn the porch light on" from the car is the product,
+and the category names it correctly.
+
+The consequence is that Android and iOS end up in the same place from opposite directions.
+Apple grants a conversational category with no canvas; Google grants a canvas only to
+navigation and has no conversational category at all. Either way the car screen is
+templates, the conversation is the page, and the agent's face is somewhere else.
+
### The open-source inventory
| Project | License | What it gives us | Verdict |
@@ -117,7 +132,20 @@ channel joins them. Written, wired, and documented in
[`ios/docs/CARPLAY.md`](../ios/docs/CARPLAY.md). It cannot ship until Apple grants
`com.apple.developer.carplay-voice-based-conversation`, which is a per-app request.
-### Lane C: the open head unit (designed, not built)
+### Lane C: native Android Auto (built)
+
+A `CarAppService` in the Android app's overlay, declaring the IOT category. The car screen
+shows the same four controls; the conversation runs in a web view inside a typed foreground
+service, because during a drive the phone screen belongs to Android Auto and a background
+process may not hold a microphone. Written, wired, compiled, and documented in
+[`solana-mobile/docs/ANDROID-AUTO.md`](../solana-mobile/docs/ANDROID-AUTO.md).
+
+That web view is never attached to a window, which means no animation frames, so
+`?surface=androidauto` turns the 3D stage off and the loop runs as audio. The same preset
+turns the confirmation card off: a person who cannot see what they are approving is a
+recognizer being trusted with a lock, and the home safety doctrine refuses that outright.
+
+### Lane D: the open head unit (designed, not built)
`node-carplay` plus a Carlinkit CPC200 dongle renders a real CarPlay or Android Auto session
inside a browser surface we own, on a Raspberry Pi or any Linux box. In that lane there is no
@@ -188,9 +216,10 @@ same class, which is what makes barge-in and "louder" real rather than decorativ
| 1 | `/drive`: the surface, the voice loop, the safety rules, the presets | **Done.** Verified in Chromium at 800x480, 1280x720, 1920x720 and 390x844, day and night, with a real `/api/chat` and `/api/tts` turn. |
| 2 | Native CarPlay scene, audio session, message channel, entitlement, Info.plist | **Done in the tree.** Cannot be compiled or run without Xcode and a car or the CarPlay simulator. |
| 3 | Apple's entitlement request | **Owner action.** See `ios/docs/CARPLAY.md`. Nothing else blocks. |
-| 4 | Android Auto: a `CarAppService` in `solana-mobile/`, same four controls, same channel | Scoped, not started. |
-| 5 | Lane C head unit: `node-carplay` beside the agent on a Pi | Scoped, needs a CPC200 dongle. |
-| 6 | The car as a home trigger: "we are ten minutes out" through `@three-ws/home-bridge` | Scoped. Both halves exist. |
+| 4 | Android Auto: `CarAppService`, foreground-service web view, the same channel, the same four controls | **Done and compiled.** Needs the Desktop Head Unit and a phone to exercise on a car screen. |
+| 5 | The home lane reaching the car: guarded actions surfaced and gated on being parked | **Done.** `/api/chat`'s home tools already work from `/drive`; the confirmation card is wired and refused while moving. |
+| 6 | Android Automotive OS: `app-automotive` plus a `CarAppActivity` in its own build flavor | Scoped, not started. |
+| 7 | Lane D head unit: `node-carplay` beside the agent on a Pi | Scoped, needs a CPC200 dongle. |
---
diff --git a/pages/drive.html b/pages/drive.html
index 2bc5a18b0..3fa968666 100644
--- a/pages/drive.html
+++ b/pages/drive.html
@@ -527,6 +527,18 @@
<div class="dr-grid" id="dr-picker-grid"></div>
</section>
+ <section class="dr-sheet" id="dr-confirm" hidden aria-label="Approve a home action">
+ <div class="dr-sheet-head">
+ <h1 class="dr-sheet-title">Approve this</h1>
+ <button type="button" class="dr-pill" id="dr-confirm-close">Not now</button>
+ </div>
+ <p class="dr-sheet-sub">
+ Your agent will not touch a lock, a garage door, or an alarm on its own. It froze the
+ action and is waiting for you.
+ </p>
+ <div id="dr-confirm-slot"></div>
+ </section>
+
<section class="dr-sheet" id="dr-type" hidden aria-label="Type to your agent">
<div class="dr-sheet-head">
<h1 class="dr-sheet-title">Type</h1>
diff --git a/solana-mobile/android-overlay/AndroidManifest.application.xml b/solana-mobile/android-overlay/AndroidManifest.application.xml
index f5f80bad2..e6081db9e 100644
--- a/solana-mobile/android-overlay/AndroidManifest.application.xml
+++ b/solana-mobile/android-overlay/AndroidManifest.application.xml
@@ -24,3 +24,31 @@
<data android:scheme="threews" android:host="glance" />
</intent-filter>
</activity>
+
+ <!-- three.ws Drive (solana-mobile/android-overlay): Android Auto.
+ The IOT category is the accurate one: a three.ws agent takes real
+ actions on the user's connected devices through the home tools wired
+ into /api/chat, with a safety gate that freezes anything physical
+ until a person approves it. Docs: docs/carplay.md. -->
+ <meta-data
+ android:name="androidx.car.app.minCarApiLevel"
+ android:value="1" />
+
+ <meta-data
+ android:name="com.google.android.gms.car.application"
+ android:resource="@xml/drive_automotive_app_desc" />
+
+ <service
+ android:name=".drive.DriveCarAppService"
+ android:exported="true"
+ android:label="@string/drive_car_title">
+ <intent-filter>
+ <action android:name="androidx.car.app.CarAppService" />
+ <category android:name="androidx.car.app.category.IOT" />
+ </intent-filter>
+ </service>
+
+ <service
+ android:name=".drive.DriveWebService"
+ android:exported="false"
+ android:foregroundServiceType="microphone|mediaPlayback" />
diff --git a/solana-mobile/android-overlay/AndroidManifest.permissions.xml b/solana-mobile/android-overlay/AndroidManifest.permissions.xml
index 1bd4dd41f..151622bbe 100644
--- a/solana-mobile/android-overlay/AndroidManifest.permissions.xml
+++ b/solana-mobile/android-overlay/AndroidManifest.permissions.xml
@@ -2,3 +2,13 @@
from an OS process, so the app itself declares the network. -->
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
+
+ <!-- three.ws Drive (solana-mobile/android-overlay): the agent in the car.
+ The loop runs in a service-hosted web view while Android Auto owns the
+ phone screen, so it is a background process by definition and needs a
+ typed foreground service to hold the microphone at all. -->
+ <uses-permission android:name="android.permission.RECORD_AUDIO"/>
+ <uses-permission android:name="android.permission.FOREGROUND_SERVICE"/>
+ <uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE"/>
+ <uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PLAYBACK"/>
+ <uses-permission android:name="android.permission.POST_NOTIFICATIONS"/>
diff --git a/solana-mobile/android-overlay/README.md b/solana-mobile/android-overlay/README.md
index fdce231ef..59a5fd431 100644
--- a/solana-mobile/android-overlay/README.md
+++ b/solana-mobile/android-overlay/README.md
@@ -9,8 +9,18 @@ fails the build if this directory is missing.
## What is in it
-The home screen widget ("Agent glance"): the owner's agent as a card, one live number, and a
-tap that opens the app on that agent. Product docs: [`docs/native-widgets.md`](../../docs/native-widgets.md).
+Two features, namespaced so they can never collide with each other or with a generated file.
+
+**The home screen widget** ("Agent glance"): the owner's agent as a card, one live number, and
+a tap that opens the app on that agent. Product docs:
+[`docs/native-widgets.md`](../../docs/native-widgets.md).
+
+**three.ws Drive** (`drive*`): the agent in the car, on Android Auto. The car screen is four
+rows rendered by Android Auto's template host; the conversation runs in a web view inside a
+foreground service, because during a drive the phone screen belongs to Android Auto and a
+background process may not hold the microphone. Docs:
+[`../docs/ANDROID-AUTO.md`](../docs/ANDROID-AUTO.md) and
+[`docs/carplay.md`](../../docs/carplay.md).
| File | Role |
| --- | --- |
@@ -22,14 +32,26 @@ tap that opens the app on that agent. Product docs: [`docs/native-widgets.md`](.
| `app/src/main/res/layout/glance_widget_{small,wide}.xml` | RemoteViews layouts for 2x2 and 4x2 / 4x3. |
| `app/src/main/res/xml/glance_widget_info.xml` | Widget metadata: sizes, resize modes, no AppWidget alarm (`updatePeriodMillis=0`, WorkManager refreshes). |
| `app/src/main/res/values/glance.xml`, `res/drawable/glance_*.xml` | Strings, colors, the rounded card background, the action chip. |
-| `AndroidManifest.permissions.xml` | Spliced after the `<manifest>` tag: `INTERNET`, `ACCESS_NETWORK_STATE`. |
-| `AndroidManifest.application.xml` | Spliced before `</application>`: the widget receiver and the link activity. |
-| `build.gradle.fragment` | Appended to `app/build.gradle`: `androidx.work:work-runtime`. |
+| `.../drive/DriveCarAppService.java` | The `CarAppService`: declares the IOT category, pins the production host allowlist. |
+| `.../drive/DriveSession.java` | One connection to the car; opens the screen. |
+| `.../drive/DriveScreen.java` | The four controls as a `ListTemplate`, and the microphone-permission screen that precedes them. |
+| `.../drive/DriveLink.java` | The channel to the page: state in, presses out, everything marshalled to the main looper. |
+| `.../drive/DriveWebService.java` | The typed foreground service hosting the web view that runs `/drive?surface=androidauto`. |
+| `app/src/main/res/xml/drive_automotive_app_desc.xml` | `<uses name="template" />`: how Android Auto discovers the car app at all. |
+| `app/src/main/res/values/drive.xml` | Every string on the car screen and in the session notification. |
+| `AndroidManifest.permissions.xml` | Spliced after the `<manifest>` tag: `INTERNET`, `ACCESS_NETWORK_STATE`, `RECORD_AUDIO`, the typed foreground-service permissions, `POST_NOTIFICATIONS`. |
+| `AndroidManifest.application.xml` | Spliced before `</application>`: the widget receiver, the link activity, the car app service, and the drive web service. |
+| `build.gradle.fragment` | Appended to `app/build.gradle`: `androidx.work:work-runtime`, `androidx.car.app:app`, `androidx.car.app:app-projected`. |
## Rules
-- Every file name and resource id starts with `glance`, so the overlay can never collide with a
- Bubblewrap-generated file. `apply-overlay.mjs` refuses to overwrite anything it does not own.
+- Every file name and resource id starts with its feature (`glance`, `drive`), so the overlay
+ can never collide with a Bubblewrap-generated file. `apply-overlay.mjs` refuses to overwrite
+ anything it does not own.
+- **Both manifest splices and the gradle fragment are fenced and REPLACED on re-apply**, not
+ skipped when a marker is present. Editing a fragment therefore works the obvious way; a
+ presence check would have left the old block in place and silently dropped the new
+ declaration.
- Java 8 source level (that is what the generated project compiles at). No Kotlin, no Compose.
- The widget never renders WebGL or HTML. It shows the PNG the server renders and nothing else.
- Applying the overlay by hand, for a debug build:
diff --git a/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveCarAppService.java b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveCarAppService.java
new file mode 100644
index 000000000..5df7d076e
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveCarAppService.java
@@ -0,0 +1,43 @@
+package ws.three.app.drive;
+
+import androidx.annotation.NonNull;
+import androidx.car.app.CarAppService;
+import androidx.car.app.Session;
+import androidx.car.app.validation.HostValidator;
+
+/**
+ * three.ws Drive on the Android Auto screen.
+ *
+ * The declared category is {@code androidx.car.app.category.IOT}: "take relevant
+ * actions on connected devices from within the car". That is literally what this
+ * is. A three.ws agent already reaches a real house through the home tools wired
+ * into /api/chat, with a safety gate that freezes anything physical until a
+ * person approves it, so "turn the porch light on" from the car is the product
+ * and not a stretch of the category.
+ *
+ * There is no conversational category on Android the way iOS 26.4 added one for
+ * CarPlay, and there is no drawing surface outside navigation. So this screen is
+ * four controls, the conversation lives in DriveWebService's web view, and the
+ * two are joined by DriveLink.
+ *
+ * @see ws.three.app.drive.DriveScreen
+ */
+public final class DriveCarAppService extends CarAppService {
+
+ @Override
+ @NonNull
+ public HostValidator createHostValidator() {
+ // The library's own allowlist of signed Android Auto / Automotive hosts.
+ // ALLOW_ALL_HOSTS_VALIDATOR exists and is a debug convenience; shipping it
+ // would let any app on the device drive the car screen as us.
+ return new HostValidator.Builder(getApplicationContext())
+ .addAllowedHosts(androidx.car.app.R.array.hosts_allowlist_sample)
+ .build();
+ }
+
+ @Override
+ @NonNull
+ public Session onCreateSession() {
+ return new DriveSession();
+ }
+}
diff --git a/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveLink.java b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveLink.java
new file mode 100644
index 000000000..97e17eba6
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveLink.java
@@ -0,0 +1,231 @@
+package ws.three.app.drive;
+
+import android.os.Handler;
+import android.os.Looper;
+
+import org.json.JSONArray;
+import org.json.JSONObject;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * The single channel between the Android Auto templates on the car screen and
+ * the /drive page running in {@link DriveWebService}'s web view.
+ *
+ * Android Auto's template host renders the car screen; an app never draws on it
+ * outside the navigation category, and three.ws is not a navigation app. So the
+ * conversation, the agent, its voice and its memory all stay in the page, and
+ * this object is the wire between them. The page posts its state out through the
+ * {@code ThreeWsDriveNative} JavaScript interface, and template presses go back
+ * in as {@code window.threeWsDrive.command(...)}. Both halves of that protocol
+ * live in src/drive/bridge.js, and the iOS half is DriveLink.swift.
+ *
+ * Every mutation lands on the main looper, because the page calls in from the
+ * WebView's JavaScript thread and the car screen may only be invalidated from
+ * the main thread.
+ */
+public final class DriveLink {
+
+ /** Protocol version. Must match PROTOCOL in src/drive/bridge.js. */
+ static final int PROTOCOL = 1;
+
+ public static final String STATE_IDLE = "idle";
+
+ /** One control mirrored onto the car screen. */
+ public static final class Action {
+ public final String id;
+ public final String label;
+ public final boolean enabled;
+
+ Action(String id, String label, boolean enabled) {
+ this.id = id;
+ this.label = label;
+ this.enabled = enabled;
+ }
+ }
+
+ /** Implemented by the car screen so it can repaint when anything changes. */
+ public interface Listener {
+ void onDriveChanged();
+ }
+
+ /** Implemented by the web service: delivers a command into the page. */
+ public interface CommandSink {
+ void sendCommand(String json);
+ }
+
+ private static final DriveLink INSTANCE = new DriveLink();
+
+ public static DriveLink get() {
+ return INSTANCE;
+ }
+
+ private final Handler main = new Handler(Looper.getMainLooper());
+
+ private String state = STATE_IDLE;
+ private String agentName;
+ private String lastSpoken;
+ private String lastHeard;
+ private String lastError;
+ private List<Action> actions = Collections.emptyList();
+
+ private Listener listener;
+ private CommandSink sink;
+
+ private DriveLink() {
+ }
+
+ // -- reads (main thread) --------------------------------------------------
+
+ public String state() {
+ return state;
+ }
+
+ /** Is a turn in flight? The car screen shows progress for exactly this window. */
+ public boolean isActive() {
+ return !STATE_IDLE.equals(state);
+ }
+
+ public String agentName() {
+ return agentName;
+ }
+
+ public String lastSpoken() {
+ return lastSpoken;
+ }
+
+ public String lastHeard() {
+ return lastHeard;
+ }
+
+ public String lastError() {
+ return lastError;
+ }
+
+ public List<Action> actions() {
+ return actions;
+ }
+
+ // -- wiring ---------------------------------------------------------------
+
+ public void setListener(Listener value) {
+ listener = value;
+ }
+
+ public void setCommandSink(CommandSink value) {
+ sink = value;
+ }
+
+ /**
+ * Send a command the page understands. See onNativeCommand in
+ * src/drive/index.js for the accepted types. Dropped, deliberately, when no
+ * page is attached yet: a press before the loop is up is not an error.
+ */
+ public void send(String type) {
+ CommandSink target = sink;
+ if (target == null) return;
+ try {
+ JSONObject payload = new JSONObject();
+ payload.put("type", type);
+ target.sendCommand(payload.toString());
+ } catch (Exception ignored) {
+ // A malformed command must never take the drive down.
+ }
+ }
+
+ public void reset() {
+ main.post(new Runnable() {
+ @Override
+ public void run() {
+ state = STATE_IDLE;
+ agentName = null;
+ lastSpoken = null;
+ lastHeard = null;
+ lastError = null;
+ actions = Collections.emptyList();
+ notifyChanged();
+ }
+ });
+ }
+
+ // -- inbound: page to car screen ------------------------------------------
+
+ /**
+ * Called from the JavaScript interface thread with one message from the
+ * page. Anything unparseable is dropped rather than thrown: the page is a
+ * remote document and this is a trust boundary.
+ */
+ void accept(String json) {
+ final JSONObject body;
+ try {
+ body = new JSONObject(json);
+ } catch (Exception e) {
+ return;
+ }
+ if (body.optInt("v", -1) != PROTOCOL) return;
+ final String type = body.optString("type", "");
+ if (type.isEmpty()) return;
+
+ main.post(new Runnable() {
+ @Override
+ public void run() {
+ apply(type, body);
+ notifyChanged();
+ }
+ });
+ }
+
+ private void apply(String type, JSONObject body) {
+ switch (type) {
+ case "ready": {
+ JSONObject agent = body.optJSONObject("agent");
+ agentName = agent == null ? null : emptyToNull(agent.optString("name", ""));
+ lastError = null;
+ break;
+ }
+ case "state":
+ state = body.optString("state", STATE_IDLE);
+ break;
+ case "heard":
+ lastHeard = emptyToNull(body.optString("text", ""));
+ break;
+ case "said":
+ lastSpoken = emptyToNull(body.optString("text", ""));
+ lastError = null;
+ break;
+ case "error":
+ lastError = emptyToNull(body.optString("message", ""));
+ break;
+ case "actions":
+ actions = readActions(body.optJSONArray("actions"));
+ break;
+ default:
+ break;
+ }
+ }
+
+ private static List<Action> readActions(JSONArray raw) {
+ if (raw == null) return Collections.emptyList();
+ List<Action> parsed = new ArrayList<>(raw.length());
+ for (int i = 0; i < raw.length(); i++) {
+ JSONObject entry = raw.optJSONObject(i);
+ if (entry == null) continue;
+ String id = entry.optString("id", "");
+ String label = entry.optString("label", "");
+ if (id.isEmpty() || label.isEmpty()) continue;
+ parsed.add(new Action(id, label, entry.optBoolean("enabled", true)));
+ }
+ return Collections.unmodifiableList(parsed);
+ }
+
+ private void notifyChanged() {
+ Listener target = listener;
+ if (target != null) target.onDriveChanged();
+ }
+
+ private static String emptyToNull(String value) {
+ return value == null || value.isEmpty() ? null : value;
+ }
+}
diff --git a/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveScreen.java b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveScreen.java
new file mode 100644
index 000000000..ba541cbd6
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveScreen.java
@@ -0,0 +1,197 @@
+package ws.three.app.drive;
+
+import android.Manifest;
+import android.content.pm.PackageManager;
+
+import androidx.annotation.NonNull;
+import androidx.car.app.CarContext;
+import androidx.car.app.CarToast;
+import androidx.car.app.Screen;
+import androidx.car.app.model.Action;
+import androidx.car.app.model.ItemList;
+import androidx.car.app.model.ListTemplate;
+import androidx.car.app.model.MessageTemplate;
+import androidx.car.app.model.Row;
+import androidx.car.app.model.Template;
+import androidx.lifecycle.DefaultLifecycleObserver;
+import androidx.lifecycle.LifecycleOwner;
+
+import java.util.Collections;
+import java.util.List;
+
+import ws.three.app.R;
+
+/**
+ * The car screen: four controls and what the agent is doing right now.
+ *
+ * They are the same four the /drive page shows and the same four CarPlay shows,
+ * in the same order, because a driver who learns one has learned all three. Four
+ * is not an arbitrary number either: it is the ceiling CarPlay's voice template
+ * puts on action buttons, and designing every surface to the tightest one keeps
+ * them honest.
+ *
+ * A row with no click listener is how "not available yet" is expressed. Repeat
+ * before the agent has said anything is not a button that does nothing; it is a
+ * line of text explaining itself.
+ */
+public final class DriveScreen extends Screen implements DriveLink.Listener, DefaultLifecycleObserver {
+
+ private boolean askedForMicrophone;
+
+ DriveScreen(@NonNull CarContext carContext) {
+ super(carContext);
+ getLifecycle().addObserver(this);
+ }
+
+ // -- lifecycle -------------------------------------------------------------
+
+ @Override
+ public void onCreate(@NonNull LifecycleOwner owner) {
+ DriveLink.get().setListener(this);
+ }
+
+ @Override
+ public void onStart(@NonNull LifecycleOwner owner) {
+ if (hasMicrophone()) startLoop();
+ }
+
+ @Override
+ public void onDestroy(@NonNull LifecycleOwner owner) {
+ DriveLink.get().setListener(null);
+ DriveWebService.stop(getCarContext());
+ }
+
+ @Override
+ public void onDriveChanged() {
+ invalidate();
+ }
+
+ // -- template --------------------------------------------------------------
+
+ @Override
+ @NonNull
+ public Template onGetTemplate() {
+ if (!hasMicrophone()) return microphoneTemplate();
+
+ DriveLink link = DriveLink.get();
+ ItemList.Builder items = new ItemList.Builder();
+ for (DriveLink.Action action : controls()) {
+ Row.Builder row = new Row.Builder().setTitle(action.label);
+ String detail = detailFor(action.id);
+ if (detail != null) row.addText(detail);
+ if (action.enabled) {
+ final String id = action.id;
+ row.setOnClickListener(() -> perform(id));
+ }
+ items.addItem(row.build());
+ }
+
+ String title = link.agentName() == null
+ ? getCarContext().getString(R.string.drive_car_title)
+ : link.agentName();
+
+ return new ListTemplate.Builder()
+ .setSingleList(items.build())
+ .setTitle(title)
+ .setHeaderAction(Action.APP_ICON)
+ .build();
+ }
+
+ /**
+ * What the page published, or the resting set until it has published
+ * anything. An empty car screen while the loop boots would read as broken.
+ */
+ private List<DriveLink.Action> controls() {
+ List<DriveLink.Action> published = DriveLink.get().actions();
+ if (!published.isEmpty()) return published;
+ CarContext context = getCarContext();
+ return Collections.unmodifiableList(java.util.Arrays.asList(
+ new DriveLink.Action("talk", context.getString(R.string.drive_action_talk), true),
+ new DriveLink.Action("hands", context.getString(R.string.drive_action_hands), true),
+ new DriveLink.Action("repeat", context.getString(R.string.drive_action_repeat), false),
+ new DriveLink.Action("hush", context.getString(R.string.drive_action_hush), false)));
+ }
+
+ /** One short line per row. A driver reads at most this much. */
+ private String detailFor(String id) {
+ DriveLink link = DriveLink.get();
+ CarContext context = getCarContext();
+ if (link.lastError() != null && "talk".equals(id)) return link.lastError();
+ switch (id) {
+ case "talk":
+ return "listening".equals(link.state())
+ ? context.getString(R.string.drive_detail_listening)
+ : context.getString(R.string.drive_detail_ask);
+ case "hands":
+ return context.getString(R.string.drive_detail_hands);
+ case "repeat":
+ return link.lastSpoken() != null
+ ? link.lastSpoken()
+ : context.getString(R.string.drive_detail_nothing_said);
+ case "hush":
+ return "speaking".equals(link.state())
+ ? context.getString(R.string.drive_detail_stop_talking)
+ : null;
+ default:
+ return null;
+ }
+ }
+
+ private void perform(String id) {
+ DriveLink link = DriveLink.get();
+ if ("talk".equals(id)) {
+ link.send("listening".equals(link.state()) ? "talk-stop" : "talk-start");
+ return;
+ }
+ link.send(id);
+ }
+
+ // -- microphone ------------------------------------------------------------
+
+ private boolean hasMicrophone() {
+ return getCarContext().checkSelfPermission(Manifest.permission.RECORD_AUDIO)
+ == PackageManager.PERMISSION_GRANTED;
+ }
+
+ /**
+ * Voice is the entire product here, so a missing microphone is a screen of
+ * its own rather than a silent failure. CarContext.requestPermissions puts
+ * the system prompt on the phone and tells the driver to look at it, which is
+ * the only place Android will ever show it.
+ */
+ private Template microphoneTemplate() {
+ CarContext context = getCarContext();
+ return new MessageTemplate.Builder(context.getString(R.string.drive_mic_needed))
+ .setTitle(context.getString(R.string.drive_car_title))
+ .setHeaderAction(Action.APP_ICON)
+ .addAction(new Action.Builder()
+ .setTitle(context.getString(R.string.drive_mic_grant))
+ .setOnClickListener(this::requestMicrophone)
+ .build())
+ .build();
+ }
+
+ private void requestMicrophone() {
+ askedForMicrophone = true;
+ getCarContext().requestPermissions(
+ Collections.singletonList(Manifest.permission.RECORD_AUDIO),
+ (granted, rejected) -> {
+ if (granted.contains(Manifest.permission.RECORD_AUDIO)) {
+ startLoop();
+ } else if (askedForMicrophone) {
+ CarToast.makeText(
+ getCarContext(),
+ R.string.drive_mic_denied,
+ CarToast.LENGTH_LONG).show();
+ }
+ invalidate();
+ });
+ }
+
+ private void startLoop() {
+ DriveWebService.start(getCarContext());
+ // Voice first: the driver opened an assistant, so it starts listening
+ // rather than waiting to be told twice.
+ DriveLink.get().send("talk-start");
+ }
+}
diff --git a/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveSession.java b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveSession.java
new file mode 100644
index 000000000..722fdc417
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveSession.java
@@ -0,0 +1,17 @@
+package ws.three.app.drive;
+
+import android.content.Intent;
+
+import androidx.annotation.NonNull;
+import androidx.car.app.Screen;
+import androidx.car.app.Session;
+
+/** One connection to the car. Owns nothing but the screen it opens. */
+public final class DriveSession extends Session {
+
+ @Override
+ @NonNull
+ public Screen onCreateScreen(@NonNull Intent intent) {
+ return new DriveScreen(getCarContext());
+ }
+}
diff --git a/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveWebService.java b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveWebService.java
new file mode 100644
index 000000000..debaf74cd
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/DriveWebService.java
@@ -0,0 +1,210 @@
+package ws.three.app.drive;
+
+import android.app.Notification;
+import android.app.NotificationChannel;
+import android.app.NotificationManager;
+import android.app.PendingIntent;
+import android.app.Service;
+import android.content.Context;
+import android.content.Intent;
+import android.content.pm.ServiceInfo;
+import android.net.Uri;
+import android.os.Build;
+import android.os.IBinder;
+import android.webkit.JavascriptInterface;
+import android.webkit.PermissionRequest;
+import android.webkit.WebChromeClient;
+import android.webkit.WebSettings;
+import android.webkit.WebView;
+
+import ws.three.app.R;
+
+/**
+ * Hosts the /drive page for the length of an Android Auto session.
+ *
+ * The page is the product: microphone in, /api/asr, /api/chat, /api/tts, and the
+ * agent's own voice back out. On Android Auto there is nowhere to draw it, so it
+ * runs here in a web view that is never attached to a window, and the page is
+ * told so with {@code ?surface=androidauto}: src/drive/surface.js turns off the
+ * 3D stage for that surface, because a windowless web view gets no animation
+ * frames. Audio, fetch and timers all keep running, which is the whole loop.
+ *
+ * A foreground service is not optional here. The phone screen belongs to Android
+ * Auto during a drive, so this process is background by definition, and a
+ * background process may not hold the microphone. The notification is the honest
+ * price of that: the user can see the agent is listening and stop it in one tap.
+ */
+public final class DriveWebService extends Service implements DriveLink.CommandSink {
+
+ /** The car screen starts and stops the loop with these. */
+ public static final String ACTION_START = "ws.three.app.drive.START";
+ public static final String ACTION_STOP = "ws.three.app.drive.STOP";
+
+ /** The surface preset the page reads. See src/drive/surface.js. */
+ private static final String DRIVE_URL = "https://three.ws/drive?surface=androidauto";
+
+ /** The name the page posts to. Must match src/drive/bridge.js. */
+ private static final String BRIDGE = "ThreeWsDriveNative";
+
+ private static final String CHANNEL_ID = "drive_session";
+ private static final int NOTIFICATION_ID = 4201;
+
+ private WebView web;
+
+ public static void start(Context context) {
+ Intent intent = new Intent(context.getApplicationContext(), DriveWebService.class);
+ intent.setAction(ACTION_START);
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
+ context.getApplicationContext().startForegroundService(intent);
+ } else {
+ context.getApplicationContext().startService(intent);
+ }
+ }
+
+ public static void stop(Context context) {
+ Intent intent = new Intent(context.getApplicationContext(), DriveWebService.class);
+ intent.setAction(ACTION_STOP);
+ context.getApplicationContext().startService(intent);
+ }
+
+ @Override
+ public IBinder onBind(Intent intent) {
+ return null;
+ }
+
+ @Override
+ public int onStartCommand(Intent intent, int flags, int startId) {
+ if (intent != null && ACTION_STOP.equals(intent.getAction())) {
+ stopSelf();
+ return START_NOT_STICKY;
+ }
+ goForeground();
+ if (web == null) createWebView();
+ return START_NOT_STICKY;
+ }
+
+ @Override
+ public void onDestroy() {
+ DriveLink.get().setCommandSink(null);
+ DriveLink.get().reset();
+ if (web != null) {
+ web.removeJavascriptInterface(BRIDGE);
+ web.loadUrl("about:blank");
+ web.destroy();
+ web = null;
+ }
+ super.onDestroy();
+ }
+
+ // -- the page -------------------------------------------------------------
+
+ private void createWebView() {
+ // Application context: a Service context is not a UI context, and some
+ // OEM web views refuse to inflate against one.
+ web = new WebView(getApplicationContext());
+ WebSettings settings = web.getSettings();
+ settings.setJavaScriptEnabled(true);
+ settings.setDomStorageEnabled(true);
+ // No window means no gesture is possible, and the whole point of the
+ // surface is that it speaks the moment it has something to say.
+ settings.setMediaPlaybackRequiresUserGesture(false);
+
+ web.setWebChromeClient(new WebChromeClient() {
+ @Override
+ public void onPermissionRequest(final PermissionRequest request) {
+ // Grant the microphone and nothing else, and only what the page
+ // actually asked for. The app's own RECORD_AUDIO grant is checked
+ // on the car screen before this service is ever started.
+ for (String resource : request.getResources()) {
+ if (PermissionRequest.RESOURCE_AUDIO_CAPTURE.equals(resource)) {
+ request.grant(new String[]{PermissionRequest.RESOURCE_AUDIO_CAPTURE});
+ return;
+ }
+ }
+ request.deny();
+ }
+ });
+
+ web.addJavascriptInterface(new Bridge(), BRIDGE);
+ DriveLink.get().setCommandSink(this);
+ web.loadUrl(DRIVE_URL);
+ }
+
+ /**
+ * The page's outbound half. Every method here is reachable from a remote
+ * document, so it takes one string, parses it defensively, and exposes
+ * nothing else.
+ */
+ private static final class Bridge {
+ @JavascriptInterface
+ public void post(String json) {
+ DriveLink.get().accept(json);
+ }
+ }
+
+ @Override
+ public void sendCommand(final String json) {
+ final WebView target = web;
+ if (target == null) return;
+ target.post(new Runnable() {
+ @Override
+ public void run() {
+ // Optional chaining on the page side: a command that arrives
+ // before the page has booted is dropped, not thrown.
+ target.evaluateJavascript("window.threeWsDrive?.command(" + json + ");", null);
+ }
+ });
+ }
+
+ // -- foreground ------------------------------------------------------------
+
+ private void goForeground() {
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
+ NotificationManager manager = getSystemService(NotificationManager.class);
+ if (manager != null && manager.getNotificationChannel(CHANNEL_ID) == null) {
+ NotificationChannel channel = new NotificationChannel(
+ CHANNEL_ID,
+ getString(R.string.drive_channel_name),
+ NotificationManager.IMPORTANCE_LOW);
+ channel.setDescription(getString(R.string.drive_channel_description));
+ channel.setShowBadge(false);
+ manager.createNotificationChannel(channel);
+ }
+ }
+
+ Notification notification = buildNotification();
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
+ startForeground(
+ NOTIFICATION_ID,
+ notification,
+ ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE
+ | ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PLAYBACK);
+ } else {
+ startForeground(NOTIFICATION_ID, notification);
+ }
+ }
+
+ private Notification buildNotification() {
+ Intent stop = new Intent(this, DriveWebService.class).setAction(ACTION_STOP);
+ int flags = PendingIntent.FLAG_UPDATE_CURRENT;
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) flags |= PendingIntent.FLAG_IMMUTABLE;
+ PendingIntent stopIntent = PendingIntent.getService(this, 1, stop, flags);
+
+ Intent open = new Intent(Intent.ACTION_VIEW, Uri.parse("https://three.ws/drive"));
+ PendingIntent openIntent = PendingIntent.getActivity(this, 2, open, flags);
+
+ Notification.Builder builder = Build.VERSION.SDK_INT >= Build.VERSION_CODES.O
+ ? new Notification.Builder(this, CHANNEL_ID)
+ : new Notification.Builder(this);
+
+ return builder
+ .setContentTitle(getString(R.string.drive_notification_title))
+ .setContentText(getString(R.string.drive_notification_text))
+ .setSmallIcon(android.R.drawable.ic_btn_speak_now)
+ .setContentIntent(openIntent)
+ .setOngoing(true)
+ .addAction(new Notification.Action.Builder(
+ null, getString(R.string.drive_notification_stop), stopIntent).build())
+ .build();
+ }
+}
diff --git a/solana-mobile/android-overlay/app/src/main/res/values/drive.xml b/solana-mobile/android-overlay/app/src/main/res/values/drive.xml
new file mode 100644
index 000000000..2dd55aa2f
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/res/values/drive.xml
@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- three.ws Drive (solana-mobile/android-overlay): car screen + session copy. -->
+<resources>
+ <string name="drive_car_title">three.ws Drive</string>
+
+ <string name="drive_action_talk">Talk</string>
+ <string name="drive_action_hands">Hands free</string>
+ <string name="drive_action_repeat">Repeat</string>
+ <string name="drive_action_hush">Quiet</string>
+
+ <string name="drive_detail_ask">Ask your agent something</string>
+ <string name="drive_detail_listening">Listening</string>
+ <string name="drive_detail_hands">Keep the mic open between turns</string>
+ <string name="drive_detail_nothing_said">Nothing said yet</string>
+ <string name="drive_detail_stop_talking">Stop talking</string>
+
+ <string name="drive_mic_needed">three.ws Drive is a voice assistant, so it needs the microphone. Grant it on your phone and it will start listening.</string>
+ <string name="drive_mic_grant">Grant microphone</string>
+ <string name="drive_mic_denied">Without the microphone there is nothing to listen with.</string>
+
+ <string name="drive_channel_name">Drive session</string>
+ <string name="drive_channel_description">Shown while your agent is riding with you, so it is never listening invisibly.</string>
+ <string name="drive_notification_title">Your agent is riding with you</string>
+ <string name="drive_notification_text">Talk to it from the car screen. Tap to open three.ws Drive.</string>
+ <string name="drive_notification_stop">Stop</string>
+</resources>
diff --git a/solana-mobile/android-overlay/app/src/main/res/xml/drive_automotive_app_desc.xml b/solana-mobile/android-overlay/app/src/main/res/xml/drive_automotive_app_desc.xml
new file mode 100644
index 000000000..ac338ed22
--- /dev/null
+++ b/solana-mobile/android-overlay/app/src/main/res/xml/drive_automotive_app_desc.xml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+ Declares which car experiences this app provides. Android Auto reads it from
+ the com.google.android.gms.car.application meta-data; without it the car app
+ service is never discovered and three.ws Drive simply does not appear.
+-->
+<automotiveApp>
+ <uses name="template" />
+</automotiveApp>
diff --git a/solana-mobile/android-overlay/build.gradle.fragment b/solana-mobile/android-overlay/build.gradle.fragment
index 2a97e9704..8b50a88a2 100644
--- a/solana-mobile/android-overlay/build.gradle.fragment
+++ b/solana-mobile/android-overlay/build.gradle.fragment
@@ -2,3 +2,12 @@
dependencies {
implementation 'androidx.work:work-runtime:2.9.1'
}
+
+// three.ws Drive (solana-mobile/android-overlay): the Android Auto car app.
+// app-projected is what makes the templates render on a phone-projected head
+// unit; app-automotive (a separate AAOS build with its own CarAppActivity) is
+// deliberately not here yet, see docs/carplay.md.
+dependencies {
+ implementation 'androidx.car.app:app:1.4.0'
+ implementation 'androidx.car.app:app-projected:1.4.0'
+}
diff --git a/solana-mobile/docs/ANDROID-AUTO.md b/solana-mobile/docs/ANDROID-AUTO.md
new file mode 100644
index 000000000..4532864c7
--- /dev/null
+++ b/solana-mobile/docs/ANDROID-AUTO.md
@@ -0,0 +1,104 @@
+# three.ws Drive on Android Auto
+
+The Android half of [`/drive`](../../docs/carplay.md). Read that document first: it explains
+why the agent's 3D face is not on the car screen, and why the same page serves every lane.
+
+This file covers the Android side only: what is in the overlay, why the category is what it
+is, and how to run it.
+
+## The one-paragraph version
+
+Android has no conversational app category, and no drawing surface outside navigation. What
+it does have is `androidx.car.app.category.IOT`, for apps that let people "take relevant
+actions on connected devices from within the car", and that is exactly what a three.ws agent
+does: the home tools wired into `/api/chat` reach a real house, with a safety gate that
+freezes anything physical until a person approves it. So the car screen is four rows rendered
+by Android Auto's template host, the conversation runs in a web view inside a foreground
+service, and `DriveLink` is the wire between them.
+
+## What is in the tree
+
+Everything lives in [`../android-overlay/`](../android-overlay/README.md), which is laid over
+the Bubblewrap-generated project on every build. File and resource names start with `drive`,
+which is what keeps the overlay from ever colliding with a generated file.
+
+| File | Role |
+|---|---|
+| `app/src/main/java/ws/three/app/drive/DriveCarAppService.java` | The `CarAppService`. Declares the IOT category and the production host allowlist. |
+| `.../DriveSession.java` | One connection to the car; opens the screen. |
+| `.../DriveScreen.java` | The four controls as a `ListTemplate`, plus the microphone-permission screen. |
+| `.../DriveLink.java` | The channel: page state in, template presses out, everything marshalled to the main looper. |
+| `.../DriveWebService.java` | The foreground service that hosts the web view running `/drive?surface=androidauto`. |
+| `app/src/main/res/xml/drive_automotive_app_desc.xml` | `<uses name="template" />`, which is how Android Auto discovers the car app at all. |
+| `app/src/main/res/values/drive.xml` | Every string on the car screen and in the notification. |
+| `AndroidManifest.permissions.xml` | Appended: `RECORD_AUDIO`, the typed foreground-service permissions, `POST_NOTIFICATIONS`. |
+| `AndroidManifest.application.xml` | Appended: `minCarApiLevel`, the automotive app descriptor, both services. |
+| `build.gradle.fragment` | Appended: `androidx.car.app:app` and `app-projected`. |
+| `../../src/drive/bridge.js` | The web half of the same protocol. Change one side, change both. |
+
+## Why a foreground service with a web view
+
+During a drive the phone screen belongs to Android Auto, so this app is a background process
+by definition, and a background process may not hold the microphone. A typed foreground
+service (`microphone|mediaPlayback`) is the only way to run the loop at all, and the
+notification it requires is the honest price: the driver can see the agent is listening and
+stop it in one tap.
+
+The web view is never attached to a window, which has one consequence worth stating: no
+window means no animation frames. `?surface=androidauto` tells the page so, and
+[`src/drive/surface.js`](../../src/drive/surface.js) turns off the 3D stage for that surface
+(`renders3d: false`). Audio, `fetch` and timers all keep running, and that is the whole
+conversation. The mouth target is safe unattached, so the identical `TalkController` runs on
+both surfaces with no branch inside it.
+
+The same flag turns off the confirmation card (`canConfirm: false`). A person who cannot see
+what they are approving is a recognizer being trusted with a lock, which the home safety
+doctrine refuses outright, so a guarded action from Android Auto is spoken back as "approve
+it on your phone" rather than taken.
+
+## The protocol
+
+Identical to CarPlay's, version 1. The page posts to the `ThreeWsDriveNative` JavaScript
+interface; commands go back through `evaluateJavascript`. The table is in
+[`ios/docs/CARPLAY.md`](../../ios/docs/CARPLAY.md) and both native sides implement it.
+
+## Permissions and the first run
+
+`RECORD_AUDIO` is a runtime permission and Android will only ever show its prompt on the
+phone. `DriveScreen` checks it before starting anything; when it is missing the car screen is
+a `MessageTemplate` with one action that calls `CarContext.requestPermissions`, which puts
+the prompt on the phone and tells the driver to look at it. That is the documented path and
+the only one that works from a car screen.
+
+## Building and running it
+
+```bash
+cd solana-mobile/build
+npx --no-install @bubblewrap/cli update --skipVersionUpgrade
+node ../scripts/apply-overlay.mjs --project . --overlay ../android-overlay
+JAVA_HOME=~/.bubblewrap/jdk/jdk-17.0.20.1+1 ./gradlew assembleDebug
+```
+
+To see it on a car screen, install the debug APK and run Google's **Desktop Head Unit**
+(`$ANDROID_HOME/extras/google/auto/desktop-head-unit`) against the connected phone with
+developer mode enabled in the Android Auto app. Unknown sources must be allowed there, or a
+debug build never appears.
+
+The web half alone needs no Android at all: `npm run dev`, then open
+`http://localhost:3000/drive?surface=androidauto`. The 3D stage stays off, the loop runs, and
+`window.threeWsDrive.command({ type: 'talk-start' })` in the console does exactly what a row
+press does.
+
+## Play Store posture
+
+Distribution of a car app is gated on Google's [car app
+quality](https://developer.android.com/docs/quality-guidelines/car-app-quality) review for
+the declared category, on top of the normal listing. Two things follow:
+
+- **The IOT claim has to stay true.** It is true today because the home lane is built and
+ wired into `/api/chat` ([docs/smart-home.md](../../docs/smart-home.md)). If that lane were
+ ever removed, the category would have to change with it.
+- **Android Automotive OS is a separate build.** AAOS runs the app on the head unit itself
+ and needs `androidx.car.app:app-automotive` plus a `CarAppActivity` entry point, in its own
+ build flavor. It is deliberately not in this overlay yet; the templates are the same, the
+ packaging is not.
diff --git a/solana-mobile/scripts/apply-overlay.mjs b/solana-mobile/scripts/apply-overlay.mjs
index 654997fad..8ec129c9b 100755
--- a/solana-mobile/scripts/apply-overlay.mjs
+++ b/solana-mobile/scripts/apply-overlay.mjs
@@ -3,7 +3,8 @@
//
// `bubblewrap update` regenerates the whole Android project from
// twa/twa-manifest.json on every build, so anything native (the glance home
-// screen widget) has to be laid on top AFTER update and BEFORE build. This
+// screen widget, the Android Auto car app) has to be laid on top AFTER update
+// and BEFORE build. This
// script is that step, and build-apk.sh runs it in exactly that slot.
//
// node scripts/apply-overlay.mjs --project build [--overlay android-overlay]
@@ -47,6 +48,8 @@ const REQUIRED = [
'build.gradle.fragment',
'app/src/main/java/ws/three/app/glance/GlanceWidget.java',
'app/src/main/res/xml/glance_widget_info.xml',
+ 'app/src/main/java/ws/three/app/drive/DriveCarAppService.java',
+ 'app/src/main/res/xml/drive_automotive_app_desc.xml',
];
for (const rel of REQUIRED) {
if (!fs.existsSync(path.join(overlayDir, rel))) fail(`overlay is missing ${rel} (${overlayDir})`);
@@ -57,8 +60,12 @@ const gradlePath = path.join(projectDir, 'app/build.gradle');
if (!fs.existsSync(manifestPath)) fail(`no generated manifest at ${manifestPath}; run bubblewrap update first`);
if (!fs.existsSync(gradlePath)) fail(`no generated build.gradle at ${gradlePath}; run bubblewrap update first`);
-// 1. Sources and resources. Every overlay file is namespaced "glance", so a
-// collision with a generated file means the generator changed under us.
+// 1. Sources and resources. Every overlay file is namespaced by its feature, so
+// a collision with a generated file means the generator changed under us.
+// Re-applying the overlay over its own previous output is normal and must not
+// trip that check, which is why ownership is by name and not by "does it exist".
+const OWNED = ['glance', 'drive'];
+const isOurs = (name) => OWNED.some((prefix) => name.includes(prefix));
let copied = 0;
function copyTree(from, to) {
for (const entry of fs.readdirSync(from, { withFileTypes: true })) {
@@ -69,7 +76,7 @@ function copyTree(from, to) {
copyTree(src, dst);
continue;
}
- if (fs.existsSync(dst) && fs.readFileSync(dst).equals(fs.readFileSync(src)) === false && !entry.name.includes('glance')) {
+ if (fs.existsSync(dst) && fs.readFileSync(dst).equals(fs.readFileSync(src)) === false && !isOurs(entry.name)) {
fail(`refusing to overwrite generated file ${dst}`);
}
fs.copyFileSync(src, dst);
@@ -80,35 +87,76 @@ copyTree(path.join(overlayDir, 'app'), path.join(projectDir, 'app'));
log(`copied ${copied} files into ${path.relative(root, projectDir)}/app`);
// 2. Manifest.
-const MARK_PERMS = '<!-- glance widget (solana-mobile/android-overlay): fetched';
-const MARK_APP = '<!-- glance widget (solana-mobile/android-overlay) -->';
-let manifest = fs.readFileSync(manifestPath, 'utf8');
-if (!manifest.includes(MARK_PERMS)) {
- const open = manifest.match(/<manifest\b[^>]*>/);
- if (!open) fail('cannot find the <manifest> opening tag');
- const at = open.index + open[0].length;
- manifest = `${manifest.slice(0, at)}\n\n${fs.readFileSync(path.join(overlayDir, 'AndroidManifest.permissions.xml'), 'utf8')}${manifest.slice(at)}`;
+//
+// Each splice is fenced by a begin/end comment and REPLACED on re-apply, rather
+// than skipped when a marker is present. The difference matters the first time
+// someone adds a permission to a fragment: a presence check leaves the old block
+// in place and the new declaration never reaches the APK, which is a silent
+// half-built feature. Replacing makes editing a fragment work the obvious way.
+function fence(key) {
+ return {
+ begin: `<!-- three.ws overlay: ${key} (begin, generated by scripts/apply-overlay.mjs) -->`,
+ end: `<!-- three.ws overlay: ${key} (end) -->`,
+ };
}
-if (!manifest.includes(MARK_APP)) {
- const at = manifest.lastIndexOf('</application>');
- if (at < 0) fail('cannot find </application>');
- manifest = `${manifest.slice(0, at)}${fs.readFileSync(path.join(overlayDir, 'AndroidManifest.application.xml'), 'utf8')}\n ${manifest.slice(at)}`;
+
+function spliceFenced(text, key, body, place) {
+ const { begin, end } = fence(key);
+ const block = `${begin}\n${body.trimEnd()}\n ${end}`;
+ const from = text.indexOf(begin);
+ const to = text.indexOf(end);
+ if (from >= 0 && to > from) {
+ return `${text.slice(0, from)}${block}${text.slice(to + end.length)}`;
+ }
+ const at = place(text);
+ if (at < 0) return null;
+ return `${text.slice(0, at)}\n ${block}\n${text.slice(at)}`;
}
+
+let manifest = fs.readFileSync(manifestPath, 'utf8');
+const permsBody = fs.readFileSync(path.join(overlayDir, 'AndroidManifest.permissions.xml'), 'utf8');
+const appBody = fs.readFileSync(path.join(overlayDir, 'AndroidManifest.application.xml'), 'utf8');
+
+const withPerms = spliceFenced(manifest, 'permissions', permsBody, (text) => {
+ const open = text.match(/<manifest\b[^>]*>/);
+ return open ? open.index + open[0].length : -1;
+});
+if (withPerms === null) fail('cannot find the <manifest> opening tag');
+manifest = withPerms;
+
+const withApp = spliceFenced(manifest, 'application', appBody, (text) => text.lastIndexOf('</application>'));
+if (withApp === null) fail('cannot find </application>');
+manifest = withApp;
+
fs.writeFileSync(manifestPath, manifest);
-for (const needle of ['.glance.GlanceWidget', '.glance.GlanceLinkActivity', 'android.permission.INTERNET']) {
+for (const needle of [
+ '.glance.GlanceWidget',
+ '.glance.GlanceLinkActivity',
+ 'android.permission.INTERNET',
+ '.drive.DriveCarAppService',
+ '.drive.DriveWebService',
+ 'androidx.car.app.category.IOT',
+ 'android.permission.RECORD_AUDIO',
+]) {
if (!manifest.includes(needle)) fail(`manifest merge lost ${needle}`);
}
-log('merged the widget receiver, link activity, and permissions into AndroidManifest.xml');
+log('merged the widget receiver, link activity, car app service, drive service, and permissions into AndroidManifest.xml');
// 3. Gradle.
const fragment = fs.readFileSync(path.join(overlayDir, 'build.gradle.fragment'), 'utf8');
let gradle = fs.readFileSync(gradlePath, 'utf8');
-const fragmentMark = fragment.split('\n')[0];
-if (!gradle.includes(fragmentMark)) {
- gradle = `${gradle.trimEnd()}\n\n${fragment}`;
- fs.writeFileSync(gradlePath, gradle);
+const GRADLE_BEGIN = '// three.ws overlay (begin, generated by scripts/apply-overlay.mjs)';
+const GRADLE_END = '// three.ws overlay (end)';
+const gradleBlock = `${GRADLE_BEGIN}\n${fragment.trimEnd()}\n${GRADLE_END}\n`;
+const gFrom = gradle.indexOf(GRADLE_BEGIN);
+const gTo = gradle.indexOf(GRADLE_END);
+gradle = gFrom >= 0 && gTo > gFrom
+ ? `${gradle.slice(0, gFrom)}${gradleBlock}${gradle.slice(gTo + GRADLE_END.length + 1)}`
+ : `${gradle.trimEnd()}\n\n${gradleBlock}`;
+fs.writeFileSync(gradlePath, gradle);
+for (const dep of ['androidx.work:work-runtime', 'androidx.car.app:app:', 'androidx.car.app:app-projected:']) {
+ if (!gradle.includes(dep)) fail(`build.gradle merge lost ${dep}`);
}
-if (!gradle.includes('androidx.work:work-runtime')) fail('build.gradle merge lost the WorkManager dependency');
-log('appended the WorkManager dependency to app/build.gradle');
+log('appended the WorkManager and Car App Library dependencies to app/build.gradle');
log('overlay applied');
diff --git a/solana-mobile/scripts/lib/hand.mjs b/solana-mobile/scripts/lib/hand.mjs
index ecc92274c..311f350b7 100644
--- a/solana-mobile/scripts/lib/hand.mjs
+++ b/solana-mobile/scripts/lib/hand.mjs
@@ -361,6 +361,17 @@ export class Hand {
/** Put an element in the comfortable middle band of the screen. */
async bringIntoView(selector, { top = 150, bottom = 190 } = {}) {
+ /* A fixed or sticky element travels with the viewport, so scrolling
+ towards it moves the page and never moves the target: the corner
+ companion widget would drag the whole tour down the document. */
+ const pinned = await this.page.locator(selector).first().evaluate((el) => {
+ for (let n = el; n && n !== document.documentElement; n = n.parentElement) {
+ const pos = getComputedStyle(n).position;
+ if (pos === 'fixed' || pos === 'sticky') return true;
+ }
+ return false;
+ }).catch(() => false);
+ if (pinned) return;
const box = await this.page.locator(selector).first().boundingBox();
if (!box) return;
const lo = top;
diff --git a/solana-mobile/scripts/lib/seeker-panel.mjs b/solana-mobile/scripts/lib/seeker-panel.mjs
index 3d6a9903d..c06264e1a 100644
--- a/solana-mobile/scripts/lib/seeker-panel.mjs
+++ b/solana-mobile/scripts/lib/seeker-panel.mjs
@@ -48,14 +48,33 @@ export const BG = '#080814';
export const AUTH_STATE = path.join(ROOT, '.auth/audit-state.json');
+/**
+ * Chromium launch flags for a recording.
+ *
+ * `--disable-dev-shm-usage` is not optional in a container: Docker gives /dev/shm
+ * 64 MB, and a heavy page (the marketplace document is 90k px tall with a live
+ * WebGL hero) blows through that. The renderer then either crashes outright with
+ * "Page crashed" or limps, which reads as the site being slow rather than as the
+ * sandbox being small: a single 1200x2670 screenshot of /marketplace measured
+ * 22s with the default shm and 4s with this flag.
+ */
+export function launchOptions() {
+ return { args: ['--disable-dev-shm-usage'] };
+}
+
+/** The panel in device pixels at a given capture density. */
+export function panelFor(dpr = DPR) {
+ return { width: CSS.width * dpr, height: CSS.height * dpr };
+}
+
/** The Android/Seeker browser context every recording runs in. */
-export function contextOptions({ authed = false } = {}) {
+export function contextOptions({ authed = false, dpr = DPR } = {}) {
if (authed && !existsSync(AUTH_STATE)) {
throw new Error(`--authed needs ${path.relative(ROOT, AUTH_STATE)}; mint it with: npm run audit:web:login`);
}
return {
viewport: CSS,
- deviceScaleFactor: DPR,
+ deviceScaleFactor: dpr,
isMobile: true,
hasTouch: true,
colorScheme: 'dark',
diff --git a/solana-mobile/scripts/make-feature-tour.mjs b/solana-mobile/scripts/make-feature-tour.mjs
index ffe4a5def..e050d7333 100644
--- a/solana-mobile/scripts/make-feature-tour.mjs
+++ b/solana-mobile/scripts/make-feature-tour.mjs
@@ -37,12 +37,17 @@
* --out= write somewhere other than marketing/seeker-video
* --fps= output frame rate (default 30)
* --seed= seed for the hand's jitter, so a rerun lands the same pixels
+ * --dpr= capture density (default 3, the Seeker's own); 2 halves render time
+ * and still exceeds the 782px the device frame draws
+ * --live-css keep CSS transitions running instead of snapping them settled
+ * --agent= open a specific agent page in the agent act instead of picking
+ * one out of the marketplace (a path, e.g. /marketplace/agents/<id>)
*/
import { chromium } from 'playwright';
import { mkdirSync, rmSync } from 'node:fs';
import path from 'node:path';
import {
- ROOT, CSS, PANEL, contextOptions, encodeScreen, encodeDevice,
+ ROOT, CSS, DPR, panelFor, contextOptions, launchOptions, encodeScreen, encodeDevice,
} from './lib/seeker-panel.mjs';
import { Hand, installHand } from './lib/hand.mjs';
@@ -54,6 +59,14 @@ const ORIGIN = String(args.origin || 'https://three.ws').replace(/\/$/, '');
const OUT = path.resolve(ROOT, String(args.out || 'marketing/seeker-video'));
const FPS = Number(args.fps || 30);
const SEED = Number(args.seed || 7);
+const CAPTURE_DPR = Number(args.dpr || DPR);
+const PANEL = panelFor(CAPTURE_DPR);
+/* Our frames are not real time, so a CSS transition captured mid-flight is
+ rendering at the wall clock's idea of progress, not the video's. Snapping
+ every transition to its settled state is both the honest frame for a stepped
+ capture and, on a page with a live WebGL hero, twice as fast to screenshot
+ (3.3s to 1.5s on /marketplace). Pass --live-css to keep them running. */
+const ANIMATIONS = args['live-css'] ? 'allow' : 'disabled';
const NAME = String(args.name || 'seeker-feature-tour');
const log = (...m) => console.log('[feature-tour]', ...m);
@@ -111,13 +124,16 @@ const ACTS = {
},
/* Search the live marketplace and open something out of the results. */
- market: async ({ page, hand, visit }) => {
- if (!page.url().includes('/marketplace')) {
+ market: async ({ page, hand, visit, origin }) => {
+ /* Walked into from another act: take the app's own route there, through
+ the nav drawer. Started here with --acts=market: just open the page. */
+ if (page.url().startsWith(origin) && !page.url().includes('/marketplace')) {
await hand.tapOn('#nav-toggle', { expect: 'a[href="/marketplace"]' });
await hand.hold(500);
await hand.tapOn('a[href="/marketplace"]', { expect: '#market-search' });
}
await visit('/marketplace', { ready: '#market-search', settle: 4200 });
+ await hand.tapIfPresent('.walk-companion-close', { after: 400 });
await hand.typeInto('#market-search', SEARCH, { settle: 900 });
await hand.waitFor(
(p) => p.locator('#market-grid a.title.card-profile-link').first().isVisible().catch(() => false),
@@ -132,11 +148,17 @@ const ACTS = {
/* An agent's own page: turn the model, then talk to it. */
agent: async ({ page, hand, visit }) => {
if (!(await page.locator('#av-stage').count())) {
- await visit('/marketplace', { ready: '#market-search', settle: 3500 });
- await hand.tapOn('#market-grid a.title.card-profile-link', { expect: '#av-stage' });
+ /* Walked in from the market act, this never runs. Started here, either
+ pick up the agent named on the command line or go and find one. */
+ if (args.agent) await visit(String(args.agent), { ready: '#av-stage', settle: 3000 });
+ else {
+ await visit('/marketplace', { ready: '#market-search', settle: 3500 });
+ await hand.tapOn('#market-grid a.title.card-profile-link', { expect: '#av-stage' });
+ }
}
await page.waitForSelector('#av-viewer', { timeout: 30_000 });
await page.waitForTimeout(6000);
+ await hand.tapIfPresent('.walk-companion-close', { after: 400 });
await hand.hold(900);
await hand.dragAcross('#av-viewer', { dx: 190, ms: 1100 });
await hand.hold(800);
@@ -181,10 +203,10 @@ rmSync(work, { recursive: true, force: true });
const frames = path.join(work, 'frames');
mkdirSync(frames, { recursive: true });
-const browser = await chromium.launch();
+const browser = await chromium.launch(launchOptions());
let count = 0;
try {
- const ctx = await browser.newContext(contextOptions({ authed: Boolean(args.authed) }));
+ const ctx = await browser.newContext(contextOptions({ authed: Boolean(args.authed), dpr: CAPTURE_DPR }));
await ctx.addInitScript(installHand);
const page = await ctx.newPage();
@@ -196,7 +218,8 @@ try {
path: path.join(frames, `${String(count).padStart(6, '0')}.jpg`),
type: 'jpeg',
quality: 92,
- timeout: 20_000,
+ animations: ANIMATIONS,
+ timeout: 30_000,
});
count += 1;
} catch (err) {
@@ -223,7 +246,7 @@ try {
for (const act of wanted) {
log(`act: ${act}`);
const at = count;
- await ACTS[act]({ page, hand, visit, log });
+ await ACTS[act]({ page, hand, visit, log, origin: ORIGIN });
log(`act ${act} added ${((count - at) / FPS).toFixed(1)}s`);
}
/* Leave the hand off the glass on the last frame. */
diff --git a/solana-mobile/scripts/make-screencast.mjs b/solana-mobile/scripts/make-screencast.mjs
index 1cf7ef56e..850321a9b 100644
--- a/solana-mobile/scripts/make-screencast.mjs
+++ b/solana-mobile/scripts/make-screencast.mjs
@@ -8,21 +8,19 @@
* the app IS this page. Driving that same page in Chromium at the Seeker's real
* panel geometry (1200x2670 device pixels, 3x density) produces the identical
* pixels the device would, and the tour below only touches links and sections
- * that exist on the live page.
- *
- * Frames are STEPPED, not recorded in real time. Playwright's video recorder
- * ignores deviceScaleFactor: ask it for a 1200x2670 video of a 400x890 CSS
- * viewport and it draws the page at 400x890 in the corner and pads the rest
- * grey. Screenshots do honour the scale factor, so the tour is advanced one
- * output frame at a time and each frame is captured at full device resolution.
- * Wall-clock capture time is then decoupled from playback time, which also
- * makes a rerun of the same tour produce the same video.
+ * that exist on the live page. The geometry, the browser context, and the two
+ * encodes live in lib/seeker-panel.mjs, which also explains why frames are
+ * stepped rather than recorded in real time.
*
* Two artefacts come out of one run:
* seeker-screen.mp4 the bare 1200x2670 panel, for dropping into an edit
* seeker-device.mp4 the same panel seated in a Seeker-proportioned body,
* 1080x1920, ready to post
*
+ * This is the short listing tour: a scripted scroll down /seeker, no hands, no
+ * typing. For a long-form video of someone actually using the app feature by
+ * feature, run make-feature-tour.mjs instead.
+ *
* What this deliberately does NOT fake: the Android status bar, the launcher,
* the Seed Vault approval sheet, and the dApp Store install. Those are system
* surfaces, not app surfaces, and inventing them would be inventing UI. Capture
@@ -33,47 +31,21 @@
* node solana-mobile/scripts/make-screencast.mjs --origin=http://localhost:3000
* node solana-mobile/scripts/make-screencast.mjs --authed # replays the audit session
*/
-import sharp from 'sharp';
import { chromium } from 'playwright';
-import { spawnSync } from 'node:child_process';
-import { existsSync, mkdirSync, rmSync } from 'node:fs';
-import { fileURLToPath } from 'node:url';
+import { mkdirSync, rmSync } from 'node:fs';
import path from 'node:path';
-
-const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..');
+import {
+ ROOT, CSS, PANEL, OUT_W, OUT_H, contextOptions, launchOptions, encodeScreen, encodeDevice,
+} from './lib/seeker-panel.mjs';
const args = Object.fromEntries(process.argv.slice(2).map((a) => {
- const m = a.match(/^--([^=]+)(?:=(.*))?$/);
- return m ? [m[1], m[2] ?? true] : [a, true];
+ const m = a.match(/^--([^=]+)(?:=(.*))?$/);
+ return m ? [m[1], m[2] ?? true] : [a, true];
}));
const ORIGIN = String(args.origin || 'https://three.ws').replace(/\/$/, '');
const OUT = path.resolve(ROOT, String(args.out || 'marketing/seeker-video'));
-const AUTH_STATE = path.join(ROOT, '.auth/audit-state.json');
const FPS = Number(args.fps || 30);
-/* Seeker panel: 6.36" AMOLED, 1200x2670 at ~460ppi, which Android reports as a
- 400x890 CSS viewport at density 3. Capturing at that density is what makes
- the frames real device pixels rather than an upscale. */
-const CSS = { width: 400, height: 890 };
-const DPR = 3;
-const PANEL = { width: CSS.width * DPR, height: CSS.height * DPR };
-
-/* Output composition. The screen keeps the panel's exact aspect ratio; the body
- reuses the bezel proportions and gradient the store frames already use, so
- the video and the listing screenshots read as one set of assets. */
-const OUT_W = 1080;
-const OUT_H = 1920;
-const SCREEN_H = 1740;
-const SCREEN_W = Math.round(SCREEN_H * (PANEL.width / PANEL.height)) & ~1;
-const BEZEL = 20;
-const BODY_W = SCREEN_W + BEZEL * 2;
-const BODY_H = SCREEN_H + BEZEL * 2;
-const BODY_X = Math.round((OUT_W - BODY_W) / 2);
-const BODY_Y = Math.round((OUT_H - BODY_H) / 2);
-const SCREEN_X = BODY_X + BEZEL;
-const SCREEN_Y = BODY_Y + BEZEL;
-const BG = '#080814';
-
/**
* The tour. Every `click` selector and every `to` selector resolves on the live
* page; a step that cannot find its target fails the run rather than silently
@@ -89,132 +61,83 @@ const BG = '#080814';
* rest of the video tells.
*/
const TOUR = [
- { hold: 3000, note: 'hero and the Seed Vault sign-in' },
- { to: '.grid', glide: 1800, hold: 1600, note: 'the Create lane' },
- { to: '#verify', glide: 1600, hold: 2800, note: 'Seeker verification' },
- { to: 0, glide: 1600, hold: 1200, note: 'back to the hero' },
+ { hold: 3000, note: 'hero and the Seed Vault sign-in' },
+ { to: '.grid', glide: 1800, hold: 1600, note: 'the Create lane' },
+ { to: '#verify', glide: 1600, hold: 2800, note: 'Seeker verification' },
+ { to: 0, glide: 1600, hold: 1200, note: 'back to the hero' },
];
const log = (...m) => console.log('[screencast]', ...m);
const frameCount = (ms) => Math.max(1, Math.round((ms / 1000) * FPS));
const ease = (x) => (x < 0.5 ? 4 * x * x * x : 1 - Math.pow(-2 * x + 2, 3) / 2);
-function ffmpeg(argv) {
- const r = spawnSync('ffmpeg', ['-hide_banner', '-loglevel', 'error', '-y', ...argv], { stdio: 'inherit' });
- if (r.error || r.status !== 0) throw new Error(`ffmpeg failed (${r.status ?? r.error?.message})`);
-}
-
/** Where a step wants the page scrolled to, resolved once before the glide. */
async function resolveTarget(page, target) {
- return page.evaluate((t) => {
- const max = Math.max(0, document.documentElement.scrollHeight - window.innerHeight);
- if (typeof t === 'string') {
- const el = document.querySelector(t);
- if (!el) return { error: `${t} is not on the page` };
- /* A section that is present but empty collapses to zero height and sits
- at the top of the document, so scrolling to it resolves to y=0 and the
- tour records a still of the hero while reporting success. Signed out,
- #agents and #mine are exactly that. Refuse it. */
- if (el.getBoundingClientRect().height < 1) {
- return { error: `${t} is on the page but has no height, so there is nothing to scroll to` };
- }
- return { y: Math.max(0, Math.min(max, window.scrollY + el.getBoundingClientRect().top - 72)) };
- }
- return { y: Math.max(0, Math.min(max, t > 0 && t < 1 ? t * max : t)) };
- }, target);
+ return page.evaluate((t) => {
+ const max = Math.max(0, document.documentElement.scrollHeight - window.innerHeight);
+ if (typeof t === 'string') {
+ const el = document.querySelector(t);
+ if (!el) return { error: `${t} is not on the page` };
+ /* A section that is present but empty collapses to zero height and sits
+ at the top of the document, so scrolling to it resolves to y=0 and the
+ tour records a still of the hero while reporting success. Signed out,
+ #agents and #mine are exactly that. Refuse it. */
+ if (el.getBoundingClientRect().height < 1) {
+ return { error: `${t} is on the page but has no height, so there is nothing to scroll to` };
+ }
+ return { y: Math.max(0, Math.min(max, window.scrollY + el.getBoundingClientRect().top - 72)) };
+ }
+ return { y: Math.max(0, Math.min(max, t > 0 && t < 1 ? t * max : t)) };
+ }, target);
}
async function capture(frames) {
- mkdirSync(frames, { recursive: true });
-
- const browser = await chromium.launch();
- const ctx = await browser.newContext({
- viewport: CSS,
- deviceScaleFactor: DPR,
- isMobile: true,
- hasTouch: true,
- colorScheme: 'dark',
- userAgent: 'Mozilla/5.0 (Linux; Android 15; Seeker) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36',
- ...(args.authed ? { storageState: AUTH_STATE } : {}),
- });
- if (args.authed && !existsSync(AUTH_STATE)) {
- throw new Error(`--authed needs ${path.relative(ROOT, AUTH_STATE)}; mint it with: npm run audit:web:login`);
- }
-
- const page = await ctx.newPage();
- let n = 0;
- const shoot = () => page.screenshot({
- path: path.join(frames, `${String(n++).padStart(6, '0')}.jpg`),
- type: 'jpeg',
- quality: 94,
- });
-
- log(`opening ${ORIGIN}/seeker at ${PANEL.width}x${PANEL.height}`);
- /* `load` never settles on this page (the 3D scene keeps streaming), so the
- ready signal is the hero being on screen plus a beat for the avatar. */
- await page.goto(`${ORIGIN}/seeker`, { waitUntil: 'domcontentloaded', timeout: 60_000 });
- await page.waitForSelector('#hero-title', { timeout: 30_000 });
- await page.waitForTimeout(4000);
-
- for (const step of TOUR) {
- log(step.note);
- if (step.click) {
- const target = page.locator(step.click).first();
- await target.waitFor({ state: 'visible', timeout: 15_000 });
- await target.click();
- await page.waitForLoadState('domcontentloaded', { timeout: 30_000 }).catch(() => {});
- await page.waitForTimeout(2500);
- }
- if (step.to !== undefined) {
- const from = await page.evaluate(() => window.scrollY);
- const { y, error } = await resolveTarget(page, step.to);
- if (error) throw new Error(`tour target not on the page: ${error}`);
- const total = frameCount(step.glide ?? 1200);
- for (let i = 1; i <= total; i += 1) {
- await page.evaluate((py) => window.scrollTo(0, py), from + (y - from) * ease(i / total));
- await shoot();
- }
- }
- for (let i = 0; i < frameCount(step.hold ?? 600); i += 1) await shoot();
- }
-
- await ctx.close();
- await browser.close();
- log(`captured ${n} frames (${(n / FPS).toFixed(1)}s at ${FPS}fps)`);
- return n;
-}
-
-/** The backdrop and the phone body, drawn once at output resolution. */
-async function chrome(dir) {
- const bg = Buffer.from(`<svg xmlns="http://www.w3.org/2000/svg" width="${OUT_W}" height="${OUT_H}">
- <defs><radialGradient id="g" cx="0.5" cy="0.4" r="0.66">
- <stop offset="0" stop-color="#4b32d6" stop-opacity="0.5"/>
- <stop offset="0.55" stop-color="#1b1650" stop-opacity="0.28"/>
- <stop offset="1" stop-color="${BG}" stop-opacity="0"/>
- </radialGradient></defs>
- <rect width="${OUT_W}" height="${OUT_H}" fill="${BG}"/>
- <rect width="${OUT_W}" height="${OUT_H}" fill="url(#g)"/>
-</svg>`);
-
- const bezel = Buffer.from(`<svg xmlns="http://www.w3.org/2000/svg" width="${OUT_W}" height="${OUT_H}">
- <defs>
- <linearGradient id="body" x1="0" y1="0" x2="1" y2="1">
- <stop offset="0" stop-color="#2f3350"/><stop offset="0.58" stop-color="#12131f"/><stop offset="1" stop-color="#262a40"/>
- </linearGradient>
- <mask id="hole">
- <rect width="${OUT_W}" height="${OUT_H}" fill="black"/>
- <rect x="${BODY_X}" y="${BODY_Y}" width="${BODY_W}" height="${BODY_H}" rx="62" fill="white"/>
- <rect x="${SCREEN_X}" y="${SCREEN_Y}" width="${SCREEN_W}" height="${SCREEN_H}" rx="44" fill="black"/>
- </mask>
- </defs>
- <rect width="${OUT_W}" height="${OUT_H}" fill="url(#body)" mask="url(#hole)"/>
-</svg>`);
-
- const bgPath = path.join(dir, 'bg.png');
- const bezelPath = path.join(dir, 'bezel.png');
- await sharp(bg).png().toFile(bgPath);
- await sharp(bezel).png().toFile(bezelPath);
- return { bgPath, bezelPath };
+ mkdirSync(frames, { recursive: true });
+
+ const browser = await chromium.launch(launchOptions());
+ const ctx = await browser.newContext(contextOptions({ authed: Boolean(args.authed) }));
+
+ const page = await ctx.newPage();
+ let n = 0;
+ const shoot = () => page.screenshot({
+ path: path.join(frames, `${String(n++).padStart(6, '0')}.jpg`),
+ type: 'jpeg',
+ quality: 94,
+ });
+
+ log(`opening ${ORIGIN}/seeker at ${PANEL.width}x${PANEL.height}`);
+ /* `load` never settles on this page (the 3D scene keeps streaming), so the
+ ready signal is the hero being on screen plus a beat for the avatar. */
+ await page.goto(`${ORIGIN}/seeker`, { waitUntil: 'domcontentloaded', timeout: 60_000 });
+ await page.waitForSelector('#hero-title', { timeout: 30_000 });
+ await page.waitForTimeout(4000);
+
+ for (const step of TOUR) {
+ log(step.note);
+ if (step.click) {
+ const target = page.locator(step.click).first();
+ await target.waitFor({ state: 'visible', timeout: 15_000 });
+ await target.click();
+ await page.waitForLoadState('domcontentloaded', { timeout: 30_000 }).catch(() => {});
+ await page.waitForTimeout(2500);
+ }
+ if (step.to !== undefined) {
+ const from = await page.evaluate(() => window.scrollY);
+ const { y, error } = await resolveTarget(page, step.to);
+ if (error) throw new Error(`tour target not on the page: ${error}`);
+ const total = frameCount(step.glide ?? 1200);
+ for (let i = 1; i <= total; i += 1) {
+ await page.evaluate((py) => window.scrollTo(0, py), from + (y - from) * ease(i / total));
+ await shoot();
+ }
+ }
+ for (let i = 0; i < frameCount(step.hold ?? 600); i += 1) await shoot();
+ }
+
+ await ctx.close();
+ await browser.close();
+ log(`captured ${n} frames (${(n / FPS).toFixed(1)}s at ${FPS}fps)`);
+ return n;
}
mkdirSync(OUT, { recursive: true });
@@ -228,26 +151,10 @@ const screenMp4 = path.join(OUT, 'seeker-screen.mp4');
const deviceMp4 = path.join(OUT, 'seeker-device.mp4');
log('encoding the raw panel');
-ffmpeg(['-framerate', String(FPS), '-i', glob, '-c:v', 'libx264', '-crf', '20', '-preset', 'medium',
- '-pix_fmt', 'yuv420p', '-movflags', '+faststart', screenMp4]);
+encodeScreen({ glob, fps: FPS, out: screenMp4 });
log('seating the panel in the device body');
-const { bgPath, bezelPath } = await chrome(work);
-/* Both stills are looped, so they never end on their own, and overlay's own
- `shortest` does not reliably bound them: it left a composite growing past
- 50 MB on a 30 second capture. The frame count does bound them, so every
- looped input and the output carry an explicit -t. */
-ffmpeg([
- '-loop', '1', '-t', seconds, '-i', bgPath,
- '-framerate', String(FPS), '-i', glob,
- '-loop', '1', '-t', seconds, '-i', bezelPath,
- '-filter_complex',
- `[1:v]scale=${SCREEN_W}:${SCREEN_H}:flags=lanczos,setpts=PTS-STARTPTS[s];` +
- `[0:v][s]overlay=${SCREEN_X}:${SCREEN_Y}[a];` +
- `[a][2:v]overlay=0:0,format=yuv420p[v]`,
- '-map', '[v]', '-t', seconds, '-r', String(FPS), '-c:v', 'libx264', '-crf', '20', '-preset', 'medium',
- '-movflags', '+faststart', deviceMp4,
-]);
+await encodeDevice({ glob, fps: FPS, out: deviceMp4, work, seconds });
rmSync(work, { recursive: true, force: true });
log(`wrote ${path.relative(ROOT, screenMp4)} (${PANEL.width}x${PANEL.height}, ${seconds}s)`);
diff --git a/src/drive/index.js b/src/drive/index.js
index a29349c00..73073ef6a 100644
--- a/src/drive/index.js
+++ b/src/drive/index.js
@@ -18,7 +18,7 @@
import { TalkScene } from '../voice/talk-scene.js';
import { AvatarMouthTarget } from '../voice/avatar-morph-target.js';
import { TalkController } from '../voice/talk-controller.js';
-import { detectSurface, surfaceProfile, applySurface } from './surface.js';
+import { detectSurface, surfaceProfile, applySurface, approvalDisposition } from './surface.js';
import { createBridge } from './bridge.js';
import { createDriveInterceptor } from './commands.js';
import { watchMotion } from './motion.js';
@@ -68,6 +68,9 @@ const ui = {
pickerClose: el('dr-picker-close'),
pickerGrid: el('dr-picker-grid'),
pickerSub: el('dr-picker-sub'),
+ confirm: el('dr-confirm'),
+ confirmClose: el('dr-confirm-close'),
+ confirmSlot: el('dr-confirm-slot'),
type: el('dr-type'),
typeClose: el('dr-type-close'),
typeForm: el('dr-type-form'),
@@ -204,6 +207,7 @@ function systemPrompt() {
'Never ask them to read anything, tap through a list, or look at the screen.',
'Never produce markdown, bullet points, code, links, or emoji.',
'If something genuinely needs a screen, say you will have it waiting when they park.',
+ 'A lock, a garage door or an alarm always needs their hands on the screen, so offer it for when they stop rather than promising it now.',
'If you are not sure what they said over road noise, ask one short clarifying question.',
].join(' ');
}
@@ -273,7 +277,7 @@ function clearRearm() {
function maybeRearm() {
clearRearm();
if (!handsFree || !controller) return;
- if (document.hidden || !ui.type.hidden || !ui.picker.hidden) return;
+ if (document.hidden || !ui.type.hidden || !ui.picker.hidden || !ui.confirm.hidden) return;
rearmTimer = setTimeout(() => {
rearmTimer = 0;
if (!handsFree || !controller || controller.state !== 'idle') return;
@@ -381,6 +385,73 @@ function greet() {
scene?.playEmoteOnce('wave', { settleTo: 'idle' }).catch(() => {});
}
+// The one thing the agent is not allowed to do on its own.
+//
+// A lock, a garage door, an alarm: the server never executes these. It freezes
+// the action and hands back a confirmation only a person can redeem, over a
+// session-and-CSRF endpoint no model can reach. The car is the hardest place to
+// honour that, because the person is driving. So the rule here is simple and it
+// does not bend: while the wheels are turning, the answer is no, said out loud,
+// with an offer for when they stop. Parked, it is one tap on the sentence the
+// SERVER composed, never the model's.
+async function onHomeEvent(evt) {
+ if (evt?.status !== 'pending_confirmation') return;
+ const payload = evt.data;
+ if (!payload?.confirmation?.id || !payload?.home?.id) return;
+ const sentence = String(payload.confirmation.summary || 'That changes something in your home.').slice(0, 240);
+
+ const disposition = approvalDisposition(profile, moving);
+ if (disposition !== 'approve') {
+ setSaid(sentence);
+ controller?.speakText(
+ disposition === 'moving'
+ ? `${sentence} Not while we are moving. Pull over and it is one tap.`
+ : `${sentence} I cannot take that one here, because you cannot see what you would be approving. Confirm it on your phone and it will go through.`,
+ );
+ return;
+ }
+
+ clearRearm();
+ stopVad();
+ let renderHomeConfirmation;
+ try {
+ ({ renderHomeConfirmation } = await import('../home-confirm-card.js'));
+ } catch (err) {
+ log.warn('[drive] confirmation card unavailable:', err?.message);
+ controller?.speakText(`${sentence} Approve it on your phone and it will go through.`);
+ return;
+ }
+
+ ui.confirmSlot.replaceChildren();
+ const card = renderHomeConfirmation(ui.confirmSlot, payload, {
+ onResolved: (result) => {
+ const spoken = result?.ok
+ ? 'Done.'
+ : result?.declined
+ ? 'Left alone.'
+ : String(result?.message || 'That did not go through.');
+ setSaid(spoken);
+ controller?.speakText(spoken);
+ // Let the card's own end state stay on screen for a beat before the
+ // sheet closes, so the outcome is read as well as heard.
+ setTimeout(closeConfirm, 2200);
+ },
+ });
+ if (!card) return;
+
+ ui.confirm.hidden = false;
+ ui.confirmClose.focus();
+ setSaid(sentence);
+ controller?.speakText(`${sentence} Approve it on the screen and it will go through.`);
+}
+
+function closeConfirm() {
+ if (ui.confirm.hidden) return;
+ ui.confirm.hidden = true;
+ ui.confirmSlot.replaceChildren();
+ if (handsFree) maybeRearm();
+}
+
// ── session ──────────────────────────────────────────────────────────────────
async function startSession(avatar) {
@@ -399,21 +470,27 @@ async function startSession(avatar) {
}
mouth?.dispose();
- scene = new TalkScene();
+ // The mouth target is safe unattached (it no-ops until a model binds), which
+ // is what lets the audio-only surface run the identical controller.
mouth = new AvatarMouthTarget();
- try {
- await scene.mount({ container: ui.stage, glbUrl: avatar.model_url, cameraPreset: 'half' });
- scene.attachMouthTarget(mouth);
- await ensureMotion();
- } catch (err) {
- setStatus('Ready');
- setFault(`Could not load ${avatar.name || 'that agent'}: ${err.message}`, {
- label: 'Choose another',
- run: () => openPicker(true),
- code: 'model',
- });
- return;
+ if (profile.renders3d) {
+ scene = new TalkScene();
+ try {
+ await scene.mount({ container: ui.stage, glbUrl: avatar.model_url, cameraPreset: 'half' });
+ scene.attachMouthTarget(mouth);
+ await ensureMotion();
+ } catch (err) {
+ setStatus('Ready');
+ setFault(`Could not load ${avatar.name || 'that agent'}: ${err.message}`, {
+ label: 'Choose another',
+ run: () => openPicker(true),
+ code: 'model',
+ });
+ return;
+ }
+ } else {
+ ui.stage.hidden = true;
}
controller = new TalkController({
@@ -421,6 +498,7 @@ async function startSession(avatar) {
systemPromptFn: systemPrompt,
mouthTarget: mouth,
commandInterceptor: interceptor,
+ onHomeEvent,
onMessage: (m) => {
if (m.role === 'user') setHeard(m.content);
else setSaid(m.content);
@@ -619,7 +697,8 @@ function wire() {
beginListening();
}
} else if (e.key === 'Escape') {
- if (!ui.type.hidden) closeType();
+ if (!ui.confirm.hidden) closeConfirm();
+ else if (!ui.type.hidden) closeType();
else if (!ui.picker.hidden && copilot) closePicker();
else hush();
}
@@ -649,6 +728,7 @@ function wire() {
});
ui.pickerClose.addEventListener('click', closePicker);
ui.typeClose.addEventListener('click', closeType);
+ ui.confirmClose.addEventListener('click', closeConfirm);
ui.typeForm.addEventListener('submit', (e) => {
e.preventDefault();
const text = ui.typeInput.value.trim();
diff --git a/src/drive/surface.js b/src/drive/surface.js
index 627a29cbe..6baecb080 100644
--- a/src/drive/surface.js
+++ b/src/drive/surface.js
@@ -63,6 +63,16 @@ export function surfaceProfile(surface = detectSurface()) {
// Hands free is opt-in everywhere, but a car panel is the only place it
// is the better default once the driver has enabled it once.
remembersHandsFree: surface !== 'browser',
+ // Is there anywhere to draw the agent? On Android Auto the page runs in a
+ // service-hosted web view with no window at all, because the phone screen
+ // is Android Auto's during a drive and the car screen is templates. No
+ // window means no animation frames, so the 3D stage is skipped and the
+ // loop runs as audio: the same conversation, no renderer.
+ renders3d: surface !== 'androidauto',
+ // Can a person tap to approve a physical action here? A confirmation the
+ // user cannot SEE is a recognizer being trusted with a lock, which the
+ // home safety doctrine refuses (src/voice/home-voice.js).
+ canConfirm: surface !== 'androidauto',
};
}
@@ -82,3 +92,28 @@ export function applySurface(profile, doc = document, win = window) {
win.addEventListener('resize', sync, { passive: true });
return () => win.removeEventListener('resize', sync);
}
+
+/**
+ * May a physical home action be approved right now?
+ *
+ * Two independent refusals, and either one is enough:
+ *
+ * - **No screen to look at.** Approving a lock you cannot see is trusting a
+ * recognizer with a door, which the home safety doctrine refuses outright
+ * (src/voice/home-voice.js). That is the Android Auto case, where the page
+ * runs in a web view with no window.
+ * - **The car is moving.** A confirmation is a sentence to read and a target
+ * to hit, which is exactly what no in-car guideline permits at speed.
+ *
+ * The answer to a refusal is never silence: the caller says so out loud and
+ * offers it for when the car stops.
+ *
+ * @param {{ canConfirm: boolean }} profile
+ * @param {boolean} moving
+ * @returns {'approve'|'no-screen'|'moving'}
+ */
+export function approvalDisposition(profile, moving) {
+ if (!profile?.canConfirm) return 'no-screen';
+ if (moving) return 'moving';
+ return 'approve';
+}
diff --git a/tests/drive-surface.test.js b/tests/drive-surface.test.js
index 5ec2184ca..1df046bd8 100644
--- a/tests/drive-surface.test.js
+++ b/tests/drive-surface.test.js
@@ -5,7 +5,7 @@
// or a command that silently eats a real question.
import { describe, it, expect, vi } from 'vitest';
-import { detectSurface, scaleFor, surfaceProfile } from '../src/drive/surface.js';
+import { detectSurface, scaleFor, surfaceProfile, approvalDisposition } from '../src/drive/surface.js';
import { matchDriveCommand, normalize, createDriveInterceptor, commandVocabulary } from '../src/drive/commands.js';
import { watchMotion } from '../src/drive/motion.js';
@@ -36,6 +36,50 @@ describe('drive surface detection', () => {
expect(surfaceProfile('carplay').native).toBe(true);
expect(surfaceProfile('browser').native).toBe(false);
});
+
+ // Android Auto runs the page in a service-hosted web view with no window, so
+ // there are no animation frames and nothing to look at. Both flags follow
+ // from that one fact, and both change behaviour, so both are pinned.
+ it('turns the 3D stage off only where there is no window to draw in', () => {
+ expect(surfaceProfile('androidauto').renders3d).toBe(false);
+ for (const name of ['carplay', 'headunit', 'cradle', 'browser']) {
+ expect(surfaceProfile(name).renders3d).toBe(true);
+ }
+ });
+
+ it('refuses tap-to-approve where the person cannot see what they are approving', () => {
+ expect(surfaceProfile('androidauto').canConfirm).toBe(false);
+ for (const name of ['carplay', 'headunit', 'cradle', 'browser']) {
+ expect(surfaceProfile(name).canConfirm).toBe(true);
+ }
+ });
+});
+
+// Approving a lock is the one thing in the car that must never quietly happen.
+// Both refusals are load-bearing, so both are pinned here rather than left to a
+// reading of the UI code.
+describe('approving a physical home action', () => {
+ it('allows a tap only on a parked car with a screen', () => {
+ expect(approvalDisposition(surfaceProfile('headunit'), false)).toBe('approve');
+ expect(approvalDisposition(surfaceProfile('cradle'), false)).toBe('approve');
+ expect(approvalDisposition(surfaceProfile('carplay'), false)).toBe('approve');
+ });
+
+ it('refuses while the car is moving, on every surface', () => {
+ for (const name of ['carplay', 'headunit', 'cradle', 'browser']) {
+ expect(approvalDisposition(surfaceProfile(name), true)).toBe('moving');
+ }
+ });
+
+ it('refuses on a surface with nothing to look at, parked or not', () => {
+ expect(approvalDisposition(surfaceProfile('androidauto'), false)).toBe('no-screen');
+ expect(approvalDisposition(surfaceProfile('androidauto'), true)).toBe('no-screen');
+ });
+
+ it('refuses when handed no profile at all rather than defaulting open', () => {
+ expect(approvalDisposition(null, false)).toBe('no-screen');
+ expect(approvalDisposition(undefined, false)).toBe('no-screen');
+ });
});
describe('drive commands', () => {