Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(glasses): document how agents reach a lens, with a Meta Ray-Ban Display build plan
nirholas7328bb2df8d1
diff --git a/data/changelog.json b/data/changelog.jsonindex 7abf7badb..be7508614 100644--- a/data/changelog.json+++ b/data/changelog.json@@ -28275,6 +28275,16 @@"improvement","fix"]+ },+ {+ "date": "2026-09-17",+ "title": "A plan for meeting agents through smart glasses",+ "summary": "three.ws already lets you pin a 3D agent at a real place and walk up to it, and /irl can already mirror the nearest-agent cue to Brilliant Labs Frame and Even Realities G1 lenses over Bluetooth. New documentation maps the whole glasses lane: how the work splits between the lens (who is near, which way, how far) and the phone (the 3D body in AR), a full build plan for Meta Ray-Ban Display glasses and the Neural Band, and an honest survey of every platform, repository and package involved, including which ones have shut down and should not be built on.",+ "tags": [+ "docs",+ "feature"+ ],+ "link": "/docs/glasses"}]}diff --git a/data/pages.json b/data/pages.jsonindex 7eef15d5c..786888dad 100644--- a/data/pages.json+++ b/data/pages.json@@ -4941,6 +4941,14 @@"priority": 0.6,"changefreq": "monthly"},+ {+ "path": "/docs/glasses",+ "title": "Docs · Smart glasses, agents on your lens",+ "description": "Wear smart glasses and the world tells you an agent is standing thirty metres away. How three.ws splits the work: the lens carries the signal, the phone carries the 3D body, and the presence-gated IRL rail carries the truth.",+ "priority": 0.6,+ "changefreq": "monthly",+ "added": "2026-09-17"+ },{"path": "/docs/js-api","title": "Docs · JavaScript API reference",diff --git a/docs/glasses.md b/docs/glasses.mdnew file mode 100644index 000000000..ed136632f--- /dev/null+++ b/docs/glasses.md@@ -0,0 +1,127 @@+# three.ws on smart glasses++Agents already stand at real GPS coordinates. [IRL](/docs/irl) pins them to a+park bench, a shop doorway, a conference table, and anyone who physically walks+up can see the avatar in AR, talk to it, and pay it. The missing half is the+walking up: today you only find an agent if you are already holding a phone with+`/irl` open.++Smart glasses close that gap. They are the sense organ. You wear them, and the+world tells you an agent is thirty metres to your left before you ever reach for+a phone.++This page is the map of that work: what ships today, what is being built, and+the one architectural constraint every glasses lane obeys.++---++## The constraint that shapes everything++**No smart-glasses display in 2026 can render a three.ws avatar.** Not one.++- Brilliant Labs Frame and Even Realities G1 are thin Bluetooth screens. No+ browser, no WebGL, no WebXR.+- Meta Ray-Ban Display is a 600x600 additive monocular panel driven over+ Bluetooth from the phone. Its native SDK offers a fixed component set (text,+ image, button, icon, video, flexbox) and states plainly that custom pixel and+ 3D rendering are not supported. The web-app lane gives you a DOM at 600x600,+ but the same Bluetooth bandwidth ceiling applies: Meta's own guidance is to+ keep layouts to text and small images, and caps video at 400px per side and+ 70,000 total pixels.+- Everything with real 6DoF passthrough (Quest, Vision Pro) is a headset you do+ not wear down the street.++So we never try. The division of labour is fixed:++| Layer | Runs on | Carries |+|---|---|---|+| **The signal** | The glasses | Who is near, which way, how far, what they said |+| **The body** | The phone (or the headset) | The 3D avatar, standing in the real world, in AR |+| **The truth** | `api/irl/*` | Presence tokens, pins, conversation, interactions |++The glasses give you the summons. The phone gives you the meeting. The one time+this is not a compromise is the best moment in the product: you are walking,+your lens quietly says `Scout, 24 m, turn left`, and you decide whether to stop.++---++## Lane 1: companion BLE glasses (live)++Live in [/irl](https://three.ws/irl) behind the "Connect glasses" button.++`/irl` already computes the nearest-agent cue for its on-screen directional+nudge ([src/irl/proximity-cue.js](../src/irl/proximity-cue.js)). The bridge+reshapes that same signal into a compact heads-up frame and streams it to a lens+over Web Bluetooth.++| Piece | File |+|---|---|+| Pure HUD protocol (formatting, arrow glyphs, wire framing) | [src/irl/glasses/protocol.js](../src/irl/glasses/protocol.js) |+| Device-agnostic controller (rate limiting, heartbeat, recovery) | [src/irl/glasses/bridge.js](../src/irl/glasses/bridge.js) |+| Web Bluetooth transport + capability gate | [src/irl/glasses/transport.js](../src/irl/glasses/transport.js) |+| Brilliant Labs Frame adapter | [src/irl/glasses/frame.js](../src/irl/glasses/frame.js) |+| Even Realities G1 adapter (two-arm staged pairing) | [src/irl/glasses/g1.js](../src/irl/glasses/g1.js) |+| Connect sheet (capability gate, picker, connected preview, retry) | [src/irl/glasses/connect-ui.js](../src/irl/glasses/connect-ui.js) |+| Tests | [tests/irl-glasses-protocol.test.js](../tests/irl-glasses-protocol.test.js) |++Push discipline: 3 Hz ceiling on display writes, single-flight (BLE writes never+overlap), a 4 s heartbeat that re-sends an unchanged frame to recover a dropped+write, and transient announcements that hold the lens for 3.5 s. A connected+link stamps its interaction telemetry as a `glasses` encounter, so the analytics+rollup can tell a lens discovery from a screen one.++Web Bluetooth is Chromium-only, so this lane does not exist on iOS Safari. The+connect sheet says so honestly instead of failing at pair time.++## Lane 2: Meta Ray-Ban Display web app (in build)++The inversion of lane 1. Ray-Ban Display exposes no Bluetooth API to third+parties; instead **the glasses run our web app**. A publicly hosted HTTPS page+loads on the lens with GPS from the paired phone, motion and orientation from+the glasses, and Neural Band input delivered as arrow keys and Enter.++That is enough to run the whole IRL discovery loop natively on your face, with+no phone in your hand. The build plan, the interaction design for the Neural+Band, and the phased checklist live in+[Meta Ray-Ban Display](/docs/glasses/meta-ray-ban-display).++## Lane 3: native Device Access Toolkit (in build)++The web-app lane has no camera and no microphone. Meta's native toolkit does:+camera streaming, photo capture, mic, and speakers over Bluetooth profiles, from+inside an existing iOS or Android app. We already ship an iOS app shell+([ios/](../ios/README.md)), which is where that integration belongs.++What it unlocks that lane 2 cannot: **spoken** conversation with a street agent+(the same `TalkController` that powers+[src/irl/pin-talk.js](../src/irl/pin-talk.js)), and camera frames good enough+for visual re-localisation, so an agent stands where it was placed to the+centimetre rather than to the five metres GPS gives you.++---++## What every lane rides on++Nothing above re-implements IRL. All three lanes speak the same presence-gated+contract, documented in [IRL](/docs/irl) and shipped as+[`@three-ws/irl`](https://www.npmjs.com/package/@three-ws/irl):++- `POST /api/irl/fix-token` mints a short-lived (180 s) proof-of-presence token+ from a real fix, anchored to a coarsened (~110 m) point.+- `GET /api/irl/pins?lat=&lng=&radius=` answers only for the area that token was+ minted in, radius clamped to 10 to 60 m, at most 50 pins, sweep-detected.+- `POST /api/chat` carries the conversation in the agent's own persona.+- `POST /api/irl/interactions` logs the encounter, never the transcript, never+ the caller's position.++There is no browseable map and no "query any point on earth", on a lens or+anywhere else. You see what is around you because you are there. The reasoning+is in [docs/irl/THREAT-MODEL.md](irl/THREAT-MODEL.md).++## Related++- [IRL: agents in the real world](/docs/irl)+- [IRL street demo runbook](/docs/irl/street-demo)+- [World Lines: proof of presence](/docs/world-lines)+- [AR Studio](/docs/ar-studio) and [AR exports](/docs/ar)+- [Platform and package research for the glasses lanes](/docs/glasses/research)diff --git a/docs/glasses/meta-ray-ban-display.md b/docs/glasses/meta-ray-ban-display.mdnew file mode 100644index 000000000..dea0b7346--- /dev/null+++ b/docs/glasses/meta-ray-ban-display.md@@ -0,0 +1,359 @@+# Meta Ray-Ban Display: agents you can walk up to++**Goal: you wear Ray-Ban Display glasses, walk down a street, and your lens+tells you a three.ws agent is standing 24 metres away. You walk up. It knows you+arrived. You talk to it with a pinch of your fingers, no phone in your hand, and+if you want to see its body you hand off to AR.**++This is the build plan for that. It rides entirely on the existing+[IRL](/docs/irl) rail: the pins, the presence tokens, the personas, and the+conversation are already live and already shipped as+[`@three-ws/irl`](https://www.npmjs.com/package/@three-ws/irl). What is new is+one client, written for a 600x600 additive panel and a wristband that speaks+four arrow keys and Enter.++Platform background and the other glasses lanes: [three.ws on smart+glasses](/docs/glasses). Verified platform facts and every repo or package+evaluated: [research](/docs/glasses/research).++---++## What the platform gives us++Verified against Meta's developer documentation on 2026-09-17. Two paths exist,+and they have opposite strengths.++| | Web Apps | Device Access Toolkit (native) |+|---|---|---|+| Runs | On the glasses, from an HTTPS URL | Inside your own iOS or Android app |+| Language | HTML, CSS, JavaScript | Swift, Kotlin |+| Display | Your own DOM at 600x600 | A fixed component set (text, image, button, icon, video, flexbox) |+| Input | Neural Band and captouch, as `ArrowUp/Down/Left/Right`, `Enter`, `Escape` | Same gestures, native callbacks |+| Sensors | `devicemotion`, `deviceorientation` (glasses), `navigator.geolocation` (phone, 5 to 50 m) | Same, plus device session state |+| Camera | Not available | Video streaming and photo capture |+| Microphone | Not available | Available |+| Speakers | Not a listed capability | Available over Bluetooth profiles |+| Storage | `localStorage` and `sessionStorage`, 5 MB each | Native |+| Custom 3D | Not available in practice (Bluetooth bandwidth) | Explicitly not supported |+| Distribution | Any HTTPS URL, added by QR or Meta AI app settings | App registration, release channels |+| Publishing | Not open during developer preview | Not open during developer preview |++Hard requirements: Meta Ray-Ban Display glasses on software v125 or later, the+Meta AI app v272 or later with Developer Mode enabled, and an HTTPS origin with+a valid certificate. HTTP is refused.++Two consequences drive the whole design:++1. **The lens is a HUD, not a canvas.** Black is transparent on an additive+ display, complex layouts lag over Bluetooth, and there is no 3D. We render+ type, one small portrait, and a direction. The avatar's body lives on the+ phone.+2. **There is no keyboard and no microphone on the web lane.** A conversation+ has to be winnable with a d-pad and a pinch. That is a design constraint, and+ it produces a better street interaction than a tiny keyboard would.++---++## The surface++A new page at **`/glass`**, served from this repo like every other page.++`/irl` is an 8,500-line three.js application. It cannot be the thing the lens+loads. `/glass` is a separate, deliberately tiny DOM surface (target: under+60 KB of JavaScript, no three.js, no WebGL) that speaks the same API and imports+the same pure helpers, so the two lanes can never disagree about what "24 m,+turn left" means.++```+pages/glass.html the 600x600 shell, additive-safe, no chrome+src/glass/main.js boot, view stack, lifecycle+src/glass/dpad.js roving focus over .focusable, Enter / Escape+src/glass/presence.js check-in, token refresh, adaptive nearby polling+src/glass/compass.js deviceorientation to screen-relative bearing+src/glass/views/*.js scanning, nearby, agent, talk, handoff, denied+src/glass/hud.css the additive design system+```++Reused unchanged, not reimplemented:++- [src/irl/glasses/protocol.js](../../src/irl/glasses/protocol.js) for+ `formatDistance()`, `normalizeAngle()` and `arrowGlyph()`. The Frame lens, the+ G1 lens and the Ray-Ban Display lens all format a distance the same way+ because they call the same pure function, already covered by+ [tests/irl-glasses-protocol.test.js](../../tests/irl-glasses-protocol.test.js).+- [src/irl/proximity-cue.js](../../src/irl/proximity-cue.js) for the+ nearest-agent selection and the screen-relative bearing convention+ (0 = ahead, positive = turn right, radians).+- The `/api/irl/*` contract, through the same calls+ [`@three-ws/irl`](https://www.npmjs.com/package/@three-ws/irl) makes.++## The design system on an additive panel++- Background: pure `#000`, which is transparent glass. Never a light surface.+- One accent (`#7aa2ff`) and one alert (`#ffd479`), both high-luminance on+ black. Never a mid-grey: it reads as haze over the real world.+- Body type 20px minimum, primary type 24 to 32px. 16px is the floor Meta+ documents, and at street distance 16px is a squint.+- Every interactive element carries `.focusable` and a focus ring that is+ visible at a glance (glow plus border, not an outline).+- One portrait maximum per view, 160px square, fetched from+ `GET /api/render/avatar-clip` (posed, camera-orbited PNG of any GLB, already+ CDN-cached) with a dark background so it composites cleanly.+- No animation that runs continuously. The panel is in someone's eye while they+ cross a road.++## The interaction model++The Neural Band and the temple captouch arrive as keyboard events. That is the+whole input vocabulary:++| Gesture | Event | In `/glass` |+|---|---|---|+| Swipe up / down | `ArrowUp` / `ArrowDown` | Move focus in the nearby list or the reply list |+| Swipe left / right | `ArrowLeft` / `ArrowRight` | Previous / next view, or scrub a long reply |+| Pinch (index) | `Enter` | Select, send, confirm |+| Pinch and hold, drag | continuous | Scrub a long list without discrete steps |+| Back | `Escape` | Pop the view stack |++There are no custom gestures. Meta does not expose the raw sEMG stream to third+parties, so anything a design needs beyond these five inputs does not exist.++## The loop++```+boot+ └─ permissions (orientation needs a user gesture: the first pinch)+ └─ checkIn() POST /api/irl/fix-token (180 s token)+ └─ nearby() GET /api/irl/pins (10 to 60 m)+ ├─ nobody near "Scanning" with a live heading ring+ └─ someone near name · distance · arrow, ranked by distance+ └─ arrival inside the pin radius: the agent greets you+ └─ talk POST /api/chat as that agent+ └─ AR hand off to the phone for the body+```++Polling cadence is adaptive, not a fixed timer: 5 s while the fix is moving,+15 s while it is not, paused entirely when the document is hidden. The presence+token is re-minted when it expires (180 s) or when the precision-7 geohash cell+changes, which is exactly the rule `@three-ws/irl` already documents.++Every state is designed, because a lens with a blank frame is worse than no+lens: permission denied, GPS unavailable, offline, scanning with nothing near,+scanning with the compass uncalibrated, list, card, talking, agent unreachable,+and handed off.++## Conversation without a keyboard or a microphone++The agent's reply comes from `POST /api/chat` in its own persona, exactly as+[src/irl/pin-talk.js](../../src/irl/pin-talk.js) does it today (the pin's+`agent_id` when it has one, the pin persona otherwise).++The visitor's side is the interesting half. Three input lanes, each capability+gated, each real:++1. **Suggested replies (always available).** The chat response carries three+ short replies the visitor could plausibly say next, generated in the same+ completion. The lens lists them; swipe to choose, pinch to send. This is the+ primary lane and it must be good enough to be the only lane.+2. **Neural Handwriting (when the composer reaches us).** Meta's web-app+ documentation lists text input as unsupported, while the on-glasses composer+ exists at the OS level. So we feature-detect it honestly: focus a real input,+ watch for a real `input` event, and only then show the "Write" affordance.+ Never a button that might do nothing.+3. **Audio out (when it plays).** Audio playback is not a documented web-app+ capability. We attempt the agent's real voice through `/api/tts` and verify+ it is actually playing (the `play()` promise resolves and `currentTime`+ advances), falling back to text-only on the lens. Both branches ship; neither+ is a stub.++The native Device Access Toolkit lane has a real microphone and real speakers,+so spoken conversation is Phase 6, not a promise made on the web lane.++## Handing off to the body++The lens cannot render the avatar, and it cannot open a browser on the phone.+So the handoff is explicit and works signed out:++- The agent card offers **See it in AR**.+- Signed-in wearers get a real push through the existing notification rail+ ([api/\_lib/feed.js](../../api/_lib/feed.js)), which opens+ `/irl?pin=<id>` on the phone, the same visit link the printable IRL sign uses.+- Signed-out wearers get a six-character code on the lens, resolving at+ `three.ws/g/<code>` to the same visit link, minted like the existing IRL share+ tokens ([api/irl/share.js](../../api/irl/share.js)).++The same code is how an account claims a lens session, since typing an email+address with a d-pad is not a thing anyone will do: the lens shows a code, the+phone claims it, and the anonymous device token upgrades to the account. Until+then the lens runs in IRL's existing anonymous device-token mode.++---++## Build plan++Each phase is shippable on its own and leaves the platform better than it found+it. Nothing here is gated on Meta approving anything, except distribution.++### Phase 0: access and proof++- [ ] Meta Ray-Ban Display glasses on v125+, Meta AI app v272+, Developer Mode on.+- [ ] Install the Meta Ray-Ban Display Web App Simulator Chrome extension and+ confirm the 600x600 additive preview against a static page.+- [ ] Add the Wearables MCP endpoint (`https://mcp.developer.meta.com/wearables`,+ tool `search_webapps_docs`) to the workspace MCP config so platform limits+ are checked against the source, not against this document.+- [ ] Serve a one-screen `/glass` stub from the live site, add it to the glasses+ by QR, and confirm it loads on the lens. Record what the lens does with+ `deviceorientation` permission prompts.++**Done when:** a page served from `https://three.ws/glass` renders on real+glasses and logs a d-pad event from a real pinch.++### Phase 1: the lens surface++- [ ] `pages/glass.html` with the documented viewport and+ `<meta name="mrbd-web-app-capable" content="yes">`, registered in+ [vite.config.js](../../vite.config.js), [vercel.json](../../vercel.json)+ and [data/pages.json](../../data/pages.json).+- [ ] `src/glass/hud.css`: the additive design system above, as tokens.+- [ ] `src/glass/dpad.js`: roving focus over `.focusable`, wrap-around, `Enter`+ to activate, `Escape` to pop, continuous-drag scrubbing. Pure and unit+ tested (no DOM assumptions beyond an element list).+- [ ] `src/glass/main.js`: view stack, document-visibility lifecycle, a single+ render pass per state change (never a per-event DOM write).+- [ ] Every designed state present, including the ones that are not the happy+ path.++**Done when:** the surface passes `npm run check:dist` and `npm run check:pages`,+renders correctly in the simulator at 600x600, and is fully operable with only+the five inputs.++### Phase 2: the presence loop++- [ ] `src/glass/presence.js`: check-in, token expiry and geohash-cell re-mint,+ adaptive polling, request coalescing, honest error surfacing.+- [ ] `src/glass/compass.js`: `deviceorientation` to a screen-relative bearing,+ `requestPermission()` behind the first pinch, a calibration state when the+ heading is unreliable.+- [ ] Scanning view: heading ring, nearby count, last-fix age.+- [ ] Nearby view: ranked list, each row name + `formatDistance()` ++ `arrowGlyph()` from the shared protocol module.+- [ ] Arrival: crossing into a pin's radius promotes that agent to the lens once,+ with a real interaction logged (`POST /api/irl/interactions`, type+ `glasses`, consistent with the BLE lane's telemetry).++**Done when:** walking toward a real pin moves the distance down and the arrow+tracks the turn, on hardware, with the phone in a pocket.++### Phase 3: conversation++- [ ] Agent card: portrait from `/api/render/avatar-clip`, name, caption, the+ actions (Talk, See it in AR, and its x402 endpoint when the pin carries one).+- [ ] `POST /api/chat` extended with an opt-in `suggestReplies` count, returning+ three short suggestions in the same completion. Wired for `/glass` first,+ then offered to [src/irl/pin-talk.js](../../src/irl/pin-talk.js), which+ wants it too.+- [ ] Talk view: transcript, focused reply list, send on pinch, honest thinking+ and error states, conversation capped to what fits on a lens.+- [ ] Capability gates for the composer and for audio, each proven at runtime+ rather than assumed.+- [ ] One `talk` interaction per conversation, never the transcript, matching+ the existing IRL privacy posture.++**Done when:** a full exchange with a real pinned agent happens on the lens with+no phone interaction, and `forge_creations`-style ground truth (the interaction+log) shows the encounter.++### Phase 4: handing off to the body++- [ ] `api/glass/handoff.js`: mint a six-character code for a pin, resolve+ `three.ws/g/<code>` to `/irl?pin=<id>`, expiring, owner-free, rate limited.+- [ ] Push lane for signed-in wearers through the existing notification rail.+- [ ] `api/glass/pair.js`: claim a lens session from the phone, upgrading the+ anonymous device token to the account.+- [ ] `/irl` recognises an arrival that started on the lens and opens the pin+ card directly, the way the visit link already does.++**Done when:** a pinch on the lens puts the avatar in your hand in AR in under+ten seconds, signed in or not.++### Phase 5: place an agent from the lens++- [ ] Place at your current fix, choosing from your own avatars, with the+ heading taken from where you are actually looking.+- [ ] Confirmation state, then the placed agent appears in your own nearby feed.+- [ ] Honest anonymous-versus-permanent expiry copy, as `/irl` already shows.++**Done when:** an agent can be placed and rediscovered without ever opening a+phone.++### Phase 6: the native lane (camera, voice, precision)++- [ ] Add the Meta Wearables Device Access Toolkit to the iOS app+ ([ios/](../../ios/README.md)) over Swift Package Manager, behind the+ existing Capacitor shell, with MockDeviceKit covering the no-hardware path.+- [ ] Spoken conversation: glasses microphone into the same `TalkController`+ pipeline, agent voice out through the glasses speakers.+- [ ] Camera frames into a re-localisation pass so a pin lands where it was+ placed rather than where GPS guesses, keeping the presence-gated contract+ intact (frames are processed for pose, never uploaded as a location feed).+- [ ] Android parity, same toolkit.++**Done when:** you can hold a spoken conversation with a street agent, hands+free, and the agent stands in the same doorway it was left in.++### Phase 7: ship it++- [ ] `data/pages.json` entry for `/glass` (this is what feeds the changelog,+ the sitemap and `llms.txt`).+- [ ] `data/changelog.json` entry, community language, `feature` tag.+- [ ] A row in [STRUCTURE.md](../../STRUCTURE.md) for the glasses surfaces.+- [ ] `npm run audit:docs` clean, `npm test` green, `npm run check:rules` clean+ on the touched paths.+- [ ] Distribution when Meta opens publishing. Until then, the QR deeplink is+ the install path and it is a perfectly good one for a demo.++---++## Non-goals++- Rendering a three.ws avatar on the lens. The hardware cannot, and pretending+ otherwise would ship a blurry sprite that undersells the product.+- A map of every agent on earth. Presence-gated reads are the contract, on every+ client. See [docs/irl/THREAT-MODEL.md](../irl/THREAT-MODEL.md).+- Custom Neural Band gestures. Meta exposes five inputs. We design for five.+- Continuous camera capture on the native lane. Frames serve a pose fix in the+ moment and are not retained.++## Decisions taken (and their alternatives)++| Decision | Alternative rejected | Why |+|---|---|---|+| A separate `/glass` page | Responsive mode inside `/irl` | `/irl` boots three.js, nipplejs and a WebGL renderer. A lens app that downloads a 3D engine it can never use is not a lens app. |+| Suggested replies as the primary input | Waiting for text input support | A pinch-selected reply is faster on a street than handwriting, and it works today on documented capabilities. |+| Anonymous device token by default | Requiring sign-in | IRL already supports anonymous placement and discovery, and no one is typing a password with a d-pad. |+| Reuse `protocol.js` formatting | A lens-specific formatter | Three lenses that disagree about "24 m" is a bug waiting to happen. One tested pure module. |+| GPS-only in phases 1 to 5 | Visual positioning first | GPS at 5 to 50 m is what the web lane exposes, and IRL's radius is 10 to 60 m, so GPS is inside the envelope. Precision is a Phase 6 upgrade on the native lane, not a prerequisite. |++## Testing++- **Simulator:** the Meta Ray-Ban Display Web App Simulator Chrome extension+ (600x600, additive blending, environment backgrounds, D-pad input, recording).+ Everything except real GPS movement can be exercised there.+- **Desktop:** Chrome DevTools at a 600x600 viewport with sensor emulation, plus+ the repo's existing Playwright lane for the page's states.+- **Unit:** `src/glass/dpad.js`, `src/glass/presence.js` and `src/glass/compass.js`+ are pure enough to test without a DOM, the same way+ [tests/irl-glasses-protocol.test.js](../../tests/irl-glasses-protocol.test.js)+ tests the BLE HUD.+- **On hardware:** add the app by QR from the simulator or the Meta AI app, then+ walk a real pin. The [IRL street demo runbook](/docs/irl/street-demo) is the+ script for that walk.++## Related++- [three.ws on smart glasses](/docs/glasses)+- [Platform and package research](/docs/glasses/research)+- [IRL](/docs/irl), the [street demo runbook](/docs/irl/street-demo) and the+ [threat model](../irl/THREAT-MODEL.md)+- [`@three-ws/irl`](https://www.npmjs.com/package/@three-ws/irl)diff --git a/docs/glasses/research.md b/docs/glasses/research.mdnew file mode 100644index 000000000..c9672fd65--- /dev/null+++ b/docs/glasses/research.md@@ -0,0 +1,115 @@+# Glasses platform research: repos, packages and what we adopt++Everything below was checked on **2026-09-17**. Versions, availability and+shutdown dates move fast in this corner of the industry, so each row says what+was verified rather than asserting a permanent truth. Re-check before depending+on any of it.++The short version: **Meta's own toolkit is the only thing we need to adopt, and+almost everything else worth having we already own.** The 3D avatars, the+geofenced presence rail, the conversation pipeline and a tested smart-glasses+HUD protocol are already in this repository.++---++## Meta's first-party platform (adopt)++| Resource | What it is | Verdict |+|---|---|---|+| [wearables.developer.meta.com/docs](https://wearables.developer.meta.com/docs) | The source of truth for both paths. The [web apps build guide](https://wearables.developer.meta.com/docs/develop/webapps/build/) carries the real constraints: input mapping, sensors, storage, design system. | Adopt. Check limits here, not in blog posts. |+| `https://mcp.developer.meta.com/wearables` | Meta's MCP endpoint, tool `search_webapps_docs`, which answers questions about current display constraints, setup, testing and publishing. | Adopt. Wire it into the workspace MCP config so an agent checks the platform instead of guessing. |+| [facebookincubator/meta-wearables-webapp](https://github.com/facebookincubator/meta-wearables-webapp) | Meta's AI-assisted toolkit for building display web apps: skills covering d-pad navigation, EMG pinch and continuous drag, sensors, storage, offline, deployment, QR generation. Not an npm package, a set of agent skills. | Adopt as reference. Read the skills, keep our own code. |+| [facebook/meta-wearables-dat-ios](https://github.com/facebook/meta-wearables-dat-ios) | Native toolkit for iOS over Swift Package Manager: video streaming, photo capture, display components, session lifecycle, MockDeviceKit for testing without hardware. | Adopt in Phase 6, inside [ios/](../../ios/README.md). |+| [facebook/meta-wearables-dat-android](https://github.com/facebook/meta-wearables-dat-android) | The Android twin. | Adopt at Android parity. |+| [Meta Ray-Ban Display Web App Simulator](https://chromewebstore.google.com/detail/meta-ray-ban-display-web/jpjlmmodokemlepklkdbimceggpbjcll) | Chrome extension: the 600x600 surface with additive blending, environment backgrounds, d-pad input, display tuning, recording, and QR deeplink generation. | Adopt. This is the development loop before hardware. |+| [displayglasses.dev](https://displayglasses.dev/) | Browser-hosted simulator, same idea, no extension install. | Reference. Useful for sharing a preview with someone who has no extension. |++Platform facts worth repeating because they change the architecture:++- Web apps get **no camera and no microphone**. Native gets both, plus speakers+ over Bluetooth profiles.+- The native display API is a **fixed component set**, and custom pixel or 3D+ rendering is explicitly unsupported. Video is capped under 400px per side and+ 70,000 total pixels.+- **No custom Neural Band gestures.** Predefined swipes and pinches only, and+ they arrive as arrow keys and Enter. The raw sEMG stream is not exposed.+- **Publishing is closed during developer preview.** Distribution today is a+ URL and a QR code, which is enough for a demo and a pilot.+- Mock Device Kit does not cover the display glasses, so display work needs+ either the simulator or hardware.++## Community work worth reading (reference, not dependency)++| Repo or site | Why it is useful |+|---|---|+| [ggoonnzzaallo/meta_display](https://github.com/ggoonnzzaallo/meta_display) | Working web apps written for real Ray-Ban Display hardware. The fastest way to see what the lens actually does with a DOM. |+| [kevinCCme/XR_WEARABLE](https://github.com/kevinCCme/XR_WEARABLE) | Early third-party experiments against the toolkit. |+| [glasskit.app/docs](https://glasskit.app/docs/meta-ray-ban-display-sdk-explained) | A clear third-party explainer of the two SDK paths, useful for onboarding someone new to the platform. |++## Neuromotor input research (context, not a dependency)++We cannot read the band ourselves, so none of this ships. It is here because it+explains what the band can and cannot do, which is what makes the five-input+design defensible rather than lazy.++- [facebookresearch/emg2qwerty](https://github.com/facebookresearch/emg2qwerty):+ the open sEMG typing dataset and baselines (108 users, 1,135 sessions, 346+ hours), with character error rates under 10% once a language model refines the+ decode. That threshold is why on-glasses handwriting exists at all.+- `emg2pose`, the companion pose-estimation dataset from the same release.+- ["A generic non-invasive neuromotor interface for human-computer+ interaction"](https://www.nature.com/articles/s41586-025-09255-w), Nature,+ 2025: the research line the Neural Band productises.++## Geospatial anchoring (mostly avoid, and here is why)++The obvious idea for "an avatar standing exactly here" is a visual positioning+system. The 2026 landscape does not reward that bet for a web client:++| Option | Status on 2026-09-17 | Verdict |+|---|---|---|+| 8th Wall (WebAR, hosted VPS) | Shut down. Hosted services offline, access ended 28 Feb 2026, existing hosted campaigns run until 28 Feb 2027. The engine went open source (MIT) minus SLAM. | Avoid. |+| Niantic Spatial VPS on the web | Not supported on the distributed engine binary or the open-source engine. Legacy hosted projects only, through Feb 2027. | Avoid for web. |+| Google ARCore Geospatial API | Alive and strong, but native Android and Unity, not a browser API. | Phase 6 candidate on the native lane only. |+| Immersal | Native SDKs plus a REST lane, needs site mapping per location. | Revisit if a specific venue needs centimetre placement. |+| [OpenFLAME](https://arxiv.org/html/2510.03915v1) | Research on federated VPS with a common API, exactly the interoperability gap that makes every option above a lock-in. | Watch. |+| [locar](https://www.npmjs.com/package/locar) (LocAR.js, AR.js org) | 0.2.12, MIT, the location-based part of AR.js rebuilt for current three.js. GPS to world coordinates plus device-orientation controls. | Not needed. [src/irl.js](../../src/irl.js) and [src/ar/](../../src/ar) already do this with WebXR hit-test anchoring, and the lens cannot render 3D at all. Worth reading if the phone-side geo projection is ever rewritten. |+| `geolib`, `@turf/turf` | Solid MIT geospatial maths. | Not needed. `api/_lib/geohash.js` and the IRL distance maths already cover the whole surface, and a lens app should not ship a geospatial library to compute one bearing. |++The conclusion that matters: **GPS is inside our envelope.** The web lane gives+5 to 50 m accuracy, IRL's discovery radius is 10 to 60 m, so a pin resolves+correctly today with no VPS at all. Precision is a Phase 6 upgrade on the native+camera lane, not a prerequisite for shipping.++## What we already own (the real head start)++| Ours | Where | What it removes from the plan |+|---|---|---|+| Geofenced presence rail | [api/irl/](../../api/irl/README.md), [packages/irl/](../../packages/irl/README.md) | Presence tokens, nearby reads, pins, interactions, money drops, world lines |+| Tested smart-glasses HUD protocol | [src/irl/glasses/protocol.js](../../src/irl/glasses/protocol.js) | Distance formatting, direction glyphs, wire framing, all unit tested |+| Two live BLE glasses adapters | [src/irl/glasses/](../../src/irl/glasses) | Proof the HUD model works on a real lens |+| Walk-up conversation | [src/irl/pin-talk.js](../../src/irl/pin-talk.js) | Persona, memory, voice, lipsync, barge-in |+| Server-side avatar rendering | `GET /api/avatar/render`, `GET /api/render/avatar-clip` ([media API](/docs/media-api)) | The lens portrait, CDN-cached, no client 3D |+| AR handoff | [src/ar/](../../src/ar), [api/ar.js](../../api/ar.js), [/irl](https://three.ws/irl) | WebXR anchoring, Quick Look, Scene Viewer, visit links |+| iOS app shell | [ios/](../../ios/README.md) | Where the native toolkit lands in Phase 6 |++## How to re-verify this page++```bash+# Platform limits, straight from Meta's MCP endpoint+# endpoint: https://mcp.developer.meta.com/wearables+# tool: search_webapps_docs++# Package facts+npm view locar version license+```++Check the [build guide](https://wearables.developer.meta.com/docs/develop/webapps/build/)+and the [FAQ](https://developers.meta.com/wearables/faq/) for capability changes,+especially camera, microphone, audio playback and publishing, all of which are+the difference between a phase being possible and being blocked.++## Related++- [three.ws on smart glasses](/docs/glasses)+- [Meta Ray-Ban Display build plan](/docs/glasses/meta-ray-ban-display)diff --git a/docs/nav.json b/docs/nav.jsonindex 781aa6d0d..5afcbd08c 100644--- a/docs/nav.json+++ b/docs/nav.json@@ -543,6 +543,18 @@"label": "IRL street demo (visit links and signs)","path": "irl/street-demo"},+ {+ "label": "Smart glasses (agents on your lens)",+ "path": "glasses"+ },+ {+ "label": "Meta Ray-Ban Display build plan",+ "path": "glasses/meta-ray-ban-display"+ },+ {+ "label": "Glasses platform research",+ "path": "glasses/research"+ },{"label": "World Lines (proof of presence)","path": "world-lines"