Code-first video & image, rendered natively on WebGPU.
replace yolo
okanasla9782addb8eb
diff --git a/.agent/rules/general-information.md b/.agent/rules/general-information.mdindex e5708be4..9083ae5e 100644--- a/.agent/rules/general-information.md+++ b/.agent/rules/general-information.md@@ -28,7 +28,7 @@ Gitframes is a code-first video engine on native WebGPU (Dawn) in Node, with no- `packages/compositions`: layout (Yoga, `utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), animation and evaluator, `to-virtual-media`.- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text, audio DSP, ChartGPU bridge.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny, `HeadlessMediaRenderer`).-- `packages/core`: shared types, `Effect` base, signals. Other packages: `renderers`, `media`, `node-sdk` (node authoring: `shared/` + `renderer/`), `server-utils`, `client-utils`, `tensor-webgpu`, `yolo`, `tsconfig`.+- `packages/core`: shared types, `Effect` base, signals. Other packages: `renderers`, `media`, `node-sdk` (node authoring: `shared/` + `renderer/`), `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.- `nodes/node-*` (59): one VFX/audio/layout node each. Layout: `src/shared/` (config and schema), `src/renderers/` (WebGPU), `src/index.ts`. Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (`packages/compositions/src/program/skillmd-check.test.ts` guards it).- `apps/renderer-service`: HTTP render service (Docker, RunPod worker).- `examples/NN_*`: reference films/ads that import `gitframes` via workspace. Study `19_gitframes_film` (best structure), `22_gitframes_launch` before writing a new film.diff --git a/.gitignore b/.gitignoreindex 2d6a478b..9e2465f7 100644--- a/.gitignore+++ b/.gitignore@@ -74,7 +74,7 @@ examples/output/examples/**/output/**/output/-# Lazy-downloaded vision model weights (never source — see specs/yolov4plan.ts §Phase F)+# Lazy-downloaded vision model weights (never source)*.onnx*.tflite*.taskdiff --git a/README.md b/README.mdindex 7737b714..42328b5d 100644--- a/README.md+++ b/README.md@@ -9,7 +9,7 @@[](https://nodejs.org)[](https://www.w3.org/TR/webgpu/)[](#monorepo-architecture)-[](#6-yolo11-neural-vision--tracking)+[](#6-on-device-vision--tracking)**Code-first video as pure software engineering** — no headless browser, no DOM reflow, no screenshot pipeline.Renders directly on GPU hardware via Dawn / WebGPU / Metal / Vulkan in Node.js and modern WebGPU browsers.@@ -17,7 +17,20 @@ Renders directly on GPU hardware via Dawn / WebGPU / Metal / Vulkan in Node.js a</div>> [!NOTE]-> **Using an AI coding agent?** Install the gitframes skills in **Claude Code** with `/plugin marketplace add gatewai-dev/gitframes` → `/plugin install gitframes@gitframes-plugins`, or in **Codex** with `codex plugin marketplace add gatewai-dev/gitframes`. See [Agent Skills & Plugins](#agent-skills--plugins).+> **Using an AI coding agent?** Install the gitframes skills straight from the marketplace.+>+> **Claude Code**+> ```text+> /plugin marketplace add gatewai-dev/gitframes+> /plugin install gitframes@gitframes-plugins+> ```+>+> **Codex**+> ```bash+> codex plugin marketplace add gatewai-dev/gitframes+> ```+>+> See [Agent Skills & Plugins](#agent-skills--plugins) for other agents.---@@ -31,7 +44,7 @@ Renders directly on GPU hardware via Dawn / WebGPU / Metal / Vulkan in Node.js a- [3. Unified 3D Scene Graph, Camera & Mesh Shading](#3-unified-3d-scene-graph-camera--mesh-shading)- [4. Audio Layers, Procedural SFX & Reactive Signals](#4-audio-layers-procedural-sfx--reactive-signals)- [5. WebGPU ChartGPU Bridge](#5-webgpu-chartgpu-bridge)- - [6. YOLO11 Neural Vision & Tracking](#6-yolo11-neural-vision--tracking)+ - [6. On-Device Vision & Tracking](#6-on-device-vision--tracking)- [7. Headless Conformance & FrameGrid Testing](#7-headless-conformance--framegrid-testing)- [Monorepo Architecture](#monorepo-architecture)- [Quickstart Guide](#quickstart-guide)@@ -39,7 +52,7 @@ Renders directly on GPU hardware via Dawn / WebGPU / Metal / Vulkan in Node.js a- [2. Unified 3D Scene with Camera & 3D Model](#2-unified-3d-scene-with-camera--3d-model)- [3. Audio Soundtrack, Procedural SFX & Reactive Signals](#3-audio-soundtrack-procedural-sfx--reactive-signals)- [4. Chained WebGPU Post-Processing VFX](#4-chained-webgpu-post-processing-vfx)- - [5. YOLO11 Vision: Pin, Matte & Reframe](#5-yolo11-vision-pin-matte--reframe)+ - [5. Vision: Pin, Matte & Reframe](#5-vision-pin-matte--reframe)- [6. Headless Video & FrameGrid Rendering](#6-headless-video--framegrid-rendering)- [Engineering Doctrines & Best Practices](#engineering-doctrines--best-practices)- [Agent Skills & Plugins](#agent-skills--plugins)@@ -61,7 +74,7 @@ Modern automated video generation is usually constrained by the architectures of| 🎨 | **Photoshop-Grade Tonal & Spatial VFX** | 50+ modular GPU shaders: Curves, Levels, Selective Color, 3D LUTs, Halftone, Film Grain, Unsharp Mask, Mesh Warp, and Screen-Space Relighting. || 🧊 | **Unified 3D & 2D Depth Compositing** | Nest 2D flex/box trees inside 3D homography planes, multiplane rigs, and meshes (OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, OFF), with PBR glass and SSAO. || 🔊 | **Built-in Procedural Audio DSP** | Multi-track soundtracks, deterministic procedural transition SFX (whoosh, impact, riser), and reactive signals that drive visuals from audio. |-| 👁️ | **On-Device Neural Vision** | Native YOLO11 ONNX inference for detection, instance segmentation, pose, OBB, classification, and open-vocabulary prompting — feeding reactive signals without a round trip to disk. |+| 👁️ | **On-Device Neural Vision** | Object tracking, instance segmentation, multi-person pose, and person mattes from Apache-2.0 ONNX models — feeding reactive signals without a round trip to disk. || ☁️ | **Cloud-Native & CI/CD Ready** | ~200–400 MB RAM per worker (vs. 2–4 GB for Chromium), ideal for serverless GPU render clusters (AWS G4/G5, Modal, RunPod, Kubernetes). |---@@ -84,7 +97,7 @@ Developers generating video programmatically commonly weigh **Remotion** (React/| **Motion Blur & Physics** | Physical 180° shutter velocity buffers in MRT + closed-form spring kinematics | CSS transitions / JS interpolation; synthetic blur hacks | Frame interpolation or manual multipass || **Audio Engine & DSP** | Native audio DSP & procedural SFX (multi-track mixing, beat grids, reactive signals) | `<Audio>` playback; basic volume curves | Basic static audio playback || **Live Charting & Telemetry** | **ChartGPU bridge** — GPU-rendered line, bar, OHLC, area charts into GPU textures | DOM chart libraries (Recharts, Chart.js) | Custom canvas draw operations |-| **AI & Computer Vision** | **YOLO11 ONNX** — detect, instance segmentation, COCO-17 pose, OBB, classify, and open-vocabulary YOLO-World; WebGPU tensor conditioning (Canny, depth-to-normals, optical flow, deflicker) | External pre-rendered assets; no native GPU tensor conditioning | External pre-rendered assets |+| **AI & Computer Vision** | **On-device ONNX vision** — COCO-80 detection + instance masks (RTMDet-Ins), COCO-17 pose (RTMO), person mattes (Selfie Segmenter); WebGPU tensor conditioning (Canny, depth-to-normals, optical flow, deflicker) | External pre-rendered assets; no native GPU tensor conditioning | External pre-rendered assets || **Headless Verification** | **FrameGrid contact sheets**, single-frame snapshots, Skia MSE pixel-invariant assertions | Playwright/Puppeteer visual snapshots | Manual frame inspection / canvas diffing || **Docker / Cloud Portability** | **Compact** (~500 MB slim image with native GPU/Vulkan drivers) | **Heavy** (~2–3 GB with Chromium, fonts, X11/Mesa) | Moderate container size |@@ -141,38 +154,38 @@ Through native [`ChartGPU`](packages/webgpu-renderers/src/chartgpu/chartgpu-brid- Render real-time financial, scientific, and metric charts (line, bar, OHLC/candlestick, area, scatter) directly onto offscreen GPU textures.- Composite live charts into 3D perspective cards, flex layouts, or HUD overlays — no canvas DOM elements.-### 6. YOLO11 Neural Vision & Tracking-The MediaPipe engine has been **replaced by a first-class YOLO11 pipeline** ([`@gitframes/yolo`](packages/yolo)). It runs ONNX models via `onnxruntime-node` (CPU EP) or `onnxruntime-web` (WebGPU), and wires every result into the same reactive signal surface the rest of Gitframes consumes.+### 6. On-Device Vision & Tracking+[`@gitframes/vision`](packages/vision) runs ONNX models via `onnxruntime-node` (CPU) or `onnxruntime-web` (WebGPU) and wires every result into the same reactive signal surface the rest of Gitframes consumes.> [!TIP]-> **Lazy by construction.** `YoloVisionRunner.create()` and `new YoloNode(...)` perform **zero I/O** — no downloads, no sessions, no file probes. A model is fetched the first time a task actually runs. Call `await vision.ready()` to warm up ahead of time.+> **Lazy by construction.** `VisionRunner.create()`, `comp.withVision(...)` and `VisionNode.attach(...)` perform **zero I/O** — no downloads, no sessions, no file probes. A model is fetched the first time a task actually runs. To warm up ahead of time, call `await runner.preload(["detect", "pose"])` (or `await vision.ready()` on an attached node).-#### Six task heads from one engine+#### Tasks and models-| Task | Default model | Classes | Output |+Every model is **Apache-2.0**, pinned to an immutable Hugging Face revision, and verified by SHA-256 after download.++| Task | Option | Model | Output ||---|---|---|---|-| **Detect** | `yolo11{n,s,m,l,x}` | COCO 80 | Axis-aligned boxes + scores |-| **Segment** | `yolo11{n,s,m}-seg` | COCO 80 | Per-instance masks (frame-aligned alpha) |-| **Pose** | `yolo11{n,s,m,l}-pose` | COCO-17 | 17 keypoints + visibility per person |-| **OBB** | `yolo11{n,s,m,l}-obb` | DOTA-15 | Rotated boxes + 4 corners + angle |-| **Classify** | `yolo11{n,s}-cls` | ImageNet-1k | Top-1 / Top-5 |-| **World** | `yolov8{s,m}-worldv2` | Open vocabulary | Prompt-driven detection |--- **Variants `n/s/m/l/x`** and **imgsz 320–1280** (640 default; OBB 1024; classify 224). Tune `confidence`, `iouThreshold`, and a COCO `classes` filter per composition.-- **Custom models** — point `customModel` at a local `path` or `url` with a `task` and `classes` array to run your own ONNX export.-- **Configurable asset origin** — override `baseUrl` per run or via `GITFRAMES_YOLO_BASE_URL`; models are cached atomically (temp + rename) and evicted if corrupt.--#### Hardware-accelerated vision (WebGPU T2)-- **GPU letterbox resize** — WGSL compute letterboxing straight into the inference tensor ([`YoloGpuPipeline`](packages/yolo/src/gpu/yolo-gpu-pipeline.ts)).-- **GPU mask upscaling** — prototype-coefficient mask assembly with threshold + feather in compute ([`mask-upscale.wgsl`](packages/yolo/src/gpu/mask-upscale.wgsl.ts)).-- **OpenPose-style skeleton textures** — rasterize COCO-17 keypoints into a VRAM conditioning texture ([`PoseSkeletonRenderer`](packages/yolo/src/gpu/pose-skeleton-renderer.ts)).-- **GPU segmentation texture pool** — reusable `r8unorm`/`rgba8unorm` silhouette textures ([`SegmentationTexturePool`](packages/yolo/src/gpu/segmentation-texture-pool.ts)).-- **Optical-flow forward warping** — advect detections and masks between inference calls for up to ~4× effective throughput ([`optical-flow-warper`](packages/yolo/src/gpu/optical-flow-warper.ts)).+| **Detect** | `enableDetection` | RTMDet-Ins `t/s/m` (OpenMMLab) | COCO-80 boxes + scores, tracked over time |+| **Segment** | `enableSegmentation` | RTMDet-Ins (same forward pass as detect) | Soft per-instance masks, frame-aligned |+| **Pose** | `enablePose` | RTMO `t/s/m` (OpenMMLab) | 17 COCO keypoints + visibility per person |+| **Matte** | `enableMatte` | MediaPipe Selfie Segmenter (Google) | Fast person-vs-background alpha for portrait / webcam framing |++- **Variants** — `variant: "t" | "s" | "m"` (default `"s"`; ~24 / 43 / 116 MB for RTMDet-Ins). Tune `confidence` and a COCO `classes` filter per composition. On CPU, a 2K frame takes roughly 200–340 ms to detect + segment, ~120 ms for pose and ~20 ms for the matte with `"s"`.+- **One pass, two tasks** — detection and segmentation share a single RTMDet-Ins inference per frame.+- **Picking a matte** — the Selfie Segmenter is tuned for a person filling much of the frame: it misses distant figures and can report "person" on close-ups with nobody in them. For anything else, cut out with instance masks (`matteSource: "instance"`, the default).+- **Whole-subject cutouts** — `mask` / `matte` / `crop` modes merge every comparably sized instance that overlaps the main subject, so a flowing dress or a held instrument stays attached to the person, while a tunnel or window framing them does not.+- **One-frame delay** — vision reads each layer's previous rendered frame, so results trail the plate by one frame and frame 0 has none. Verify vision layers with the exported video or consecutive frames, not frame grids.+- **Model cache & mirrors** — models are cached atomically (temp + rename) in `$GITFRAMES_MODELS_DIR` (default `~/.cache/gitframes/models`). Point `baseUrl` or `GITFRAMES_MODELS_BASE_URL` at your own mirror for air-gapped or CI renders.++#### GPU helpers+- **OpenPose-style skeleton textures** — rasterize COCO-17 keypoints into a VRAM conditioning texture ([`PoseSkeletonRenderer`](packages/vision/src/gpu/pose-skeleton-renderer.ts)).+- **GPU segmentation texture pool** — reusable silhouette textures ([`SegmentationTexturePool`](packages/vision/src/gpu/segmentation-texture-pool.ts)).#### Temporal tracking & analysis-- **Multi-object tracker** ([`TemporalObjectTracker`](packages/yolo/src/tracking/temporal-object-tracker.ts)) assigns stable `trackId`s via IoU association, with configurable `minHits`, `positionSmoothing`, and velocity-based **coasting** for up to `maxMissedFrames` (default 15) so a transient miss holds the track instead of flashing.-- **Pose↔track matching** ([`pose-track-matcher`](packages/yolo/src/tracking/pose-track-matcher.ts)) binds keypoints to the right track by id, then by spatial IoU fallback.-- **One-shot sequence analysis** — `comp.analyzeVisionSequence(src, { tasks, categories })` decodes frames through the mediabunny pipeline, tracks them, and returns a **zod-serializable** report (per-track frame ranges, mean speed, sampled center paths, per-class presence/confidence, mean mask coverage, model download bytes/timing) ([`analyzeSequence`](packages/yolo/src/analysis/analyze-sequence.ts)).+- **Multi-object tracker** ([`TemporalObjectTracker`](packages/vision/src/tracking/temporal-object-tracker.ts)) assigns stable `trackId`s via IoU association, with configurable `minHits`, `positionSmoothing`, and velocity-based **coasting** for up to `maxMissedFrames` (default 15) so a transient miss holds the track instead of flashing.+- **Pose↔track matching** ([`pose-track-matcher`](packages/vision/src/tracking/pose-track-matcher.ts)) binds keypoints to the right track by id, then by spatial IoU fallback.+- **One-shot sequence analysis** — `comp.analyzeVisionSequence(src, { tasks, categories })` decodes frames through the mediabunny pipeline, tracks them, and returns a **zod-serializable** report (per-track frame ranges, mean speed, sampled center paths, per-class presence/confidence, mean mask coverage, model download bytes/timing) ([`analyzeSequence`](packages/vision/src/analysis/analyze-sequence.ts)).#### Reactive vision signalsEvery tracked entity is exposed as reactive `ProgrammaticSignal`s that animate layers and shader uniforms:@@ -180,35 +193,32 @@ Every tracked entity is exposed as reactive `ProgrammaticSignal`s that animate l| Group | Highlights ||---|---|| `objects` | `get(trackId)`, `byCategory(cat, rank)`, `primary`, `count`, `hasCategory`, `detectedCategories` |-| `objects.*.bounds` | `x/y/width/height`, `screenX/screenY/screenWidth/screenHeight`, `aspectRatio`, `area`, `angle` |+| `objects.*.bounds` | `x/y/width/height`, `screenX/screenY/screenWidth/screenHeight`, `aspectRatio`, `area` || `objects.*.anchors` | 9 anchors (corners, edges, center) ready for pinning || `objects.*.kinematics` | `vx`, `vy`, `speed`, `acceleration`, `headingRad/Deg` || `objects.*.pose` | All 17 COCO keypoints, plus `hasPose`, `wristSpeed`, `handRaised`, `bodyTiltAngle` || `masks` | `get(trackId)`, `subject`, `count`; per-mask `area`, `coverage`, `solidity`, `bboxFill` |-| `segmentation` | `subject`, `humanSilhouette`, `instanceMasks`, GPU `stencilTexture` |+| `segmentation` | `subject`, `humanSilhouette`, `instanceMasks`, `matte.coverage`, GPU `stencilTexture` || `classes` | Per-class `count`, `maxConfidence`, `present`, `primary`, plus a detection `histogram` |-| `classification` | `top1`, `top1Confidence`, `top5`, `top5At(frame)` || Tensors | `poseLandmarksTensor [17,3]`, `objectsTensor [16,8]`, `masksTensor [16,2]`, `histogramTensor [80]` |#### Spatial pinning-Project normalized landmarks to screen space with a configurable camera FOV, then bind any node to a track or landmark ([`SpatialLandmarkTransformer`](packages/yolo/src/spatial/camera-space-transformer.ts), [`spatial-pin`](packages/yolo/src/spatial/spatial-pin.ts)):+Project normalized landmarks to screen space with a configurable camera FOV, then bind any node to a track or landmark ([`SpatialLandmarkTransformer`](packages/vision/src/spatial/camera-space-transformer.ts), [`spatial-pin`](packages/vision/src/spatial/spatial-pin.ts)):- `pinToObject(track, { anchor, offsetX/Y/Z, matchWidth, matchHeight, smoothFrames, hideWhenLost })`- `pinToLandmark(coord, { offsetX/Y/Z })`#### High-level composition helpers- **Subject Sandwich** — `comp.addSubjectSandwich({ source, behind, feather, fit })` cuts the foreground subject out and places typography/graphics behind them.-- **Smart Reframing** — `comp.addSmartFraming({ source, targetAspect, damping, leadHeadroom })` auto-crops 16:9 → 9:16 while tracking the focal subject.-- **Subject Outline** — `comp.addSubjectOutline({ source, color, width, blur })` strokes the segmented boundary as an audio-reactive contour glow.+- **Smart Reframing** — `comp.addSmartFraming({ source, target, targetAspect, damping, leadHeadroom })` auto-crops 16:9 → 9:16 while tracking `target`.+- **Subject Outline** — `comp.addSubjectOutline(vision.segmentation.subject, { source, color, width, blur })` strokes the segmented boundary as an audio-reactive contour glow.- **Tracked Region Blur** — `layer.blurRegion(track, { strength })` blurs faces, plates, or any detected class.-- **Node modes** — `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`, `obb`, plus optional `keyBackground`/`backgroundKeyThreshold` to grow a person mask into connected foreground (e.g. a dress).+- **Node modes** — `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`; pick the cutout alpha with `matteSource: "instance" | "selfie"`, and optionally `keyBackground` to grow the subject into connected foreground.#### Agent-first DX-- **Runtime config is zod-validated** and available from a **zod-only entry** (`@gitframes/yolo/schemas`) so the hot path stays zod-free.+- **Runtime config is zod-validated** and available from a **zod-only entry** (`@gitframes/vision/schemas`) so the hot path stays zod-free. Unknown or removed options are rejected, not silently ignored.- **`vision.summary(frame)`** returns a deterministic, serializable snapshot (objects, classes, masks) safe to call inside a frame hook.-- **Browser entry** — `@gitframes/yolo/web` re-exports the engine plus `createWebGPUProvider()` / `hasWebGPU()`; `onnxruntime-web` is an optional lazy peer.--#### Migration from MediaPipe-The `MediaPipe` effect class and `comp.withMediaPipe(...)` remain as **deprecated, source-compatible aliases** that normalize legacy options (`enablePoseLandmarks`, `enableSegmentation`, `mode: "skeleton"`, `objectScoreThreshold`, …) onto the YOLO engine. Prefer [`Yolo`](packages/gitframes/src/effects/classes.ts) and `comp.withYolo(...)`. Face blendshape signals were removed with the old engine; `driveBlendshapes` is accepted as a no-op.+- **Clear failures** — a model that is the wrong size, fails its checksum, or lacks an expected output raises an error naming the model and its source.+- **Browser entry** — `@gitframes/vision/web` re-exports the engine plus `createWebGPUProvider()` / `hasWebGPU()`; `onnxruntime-web` is an optional lazy peer.### 7. Headless Conformance & FrameGrid Testing- **Pixel-sampling invariant assertions** — test compositions in Vitest with `skia-canvas` to verify shader math, font coverage, and Mean Squared Error (MSE) temporal deltas.@@ -228,14 +238,14 @@ gitframes/│ ├── compositions/ # Layout engine, Flex/Box AST compiler, timeline evaluator│ ├── webgpu-renderers/ # WGSL shaders, Slug text engine, 3D renderer, camera, lights, materials│ ├── tensor-webgpu/ # WebGPU compute pipelines (Canny, depth-to-normals, flow, deflicker, landmarks)-│ ├── yolo/ # YOLO11 ONNX engine: tasks, decode, tracking, GPU accel, signals+│ ├── vision/ # ONNX vision engine: detect, segment, pose, matte, tracking, signals│ ├── renderer/ # Headless Node.js WebGPU renderer via Dawn, WebCodecs, skia-canvas│ ├── renderers/ # Higher-level render orchestration│ ├── media/ # Media decoding / encoding adapters│ ├── node-sdk/ # Node renderer contracts and result schemas│ ├── server-utils/ # Server infrastructure, storage, asset caches│ └── client-utils/ # Shared browser utilities-├── nodes/ # 58+ specialized domain nodes (VFX, audio, layout, node-yolo)+├── nodes/ # 58+ specialized domain nodes (VFX, audio, layout, node-vision)├── apps/│ └── renderer-service/ # Production HTTP / gRPC rendering microservice container├── examples/ # Reference compositions and films@@ -418,15 +428,15 @@ comp.apply(new ColorBalance({ shadows: [0, 2, 6], highlights: [4, 1, -2] }));----### 5. YOLO11 Vision: Pin, Matte & Reframe+### 5. Vision: Pin, Matte & Reframe```typescriptimport { Composition, Layer, Vignette } from "gitframes";const comp = new Composition({ width: 1920, height: 1080, fps: 30 });-// Attach YOLO11 to the whole composition. Models download lazily on first use.-const vision = comp.withYolo({+// Run vision on the whole composition. Models download lazily on first use.+const vision = comp.withVision({enableDetection: true,enableSegmentation: true,enablePose: true,@@ -451,8 +461,8 @@ comp.add(// Or use the one-liners for the common editorial moves:// comp.addSubjectSandwich({ source: "assets/dancer.mp4", behind: [headline], feather: 4 });-// comp.addSmartFraming({ source: "assets/action.mp4", targetAspect: 9 / 16, damping: 0.15 });-// comp.addSubjectOutline({ source: "assets/character.mp4", color: "#FF5A1F", width: 6, blur: 12 });+// comp.addSmartFraming({ source: "assets/action.mp4", target: vision.objects.primary, targetAspect: 9 / 16 });+// comp.addSubjectOutline(vision.segmentation.subject, { source: "assets/character.mp4", color: "#FF5A1F", width: 6 });// Inspect a source before authoring: one-shot, ffmpeg-free, zod-serializable reportconst report = await comp.analyzeVisionSequence("assets/street.mp4", {@@ -465,20 +475,23 @@ console.log(report.tracks.map((t) => `${t.category}#${t.trackId} ${t.frames.join**Standalone runner (no composition):**```typescript-import { YoloVisionRunner } from "@gitframes/yolo";+import { VisionRunner } from "@gitframes/vision";-const runner = YoloVisionRunner.create({ variant: "n", confidence: 0.25 }); // zero I/O-const boxes = await runner.detect({ data: rgba, width: 1920, height: 1080 });+const runner = VisionRunner.create({ variant: "s", confidence: 0.3 }); // zero I/O+const frame = { data: rgba, width: 1920, height: 1080 };+const boxes = await runner.detect(frame); // downloads RTMDet-Ins on first call+const { masks } = await runner.segment(frame); // same forward pass, no second inference+const { people } = await runner.pose(frame); // RTMO, COCO-17 keypointsrunner.close();```**In the browser (WebGPU EP):**```typescript-import { YoloVisionRunner, createWebGPUProvider, hasWebGPU } from "@gitframes/yolo/web";+import { VisionRunner, createWebGPUProvider, hasWebGPU } from "@gitframes/vision/web";if (hasWebGPU()) {- const runner = YoloVisionRunner.create({ provider: createWebGPUProvider() });+ const runner = VisionRunner.create({ provider: createWebGPUProvider() });}```@@ -532,7 +545,7 @@ Gitframes ships agent skills that teach Claude, Codex, and other coding agents h| Skill | Use it for || --- | --- || `gitframes-compose` | Compositions, layer trees, layout, animation and easing, beat grids, film structure |-| `gitframes-effects` | Effect classes, the unified section architecture, premultiplied-alpha invariants, YOLO vision conditioning |+| `gitframes-effects` | Effect classes, the unified section architecture, premultiplied-alpha invariants, vision conditioning || `gitframes-render` | Headless rendering, FrameGrid inspection, pixel probes, MP4 delivery checks |Once installed, skills load automatically when a task matches (e.g. *"add a film-grain pass to this scene"* or *"render a frame grid of intro.ts"*).@@ -627,6 +640,9 @@ pnpm build# Run conformance testspnpm test+# Check the vision models end to end (downloads ~380 MB of weights once)+pnpm --filter @gitframes/vision test:models+# Render a specific showcase examplepnpm --filter @gitframes/example-21-full-circle render@@ -646,4 +662,4 @@ docker build -t gitframes-renderer -f Dockerfile.renderer .## License-Gitframes is open-source software licensed under [Apache-2.0](LICENCE). Downloaded YOLO model weights follow their upstream Ultralytics licensing.+Gitframes is open-source software licensed under [Apache-2.0](LICENCE). The vision models it downloads on demand — RTMDet-Ins and RTMO (OpenMMLab) and the Selfie Segmenter (Google) — are also Apache-2.0; see [`registry.ts`](packages/vision/src/model/registry.ts) for exact sources and checksums.diff --git a/examples/19_gitframes_film/src/render-track.ts b/examples/19_gitframes_film/src/render-track.tsindex 1916d61e..c11e41fd 100644--- a/examples/19_gitframes_film/src/render-track.ts+++ b/examples/19_gitframes_film/src/render-track.ts@@ -41,7 +41,7 @@ async function main(): Promise<void> {{frame: 12,label:- "YOLO11 Soft-Alpha Cutout · 3D Ground Contact Rings · Reticle Lock",+ "Soft-Alpha Cutout · 3D Ground Contact Rings · Reticle Lock",},{frame: 48,diff --git a/examples/19_gitframes_film/src/scenes/track.ts b/examples/19_gitframes_film/src/scenes/track.tsindex bec971fd..9f13e86f 100644--- a/examples/19_gitframes_film/src/scenes/track.ts+++ b/examples/19_gitframes_film/src/scenes/track.ts@@ -1,8 +1,8 @@/*** Chapter 05: Vision — "Track it."*- * Demonstrates Gitframes's latest YOLO11 vision updates with unmistakable visual punch:- * 1. YOLO11 temporal object tracking (`vision.objects.byCategory("person")`)+ * Demonstrates Gitframes's on-device vision with unmistakable visual punch:+ * 1. Temporal object tracking (`vision.objects.byCategory("person")`)* anchoring precision spatial telemetry and bounding reticles.* 2. High-energy 2.5D perspective homography section (`Layer.section`) with 3D banking,* rapid 5Hz shutter strobe (`fastStrobe`), harmonic spectral hue cycling (`fastHue`),@@ -19,7 +19,7 @@ import {LayerAnimation,type ObjectTrackSignals,type TrackedObject,- YoloVisionBundle,+ VisionBundle,} from "gitframes";import {asset,@@ -249,8 +249,8 @@ export function trackScene(options: TrackSceneOptions = {}) {const to = options.to ?? TRACK_TO;const len = to - from;- // 1. Initialize YOLO Vision Bundle populated with temporal object tracking- const vision = new YoloVisionBundle({+ // 1. Initialize the vision bundle populated with temporal object tracking+ const vision = new VisionBundle({width: W,height: H,fps: FPS,@@ -279,8 +279,8 @@ export function trackScene(options: TrackSceneOptions = {}) {// 2. Neural Isolated Smoke Dancer ("Smoke Dancing"):// Her entire body (head, arms, torso, spinning dress) is converted into billowing ember smoke- // via YOLO11 instance segmentation and background keying:- // - Layer A (Silhouette Base): Full body & dress matte via background-keyed YOLO segmentation.+ // via instance segmentation and background keying:+ // - Layer A (Silhouette Base): Full body & dress matte via background-keyed instance segmentation.// - Layer B (Smoke Fill): Billowing ember smoke video composited in source-in mode,// strictly confined to her entire silhouette.// Seamlessly isolated inside a transparent container box without rectangular framing cards.@@ -293,7 +293,7 @@ export function trackScene(options: TrackSceneOptions = {}) {height: H,fit: "cover",muted: true,- }).withYolo({+ }).withVision({mode: "matte",enableSegmentation: true,variant: "s",@@ -431,7 +431,7 @@ export function trackScene(options: TrackSceneOptions = {}) {}),label({id: "track-caption",- text: "YOLO11 Neural Tracking · Full Smoke Silhouette Blend · 35mm Spatial Analysis",+ text: "Neural Tracking · Full Smoke Silhouette Blend · 35mm Spatial Analysis",x: W - 96 - 920,y: H - BAR / 2 - 11,width: 920,diff --git a/nodes/node-yolo/package.json b/nodes/node-vision/package.jsonsimilarity index 86%rename from nodes/node-yolo/package.jsonrename to nodes/node-vision/package.jsonindex 0a8cda75..89aa8247 100644--- a/nodes/node-yolo/package.json+++ b/nodes/node-vision/package.json@@ -1,13 +1,10 @@{- "name": "@gitframes/node-yolo",+ "name": "@gitframes/node-vision","version": "1.2.35","private": false,"gitframes": {- "type": "Yolo",- "kind": "effect",- "aliases": [- "MediaPipe"- ]+ "type": "Vision",+ "kind": "effect"},"type": "module","module": "./dist/index.mjs",@@ -34,7 +31,7 @@"dependencies": {"@gitframes/core": "workspace:*","@gitframes/node-sdk": "workspace:*",- "@gitframes/yolo": "workspace:*",+ "@gitframes/vision": "workspace:*","zod": "^4.3.6"},"devDependencies": {diff --git a/nodes/node-yolo/src/index.ts b/nodes/node-vision/src/index.tssimilarity index 100%rename from nodes/node-yolo/src/index.tsrename to nodes/node-vision/src/index.tsdiff --git a/nodes/node-yolo/src/renderers/index.ts b/nodes/node-vision/src/renderers/index.tssimilarity index 50%rename from nodes/node-yolo/src/renderers/index.tsrename to nodes/node-vision/src/renderers/index.tsindex e2e19cf4..eab77dd3 100644--- a/nodes/node-yolo/src/renderers/index.ts+++ b/nodes/node-vision/src/renderers/index.ts@@ -1,6 +1,6 @@import { defineRenderer } from "@gitframes/node-sdk/renderer";-import { YoloWebGPURenderer } from "./webgpu-renderer.js";+import { VisionWebGPURenderer } from "./webgpu-renderer.js";export default defineRenderer({- WebGPURenderer: YoloWebGPURenderer,+ WebGPURenderer: VisionWebGPURenderer,});diff --git a/nodes/node-yolo/src/renderers/webgpu-renderer.ts b/nodes/node-vision/src/renderers/webgpu-renderer.tssimilarity index 63%rename from nodes/node-yolo/src/renderers/webgpu-renderer.tsrename to nodes/node-vision/src/renderers/webgpu-renderer.tsindex a02af069..c695589e 100644--- a/nodes/node-yolo/src/renderers/webgpu-renderer.ts+++ b/nodes/node-vision/src/renderers/webgpu-renderer.ts@@ -1,21 +1,27 @@/// <reference types="webgpu" />import type { WebGPUNodeRenderer } from "@gitframes/node-sdk";import {+ type InstanceMask,+ type MaskBounds,+ maskBounds,matchPoseToTracks,+ mergeSubjectMask,+ type PersonMatte,+ type PoseResult,PoseSkeletonRenderer,SegmentationTexturePool,TemporalObjectTracker,- type YoloInstanceMask,- type YoloPoseResult,- type YoloVisionBundle,- YoloVisionRunner,-} from "@gitframes/yolo";-import type { YoloOperation } from "../shared/config.js";--let sharedRunner: YoloVisionRunner | null = null;+ type TrackedObject,+ type VisionBundle,+ VisionRunner,+} from "@gitframes/vision";+import type { VisionOperation } from "../shared/config.js";++let sharedRunner: { key: string; runner: VisionRunner } | null = null;let sharedSkeletonRenderer: PoseSkeletonRenderer | null = null;let sharedTexturePool: SegmentationTexturePool | null = null;-let sharedObjectTracker: TemporalObjectTracker | null = null;+/** One tracker per vision node — track ids must not mix across nodes/sources. */+const objectTrackers = new Map<string, TemporalObjectTracker>();/*** Persistent per-node child textures. The shared frame encoder is submitted at@@ -24,64 +30,58 @@ let sharedObjectTracker: TemporalObjectTracker | null = null;* the start of the next call — a stable one-frame delay instead of the* unpredictable multi-frame lag you get from pooled textures.*- * Keyed on the stable Yolo `Effect` instance (render ids change every frame),+ * Keyed on the stable Vision `Effect` instance (render ids change every frame),* with a size-keyed fallback for raw operation nodes.*/-const yoloChildTexturesByEffect = new WeakMap<object, GPUTexture>();-const yoloChildTexturesByKey = new Map<string, GPUTexture>();+const childTexturesByEffect = new WeakMap<object, GPUTexture>();+const childTexturesByKey = new Map<string, GPUTexture>();/*** Last successful subject mask, reused for a few frames when the model misses —* a transient miss then holds the silhouette instead of flashing the raw plate.- * Keyed per node/effect so multiple YOLO nodes do not collide.+ * Keyed per node/effect so multiple vision nodes do not collide.*/const lastSubjectByNode = new Map<string,- { mask: YoloInstanceMask; frame: number }+ { mask: InstanceMask; frame: number }>();const SUBJECT_HOLD_FRAMES = 3;/*** Lazy shared runner — `create()` is a pure constructor (zero I/O); the first frame that* requests a task triggers that task's model download at inference time (never at init).+ * Recreated only when an option that changes inference changes.*/-function getSharedRunner(op: YoloOperation): YoloVisionRunner {- if (- !sharedRunner ||- sharedRunner.variant !== op.variant ||- sharedRunner.enableWorld !== op.enableWorld ||- sharedRunner.prompts !== op.prompts ||- sharedRunner.customModel !== op.customModel ||- sharedRunner.featherRadius !== op.featherRadius ||- sharedRunner.maskThreshold !== op.maskThreshold- ) {- sharedRunner = YoloVisionRunner.create({- variant: op.variant,- imgsz: op.imgsz,- confidence: op.confidence,- iouThreshold: op.iouThreshold,- classes: op.classes,- enableWorld: op.enableWorld,- prompts: op.prompts,- customModel: op.customModel,- maskThreshold: op.maskThreshold,- featherRadius: op.featherRadius,- modelsDir: op.modelsDir,- baseUrl: op.baseUrl,- });+function getSharedRunner(op: VisionOperation): VisionRunner {+ const options = {+ variant: op.variant,+ confidence: op.confidence,+ classes: op.classes,+ maskThreshold: op.maskThreshold,+ featherRadius: op.featherRadius,+ modelsDir: op.modelsDir,+ baseUrl: op.baseUrl,+ };+ const key = JSON.stringify(options);+ if (sharedRunner?.key !== key) {+ sharedRunner?.runner.close();+ sharedRunner = { key, runner: VisionRunner.create(options) };}- return sharedRunner;+ return sharedRunner.runner;}-/** Lazy shared tracker — detections (incl. OBB) are tracked under one stable identity space. */-function getSharedObjectTracker(op: YoloOperation): TemporalObjectTracker {- if (!sharedObjectTracker) {- sharedObjectTracker = new TemporalObjectTracker({- iouThreshold: op.iouThreshold ?? 0.25,+function getObjectTracker(+ nodeKey: string,+ op: VisionOperation,+): TemporalObjectTracker {+ let tracker = objectTrackers.get(nodeKey);+ if (!tracker) {+ tracker = new TemporalObjectTracker({maxMissedFrames: op.maxMissedFrames ?? 15,});+ objectTrackers.set(nodeKey, tracker);}- return sharedObjectTracker;+ return tracker;}/**@@ -89,10 +89,10 @@ function getSharedObjectTracker(op: YoloOperation): TemporalObjectTracker {* the contract the signal bundle and the skeleton pipeline expect.*/function normalizePoseKeypoints(- result: YoloPoseResult,+ result: PoseResult,width: number,height: number,-): YoloPoseResult {+): PoseResult {if (width <= 0 || height <= 0) return result;return {people: result.people.map((person) => ({@@ -108,9 +108,9 @@ function normalizePoseKeypoints(/** Assigns each decoded instance mask to the temporal track whose box center is nearest the mask's centroid. */function assignMasksToTracks(- masks: readonly YoloInstanceMask[],+ masks: readonly InstanceMask[],tracked: readonly { trackId: number; centerX: number; centerY: number }[],-): YoloInstanceMask[] {+): InstanceMask[] {return masks.map((mask) => {const { cx, cy } = maskCentroid(mask);let best: { trackId: number; distance: number } | null = null;@@ -126,29 +126,11 @@ function assignMasksToTracks(});}-/** Picks the subject mask: the largest instance, preferring a person when present. */-function selectSubjectMask(- masks: readonly YoloInstanceMask[],-): YoloInstanceMask | undefined {- if (masks.length === 0) return undefined;- let best = masks[0];- for (const m of masks) {- const mIsPerson = m.category.toLowerCase() === "person";- const bestIsPerson = best.category.toLowerCase() === "person";- if (mIsPerson && !bestIsPerson) {- best = m;- continue;- }- if (mIsPerson === bestIsPerson && m.area > best.area) best = m;- }- return best;-}-/**- * Grows a person mask into connected foreground pixels — the COCO person head- * drops a flowing dress (treats it as non-person), so seeding a flood fill from- * the body through pixels that differ from the sampled backdrop recovers the- * fabric without adding the near-uniform studio wall or its soft grey shadow.+ * Grows a subject mask into connected foreground pixels — seeding a flood fill+ * from the subject through pixels that differ from the sampled backdrop recovers+ * thin or fast-moving edges (hair, fabric) without adding a near-uniform studio+ * wall or its soft grey shadow.*/function fillInternalHoles(mask: Uint8Array,@@ -322,7 +304,7 @@ function growMaskIntoForeground(return smoothMaskBoundary(mask, width, height, blurRadius);}-function maskCentroid(mask: YoloInstanceMask): { cx: number; cy: number } {+function maskCentroid(mask: InstanceMask): { cx: number; cy: number } {// First moment of the frame-aligned mask — cheap, deterministic association.const { mask: data, width, height } = mask;let sumX = 0;@@ -342,7 +324,7 @@ function maskCentroid(mask: YoloInstanceMask): { cx: number; cy: number } {return { cx: sumX / count, cy: sumY / count };}-export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {+export const VisionWebGPURenderer: WebGPUNodeRenderer = async (args) => {const {ctx,encoder,@@ -356,12 +338,11 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {const { virtualMedia } = props;const rawOp = virtualMedia?.operation as- | YoloOperation+ | VisionOperation| Record<string, unknown>| undefined;- if (!rawOp) return;- const op = normalizeOperation(rawOp);- if (!op) return;+ if (rawOp?.op !== "Vision") return;+ const op = rawOp as VisionOperation;// Close the incoming render pass to allow custom render passespass.end();@@ -369,7 +350,7 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {const childMedia = virtualMedia?.children?.[0];if (!childMedia) return;- // 1. Persistent per-node child texture (see yoloChildTextures*). Reading the+ // 1. Persistent per-node child texture (see childTextures*). Reading the// texture we just drew would race the end-of-frame submit; reading last// frame's already-submitted pixels at the start of this call is correct.const effectKey = (op as { effect?: object }).effect;@@ -378,18 +359,18 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {(virtualMedia as any)?.id ??(rawOp as any)?.id ??`${targetWidth}x${targetHeight}_${op.mode}_${op.variant}_${op.keyBackground}_${op.backgroundKeyThreshold}`;- const fallbackKey = `yolo-${nodeKeyStr}`;+ const fallbackKey = `vision-${nodeKeyStr}`;let childTex = hasEffectKey- ? yoloChildTexturesByEffect.get(effectKey as object)- : yoloChildTexturesByKey.get(fallbackKey);+ ? childTexturesByEffect.get(effectKey as object)+ : childTexturesByKey.get(fallbackKey);if (childTex &&(childTex.width !== targetWidth || childTex.height !== targetHeight)) {childTex.destroy();childTex = undefined;- if (hasEffectKey) yoloChildTexturesByEffect.delete(effectKey as object);- else yoloChildTexturesByKey.delete(fallbackKey);+ if (hasEffectKey) childTexturesByEffect.delete(effectKey as object);+ else childTexturesByKey.delete(fallbackKey);}const hasPreviousFrame = childTex !== undefined;if (!childTex) {@@ -400,17 +381,16 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {GPUTextureUsage.RENDER_ATTACHMENT |GPUTextureUsage.TEXTURE_BINDING |GPUTextureUsage.COPY_SRC,- label: "yolo_child_persistent",+ label: "vision_child_persistent",});- if (hasEffectKey)- yoloChildTexturesByEffect.set(effectKey as object, childTex);- else yoloChildTexturesByKey.set(fallbackKey, childTex);+ if (hasEffectKey) childTexturesByEffect.set(effectKey as object, childTex);+ else childTexturesByKey.set(fallbackKey, childTex);// keep the fallback cache bounded across many renders in one process- if (yoloChildTexturesByKey.size > 4) {- const oldest = yoloChildTexturesByKey.keys().next().value;+ if (childTexturesByKey.size > 4) {+ const oldest = childTexturesByKey.keys().next().value;if (oldest && oldest !== fallbackKey) {- yoloChildTexturesByKey.get(oldest)?.destroy();- yoloChildTexturesByKey.delete(oldest);+ childTexturesByKey.get(oldest)?.destroy();+ childTexturesByKey.delete(oldest);}}}@@ -426,11 +406,11 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {const stagingBuffer = ctx.device.createBuffer({size: bufferSize,usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ,- label: "yolo_frame_staging",+ label: "vision_frame_staging",});const readbackEncoder = ctx.device.createCommandEncoder({- label: "yolo_readback_encoder",+ label: "vision_readback_encoder",});readbackEncoder.copyTextureToBuffer({ texture: childTex },@@ -481,108 +461,94 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {targetHeight,);- // 4. YOLO inference on the previous frame's pixels (models download lazily on- // first use of each task). Masks are therefore one frame behind the plate.+ // 4. Inference on the previous frame's pixels (models download lazily on first+ // use of each task). Results are therefore one frame behind the plate.const frameIdx = props.frame ?? 0;+ const fps = props.fps ?? 24;+ const mode = op.mode ?? "passthrough";+ const isMatteMode = mode === "mask" || mode === "matte" || mode === "crop";const visionBundle =- (op.visionBundle as YoloVisionBundle | undefined) ??+ (op.visionBundle as VisionBundle | undefined) ??((virtualMedia as Record<string, unknown>).visionBundle as- | YoloVisionBundle+ | VisionBundle| undefined);- const runner = await getSharedRunner(op);+ const runner = getSharedRunner(op);const image = framePixels? { data: framePixels, width: targetWidth, height: targetHeight }: null;+ // detect + segment share one RTMDet-Ins pass per frame inside the runner.+ // Tracked objects are kept locally (like masks below) so `boxes` / `tracking`+ // draw even without an attached visionBundle.+ let frameObjects: readonly TrackedObject[] = [];if (image &&- op.mode !== "obb" &&- (op.enableDetection !== false ||- op.mode === "boxes" ||- op.mode === "tracking")+ (op.enableDetection !== false || mode === "boxes" || mode === "tracking")) {const detections = await runner.detect(image);- const tracker = getSharedObjectTracker(op);+ const tracker = getObjectTracker(nodeKeyStr, op);if (frameIdx === 0) {tracker.reset();}- const tracked = tracker.update(detections, frameIdx, props.fps ?? 24);+ frameObjects = tracker.update(detections, frameIdx, fps);visionBundle?.setObjectResult(frameIdx, {- objects: tracked,+ objects: frameObjects,rawDetections: detections,});}- const needsSegmentation =- op.enableSegmentation === true ||- op.mode === "mask" ||- op.mode === "matte" ||- op.mode === "crop";-// Decoded masks are kept locally so mask-family modes render even without an// attached visionBundle (the round-trip through the bundle is for signal DX).- let frameMasks: readonly YoloInstanceMask[] = [];- if (image && needsSegmentation) {+ let frameMasks: readonly InstanceMask[] = [];+ if (+ image &&+ (op.enableSegmentation === true ||+ (isMatteMode && op.matteSource !== "selfie"))+ ) {const segRes = await runner.segment(image);- const tracked = visionBundle?.getObjectResult(frameIdx).objects ?? [];- frameMasks = assignMasksToTracks(segRes.masks, tracked);+ frameMasks = assignMasksToTracks(segRes.masks, frameObjects);visionBundle?.setMaskResult(frameIdx, frameMasks);}- let currentPoseRes: YoloPoseResult | undefined;- if (image && (op.enablePose === true || op.mode === "skeleton")) {+ let personMatte: PersonMatte | undefined;+ if (+ image &&+ (op.enableMatte === true || (isMatteMode && op.matteSource === "selfie"))+ ) {+ personMatte = await runner.matte(image);+ visionBundle?.setMatteResult(frameIdx, personMatte);+ }++ let currentPoseRes: PoseResult | undefined;+ if (image && (op.enablePose === true || mode === "skeleton")) {const poseRes = await runner.pose(image);const normalized = normalizePoseKeypoints(poseRes,image.width,image.height,);- const tracked = visionBundle?.getObjectResult(frameIdx).objects ?? [];- const matchedPeople = matchPoseToTracks(normalized.people, tracked);- currentPoseRes = { people: matchedPeople };+ currentPoseRes = {+ people: matchPoseToTracks(normalized.people, frameObjects),+ };visionBundle?.setPoseResult(frameIdx, currentPoseRes);}- if (image && op.enableClassification === true) {- const classifyRes = await runner.classify(image);- visionBundle?.setClassifyResult(frameIdx, classifyRes);- }-- if (image && (op.enableObb === true || op.mode === "obb")) {- const obbRes = await runner.detectObb(image);- visionBundle?.setObbResult(frameIdx, obbRes);- // OBB detections also flow through the tracker so `objects.*` (and `bounds.angle`)- // work in `obb` mode without a separate detect pass.- const detections = obbRes.detections.map((d) => ({- category: d.category,- score: d.score,- boundingBox: d.boundingBox,- }));- const tracker = getSharedObjectTracker(op);- if (frameIdx === 0) {- tracker.reset();- }- const tracked = tracker.update(detections, frameIdx, props.fps ?? 24);- visionBundle?.setObjectResult(frameIdx, {- objects: tracked,- rawDetections: detections,- });- }-// 5. Output to destination texture based on mode- const mode = op.mode ?? "passthrough";// Mask-family modes composite from CPU pixels we already hold (frame + masks)- if (mode === "mask" || mode === "matte" || mode === "crop") {+ if (isMatteMode) {if (!sharedTexturePool) {sharedTexturePool = new SegmentationTexturePool(ctx.device);}- const masks =- frameMasks.length > 0- ? frameMasks- : (visionBundle?.getMaskResult(frameIdx) ?? []);- const selected = selectSubjectMask(masks);- let subject: YoloInstanceMask | undefined;+ const selected =+ op.matteSource === "selfie"+ ? personMatteAsMask(personMatte)+ : mergeSubjectMask(+ frameMasks.length > 0+ ? frameMasks+ : (visionBundle?.getMaskResult(frameIdx) ?? []),+ );+ let subject: InstanceMask | undefined;const lastSubjectEntry = lastSubjectByNode.get(nodeKeyStr);const lastSubjectMask = lastSubjectEntry?.mask ?? null;const lastSubjectFrame =@@ -612,8 +578,8 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {return;}- // Opt-in: grow the person mask into the flowing dress (connected colourful- // /dark fabric) so the dancer reads as a full body, not a torso cut-out.+ // Opt-in: grow the subject into connected pixels that stand out from the+ // backdrop (catches edges the model's soft mask leaves behind).if (op.keyBackground === true) {const rawThreshold = (op as Record<string, unknown>).backgroundKeyThreshold;@@ -704,50 +670,6 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {}}- if (mode === "obb") {- const outPass = ctx.renderer.beginFrame(- encoder,- targetView,- { r: 0, g: 0, b: 0, a: 0 },- targetWidth,- targetHeight,- "clear",- );- ctx.renderer.drawTexture(outPass, childTex, {- x: 0,- y: 0,- width: targetWidth,- height: targetHeight,- });-- const detections = visionBundle?.getObbResult(frameIdx).detections ?? [];- const obbColor = "#f59e0b";- for (const det of detections) {- if (det.corners.length < 4) continue;- // Rotated quad stroked as an SVG path (corners are already source px).- const path =- det.corners- .map(- (corner, i) => `${i === 0 ? "M" : "L"} ${corner[0]} ${corner[1]}`,- )- .join(" ") + " Z";- try {- ctx.renderer.drawPath(outPass, path, obbColor, 2);- } catch {- // Fall back to the axis-aligned box if the path pipeline is unavailable.- const b = det.boundingBox;- ctx.renderer.drawRect(- outPass,- { x: b.originX, y: b.originY, width: b.width, height: b.height },- obbColor,- 2,- );- }- }- outPass.end();- return;- }-if (mode === "boxes" || mode === "tracking") {const outPass = ctx.renderer.beginFrame(encoder,@@ -764,12 +686,15 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {height: targetHeight,});- const objRes = visionBundle?.getObjectResult(frameIdx);- if (objRes && objRes.objects.length > 0) {+ const objects =+ frameObjects.length > 0+ ? frameObjects+ : (visionBundle?.getObjectResult(frameIdx).objects ?? []);+ if (objects.length > 0) {const boxColor = "#38bdf8";const stroke = 2;- for (const obj of objRes.objects) {+ for (const obj of objects) {if (!obj.active) continue;const b = obj.boundingBox;@@ -857,7 +782,7 @@ export const YoloWebGPURenderer: WebGPUNodeRenderer = async (args) => {function uploadComposite(pool: SegmentationTexturePool,mode: "mask" | "matte" | "crop",- subject: YoloInstanceMask,+ subject: InstanceMask,framePixels: Uint8ClampedArray,width: number,height: number,@@ -886,95 +811,65 @@ function uploadComposite(}}- const key = `yolo_composite_${nodeKeyStr}_${mode}_${frameIdx}`;+ const key = `vision_composite_${nodeKeyStr}_${mode}_${frameIdx}`;const tex = pool.uploadMask(key, out, width, height);if (mode === "crop") {- // punch-in: re-upload centered on the subject bbox (scaled to full frame)- const box = subjectBoundingBox(subject);+ // Punch in: stretch the subject's bounds to fill the frame.+ const box = maskBounds(subject);if (box) {- const cropW = Math.max(1, Math.round(box.width));- const cropH = Math.max(1, Math.round(box.height));- const crop = new Uint8Array(width * height * 4);- for (let y = 0; y < height; y++) {- const srcY = Math.min(- cropH - 1,- Math.max(0, Math.round((y / height) * cropH)),- );- for (let x = 0; x < width; x++) {- const srcX = Math.min(- cropW - 1,- Math.max(Math.round((x / width) * cropW)),- );- const srcIdx = (srcY * width + srcX) * 4;- const dstIdx = (y * width + x) * 4;- crop[dstIdx] = out[srcIdx];- crop[dstIdx + 1] = out[srcIdx + 1];- crop[dstIdx + 2] = out[srcIdx + 2];- crop[dstIdx + 3] = out[srcIdx + 3];- }- }- return pool.uploadMask(`yolo_crop_${frameIdx}`, crop, width, height);+ return pool.uploadMask(+ `vision_crop_${nodeKeyStr}_${frameIdx}`,+ cropToBounds(out, width, height, box),+ width,+ height,+ );}}return tex;}-function subjectBoundingBox(- subject: YoloInstanceMask,-): { width: number; height: number } | null {- // Derive the bbox from the mask's non-zero extent (cheap one-pass)- const { mask, width } = subject;- let minX = width;- let minY = Infinity;- let maxX = -1;- let maxY = -1;- for (let y = 0; y < subject.height; y++) {+/** Nearest-neighbour resample of `bounds` inside an RGBA frame up to the full frame. */+function cropToBounds(+ rgba: Uint8Array,+ width: number,+ height: number,+ bounds: MaskBounds,+): Uint8Array {+ const cropW = bounds.x1 - bounds.x0 + 1;+ const cropH = bounds.y1 - bounds.y0 + 1;+ const crop = new Uint8Array(width * height * 4);+ for (let y = 0; y < height; y++) {+ const srcY =+ bounds.y0 + Math.min(cropH - 1, Math.floor((y / height) * cropH));for (let x = 0; x < width; x++) {- if (mask[y * width + x] > 0) {- if (x < minX) minX = x;- if (x > maxX) maxX = x;- if (y < minY) minY = y;- if (y > maxY) maxY = y;- }+ const srcX =+ bounds.x0 + Math.min(cropW - 1, Math.floor((x / width) * cropW));+ const src = (srcY * width + srcX) * 4;+ const dst = (y * width + x) * 4;+ crop[dst] = rgba[src];+ crop[dst + 1] = rgba[src + 1];+ crop[dst + 2] = rgba[src + 2];+ crop[dst + 3] = rgba[src + 3];}}- if (maxX < minX || maxY < minY) return null;- return { width: maxX - minX + 1, height: maxY - minY + 1 };+ return crop;}-/**- * Legacy-compat: accepts a `MediaPipe` operation (from compositions targeting the old- * node) and maps its flags onto the YOLO config — pose, object tracking, segmentation,- * and passthrough/mask/skeleton/boxes/tracking modes all carry over.- */-function normalizeOperation(- raw: YoloOperation | Record<string, unknown>,-): YoloOperation | null {- if (raw.op === "Yolo") return raw as YoloOperation;-- if (raw.op === "MediaPipe") {- const legacy = raw as Record<string, unknown>;- return {- ...raw,- op: "Yolo",- enableDetection:- legacy.enableObjectTracking === true ||- legacy.mode === "boxes" ||- legacy.mode === "tracking",- enablePose:- legacy.enablePoseLandmarks === true || legacy.mode === "skeleton",- enableSegmentation:- legacy.enableSegmentation === true ||- legacy.mode === "mask" ||- legacy.mode === "matte",- confidence: (legacy.objectScoreThreshold as number | undefined) ?? 0.25,- iouThreshold: (legacy.iouThreshold as number | undefined) ?? 0.45,- classes: (legacy.objectCategories as string[] | undefined) ?? undefined,- modelsDir: (legacy.modelsDir as string | undefined) ?? undefined,- } as unknown as YoloOperation;- }-- return null;+/** Adapts a Selfie Segmenter matte to the instance-mask shape the composite path takes. */+function personMatteAsMask(+ matte: PersonMatte | undefined,+): InstanceMask | undefined {+ if (!matte || matte.coverage <= 0) return undefined;+ const area = Math.round(matte.coverage * matte.width * matte.height);+ return {+ category: "person",+ mask: matte.mask,+ width: matte.width,+ height: matte.height,+ area,+ coverage: matte.coverage,+ detectionIndex: -1,+ };}diff --git a/nodes/node-vision/src/shared/config.ts b/nodes/node-vision/src/shared/config.tsnew file mode 100644index 00000000..7f7a89c2--- /dev/null+++ b/nodes/node-vision/src/shared/config.ts@@ -0,0 +1,88 @@+import {+ createOutputItemSchema,+ SingleOutputGenericSchema,+ VirtualMediaDataSchema,+} from "@gitframes/core";+import {+ ImageResultSchema,+ MultiOutputGenericSchema,+} from "@gitframes/node-sdk";+import { z } from "zod";++/**+ * What the node draws:+ * - passthrough — the child unchanged (vision signals still update)+ * - mask — white subject silhouette on black+ * - matte — the child cut out by the subject alpha (transparent background)+ * - crop — the matte, punched in to the subject's bounds+ * - skeleton — COCO-17 pose skeleton of the primary person+ * - boxes — the child with tracked-object boxes+ * - tracking — boxes plus track centers+ */+export const VISION_MODES = [+ "passthrough",+ "mask",+ "matte",+ "crop",+ "skeleton",+ "boxes",+ "tracking",+] as const;++export type VisionMode = (typeof VISION_MODES)[number];++export const VisionNodeConfigSchema = z+ .object({+ enableDetection: z.boolean().default(true),+ enableSegmentation: z.boolean().default(false),+ enablePose: z.boolean().default(false),+ enableMatte: z.boolean().default(false),+ classes: z.array(z.string()).optional(),+ confidence: z.number().min(0).max(1).default(0.3),+ variant: z.enum(["t", "s", "m"]).default("s"),+ mode: z.enum(VISION_MODES).default("passthrough"),+ /**+ * Source of the subject alpha for mask/matte/crop: `instance` (RTMDet-Ins, any COCO+ * class, overlapping parts merged) or `selfie` (Selfie Segmenter, people only, fastest).+ */+ matteSource: z.enum(["instance", "selfie"]).default("instance"),+ /** Frames a track may go unseen before it is dropped. */+ maxMissedFrames: z.number().int().positive().default(15),+ maskThreshold: z.number().min(0).max(1).default(0.5),+ featherRadius: z.number().min(0).max(1).optional(),+ /** Grow the subject into connected pixels that differ from the frame-border backdrop. */+ keyBackground: z.boolean().default(false),+ backgroundKeyThreshold: z.number().min(0).max(255).default(70),+ modelsDir: z.string().optional(),+ baseUrl: z.string().optional(),+ visionBundle: z.custom<unknown>().optional(),+ })+ .strict();++export type VisionNodeConfig = z.infer<typeof VisionNodeConfigSchema>;++export const VisionOperationSchema = VisionNodeConfigSchema.extend({+ op: z.literal("Vision"),+ dataType: z.enum(["Image", "Video", "SVG", "GIF", "Caption"]).optional(),+ inputs: z.record(z.string(), z.unknown()).optional(),+ metadata: z.record(z.string(), z.unknown()).optional(),+});++export type VisionOperation = z.infer<typeof VisionOperationSchema>;++export const ImageVisionResultSchema = ImageResultSchema;+export type ImageVisionResult = z.infer<typeof ImageVisionResultSchema>;++const VisualOutputSchema = z.union([+ createOutputItemSchema(z.literal("Video"), VirtualMediaDataSchema),+ createOutputItemSchema(z.literal("Image"), VirtualMediaDataSchema),+ createOutputItemSchema(z.literal("SVG"), VirtualMediaDataSchema),+ createOutputItemSchema(z.literal("GIF"), VirtualMediaDataSchema),+]);++export const VideoVisionResultSchema =+ SingleOutputGenericSchema(VisualOutputSchema);+export type VideoVisionResult = z.infer<typeof VideoVisionResultSchema>;++export const VisionResultSchema = MultiOutputGenericSchema(VisualOutputSchema);+export type VisionResult = z.infer<typeof VisionResultSchema>;diff --git a/nodes/node-yolo/src/shared/index.ts b/nodes/node-vision/src/shared/index.tssimilarity index 100%rename from nodes/node-yolo/src/shared/index.tsrename to nodes/node-vision/src/shared/index.tsdiff --git a/nodes/node-yolo/tsconfig.json b/nodes/node-vision/tsconfig.jsonsimilarity index 100%rename from nodes/node-yolo/tsconfig.jsonrename to nodes/node-vision/tsconfig.jsondiff --git a/nodes/node-yolo/tsdown.config.ts b/nodes/node-vision/tsdown.config.tssimilarity index 100%rename from nodes/node-yolo/tsdown.config.tsrename to nodes/node-vision/tsdown.config.tsdiff --git a/nodes/node-yolo/src/shared/config.ts b/nodes/node-yolo/src/shared/config.tsdeleted file mode 100644index 75d625ab..00000000--- a/nodes/node-yolo/src/shared/config.ts+++ /dev/null@@ -1,91 +0,0 @@-import {- createOutputItemSchema,- SingleOutputGenericSchema,- VirtualMediaDataSchema,-} from "@gitframes/core";-import {- ImageResultSchema,- MultiOutputGenericSchema,-} from "@gitframes/node-sdk";-import { z } from "zod";--export const YoloNodeConfigSchema = z- .object({- enableDetection: z.boolean().default(true),- enableSegmentation: z.boolean().default(false),- enablePose: z.boolean().default(false),- enableClassification: z.boolean().default(false),- enableObb: z.boolean().default(false),- enableWorld: z.boolean().default(false),- prompts: z.array(z.string()).optional(),- customModel: z- .object({- path: z.string().optional(),- url: z.string().optional(),- task: z.enum(["detect", "segment", "pose", "obb", "classify", "world"]),- classes: z.array(z.string()),- imgsz: z.number().int().min(320).max(1280).optional(),- })- .optional(),- classes: z.array(z.string()).optional(),- confidence: z.number().min(0).max(1).default(0.25),- iouThreshold: z.number().min(0).max(1).default(0.45),- variant: z.enum(["n", "s", "m", "l", "x"]).default("n"),- imgsz: z.number().int().min(320).max(1280).default(640),- mode: z- .enum([- "passthrough",- "mask",- "matte",- "crop",- "skeleton",- "boxes",- "tracking",- "obb",- ])- .default("passthrough"),- delegate: z.enum(["CPU", "GPU"]).default("CPU"),- maxMissedFrames: z.number().int().positive().default(15),- maskThreshold: z.number().min(0).max(1).default(0.5),- featherRadius: z.number().min(0).max(1).optional(),- keyBackground: z.boolean().default(false),- backgroundKeyThreshold: z.number().min(0).max(255).default(70),- modelsDir: z.string().optional(),- baseUrl: z.string().optional(),- visionBundle: z.custom<unknown>().optional(),- })- .strict();--export type YoloNodeConfig = z.infer<typeof YoloNodeConfigSchema>;--export const YoloOperationSchema = YoloNodeConfigSchema.extend({- op: z.literal("Yolo"),- dataType: z.enum(["Image", "Video", "SVG", "GIF", "Caption"]).optional(),- inputs: z.record(z.string(), z.unknown()).optional(),- metadata: z.record(z.string(), z.unknown()).optional(),-});--export type YoloOperation = z.infer<typeof YoloOperationSchema>;--export const ImageYoloResultSchema = ImageResultSchema;-export type ImageYoloResult = z.infer<typeof ImageYoloResultSchema>;--export const VideoYoloResultSchema = SingleOutputGenericSchema(- z.union([- createOutputItemSchema(z.literal("Video"), VirtualMediaDataSchema),- createOutputItemSchema(z.literal("Image"), VirtualMediaDataSchema),- createOutputItemSchema(z.literal("SVG"), VirtualMediaDataSchema),- createOutputItemSchema(z.literal("GIF"), VirtualMediaDataSchema),- ]),-);-export type VideoYoloResult = z.infer<typeof VideoYoloResultSchema>;--export const YoloResultSchema = MultiOutputGenericSchema(- z.union([- createOutputItemSchema(z.literal("Video"), VirtualMediaDataSchema),- createOutputItemSchema(z.literal("Image"), VirtualMediaDataSchema),- createOutputItemSchema(z.literal("SVG"), VirtualMediaDataSchema),- createOutputItemSchema(z.literal("GIF"), VirtualMediaDataSchema),- ]),-);-export type YoloResult = z.infer<typeof YoloResultSchema>;diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.tsindex be264c78..482b53cf 100644--- a/packages/compositions/src/program/schema.ts+++ b/packages/compositions/src/program/schema.ts@@ -575,8 +575,8 @@ const BaseNodeFields = {envIntensity: z.number().min(0).max(10.0).optional(),/** Per-node motion: tracks keyed by node id. */animation: LayerAnimationSchema.default({ tracks: [] }),- /** YOLO vision configuration or effect attached directly to the node. */- yolo: z.unknown().optional(),+ /** Vision configuration or effect attached directly to the node. */+ vision: z.unknown().optional(),/** Relighting configuration attached directly to the node. */relighting: z.unknown().optional(),} as const;@@ -630,7 +630,7 @@ export interface CompositionNodeBase {durationFrames?: number;motionBlurShutter?: number;animation?: LayerAnimationSpec;- yolo?: unknown;+ vision?: unknown;relighting?: unknown;}@@ -1636,8 +1636,8 @@ export const CompositorProgramSchema = zsignals: z.record(z.string(), z.unknown()).optional(),/** Composition-wide effect chain (`comp.apply(effect)`); resolved to operations downstream. */effects: z.array(z.unknown()).optional(),- /** Composition-wide YOLO vision config or bundle. */- yolo: z.unknown().optional(),+ /** Composition-wide vision config or bundle. */+ vision: z.unknown().optional(),/** Per-frame notification hook. */onRequestFrame: z.custom<unknown>().optional(),})diff --git a/packages/compositions/src/program/to-virtual-media.ts b/packages/compositions/src/program/to-virtual-media.tsindex 59b9cf1a..166caa15 100644--- a/packages/compositions/src/program/to-virtual-media.ts+++ b/packages/compositions/src/program/to-virtual-media.ts@@ -104,19 +104,19 @@ function nodeToVirtualMedia(} as unknown as VirtualMediaData;}}- if (media && (node as any).yolo) {- const yoloRaw = (node as any).yolo;- const yoloOp =- typeof yoloRaw.toOperation === "function"- ? yoloRaw.toOperation()- : yoloRaw;+ if (media && (node as any).vision) {+ const visionRaw = (node as any).vision;+ const visionOp =+ typeof visionRaw.toOperation === "function"+ ? visionRaw.toOperation()+ : visionRaw;media = {metadata: { ...media.metadata },operation: {- op: "Yolo",+ op: "Vision",id: (node as any).id,- effect: yoloRaw,- ...yoloOp,+ effect: visionRaw,+ ...visionOp,dataType: media.operation.dataType,},children: [media],@@ -314,17 +314,17 @@ export function compositorToProgram(} as unknown as VirtualMediaData;}- const yoloRaw = (config as any).yolo ?? (options as any).yolo;- if (yoloRaw) {- const yoloOp =- typeof yoloRaw.toOperation === "function"- ? yoloRaw.toOperation()- : yoloRaw;+ const visionRaw = (config as any).vision ?? (options as any).vision;+ if (visionRaw) {+ const visionOp =+ typeof visionRaw.toOperation === "function"+ ? visionRaw.toOperation()+ : visionRaw;rootMedia = {metadata: { ...rootMedia.metadata },operation: {- op: "Yolo",- ...yoloOp,+ op: "Vision",+ ...visionOp,dataType: rootMedia.operation.dataType,},children: [rootMedia],diff --git a/packages/core/src/types/index.ts b/packages/core/src/types/index.tsindex 736db7ca..31769f81 100644--- a/packages/core/src/types/index.ts+++ b/packages/core/src/types/index.ts@@ -8,4 +8,3 @@ export * from "./services.js";export * from "./tensor.js";export * from "./video/index.js";export * from "./vision.js";-export * from "./yolo.js";diff --git a/packages/core/src/types/vision.ts b/packages/core/src/types/vision.tsindex cac60e04..492bd816 100644--- a/packages/core/src/types/vision.ts+++ b/packages/core/src/types/vision.ts@@ -1,11 +1,51 @@/**- * Shared vision types (engine-agnostic).+ * Shared vision types (engine-agnostic — no ONNX imports here).*- * These were historically declared in `mediapipe.ts`; the MediaPipe engine has been removed- * (see specs/yolov4plan.ts §Phase F), so the neutral object/landmark shapes now live here and- * are consumed by @gitframes/yolo and @gitframes/gitframes.+ * Consumed by @gitframes/vision (the inference engine) and @gitframes/gitframes (the SDK).*/+/**+ * Vision tasks and the model that serves each:+ * - `detect` / `segment` — RTMDet-Ins (COCO 80 boxes + instance masks, one forward pass)+ * - `pose` — RTMO (COCO-17 keypoints, multi-person)+ * - `matte` — MediaPipe Selfie Segmenter (fast person-vs-background alpha)+ */+export type VisionTask = "detect" | "segment" | "pose" | "matte";++/** Model size: `t` (tiny, fastest) · `s` (default) · `m` (most accurate). */+export type VisionVariant = "t" | "s" | "m";++export interface VisionConfig {+ readonly enableDetection?: boolean;+ readonly enableSegmentation?: boolean;+ readonly enablePose?: boolean;+ /** Fast person matte (Selfie Segmenter) alongside or instead of instance masks. */+ readonly enableMatte?: boolean;+ /** COCO class-name filter (all classes when omitted). */+ readonly classes?: readonly string[];+ /** Minimum detection score, 0..1. Default 0.3. */+ readonly confidence?: number;+ /** Model size. Default "s". */+ readonly variant?: VisionVariant;+ /** Model cache directory. Default `$GITFRAMES_MODELS_DIR` or `~/.cache/gitframes/models`. */+ readonly modelsDir?: string;+ /** Mirror origin for model downloads (`<baseUrl>/<filename>`). Default: pinned Hugging Face revisions. */+ readonly baseUrl?: string;+ /** Mask probability threshold (lower → fuller masks). Default 0.5. */+ readonly maskThreshold?: number;+ /** Soft-edge band around `maskThreshold`, 0..1. Default 0.05. */+ readonly featherRadius?: number;+ /** Vertical FOV (degrees) used to project landmarks into camera space. */+ readonly cameraFov?: number;+}++export interface VisionNodeSpec {+ readonly id: string;+ readonly kind: "vision";+ readonly source?: unknown;+ readonly config: VisionConfig;+}+export interface Landmark3D {readonly x: number;readonly y: number;diff --git a/packages/core/src/types/yolo.ts b/packages/core/src/types/yolo.tsdeleted file mode 100644index 7a35ddd7..00000000--- a/packages/core/src/types/yolo.ts+++ /dev/null@@ -1,51 +0,0 @@-/**- * YOLO11 AST + config types (engine-agnostic — no ONNX/mediapipe imports here).- */--export type YoloCoreTask = "detect" | "segment" | "pose" | "obb" | "classify" | "world";--export interface CustomModelConfig {- readonly path?: string;- readonly url?: string;- readonly task: YoloCoreTask;- readonly classes: readonly string[];- readonly imgsz?: number;-}--export interface YoloConfig {- readonly enableDetection?: boolean;- readonly enableSegmentation?: boolean;- readonly enablePose?: boolean;- readonly enableClassification?: boolean;- /** Rotated boxes (DOTA-style OBB models). */- readonly enableObb?: boolean;- /** Open-vocabulary YOLO-World detection. */- readonly enableWorld?: boolean;- /** Target open-vocabulary prompts (e.g. ["vintage guitar", "neon sunglasses"]). */- readonly prompts?: readonly string[];- /** Bring-your-own fine-tuned ONNX model configuration. */- readonly customModel?: CustomModelConfig;- /** COCO class-name filter (all classes when omitted). */- readonly classes?: readonly string[];- readonly confidence?: number; // default 0.25- readonly iouThreshold?: number; // NMS, default 0.45- readonly variant?: "n" | "s" | "m" | "l" | "x"; // default "n"- readonly imgsz?: number; // default 640- readonly delegate?: "cpu" | "webgpu"; // webgpu is a no-op on Node (CPU EP)- readonly modelsDir?: string;- readonly baseUrl?: string;- /** Sigmoid threshold for instance-mask pixels (lower → fuller masks). Default 0.5. */- readonly maskThreshold?: number;- /** Transition band radius around maskThreshold for anti-aliased soft alpha. */- readonly featherRadius?: number;- /** Compatibility no-op: YOLO models are ALWAYS lazy (downloaded on first use). */- readonly autoDownload?: boolean;- readonly cameraFov?: number;-}--export interface YoloNode {- readonly id: string;- readonly kind: "yolo";- readonly source?: unknown;- readonly config: YoloConfig;-}diff --git a/packages/gitframes/package.json b/packages/gitframes/package.jsonindex f82fb9c6..97aa20d2 100644--- a/packages/gitframes/package.json+++ b/packages/gitframes/package.json@@ -148,13 +148,13 @@"@gitframes/node-tile-offset": "workspace:*","@gitframes/node-unsharp-mask": "workspace:*","@gitframes/node-vignette": "workspace:*",- "@gitframes/node-yolo": "workspace:*",+ "@gitframes/node-vision": "workspace:*","@gitframes/renderer": "workspace:*","@gitframes/renderers": "workspace:*","@gitframes/tensor-webgpu": "workspace:*","@gitframes/tsconfig": "workspace:*","@gitframes/webgpu-renderers": "workspace:*",- "@gitframes/yolo": "workspace:*",+ "@gitframes/vision": "workspace:*","@types/fontkit": "^2.0.9","@types/node": "^24.1.0","@types/react": "^19.2.4",diff --git a/packages/gitframes/src/effects/classes.ts b/packages/gitframes/src/effects/classes.tsindex aef5a3e7..5b85117d 100644--- a/packages/gitframes/src/effects/classes.ts+++ b/packages/gitframes/src/effects/classes.ts@@ -1,5 +1,4 @@import { Effect } from "@gitframes/core";-import type { CustomModelConfig } from "@gitframes/yolo";import {AudioSignal,type AudioSignalOptions,@@ -689,35 +688,21 @@ export class Crop extends Effect<CropProps> {}}-/**- * @deprecated The MediaPipe engine was removed (specs/yolov4plan.ts §Phase F).- * Legacy option shape, mapped onto YOLO by the `MediaPipe` alias class below.- */-export interface MediaPipeProps {- enablePoseLandmarks?: boolean;- enableFaceLandmarks?: boolean;- enableFaceBlendshapes?: boolean;- enableSegmentation?: boolean;- mode?: "passthrough" | "mask" | "matte" | "skeleton";- delegate?: "CPU" | "GPU";- modelsDir?: string;- visionBundle?: unknown;-}--export interface YoloProps {+export interface VisionProps {+ /** Track COCO-80 objects (RTMDet-Ins boxes). Default true. */enableDetection?: boolean;+ /** Per-instance masks (RTMDet-Ins). Shares the detection pass. */enableSegmentation?: boolean;+ /** COCO-17 keypoints per person (RTMO). */enablePose?: boolean;- enableClassification?: boolean;- enableObb?: boolean;- enableWorld?: boolean;- prompts?: readonly string[];- customModel?: CustomModelConfig;+ /** Fast person-vs-background alpha (Selfie Segmenter). */+ enableMatte?: boolean;+ /** COCO class-name filter (all classes when omitted). */classes?: readonly string[];+ /** Minimum detection score, 0..1. Default 0.3. */confidence?: number;- iouThreshold?: number;- variant?: "n" | "s" | "m" | "l" | "x";- imgsz?: number;+ /** Model size: `t` fastest · `s` default · `m` most accurate. */+ variant?: "t" | "s" | "m";mode?:| "passthrough"| "mask"@@ -725,38 +710,42 @@ export interface YoloProps {| "crop"| "skeleton"| "boxes"- | "tracking"- | "obb";- delegate?: "CPU" | "GPU";+ | "tracking";+ /**+ * Subject alpha for mask/matte/crop: instance masks (default; any COCO class) or the+ * selfie matte (fastest, but only reliable when a person fills much of the frame).+ */+ matteSource?: "instance" | "selfie";+ /** Frames a track may go unseen before it is dropped. Default 15. */+ maxMissedFrames?: number;modelsDir?: string;baseUrl?: string;maskThreshold?: number;featherRadius?: number;- /** Grow the person mask into connected colourful/dark foreground (e.g. a dress). */+ /** Grow the subject into connected pixels that stand out from the backdrop. */keyBackground?: boolean;/** Colour-distance threshold for the background key (lower → more growth). Default 70. */backgroundKeyThreshold?: number;visionBundle?: unknown;}-export class Yolo extends Effect<YoloProps> {- public readonly op = "Yolo";+/**+ * On-device vision: tracks objects, segments instances, estimates pose and mattes people on+ * the rendered frame, feeding reactive signals. Models download lazily on first use.+ */+export class Vision extends Effect<VisionProps> {+ public readonly op = "Vision";public declare enableDetection?: boolean;public declare enableSegmentation?: boolean;public declare enablePose?: boolean;- public declare enableClassification?: boolean;- public declare enableObb?: boolean;- public declare enableWorld?: boolean;- public declare prompts?: readonly string[];- public declare customModel?: CustomModelConfig;+ public declare enableMatte?: boolean;public declare classes?: readonly string[];public declare confidence?: number;- public declare iouThreshold?: number;- public declare variant?: "n" | "s" | "m" | "l" | "x";- public declare imgsz?: number;- public declare mode?: YoloProps["mode"];- public declare delegate?: "CPU" | "GPU";+ public declare variant?: VisionProps["variant"];+ public declare mode?: VisionProps["mode"];+ public declare matteSource?: VisionProps["matteSource"];+ public declare maxMissedFrames?: number;public declare modelsDir?: string;public declare baseUrl?: string;public declare maskThreshold?: number;@@ -765,49 +754,21 @@ export class Yolo extends Effect<YoloProps> {public declare backgroundKeyThreshold?: number;public declare visionBundle?: unknown;- constructor(config: YoloProps = {}) {+ constructor(config: VisionProps = {}) {super({enableDetection: config.enableDetection !== false,enableSegmentation: config.enableSegmentation === true,enablePose: config.enablePose === true,- enableClassification: config.enableClassification === true,- enableObb: config.enableObb === true,- confidence: config.confidence ?? 0.25,- iouThreshold: config.iouThreshold ?? 0.45,- variant: config.variant ?? "n",- imgsz: config.imgsz ?? 640,+ enableMatte: config.enableMatte === true,+ confidence: config.confidence ?? 0.3,+ variant: config.variant ?? "s",mode: config.mode ?? "passthrough",- delegate: config.delegate ?? "CPU",+ matteSource: config.matteSource ?? "instance",...config,});}}-/**- * @deprecated Use `Yolo`. Kept as a source-compatible alias that maps the legacy MediaPipe- * option shape onto the YOLO engine (`op` is `"Yolo"`).- */-export class MediaPipe extends Yolo {- constructor(config: MediaPipeProps = {}) {- super({- enableDetection: true,- enablePose: config.enablePoseLandmarks !== false,- enableSegmentation: config.enableSegmentation === true,- mode:- config.mode === "skeleton"- ? "skeleton"- : (config.mode ?? "passthrough"),- ...(config.delegate !== undefined ? { delegate: config.delegate } : {}),- ...(config.modelsDir !== undefined- ? { modelsDir: config.modelsDir }- : {}),- ...(config.visionBundle !== undefined- ? { visionBundle: config.visionBundle }- : {}),- });- }-}-export interface Relight3DProps {lightType?: "Point" | "Spot" | "Directional" | "Rim";intensity?: EffectProp<number>;diff --git a/packages/gitframes/src/effects/custom/blur.ts b/packages/gitframes/src/effects/custom/blur.tsindex edc6a68c..1f9d1ae2 100644--- a/packages/gitframes/src/effects/custom/blur.ts+++ b/packages/gitframes/src/effects/custom/blur.ts@@ -1,4 +1,4 @@-import type { ObjectTrackSignals } from "@gitframes/yolo";+import type { ObjectTrackSignals } from "@gitframes/vision";import { BlurBase, type BlurBaseProps } from "../generated/classes.js";export interface BlurProps extends BlurBaseProps {diff --git a/packages/gitframes/src/effects/generated.test.ts b/packages/gitframes/src/effects/generated.test.tsindex 36c889ea..20cbc3fe 100644--- a/packages/gitframes/src/effects/generated.test.ts+++ b/packages/gitframes/src/effects/generated.test.ts@@ -69,7 +69,7 @@ const NOT_YET_MIGRATED = ["node-stereo-panning","node-tile-offset","node-unsharp-mask",- "node-yolo",+ "node-vision",];const exported = effects as unknown as Record<string, unknown>;diff --git a/packages/gitframes/src/effects/golden.test.ts b/packages/gitframes/src/effects/golden.test.tsindex a2bef080..df7d6c23 100644--- a/packages/gitframes/src/effects/golden.test.ts+++ b/packages/gitframes/src/effects/golden.test.ts@@ -7,7 +7,7 @@ import {Layer,type ObjectTrackSignals,Vignette,- YoloNode,+ VisionNode,} from "../index.js";/**@@ -51,7 +51,7 @@ function describeEffect(effect: Effect<object>) {function syntheticTrack(): ObjectTrackSignals {const video = Layer.video("test.mp4");- const vision = YoloNode.attach(video, {+ const vision = VisionNode.attach(video, {enableDetection: true,classes: ["person"],});diff --git a/packages/gitframes/src/effects/index.ts b/packages/gitframes/src/effects/index.tsindex 9b4250e6..d7c36486 100644--- a/packages/gitframes/src/effects/index.ts+++ b/packages/gitframes/src/effects/index.ts@@ -34,5 +34,5 @@ export { default as shadowsHighlightsRenderer } from "@gitframes/node-shadows-hiexport { default as tileOffsetRenderer } from "@gitframes/node-tile-offset/renderer";export { default as unsharpMaskRenderer } from "@gitframes/node-unsharp-mask/renderer";export { default as vignetteRenderer } from "@gitframes/node-vignette/renderer";-export { default as yoloRenderer } from "@gitframes/node-yolo/renderer";+export { default as visionRenderer } from "@gitframes/node-vision/renderer";export * from "./classes.js";diff --git a/packages/gitframes/src/index.ts b/packages/gitframes/src/index.tsindex 49dbd0c2..ac19c6a6 100644--- a/packages/gitframes/src/index.ts+++ b/packages/gitframes/src/index.ts@@ -74,13 +74,15 @@ export {type TensorViewExport,} from "@gitframes/tensor-webgpu";export {- // Shared vision types (engine-agnostic, re-exported by @gitframes/yolo from @gitframes/core)+ COCO_CLASSES,+ COCO17_BONES,+ COCO17_KEYPOINT_NAMES,+ COCO17_KEYPOINTS,type DetectedObject,+ type InstanceMask,type Landmark3D,type LandmarkCoordinateSignals,type MaskTrackSignals,- // YOLO engine surface- MEDIAPIPE_TO_YOLO_POSE_INDEX,type NormalizedLandmarkList,type ObjectAnchorName,type ObjectAnchorsSignals,@@ -89,29 +91,26 @@ export {type ObjectCollectionSignals,type ObjectKinematicsSignals,type ObjectTrackSignals,- type ObjectTrackSignals,+ type PersonMatte,type PinToLandmarkOptions,type PinToObjectOptions,- POSE_LANDMARKS_YOLO,type PoseLandmarkSignals,+ type PoseResult,PoseSkeletonRenderer,pinNodeToLandmark,pinNodeToObject,+ type SegmentationResult,SegmentationTexturePool,SpatialLandmarkTransformer,TemporalObjectTracker,type TrackedObject,- YOLO_COCO17_BONES,- YOLO_POSE_KEYPOINT_NAMES,- type YoloConfig,- type YoloInstanceMask,- YoloNode,- type YoloOBBResult,- type YoloPoseResult,- type YoloSegmentationResult,- YoloVisionBundle,- YoloVisionRunner,-} from "@gitframes/yolo";+ VisionBundle,+ type VisionConfig,+ VisionNode,+ VisionRunner,+ type VisionTask,+ type VisionVariant,+} from "@gitframes/vision";import fs from "node:fs/promises";import path from "node:path";@@ -148,7 +147,7 @@ import {type VirtualMediaData,} from "@gitframes/core";import {- analyzeSequence as analyzeYoloSequence,+ analyzeSequence as analyzeVisionFrames,type LandmarkCoordinateSignals,type ObjectTrackSignals,type PinToLandmarkOptions,@@ -156,15 +155,16 @@ import {pinNodeToLandmark,pinNodeToObject,type VisionAnalysisReport,- YoloNode,- YoloVisionBundle,- YoloVisionRunner,-} from "@gitframes/yolo";+ VisionBundle,+ VisionNode,+ VisionRunner,+ type VisionTask,+} from "@gitframes/vision";export {analyzeSequence,type VisionAnalysisReport,-} from "@gitframes/yolo";+} from "@gitframes/vision";import {type DeflickerOptions,@@ -199,8 +199,6 @@ import {type HighPassProps,Levels,type LevelsProps,- MediaPipe,- type MediaPipeProps,Modulate,type ModulateProps,MotionBlur,@@ -222,8 +220,8 @@ import {type UnsharpMaskProps,Vignette,type VignetteProps,- Yolo,- type YoloProps,+ Vision,+ type VisionProps,} from "./effects/index.js";import {type FrameGridOptions,@@ -341,13 +339,13 @@ function decorateTrackSignals<T extends ObjectTrackSignals>(return track;}-const origYoloAttach = YoloNode.attach;-YoloNode.attach = function (- source: Parameters<typeof origYoloAttach>[0],- config?: Parameters<typeof origYoloAttach>[1],- bundleOptions?: Parameters<typeof origYoloAttach>[2],-): ReturnType<typeof origYoloAttach> {- const bundle = origYoloAttach.call(this, source, config, bundleOptions);+const origVisionAttach = VisionNode.attach;+VisionNode.attach = function (+ source: Parameters<typeof origVisionAttach>[0],+ config?: Parameters<typeof origVisionAttach>[1],+ bundleOptions?: Parameters<typeof origVisionAttach>[2],+): ReturnType<typeof origVisionAttach> {+ const bundle = origVisionAttach.call(this, source, config, bundleOptions);if (bundle?.objects) {const src = source as string | MediaNode;const origGet = bundle.objects.get.bind(bundle.objects);@@ -656,8 +654,6 @@ export {type LevelChannel,Levels,type LevelsProps,- MediaPipe,- type MediaPipeProps,Modulate,type ModulateProps,MotionBlur,@@ -681,8 +677,8 @@ export {type UnsharpMaskProps,Vignette,type VignetteProps,- Yolo,- type YoloProps,+ Vision,+ type VisionProps,} from "./effects/index.js";export {type Carousel3DOptions,@@ -862,7 +858,7 @@ export interface SubjectSandwichOptions {readonly behind: readonly LayoutNode[];readonly feather?: number;readonly fit?: "cover" | "contain" | "fill";- readonly variant?: "n" | "s" | "m" | "l" | "x";+ readonly variant?: VisionProps["variant"];readonly confidence?: number;readonly keyBackground?: boolean;readonly backgroundKeyThreshold?: number;@@ -878,7 +874,7 @@ export interface SmartFramingOptions {readonly damping?: number;readonly leadHeadroom?: number;readonly fit?: "cover" | "contain" | "fill";- readonly variant?: "n" | "s" | "m" | "l" | "x";+ readonly variant?: VisionProps["variant"];readonly muted?: boolean;readonly volume?: number;}@@ -907,21 +903,16 @@ export class Composition {public audioTracks: MediaNode[] = [];public signals: Record<string, unknown> = {};private _frameHooks: Array<(ctx: FrameContext) => void> = [];- private _yoloConfig?: YoloProps | Yolo;- private _yoloBundle?: YoloVisionBundle;+ private _visionConfig?: VisionProps;+ private _visionBundle?: VisionBundle;private _effects: unknown[] = [];/*** Applies a whole-composition post-processing effect or vision pipeline.*/public apply(effect: unknown): this {- if (- effect &&- typeof effect === "object" &&- ((effect as { op?: string }).op === "MediaPipe" ||- (effect as { op?: string }).op === "Yolo")- ) {- this.withYolo(effect as Yolo);+ if (effect instanceof Vision) {+ this.withVision(effect);return this;}this._effects.push(effect);@@ -929,53 +920,37 @@ export class Composition {}/**- * @deprecated Use `withYolo`. Alias that maps the legacy MediaPipe config onto the YOLO- * engine; the legacy option shape (`enablePoseLandmarks`, `enableSegmentation`, …) is accepted.- */- public withMediaPipe(- config: MediaPipeProps | MediaPipe = {},- ): YoloVisionBundle {- const props =- config instanceof MediaPipe ? (config.config as MediaPipeProps) : config;- return this.withYolo(new MediaPipe(props));- }-- /**- * Attaches YOLO11 on top of the whole composition in one go — the rendered frame- * (composition, blur, modulate, crop — anything upstream) is the inference input.- * Models are LAZY: nothing downloads until the first frame actually runs a task.- * Returns a reactive YoloVisionBundle (objects / masks / pose / classes signals).+ * Runs vision on the whole composition — the rendered frame (composition, blur, crop,+ * anything upstream) is the inference input. Models are LAZY: nothing downloads until+ * the first frame actually runs a task.+ * Returns the reactive VisionBundle (objects / masks / pose / classes signals).*/- public withYolo(config: YoloProps | Yolo = {}): YoloVisionBundle {- if (!this._yoloBundle) {- this._yoloBundle = new YoloVisionBundle({- width: this.width,- height: this.height,- fps: this.fps,- });- }+ public withVision(config: VisionProps | Vision = {}): VisionBundle {+ this._visionBundle ??= new VisionBundle({+ width: this.width,+ height: this.height,+ fps: this.fps,+ });const rawConfig =- config instanceof Yolo- ? (config as unknown as { config: YoloProps }).config- : config;- this._yoloConfig = {+ config instanceof Vision ? (config.config as VisionProps) : config;+ this._visionConfig = {...rawConfig,- visionBundle: this._yoloBundle,+ visionBundle: this._visionBundle,};- return this._yoloBundle;+ return this._visionBundle;}/**- * One-shot, ffmpeg-free vision analysis of a media source (specs/yolov4plan.ts §Phase C).+ * One-shot, ffmpeg-free vision analysis of a media source.** Decodes frames through the same mediabunny pipeline the renderer uses, runs the requested- * YOLO tasks lazily (only those models download), temporally tracks detections, and returns- * a zod-serializable report. The decoder is released and the runner closed on completion.+ * tasks lazily (only those models download), temporally tracks detections, and returns a+ * zod-serializable report. The decoder is released and the runner closed on completion.*/public async analyzeVisionSequence(source: string,options: {- tasks?: readonly ("detect" | "segment" | "pose" | "obb" | "classify")[];+ tasks?: readonly VisionTask[];categories?: readonly string[];fps?: number;totalFrames?: number;@@ -984,6 +959,8 @@ export class Composition {modelsDir?: string;baseUrl?: string;confidence?: number;+ variant?: VisionProps["variant"];+ /** Tracker box-association IoU. Default 0.25. */iouThreshold?: number;} = {},): Promise<VisionAnalysisReport> {@@ -992,17 +969,13 @@ export class Composition {const totalFrames =options.totalFrames ?? Math.max(1, Math.round((durationMs / 1000) * fps));- const runner = YoloVisionRunner.create({- ...(options.modelsDir !== undefined- ? { modelsDir: options.modelsDir }- : {}),- ...(options.baseUrl !== undefined ? { baseUrl: options.baseUrl } : {}),+ const runner = VisionRunner.create({+ modelsDir: options.modelsDir,+ baseUrl: options.baseUrl,...(options.confidence !== undefined? { confidence: options.confidence }: {}),- ...(options.iouThreshold !== undefined- ? { iouThreshold: options.iouThreshold }- : {}),+ ...(options.variant !== undefined ? { variant: options.variant } : {}),});const decoder = mediaDecoderCache.getVideo(source, true);@@ -1024,7 +997,7 @@ export class Composition {}try {- return await analyzeYoloSequence(frameSource(), {+ return await analyzeVisionFrames(frameSource(), {runner,source,fps,@@ -1091,13 +1064,13 @@ export class Composition {| LayoutNode| LayoutNode[]| TensorPipeline- | YoloNode+ | VisionNode| { node: unknown },...rest: Array<| LayoutNode| LayoutNode[]| TensorPipeline- | YoloNode+ | VisionNode| { node: unknown }| LayerAnimation| { tracks: AnimationTrack[] }@@ -1105,7 +1078,7 @@ export class Composition {): this {if (layerOrFirst instanceof TensorPipeline) {this.children.push(layerOrFirst.toNode() as unknown as LayoutNode);- } else if (layerOrFirst instanceof YoloNode) {+ } else if (layerOrFirst instanceof VisionNode) {this.children.push(layerOrFirst.toNode() as unknown as LayoutNode);} else if (layerOrFirst &&@@ -1209,7 +1182,7 @@ export class Composition {fit,muted: true,volume: 0,- }).withYolo({+ }).withVision({mode: "matte",enableSegmentation: true,variant,@@ -1401,7 +1374,7 @@ export class Composition {fit: "cover",muted: true,volume: 0,- }).withYolo({+ }).withVision({mode: "mask",enableSegmentation: true,});@@ -1578,7 +1551,7 @@ export class Composition {fonts: mergedFonts,signals: this.signals,...(this._effects.length > 0 && { effects: this._effects }),- ...(this._yoloConfig && { yolo: this._yoloConfig }),+ ...(this._visionConfig && { vision: this._visionConfig }),...(this._frameHooks.length > 0 && {onRequestFrame: (ctx: FrameContext) => this.notifyFrame(ctx),}),@@ -1597,7 +1570,7 @@ export class Composition {durationMs: options.durationMs ?? this.durationMs,signals: options.signals ?? this.signals,...(this._effects.length > 0 && { effects: this._effects }),- ...(this._yoloConfig && { yolo: this._yoloConfig }),+ ...(this._visionConfig && { vision: this._visionConfig }),...options,});}@@ -1617,7 +1590,7 @@ export class Composition {durationMs,signals: options.signals ?? this.signals,...(this._effects.length > 0 && { effects: this._effects }),- ...(this._yoloConfig && { yolo: this._yoloConfig }),+ ...(this._visionConfig && { vision: this._visionConfig }),...options,});}@@ -1717,13 +1690,11 @@ export class Composition {*/export type AnimatableNode<T extends LayoutNode> = T & {- blendshapes?: Record<string, unknown>;- hasBlendshapes?: boolean;mask?: unknown;glass?: Record<string, unknown>;audioDeform?: MeshAudioDeformConfig | Record<string, unknown>;crop?: CropProps | Crop;- yolo?: YoloProps | Yolo;+ vision?: Vision;relighting?: NormalRelightingOptions;deflicker?: DeflickerOptions;effects?: unknown[];@@ -1761,16 +1732,14 @@ export type AnimatableNode<T extends LayoutNode> = T & {},options?: PinToObjectOptions,): AnimatableNode<T>;- /** @deprecated Face blendshapes were removed with MediaPipe; accepted but a no-op. */- driveBlendshapes(blendshapes: Record<string, unknown>): AnimatableNode<T>;withMask(mask: unknown): AnimatableNode<T>;withPbrGlass(options?: Record<string, unknown>): AnimatableNode<T>;deformWithAudio(config?: MeshAudioDeformConfig | Record<string, unknown>,): AnimatableNode<T>;withCrop(cropConfig: CropProps | Crop): AnimatableNode<T>;- withMediaPipe(config?: MediaPipeProps | MediaPipe): AnimatableNode<T>;- withYolo(config?: YoloProps | Yolo): AnimatableNode<T>;+ /** Runs vision on this node's rendered output (see `Vision` for options and modes). */+ withVision(config?: VisionProps | Vision): AnimatableNode<T>;withRelighting(options?: NormalRelightingOptions): AnimatableNode<T>;relight(options?: NormalRelightingOptions): AnimatableNode<T>;withDeflicker(options?: DeflickerOptions): AnimatableNode<T>;@@ -1880,17 +1849,6 @@ function withAnimation<T extends LayoutNode>(node: T): AnimatableNode<T> {configurable: true,});- Object.defineProperty(target, "driveBlendshapes", {- value: (blendshapes: Record<string, unknown>) => {- target.blendshapes = blendshapes;- target.hasBlendshapes = true;- return target;- },- enumerable: false,- writable: true,- configurable: true,- });-Object.defineProperty(target, "withMask", {value: (mask: unknown) => {target.mask = mask;@@ -1932,21 +1890,9 @@ function withAnimation<T extends LayoutNode>(node: T): AnimatableNode<T> {configurable: true,});- Object.defineProperty(target, "withMediaPipe", {- value: (config?: MediaPipeProps | MediaPipe) => {- // Deprecated alias — routes through the YOLO engine (`target.yolo`).- target.yolo =- config instanceof MediaPipe ? config : new MediaPipe(config);- return target;- },- enumerable: false,- writable: true,- configurable: true,- });-- Object.defineProperty(target, "withYolo", {- value: (config?: YoloProps | Yolo) => {- target.yolo = config instanceof Yolo ? config : new Yolo(config);+ Object.defineProperty(target, "withVision", {+ value: (config?: VisionProps | Vision) => {+ target.vision = config instanceof Vision ? config : new Vision(config);return target;},enumerable: false,@@ -2750,8 +2696,7 @@ export const Effect = Object.assign(CoreEffect, {pbrGlass: (config?: PBRGlassProps) => new PBRGlass(config),depthOfField: (config?: DepthOfFieldProps) => new DepthOfField(config),crop: (config?: CropProps) => new Crop(config),- mediaPipe: (config?: MediaPipeProps) => new MediaPipe(config),- yolo: (config?: YoloProps) => new Yolo(config),+ vision: (config?: VisionProps) => new Vision(config),relight3d: (config?: Relight3DProps) => new Relight3D(config),modulate: (config?: ModulateProps) => new Modulate(config),deflicker: (config?: DeflickerProps) => new TemporalDeflicker(config),diff --git a/packages/gitframes/src/section-effects.test.ts b/packages/gitframes/src/section-effects.test.tsindex c75b522d..ed7902a6 100644--- a/packages/gitframes/src/section-effects.test.ts+++ b/packages/gitframes/src/section-effects.test.ts@@ -7,17 +7,17 @@ import {Layer,type ObjectTrackSignals,Vignette,- YoloNode,+ VisionNode,} from "./index.js";describe("Unified Composition & Layer Section Architecture Suite", () => {function setupVisionWithSyntheticTrack(): {video: ReturnType<typeof Layer.video>;- vision: ReturnType<typeof YoloNode.attach>;+ vision: ReturnType<typeof VisionNode.attach>;targetPerson: ObjectTrackSignals;} {const video = Layer.video("test_source.mp4");- const vision = YoloNode.attach(video, {+ const vision = VisionNode.attach(video, {enableDetection: true,classes: ["person"],});diff --git a/packages/gitframes/src/tracked-region-blur.test.ts b/packages/gitframes/src/tracked-region-blur.test.tsindex 9869aec0..72df46b0 100644--- a/packages/gitframes/src/tracked-region-blur.test.ts+++ b/packages/gitframes/src/tracked-region-blur.test.ts@@ -5,16 +5,16 @@ import {Effect,Layer,type ObjectTrackSignals,- YoloNode,+ VisionNode,} from "./index.js";describe("Tracked Section Editing & Blur Engine Suite", () => {function setupVisionWithSyntheticTrack(): {- vision: ReturnType<typeof YoloNode.attach>;+ vision: ReturnType<typeof VisionNode.attach>;targetPerson: ObjectTrackSignals;} {const video = Layer.video("test.mp4");- const vision = YoloNode.attach(video, {+ const vision = VisionNode.attach(video, {enableDetection: true,classes: ["person"],});diff --git a/packages/gitframes/src/yolo-composability.test.ts b/packages/gitframes/src/vision-composability.test.tssimilarity index 61%rename from packages/gitframes/src/yolo-composability.test.tsrename to packages/gitframes/src/vision-composability.test.tsindex 283c3ac3..6c793b1e 100644--- a/packages/gitframes/src/yolo-composability.test.ts+++ b/packages/gitframes/src/vision-composability.test.ts@@ -1,8 +1,8 @@import { describe, expect, it } from "vitest";-import { Blur, Composition, Layer, Modulate, Yolo } from "./index.js";+import { Blur, Composition, Layer, Modulate, Vision } from "./index.js";-describe("YOLO composability (composition / crop / blur / modulate chains)", () => {- it("yolo effect applies on top of Crop -> Blur -> Modulate on a single layer", () => {+describe("Vision composability (composition / crop / blur / modulate chains)", () => {+ it("Vision effect applies on top of Crop -> Blur -> Modulate on a single layer", () => {const videoLayer = Layer.video("test_source.mp4").withCrop({leftPercentage: 10,@@ -12,7 +12,7 @@ describe("YOLO composability (composition / crop / blur / modulate chains)", ()}).apply(new Blur({ strength: 8 })).apply(new Modulate({ saturation: 1.2 }))- .apply(new Yolo({ enableDetection: true }));+ .apply(new Vision({ enableDetection: true }));const comp = Composition.create({width: 1080,@@ -26,25 +26,25 @@ describe("YOLO composability (composition / crop / blur / modulate chains)", ()const layerNode = vm.children[0];expect(layerNode.operation.op).toBe("CompositorLayer");- // pipeline wraps the video: Yolo -> Modulate -> Blur -> Crop -> Video- const yoloOp = layerNode.children[0];- expect(yoloOp.operation.op).toBe("Yolo");+ // pipeline wraps the video: Vision -> Modulate -> Blur -> Crop -> Video+ const visionOp = layerNode.children[0];+ expect(visionOp.operation.op).toBe("Vision");expect(- (yoloOp.operation as unknown as { enableDetection: boolean })+ (visionOp.operation as unknown as { enableDetection: boolean }).enableDetection,).toBe(true);// effect chains nest right-deep — walk the spine to verify orderconst ops: string[] = [];let walk: { children?: unknown[]; operation: { op: string } } | undefined =- yoloOp;+ visionOp;while (walk) {ops.push(walk.operation.op);walk = walk.children?.[0] as typeof walk | undefined;}- expect(ops).toEqual(["Yolo", "Modulate", "Blur", "Crop", "source"]);+ expect(ops).toEqual(["Vision", "Modulate", "Blur", "Crop", "source"]);});- it("withYolo() lifts a whole composition (blur + modulate upstream become the inference input)", () => {+ it("withVision() lifts a whole composition (blur + modulate upstream become the inference input)", () => {const comp = Composition.create({width: 1920,height: 1080,@@ -55,52 +55,52 @@ describe("YOLO composability (composition / crop / blur / modulate chains)", ().apply(new Blur({ strength: 4 })).apply(new Modulate({ saturation: 0.9 })),);- comp.withYolo({ enablePose: true, enableSegmentation: true });+ comp.withVision({ enablePose: true, enableSegmentation: true });- // composition-level yolo config passes through every serialization path- // (runtime passthrough, same convention as mediapipe — the zod program- // schema intentionally stays engine-agnostic)+ // composition-level vision config passes through every serialization path+ // (runtime passthrough — the zod program schema stays engine-agnostic)const spec = comp.toSpec() as unknown as {- yolo?: { enablePose?: boolean; enableSegmentation?: boolean };+ vision?: { enablePose?: boolean; enableSegmentation?: boolean };};- expect(spec.yolo).toBeDefined();- expect(spec.yolo?.enablePose).toBe(true);- expect(spec.yolo?.enableSegmentation).toBe(true);+ expect(spec.vision).toBeDefined();+ expect(spec.vision?.enablePose).toBe(true);+ expect(spec.vision?.enableSegmentation).toBe(true);- // root-level yolo becomes a wrapping op in the render tree, above Compositor+ // root-level vision becomes a wrapping op in the render tree, above Compositorconst vm = comp.toVirtualMedia();- expect(vm.operation.op).toBe("Yolo");+ expect(vm.operation.op).toBe("Vision");expect(vm.children[0]?.operation.op).toBe("Compositor");});- it("Layer.withYolo() accepts the engine node and keeps zero-I/O laziness", () => {+ it("Layer.withVision() accepts the engine node and keeps zero-I/O laziness", () => {const comp = Composition.create({width: 640,height: 480,fps: 30,durationFrames: 30,- }).add(Layer.video("clip.mp4").withYolo({ enableDetection: true }));+ }).add(Layer.video("clip.mp4").withVision({ enableDetection: true }));const vm = comp.toVirtualMedia();const layerNode = vm.children[0];- const yoloOp = layerNode.children[0];- expect(yoloOp.operation.op).toBe("Yolo");+ const visionOp = layerNode.children[0];+ expect(visionOp.operation.op).toBe("Vision");});- it("apply() dispatches a Yolo effect onto an existing composition", () => {+ it("apply() dispatches a Vision effect onto an existing composition", () => {const comp = Composition.create({width: 1280,height: 720,fps: 30,durationFrames: 90,}).add(Layer.video("clip.mp4"));- comp.apply(new Yolo({ enableClassification: true }));+ comp.apply(new Vision({ enablePose: true, enableMatte: true }));const spec = comp.toSpec() as unknown as {- yolo?: { enableClassification?: boolean };+ vision?: { enablePose?: boolean; enableMatte?: boolean };};- expect(spec.yolo?.enableClassification).toBe(true);+ expect(spec.vision?.enablePose).toBe(true);+ expect(spec.vision?.enableMatte).toBe(true);const vm = comp.toVirtualMedia();- expect(vm.operation.op).toBe("Yolo");+ expect(vm.operation.op).toBe("Vision");});it("addSubjectSandwich() creates bottom plate, behind elements, and top matte cutout", () => {@@ -128,12 +128,12 @@ describe("YOLO composability (composition / crop / blur / modulate chains)", ()expect(comp.children[0].kind).toBe("media");// Middle behind layerexpect(comp.children[1].kind).toBe("text");- // Top cutout layer with yolo mode: matte+ // Top cutout layer with vision mode: matteconst topNode = comp.children[2] as unknown as {- yolo?: { config?: { mode: string; featherRadius: number } };+ vision?: { config?: { mode: string; featherRadius: number } };};- expect(topNode.yolo?.config?.mode).toBe("matte");- expect(topNode.yolo?.config?.featherRadius).toBeCloseTo(0.04, 3);+ expect(topNode.vision?.config?.mode).toBe("matte");+ expect(topNode.vision?.config?.featherRadius).toBeCloseTo(0.04, 3);});it("addSmartFraming() wraps video in an auto-reframing box with smoothed camera pan", () => {@@ -144,7 +144,7 @@ describe("YOLO composability (composition / crop / blur / modulate chains)", ()durationFrames: 30,});- const vision = comp.withYolo({ enableDetection: true });+ const vision = comp.withVision({ enableDetection: true });comp.addSmartFraming({source: "landscape_16_9.mp4",target: vision.objects.primary,@@ -171,7 +171,7 @@ describe("YOLO composability (composition / crop / blur / modulate chains)", ()durationFrames: 30,});- const vision = comp.withYolo({ enableSegmentation: true });+ const vision = comp.withVision({ enableSegmentation: true });comp.addSubjectOutline(vision.segmentation.subject, {source: "footage.mp4",color: "#38bdf8",diff --git a/packages/gitframes/src/vision-pipeline.test.ts b/packages/gitframes/src/vision-pipeline.test.tsindex 92ef4c6a..6fe67fb3 100644--- a/packages/gitframes/src/vision-pipeline.test.ts+++ b/packages/gitframes/src/vision-pipeline.test.ts@@ -1,15 +1,8 @@import { describe, expect, it } from "vitest";-import {- Blur,- Composition,- Layer,- MediaPipe,- Vignette,- Yolo,-} from "./index.js";+import { Blur, Composition, Layer, Vignette } from "./index.js";-describe("WebGPU vision pipeable architecture (YOLO engine)", () => {- it("chains Crop -> Blur -> Yolo on a single video layer in one go", () => {+describe("WebGPU vision pipeable architecture", () => {+ it("chains Crop -> Blur -> Vision on a single video layer in one go", () => {const videoLayer = Layer.video("test_source.mp4").withCrop({leftPercentage: 27.8,@@ -18,7 +11,7 @@ describe("WebGPU vision pipeable architecture (YOLO engine)", () => {heightPercentage: 100,}).apply(new Blur({ strength: 10 }))- .withYolo({ mode: "skeleton", enablePose: true });+ .withVision({ mode: "skeleton", enablePose: true });const comp = Composition.create({width: 1080,@@ -34,12 +27,14 @@ describe("WebGPU vision pipeable architecture (YOLO engine)", () => {const layerNode = vm.children[0];expect(layerNode.operation.op).toBe("CompositorLayer");- // Layer pipeline: Yolo -> Blur -> Crop -> Video- const yoloOp = layerNode.children[0];- expect(yoloOp.operation.op).toBe("Yolo");- expect((yoloOp.operation as Record<string, unknown>).mode).toBe("skeleton");+ // Layer pipeline: Vision -> Blur -> Crop -> Video+ const visionOp = layerNode.children[0];+ expect(visionOp.operation.op).toBe("Vision");+ expect((visionOp.operation as Record<string, unknown>).mode).toBe(+ "skeleton",+ );- const blurOp = yoloOp.children[0];+ const blurOp = visionOp.children[0];expect(blurOp.operation.op).toBe("Blur");const cropOp = blurOp.children[0];@@ -52,16 +47,15 @@ describe("WebGPU vision pipeable architecture (YOLO engine)", () => {);});- it("withMediaPipe is a deprecated alias that renders through YOLO", () => {+ it("pins nodes to pose landmarks from a composition-wide vision bundle", () => {const comp = Composition.create({width: 1920,height: 1080,fps: 30,durationFrames: 90,});- const vision = comp.withMediaPipe({- mode: "passthrough",- enablePoseLandmarks: true,+ const vision = comp.withVision({+ enablePose: true,enableSegmentation: true,});@@ -73,24 +67,12 @@ describe("WebGPU vision pipeable architecture (YOLO engine)", () => {comp.add(Layer.video("background.mp4"), floatingCard);const vm = comp.toVirtualMedia();- expect(vm.operation.op).toBe("Yolo");+ expect(vm.operation.op).toBe("Vision");expect(vm.children[0]?.operation.op).toBe("Compositor");expect(floatingCard.x).toBeDefined();expect(vision.poseLandmarksTensor).toBeDefined();- expect(vision.segmentation.humanSilhouette).toBeDefined();- });-- it("accepts a `new MediaPipe(...)` effect and normalizes it onto YOLO", () => {- const comp = Composition.create({- width: 640,- height: 480,- fps: 24,- durationFrames: 24,- })- .add(Layer.video("clip.mp4"))- .apply(new MediaPipe({ enableSegmentation: true, mode: "matte" }));- const vm = comp.toVirtualMedia();- expect(vm.operation.op).toBe("Yolo");+ expect(vision.segmentation.subject).toBeDefined();+ expect(vision.segmentation.matte.coverage).toBeDefined();});it("supports whole-composition post-processing effect chaining", () => {@@ -105,9 +87,4 @@ describe("WebGPU vision pipeable architecture (YOLO engine)", () => {expect(vignetteNode.operation.op).toBe("Vignette");expect(vignetteNode.children[0].operation.op).toBe("Compositor");});-- it("Yolo and MediaPipe effects are distinct classes", () => {- expect(new Yolo()).not.toBeInstanceOf(MediaPipe);- expect(new MediaPipe()).toBeInstanceOf(Yolo);- });});diff --git a/packages/renderer/src/tests/node-manifest.test.ts b/packages/renderer/src/tests/node-manifest.test.tsindex e3272521..715559e6 100644--- a/packages/renderer/src/tests/node-manifest.test.ts+++ b/packages/renderer/src/tests/node-manifest.test.ts@@ -45,7 +45,8 @@ describe("node manifests (package.json `gitframes` block)", () => {"RefractionCaustics3D",);expect(byDir.get("node-extract-lut")?.type).toBe("ExtractLUT");- expect(byDir.get("node-yolo")?.aliases).toEqual(["MediaPipe"]);+ expect(byDir.get("node-vision")?.type).toBe("Vision");+ expect(byDir.get("node-vision")?.aliases).toEqual([]);expect(byDir.get("node-deflicker")?.aliases).toEqual(["Deflicker"]);});@@ -156,8 +157,7 @@ describe("discoverAndRegisterNodeRenderers", () => {"Blur","Vignette","ColorBalance",- "Yolo",- "MediaPipe",+ "Vision","TemporalDeflicker","Deflicker","Relight3D",@@ -174,6 +174,9 @@ describe("discoverAndRegisterNodeRenderers", () => {"RefractionCaustics","ExtractLut","Relight3d",+ // Removed engines.+ "Yolo",+ "MediaPipe",]) {expect(webgpuRegistry.has(op), op).toBe(false);}diff --git a/packages/yolo/package.json b/packages/vision/package.jsonsimilarity index 79%rename from packages/yolo/package.jsonrename to packages/vision/package.jsonindex 55e76a60..40179598 100644--- a/packages/yolo/package.json+++ b/packages/vision/package.json@@ -1,5 +1,6 @@{- "name": "@gitframes/yolo",+ "name": "@gitframes/vision",+ "description": "On-device vision for gitframes: RTMDet-Ins detection + instance masks, RTMO pose, and Selfie Segmenter mattes over ONNX Runtime.","version": "1.2.35","private": false,"publishConfig": {@@ -7,7 +8,7 @@},"type": "module","sideEffects": false,- "license": "AGPL-3.0",+ "license": "Apache-2.0","module": "./dist/index.mjs","types": "./dist/index.d.mts","exports": {@@ -37,7 +38,8 @@"build": "tsdown","dev": "tsdown --watch","clean": "rm -rf .turbo && rm -rf node_modules && rm -rf dist",- "test": "vitest run"+ "test": "vitest run",+ "test:models": "GITFRAMES_VISION_E2E=1 vitest run src/models.e2e.test.ts"},"peerDependencies": {"onnxruntime-web": "^1.24.0"@@ -57,6 +59,7 @@"devDependencies": {"@gitframes/tsconfig": "workspace:*","@types/node": "^24.1.0",+ "skia-canvas": "^3.0.8","tsdown": "^0.19.0","typescript": "^5.8.3","vitest": "^4.1.5"diff --git a/packages/vision/src/analysis/analyze-sequence.test.ts b/packages/vision/src/analysis/analyze-sequence.test.tsnew file mode 100644index 00000000..f8c1f6c3--- /dev/null+++ b/packages/vision/src/analysis/analyze-sequence.test.ts@@ -0,0 +1,136 @@+import { describe, expect, it } from "vitest";+import { VisionRunner } from "../runner/vision-runner.js";+import type {+ SessionProvider,+ VisionSession,+ VisionTensorOutputs,+} from "../runtime/session-provider.js";+import {+ fakeStore,+ rtmdetInsOutput,+ rtmoOutput,+ ScriptedProvider,+} from "../test-support.js";+import type { VisionImageInput } from "../types.js";+import { analyzeSequence } from "./analyze-sequence.js";++/** RTMDet-Ins session whose single person moves right by 10px on every call. */+class MovingProvider implements SessionProvider {+ public readonly kind = "node" as const;+ async createSession(): Promise<VisionSession> {+ let step = 0;+ return {+ run: async (): Promise<VisionTensorOutputs> => {+ const out = rtmdetInsOutput();+ const dets = out.dets.data as Float32Array;+ dets[0] += step * 10;+ dets[2] += step * 10;+ step++;+ return out;+ },+ release: () => {},+ };+ }+}++async function* frames(count: number): AsyncIterable<VisionImageInput> {+ for (let i = 0; i < count; i++) {+ yield {+ data: new Uint8ClampedArray(640 * 640 * 4).fill(128),+ width: 640,+ height: 640,+ };+ }+}++describe("analyzeSequence", () => {+ it("produces a deterministic, zod-valid report for a moving object", async () => {+ const { store } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new MovingProvider(),+ });+ const report = await analyzeSequence(frames(12), {+ runner,+ source: "clip.mp4",+ fps: 24,+ tasks: ["detect"],+ });++ expect(report.source).toBe("clip.mp4");+ expect(report.totalFrames).toBe(12);+ expect(report.durationSec).toBeCloseTo(0.5, 6);+ expect(report.tasks).toEqual(["detect"]);+ expect(Object.keys(report.modelDownloads)).toEqual(["rtmdet-ins-s"]);+ expect(report.modelDownloads["rtmdet-ins-s"].bytes).toBeGreaterThan(0);+ expect(report.inferenceMs).toBeGreaterThanOrEqual(0);++ expect(report.trackCount).toBe(1);+ expect(report.classes.person).toMatchObject({+ framesPresent: 12,+ totalFrames: 12,+ tracks: 1,+ });+ expect(report.classes.person.maxConfidence).toBeCloseTo(0.9, 5);++ const track = report.tracks[0];+ expect(track.category).toBe("person");+ expect(track.frames).toEqual([0, 11]);+ expect(track.centerPath.length).toBeGreaterThan(0);+ expect(track.speedMeanPxS).toBeGreaterThan(0);++ // zod round-trip+ const { VisionAnalysisReportSchema } = await import("../schemas.js");+ expect(VisionAnalysisReportSchema.safeParse(report).success).toBe(true);+ runner.close();+ });++ it("runs only the requested tasks' models", async () => {+ const { store, fetches } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new ScriptedProvider([rtmoOutput()]),+ });+ const report = await analyzeSequence(frames(3), {+ runner,+ tasks: ["pose"],+ });+ expect(report.tasks).toEqual(["pose"]);+ expect(Object.keys(report.modelDownloads)).toEqual(["rtmo-s"]);+ expect(fetches).toHaveLength(1);+ expect(report.trackCount).toBe(0);+ runner.close();+ });++ it("aggregates mask coverage from the shared segmentation pass", async () => {+ const { store } = fakeStore();+ const provider = new ScriptedProvider([rtmdetInsOutput()]);+ const runner = VisionRunner.create({ store, provider });+ const report = await analyzeSequence(frames(2), {+ runner,+ tasks: ["detect", "segment"],+ includeMasks: true,+ });+ expect(provider.created).toHaveLength(1);+ expect(provider.created[0].inputs).toHaveLength(2); // one pass per frame+ expect(report.masks.person.meanCoverage).toBeGreaterThan(0);+ expect(report.trackCount).toBe(1);+ runner.close();+ });++ it("respects the category filter", async () => {+ const { store } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new MovingProvider(),+ });+ const report = await analyzeSequence(frames(4), {+ runner,+ tasks: ["detect"],+ categories: ["car"],+ });+ expect(report.trackCount).toBe(0);+ expect(report.classes).toEqual({});+ runner.close();+ });+});diff --git a/packages/yolo/src/analysis/analyze-sequence.ts b/packages/vision/src/analysis/analyze-sequence.tssimilarity index 77%rename from packages/yolo/src/analysis/analyze-sequence.tsrename to packages/vision/src/analysis/analyze-sequence.tsindex acd9f3f9..ccdb73ae 100644--- a/packages/yolo/src/analysis/analyze-sequence.ts+++ b/packages/vision/src/analysis/analyze-sequence.ts@@ -1,33 +1,34 @@import type { DetectedObject } from "@gitframes/core";-import { YOLO_MODELS, type YoloTask } from "../model/registry.js";-import type { YoloVisionRunner } from "../runner/yolo-runner.js";+import { VISION_MODELS, type VisionTask } from "../model/registry.js";+import type { VisionRunner } from "../runner/vision-runner.js";import { TemporalObjectTracker } from "../tracking/temporal-object-tracker.js";import type {+ InstanceMask,VisionAnalysisClassSummary,VisionAnalysisReport,VisionAnalysisTrackSummary,- YoloImageInput,- YoloInstanceMask,+ VisionImageInput,} from "../types.js";/**- * One-shot, ffmpeg-free sequence analysis (specs/yolov4plan.ts §Phase C).+ * One-shot, ffmpeg-free sequence analysis.** Pure frame-iterable analyzer: it never decodes media itself, so it stays engine-only and* offline-testable. Callers (e.g. the gitframes ingestion adapter) supply decoded frames.* Model loading remains lazy — the runner downloads each task's model on the first frame- * that actually runs that task (I1), and only the tasks requested here are ever touched (I2).+ * that actually runs that task, and only the tasks requested here are ever touched.*/export interface AnalyzeSequenceOptions {- readonly runner: YoloVisionRunner;+ readonly runner: VisionRunner;/** Label for the report ("clip.mp4" or similar). Defaults to "frames". */readonly source?: string;readonly fps?: number;/** Tasks to run. Defaults to `["detect"]`. Only these models are ever downloaded. */- readonly tasks?: readonly YoloTask[];+ readonly tasks?: readonly VisionTask[];/** Restrict class summaries to these categories (all when omitted). */readonly categories?: readonly string[];+ /** Tracker box-association IoU. Default 0.25. */readonly iouThreshold?: number;readonly maxMissedFrames?: number;/** Sample the per-track center path every N frames. Default: ~50 points across the clip. */@@ -56,7 +57,7 @@ const now = (): number =>: Date.now();export async function analyzeSequence(- frames: AsyncIterable<YoloImageInput> | Iterable<YoloImageInput>,+ frames: AsyncIterable<VisionImageInput> | Iterable<VisionImageInput>,options: AnalyzeSequenceOptions,): Promise<VisionAnalysisReport> {const { runner } = options;@@ -65,20 +66,13 @@ export async function analyzeSequence(options.tasks && options.tasks.length > 0? options.tasks: (["detect"] as const);- const taskSet = new Set<YoloTask>(tasks);+ const taskSet = new Set<VisionTask>(tasks);const categoryFilter = options.categories? new Set(options.categories.map((c) => c.toLowerCase())): null;const includeMasks = options.includeMasks === true && taskSet.has("segment");-- // Detection source, in preference order (detect → segment → obb).- const detectionTask: YoloTask | null = taskSet.has("detect")- ? "detect"- : taskSet.has("segment")- ? "segment"- : taskSet.has("obb")- ? "obb"- : null;+ // detect and segment share one RTMDet-Ins pass, so either yields detections.+ const tracksObjects = taskSet.has("detect") || taskSet.has("segment");const perFrameDetections: Array<readonly DetectedObject[]> = [];const modelDownloads: Record<string, { bytes: number; ms: number }> = {};@@ -90,22 +84,23 @@ export async function analyzeSequence(!categoryFilter || categoryFilter.has(category.toLowerCase());const runTask = async <T>(- task: YoloTask,+ task: VisionTask,+ frame: VisionImageInput,fn: () => Promise<T>,): Promise<T> => {- const key = runner.modelKeyFor(task);+ const key = runner.modelKeyFor(task, frame.width / frame.height);const wasReady = runner.downloadStatus.get(key) === "ready";const start = now();const result = await fn();const elapsed = now() - start;inferenceMs += elapsed;if (!wasReady && !modelDownloads[key]) {- modelDownloads[key] = { bytes: YOLO_MODELS[key].bytesFp32, ms: elapsed };+ modelDownloads[key] = { bytes: VISION_MODELS[key].bytes, ms: elapsed };}return result;};- const recordMasks = (masks: readonly YoloInstanceMask[]): void => {+ const recordMasks = (masks: readonly InstanceMask[]): void => {for (const m of masks) {if (!accepts(m.category)) continue;const entry = maskCoverage.get(m.category) ?? { sum: 0, count: 0 };@@ -118,29 +113,18 @@ export async function analyzeSequence(for await (const frame of frames) {let detections: readonly DetectedObject[] = [];- // Segmentation runs once if it is either the detection source or a mask source.- if (includeMasks || detectionTask === "segment") {- const seg = await runTask("segment", () => runner.segment(frame));- if (includeMasks) recordMasks(seg.masks);- if (detectionTask === "segment") detections = seg.detections;- }-- if (detectionTask === "detect") {- detections = await runTask("detect", () => runner.detect(frame));- } else if (detectionTask === "obb") {- const obb = await runTask("obb", () => runner.detectObb(frame));- detections = obb.detections.map((d) => ({- category: d.category,- score: d.score,- boundingBox: d.boundingBox,- }));+ if (includeMasks) {+ const seg = await runTask("segment", frame, () => runner.segment(frame));+ recordMasks(seg.masks);+ detections = seg.detections;+ } else if (tracksObjects) {+ detections = await runTask("detect", frame, () => runner.detect(frame));}-if (taskSet.has("pose")) {- await runTask("pose", () => runner.pose(frame));+ await runTask("pose", frame, () => runner.pose(frame));}- if (taskSet.has("classify")) {- await runTask("classify", () => runner.classify(frame));+ if (taskSet.has("matte")) {+ await runTask("matte", frame, () => runner.matte(frame));}perFrameDetections.push(diff --git a/packages/vision/src/decode/preprocess.ts b/packages/vision/src/decode/preprocess.tsnew file mode 100644index 00000000..32c2c590--- /dev/null+++ b/packages/vision/src/decode/preprocess.ts@@ -0,0 +1,214 @@+/**+ * Image → NCHW float tensor preprocessing, plus the inverse mapping back to source pixels.+ *+ * Each model family was exported with its own preprocessing contract:+ *+ * | Model | Resize | Pad | Channels | Values |+ * | ----------- | ---------------------------- | ---- | -------- | -------------------------- |+ * | RTMDet-Ins | keep ratio, content top-left | 114 | BGR | (v − mean) / std (ImageNet) |+ * | RTMO | keep ratio, content centered | 114 | BGR | raw 0..255 |+ * | Selfie | stretch to input | — | RGB | v / 255 |+ *+ * Getting any of these wrong does not crash — it silently degrades accuracy — so the specs+ * live here and every runner path goes through `imageToTensor`. Input sizes differ per model+ * (RTMO-t is 416², the rest of RTMO 640²), so they come from the registry, never from here.+ */++export interface RgbaImage {+ readonly data: Uint8ClampedArray | Uint8Array;+ readonly width: number;+ readonly height: number;+}++export interface PreprocessSpec {+ /** Model input size in pixels. */+ readonly width: number;+ readonly height: number;+ /** `topLeft` / `center` keep aspect ratio and pad; `stretch` fills the input. */+ readonly fit: "topLeft" | "center" | "stretch";+ readonly channels: "rgb" | "bgr";+ /** Per-channel mean, in output channel order, on the 0..255 scale. */+ readonly mean: readonly [number, number, number];+ /** Per-channel std, in output channel order, on the 0..255 scale. */+ readonly std: readonly [number, number, number];+ /** Padding value on the 0..255 scale (normalized like any other pixel). */+ readonly padValue: number;+}++/** Maps model-input pixels back to source pixels: `src = (input − offset) / scale`. */+export interface InputTransform {+ readonly scaleX: number;+ readonly scaleY: number;+ readonly offsetX: number;+ readonly offsetY: number;+ readonly inputWidth: number;+ readonly inputHeight: number;+}++const IMAGENET_BGR_MEAN = [103.53, 116.28, 123.675] as const;+const IMAGENET_BGR_STD = [57.375, 57.12, 58.395] as const;++type Normalization = Omit<PreprocessSpec, "width" | "height">;++/** Per-family normalization; the input size always comes from the model's registry entry. */+export const PREPROCESS_BY_FAMILY: Readonly<+ Record<"rtmdet-ins" | "rtmo" | "selfie", Normalization>+> = {+ "rtmdet-ins": {+ fit: "topLeft",+ channels: "bgr",+ mean: IMAGENET_BGR_MEAN,+ std: IMAGENET_BGR_STD,+ padValue: 114,+ },+ rtmo: {+ fit: "center",+ channels: "bgr",+ mean: [0, 0, 0],+ std: [1, 1, 1],+ padValue: 114,+ },+ selfie: {+ fit: "stretch",+ channels: "rgb",+ mean: [0, 0, 0],+ std: [255, 255, 255],+ padValue: 0,+ },+};++export function preprocessSpec(+ family: keyof typeof PREPROCESS_BY_FAMILY,+ input: readonly [number, number],+): PreprocessSpec {+ return { ...PREPROCESS_BY_FAMILY[family], width: input[0], height: input[1] };+}++export function computeInputTransform(+ srcW: number,+ srcH: number,+ spec: Pick<PreprocessSpec, "width" | "height" | "fit">,+): InputTransform {+ if (spec.fit === "stretch") {+ return {+ scaleX: spec.width / srcW,+ scaleY: spec.height / srcH,+ offsetX: 0,+ offsetY: 0,+ inputWidth: spec.width,+ inputHeight: spec.height,+ };+ }+ const scale = Math.min(spec.width / srcW, spec.height / srcH);+ const contentW = Math.round(srcW * scale);+ const contentH = Math.round(srcH * scale);+ const center = spec.fit === "center";+ return {+ scaleX: scale,+ scaleY: scale,+ offsetX: center ? (spec.width - contentW) / 2 : 0,+ offsetY: center ? (spec.height - contentH) / 2 : 0,+ inputWidth: spec.width,+ inputHeight: spec.height,+ };+}++/** Model-input point → source pixel point. */+export function toSourcePoint(+ x: number,+ y: number,+ t: InputTransform,+): { x: number; y: number } {+ return { x: (x - t.offsetX) / t.scaleX, y: (y - t.offsetY) / t.scaleY };+}++/**+ * Resizes RGBA → NCHW float32 `[1, 3, height, width]` with bilinear sampling, padding,+ * channel reordering and normalization per `spec`. Reuses `out` when large enough+ * (hot-path frame loop).+ */+export function imageToTensor(+ image: RgbaImage,+ spec: PreprocessSpec,+ out?: Float32Array,+): { tensor: Float32Array; transform: InputTransform } {+ const { width: srcW, height: srcH, data: src } = image;+ const { width: W, height: H } = spec;+ const transform = computeInputTransform(srcW, srcH, spec);+ const { scaleX, scaleY, offsetX, offsetY } = transform;++ const plane = W * H;+ if (!out || out.length < 3 * plane) {+ out = new Float32Array(3 * plane);+ }++ // Source channel feeding each output plane (RGBA byte offsets).+ const order = spec.channels === "bgr" ? [2, 1, 0] : [0, 1, 2];+ const [m0, m1, m2] = spec.mean;+ const inv0 = 1 / spec.std[0];+ const inv1 = 1 / spec.std[1];+ const inv2 = 1 / spec.std[2];+ const pad0 = (spec.padValue - m0) * inv0;+ const pad1 = (spec.padValue - m1) * inv1;+ const pad2 = (spec.padValue - m2) * inv2;+ const [c0, c1, c2] = order;++ const x0 = Math.ceil(offsetX);+ const y0 = Math.ceil(offsetY);+ const x1 = Math.min(W, Math.floor(offsetX + srcW * scaleX)) - 1;+ const y1 = Math.min(H, Math.floor(offsetY + srcH * scaleY)) - 1;++ for (let y = 0; y < H; y++) {+ const rowBase = y * W;+ const paddedRow = y < y0 || y > y1;+ // Pixel-center sampling keeps stretch/letterbox resizes unbiased.+ const sy = Math.max(0, (y + 0.5 - offsetY) / scaleY - 0.5);+ const iy0 = Math.min(srcH - 1, Math.floor(sy));+ const iy1 = Math.min(srcH - 1, iy0 + 1);+ const fy = sy - iy0;+ for (let x = 0; x < W; x++) {+ const o = rowBase + x;+ if (paddedRow || x < x0 || x > x1) {+ out[o] = pad0;+ out[plane + o] = pad1;+ out[2 * plane + o] = pad2;+ continue;+ }+ const sx = Math.max(0, (x + 0.5 - offsetX) / scaleX - 0.5);+ const ix0 = Math.min(srcW - 1, Math.floor(sx));+ const ix1 = Math.min(srcW - 1, ix0 + 1);+ const fx = sx - ix0;+ const i00 = (iy0 * srcW + ix0) * 4;+ const i10 = (iy0 * srcW + ix1) * 4;+ const i01 = (iy1 * srcW + ix0) * 4;+ const i11 = (iy1 * srcW + ix1) * 4;+ const w00 = (1 - fx) * (1 - fy);+ const w10 = fx * (1 - fy);+ const w01 = (1 - fx) * fy;+ const w11 = fx * fy;+ out[o] =+ (src[i00 + c0] * w00 ++ src[i10 + c0] * w10 ++ src[i01 + c0] * w01 ++ src[i11 + c0] * w11 -+ m0) *+ inv0;+ out[plane + o] =+ (src[i00 + c1] * w00 ++ src[i10 + c1] * w10 ++ src[i01 + c1] * w01 ++ src[i11 + c1] * w11 -+ m1) *+ inv1;+ out[2 * plane + o] =+ (src[i00 + c2] * w00 ++ src[i10 + c2] * w10 ++ src[i01 + c2] * w01 ++ src[i11 + c2] * w11 -+ m2) *+ inv2;+ }+ }++ return { tensor: out, transform };+}diff --git a/packages/vision/src/decode/rtmdet-ins.ts b/packages/vision/src/decode/rtmdet-ins.tsnew file mode 100644index 00000000..84b1416a--- /dev/null+++ b/packages/vision/src/decode/rtmdet-ins.ts@@ -0,0 +1,180 @@+import type { DetectedObject } from "@gitframes/core";+import type { InstanceMask, SegmentationResult } from "../types.js";+import { type InputTransform, toSourcePoint } from "./preprocess.js";++/**+ * RTMDet-Ins decode (mmdeploy end2end export — NMS and mask assembly are baked into the graph).+ *+ * Outputs:+ * dets [1, N, 5] x0, y0, x1, y1 (input px), score+ * labels [1, N] COCO-80 class index (int64)+ * masks [1, N, Hm, Wm] per-instance probabilities (already sigmoided), input-aligned+ */+export interface RtmdetInsOutputs {+ readonly dets: ArrayLike<number>;+ readonly labels: ArrayLike<number | bigint>;+ /** Omit to decode boxes only (detection without masks). */+ readonly masks?: ArrayLike<number>;+ readonly count: number;+ readonly maskWidth: number;+ readonly maskHeight: number;+}++export interface RtmdetInsDecodeOptions {+ readonly confidence: number;+ /** Class-name filter, case-insensitive (all classes when omitted). */+ readonly classes?: readonly string[];+ readonly classNames: readonly string[];+ readonly transform: InputTransform;+ readonly sourceWidth: number;+ readonly sourceHeight: number;+ /** Mask probability threshold. Default 0.5. */+ readonly maskThreshold?: number;+ /** Soft-edge band around the threshold. Default 0.05. */+ readonly featherRadius?: number;+}++/** Source pixels a mask may extend past its box (instance masks are not box-clipped). */+const MASK_BOX_PAD_PX = 8;++export function decodeRtmdetIns(+ out: RtmdetInsOutputs,+ opts: RtmdetInsDecodeOptions,+): SegmentationResult {+ const { sourceWidth: sw, sourceHeight: sh, transform } = opts;+ const classFilter = opts.classes+ ? new Set(opts.classes.map((c) => c.toLowerCase()))+ : null;++ const kept: Array<{ index: number; detection: DetectedObject }> = [];+ for (let i = 0; i < out.count; i++) {+ const score = out.dets[i * 5 + 4];+ if (!(score >= opts.confidence)) continue;+ const classIndex = Number(out.labels[i]);+ const category = opts.classNames[classIndex] ?? `class_${classIndex}`;+ if (classFilter && !classFilter.has(category.toLowerCase())) continue;++ const p0 = toSourcePoint(out.dets[i * 5], out.dets[i * 5 + 1], transform);+ const p1 = toSourcePoint(+ out.dets[i * 5 + 2],+ out.dets[i * 5 + 3],+ transform,+ );+ const x0 = clamp(p0.x, 0, sw);+ const y0 = clamp(p0.y, 0, sh);+ const width = clamp(p1.x, 0, sw) - x0;+ const height = clamp(p1.y, 0, sh) - y0;+ if (width <= 0 || height <= 0) continue;++ kept.push({+ index: i,+ detection: {+ category,+ score,+ boundingBox: {+ originX: x0,+ originY: y0,+ width,+ height,+ normalizedX: x0 / sw,+ normalizedY: y0 / sh,+ normalizedWidth: width / sw,+ normalizedHeight: height / sh,+ },+ },+ });+ }+ kept.sort((a, b) => b.detection.score - a.detection.score);++ const detections = kept.map((k) => k.detection);+ const masks: InstanceMask[] = [];+ if (!out.masks) return { detections, masks };++ const alphaOf = probabilityToAlpha(+ opts.maskThreshold ?? 0.5,+ opts.featherRadius ?? 0.05,+ );+ const { maskWidth: mw, maskHeight: mh } = out;+ const plane = mw * mh;+ // Mask grid px per input px (exports may emit masks below input resolution).+ const gx = mw / transform.inputWidth;+ const gy = mh / transform.inputHeight;++ for (let d = 0; d < kept.length; d++) {+ const { index, detection } = kept[d];+ const box = detection.boundingBox;+ const base = index * plane;+ const bx0 = Math.max(0, Math.floor(box.originX) - MASK_BOX_PAD_PX);+ const by0 = Math.max(0, Math.floor(box.originY) - MASK_BOX_PAD_PX);+ const bx1 = Math.min(+ sw - 1,+ Math.ceil(box.originX + box.width) + MASK_BOX_PAD_PX,+ );+ const by1 = Math.min(+ sh - 1,+ Math.ceil(box.originY + box.height) + MASK_BOX_PAD_PX,+ );++ const mask = new Uint8Array(sw * sh);+ let area = 0;+ for (let sy = by0; sy <= by1; sy++) {+ // source pixel center → input px → mask grid (pixel-center aligned)+ const my = ((sy + 0.5) * transform.scaleY + transform.offsetY) * gy - 0.5;+ const my0 = clamp(Math.floor(my), 0, mh - 1);+ const my1 = Math.min(mh - 1, my0 + 1);+ const fy = clamp(my - my0, 0, 1);+ const row = sy * sw;+ for (let sx = bx0; sx <= bx1; sx++) {+ const mx =+ ((sx + 0.5) * transform.scaleX + transform.offsetX) * gx - 0.5;+ const mx0 = clamp(Math.floor(mx), 0, mw - 1);+ const mx1 = Math.min(mw - 1, mx0 + 1);+ const fx = clamp(mx - mx0, 0, 1);+ const p =+ out.masks[base + my0 * mw + mx0] * (1 - fx) * (1 - fy) ++ out.masks[base + my0 * mw + mx1] * fx * (1 - fy) ++ out.masks[base + my1 * mw + mx0] * (1 - fx) * fy ++ out.masks[base + my1 * mw + mx1] * fx * fy;+ const alpha = alphaOf(p);+ if (alpha > 0) {+ mask[row + sx] = alpha;+ area += alpha / 255;+ }+ }+ }++ area = Math.round(area);+ if (area === 0) continue;+ masks.push({+ category: detection.category,+ mask,+ width: sw,+ height: sh,+ area,+ coverage: area / (sw * sh),+ detectionIndex: d,+ });+ }++ return { detections, masks };+}++/**+ * Probability → 0..255 alpha. A hard threshold when `feather` is 0, otherwise a linear ramp+ * across `[threshold − feather, threshold + feather]` for anti-aliased edges.+ */+export function probabilityToAlpha(+ threshold: number,+ feather: number,+): (p: number) => number {+ if (feather <= 0) {+ return (p) => (p > threshold ? 255 : 0);+ }+ const lo = threshold - feather;+ const inv = 255 / (2 * feather);+ return (p) => (p <= lo ? 0 : p >= threshold + feather ? 255 : (p - lo) * inv);+}++function clamp(v: number, lo: number, hi: number): number {+ return v < lo ? lo : v > hi ? hi : v;+}diff --git a/packages/vision/src/decode/rtmo.test.ts b/packages/vision/src/decode/rtmo.test.tsnew file mode 100644index 00000000..e906f4a0--- /dev/null+++ b/packages/vision/src/decode/rtmo.test.ts@@ -0,0 +1,71 @@+import { describe, expect, it } from "vitest";+import { computeInputTransform } from "./preprocess.js";+import { decodeRtmo } from "./rtmo.js";++const transform = computeInputTransform(640, 640, {+ width: 640,+ height: 640,+ fit: "center",+});++/** People as [x0, y0, x1, y1, score, visibleKeypoints]. */+function outputs(people: number[][]) {+ const dets = new Float32Array(people.length * 5);+ const keypoints = new Float32Array(people.length * 17 * 3);+ people.forEach(([x0, y0, x1, y1, score, visible], i) => {+ dets.set([x0, y0, x1, y1, score], i * 5);+ for (let k = 0; k < 17; k++) {+ const o = (i * 17 + k) * 3;+ keypoints.set([x0 + 1, y0 + k, k < visible ? 0.9 : 0.05], o);+ }+ });+ return { dets, keypoints, count: people.length, keypointStride: 3 };+}++const opts = {+ confidence: 0.3,+ transform,+ sourceWidth: 640,+ sourceHeight: 640,+};++describe("decodeRtmo", () => {+ it("suppresses near-duplicate people, keeping the higher score", () => {+ const res = decodeRtmo(+ outputs([+ [100, 100, 200, 400, 0.42, 17],+ [102, 101, 201, 398, 0.67, 17],+ [400, 100, 500, 400, 0.5, 17],+ ]),+ opts,+ );+ expect(res.people.map((p) => p.score)).toEqual([+ expect.closeTo(0.67, 5),+ expect.closeTo(0.5, 5),+ ]);+ });++ it("suppresses duplicates whose boxes differ but whose joints coincide", () => {+ const res = decodeRtmo(+ outputs([+ [100, 100, 200, 400, 0.67, 17],+ [100, 100, 340, 420, 0.42, 17], // wider box, same joints+ ]),+ opts,+ );+ expect(res.people).toHaveLength(1);+ expect(res.people[0].score).toBeCloseTo(0.67);+ });++ it("drops detections with too few visible keypoints", () => {+ const res = decodeRtmo(+ outputs([+ [100, 100, 200, 400, 0.9, 1],+ [400, 100, 500, 400, 0.9, 5],+ ]),+ opts,+ );+ expect(res.people).toHaveLength(1);+ expect(res.people[0].boundingBox.originX).toBeCloseTo(400);+ });+});diff --git a/packages/vision/src/decode/rtmo.ts b/packages/vision/src/decode/rtmo.tsnew file mode 100644index 00000000..7e2b677f--- /dev/null+++ b/packages/vision/src/decode/rtmo.ts@@ -0,0 +1,129 @@+import { computeIoU } from "../tracking/temporal-object-tracker.js";+import type { PoseKeypoint, PosePerson, PoseResult } from "../types.js";+import { type InputTransform, toSourcePoint } from "./preprocess.js";++/**+ * RTMO decode (one-stage multi-person pose; NMS is baked into the export).+ *+ * Outputs:+ * dets [1, N, 5] x0, y0, x1, y1 (input px), person score+ * keypoints [1, N, 17, 3] x, y (input px), keypoint score — COCO-17 order+ */+export interface RtmoOutputs {+ readonly dets: ArrayLike<number>;+ readonly keypoints: ArrayLike<number>;+ readonly count: number;+ /** Values per keypoint (3 for x, y, score). */+ readonly keypointStride: number;+}++export interface RtmoDecodeOptions {+ readonly confidence: number;+ readonly transform: InputTransform;+ readonly sourceWidth: number;+ readonly sourceHeight: number;+ /** Drop people with fewer keypoints at visibility ≥ 0.3. Default 3. */+ readonly minVisibleKeypoints?: number;+ /** Suppress a person whose box overlaps a higher-scored one above this IoU. Default 0.6. */+ readonly iouThreshold?: number;+}++export const COCO17_KEYPOINT_COUNT = 17;++/** Keypoint visibility at which a joint counts as seen. */+const VISIBLE = 0.3;++export function decodeRtmo(+ out: RtmoOutputs,+ opts: RtmoDecodeOptions,+): PoseResult {+ const { sourceWidth: sw, sourceHeight: sh, transform } = opts;+ const stride = out.keypointStride;+ const people: PosePerson[] = [];++ for (let i = 0; i < out.count; i++) {+ const score = out.dets[i * 5 + 4];+ if (!(score >= opts.confidence)) continue;++ const p0 = toSourcePoint(out.dets[i * 5], out.dets[i * 5 + 1], transform);+ const p1 = toSourcePoint(+ out.dets[i * 5 + 2],+ out.dets[i * 5 + 3],+ transform,+ );+ const x0 = clamp(p0.x, 0, sw);+ const y0 = clamp(p0.y, 0, sh);+ const width = clamp(p1.x, 0, sw) - x0;+ const height = clamp(p1.y, 0, sh) - y0;+ if (width <= 0 || height <= 0) continue;++ const keypoints: PoseKeypoint[] = [];+ const base = i * COCO17_KEYPOINT_COUNT * stride;+ for (let k = 0; k < COCO17_KEYPOINT_COUNT; k++) {+ const o = base + k * stride;+ const p = toSourcePoint(+ out.keypoints[o],+ out.keypoints[o + 1],+ transform,+ );+ keypoints.push({+ x: clamp(p.x, 0, sw),+ y: clamp(p.y, 0, sh),+ visibility: clamp(out.keypoints[o + 2], 0, 1),+ });+ }++ let visible = 0;+ for (const k of keypoints) if (k.visibility >= VISIBLE) visible++;+ if (visible < (opts.minVisibleKeypoints ?? 3)) continue;++ people.push({+ score,+ boundingBox: {+ originX: x0,+ originY: y0,+ width,+ height,+ normalizedX: x0 / sw,+ normalizedY: y0 / sh,+ normalizedWidth: width / sw,+ normalizedHeight: height / sh,+ },+ keypoints,+ });+ }++ // Highest-confidence person first — the bundle's named signals track it as primary.+ // The export's built-in NMS lets near-identical low-score duplicates through; drop them.+ people.sort((a, b) => b.score - a.score);+ const iou = opts.iouThreshold ?? 0.6;+ const kept: PosePerson[] = [];+ for (const p of people) {+ if (!kept.some((k) => isDuplicate(k, p, iou))) kept.push(p);+ }+ return { people: kept };+}++function clamp(v: number, lo: number, hi: number): number {+ return v < lo ? lo : v > hi ? hi : v;+}++/**+ * Same person twice: boxes overlap above `iou`, or the joints both detections can see sit+ * within 10% of the stronger detection's box size of each other (boxes may differ when one+ * of them also covers a flowing garment or a shadow).+ */+function isDuplicate(a: PosePerson, b: PosePerson, iou: number): boolean {+ if (computeIoU(a.boundingBox, b.boundingBox) > iou) return true;+ const scale = Math.sqrt(a.boundingBox.width * a.boundingBox.height);+ let sum = 0;+ let n = 0;+ for (let k = 0; k < a.keypoints.length; k++) {+ const ka = a.keypoints[k];+ const kb = b.keypoints[k];+ if (ka.visibility < VISIBLE || kb.visibility < VISIBLE) continue;+ sum += Math.hypot(ka.x - kb.x, ka.y - kb.y);+ n++;+ }+ return n >= 3 && sum / n < 0.1 * scale;+}diff --git a/packages/vision/src/decode/selfie.ts b/packages/vision/src/decode/selfie.tsnew file mode 100644index 00000000..28eeee5b--- /dev/null+++ b/packages/vision/src/decode/selfie.ts@@ -0,0 +1,60 @@+import type { PersonMatte } from "../types.js";+import { probabilityToAlpha } from "./rtmdet-ins.js";++/**+ * Selfie Segmenter decode: `alphas [1, 1, h, w]` person probability (stretched input) →+ * frame-sized 0..255 alpha, bilinear-upsampled to the source resolution.+ */+export interface SelfieDecodeOptions {+ readonly sourceWidth: number;+ readonly sourceHeight: number;+ /** Probability threshold. Default 0.5. */+ readonly maskThreshold?: number;+ /** Soft-edge band; the low-res matte reads best with a wide one. Default 0.2. */+ readonly featherRadius?: number;+}++export function decodeSelfie(+ alphas: ArrayLike<number>,+ matteWidth: number,+ matteHeight: number,+ opts: SelfieDecodeOptions,+): PersonMatte {+ const { sourceWidth: sw, sourceHeight: sh } = opts;+ const alphaOf = probabilityToAlpha(+ opts.maskThreshold ?? 0.5,+ opts.featherRadius ?? 0.2,+ );+ const mask = new Uint8Array(sw * sh);+ const kx = matteWidth / sw;+ const ky = matteHeight / sh;+ let sum = 0;++ for (let y = 0; y < sh; y++) {+ const my = Math.max(0, (y + 0.5) * ky - 0.5);+ const my0 = Math.min(matteHeight - 1, Math.floor(my));+ const my1 = Math.min(matteHeight - 1, my0 + 1);+ const fy = my - my0;+ for (let x = 0; x < sw; x++) {+ const mx = Math.max(0, (x + 0.5) * kx - 0.5);+ const mx0 = Math.min(matteWidth - 1, Math.floor(mx));+ const mx1 = Math.min(matteWidth - 1, mx0 + 1);+ const fx = mx - mx0;+ const p =+ alphas[my0 * matteWidth + mx0] * (1 - fx) * (1 - fy) ++ alphas[my0 * matteWidth + mx1] * fx * (1 - fy) ++ alphas[my1 * matteWidth + mx0] * (1 - fx) * fy ++ alphas[my1 * matteWidth + mx1] * fx * fy;+ const a = alphaOf(p);+ mask[y * sw + x] = a;+ sum += a;+ }+ }++ return {+ mask,+ width: sw,+ height: sh,+ coverage: sw * sh > 0 ? sum / (255 * sw * sh) : 0,+ };+}diff --git a/packages/yolo/src/gpu/pose-skeleton-renderer.ts b/packages/vision/src/gpu/pose-skeleton-renderer.tssimilarity index 64%rename from packages/yolo/src/gpu/pose-skeleton-renderer.tsrename to packages/vision/src/gpu/pose-skeleton-renderer.tsindex e71e0599..2e786c2e 100644--- a/packages/yolo/src/gpu/pose-skeleton-renderer.ts+++ b/packages/vision/src/gpu/pose-skeleton-renderer.ts@@ -1,17 +1,16 @@import type { PoseSkeletonOptions } from "@gitframes/tensor-webgpu";import { PoseSkeletonComputePipeline } from "@gitframes/tensor-webgpu";-import { YOLO_COCO17_BONES } from "../extractors/canonical-indices.js";-import type { YoloPoseKeypoint } from "../types.js";+import { COCO17_BONES } from "../pose/keypoints.js";+import type { PoseKeypoint } from "../types.js";export type { PoseSkeletonOptions };export type { CanonicalBoneDef as SkeletonBone } from "@gitframes/tensor-webgpu";/**- * WebGPU skeleton renderer for YOLO pose results.+ * WebGPU skeleton renderer for pose results.** Rasterizes the primary person's COCO-17 keypoints into an OpenPose-style- * conditioning texture natively in VRAM — same compute pipeline as the MediaPipe- * renderer, with the COCO-17 bone set instead of the 33-point OpenPose set.+ * conditioning texture natively in VRAM, using the COCO-17 bone set.*/export class PoseSkeletonRenderer {private pipeline: PoseSkeletonComputePipeline;@@ -21,9 +20,9 @@ export class PoseSkeletonRenderer {}public renderToTexture(- keypoints: readonly YoloPoseKeypoint[],+ keypoints: readonly PoseKeypoint[],options: PoseSkeletonOptions,- bones = YOLO_COCO17_BONES,+ bones = COCO17_BONES,): GPUTexture {return this.pipeline.execute(keypoints as never, options, bones);}diff --git a/packages/yolo/src/gpu/segmentation-texture-pool.ts b/packages/vision/src/gpu/segmentation-texture-pool.tssimilarity index 97%rename from packages/yolo/src/gpu/segmentation-texture-pool.tsrename to packages/vision/src/gpu/segmentation-texture-pool.tsindex 8e669994..185527b1 100644--- a/packages/yolo/src/gpu/segmentation-texture-pool.ts+++ b/packages/vision/src/gpu/segmentation-texture-pool.ts@@ -32,7 +32,7 @@ export class SegmentationTexturePool {const format = options.format ?? "rgba8unorm";const texture = this.device.createTexture({- label: options.label ?? `Yolo_Segmentation_${key}`,+ label: options.label ?? `vision_segmentation_${key}`,size: [options.width, options.height, 1],format,usage:diff --git a/packages/vision/src/index.ts b/packages/vision/src/index.tsnew file mode 100644index 00000000..9a1734c8--- /dev/null+++ b/packages/vision/src/index.ts@@ -0,0 +1,21 @@+export * from "./analysis/analyze-sequence.js";+export * from "./decode/preprocess.js";+export * from "./decode/rtmdet-ins.js";+export * from "./decode/rtmo.js";+export * from "./decode/selfie.js";+export * from "./gpu/pose-skeleton-renderer.js";+export * from "./gpu/segmentation-texture-pool.js";+export * from "./model/model-store.js";+export * from "./model/registry.js";+export * from "./pose/keypoints.js";+export * from "./runner/canonical-baselines.js";+export * from "./runner/vision-runner.js";+export * from "./runtime/session-provider.js";+export * from "./segmentation/subject.js";+export * from "./signals/vision-bundle.js";+export * from "./spatial/camera-space-transformer.js";+export * from "./spatial/spatial-pin.js";+export * from "./tracking/pose-track-matcher.js";+export * from "./tracking/temporal-object-tracker.js";+export * from "./types.js";+export { type VisionAttachedBundle, VisionNode } from "./vision-node.js";diff --git a/packages/vision/src/model/model-store.test.ts b/packages/vision/src/model/model-store.test.tsnew file mode 100644index 00000000..976ffdc4--- /dev/null+++ b/packages/vision/src/model/model-store.test.ts@@ -0,0 +1,198 @@+import { createHash } from "node:crypto";+import { existsSync, mkdtempSync } from "node:fs";+import fs from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import { VisionModelStore } from "./model-store.js";+import { VISION_MODELS } from "./registry.js";++describe("VisionModelStore (lazy download semantics)", () => {+ let dir: string;+ let fetches: Array<{ url: string }>;++ beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "vision-store-"));+ fetches = [];+ });++ afterEach(() => {+ fetches = [];+ });++ const store = () =>+ new VisionModelStore({+ modelsDir: dir,+ baseUrl: "https://example.test/models",+ verify: false,+ fetchImpl: (async (url: unknown) => {+ fetches.push({ url: String(url) });+ return new Response(new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]));+ }) as typeof fetch,+ timeoutMs: 5000,+ });++ it("constructor performs ZERO I/O (no fetch, no status change)", () => {+ const s = store();+ expect(fetches).toHaveLength(0);+ expect(s.status("rtmdet-ins-s")).toBe("pending");+ });++ it("downloads only when ensure() is called, caches in flight, and persists", async () => {+ const s = store();+ const [a, b] = await Promise.all([+ s.ensure("rtmdet-ins-s"),+ s.ensure("rtmdet-ins-s"),+ ]);+ expect(fetches).toHaveLength(1); // in-flight dedupe+ expect(a).toEqual(b);+ expect(s.status("rtmdet-ins-s")).toBe("ready");+ // second ensure hits the memory cache — no new fetch+ await s.ensure("rtmdet-ins-s");+ expect(fetches).toHaveLength(1);+ // file persisted for future processes+ expect(existsSync(s.pathFor("rtmdet-ins-s"))).toBe(true);+ });++ it("distinct models fetch independently from the mirror", async () => {+ const s = store();+ await s.ensure("rtmo-s");+ expect(fetches).toHaveLength(1);+ expect(fetches[0].url).toBe(+ `https://example.test/models/${VISION_MODELS["rtmo-s"].filename}`,+ );+ expect(s.status("rtmdet-ins-s")).toBe("pending");+ await s.ensure("rtmdet-ins-s");+ expect(fetches).toHaveLength(2);+ });++ it("uses the pinned Hugging Face URL when no mirror is set", () => {+ const s = new VisionModelStore({ modelsDir: dir });+ for (const desc of Object.values(VISION_MODELS)) {+ if (process.env.GITFRAMES_MODELS_BASE_URL) break;+ expect(s.urlFor(desc.key)).toBe(desc.url);+ expect(desc.url).toMatch(/\/resolve\/[0-9a-f]{40}\//);+ }+ });++ it("preload() downloads only the requested keys", async () => {+ const s = store();+ await s.preload(["rtmdet-ins-s", "rtmo-s", "rtmo-s"]);+ expect(fetches).toHaveLength(2);+ expect(s.status("rtmo-s")).toBe("ready");+ expect(s.status("selfie-square")).toBe("pending");+ });++ it("rejects bytes whose size or SHA-256 does not match the registry", async () => {+ const s = new VisionModelStore({+ modelsDir: dir,+ fetchImpl: (async () =>+ new Response(new Uint8Array([1, 2, 3]))) as typeof fetch,+ });+ await expect(s.ensure("selfie-square")).rejects.toThrow(/bytes, expected/);+ expect(s.status("selfie-square")).toBe("error");+ expect(existsSync(s.pathFor("selfie-square"))).toBe(false);+ });++ it("re-downloads a partial cached file instead of trusting it", async () => {+ const desc = VISION_MODELS["selfie-square"];+ await fs.mkdir(dir, { recursive: true });+ await fs.writeFile(join(dir, desc.filename), new Uint8Array(4096));+ const good = new Uint8Array(desc.bytes);+ const s = new VisionModelStore({+ modelsDir: dir,+ fetchImpl: (async () => new Response(good)) as typeof fetch,+ });+ expect(s.has("selfie-square")).toBe(false);+ // the fake bytes are the right size but the wrong hash → loud failure+ const digest = createHash("sha256").update(good).digest("hex");+ expect(digest).not.toBe(desc.sha256);+ await expect(s.ensure("selfie-square")).rejects.toThrow(/SHA-256/);+ });++ it("evict() drops the file and resets to pending so the next ensure re-downloads", async () => {+ const s = store();+ await s.ensure("rtmdet-ins-s");+ expect(existsSync(s.pathFor("rtmdet-ins-s"))).toBe(true);+ await s.evict("rtmdet-ins-s");+ expect(existsSync(s.pathFor("rtmdet-ins-s"))).toBe(false);+ expect(s.has("rtmdet-ins-s")).toBe(false);+ expect(s.status("rtmdet-ins-s")).toBe("pending");+ await s.ensure("rtmdet-ins-s");+ expect(fetches).toHaveLength(2); // re-downloaded, not served from memory+ });++ it("surfaces an error status and can retry after failure", async () => {+ let calls = 0;+ const s = new VisionModelStore({+ modelsDir: dir,+ baseUrl: "https://example.test/models",+ verify: false,+ fetchImpl: (async () => {+ calls += 1;+ if (calls === 1) throw new Error("boom");+ return new Response(new Uint8Array([9, 9, 9]));+ }) as typeof fetch,+ });+ await expect(s.ensure("rtmdet-ins-s")).rejects.toThrow("boom");+ expect(s.status("rtmdet-ins-s")).toBe("error");+ expect(s.has("rtmdet-ins-s")).toBe(false);+ const bytes = await s.ensure("rtmdet-ins-s");+ expect([...bytes]).toEqual([9, 9, 9]);+ expect(s.status("rtmdet-ins-s")).toBe("ready");+ });++ it("retries transient network and 5xx failures, but not 4xx", async () => {+ let calls = 0;+ const flaky = new VisionModelStore({+ modelsDir: dir,+ verify: false,+ retryDelayMs: 1,+ fetchImpl: (async () => {+ calls += 1;+ if (calls === 1) throw new TypeError("fetch failed");+ if (calls === 2) return new Response("busy", { status: 503 });+ return new Response(new Uint8Array([7]));+ }) as typeof fetch,+ });+ expect([...(await flaky.ensure("rtmo-s"))]).toEqual([7]);+ expect(calls).toBe(3);++ calls = 0;+ const missing = new VisionModelStore({+ modelsDir: dir,+ verify: false,+ retryDelayMs: 1,+ fetchImpl: (async () => {+ calls += 1;+ return new Response("nope", { status: 404 });+ }) as typeof fetch,+ });+ await expect(missing.ensure("rtmo-t")).rejects.toThrow(/HTTP 404/);+ expect(calls).toBe(1);+ });++ it("gives up after the configured retries with the cause attached", async () => {+ const s = new VisionModelStore({+ modelsDir: dir,+ verify: false,+ retries: 1,+ retryDelayMs: 1,+ fetchImpl: (async () => {+ throw new TypeError("fetch failed");+ }) as typeof fetch,+ });+ await expect(s.ensure("rtmo-m")).rejects.toThrow(/after 2 attempts/);+ });++ it("treats a zero-length download as an error", async () => {+ const s = new VisionModelStore({+ modelsDir: dir,+ baseUrl: "https://example.test/models",+ verify: false,+ fetchImpl: (async () => new Response(new Uint8Array(0))) as typeof fetch,+ });+ await expect(s.ensure("rtmdet-ins-s")).rejects.toThrow(/empty/i);+ expect(s.status("rtmdet-ins-s")).toBe("error");+ });+});diff --git a/packages/vision/src/model/model-store.ts b/packages/vision/src/model/model-store.tsnew file mode 100644index 00000000..6b544072--- /dev/null+++ b/packages/vision/src/model/model-store.ts@@ -0,0 +1,289 @@+import { createHash } from "node:crypto";+import { existsSync, mkdirSync, readFileSync, statSync } from "node:fs";+import { rename, unlink, writeFile } from "node:fs/promises";+import { homedir } from "node:os";+import { dirname, join, resolve } from "node:path";+import {+ VISION_MODELS,+ type VisionModelDescriptor,+ type VisionModelKey,+} from "./registry.js";++export type ModelDownloadStatus = "pending" | "downloading" | "ready" | "error";++export interface VisionModelStoreOptions {+ readonly modelsDir?: string;+ /**+ * Mirror origin: models are fetched from `<baseUrl>/<filename>` instead of their pinned+ * Hugging Face URLs. Defaults to `$GITFRAMES_MODELS_BASE_URL` when set.+ */+ readonly baseUrl?: string;+ readonly timeoutMs?: number;+ /** Extra attempts after a network error, HTTP 429 or 5xx. Default 2. */+ readonly retries?: number;+ /** Base backoff between attempts (doubles each retry). Default 1000 ms. */+ readonly retryDelayMs?: number;+ /**+ * Verify byte size + SHA-256 against the registry (default true). Only offline tests+ * that serve fake bytes should turn this off.+ */+ readonly verify?: boolean;+ /** Injectable fetch — offline unit tests inject a mock. */+ readonly fetchImpl?: typeof fetch;+ readonly onProgress?: (+ key: VisionModelKey,+ bytesLoaded: number,+ totalBytes: number,+ ) => void;+}++/**+ * Resolves the default model cache directory:+ * 1. `GITFRAMES_MODELS_DIR`+ * 2. `~/.cache/gitframes/models`+ */+export function getDefaultModelsDir(): string {+ if (process.env.GITFRAMES_MODELS_DIR) {+ return resolve(process.env.GITFRAMES_MODELS_DIR);+ }+ return resolve(homedir(), ".cache/gitframes/models");+}++/**+ * Lazy model store. The constructor performs ZERO I/O; the only entry that can hit the+ * network is `ensure(key)`, called by an inference function the first time it runs.+ * Concurrent callers of the same key share a single in-flight download (promise dedupe),+ * downloads are written atomically (temp + rename), and verified files on disk are reused+ * across processes.+ */+export class VisionModelStore {+ private readonly _modelsDir: string;+ private readonly _baseUrl?: string;+ private readonly _timeoutMs?: number;+ private readonly _retries: number;+ private readonly _retryDelayMs: number;+ private readonly _verify: boolean;+ private readonly _fetch: typeof fetch;+ private readonly _onProgress?: VisionModelStoreOptions["onProgress"];++ private _inflight = new Map<VisionModelKey, Promise<Uint8Array>>();+ private _buffers = new Map<VisionModelKey, Uint8Array>();+ private _status = new Map<VisionModelKey, ModelDownloadStatus>();++ constructor(options: VisionModelStoreOptions = {}) {+ this._modelsDir = options.modelsDir+ ? resolve(options.modelsDir)+ : getDefaultModelsDir();+ this._baseUrl =+ options.baseUrl ?? process.env.GITFRAMES_MODELS_BASE_URL ?? undefined;+ this._timeoutMs = options.timeoutMs;+ this._retries = Math.max(0, options.retries ?? 2);+ this._retryDelayMs = options.retryDelayMs ?? 1000;+ this._verify = options.verify !== false;+ this._fetch = options.fetchImpl ?? globalThis.fetch;+ this._onProgress = options.onProgress;+ }++ public get modelsDir(): string {+ return this._modelsDir;+ }++ public pathFor(key: VisionModelKey): string {+ return join(this._modelsDir, VISION_MODELS[key].filename);+ }++ public descriptor(key: VisionModelKey): VisionModelDescriptor {+ return VISION_MODELS[key];+ }++ /** The URL `ensure(key)` downloads from (mirror when `baseUrl` is set). */+ public urlFor(key: VisionModelKey): string {+ const desc = VISION_MODELS[key];+ if (!this._baseUrl) return desc.url;+ return `${this._baseUrl.replace(/\/+$/, "")}/${desc.filename}`;+ }++ /** True when a complete cached file exists (exact registry size when verifying). */+ public has(key: VisionModelKey): boolean {+ const targetPath = this.pathFor(key);+ if (!existsSync(targetPath)) {+ return false;+ }+ try {+ const size = statSync(targetPath).size;+ return this._verify ? size === VISION_MODELS[key].bytes : size > 0;+ } catch {+ return false;+ }+ }++ /** Drops a cached model (file + memory) so the next `ensure` re-downloads it. */+ public async evict(key: VisionModelKey): Promise<void> {+ this._buffers.delete(key);+ this._status.set(key, "pending");+ await unlink(this.pathFor(key)).catch(() => undefined);+ }++ public status(key: VisionModelKey): ModelDownloadStatus {+ return this._status.get(key) ?? "pending";+ }++ /** Snapshot of every tracked model's status (empty until anything is fetched). */+ public statuses(): ReadonlyMap<VisionModelKey, ModelDownloadStatus> {+ return new Map(this._status);+ }++ /** True when the model is resident in memory (loaded or downloaded this process). */+ public isLoaded(key: VisionModelKey): boolean {+ return this._buffers.has(key);+ }++ /**+ * THE only download entry point. Returns the model bytes from memory (cached),+ * loading from disk or fetching from the registry URL as needed.+ */+ public async ensure(key: VisionModelKey): Promise<Uint8Array> {+ const inMemory = this._buffers.get(key);+ if (inMemory) {+ return inMemory;+ }+ const inflight = this._inflight.get(key);+ if (inflight) {+ return inflight;+ }+ const promise = this.load(key).finally(() => this._inflight.delete(key));+ this._inflight.set(key, promise);+ return promise;+ }++ /** Explicit warm-up — fetch multiple models ahead of use (agents, renderers). */+ public async preload(keys: readonly VisionModelKey[]): Promise<void> {+ await Promise.all([...new Set(keys)].map((key) => this.ensure(key)));+ }++ private async load(key: VisionModelKey): Promise<Uint8Array> {+ const targetPath = this.pathFor(key);++ if (this.has(key)) {+ this._status.set(key, "ready");+ return this.readBuffer(key, targetPath);+ }++ const downloadUrl = this.urlFor(key);+ this._status.set(key, "downloading");++ if (!existsSync(dirname(targetPath))) {+ mkdirSync(dirname(targetPath), { recursive: true });+ }++ const tempPath = `${targetPath}.tmp.${Date.now()}`;+ try {+ const { buffer, totalBytes } = await this.download(key, downloadUrl);++ if (buffer.byteLength === 0) {+ throw new Error(`Downloaded vision model '${key}' is empty (0 bytes).`);+ }+ if (this._verify) {+ this.verifyBytes(key, buffer, downloadUrl);+ }++ this._onProgress?.(+ key,+ buffer.byteLength,+ totalBytes || buffer.byteLength,+ );++ // Write-then-rename keeps the cache atomic: a crash never leaves a partial model+ // at the final path.+ await writeFile(tempPath, buffer);+ await rename(tempPath, targetPath);++ this._buffers.set(key, buffer);+ this._status.set(key, "ready");+ return buffer;+ } catch (error) {+ this._status.set(key, "error");+ if (existsSync(tempPath)) {+ await unlink(tempPath).catch(() => undefined);+ }+ throw error;+ }+ }++ /** Fetches with retries on transient failures (network errors, HTTP 429/5xx). */+ private async download(+ key: VisionModelKey,+ url: string,+ ): Promise<{ buffer: Uint8Array; totalBytes: number }> {+ for (let attempt = 0; ; attempt++) {+ let transient: unknown;+ try {+ const response = await this._fetch(url, {+ signal: this._timeoutMs+ ? AbortSignal.timeout(this._timeoutMs)+ : undefined,+ });+ if (response.ok) {+ const totalBytes = Number(+ response.headers.get("content-length") ?? 0,+ );+ return {+ buffer: new Uint8Array(await response.arrayBuffer()),+ totalBytes,+ };+ }+ const error = new Error(+ `Failed to download vision model '${key}' from ${url}: HTTP ${response.status} ${response.statusText}`,+ );+ if (response.status !== 429 && response.status < 500) throw error;+ transient = error;+ } catch (error) {+ // fetch() rejects with a TypeError on network failures (reset, DNS, TLS).+ if (!(error instanceof TypeError)) throw error;+ transient = error;+ }+ if (attempt >= this._retries) {+ throw new Error(+ `Failed to download vision model '${key}' from ${url} after ${attempt + 1} attempts: ${String(transient)}`,+ { cause: transient },+ );+ }+ await new Promise((r) => setTimeout(r, this._retryDelayMs * 2 ** attempt));+ }+ }++ private verifyBytes(+ key: VisionModelKey,+ buffer: Uint8Array,+ url: string,+ ): void {+ const desc = VISION_MODELS[key];+ if (buffer.byteLength !== desc.bytes) {+ throw new Error(+ `Vision model '${key}' from ${url} is ${buffer.byteLength} bytes, expected ${desc.bytes}.`,+ );+ }+ const digest = createHash("sha256").update(buffer).digest("hex");+ if (digest !== desc.sha256) {+ throw new Error(+ `Vision model '${key}' from ${url} failed its SHA-256 check (got ${digest}, expected ${desc.sha256}).`,+ );+ }+ }++ private readBuffer(key: VisionModelKey, targetPath: string): Uint8Array {+ const fileBuffer = readFileSync(targetPath);+ const buffer = new Uint8Array(+ fileBuffer.buffer,+ fileBuffer.byteOffset,+ fileBuffer.byteLength,+ );+ this._buffers.set(key, buffer);+ return buffer;+ }++ /** Drops in-memory buffers (model stays on disk for the next run). */+ public clearMemoryCache(): void {+ this._buffers.clear();+ }+}diff --git a/packages/vision/src/model/registry.ts b/packages/vision/src/model/registry.tsnew file mode 100644index 00000000..018e63cb--- /dev/null+++ b/packages/vision/src/model/registry.ts@@ -0,0 +1,286 @@+import type { VisionTask, VisionVariant } from "@gitframes/core";++/**+ * Vision model registry — single source of truth for what exists, where it lives, and which+ * model serves which task + variant.+ *+ * Every model is Apache-2.0 and pinned to an immutable Hugging Face revision; the store+ * verifies the SHA-256 below after download, so a moved or tampered file fails loudly.+ *+ * Downloads are ALWAYS lazy: nothing here touches the network. The store (model-store.ts)+ * fetches a model only when an inference function that needs it is first called.+ */++export type { VisionTask, VisionVariant };++export const VISION_TASKS: readonly VisionTask[] = [+ "detect",+ "segment",+ "pose",+ "matte",+];++export const VISION_VARIANTS: readonly VisionVariant[] = ["t", "s", "m"];++export type VisionModelKey =+ | "rtmdet-ins-t"+ | "rtmdet-ins-s"+ | "rtmdet-ins-m"+ | "rtmo-t"+ | "rtmo-s"+ | "rtmo-m"+ | "selfie-square"+ | "selfie-landscape";++/** Which decoder a model's outputs feed (see runner/vision-runner.ts). */+export type VisionModelFamily = "rtmdet-ins" | "rtmo" | "selfie";++export interface VisionModelDescriptor {+ readonly key: VisionModelKey;+ readonly family: VisionModelFamily;+ /** Cache filename (also the path appended to a custom `baseUrl`). */+ readonly filename: string;+ /** Pinned download URL (immutable revision). */+ readonly url: string;+ /** Exact byte size of the pinned file. */+ readonly bytes: number;+ /** Hex SHA-256 of the pinned file. */+ readonly sha256: string;+ /** Model input size, `[width, height]` — must match the ONNX graph's input shape. */+ readonly input: readonly [number, number];+ /** Upstream project and license, surfaced in docs and errors. */+ readonly source: string;+}++const hf = (repo: string, revision: string, path: string): string =>+ `https://huggingface.co/${repo}/resolve/${revision}/${path}`;++const RTMDET_INS_REPO = "tori29umai/rtmdet-ins-onnx-person-masks";+const RTMDET_INS_REV = "e5218e8d47d7c50077f27f9893a28e0ef2ad5656";+const RTMDET_INS_SOURCE =+ "RTMDet-Ins (OpenMMLab mmdetection, Apache-2.0), mmdeploy ONNX export";+const RTMO_SOURCE = "RTMO (OpenMMLab mmpose, Apache-2.0), ONNX export";+const SELFIE_SOURCE =+ "MediaPipe Selfie Segmenter (Google, Apache-2.0), ONNX export";++export const VISION_MODELS: Readonly<+ Record<VisionModelKey, VisionModelDescriptor>+> = {+ // ── Detection + instance segmentation (COCO 80) ─────────────────────+ "rtmdet-ins-t": {+ key: "rtmdet-ins-t",+ family: "rtmdet-ins",+ filename: "rtmdet-ins-t.onnx",+ url: hf(+ RTMDET_INS_REPO,+ RTMDET_INS_REV,+ "models/tiny/rtmdet-ins_tiny_640x640.onnx",+ ),+ bytes: 24_032_356,+ sha256: "99ee670820c4493aa13e20578b6a8e39781c319846dc41796a77765734beac5c",+ input: [640, 640],+ source: RTMDET_INS_SOURCE,+ },+ "rtmdet-ins-s": {+ key: "rtmdet-ins-s",+ family: "rtmdet-ins",+ filename: "rtmdet-ins-s.onnx",+ url: hf(+ RTMDET_INS_REPO,+ RTMDET_INS_REV,+ "models/s/rtmdet-ins_s_640x640.onnx",+ ),+ bytes: 43_236_976,+ sha256: "9cd1787fbf3eb2bd64cc2d8154f41c036f5484b85599e1cbe0227a2dc5a4a08b",+ input: [640, 640],+ source: RTMDET_INS_SOURCE,+ },+ "rtmdet-ins-m": {+ key: "rtmdet-ins-m",+ family: "rtmdet-ins",+ filename: "rtmdet-ins-m.onnx",+ url: hf(+ RTMDET_INS_REPO,+ RTMDET_INS_REV,+ "models/m/rtmdet-ins_m_640x640.onnx",+ ),+ bytes: 115_735_979,+ sha256: "a003bce80316e03467c2c2b483df2d5c0398220c2357a44f9bf985929f7eb052",+ input: [640, 640],+ source: RTMDET_INS_SOURCE,+ },+ // ── Multi-person pose (COCO-17) ─────────────────────────────────────+ "rtmo-t": {+ key: "rtmo-t",+ family: "rtmo",+ filename: "rtmo-t.onnx",+ url: hf(+ "Xenova/RTMO-t",+ "f29d13ce5ee291fbe505e9d06a049992f05699dc",+ "onnx/model.onnx",+ ),+ bytes: 27_360_569,+ sha256: "cb54dd042bf997df86d938721c94238940f2bff98ba8d2ccc834430de5c65c13",+ input: [416, 416],+ source: RTMO_SOURCE,+ },+ "rtmo-s": {+ key: "rtmo-s",+ family: "rtmo",+ filename: "rtmo-s.onnx",+ url: hf(+ "Xenova/RTMO-s",+ "d8c526187f341d287753831c9c8b1ecc4855bba1",+ "onnx/model.onnx",+ ),+ bytes: 39_636_400,+ sha256: "1cd6a3517658903233e9ccb78666554862f57f0944b6886e2048093594aef4e7",+ input: [640, 640],+ source: RTMO_SOURCE,+ },+ "rtmo-m": {+ key: "rtmo-m",+ family: "rtmo",+ filename: "rtmo-m.onnx",+ url: hf(+ "Xenova/RTMO-m",+ "3aba1280472b98ee2bf663482e27e243038d23fe",+ "onnx/model.onnx",+ ),+ bytes: 89_291_929,+ sha256: "76d82c45e5c4810baf587ecf2638d15cbf7ed196afb2055e977382532021b59e",+ input: [640, 640],+ source: RTMO_SOURCE,+ },+ // ── Person matte ────────────────────────────────────────────────────+ "selfie-square": {+ key: "selfie-square",+ family: "selfie",+ filename: "selfie-square.onnx",+ url: hf(+ "onnx-community/mediapipe_selfie_segmentation",+ "be49485c8e027524be38591817fc5cd31bd9d00e",+ "onnx/model.onnx",+ ),+ bytes: 462_352,+ sha256: "3241ac4ad8aa35bdaf33946776db29f7c283a413aa0b0dacb9483594b4531aad",+ input: [256, 256],+ source: SELFIE_SOURCE,+ },+ "selfie-landscape": {+ key: "selfie-landscape",+ family: "selfie",+ filename: "selfie-landscape.onnx",+ url: hf(+ "onnx-community/mediapipe_selfie_segmentation_landscape",+ "2497d5bec26c626c7b3c4edc6e1fefc21b64f6c3",+ "onnx/model.onnx",+ ),+ bytes: 462_338,+ sha256: "7a0adcfdb1715d3b0ff0f61486d8a181c33f4a208343a5abfccbc6540e872d24",+ input: [256, 144],+ source: SELFIE_SOURCE,+ },+};++/**+ * Resolves the model for a task. `detect` and `segment` share RTMDet-Ins (one forward pass+ * yields both); `matte` picks the landscape Selfie Segmenter for wide frames.+ */+export function modelKeyFor(+ task: VisionTask,+ variant: VisionVariant = "s",+ aspect = 1,+): VisionModelKey {+ if (task === "pose") return `rtmo-${variant}` as const;+ if (task === "matte") {+ return aspect > 1.3 ? "selfie-landscape" : "selfie-square";+ }+ return `rtmdet-ins-${variant}` as const;+}++/** COCO 80 class names — index order matches the RTMDet-Ins `labels` output. */+export const COCO_CLASSES = [+ "person",+ "bicycle",+ "car",+ "motorcycle",+ "airplane",+ "bus",+ "train",+ "truck",+ "boat",+ "traffic light",+ "fire hydrant",+ "stop sign",+ "parking meter",+ "bench",+ "bird",+ "cat",+ "dog",+ "horse",+ "sheep",+ "cow",+ "elephant",+ "bear",+ "zebra",+ "giraffe",+ "backpack",+ "umbrella",+ "handbag",+ "tie",+ "suitcase",+ "frisbee",+ "skis",+ "snowboard",+ "sports ball",+ "kite",+ "baseball bat",+ "baseball glove",+ "skateboard",+ "surfboard",+ "tennis racket",+ "bottle",+ "wine glass",+ "cup",+ "fork",+ "knife",+ "spoon",+ "bowl",+ "banana",+ "apple",+ "sandwich",+ "orange",+ "broccoli",+ "carrot",+ "hot dog",+ "pizza",+ "donut",+ "cake",+ "chair",+ "couch",+ "potted plant",+ "bed",+ "dining table",+ "toilet",+ "tv",+ "laptop",+ "mouse",+ "remote",+ "keyboard",+ "cell phone",+ "microwave",+ "oven",+ "toaster",+ "sink",+ "refrigerator",+ "book",+ "clock",+ "vase",+ "scissors",+ "teddy bear",+ "hair drier",+ "toothbrush",+] as const;++export type CocoClass = (typeof COCO_CLASSES)[number];diff --git a/packages/vision/src/models.e2e.test.ts b/packages/vision/src/models.e2e.test.tsnew file mode 100644index 00000000..65c3e3b2--- /dev/null+++ b/packages/vision/src/models.e2e.test.ts@@ -0,0 +1,129 @@+/**+ * Real-model end-to-end checks — opt-in, because they download ~380 MB of weights:+ *+ * pnpm --filter @gitframes/vision test:models+ *+ * Verifies every pinned model downloads, passes its size + SHA-256 check and matches the+ * registry's input shape, then runs each size on reference frames from `examples/` and checks+ * behaviour that regressed during development (subject selection, merging, pose duplicates).+ * Models are cached in `$GITFRAMES_MODELS_DIR` (default `~/.cache/gitframes/models`).+ */++import { fileURLToPath } from "node:url";+import { Canvas, loadImage } from "skia-canvas";+import { beforeAll, describe, expect, it } from "vitest";+import { VisionModelStore } from "./model/model-store.js";+import {+ VISION_MODELS,+ VISION_VARIANTS,+ type VisionModelKey,+} from "./model/registry.js";+import { VisionRunner } from "./runner/vision-runner.js";+import { mergeSubjectMask } from "./segmentation/subject.js";+import type { VisionImageInput } from "./types.js";++const enabled = process.env.GITFRAMES_VISION_E2E === "1";+const EXAMPLES = fileURLToPath(new URL("../../../examples/", import.meta.url));+const ASSETS = {+ dancer: "19_gitframes_film/assets/dancer.png",+ portrait: "19_gitframes_film/assets/portrait.png",+ ink: "19_gitframes_film/assets/ink.png",+ crema: "21_full_circle/assets/crema.png",+ eclipse: "21_full_circle/assets/eclipse.png",+ spotlight: "21_full_circle/assets/spotlight.png",+ tunnel: "21_full_circle/assets/tunnel.png",+} as const;++async function loadFrame(path: string): Promise<VisionImageInput> {+ const image = await loadImage(EXAMPLES + path);+ const canvas = new Canvas(image.width, image.height);+ const ctx = canvas.getContext("2d");+ ctx.drawImage(image, 0, 0);+ return {+ data: ctx.getImageData(0, 0, image.width, image.height).data,+ width: image.width,+ height: image.height,+ };+}++describe.skipIf(!enabled)("vision models (real weights)", () => {+ const frames = {} as Record<keyof typeof ASSETS, VisionImageInput>;++ beforeAll(async () => {+ for (const [name, path] of Object.entries(ASSETS)) {+ frames[name as keyof typeof ASSETS] = await loadFrame(path);+ }+ });++ it.each(+ Object.keys(VISION_MODELS) as VisionModelKey[],+ )("%s downloads, verifies and matches its registry input shape", async (key) => {+ const bytes = await new VisionModelStore().ensure(key);+ const ort = await import("onnxruntime-node");+ const session = await ort.InferenceSession.create(bytes);+ const meta = session.inputMetadata[0];+ const shape = meta?.isTensor ? meta.shape : [];+ const [w, h] = VISION_MODELS[key].input;+ expect(shape.slice(2)).toEqual([h, w]);+ await session.release();+ }, 600_000);++ describe.each(VISION_VARIANTS)("variant %s", (variant) => {+ const runner = VisionRunner.create({ variant });++ it("cuts out the whole dancer, dress included", async () => {+ const seg = await runner.segment(frames.dancer);+ const subject = mergeSubjectMask(seg.masks);+ expect(subject?.category).toBe("person");+ // the bare person mask is ~3% of the frame; with the dress merged it is ~12.5%+ expect(subject?.coverage).toBeGreaterThan(0.08);+ expect(subject?.coverage).toBeLessThan(0.2);+ }, 120_000);++ it("finds one dancer with a full skeleton", async () => {+ const { people } = await runner.pose(frames.dancer);+ expect(people).toHaveLength(1);+ const visible = people[0].keypoints.filter((k) => k.visibility > 0.5);+ expect(visible.length).toBeGreaterThanOrEqual(12);+ const nose = people[0].keypoints[0];+ expect(nose.x).toBeGreaterThan(1000);+ expect(nose.x).toBeLessThan(1120);+ expect(nose.y).toBeGreaterThan(250);+ expect(nose.y).toBeLessThan(360);+ }, 120_000);++ it("reports the spotlight dancer once (no duplicate skeletons)", async () => {+ const { people } = await runner.pose(frames.spotlight);+ expect(people).toHaveLength(1);+ }, 120_000);++ it("picks the cup, not the table, as a non-person subject", async () => {+ const seg = await runner.segment(frames.crema);+ expect(mergeSubjectMask(seg.masks)?.category).toBe("cup");+ }, 120_000);++ it("keeps a tiny figure separate from the tunnel around it", async () => {+ const seg = await runner.segment(frames.tunnel);+ const subject = mergeSubjectMask(seg.masks);+ expect(subject?.category).toBe("person");+ expect(subject?.coverage).toBeLessThan(0.02);+ }, 120_000);++ it("sees no confident person in abstract frames", async () => {+ for (const frame of [frames.ink, frames.eclipse]) {+ const detections = await runner.detect(frame);+ const people = detections.filter(+ (d) => d.category === "person" && d.score >= 0.5,+ );+ expect(people).toHaveLength(0);+ }+ }, 120_000);++ it("mattes a portrait with the Selfie Segmenter", async () => {+ const matte = await runner.matte(frames.portrait);+ expect(matte.width).toBe(frames.portrait.width);+ expect(matte.coverage).toBeGreaterThan(0.3);+ expect(matte.coverage).toBeLessThan(0.6);+ }, 120_000);+ });+});diff --git a/packages/vision/src/pose/keypoints.ts b/packages/vision/src/pose/keypoints.tsnew file mode 100644index 00000000..5db25abf--- /dev/null+++ b/packages/vision/src/pose/keypoints.ts@@ -0,0 +1,114 @@+/** COCO-17 keypoint indices (RTMO output order). */+export const COCO17_KEYPOINTS = {+ NOSE: 0,+ LEFT_EYE: 1,+ RIGHT_EYE: 2,+ LEFT_EAR: 3,+ RIGHT_EAR: 4,+ LEFT_SHOULDER: 5,+ RIGHT_SHOULDER: 6,+ LEFT_ELBOW: 7,+ RIGHT_ELBOW: 8,+ LEFT_WRIST: 9,+ RIGHT_WRIST: 10,+ LEFT_HIP: 11,+ RIGHT_HIP: 12,+ LEFT_KNEE: 13,+ RIGHT_KNEE: 14,+ LEFT_ANKLE: 15,+ RIGHT_ANKLE: 16,+} as const;++export const COCO17_KEYPOINT_NAMES = [+ "nose",+ "leftEye",+ "rightEye",+ "leftEar",+ "rightEar",+ "leftShoulder",+ "rightShoulder",+ "leftElbow",+ "rightElbow",+ "leftWrist",+ "rightWrist",+ "leftHip",+ "rightHip",+ "leftKnee",+ "rightKnee",+ "leftAnkle",+ "rightAnkle",+] as const;++/** COCO-17 skeleton bones for the WebGPU skeleton renderer (indices above). */+export const COCO17_BONES = [+ {+ from: COCO17_KEYPOINTS.NOSE,+ to: COCO17_KEYPOINTS.LEFT_SHOULDER,+ color: [0.0, 1.0, 1.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.NOSE,+ to: COCO17_KEYPOINTS.RIGHT_SHOULDER,+ color: [0.0, 1.0, 1.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_SHOULDER,+ to: COCO17_KEYPOINTS.RIGHT_SHOULDER,+ color: [1.0, 0.0, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_SHOULDER,+ to: COCO17_KEYPOINTS.LEFT_ELBOW,+ color: [1.0, 0.333, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_ELBOW,+ to: COCO17_KEYPOINTS.LEFT_WRIST,+ color: [1.0, 0.667, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.RIGHT_SHOULDER,+ to: COCO17_KEYPOINTS.RIGHT_ELBOW,+ color: [1.0, 1.0, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.RIGHT_ELBOW,+ to: COCO17_KEYPOINTS.RIGHT_WRIST,+ color: [0.667, 1.0, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_SHOULDER,+ to: COCO17_KEYPOINTS.LEFT_HIP,+ color: [0.333, 1.0, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.RIGHT_SHOULDER,+ to: COCO17_KEYPOINTS.RIGHT_HIP,+ color: [0.0, 1.0, 0.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_HIP,+ to: COCO17_KEYPOINTS.RIGHT_HIP,+ color: [0.0, 1.0, 0.333, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_HIP,+ to: COCO17_KEYPOINTS.LEFT_KNEE,+ color: [0.0, 1.0, 0.667, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.LEFT_KNEE,+ to: COCO17_KEYPOINTS.LEFT_ANKLE,+ color: [0.0, 1.0, 1.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.RIGHT_HIP,+ to: COCO17_KEYPOINTS.RIGHT_KNEE,+ color: [0.0, 0.667, 1.0, 1.0] as const,+ },+ {+ from: COCO17_KEYPOINTS.RIGHT_KNEE,+ to: COCO17_KEYPOINTS.RIGHT_ANKLE,+ color: [0.0, 0.333, 1.0, 1.0] as const,+ },+] as const;diff --git a/packages/yolo/src/runner/canonical-baselines.ts b/packages/vision/src/runner/canonical-baselines.tssimilarity index 86%rename from packages/yolo/src/runner/canonical-baselines.tsrename to packages/vision/src/runner/canonical-baselines.tsindex f17ba763..108ac266 100644--- a/packages/yolo/src/runner/canonical-baselines.ts+++ b/packages/vision/src/runner/canonical-baselines.ts@@ -3,7 +3,7 @@ import type {Landmark3D,TrackedObject,} from "@gitframes/core";-import type { YoloPoseResult } from "../types.js";+import type { PoseResult } from "../types.js";/** Neutral fallbacks so signal evaluators always have a well-formed frame to read. */@@ -14,7 +14,7 @@ export function createNeutralObjectResult(): {return { objects: [], rawDetections: [] };}-export function createNeutralPoseResult(): YoloPoseResult {+export function createNeutralPoseResult(): PoseResult {return { people: [] };}diff --git a/packages/vision/src/runner/vision-runner.test.ts b/packages/vision/src/runner/vision-runner.test.tsnew file mode 100644index 00000000..798655ca--- /dev/null+++ b/packages/vision/src/runner/vision-runner.test.ts@@ -0,0 +1,192 @@+import { describe, expect, it } from "vitest";+import { PREPROCESS_BY_FAMILY } from "../decode/preprocess.js";+import { VISION_MODELS } from "../model/registry.js";+import {+ fakeStore,+ grayImage,+ rtmdetInsOutput,+ rtmoOutput,+ ScriptedProvider,+ selfieOutput,+} from "../test-support.js";+import { VisionRunner } from "./vision-runner.js";++describe("VisionRunner (lazy + decode integration)", () => {+ it("create() performs zero I/O; the first detect() downloads + runs RTMDet-Ins", async () => {+ const { store, fetches } = fakeStore();+ const provider = new ScriptedProvider([rtmdetInsOutput()]);+ const runner = VisionRunner.create({ store, provider });++ expect(fetches).toHaveLength(0);+ expect(provider.created).toHaveLength(0);+ expect(runner.downloadStatus.get("rtmdet-ins-s")).toBe("pending");++ const detections = await runner.detect(grayImage());+ expect(fetches).toEqual([+ `https://example.test/models/${VISION_MODELS["rtmdet-ins-s"].filename}`,+ ]);+ expect(runner.downloadStatus.get("rtmdet-ins-s")).toBe("ready");+ expect(runner.downloadStatus.get("rtmo-s")).toBe("pending");+ expect(detections).toHaveLength(1);+ expect(detections[0].category).toBe("person");+ expect(detections[0].score).toBeCloseTo(0.9);+ // 640×640 source, top-left letterbox at scale 1 → input box == source box+ expect(detections[0].boundingBox.originX).toBeCloseTo(220);+ expect(detections[0].boundingBox.width).toBeCloseTo(200);+ });++ it("detect() and segment() on the same frame share one forward pass", async () => {+ const { store } = fakeStore();+ const provider = new ScriptedProvider([rtmdetInsOutput()]);+ const runner = VisionRunner.create({ store, provider });+ const frame = grayImage();++ await runner.detect(frame);+ const seg = await runner.segment(frame);+ expect(provider.created).toHaveLength(1);+ expect(provider.created[0].inputs).toHaveLength(1);++ expect(seg.masks).toHaveLength(1);+ const m = seg.masks[0];+ expect(m.width).toBe(640);+ expect(m.mask[300 * 640 + 300]).toBe(255); // inside the blob+ expect(m.mask[180 * 640 + 230]).toBe(0); // inside the box, outside the blob+ expect(m.coverage).toBeGreaterThan(0);++ // A new frame buffer runs the model again.+ await runner.segment(grayImage());+ expect(provider.created[0].inputs).toHaveLength(2);+ });++ it("maps boxes and masks back through a top-left letterbox on wide frames", async () => {+ const { store } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new ScriptedProvider([rtmdetInsOutput()]),+ });+ // 1280×720 → scale 0.5, content at input (0,0)–(640,360), no offset+ const seg = await runner.segment(grayImage(1280, 720));+ const box = seg.detections[0].boundingBox;+ expect(box.originX).toBeCloseTo(440);+ expect(box.originY).toBeCloseTo(340);+ expect(seg.masks[0].width).toBe(1280);+ expect(seg.masks[0].mask[600 * 1280 + 600]).toBe(255);+ });++ it("applies confidence and class filters", async () => {+ const { store } = fakeStore();+ const low = VisionRunner.create({+ store,+ confidence: 0.95,+ provider: new ScriptedProvider([rtmdetInsOutput()]),+ });+ expect(await low.detect(grayImage())).toHaveLength(0);++ const cars = VisionRunner.create({+ store,+ classes: ["car"],+ provider: new ScriptedProvider([rtmdetInsOutput()]),+ });+ expect(await cars.detect(grayImage())).toHaveLength(0);++ const horse = VisionRunner.create({+ store,+ classes: ["Horse"],+ provider: new ScriptedProvider([rtmdetInsOutput({ label: 17 })]),+ });+ expect((await horse.detect(grayImage()))[0].category).toBe("horse");+ });++ it("preprocesses RTMDet-Ins input as BGR, ImageNet-normalized, padded with 114", async () => {+ const { store } = fakeStore();+ const provider = new ScriptedProvider([rtmdetInsOutput()]);+ const runner = VisionRunner.create({ store, provider });+ const image = grayImage(640, 320);+ image.data.fill(0);+ for (let i = 0; i < image.data.length; i += 4) {+ image.data[i] = 255; // pure red, opaque+ image.data[i + 3] = 255;+ }+ await runner.detect(image);+ const input = provider.created[0].inputs[0];+ expect(input.name).toBe("input");+ expect(input.dims).toEqual([1, 3, 640, 640]);+ const plane = 640 * 640;+ const { mean, std } = PREPROCESS_BY_FAMILY["rtmdet-ins"];+ // content row: B plane = 0, R plane (index 2) = 255+ expect(input.data[0]).toBeCloseTo((0 - mean[0]) / std[0], 4);+ expect(input.data[2 * plane]).toBeCloseTo((255 - mean[2]) / std[2], 4);+ // padded row (y = 400, below the 320-row content)+ expect(input.data[400 * 640]).toBeCloseTo((114 - mean[0]) / std[0], 4);+ });++ it("pose() decodes RTMO keypoints through the centered letterbox inverse", async () => {+ const { store, fetches } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new ScriptedProvider([rtmoOutput()]),+ });+ // 1280×640 → scale 0.5, content centered vertically: offsetY = 160+ const res = await runner.pose(grayImage(1280, 640));+ expect(fetches[0]).toContain(VISION_MODELS["rtmo-s"].filename);+ expect(res.people).toHaveLength(1);+ const nose = res.people[0].keypoints[0];+ expect(nose.x).toBeCloseTo(640);+ expect(nose.y).toBeCloseTo(80);+ expect(nose.visibility).toBeCloseTo(0.9);+ expect(res.people[0].keypoints).toHaveLength(17);+ });++ it("matte() picks the aspect-matched Selfie Segmenter and upsamples to the frame", async () => {+ const { store, fetches } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new ScriptedProvider([selfieOutput(256, 144)]),+ });+ const matte = await runner.matte(grayImage(1280, 720));+ expect(fetches[0]).toContain(VISION_MODELS["selfie-landscape"].filename);+ expect(matte.width).toBe(1280);+ expect(matte.mask[360 * 1280 + 100]).toBe(255);+ expect(matte.mask[360 * 1280 + 1200]).toBe(0);+ expect(matte.coverage).toBeGreaterThan(0.4);+ expect(matte.coverage).toBeLessThan(0.6);+ });++ it("preload() fetches only the requested tasks' models", async () => {+ const { store, fetches } = fakeStore();+ const runner = VisionRunner.create({+ store,+ variant: "t",+ provider: new ScriptedProvider([]),+ });+ await runner.preload(["segment", "pose"]);+ expect(fetches).toHaveLength(2);+ expect(runner.downloadStatus.get("rtmdet-ins-t")).toBe("ready");+ expect(runner.downloadStatus.get("rtmo-t")).toBe("ready");+ expect(runner.downloadStatus.get("rtmdet-ins-s")).toBe("pending");+ });++ it("names the missing output when a model does not match its family", async () => {+ const { store } = fakeStore();+ const runner = VisionRunner.create({+ store,+ provider: new ScriptedProvider([{}]),+ });+ await expect(runner.pose(grayImage())).rejects.toThrow(+ /'rtmo-s' returned no 'dets' output/,+ );+ });++ it("close() releases sessions and recreates them on next use", async () => {+ const { store } = fakeStore();+ const provider = new ScriptedProvider([+ rtmdetInsOutput(),+ rtmdetInsOutput(),+ ]);+ const runner = VisionRunner.create({ store, provider });+ await runner.detect(grayImage());+ runner.close();+ await runner.detect(grayImage());+ expect(provider.created).toHaveLength(2);+ });+});diff --git a/packages/vision/src/runner/vision-runner.ts b/packages/vision/src/runner/vision-runner.tsnew file mode 100644index 00000000..66d536ad--- /dev/null+++ b/packages/vision/src/runner/vision-runner.ts@@ -0,0 +1,337 @@+import type { DetectedObject } from "@gitframes/core";+import {+ type InputTransform,+ imageToTensor,+ preprocessSpec,+} from "../decode/preprocess.js";+import { decodeRtmdetIns } from "../decode/rtmdet-ins.js";+import { decodeRtmo } from "../decode/rtmo.js";+import { decodeSelfie } from "../decode/selfie.js";+import {+ type ModelDownloadStatus,+ VisionModelStore,+} from "../model/model-store.js";+import {+ COCO_CLASSES,+ modelKeyFor,+ VISION_MODELS,+ type VisionModelKey,+ type VisionTask,+ type VisionVariant,+} from "../model/registry.js";+import {+ NodeSessionProvider,+ type SessionProvider,+ type VisionSession,+ type VisionTensor,+ type VisionTensorOutputs,+} from "../runtime/session-provider.js";+import type {+ PersonMatte,+ PoseResult,+ SegmentationResult,+ VisionImageInput,+} from "../types.js";++export interface VisionRunnerOptions {+ /** Model size for detect/segment/pose. Default "s". */+ readonly variant?: VisionVariant;+ /** Minimum detection / person score, 0..1. Default 0.3. */+ readonly confidence?: number;+ /** COCO class-name filter for detect/segment (all classes when omitted). */+ readonly classes?: readonly string[];+ readonly maskThreshold?: number;+ readonly featherRadius?: number;+ readonly modelsDir?: string;+ /** Mirror origin for model downloads (see VisionModelStore). */+ readonly baseUrl?: string;+ readonly timeoutMs?: number;+ /** Session backend. Default: onnxruntime-node (CPU). Browser: `createWebGPUProvider()`. */+ readonly provider?: SessionProvider;+ /** Inject a model store (tests, shared caches). Overrides modelsDir/baseUrl/timeoutMs. */+ readonly store?: VisionModelStore;+ readonly onProgress?: (+ key: VisionModelKey,+ bytesLoaded: number,+ totalBytes: number,+ ) => void;+}++/**+ * Lazy vision runner.+ *+ * `create()` performs ZERO I/O: no downloads, no sessions. Each inference method downloads+ * its model and warms its session on FIRST USE ONLY, then reuses them for the process+ * lifetime. `detect()` and `segment()` share one RTMDet-Ins forward pass per frame.+ * Call `close()` to release sessions.+ */+export class VisionRunner {+ public readonly variant: VisionVariant;+ public readonly confidence: number;+ public readonly classes?: readonly string[];+ public readonly maskThreshold: number;+ public readonly featherRadius?: number;++ private readonly _store: VisionModelStore;+ private readonly _provider: SessionProvider;+ private readonly _sessions = new Map<+ VisionModelKey,+ Promise<VisionSession>+ >();+ private readonly _tensorScratch = new Map<VisionModelKey, Float32Array>();+ /** One RTMDet-Ins pass per frame buffer, shared by detect() and segment(). */+ private _instancePass?: {+ readonly frame: VisionImageInput["data"];+ readonly result: Promise<InstancePass>;+ };++ private constructor(options: VisionRunnerOptions) {+ this.variant = options.variant ?? "s";+ this.confidence = options.confidence ?? 0.3;+ this.classes = options.classes;+ this.maskThreshold = options.maskThreshold ?? 0.5;+ this.featherRadius = options.featherRadius;+ this._store =+ options.store ??+ new VisionModelStore({+ modelsDir: options.modelsDir,+ baseUrl: options.baseUrl,+ timeoutMs: options.timeoutMs,+ onProgress: options.onProgress,+ });+ this._provider = options.provider ?? new NodeSessionProvider();+ }++ /** Cheap: no downloads, no sessions, no side effects. */+ public static create(options: VisionRunnerOptions = {}): VisionRunner {+ return new VisionRunner(options);+ }++ public get modelsDir(): string {+ return this._store.modelsDir;+ }++ /** Every registry model keyed to its current download state (downloaded → "ready"). */+ public get downloadStatus(): ReadonlyMap<+ VisionModelKey,+ ModelDownloadStatus+ > {+ const statuses = new Map<VisionModelKey, ModelDownloadStatus>();+ for (const key of Object.keys(VISION_MODELS) as VisionModelKey[]) {+ statuses.set(key, "pending");+ }+ for (const [key, status] of this._store.statuses()) {+ statuses.set(key, status);+ }+ return statuses;+ }++ /** The model serving `task` (matte depends on frame aspect; defaults to square). */+ public modelKeyFor(task: VisionTask, aspect = 1): VisionModelKey {+ return modelKeyFor(task, this.variant, aspect);+ }++ /**+ * Explicit warm-up: downloads + sessions for the given tasks, ahead of first use.+ * `matte` warms both aspect variants (the frame aspect is unknown until inference).+ */+ public async preload(+ tasks: readonly VisionTask[] = ["detect"],+ ): Promise<void> {+ const keys = new Set<VisionModelKey>();+ for (const task of tasks) {+ if (task === "matte") {+ keys.add("selfie-square");+ keys.add("selfie-landscape");+ } else {+ keys.add(this.modelKeyFor(task));+ }+ }+ await this._store.preload([...keys]);+ await Promise.all([...keys].map((key) => this.session(key)));+ }++ /** COCO-80 boxes (sorted by score, source pixels). Masks are not decoded. */+ public async detect(+ image: VisionImageInput,+ ): Promise<readonly DetectedObject[]> {+ const pass = await this.instancePass(image);+ return decodeRtmdetIns(+ { ...pass.outputs, masks: undefined },+ this.instanceDecodeOptions(image, pass.transform),+ ).detections;+ }++ /** COCO-80 boxes plus a frame-aligned soft mask per instance. */+ public async segment(image: VisionImageInput): Promise<SegmentationResult> {+ const pass = await this.instancePass(image);+ return decodeRtmdetIns(+ pass.outputs,+ this.instanceDecodeOptions(image, pass.transform),+ );+ }++ /** COCO-17 keypoints per person (source pixels), highest score first. */+ public async pose(image: VisionImageInput): Promise<PoseResult> {+ const key = this.modelKeyFor("pose");+ const { outputs, transform } = await this.infer(key, image);+ const dets = output(outputs, "dets", key);+ const keypoints = output(outputs, "keypoints", key);+ return decodeRtmo(+ {+ dets: dets.data as Float32Array,+ keypoints: keypoints.data as Float32Array,+ count: dets.dims[1] ?? 0,+ keypointStride: keypoints.dims[3] ?? 3,+ },+ {+ confidence: this.confidence,+ transform,+ sourceWidth: image.width,+ sourceHeight: image.height,+ },+ );+ }++ /** Person-vs-background alpha for the whole frame (Selfie Segmenter). */+ public async matte(image: VisionImageInput): Promise<PersonMatte> {+ const key = this.modelKeyFor("matte", image.width / image.height);+ const [w, h] = VISION_MODELS[key].input;+ const { outputs } = await this.infer(key, image);+ const alphas = output(outputs, "alphas", key);+ return decodeSelfie(alphas.data as Float32Array, w, h, {+ sourceWidth: image.width,+ sourceHeight: image.height,+ maskThreshold: this.maskThreshold,+ ...(this.featherRadius !== undefined+ ? { featherRadius: this.featherRadius }+ : {}),+ });+ }++ public close(): void {+ for (const promise of this._sessions.values()) {+ void promise.then((session) => session.release()).catch(() => undefined);+ }+ this._sessions.clear();+ this._instancePass = undefined;+ this._store.clearMemoryCache();+ }++ // ── internals ──────────────────────────────────────────────────────────++ private instancePass(image: VisionImageInput): Promise<InstancePass> {+ if (this._instancePass?.frame === image.data) {+ return this._instancePass.result;+ }+ const key = this.modelKeyFor("segment");+ const result = this.infer(key, image).then(+ ({ outputs, transform }) => {+ const dets = output(outputs, "dets", key);+ const masks = output(outputs, "masks", key);+ return {+ transform,+ outputs: {+ dets: dets.data as Float32Array,+ labels: output(outputs, "labels", key).data,+ masks: masks.data as Float32Array,+ count: dets.dims[1] ?? 0,+ maskHeight: masks.dims[masks.dims.length - 2] ?? 0,+ maskWidth: masks.dims[masks.dims.length - 1] ?? 0,+ },+ };+ },+ );+ this._instancePass = { frame: image.data, result };+ result.catch(() => {+ if (this._instancePass?.result === result) this._instancePass = undefined;+ });+ return result;+ }++ private instanceDecodeOptions(+ image: VisionImageInput,+ transform: InputTransform,+ ) {+ return {+ confidence: this.confidence,+ classes: this.classes,+ classNames: COCO_CLASSES,+ transform,+ sourceWidth: image.width,+ sourceHeight: image.height,+ maskThreshold: this.maskThreshold,+ ...(this.featherRadius !== undefined+ ? { featherRadius: this.featherRadius }+ : {}),+ };+ }++ private async infer(+ key: VisionModelKey,+ image: VisionImageInput,+ ): Promise<{ outputs: VisionTensorOutputs; transform: InputTransform }> {+ const session = await this.session(key);+ const desc = VISION_MODELS[key];+ const spec = preprocessSpec(desc.family, desc.input);+ const size = 3 * spec.width * spec.height;+ let scratch = this._tensorScratch.get(key);+ if (!scratch || scratch.length < size) {+ scratch = new Float32Array(size);+ this._tensorScratch.set(key, scratch);+ }+ const { tensor, transform } = imageToTensor(image, spec, scratch);+ const outputs = await session.run({+ name: "input",+ data: tensor,+ dims: [1, 3, spec.height, spec.width],+ });+ return { outputs, transform };+ }++ private session(key: VisionModelKey): Promise<VisionSession> {+ let promise = this._sessions.get(key);+ if (!promise) {+ promise = this._store+ .ensure(key)+ .then((bytes) => this._provider.createSession(bytes))+ .catch((err: unknown) => {+ // Drop both the session cache entry AND the cached file: a parse+ // failure means the on-disk bytes are corrupt — the next call must+ // re-download instead of replaying the same bad file.+ this._sessions.delete(key);+ void this._store.evict(key);+ throw err;+ });+ this._sessions.set(key, promise);+ }+ return promise;+ }+}++interface InstancePass {+ readonly transform: InputTransform;+ readonly outputs: {+ readonly dets: Float32Array;+ readonly labels: VisionTensor["data"];+ readonly masks: Float32Array;+ readonly count: number;+ readonly maskWidth: number;+ readonly maskHeight: number;+ };+}++function output(+ outputs: VisionTensorOutputs,+ name: string,+ key: VisionModelKey,+): VisionTensor {+ const tensor = outputs[name];+ if (!tensor) {+ throw new Error(+ `Vision model '${key}' returned no '${name}' output (got: ${Object.keys(outputs).join(", ") || "none"}). ` ++ `Expected the ${VISION_MODELS[key].source}.`,+ );+ }+ return tensor;+}diff --git a/packages/yolo/src/runtime/session-provider.ts b/packages/vision/src/runtime/session-provider.tssimilarity index 60%rename from packages/yolo/src/runtime/session-provider.tsrename to packages/vision/src/runtime/session-provider.tsindex e3d12b73..d21ed222 100644--- a/packages/yolo/src/runtime/session-provider.ts+++ b/packages/vision/src/runtime/session-provider.ts@@ -4,32 +4,28 @@* entry can swap in a WebGPU-backed provider with the same interface.*/-export interface YoloTensor {- readonly data: Float32Array;+/** Model output tensor. Integer outputs (e.g. RTMDet-Ins `labels`) arrive as BigInt64Array. */+export interface VisionTensor {+ readonly data: Float32Array | BigInt64Array | Int32Array;readonly dims: readonly number[];}-export interface YoloTensorInput {+export interface VisionTensorInput {readonly name: string;readonly data: Float32Array;readonly dims: readonly number[];}-export type YoloTensorOutputs = Readonly<Record<string, YoloTensor>>;+export type VisionTensorOutputs = Readonly<Record<string, VisionTensor>>;-export interface YoloSession {- run(- input: YoloTensorInput | readonly YoloTensorInput[],- ): Promise<YoloTensorOutputs>;+export interface VisionSession {+ run(input: VisionTensorInput): Promise<VisionTensorOutputs>;release(): void;}export interface SessionProvider {readonly kind: "node" | "webgpu";- createSession(- modelBytes: Uint8Array,- opts: { imgsz: number; fp16?: boolean },- ): Promise<YoloSession>;+ createSession(modelBytes: Uint8Array): Promise<VisionSession>;}/** onnxruntime-node provider. The native module is imported lazily on first session. */@@ -51,10 +47,7 @@ export class NodeSessionProvider implements SessionProvider {return this.ortPromise;}- public async createSession(- modelBytes: Uint8Array,- _opts: { imgsz: number; fp16?: boolean },- ): Promise<YoloSession> {+ public async createSession(modelBytes: Uint8Array): Promise<VisionSession> {const ort = await this.ort();const session = await ort.InferenceSession.create(modelBytes, {executionProviders: ["cpu"],@@ -63,22 +56,18 @@ export class NodeSessionProvider implements SessionProvider {type OrtTensor = InstanceType<typeof ort.Tensor>;return {- run: async (- input: YoloTensorInput | readonly YoloTensorInput[],- ): Promise<YoloTensorOutputs> => {- const inputs = Array.isArray(input) ? input : [input];- const feeds: Record<string, OrtTensor> = {};- for (const inp of inputs) {- const inputName = session.inputNames.includes(inp.name)- ? inp.name- : (session.inputNames[0] ?? inp.name);- feeds[inputName] = new ort.Tensor("float32", inp.data, [...inp.dims]);- }+ run: async (input: VisionTensorInput): Promise<VisionTensorOutputs> => {+ const inputName = session.inputNames.includes(input.name)+ ? input.name+ : (session.inputNames[0] ?? input.name);+ const feeds: Record<string, OrtTensor> = {+ [inputName]: new ort.Tensor("float32", input.data, [...input.dims]),+ };const results = await session.run(feeds);- const outputs: Record<string, YoloTensor> = {};+ const outputs: Record<string, VisionTensor> = {};for (const [name, tensor] of Object.entries(results)) {outputs[name] = {- data: tensor.data as Float32Array,+ data: tensor.data as VisionTensor["data"],dims: [...tensor.dims],};}diff --git a/packages/yolo/src/runtime/webgpu-provider.test.ts b/packages/vision/src/runtime/webgpu-provider.test.tssimilarity index 87%rename from packages/yolo/src/runtime/webgpu-provider.test.tsrename to packages/vision/src/runtime/webgpu-provider.test.tsindex 86bafef8..7a4620a2 100644--- a/packages/yolo/src/runtime/webgpu-provider.test.ts+++ b/packages/vision/src/runtime/webgpu-provider.test.ts@@ -7,7 +7,7 @@ import {} from "./webgpu-provider.js";class FakeOrtSession implements OrtWebSession {- public readonly inputNames = ["images"];+ public readonly inputNames = ["input"];async run(): Promise<Record<string, { data: Float32Array; dims: readonly number[] }>> {@@ -51,11 +51,9 @@ describe("WebGPUProvider", () => {});expect(provider.kind).toBe("webgpu");- const session = await provider.createSession(new Uint8Array([1, 2, 3]), {- imgsz: 640,- });+ const session = await provider.createSession(new Uint8Array([1, 2, 3]));const outputs = await session.run({- name: "images",+ name: "input",data: new Float32Array(3 * 8 * 8),dims: [1, 3, 8, 8],});@@ -64,14 +62,14 @@ describe("WebGPUProvider", () => {expect(calls[0].executionProviders).toEqual(["webgpu", "wasm"]);// loader is memoized across sessions- await provider.createSession(new Uint8Array([1]), { imgsz: 640 });+ await provider.createSession(new Uint8Array([1]));expect(loads).toBe(1);});it("falls back to wasm when the WebGPU session fails", async () => {const { module, calls } = fakeModule({ failWebGPU: true });const provider = new WebGPUProvider({ loader: async () => module });- await provider.createSession(new Uint8Array([1]), { imgsz: 640 });+ await provider.createSession(new Uint8Array([1]));expect(calls.map((c) => c.executionProviders)).toEqual([["webgpu", "wasm"],["wasm"],@@ -84,9 +82,9 @@ describe("WebGPUProvider", () => {loader: async () => module,wasmFallback: false,});- await expect(- provider.createSession(new Uint8Array([1]), { imgsz: 640 }),- ).rejects.toThrow("webgpu unavailable");+ await expect(provider.createSession(new Uint8Array([1]))).rejects.toThrow(+ "webgpu unavailable",+ );});it("hasWebGPU() is false in Node", () => {diff --git a/packages/yolo/src/runtime/webgpu-provider.ts b/packages/vision/src/runtime/webgpu-provider.tssimilarity index 78%rename from packages/yolo/src/runtime/webgpu-provider.tsrename to packages/vision/src/runtime/webgpu-provider.tsindex 75ea8364..90ac8889 100644--- a/packages/yolo/src/runtime/webgpu-provider.ts+++ b/packages/vision/src/runtime/webgpu-provider.ts@@ -1,25 +1,25 @@/**- * Browser / WebGPU session provider (specs/yolov4plan.ts §Phase E).+ * Browser / WebGPU session provider .** Mirrors `NodeSessionProvider` but uses `onnxruntime-web` with the WebGPU execution provider* and a WASM fallback. The ORT module is imported lazily behind an injectable loader, so:* - the Node core entry never pulls browser code, and* - unit tests can run fully offline with a fake module.*- * `onnxruntime-web` is an OPTIONAL peer dependency — consumers of `@gitframes/yolo/web`- * provide it; the main `@gitframes/yolo` entry never touches it.+ * `onnxruntime-web` is an OPTIONAL peer dependency — consumers of `@gitframes/vision/web`+ * provide it; the main `@gitframes/vision` entry never touches it.*/import type {SessionProvider,- YoloSession,- YoloTensorInput,- YoloTensorOutputs,+ VisionSession,+ VisionTensorInput,+ VisionTensorOutputs,} from "./session-provider.js";/** Minimal structural view of the `onnxruntime-web` module — isolates the ORT API surface. */export interface OrtWebTensor {- readonly data: Float32Array;+ readonly data: Float32Array | BigInt64Array | Int32Array;readonly dims: readonly number[];}@@ -90,10 +90,7 @@ export class WebGPUProvider implements SessionProvider {return this._ortPromise;}- public async createSession(- modelBytes: Uint8Array,- _opts: { imgsz: number; fp16?: boolean },- ): Promise<YoloSession> {+ public async createSession(modelBytes: Uint8Array): Promise<VisionSession> {const ort = await this.ort();let session: OrtWebSession;@@ -111,22 +108,17 @@ export class WebGPUProvider implements SessionProvider {}return {- run: async (- input: YoloTensorInput | readonly YoloTensorInput[],- ): Promise<YoloTensorOutputs> => {- const inputs = Array.isArray(input) ? input : [input];- const feeds: Record<string, unknown> = {};- for (const inp of inputs) {- const inputName = session.inputNames.includes(inp.name)- ? inp.name- : (session.inputNames[0] ?? inp.name);- feeds[inputName] = new ort.Tensor("float32", inp.data, [...inp.dims]);- }- const results = await session.run(feeds);- const outputs: Record<string, YoloTensorOutputs[string]> = {};+ run: async (input: VisionTensorInput): Promise<VisionTensorOutputs> => {+ const inputName = session.inputNames.includes(input.name)+ ? input.name+ : (session.inputNames[0] ?? input.name);+ const results = await session.run({+ [inputName]: new ort.Tensor("float32", input.data, [...input.dims]),+ });+ const outputs: Record<string, VisionTensorOutputs[string]> = {};for (const [name, tensor] of Object.entries(results)) {outputs[name] = {- data: tensor.data as Float32Array,+ data: tensor.data,dims: [...tensor.dims],};}@@ -141,7 +133,7 @@ export class WebGPUProvider implements SessionProvider {/*** Convenience factory: a `SessionProvider` type guard for the browser entry. Kept tiny so- * `@gitframes/yolo/web` stays tree-shakeable.+ * `@gitframes/vision/web` stays tree-shakeable.*/export function createWebGPUProvider(options?: WebGPUProviderOptions,diff --git a/packages/vision/src/schemas.test.ts b/packages/vision/src/schemas.test.tsnew file mode 100644index 00000000..1d497d91--- /dev/null+++ b/packages/vision/src/schemas.test.ts@@ -0,0 +1,72 @@+import { describe, expect, it } from "vitest";+import {+ PoseResultSchema,+ VisionConfigSchema,+ VisionSummarySchema,+} from "./schemas.js";++describe("vision schemas", () => {+ it("validates a frame summary", () => {+ const summary = {+ frame: 12,+ objects: [+ {+ trackId: 1,+ category: "person",+ score: 0.91,+ center: [540, 312] as [number, number],+ speed: 128,+ active: true,+ },+ ],+ classes: ["person"],+ masks: { person: 1 },+ };+ expect(VisionSummarySchema.safeParse(summary).success).toBe(true);+ expect(+ VisionSummarySchema.safeParse({ ...summary, frame: -1 }).success,+ ).toBe(false);+ });++ it("validates the runtime config surface and rejects removed options", () => {+ expect(+ VisionConfigSchema.safeParse({+ enableMatte: true,+ variant: "s",+ confidence: 0.4,+ }).success,+ ).toBe(true);+ expect(VisionConfigSchema.safeParse({ confidence: 1.5 }).success).toBe(+ false,+ );+ expect(VisionConfigSchema.safeParse({ variant: "n" }).success).toBe(false);+ expect(VisionConfigSchema.safeParse({ enableObb: true }).success).toBe(+ false,+ );+ });++ it("requires exactly 17 COCO keypoints per person", () => {+ const box = {+ originX: 0,+ originY: 0,+ width: 1,+ height: 1,+ normalizedX: 0,+ normalizedY: 0,+ normalizedWidth: 1,+ normalizedHeight: 1,+ };+ const kp = { x: 0, y: 0, visibility: 1 };+ const person = {+ score: 0.9,+ boundingBox: box,+ keypoints: Array(17).fill(kp),+ };+ expect(PoseResultSchema.safeParse({ people: [person] }).success).toBe(true);+ expect(+ PoseResultSchema.safeParse({+ people: [{ ...person, keypoints: Array(16).fill(kp) }],+ }).success,+ ).toBe(false);+ });+});diff --git a/packages/yolo/src/schemas.ts b/packages/vision/src/schemas.tssimilarity index 50%rename from packages/yolo/src/schemas.tsrename to packages/vision/src/schemas.tsindex c12e1f97..63af6027 100644--- a/packages/yolo/src/schemas.ts+++ b/packages/vision/src/schemas.ts@@ -1,16 +1,17 @@import { z } from "zod";+import { VISION_VARIANTS } from "./model/registry.js";/**- * Shared zod schemas for YOLO outputs (specs/yolov4plan.ts §Phase A).+ * Zod schemas for vision outputs and config.*- * Deliberately a SEPARATE entry point (`@gitframes/yolo/schemas`) so the core runtime entry+ * Deliberately a SEPARATE entry point (`@gitframes/vision/schemas`) so the core runtime entry* stays zod-free: the hot path (runner, decode, signals) never imports this module. Agents,* node config validation, and report round-tripping consume these.*/-// ── Bounding box / detection ────────────────────────────────────────────────+// ── Detection ───────────────────────────────────────────────────────────────-export const YoloBoundingBoxSchema = z.object({+export const BoundingBoxSchema = z.object({originX: z.number(),originY: z.number(),width: z.number(),@@ -19,19 +20,17 @@ export const YoloBoundingBoxSchema = z.object({normalizedY: z.number(),normalizedWidth: z.number(),normalizedHeight: z.number(),- angle: z.number().optional(),});-export const YoloDetectionSchema = z.object({+export const DetectedObjectSchema = z.object({category: z.string(),- classIndex: z.number().int(),score: z.number(),- boundingBox: YoloBoundingBoxSchema,+ boundingBox: BoundingBoxSchema,});-// ── Instance segmentation ───────────────────────────────────────────────────+// ── Instance segmentation / matte ───────────────────────────────────────────-export const YoloInstanceMaskSchema = z.object({+export const InstanceMaskSchema = z.object({category: z.string(),mask: z.instanceof(Uint8Array),width: z.number().int().positive(),@@ -42,55 +41,40 @@ export const YoloInstanceMaskSchema = z.object({trackId: z.number().int().optional(),});-export const YoloSegmentationResultSchema = z.object({- detections: z.array(YoloDetectionSchema),- masks: z.array(YoloInstanceMaskSchema),+export const SegmentationResultSchema = z.object({+ detections: z.array(DetectedObjectSchema),+ masks: z.array(InstanceMaskSchema),+});++export const PersonMatteSchema = z.object({+ mask: z.instanceof(Uint8Array),+ width: z.number().int().positive(),+ height: z.number().int().positive(),+ coverage: z.number().min(0).max(1),});// ── Pose ────────────────────────────────────────────────────────────────────-export const YoloPoseKeypointSchema = z.object({+export const PoseKeypointSchema = z.object({x: z.number(),y: z.number(),visibility: z.number(),});-export const YoloPosePersonSchema = z.object({+export const PosePersonSchema = z.object({score: z.number(),- boundingBox: YoloBoundingBoxSchema,- keypoints: z.array(YoloPoseKeypointSchema).length(17),+ boundingBox: BoundingBoxSchema,+ keypoints: z.array(PoseKeypointSchema).length(17),trackId: z.number().int().optional(),});-export const YoloPoseResultSchema = z.object({- people: z.array(YoloPosePersonSchema),-});--// ── Classification ──────────────────────────────────────────────────────────--export const YoloClassifyResultSchema = z.object({- top1: z.number().int(),- top1Score: z.number(),- top5: z.array(z.object({ index: z.number().int(), score: z.number() })),-});--// ── OBB ─────────────────────────────────────────────────────────────────────--export const YoloOBBDetectionSchema = z.object({- category: z.string(),- classIndex: z.number().int(),- score: z.number(),- boundingBox: YoloBoundingBoxSchema.extend({ angle: z.number() }),- corners: z.array(z.tuple([z.number(), z.number()])).length(4),-});--export const YoloOBBResultSchema = z.object({- detections: z.array(YoloOBBDetectionSchema),+export const PoseResultSchema = z.object({+ people: z.array(PosePersonSchema),});// ── Frame summary (agent DX) ────────────────────────────────────────────────-export const YoloSummarySchema = z.object({+export const VisionSummarySchema = z.object({frame: z.number().int().nonnegative(),objects: z.array(z.object({@@ -108,36 +92,22 @@ export const YoloSummarySchema = z.object({// ── Runtime config ──────────────────────────────────────────────────────────-export const CustomModelConfigSchema = z.object({- path: z.string().optional(),- url: z.string().optional(),- task: z.enum(["detect", "segment", "pose", "obb", "classify", "world"]),- classes: z.array(z.string()),- imgsz: z.number().int().min(320).max(1280).optional(),-});--export const YoloConfigSchema = z.object({- enableDetection: z.boolean().optional(),- enableSegmentation: z.boolean().optional(),- enablePose: z.boolean().optional(),- enableClassification: z.boolean().optional(),- enableObb: z.boolean().optional(),- enableWorld: z.boolean().optional(),- prompts: z.array(z.string()).optional(),- customModel: CustomModelConfigSchema.optional(),- classes: z.array(z.string()).optional(),- confidence: z.number().min(0).max(1).optional(),- iouThreshold: z.number().min(0).max(1).optional(),- variant: z.enum(["n", "s", "m", "l", "x"]).optional(),- imgsz: z.number().int().min(320).max(1280).optional(),- delegate: z.enum(["cpu", "webgpu"]).optional(),- modelsDir: z.string().optional(),- baseUrl: z.string().optional(),- maskThreshold: z.number().min(0).max(1).optional(),- featherRadius: z.number().min(0).max(1).optional(),- autoDownload: z.boolean().optional(),- cameraFov: z.number().optional(),-});+export const VisionConfigSchema = z+ .object({+ enableDetection: z.boolean().optional(),+ enableSegmentation: z.boolean().optional(),+ enablePose: z.boolean().optional(),+ enableMatte: z.boolean().optional(),+ classes: z.array(z.string()).optional(),+ confidence: z.number().min(0).max(1).optional(),+ variant: z.enum(VISION_VARIANTS).optional(),+ modelsDir: z.string().optional(),+ baseUrl: z.string().optional(),+ maskThreshold: z.number().min(0).max(1).optional(),+ featherRadius: z.number().min(0).max(1).optional(),+ cameraFov: z.number().optional(),+ })+ .strict();// ── Analysis report (Phase C) ───────────────────────────────────────────────diff --git a/packages/vision/src/segmentation/subject.test.ts b/packages/vision/src/segmentation/subject.test.tsnew file mode 100644index 00000000..b815942a--- /dev/null+++ b/packages/vision/src/segmentation/subject.test.ts@@ -0,0 +1,78 @@+import { describe, expect, it } from "vitest";+import type { InstanceMask } from "../types.js";+import { maskBounds, mergeSubjectMask, selectSubjectMask } from "./subject.js";++const W = 20;+const H = 20;++function rect(+ category: string,+ x0: number,+ y0: number,+ x1: number,+ y1: number,+ detectionIndex = 0,+): InstanceMask {+ const mask = new Uint8Array(W * H);+ let area = 0;+ for (let y = y0; y <= y1; y++) {+ for (let x = x0; x <= x1; x++) {+ mask[y * W + x] = 255;+ area++;+ }+ }+ return {+ category,+ mask,+ width: W,+ height: H,+ area,+ coverage: area / (W * H),+ detectionIndex,+ };+}++describe("subject selection", () => {+ it("prefers a person over a larger non-person instance", () => {+ const person = rect("person", 0, 0, 2, 2);+ const car = rect("car", 5, 5, 15, 15);+ expect(selectSubjectMask([car, person])).toBe(person);+ expect(selectSubjectMask([car])).toBe(car);+ expect(selectSubjectMask([])).toBeUndefined();+ });++ it("without a person, prefers the most confident instance over a large backdrop", () => {+ const table = rect("dining table", 0, 0, 19, 19, 1);+ const cup = rect("cup", 6, 6, 12, 12, 0);+ expect(selectSubjectMask([table, cup])).toBe(cup);+ });++ it("does not merge a container much larger than the subject", () => {+ const figure = rect("person", 9, 9, 10, 11);+ const tunnel = rect("clock", 0, 0, 19, 19, 1);+ expect(mergeSubjectMask([tunnel, figure])).toBe(figure);+ });++ it("merges instances overlapping the subject (e.g. a dress detected as another class)", () => {+ const body = rect("person", 8, 2, 12, 17);+ const dress = rect("horse", 4, 6, 16, 16);+ const farCar = rect("car", 0, 18, 3, 19);+ const merged = mergeSubjectMask([farCar, dress, body]);+ expect(merged?.category).toBe("person");+ expect(merged?.mask[10 * W + 5]).toBe(255); // dress pixel joined the subject+ expect(merged?.mask[19 * W + 1]).toBe(0); // unrelated car stays out+ expect(merged?.area).toBe(body.area + dress.area - 5 * 11);+ });++ it("maskBounds returns tight bounds or null", () => {+ expect(maskBounds(rect("x", 3, 4, 7, 9))).toEqual({+ x0: 3,+ y0: 4,+ x1: 7,+ y1: 9,+ });+ expect(+ maskBounds({ ...rect("x", 0, 0, 0, 0), mask: new Uint8Array(W * H) }),+ ).toBeNull();+ });+});diff --git a/packages/vision/src/segmentation/subject.ts b/packages/vision/src/segmentation/subject.tsnew file mode 100644index 00000000..da936207--- /dev/null+++ b/packages/vision/src/segmentation/subject.ts@@ -0,0 +1,120 @@+import type { InstanceMask } from "../types.js";++/**+ * The frame's primary instance: the largest person when present, else the most confident+ * instance (lowest `detectionIndex` — detections are score-sorted). Size alone is a poor+ * signal without a person: the largest instance is usually a backdrop ("dining table").+ */+export function selectSubjectMask(+ masks: readonly InstanceMask[],+): InstanceMask | undefined {+ let best: InstanceMask | undefined;+ for (const m of masks) {+ if (!best) {+ best = m;+ continue;+ }+ const mIsPerson = isPerson(m);+ const bestIsPerson = isPerson(best);+ if (mIsPerson !== bestIsPerson) {+ if (mIsPerson) best = m;+ } else if (+ mIsPerson ? m.area > best.area : m.detectionIndex < best.detectionIndex+ ) {+ best = m;+ }+ }+ return best;+}++function isPerson(mask: InstanceMask): boolean {+ return mask.category.toLowerCase() === "person";+}++export interface MaskBounds {+ readonly x0: number;+ readonly y0: number;+ readonly x1: number;+ readonly y1: number;+}++/** Tight pixel bounds of a mask's non-zero alpha, or null when empty. */+export function maskBounds(mask: InstanceMask): MaskBounds | null {+ const { mask: data, width, height } = mask;+ let x0 = width;+ let y0 = height;+ let x1 = -1;+ let y1 = -1;+ for (let y = 0; y < height; y++) {+ const row = y * width;+ for (let x = 0; x < width; x++) {+ if (data[row + x] === 0) continue;+ if (x < x0) x0 = x;+ if (x > x1) x1 = x;+ if (y < y0) y0 = y;+ if (y > y1) y1 = y;+ }+ }+ return x1 < 0 ? null : { x0, y0, x1, y1 };+}++/**+ * Builds the full subject silhouette: the primary instance plus every comparably sized+ * instance whose bounds sit mostly inside or across it (`overlap` = intersection / smaller+ * box area; `maxGrowth` caps a part's box area relative to the subject's).+ *+ * COCO has no "clothing" class, so a flowing dress, a held guitar or a ridden bike comes back+ * as its own instance (often mislabeled) — merging them keeps the whole figure in the cutout.+ * The size cap keeps containers out: a small figure inside a tunnel or window detected as a+ * huge "clock" must not drag the whole frame into the subject.+ */+export function mergeSubjectMask(+ masks: readonly InstanceMask[],+ overlap = 0.5,+ maxGrowth = 2,+): InstanceMask | undefined {+ const subject = selectSubjectMask(masks);+ if (!subject) return undefined;+ const sb = maskBounds(subject);+ if (!sb) return subject;++ const subjectArea = boundsArea(sb);+ const parts = masks.filter((m) => {+ if (m === subject) return false;+ const b = maskBounds(m);+ return (+ b !== null &&+ boundsArea(b) <= subjectArea * maxGrowth &&+ boundsOverlap(sb, b) >= overlap+ );+ });+ if (parts.length === 0) return subject;++ const merged = subject.mask.slice();+ for (const part of parts) {+ const data = part.mask;+ for (let i = 0; i < merged.length; i++) {+ if (data[i] > merged[i]) merged[i] = data[i];+ }+ }+ let area = 0;+ for (let i = 0; i < merged.length; i++) area += merged[i];+ area = Math.round(area / 255);+ return {+ ...subject,+ mask: merged,+ area,+ coverage: area / (subject.width * subject.height),+ };+}++function boundsArea(b: MaskBounds): number {+ return (b.x1 - b.x0 + 1) * (b.y1 - b.y0 + 1);+}++function boundsOverlap(a: MaskBounds, b: MaskBounds): number {+ const iw = Math.min(a.x1, b.x1) - Math.max(a.x0, b.x0) + 1;+ const ih = Math.min(a.y1, b.y1) - Math.max(a.y0, b.y0) + 1;+ if (iw <= 0 || ih <= 0) return 0;+ return (iw * ih) / Math.min(boundsArea(a), boundsArea(b));+}diff --git a/packages/yolo/src/signals/yolo-signals-bundle.test.ts b/packages/vision/src/signals/vision-bundle.test.tssimilarity index 80%rename from packages/yolo/src/signals/yolo-signals-bundle.test.tsrename to packages/vision/src/signals/vision-bundle.test.tsindex 86b0e063..cad3de66 100644--- a/packages/yolo/src/signals/yolo-signals-bundle.test.ts+++ b/packages/vision/src/signals/vision-bundle.test.ts@@ -6,11 +6,11 @@ import {} from "@gitframes/core";import { describe, expect, it } from "vitest";import type {+ InstanceMask,LandmarkCoordinateSignals,- YoloInstanceMask,- YoloPoseResult,+ PoseResult,} from "../types.js";-import { YoloVisionBundle } from "./yolo-signals-bundle.js";+import { VisionBundle } from "./vision-bundle.js";const ctx = (frame: number): FrameContext => ({frame,@@ -58,7 +58,7 @@ function track(};}-function poseResult(score = 0.8): YoloPoseResult {+function poseResult(score = 0.8): PoseResult {const keypoints = Array.from({ length: 17 }, (_, i) => ({x: 0.1 + i * 0.02,y: 0.3 + i * 0.01,@@ -81,7 +81,7 @@ function mask(area: number,width = 100,height = 100,-): YoloInstanceMask {+): InstanceMask {return {category,mask: new Uint8Array(width * height),@@ -94,9 +94,9 @@ function mask(};}-describe("YoloVisionBundle", () => {+describe("VisionBundle", () => {it("named pose landmarks mirror the primary person's COCO-17 keypoints", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -121,7 +121,7 @@ describe("YoloVisionBundle", () => {});it("pose signals fall back to neutral when nobody is detected", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -132,7 +132,7 @@ describe("YoloVisionBundle", () => {});it("objects signals expose tracked state, categories and kinematics", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -160,7 +160,7 @@ describe("YoloVisionBundle", () => {});it("classes surface per-category count/maxConfidence/present with stable identity", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -185,7 +185,7 @@ describe("YoloVisionBundle", () => {});it("masks subject = largest instance, person preferred", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -215,7 +215,7 @@ describe("YoloVisionBundle", () => {});it("segmentation mirrors subject + holds the stencil texture", () => {- const bundle = new YoloVisionBundle({ totalFrames: 10 });+ const bundle = new VisionBundle({ totalFrames: 10 });bundle.setMaskResult(1, [mask(5, "person", 900)]);frameSignal.set(1);expect(bundle.segmentation.subject.trackId).toBe(5);@@ -226,29 +226,8 @@ describe("YoloVisionBundle", () => {expect(bundle.segmentation.stencilTexture).toBe(stencil);});- it("classification signals read the frame cache", () => {- const bundle = new YoloVisionBundle({- totalFrames: 10,- width: 100,- height: 100,- });- bundle.setClassifyResult(0, {- top1: 0,- top1Score: 0.934,- top5: [- { index: 0, score: 0.934 },- { index: 14, score: 0.03 },- ],- });- frameSignal.set(0);- expect(bundle.classification.top1.get()).toBe(0);- expect(bundle.classification.top1Confidence.get()).toBeCloseTo(0.934, 5);- expect(bundle.classification.top5[0].index).toBe(0);- expect(bundle.classification.top5At(0)[1].index).toBe(14);- });-it("tensors are well-formed and frame-scoped", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -285,14 +264,14 @@ describe("YoloVisionBundle", () => {});it("clamps out-of-range frame reads to neutral values (no crash)", () => {- const bundle = new YoloVisionBundle({ totalFrames: 10 });+ const bundle = new VisionBundle({ totalFrames: 10 });frameSignal.set(47); // beyond cacheexpect(bundle.objects.count.get()).toBe(0);expect(bundle.masks.subject.coverage.get()).toBe(0);});it("summary() snapshots objects/classes/masks for a frame", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 100,height: 100,@@ -319,89 +298,28 @@ describe("YoloVisionBundle", () => {expect(snap.masks).toEqual({ person: 1, car: 1 });});- it("bounds.angle reads the nearest matching OBB detection (0 when none)", () => {- const bundle = new YoloVisionBundle({- totalFrames: 10,- width: 100,- height: 100,- });- bundle.setObjectResult(0, {- objects: [track(1, "person", 10, 20, 30, 40)],- rawDetections: [],- });- bundle.setObbResult(0, {- detections: [- {- category: "person",- classIndex: 0,- score: 0.9,- boundingBox: {- originX: 12,- originY: 22,- width: 26,- height: 36,- normalizedX: 0.12,- normalizedY: 0.22,- normalizedWidth: 0.26,- normalizedHeight: 0.36,- angle: Math.PI / 4,- },- corners: [- [0, 0],- [1, 0],- [1, 1],- [0, 1],- ],- },- ],- });-- frameSignal.set(0);- expect(bundle.objects.get(1).bounds.angle.get()).toBeCloseTo(- Math.PI / 4,- 6,- );- // frame 1 has no obb result cached directly, but lookback retains frame 0's result- frameSignal.set(1);- expect(bundle.objects.get(1).bounds.angle.get()).toBeCloseTo(- Math.PI / 4,- 6,- );- // frame 10 is >8 frames past frame 0, so it falls back to 0- frameSignal.set(10);- expect(bundle.objects.get(1).bounds.angle.get()).toBe(0);- });-- it("cache lookback retains mask, obb, and classification across intermediate frames", () => {- const bundle = new YoloVisionBundle({+ it("cache lookback retains masks across intermediate frames", () => {+ const bundle = new VisionBundle({totalFrames: 15,width: 100,height: 100,});const m = mask(1, "person", 500);bundle.setMaskResult(2, [m]);- bundle.setClassifyResult(2, {- top1: 5,- top1Score: 0.95,- top5: [{ index: 5, score: 0.95 }],- });// Exactly at frame 2expect(bundle.getMaskResult(2)).toHaveLength(1);- expect(bundle.getClassifyResult(2).top1).toBe(5);// Intermediate query at frame 5 (lookback <= 8 frames)expect(bundle.getMaskResult(5)).toHaveLength(1);expect(bundle.getMaskResult(5)[0].category).toBe("person");- expect(bundle.getClassifyResult(5).top1).toBe(5);// Intermediate query at frame 12 (> 8 frames back from 2) falls back to empty defaultsexpect(bundle.getMaskResult(12)).toHaveLength(0);- expect(bundle.getClassifyResult(12).top1Score).toBe(0);});it("track.pose exposes per-track landmark signals and kinematics", () => {- const bundle = new YoloVisionBundle({+ const bundle = new VisionBundle({totalFrames: 10,width: 1000,height: 1000,@@ -464,7 +382,7 @@ describe("YoloVisionBundle", () => {backupKeypointsF1[6] = { x: 0.68, y: 0.2, visibility: 0.9 };backupKeypointsF1[10] = { x: 0.7, y: 0.35, visibility: 0.9 };- const poseResF0: YoloPoseResult = {+ const poseResF0: PoseResult = {people: [{trackId: 1,@@ -481,7 +399,7 @@ describe("YoloVisionBundle", () => {],};- const poseResF1: YoloPoseResult = {+ const poseResF1: PoseResult = {people: [{trackId: 1,diff --git a/packages/yolo/src/signals/yolo-signals-bundle.ts b/packages/vision/src/signals/vision-bundle.tssimilarity index 79%rename from packages/yolo/src/signals/yolo-signals-bundle.tsrename to packages/vision/src/signals/vision-bundle.tsindex b7cb4cbc..7bd9cf14 100644--- a/packages/yolo/src/signals/yolo-signals-bundle.ts+++ b/packages/vision/src/signals/vision-bundle.ts@@ -9,33 +9,34 @@ import {type TensorData,type TrackedObject,} from "@gitframes/core";-import { POSE_LANDMARKS_YOLO } from "../extractors/canonical-indices.js";+import { COCO_CLASSES } from "../model/registry.js";+import { COCO17_KEYPOINT_NAMES, COCO17_KEYPOINTS } from "../pose/keypoints.js";import {createNeutralLandmarks,createNeutralObjectResult,createNeutralPoseResult,} from "../runner/canonical-baselines.js";+import { selectSubjectMask } from "../segmentation/subject.js";import { SpatialLandmarkTransformer } from "../spatial/camera-space-transformer.js";import { getPersonBoundingBox } from "../tracking/pose-track-matcher.js";import { computeIoU } from "../tracking/temporal-object-tracker.js";import type {+ InstanceMask,LandmarkCoordinateSignals,- YoloClassifyResult,- YoloConfig,- YoloInstanceMask,- YoloOBBResult,- YoloPosePerson,- YoloPoseResult,- YoloSummary,+ PersonMatte,+ PosePerson,+ PoseResult,+ VisionConfig,+ VisionSummary,} from "../types.js";-export interface YoloVisionBundleOptions {+export interface VisionBundleOptions {readonly totalFrames?: number;readonly fps?: number;readonly width?: number;readonly height?: number;readonly cameraFov?: number;- readonly config?: YoloConfig;+ readonly config?: VisionConfig;}// ── Signal surface types ────────────────────────────────────────────────────@@ -81,8 +82,6 @@ export interface ObjectBoundingBoxSignals {readonly screenHeight: ProgrammaticSignal;readonly aspectRatio: ProgrammaticSignal;readonly area: ProgrammaticSignal;- /** Rotation of the matching OBB detection (radians, screen space); 0 when none. */- readonly angle: ProgrammaticSignal;}export interface ObjectAnchorsSignals {@@ -153,9 +152,18 @@ export interface SegmentationSignals {readonly humanSilhouette: MaskTrackSignals;readonly subject: MaskTrackSignals;readonly instanceMasks: MaskCollectionSignals;+ /** Person-vs-background alpha from the Selfie Segmenter (`enableMatte`). */+ readonly matte: PersonMatteSignals;readonly stencilTexture?: GPUTexture;}+export interface PersonMatteSignals {+ /** Mean person alpha over the frame, 0..1 (0 when no matte has run). */+ readonly coverage: ProgrammaticSignal;+ /** Frame-sized 0..255 alpha for the given frame, if a matte has run. */+ at(frame: number): PersonMatte | undefined;+}+export interface ClassSignals {readonly count: ProgrammaticSignal;readonly maxConfidence: ProgrammaticSignal;@@ -174,18 +182,6 @@ export interface ClassCollectionSignals {};}-export interface ClassificationSignals {- readonly top1: ProgrammaticSignal;- readonly top1Confidence: ProgrammaticSignal;- readonly top5: ReadonlyArray<{- readonly index: number;- readonly score: number;- }>;- readonly top5At: (- frame: number,- ) => ReadonlyArray<{ index: number; score: number }>;-}-export interface ObjectCollectionSignals {get(trackId: number): ObjectTrackSignals;byCategory(category: string, rank?: number): ObjectTrackSignals;@@ -198,13 +194,12 @@ export interface ObjectCollectionSignals {// ── The bundle ──────────────────────────────────────────────────────────────-export class YoloVisionBundle {+export class VisionBundle {public readonly objects: ObjectCollectionSignals;public readonly poseLandmarks: PoseLandmarkSignals;public readonly masks: MaskCollectionSignals;public readonly segmentation: SegmentationSignals;public readonly classes: ClassCollectionSignals;- public readonly classification: ClassificationSignals;public readonly poseLandmarksTensor: {get(ctx?: FrameContext): TensorData;@@ -231,7 +226,7 @@ export class YoloVisionBundle {private _transformer: SpatialLandmarkTransformer;private _stencilTexture?: GPUTexture;- private _poseCache = new Map<number, YoloPoseResult>();+ private _poseCache = new Map<number, PoseResult>();private _objectCache = new Map<number,{@@ -239,9 +234,8 @@ export class YoloVisionBundle {rawDetections: readonly DetectedObject[];}>();- private _maskCache = new Map<number, readonly YoloInstanceMask[]>();- private _classifyCache = new Map<number, YoloClassifyResult>();- private _obbCache = new Map<number, YoloOBBResult>();+ private _maskCache = new Map<number, readonly InstanceMask[]>();+ private _matteCache = new Map<number, PersonMatte>();private _poseLmCache = new Map<number, LandmarkCoordinateSignals>();private _trackCache = new Map<number, ObjectTrackSignals>();@@ -250,7 +244,7 @@ export class YoloVisionBundle {private _maskTrackCache = new Map<number, MaskTrackSignals>();private _registeredSignals = new Set<ProgrammaticSignal>();- constructor(options: YoloVisionBundleOptions = {}) {+ constructor(options: VisionBundleOptions = {}) {this._totalFrames = options.totalFrames ?? 120;this._fps = options.fps ?? 24;this._width = options.width ?? 1920;@@ -262,7 +256,7 @@ export class YoloVisionBundle {});// `self` avoids `this` contextual-typing issues inside object literals below.- const self: YoloVisionBundle = this;+ const self: VisionBundle = this;const programmaticSignal = (evaluator: (ctx: FrameContext) => number,@@ -318,7 +312,7 @@ export class YoloVisionBundle {if (!sig) {sig = createPoseCoord(index,- POSE_KEYPOINT_NAMES[index] ?? `kpt_${index}`,+ COCO17_KEYPOINT_NAMES[index] ?? `kpt_${index}`,);this._poseLmCache.set(index, sig);}@@ -326,24 +320,24 @@ export class YoloVisionBundle {};const poseNamed: PoseLandmarkSignals = {- nose: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.NOSE),- leftEye: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_EYE),- rightEye: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_EYE),- leftEar: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_EAR),- rightEar: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_EAR),- shoulder: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_SHOULDER),- leftShoulder: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_SHOULDER),- rightShoulder: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_SHOULDER),- leftElbow: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_ELBOW),- rightElbow: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_ELBOW),- leftWrist: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_WRIST),- rightWrist: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_WRIST),- leftHip: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_HIP),- rightHip: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_HIP),- leftKnee: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_KNEE),- rightKnee: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_KNEE),- leftAnkle: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.LEFT_ANKLE),- rightAnkle: getOrCreatePoseLm(POSE_LANDMARKS_YOLO.RIGHT_ANKLE),+ nose: getOrCreatePoseLm(COCO17_KEYPOINTS.NOSE),+ leftEye: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_EYE),+ rightEye: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_EYE),+ leftEar: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_EAR),+ rightEar: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_EAR),+ shoulder: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_SHOULDER),+ leftShoulder: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_SHOULDER),+ rightShoulder: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_SHOULDER),+ leftElbow: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_ELBOW),+ rightElbow: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_ELBOW),+ leftWrist: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_WRIST),+ rightWrist: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_WRIST),+ leftHip: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_HIP),+ rightHip: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_HIP),+ leftKnee: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_KNEE),+ rightKnee: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_KNEE),+ leftAnkle: getOrCreatePoseLm(COCO17_KEYPOINTS.LEFT_ANKLE),+ rightAnkle: getOrCreatePoseLm(COCO17_KEYPOINTS.RIGHT_ANKLE),get: getOrCreatePoseLm,};this.poseLandmarks = new Proxy(poseNamed, {@@ -358,7 +352,7 @@ export class YoloVisionBundle {},}) as PoseLandmarkSignals;- // ── Object track signals (contract identical to the MediaPipe bundle) ──+ // ── Object track signals ──const neutralTrack = (trackId: number,category = "unknown",@@ -420,7 +414,7 @@ export class YoloVisionBundle {label: string,fixedId?: number,): TrackPoseSignals => {- const resolvePerson = (frame: number): YoloPosePerson | undefined => {+ const resolvePerson = (frame: number): PosePerson | undefined => {const poseRes = self.getPoseResult(frame);if (poseRes.people.length === 0) return undefined;@@ -438,7 +432,7 @@ export class YoloVisionBundle {return poseRes.people[0];}- let bestPerson: YoloPosePerson | undefined;+ let bestPerson: PosePerson | undefined;let bestIoU = 0.1;for (const p of poseRes.people) {const pBox = getPersonBoundingBox(p);@@ -513,7 +507,7 @@ export class YoloVisionBundle {const getOrCreateCoord = (idx: number): LandmarkCoordinateSignals => {let c = coordsCache.get(idx);if (!c) {- c = createCoord(idx, POSE_KEYPOINT_NAMES[idx] ?? `kpt_${idx}`);+ c = createCoord(idx, COCO17_KEYPOINT_NAMES[idx] ?? `kpt_${idx}`);coordsCache.set(idx, c);}return c;@@ -528,8 +522,8 @@ export class YoloVisionBundle {const currKp = curr.keypoints;const prevKp = prev?.keypoints;- const rwCurr = currKp[POSE_LANDMARKS_YOLO.RIGHT_WRIST];- const rwPrev = prevKp?.[POSE_LANDMARKS_YOLO.RIGHT_WRIST];+ const rwCurr = currKp[COCO17_KEYPOINTS.RIGHT_WRIST];+ const rwPrev = prevKp?.[COCO17_KEYPOINTS.RIGHT_WRIST];let rwSpeed = 0;if (rwCurr &&@@ -542,8 +536,8 @@ export class YoloVisionBundle {rwSpeed = Math.sqrt(dx * dx + dy * dy) * this._fps;}- const lwCurr = currKp[POSE_LANDMARKS_YOLO.LEFT_WRIST];- const lwPrev = prevKp?.[POSE_LANDMARKS_YOLO.LEFT_WRIST];+ const lwCurr = currKp[COCO17_KEYPOINTS.LEFT_WRIST];+ const lwPrev = prevKp?.[COCO17_KEYPOINTS.LEFT_WRIST];let lwSpeed = 0;if (lwCurr &&@@ -566,10 +560,10 @@ export class YoloVisionBundle {const p = resolvePerson(ctx.frame);if (!p) return 0.0;const k = p.keypoints;- const lw = k[POSE_LANDMARKS_YOLO.LEFT_WRIST];- const ls = k[POSE_LANDMARKS_YOLO.LEFT_SHOULDER];- const rw = k[POSE_LANDMARKS_YOLO.RIGHT_WRIST];- const rs = k[POSE_LANDMARKS_YOLO.RIGHT_SHOULDER];+ const lw = k[COCO17_KEYPOINTS.LEFT_WRIST];+ const ls = k[COCO17_KEYPOINTS.LEFT_SHOULDER];+ const rw = k[COCO17_KEYPOINTS.RIGHT_WRIST];+ const rs = k[COCO17_KEYPOINTS.RIGHT_SHOULDER];const lRaised =lw &&@@ -593,10 +587,10 @@ export class YoloVisionBundle {const p = resolvePerson(ctx.frame);if (!p) return 0.0;const k = p.keypoints;- const ls = k[POSE_LANDMARKS_YOLO.LEFT_SHOULDER];- const rs = k[POSE_LANDMARKS_YOLO.RIGHT_SHOULDER];- const lh = k[POSE_LANDMARKS_YOLO.LEFT_HIP];- const rh = k[POSE_LANDMARKS_YOLO.RIGHT_HIP];+ const ls = k[COCO17_KEYPOINTS.LEFT_SHOULDER];+ const rs = k[COCO17_KEYPOINTS.RIGHT_SHOULDER];+ const lh = k[COCO17_KEYPOINTS.LEFT_HIP];+ const rh = k[COCO17_KEYPOINTS.RIGHT_HIP];if (!ls || !rs || !lh || !rh) return 0.0;const sx = ((ls.x + rs.x) / 2) * this._width;@@ -617,24 +611,24 @@ export class YoloVisionBundle {wristSpeed,handRaised,bodyTiltAngle,- nose: getOrCreateCoord(POSE_LANDMARKS_YOLO.NOSE),- leftEye: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_EYE),- rightEye: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_EYE),- leftEar: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_EAR),- rightEar: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_EAR),- shoulder: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_SHOULDER),- leftShoulder: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_SHOULDER),- rightShoulder: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_SHOULDER),- leftElbow: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_ELBOW),- rightElbow: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_ELBOW),- leftWrist: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_WRIST),- rightWrist: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_WRIST),- leftHip: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_HIP),- rightHip: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_HIP),- leftKnee: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_KNEE),- rightKnee: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_KNEE),- leftAnkle: getOrCreateCoord(POSE_LANDMARKS_YOLO.LEFT_ANKLE),- rightAnkle: getOrCreateCoord(POSE_LANDMARKS_YOLO.RIGHT_ANKLE),+ nose: getOrCreateCoord(COCO17_KEYPOINTS.NOSE),+ leftEye: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_EYE),+ rightEye: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_EYE),+ leftEar: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_EAR),+ rightEar: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_EAR),+ shoulder: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_SHOULDER),+ leftShoulder: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_SHOULDER),+ rightShoulder: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_SHOULDER),+ leftElbow: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_ELBOW),+ rightElbow: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_ELBOW),+ leftWrist: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_WRIST),+ rightWrist: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_WRIST),+ leftHip: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_HIP),+ rightHip: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_HIP),+ leftKnee: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_KNEE),+ rightKnee: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_KNEE),+ leftAnkle: getOrCreateCoord(COCO17_KEYPOINTS.LEFT_ANKLE),+ rightAnkle: getOrCreateCoord(COCO17_KEYPOINTS.RIGHT_ANKLE),get: getOrCreateCoord,};@@ -733,26 +727,6 @@ export class YoloVisionBundle {},{ fps: this._fps, label: `${label}_area` },),- angle: programmaticSignal(- (ctx) => {- const b = getBox(ctx.frame);- const tcx = b.originX + b.width / 2;- const tcy = b.originY + b.height / 2;- let bestAngle = 0;- let bestDist = Infinity;- for (const d of self.getObbResult(ctx.frame).detections) {- const cx = d.boundingBox.originX + d.boundingBox.width / 2;- const cy = d.boundingBox.originY + d.boundingBox.height / 2;- const dist = (cx - tcx) * (cx - tcx) + (cy - tcy) * (cy - tcy);- if (dist < bestDist) {- bestDist = dist;- bestAngle = d.boundingBox.angle;- }- }- return bestAngle;- },- { fps: this._fps, label: `${label}_angle` },- ),},anchors: {topLeft: anchor(@@ -1006,29 +980,14 @@ export class YoloVisionBundle {const maskForTrack = (frame: number,trackId: number,- ): YoloInstanceMask | undefined =>+ ): InstanceMask | undefined =>this.getMaskResult(frame).find((m) => m.trackId === trackId);- const largestMask = (frame: number): YoloInstanceMask | undefined => {- const masks = this.getMaskResult(frame);- if (masks.length === 0) return undefined;- let best = masks[0];- for (const m of masks) {- const bestIsPerson = best.category.toLowerCase() === "person";- const mIsPerson = m.category.toLowerCase() === "person";- if (mIsPerson && !bestIsPerson) {- best = m;- continue;- }- if (mIsPerson === bestIsPerson && m.area > best.area) {- best = m;- }- }- return best;- };+ const subjectMask = (frame: number): InstanceMask | undefined =>+ selectSubjectMask(this.getMaskResult(frame));const buildMaskTrackSignals = (- resolve: (frame: number) => YoloInstanceMask | undefined,+ resolve: (frame: number) => InstanceMask | undefined,label: string,): MaskTrackSignals => {const boxFor = (frame: number) => {@@ -1094,7 +1053,7 @@ export class YoloVisionBundle {}return sig;},- subject: buildMaskTrackSignals(largestMask, "subject"),+ subject: buildMaskTrackSignals(subjectMask, "subject"),count: programmaticSignal((ctx) =>new Set(this.getMaskResult(ctx.frame).map((m) => m.trackId ?? -1))@@ -1107,33 +1066,18 @@ export class YoloVisionBundle {humanSilhouette: this.masks.subject,subject: this.masks.subject,instanceMasks: this.masks,+ matte: {+ coverage: programmaticSignal(+ (ctx) => self.getMatteResult(ctx.frame)?.coverage ?? 0,+ { fps: this._fps, label: "matte_coverage" },+ ),+ at: (frame: number) => self.getMatteResult(frame),+ },get stencilTexture() {return self._stencilTexture;},};- // ── Classification signals ────────────────────────────────────────- this.classification = {- top1: programmaticSignal(- (ctx) => self.getClassifyResult(ctx.frame).top1,- {- fps: this._fps,- label: "classify_top1",- },- ),- top1Confidence: programmaticSignal(- (ctx) => self.getClassifyResult(ctx.frame).top1Score,- {- fps: this._fps,- label: "classify_top1_conf",- },- ),- get top5() {- return self.getClassifyResult(frameSignal.value).top5;- },- top5At: (frame: number) => self.getClassifyResult(frame).top5,- };-// ── Tensor views ──────────────────────────────────────────────────this.poseLandmarksTensor = {get: (ctx?: FrameContext) =>@@ -1202,12 +1146,12 @@ export class YoloVisionBundle {}}- public setPoseResult(frame: number, result: YoloPoseResult): void {+ public setPoseResult(frame: number, result: PoseResult): void {this._poseCache.set(this.clamp(frame), result);this.invalidateSignals();}- public getPoseResult(frame: number): YoloPoseResult {+ public getPoseResult(frame: number): PoseResult {return (this.latestAtOrBefore(this._poseCache, frame) ?? createNeutralPoseResult());@@ -1243,49 +1187,31 @@ export class YoloVisionBundle {);}- public setMaskResult(- frame: number,- masks: readonly YoloInstanceMask[],- ): void {+ public setMaskResult(frame: number, masks: readonly InstanceMask[]): void {this._maskCache.set(this.clamp(frame), masks);this.invalidateSignals();}- public getMaskResult(frame: number): readonly YoloInstanceMask[] {+ public getMaskResult(frame: number): readonly InstanceMask[] {return this.latestAtOrBefore(this._maskCache, frame) ?? [];}- public setClassifyResult(frame: number, result: YoloClassifyResult): void {- this._classifyCache.set(this.clamp(frame), result);- this.invalidateSignals();- }-- public getClassifyResult(frame: number): YoloClassifyResult {- return (- this.latestAtOrBefore(this._classifyCache, frame) ?? {- top1: 0,- top1Score: 0,- top5: [],- }- );- }-- public setObbResult(frame: number, result: YoloOBBResult): void {- this._obbCache.set(this.clamp(frame), result);+ public setMatteResult(frame: number, matte: PersonMatte): void {+ this._matteCache.set(this.clamp(frame), matte);this.invalidateSignals();}- public getObbResult(frame: number): YoloOBBResult {- return this.latestAtOrBefore(this._obbCache, frame) ?? { detections: [] };+ public getMatteResult(frame: number): PersonMatte | undefined {+ return this.latestAtOrBefore(this._matteCache, frame);}// ── Agent DX snapshot ─────────────────────────────────────────────────/**- * Serializable snapshot of one frame's object / class / mask signals (specs/yolov4plan.ts- * §Phase A). Synchronous — reads only the per-frame caches, never triggers inference.+ * Serializable snapshot of one frame's object / class / mask signals.+ * Synchronous — reads only the per-frame caches, never triggers inference.*/- public summary(frame: number = frameSignal.value): YoloSummary {+ public summary(frame: number = frameSignal.value): VisionSummary {const f = this.clamp(frame);const res = this.getObjectResult(f);@@ -1372,7 +1298,7 @@ export class YoloVisionBundle {const data = new Float32Array(nc);for (const o of this.getObjectResult(frame).objects) {if (o.active) {- const idx = YOLO_CLASS_INDEX_BY_NAME.get(o.category.toLowerCase());+ const idx = COCO_CLASS_INDEX_BY_NAME.get(o.category.toLowerCase());if (idx !== undefined) data[idx] += 1;}}@@ -1384,115 +1310,14 @@ export class YoloVisionBundle {}}-export function createYoloVisionBundle(- options?: YoloVisionBundleOptions,-): YoloVisionBundle {- return new YoloVisionBundle(options);+export function createVisionBundle(+ options?: VisionBundleOptions,+): VisionBundle {+ return new VisionBundle(options);}// ── Module-local constants ──────────────────────────────────────────────────-const POSE_KEYPOINT_NAMES = [- "nose",- "leftEye",- "rightEye",- "leftEar",- "rightEar",- "leftShoulder",- "rightShoulder",- "leftElbow",- "rightElbow",- "leftWrist",- "rightWrist",- "leftHip",- "rightHip",- "leftKnee",- "rightKnee",- "leftAnkle",- "rightAnkle",-] as const;--const YOLO_CLASS_INDEX_BY_NAME = new Map<string, number>(- [- "person",- "bicycle",- "car",- "motorcycle",- "airplane",- "bus",- "train",- "truck",- "boat",- "traffic light",- "fire hydrant",- "stop sign",- "parking meter",- "bench",- "bird",- "cat",- "dog",- "horse",- "sheep",- "cow",- "elephant",- "bear",- "zebra",- "giraffe",- "backpack",- "umbrella",- "handbag",- "tie",- "suitcase",- "frisbee",- "skis",- "snowboard",- "sports ball",- "kite",- "baseball bat",- "baseball glove",- "skateboard",- "surfboard",- "tennis racket",- "bottle",- "wine glass",- "cup",- "fork",- "knife",- "spoon",- "bowl",- "banana",- "apple",- "sandwich",- "orange",- "broccoli",- "carrot",- "hot dog",- "pizza",- "donut",- "cake",- "chair",- "couch",- "potted plant",- "bed",- "dining table",- "toilet",- "tv",- "laptop",- "mouse",- "remote",- "keyboard",- "cell phone",- "microwave",- "oven",- "toaster",- "sink",- "refrigerator",- "book",- "clock",- "vase",- "scissors",- "teddy bear",- "hair drier",- "toothbrush",- ].map((name, index) => [name, index]),+const COCO_CLASS_INDEX_BY_NAME = new Map<string, number>(+ COCO_CLASSES.map((name, index) => [name, index]),);diff --git a/packages/yolo/src/spatial/camera-space-transformer.ts b/packages/vision/src/spatial/camera-space-transformer.tssimilarity index 97%rename from packages/yolo/src/spatial/camera-space-transformer.tsrename to packages/vision/src/spatial/camera-space-transformer.tsindex 3bcc86da..1b8abc94 100644--- a/packages/yolo/src/spatial/camera-space-transformer.ts+++ b/packages/vision/src/spatial/camera-space-transformer.ts@@ -38,7 +38,7 @@ export class SpatialLandmarkTransformer {): ProjectedScreenCoordinate {const screenX = landmark.x * this.width;const screenY = landmark.y * this.height;- // MediaPipe normalized z is roughly scaled relative to width+ // Normalized landmark z is roughly scaled relative to widthconst rawZ = landmark.z * this.width + offsetZ;// Perspective foreshortening: S = d / (d - z)diff --git a/packages/yolo/src/spatial/spatial-pin.ts b/packages/vision/src/spatial/spatial-pin.tssimilarity index 97%rename from packages/yolo/src/spatial/spatial-pin.tsrename to packages/vision/src/spatial/spatial-pin.tsindex 2d119e53..9a5df229 100644--- a/packages/yolo/src/spatial/spatial-pin.ts+++ b/packages/vision/src/spatial/spatial-pin.ts@@ -1,4 +1,4 @@-import type { ObjectTrackSignals } from "../signals/yolo-signals-bundle.js";+import type { ObjectTrackSignals } from "../signals/vision-bundle.js";import type {LandmarkCoordinateSignals,PinToLandmarkOptions,diff --git a/packages/vision/src/test-support.ts b/packages/vision/src/test-support.tsnew file mode 100644index 00000000..3bef6262--- /dev/null+++ b/packages/vision/src/test-support.ts@@ -0,0 +1,119 @@+/**+ * Offline test doubles shared by the package's unit tests (not part of any build entry).+ *+ * `ScriptedProvider` hands out sessions that replay canned model outputs, so runner, node+ * and analysis tests exercise real preprocessing + decoding without ONNX Runtime or network.+ */++import { mkdtempSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { VisionModelStore } from "./model/model-store.js";+import type {+ SessionProvider,+ VisionSession,+ VisionTensorInput,+ VisionTensorOutputs,+} from "./runtime/session-provider.js";++export class ScriptedSession implements VisionSession {+ public inputs: VisionTensorInput[] = [];+ constructor(private readonly outputs: VisionTensorOutputs) {}+ async run(input: VisionTensorInput): Promise<VisionTensorOutputs> {+ this.inputs.push({+ name: input.name,+ data: input.data.slice(),+ dims: [...input.dims],+ });+ return this.outputs;+ }+ release(): void {}+}++/** Replays `script[i]` for the i-th session created (one per model). */+export class ScriptedProvider implements SessionProvider {+ public readonly kind = "node" as const;+ public created: ScriptedSession[] = [];+ private readonly script: ScriptedSession[];+ constructor(script: readonly VisionTensorOutputs[]) {+ this.script = script.map((outputs) => new ScriptedSession(outputs));+ }+ async createSession(): Promise<VisionSession> {+ const session = this.script[this.created.length] ?? new ScriptedSession({});+ this.created.push(session);+ return session;+ }+}++/** A store that serves fake bytes from a temp dir and records every download URL. */+export function fakeStore(): { store: VisionModelStore; fetches: string[] } {+ const fetches: string[] = [];+ const store = new VisionModelStore({+ modelsDir: mkdtempSync(join(tmpdir(), "vision-test-")),+ baseUrl: "https://example.test/models",+ verify: false,+ fetchImpl: (async (url: unknown) => {+ fetches.push(String(url));+ return new Response(new Uint8Array([1, 2, 3, 4]));+ }) as typeof fetch,+ });+ return { store, fetches };+}++export function grayImage(width = 640, height = 640) {+ return {+ data: new Uint8ClampedArray(width * height * 4).fill(128),+ width,+ height,+ };+}++/**+ * One COCO instance (default: person) at input box (220,170)–(420,470), score 0.9, with a+ * solid mask over input pixels 260..380 × 220..420.+ */+export function rtmdetInsOutput(+ opts: { label?: number; score?: number } = {},+): VisionTensorOutputs {+ const masks = new Float32Array(640 * 640);+ for (let y = 220; y < 420; y++) {+ for (let x = 260; x < 380; x++) masks[y * 640 + x] = 1;+ }+ return {+ dets: {+ data: new Float32Array([220, 170, 420, 470, opts.score ?? 0.9]),+ dims: [1, 1, 5],+ },+ labels: {+ data: new BigInt64Array([BigInt(opts.label ?? 0)]),+ dims: [1, 1],+ },+ masks: { data: masks, dims: [1, 1, 640, 640] },+ };+}++/** One person, box (220,170)–(420,470), nose at input (320, 200). */+export function rtmoOutput(score = 0.8): VisionTensorOutputs {+ const keypoints = new Float32Array(17 * 3);+ for (let k = 0; k < 17; k++) {+ keypoints[k * 3] = 320;+ keypoints[k * 3 + 1] = 200 + k * 15;+ keypoints[k * 3 + 2] = 0.9;+ }+ return {+ dets: {+ data: new Float32Array([220, 170, 420, 470, score]),+ dims: [1, 1, 5],+ },+ keypoints: { data: keypoints, dims: [1, 1, 17, 3] },+ };+}++/** Left half person (1), right half background (0). */+export function selfieOutput(width = 256, height = 256): VisionTensorOutputs {+ const alphas = new Float32Array(width * height);+ for (let y = 0; y < height; y++) {+ for (let x = 0; x < width / 2; x++) alphas[y * width + x] = 1;+ }+ return { alphas: { data: alphas, dims: [1, 1, height, width] } };+}diff --git a/packages/yolo/src/tracking/pose-track-matcher.test.ts b/packages/vision/src/tracking/pose-track-matcher.test.tssimilarity index 96%rename from packages/yolo/src/tracking/pose-track-matcher.test.tsrename to packages/vision/src/tracking/pose-track-matcher.test.tsindex a7ee07d7..c1815d57 100644--- a/packages/yolo/src/tracking/pose-track-matcher.test.ts+++ b/packages/vision/src/tracking/pose-track-matcher.test.ts@@ -1,6 +1,6 @@import type { TrackedObject } from "@gitframes/core";import { describe, expect, it } from "vitest";-import type { YoloPosePerson } from "../types.js";+import type { PosePerson } from "../types.js";import { matchPoseToTracks } from "./pose-track-matcher.js";function makePersonTrack(@@ -43,7 +43,7 @@ function makePosePerson(originY: number,width: number,height: number,-): YoloPosePerson {+): PosePerson {const keypoints = Array.from({ length: 17 }, () => ({x: (originX + width / 2) / 1000,y: (originY + height / 2) / 1000,@@ -99,7 +99,7 @@ describe("matchPoseToTracks", () => {it("preserves trackId on existing annotated people", () => {const trackA = makePersonTrack(1, 100, 100, 50, 50);- const poseA: YoloPosePerson = {+ const poseA: PosePerson = {...makePosePerson(100, 100, 50, 50),trackId: 999, // explicitly assigned};diff --git a/packages/yolo/src/tracking/pose-track-matcher.ts b/packages/vision/src/tracking/pose-track-matcher.tssimilarity index 89%rename from packages/yolo/src/tracking/pose-track-matcher.tsrename to packages/vision/src/tracking/pose-track-matcher.tsindex 9125c90a..4c35084b 100644--- a/packages/yolo/src/tracking/pose-track-matcher.ts+++ b/packages/vision/src/tracking/pose-track-matcher.ts@@ -1,10 +1,8 @@-import type { ObjectBoundingBox, TrackedObject } from "@gitframes/core";-import type { YoloPosePerson } from "../types.js";-import { computeIoU } from "./temporal-object-tracker.js";+import type { TrackedObject } from "@gitframes/core";+import type { PosePerson } from "../types.js";+import { computeIoU, type PixelBox } from "./temporal-object-tracker.js";-export function getPersonBoundingBox(- person: YoloPosePerson,-): ObjectBoundingBox {+export function getPersonBoundingBox(person: PosePerson): PixelBox {if (person.boundingBox &&person.boundingBox.width > 0 &&@@ -57,10 +55,10 @@ export function getPersonBoundingBox(* Annotates each person with the matched trackId.*/export function matchPoseToTracks(- people: readonly YoloPosePerson[],+ people: readonly PosePerson[],trackedObjects: readonly TrackedObject[],minIoU = 0.15,-): YoloPosePerson[] {+): PosePerson[] {if (people.length === 0) return [];const personTracks = trackedObjects.filter(@@ -85,7 +83,7 @@ export function matchPoseToTracks(const pBox = getPersonBoundingBox(person);const isNormalized = pBox.width <= 1.05 && pBox.height <= 1.05;for (const trk of personTracks) {- const targetBox: ObjectBoundingBox =+ const targetBox: PixelBox =isNormalized && trk.boundingBox.normalizedWidth > 0? {originX: trk.boundingBox.normalizedX,diff --git a/packages/yolo/src/tracking/temporal-object-tracker.ts b/packages/vision/src/tracking/temporal-object-tracker.tssimilarity index 98%rename from packages/yolo/src/tracking/temporal-object-tracker.tsrename to packages/vision/src/tracking/temporal-object-tracker.tsindex 5663f209..047ffba7 100644--- a/packages/yolo/src/tracking/temporal-object-tracker.ts+++ b/packages/vision/src/tracking/temporal-object-tracker.ts@@ -17,10 +17,13 @@ export interface TemporalTrackerOptions {readonly activeDuringCoast?: boolean;}-export function computeIoU(- boxA: ObjectBoundingBox,- boxB: ObjectBoundingBox,-): number {+/** Axis-aligned box in any consistent unit (pixels or normalized). */+export type PixelBox = Pick<+ ObjectBoundingBox,+ "originX" | "originY" | "width" | "height"+>;++export function computeIoU(boxA: PixelBox, boxB: PixelBox): number {const xA = Math.max(boxA.originX, boxB.originX);const yA = Math.max(boxA.originY, boxB.originY);const xB = Math.min(boxA.originX + boxA.width, boxB.originX + boxB.width);diff --git a/packages/yolo/src/types.ts b/packages/vision/src/types.tssimilarity index 55%rename from packages/yolo/src/types.tsrename to packages/vision/src/types.tsindex 9b5374e0..648526df 100644--- a/packages/yolo/src/types.ts+++ b/packages/vision/src/types.ts@@ -1,37 +1,20 @@export * from "@gitframes/core";-import type { DetectedObject } from "@gitframes/core";+import type { DetectedObject, ObjectBoundingBox } from "@gitframes/core";-export type YoloInputSource =+export type VisionInputSource =| string| { inputHandleId?: string; id?: string; [key: string]: unknown };-/** RGBA pixel buffer accepted by the runner (same shape MediaPipe consumed). */-export interface YoloImageInput {+/** RGBA pixel buffer accepted by the runner. */+export interface VisionImageInput {readonly data: Uint8ClampedArray | Uint8Array;readonly width: number;readonly height: number;}-/** A single decoded detection, in ORIGINAL source pixel space (positional), normalized included. */-export interface YoloDetection {- readonly category: string;- readonly classIndex: number;- readonly score: number;- readonly boundingBox: {- readonly originX: number;- readonly originY: number;- readonly width: number;- readonly height: number;- readonly normalizedX: number;- readonly normalizedY: number;- readonly normalizedWidth: number;- readonly normalizedHeight: number;- };-}-/** Instance segmentation mask aligned to the ORIGINAL source frame. */-export interface YoloInstanceMask {+export interface InstanceMask {readonly category: string;readonly mask: Uint8Array; // frame-sized, continuous alpha 0..255readonly width: number;@@ -44,65 +27,42 @@ export interface YoloInstanceMask {readonly trackId?: number;}-export interface YoloSegmentationResult {+export interface SegmentationResult {readonly detections: readonly DetectedObject[];- readonly masks: readonly YoloInstanceMask[];+ readonly masks: readonly InstanceMask[];}-export interface YoloPoseKeypoint {+/** Person-vs-background alpha (Selfie Segmenter), aligned to the ORIGINAL source frame. */+export interface PersonMatte {+ readonly mask: Uint8Array; // frame-sized, continuous alpha 0..255+ readonly width: number;+ readonly height: number;+ readonly coverage: number; // mean alpha, 0..1+}++export interface PoseKeypoint {readonly x: number;readonly y: number;readonly visibility: number; // 0..1}-export interface YoloPosePerson {+export interface PosePerson {readonly score: number;- readonly boundingBox: YoloDetection["boundingBox"];- /** 17 keypoints in COCO-17 order (see POSE_LANDMARKS_YOLO). */- readonly keypoints: readonly YoloPoseKeypoint[];+ readonly boundingBox: ObjectBoundingBox;+ /** 17 keypoints in COCO-17 order (see COCO17_KEYPOINTS). */+ readonly keypoints: readonly PoseKeypoint[];readonly trackId?: number;}-export interface YoloPoseResult {- readonly people: readonly YoloPosePerson[];-}--export interface YoloClassifyResult {- readonly top1: number; // class index (COCO 80)- readonly top1Score: number;- readonly top5: ReadonlyArray<{- readonly index: number;- readonly score: number;- }>;-}--export interface YoloConfig {- readonly enableDetection?: boolean;- readonly enableSegmentation?: boolean;- readonly enablePose?: boolean;- readonly enableClassification?: boolean;- readonly enableObb?: boolean;- readonly enableWorld?: boolean;- readonly prompts?: readonly string[];- readonly customModel?: import("@gitframes/core").CustomModelConfig;- readonly classes?: readonly string[]; // COCO names filter (all classes if omitted)- readonly confidence?: number; // default 0.25- readonly iouThreshold?: number; // NMS, default 0.45- readonly variant?: "n" | "s" | "m" | "l" | "x"; // default "n"- readonly imgsz?: number; // default 640- readonly delegate?: "cpu" | "webgpu";- readonly modelsDir?: string;- readonly baseUrl?: string;- readonly maskThreshold?: number;- readonly featherRadius?: number;- readonly autoDownload?: boolean; // compatibility no-op: models are ALWAYS lazy now+export interface PoseResult {+ readonly people: readonly PosePerson[];}/**- * Deterministic, serializable snapshot of one frame's signals (specs/yolov4plan.ts §Phase A).+ * Deterministic, serializable snapshot of one frame's signals.* Built synchronously from the per-frame caches, so it is safe to call inside a frame hook.*/-export interface YoloSummary {+export interface VisionSummary {readonly frame: number;readonly objects: ReadonlyArray<{readonly trackId: number;@@ -118,23 +78,7 @@ export interface YoloSummary {readonly masks: Readonly<Record<string, number>>;}-/** One rotated detection from the OBB head (specs/yolov4plan.ts §Phase B). */-export interface YoloOBBDetection {- readonly category: string;- readonly classIndex: number;- readonly score: number;- readonly boundingBox: YoloDetection["boundingBox"] & {- readonly angle: number;- };- /** Four rotated corners in source pixels, ordered TL, TR, BR, BL around the box. */- readonly corners: ReadonlyArray<readonly [number, number]>;-}--export interface YoloOBBResult {- readonly detections: readonly YoloOBBDetection[];-}--/** Per-track aggregate in an analysis report (specs/yolov4plan.ts §Phase C). */+/** Per-track aggregate in an analysis report. */export interface VisionAnalysisTrackSummary {readonly trackId: number;readonly category: string;diff --git a/packages/vision/src/vision-node.test.ts b/packages/vision/src/vision-node.test.tsnew file mode 100644index 00000000..91256091--- /dev/null+++ b/packages/vision/src/vision-node.test.ts@@ -0,0 +1,61 @@+import { describe, expect, it } from "vitest";+import { fakeStore, ScriptedProvider } from "./test-support.js";+import { enabledTasks, VisionNode } from "./vision-node.js";++describe("VisionNode (lazy + composition AST)", () => {+ it("constructor performs zero I/O and creates the signals bundle", () => {+ const node = new VisionNode("https://example.test/video.mp4", {+ enableDetection: true,+ enablePose: true,+ });+ expect(node.kind).toBe("vision");+ expect(node.config.enableDetection).toBe(true);+ expect(node.vision.objects).toBeDefined();+ expect(node.vision.poseLandmarks).toBeDefined();+ // runner not created until demanded+ expect(node["_runner"]).toBeUndefined();+ });++ it("toNode() + attach() produce a composition-ready AST with the same identity", () => {+ const node = new VisionNode("clip.mp4", { enableSegmentation: true });+ const ast = node.toNode();+ expect(ast.kind).toBe("vision");+ expect(ast.source).toBe("clip.mp4");+ expect(ast.config.enableSegmentation).toBe(true);++ const attached = VisionNode.attach("clip.mp4", {+ enableSegmentation: true,+ });+ expect(attached.node.kind).toBe("vision");+ expect(attached.node.source).toBe("clip.mp4");+ expect(typeof attached.ready).toBe("function");+ expect(typeof attached.runner).toBe("function");+ expect(typeof attached.close).toBe("function");+ });++ it("enabledTasks() defaults to detection", () => {+ expect(enabledTasks({})).toEqual(["detect"]);+ expect(enabledTasks({ enablePose: true, enableMatte: true })).toEqual([+ "pose",+ "matte",+ ]);+ });++ it("ready() preloads only the enabled tasks' models", async () => {+ const { store, fetches } = fakeStore();+ const provider = new ScriptedProvider([]);+ const node = new VisionNode(+ "clip.mp4",+ { enablePose: true, variant: "t" },+ {},+ { provider, store },+ );+ const runner = await node.ready();+ expect(fetches).toHaveLength(1); // only the pose model+ expect(provider.created).toHaveLength(1);+ expect(runner.downloadStatus.get("rtmo-t")).toBe("ready");+ expect(runner.downloadStatus.get("rtmdet-ins-t")).toBe("pending");+ node.close();+ expect(node["_runner"]).toBeUndefined();+ });+});diff --git a/packages/vision/src/vision-node.ts b/packages/vision/src/vision-node.tsnew file mode 100644index 00000000..cb86582c--- /dev/null+++ b/packages/vision/src/vision-node.ts@@ -0,0 +1,139 @@+import type { VisionConfig, VisionNodeSpec } from "@gitframes/core";+import type { VisionTask } from "./model/registry.js";+import {+ VisionRunner,+ type VisionRunnerOptions,+} from "./runner/vision-runner.js";+import {+ VisionBundle,+ type VisionBundleOptions,+} from "./signals/vision-bundle.js";+import type { VisionInputSource } from "./types.js";++/**+ * Bundle returned by `VisionNode.attach()`: the reactive signals surface plus the lazy lifecycle+ * helpers (all zero-I/O until `runner()` really needs a model).+ */+export interface VisionAttachedBundle extends VisionBundle {+ readonly node: VisionNodeSpec;+ /** Warm the enabled tasks' models/sessions ahead of first use. */+ ready(): Promise<VisionRunner>;+ /** Lazily-created runner (downloads happen on first real inference). */+ runner(): VisionRunner;+ /** Release this node's runner/sessions. */+ close(): void;+}++/** Tasks a config turns on; detection is the default when nothing is enabled explicitly. */+export function enabledTasks(config: VisionConfig): VisionTask[] {+ const tasks: VisionTask[] = [];+ if (config.enableDetection === true) tasks.push("detect");+ if (config.enableSegmentation === true) tasks.push("segment");+ if (config.enablePose === true) tasks.push("pose");+ if (config.enableMatte === true) tasks.push("matte");+ return tasks.length > 0 ? tasks : ["detect"];+}++/**+ * Vision node.+ *+ * LAZY BY CONSTRUCTION: creating a VisionNode performs ZERO I/O — no model downloads, no+ * sessions, no file probes. The first inference call (or an explicit `await vision.ready()`)+ * downloads the required models.+ */+export class VisionNode {+ public readonly id: string;+ public readonly kind = "vision" as const;+ public readonly source: VisionInputSource;+ public readonly config: VisionConfig;+ public readonly vision: VisionBundle;++ private _runner?: VisionRunner;++ /** Runner construction options (modelsDir, provider, store, …) — injectable for tests/agents. */+ private readonly _runnerOptions: VisionRunnerOptions;++ constructor(+ source: VisionInputSource,+ config: VisionConfig = {},+ bundleOptions: VisionBundleOptions = {},+ runnerOptions: VisionRunnerOptions = {},+ ) {+ this.id = `vision-${Math.random().toString(36).slice(2, 9)}`;+ this.source = source;+ this.config = config;+ this._runnerOptions = runnerOptions;+ this.vision = new VisionBundle({+ ...bundleOptions,+ config,+ });+ }++ /** Lazily-created runner; downloads happen on its first inference. */+ public runner(): VisionRunner {+ if (!this._runner) {+ const {+ variant,+ confidence,+ classes,+ modelsDir,+ baseUrl,+ maskThreshold,+ featherRadius,+ } = this.config;+ // Config values win over injected runner options, but only when set —+ // an explicit undefined must not clobber an injected modelsDir/provider.+ this._runner = VisionRunner.create({+ ...this._runnerOptions,+ ...(variant !== undefined ? { variant } : {}),+ ...(confidence !== undefined ? { confidence } : {}),+ ...(classes !== undefined ? { classes } : {}),+ ...(modelsDir !== undefined ? { modelsDir } : {}),+ ...(baseUrl !== undefined ? { baseUrl } : {}),+ ...(maskThreshold !== undefined ? { maskThreshold } : {}),+ ...(featherRadius !== undefined ? { featherRadius } : {}),+ });+ }+ return this._runner;+ }++ /** Explicit warm-up — downloads the models for the enabled tasks, ahead of first inference. */+ public async ready(): Promise<VisionRunner> {+ const runner = this.runner();+ await runner.preload(enabledTasks(this.config));+ return runner;+ }++ public close(): void {+ this._runner?.close();+ this._runner = undefined;+ }++ public static attach(+ source: VisionInputSource,+ config: VisionConfig = {},+ bundleOptions: VisionBundleOptions = {},+ ): VisionAttachedBundle {+ const node = new VisionNode(source, config, bundleOptions);+ const vision = node.vision;++ // AST node for composition graph inclusion, plus the lazy lifecycle helpers.+ Object.defineProperties(vision, {+ node: { value: node.toNode(), enumerable: true },+ ready: { value: () => node.ready() },+ runner: { value: () => node.runner() },+ close: { value: () => node.close() },+ });++ return vision as VisionAttachedBundle;+ }++ public toNode(): VisionNodeSpec {+ return {+ id: this.id,+ kind: "vision",+ source: this.source,+ config: this.config,+ };+ }+}diff --git a/packages/yolo/src/web.ts b/packages/vision/src/web.tssimilarity index 69%rename from packages/yolo/src/web.tsrename to packages/vision/src/web.tsindex 7cdc8d6c..cc9bd2d7 100644--- a/packages/yolo/src/web.ts+++ b/packages/vision/src/web.ts@@ -1,10 +1,10 @@/**- * Browser entry (`@gitframes/yolo/web`, specs/yolov4plan.ts §Phase E).+ * Browser entry (`@gitframes/vision/web`).** Re-exports the full engine plus the WebGPU session provider. Consumers pass a provider to* the runner explicitly (keeps the runner environment-agnostic and tree-shakeable):*- * const runner = YoloVisionRunner.create({ provider: createWebGPUProvider() });+ * const runner = VisionRunner.create({ provider: createWebGPUProvider() });** `onnxruntime-web` is an optional peer dependency, loaded lazily on the first session.*/diff --git a/packages/yolo/tsconfig.json b/packages/vision/tsconfig.jsonsimilarity index 100%rename from packages/yolo/tsconfig.jsonrename to packages/vision/tsconfig.jsondiff --git a/packages/yolo/tsdown.config.ts b/packages/vision/tsdown.config.tssimilarity index 100%rename from packages/yolo/tsdown.config.tsrename to packages/vision/tsdown.config.tsdiff --git a/packages/yolo/vitest.config.ts b/packages/vision/vitest.config.tssimilarity index 100%rename from packages/yolo/vitest.config.tsrename to packages/vision/vitest.config.tsdiff --git a/packages/yolo/src/analysis/analyze-sequence.test.ts b/packages/yolo/src/analysis/analyze-sequence.test.tsdeleted file mode 100644index adf96f47..00000000--- a/packages/yolo/src/analysis/analyze-sequence.test.ts+++ /dev/null@@ -1,166 +0,0 @@-import { mkdtempSync } from "node:fs";-import { tmpdir } from "node:os";-import { join } from "node:path";-import { describe, expect, it } from "vitest";-import { YoloVisionRunner } from "../runner/yolo-runner.js";-import type {- SessionProvider,- YoloSession,- YoloTensorInput,-} from "../runtime/session-provider.js";-import type { YoloImageInput } from "../types.js";-import { analyzeSequence } from "./analyze-sequence.js";--/** Session that shifts a single person detection right by 10px on every call. */-class MovingSession implements YoloSession {- private step = 0;- async run(- _input: YoloTensorInput,- ): Promise<Record<string, { data: Float32Array; dims: readonly number[] }>> {- const out = new Float32Array(5);- out[0] = 300 + this.step * 10;- out[1] = 320;- out[2] = 100;- out[3] = 200;- out[4] = 0.9;- this.step++;- return { output0: { data: out, dims: [1, 5, 1] } };- }- async release(): Promise<void> {}-}--class MovingProvider implements SessionProvider {- public readonly kind = "node" as const;- public created = 0;- private session?: MovingSession;- async createSession(): Promise<YoloSession> {- this.created++;- this.session ??= new MovingSession();- return this.session;- }-}--/** OBB-only session: one rotated detection at a fixed spot. */-class ObbSession implements YoloSession {- async run(- _input: YoloTensorInput,- ): Promise<Record<string, { data: Float32Array; dims: readonly number[] }>> {- const nc = 80;- const rows = 4 + nc + 1;- const out = new Float32Array(rows);- out[0] = 320;- out[1] = 320;- out[2] = 100;- out[3] = 100;- out[4] = 0.8;- out[4 + nc] = Math.PI / 8;- return { output0: { data: out, dims: [1, rows, 1] } };- }- async release(): Promise<void> {}-}--class ObbProvider implements SessionProvider {- public readonly kind = "node" as const;- async createSession(): Promise<YoloSession> {- return new ObbSession();- }-}--async function* frames(count: number): AsyncIterable<YoloImageInput> {- for (let i = 0; i < count; i++) {- yield {- data: new Uint8ClampedArray(640 * 640 * 4).fill(128),- width: 640,- height: 640,- };- }-}--function withFetch<T>(fn: () => Promise<T>): Promise<T> {- const orig = globalThis.fetch;- globalThis.fetch = (async () =>- new Response(new Uint8Array([1, 2, 3, 4]))) as typeof fetch;- return fn().finally(() => {- globalThis.fetch = orig;- });-}--const MODELS_DIR = (): string => mkdtempSync(join(tmpdir(), "yolo-analysis-"));--describe("analyzeSequence", () => {- it("produces a deterministic, zod-valid report for a moving object", async () => {- await withFetch(async () => {- const runner = YoloVisionRunner.create({- provider: new MovingProvider(),- modelsDir: MODELS_DIR(),- });- const report = await analyzeSequence(frames(12), {- runner,- source: "clip.mp4",- fps: 24,- tasks: ["detect"],- });-- expect(report.source).toBe("clip.mp4");- expect(report.totalFrames).toBe(12);- expect(report.durationSec).toBeCloseTo(0.5, 6);- expect(report.tasks).toEqual(["detect"]);- expect(report.modelDownloads.yolo11n.bytes).toBeGreaterThan(0);- expect(report.modelDownloads.yolo11n.ms).toBeGreaterThanOrEqual(0);- expect(report.inferenceMs).toBeGreaterThanOrEqual(0);-- expect(report.trackCount).toBe(1);- expect(report.classes.person).toMatchObject({- framesPresent: 12,- totalFrames: 12,- tracks: 1,- });- expect(report.classes.person.maxConfidence).toBeCloseTo(0.9, 5);-- const track = report.tracks[0];- expect(track.category).toBe("person");- expect(track.frames).toEqual([0, 11]);- expect(track.centerPath.length).toBeGreaterThan(0);- expect(Number.isFinite(track.speedMeanPxS)).toBe(true);-- // zod round-trip- const { VisionAnalysisReportSchema } = await import("../schemas.js");- expect(VisionAnalysisReportSchema.safeParse(report).success).toBe(true);- runner.close();- });- });-- it("runs only the requested task's model (I2)", async () => {- await withFetch(async () => {- const runner = YoloVisionRunner.create({- provider: new ObbProvider(),- modelsDir: MODELS_DIR(),- });- const report = await analyzeSequence(frames(3), {- runner,- tasks: ["obb"],- });- expect(report.tasks).toEqual(["obb"]);- expect(Object.keys(report.modelDownloads)).toEqual(["yolo11n-obb"]);- expect(runner.downloadStatus.get("yolo11n")).toBe("pending");- runner.close();- });- });-- it("respects the category filter", async () => {- await withFetch(async () => {- const runner = YoloVisionRunner.create({- provider: new MovingProvider(),- modelsDir: MODELS_DIR(),- });- const report = await analyzeSequence(frames(4), {- runner,- tasks: ["detect"],- categories: ["car"],- });- expect(report.trackCount).toBe(0);- expect(report.classes).toEqual({});- runner.close();- });- });-});diff --git a/packages/yolo/src/decode/classify.ts b/packages/yolo/src/decode/classify.tsdeleted file mode 100644index 94a6f770..00000000--- a/packages/yolo/src/decode/classify.ts+++ /dev/null@@ -1,64 +0,0 @@-import type { DetectedObject } from "@gitframes/core";-import type { YoloClassifyResult } from "../types.js";--/** Classification head: [1, nc] logits → softmax → top-5. */-export function decodeClassifyOutput(- output: Float32Array,- nc: number,-): YoloClassifyResult {- const count = Math.min(nc, output.length);-- // Numerically stable softmax- let max = -Infinity;- for (let i = 0; i < count; i++) {- if (output[i] > max) max = output[i];- }- let sum = 0;- const probs = new Float32Array(count);- for (let i = 0; i < count; i++) {- const p = Math.exp(output[i] - max);- probs[i] = p;- sum += p;- }- const inv = sum > 0 ? 1 / sum : 0;- for (let i = 0; i < count; i++) {- probs[i] *= inv;- }-- const top5: Array<{ index: number; score: number }> = [];- for (let i = 0; i < Math.min(5, probs.length); i++) {- let best = -1;- let bestScore = -1;- for (let c = 0; c < probs.length; c++) {- if (probs[c] > bestScore) {- bestScore = probs[c];- best = c;- }- }- if (best < 0) break;- top5.push({ index: best, score: bestScore });- probs[best] = -1;- }-- return {- top1: top5[0]?.index ?? 0,- top1Score: top5[0]?.score ?? 0,- top5,- };-}--/** Optional per-frame class histogram tensor for ML pipelines: [nc]. */-export function classHistogram(- detections: readonly { readonly classIndex: number }[],- nc: number,-): Float32Array {- const hist = new Float32Array(nc);- for (const d of detections) {- if (d.classIndex >= 0 && d.classIndex < nc) {- hist[d.classIndex] += 1;- }- }- return hist;-}--export type { DetectedObject };diff --git a/packages/yolo/src/decode/decode.test.ts b/packages/yolo/src/decode/decode.test.tsdeleted file mode 100644index b7b67f6b..00000000--- a/packages/yolo/src/decode/decode.test.ts+++ /dev/null@@ -1,450 +0,0 @@-import { describe, expect, it } from "vitest";-import { decodeClassifyOutput } from "./classify.js";-import { decodeDetectOutput } from "./detect.js";-import {- computeLetterbox,- letterboxToTensor,- unletterboxPoint,-} from "./letterbox.js";-import { computeIoU, type NmsBox, nms } from "./nms.js";-import { decodeObbOutput } from "./obb.js";-import { decodePoseOutput } from "./pose.js";-import { decodeSegmentOutput } from "./segment.js";--const COCO = [- "person",- "bicycle",- "car",- "motorcycle",- "airplane",- "bus",- "train",- "truck",- "boat",- "traffic light",- "fire hydrant",- "stop sign",- "parking meter",- "bench",- "bird",- "cat",- "dog",- "horse",- "sheep",- "cow",- "elephant",- "bear",- "zebra",- "giraffe",- "backpack",-];--describe("computeLetterbox", () => {- it("pads 1920x1080 to 640x640 (width-limited)", () => {- const p = computeLetterbox(1920, 1080, 640);- expect(p.scale).toBeCloseTo(640 / 1920, 6);- expect(p.inputWidth).toBe(640);- expect(p.inputHeight).toBe(360);- expect(p.dw).toBe(0);- expect(p.dh).toBe(140);- });-- it("pads 640x640 exactly (no padding)", () => {- const p = computeLetterbox(640, 640, 640);- expect(p.inputWidth).toBe(640);- expect(p.inputHeight).toBe(640);- expect(p.dw).toBe(0);- expect(p.dh).toBe(0);- });-- it("unletterboxes points back to source space", () => {- const p = computeLetterbox(1920, 1080, 640);- // center of the input square == center of the source- const { x, y } = unletterboxPoint(320, 320, p);- expect(x).toBeCloseTo(960, 6);- expect(y).toBeCloseTo(540, 6);- // top-left of the content area -> source origin- const tl = unletterboxPoint(0, 140, p);- expect(tl.x).toBeCloseTo(0, 6);- expect(tl.y).toBeCloseTo(0, 6);- });-});--describe("letterboxToTensor", () => {- it("writes NCHW planes with /255 values and 114 padding", () => {- const w = 2;- const h = 2;- const data = new Uint8ClampedArray([- 255, 0, 0, 255, 0, 255, 0, 255, 0, 0, 255, 255, 255, 255, 255, 255,- ]);- const { tensor, params } = letterboxToTensor(- { data, width: w, height: h },- 320,- );- expect(params.dw).toBe(0);- expect(params.dh).toBe(0);- expect(tensor.length).toBe(3 * 320 * 320);- // channel 0 (R): source pixel (0,0) = 255 -> 1.0- expect(tensor[0]).toBe(1.0);- // input x=160 maps exactly onto source x=1 (scale = 160)- // channel 1 (G) there = 255 -> 1.0- expect(tensor[1 * 320 * 320 + 160]).toBe(1.0);- // input y=160 maps onto source y=1 — channel 2 (B) = 255 -> 1.0- expect(tensor[2 * 320 * 320 + 160 * 320]).toBe(1.0);- // bottom-right corner samples source (1,1) = white -> 1.0- expect(tensor[319 * 320 + 319]).toBe(1.0);- });-- it("fills padding with 114/255 for non-square input", () => {- const data = new Uint8ClampedArray(4 * 2 * 4).fill(0);- const { tensor, params } = letterboxToTensor(- { data, width: 4, height: 2 },- 8,- );- expect(params.dw).toBe(0);- expect(params.dh).toBe(2);- // any content row is 0 (black source)- expect(tensor[2 * 8 + 2]).toBe(0);- // padded row (y < 2) == 114/255- expect(tensor[0]).toBeCloseTo(114 / 255, 5);- expect(tensor[1 * 8 + 0]).toBeCloseTo(114 / 255, 5);- });-});--describe("nms", () => {- const box = (- x0: number,- y0: number,- x1: number,- y1: number,- score: number,- classIndex: number,- ): NmsBox => ({- x0,- y0,- x1,- y1,- score,- classIndex,- anchorIndex: 0,- });-- it("keeps the highest-scoring overlapping box", () => {- const boxes = [- box(0, 0, 100, 100, 0.9, 0),- box(10, 10, 110, 110, 0.8, 0),- box(200, 200, 300, 300, 0.95, 0),- ];- const kept = nms(boxes, 0.45);- expect(kept).toHaveLength(2);- expect(kept[0].score).toBe(0.95);- expect(kept[1].score).toBe(0.9);- });-- it("does not suppress across classes (class-aware)", () => {- const boxes = [box(0, 0, 100, 100, 0.9, 0), box(10, 10, 110, 110, 0.8, 1)];- expect(nms(boxes, 0.45)).toHaveLength(2);- });-- it("computeIoU", () => {- expect(- computeIoU(box(0, 0, 100, 100, 1, 0), box(0, 0, 100, 100, 1, 0)),- ).toBe(1);- expect(- computeIoU(box(0, 0, 100, 100, 1, 0), box(200, 200, 300, 300, 1, 0)),- ).toBe(0);- expect(- computeIoU(box(0, 0, 100, 100, 1, 0), box(50, 50, 150, 150, 1, 0)),- ).toBeCloseTo(1 / 7, 6);- });-});--describe("decodeDetectOutput", () => {- const anchors = 2;- const nc = 4;- const params = computeLetterbox(100, 100, 640);- // anchor 0: box (320, 320, 200, 150), best class 0 (person) @ 0.9- // anchor 1: box (500, 400, 60, 40), best class 2 (car) @ 0.7 — low IoU w/ anchor 0? no: separate- const output = new Float32Array((4 + nc) * anchors);- output[0] = 320;- output[anchors] = 320;- output[2 * anchors] = 200;- output[3 * anchors] = 150;- output[(4 + 0) * anchors] = 0.9;- output[(4 + 1) * anchors] = 0.1;- output[1] = 500;- output[anchors + 1] = 400;- output[2 * anchors + 1] = 60;- output[3 * anchors + 1] = 40;- output[(4 + 2) * anchors + 1] = 0.7;- output[(4 + 3) * anchors + 1] = 0.05;-- it("decodes to NMS'd DetectedObjects in source pixel space", () => {- const res = decodeDetectOutput(output, anchors, nc, {- confidence: 0.25,- iouThreshold: 0.45,- classNames: COCO.slice(0, nc),- params,- sourceWidth: 100,- sourceHeight: 100,- });- expect(res).toHaveLength(2);- const [person, car] = res;- expect(person.category).toBe("person");- expect(person.score).toBeCloseTo(0.9, 6);- // input center (320,320) is the source center (50,50); box 200x150 -> source 31.25x23.4375- expect(person.boundingBox.originX).toBeCloseTo(- 50 - 200 / 2 / (640 / 100),- 5,- );- expect(person.boundingBox.originY).toBeCloseTo(- 50 - 150 / 2 / (640 / 100),- 5,- );- expect(person.boundingBox.width).toBeCloseTo(200 / 6.4, 5);- expect(person.boundingBox.height).toBeCloseTo(150 / 6.4, 5);- expect(person.boundingBox.normalizedX).toBeCloseTo(- person.boundingBox.originX / 100,- 5,- );- expect(car.category).toBe("car");- });-- it("applies the class-name filter", () => {- const res = decodeDetectOutput(output, anchors, nc, {- confidence: 0.25,- iouThreshold: 0.45,- classes: ["car"],- classNames: COCO.slice(0, nc),- params,- sourceWidth: 100,- sourceHeight: 100,- });- expect(res).toHaveLength(1);- expect(res[0].category).toBe("car");- });-- it("suppresses a duplicate prediction overlapping the same object", () => {- const dup = new Float32Array((4 + nc) * anchors);- dup[0] = 320;- dup[anchors] = 320;- dup[2 * anchors] = 200;- dup[3 * anchors] = 150;- dup[(4 + 0) * anchors] = 0.9;- dup[1] = 330;- dup[anchors + 1] = 330;- dup[2 * anchors + 1] = 190;- dup[3 * anchors + 1] = 140;- dup[(4 + 0) * anchors + 1] = 0.88;- const res = decodeDetectOutput(dup, anchors, nc, {- confidence: 0.25,- iouThreshold: 0.45,- classNames: COCO.slice(0, nc),- params,- sourceWidth: 100,- sourceHeight: 100,- });- expect(res).toHaveLength(1);- expect(res[0].score).toBeCloseTo(0.9, 6);- });-});--describe("decodePoseOutput", () => {- it("decodes a person with 17 keypoints, unletterboxed", () => {- const anchors = 1;- const rows = 4 + 1 + 17 * 3;- const out = new Float32Array(rows * anchors);- out[0] = 320;- out[1 * anchors] = 320;- out[2 * anchors] = 200;- out[3 * anchors] = 300;- out[4 * anchors] = 0.85; // confidence- // nose keypoint at input (320, 300)- out[(5 + 0 * 3) * anchors] = 320;- out[(6 + 0 * 3) * anchors] = 300;- out[(7 + 0 * 3) * anchors] = 0.95;- // left shoulder at input (280, 360)- out[(5 + 5 * 3) * anchors] = 280;- out[(6 + 5 * 3) * anchors] = 360;- out[(7 + 5 * 3) * anchors] = 0.9;-- const params = computeLetterbox(100, 100, 640);- const res = decodePoseOutput(out, anchors, {- confidence: 0.25,- params,- sourceWidth: 100,- sourceHeight: 100,- });- expect(res.people).toHaveLength(1);- const person = res.people[0];- expect(person.score).toBeCloseTo(0.85, 6);- expect(person.keypoints).toHaveLength(17);- // nose: input (320,300) -> source (50, 300/6.4)- expect(person.keypoints[0].x).toBeCloseTo(50, 5);- expect(person.keypoints[0].y).toBeCloseTo(300 / 6.4, 5);- expect(person.keypoints[0].visibility).toBeCloseTo(0.95, 5);- // left shoulder index 5- expect(person.keypoints[5].x).toBeCloseTo(280 / 6.4, 5);- expect(person.keypoints[5].y).toBeCloseTo(360 / 6.4, 5);- });-- it("drops anchors below the confidence threshold", () => {- const anchors = 2;- const rows = 4 + 1 + 17 * 3;- const out = new Float32Array(rows * anchors);- out[0] = 320;- out[1 * anchors] = 320;- out[2 * anchors] = 100;- out[3 * anchors] = 100;- out[4 * anchors] = 0.9;- out[1] = 200;- out[anchors + 1] = 200;- out[2 * anchors + 1] = 50;- out[3 * anchors + 1] = 50;- out[4 * anchors + 1] = 0.05; // below threshold- const res = decodePoseOutput(out, anchors, {- confidence: 0.25,- params: computeLetterbox(100, 100, 640),- sourceWidth: 100,- sourceHeight: 100,- });- expect(res.people).toHaveLength(1);- });-});--describe("decodeClassifyOutput", () => {- it("softmaxes and returns top-5", () => {- const out = new Float32Array(5);- out[0] = 2;- out[1] = 1;- out[2] = 0;- out[3] = 3;- out[4] = -1;- const res = decodeClassifyOutput(out, 5);- expect(res.top1).toBe(3);- expect(res.top5).toHaveLength(5);- expect(res.top5[0].index).toBe(3);- expect(res.top1Score).toBeGreaterThan(0.5);- // sum of softmax == 1- const sum = res.top5.reduce((acc, x) => acc + x.score, 0);- expect(sum).toBeCloseTo(1, 5);- });-});--describe("decodeObbOutput", () => {- it("projects rotated corners and the axis-aligned extent into source space", () => {- const anchors = 2;- const nc = 4;- const angleRow = 4 + nc;- const out = new Float32Array((5 + nc) * anchors);- // anchor 0: center (320,320), 100x40, 90°, class 0 @ 0.9- out[0] = 320;- out[anchors] = 320;- out[2 * anchors] = 100;- out[3 * anchors] = 40;- out[(4 + 0) * anchors] = 0.9;- out[angleRow * anchors] = Math.PI / 2;- // anchor 1: below threshold- out[1] = 320;- out[anchors + 1] = 320;- out[2 * anchors + 1] = 100;- out[3 * anchors + 1] = 40;- out[(4 + 0) * anchors + 1] = 0.05;- out[angleRow * anchors + 1] = 0;-- const params = computeLetterbox(100, 100, 640);- const res = decodeObbOutput(out, anchors, nc, {- confidence: 0.25,- iouThreshold: 0.45,- classNames: COCO.slice(0, nc),- params,- sourceWidth: 100,- sourceHeight: 100,- });-- expect(res.detections).toHaveLength(1);- const det = res.detections[0];- expect(det.category).toBe("person");- expect(det.boundingBox.angle).toBeCloseTo(Math.PI / 2, 6);- expect(det.corners).toHaveLength(4);- // w=100,h=40 at scale 6.4 -> 15.625 x 6.25; rotated 90° around (50,50)- expect(det.boundingBox.originX).toBeCloseTo(50 - 6.25 / 2, 4);- expect(det.boundingBox.originY).toBeCloseTo(50 - 15.625 / 2, 4);- expect(det.boundingBox.width).toBeCloseTo(6.25, 4);- expect(det.boundingBox.height).toBeCloseTo(15.625, 4);- });-- it("honours an overridden angle row for third-party exports", () => {- const anchors = 1;- const nc = 2;- const out = new Float32Array(6 * anchors);- out[0] = 320;- out[anchors] = 320;- out[2 * anchors] = 64;- out[3 * anchors] = 64;- out[(4 + 0) * anchors] = 0.9;- out[4 * anchors] = 0.5; // angle stored at row 4 instead of 4+nc- const res = decodeObbOutput(out, anchors, nc, {- confidence: 0.25,- iouThreshold: 0.45,- classNames: ["person", "car"],- params: computeLetterbox(100, 100, 640),- sourceWidth: 100,- sourceHeight: 100,- angleRow: 4,- });- expect(res.detections[0].boundingBox.angle).toBeCloseTo(0.5, 6);- });-});--describe("decodeSegmentOutput", () => {- it("produces continuous anti-aliased soft alpha values (0..255)", () => {- const anchors = 1;- const nc = 1;- const protoDim = 4;- const rows = 4 + nc + 32;- const output = new Float32Array(rows * anchors);- // Box at center 320, 320, w=200, h=200, class 0 score 0.9- output[0] = 320;- output[1 * anchors] = 320;- output[2 * anchors] = 200;- output[3 * anchors] = 200;- output[4 * anchors] = 0.9;- // Coefficient for channel 0 = 1.0, rest 0- output[(4 + nc) * anchors] = 1.0;-- // Proto tensor: 32 channels of 4x4.- // Channel 0 has a gradient so sigmoid outputs values in transition band- const proto = new Float32Array(32 * protoDim * protoDim);- // Let logit(0.5) = 0. Center proto values around 0 with gradient -0.5 to +0.5- for (let y = 0; y < protoDim; y++) {- for (let x = 0; x < protoDim; x++) {- proto[y * protoDim + x] = (x - 1.5) * 0.4;- }- }-- const params = computeLetterbox(64, 64, 640);- const res = decodeSegmentOutput(output, proto, anchors, nc, protoDim, {- confidence: 0.25,- iouThreshold: 0.45,- classNames: ["person"],- params,- sourceWidth: 64,- sourceHeight: 64,- maskThreshold: 0.5,- featherRadius: 0.1,- });-- expect(res.masks).toHaveLength(1);- const mask = res.masks[0].mask;- // Check that mask has intermediate values between 0 and 255 (anti-aliased soft alpha)- let hasIntermediate = false;- for (let i = 0; i < mask.length; i++) {- if (mask[i] > 0 && mask[i] < 255) {- hasIntermediate = true;- break;- }- }- expect(hasIntermediate).toBe(true);- });-});diff --git a/packages/yolo/src/decode/detect.ts b/packages/yolo/src/decode/detect.tsdeleted file mode 100644index caeb660f..00000000--- a/packages/yolo/src/decode/detect.ts+++ /dev/null@@ -1,116 +0,0 @@-import type { DetectedObject } from "@gitframes/core";-import type { LetterboxParams } from "./letterbox.js";-import { type NmsBox, nms } from "./nms.js";--/**- * Shared YOLO head decode core.- *- * The ultralytics ONNX export bakes the decode into the graph:- * - box rows are in INPUT pixel space (0..imgsz, letterboxed), xywh format- * - class rows are already sigmoided (probabilities, 0..1)- * - layout per anchor: [cx, cy, w, h, cls_0..cls_{nc-1}], column-major over anchors- */-export interface DetectDecodeOptions {- readonly confidence: number; // default 0.25- readonly iouThreshold: number; // default 0.45- readonly classes?: readonly DetectedObject["category"][]; // COCO name filter- readonly classNames: readonly string[];- readonly params: LetterboxParams;- readonly sourceWidth: number;- readonly sourceHeight: number;- /** Sigmoid threshold for instance-mask pixels. Lower → fuller, blobbier masks. Default 0.5. */- readonly maskThreshold?: number;- /** Transition band radius around maskThreshold for anti-aliased soft alpha. Default 0.05. */- readonly featherRadius?: number;-}--export function decodeYoloBoxes(- output: Float32Array,- anchors: number,- nc: number,- opts: DetectDecodeOptions,-): NmsBox[] {- const candidates: NmsBox[] = [];- const classFilter = opts.classes- ? new Set(opts.classes.map((c) => c.toLowerCase()))- : null;-- for (let a = 0; a < anchors; a++) {- let bestScore = -1;- let bestClass = -1;- for (let c = 0; c < nc; c++) {- const score = output[(4 + c) * anchors + a];- if (score > bestScore) {- bestScore = score;- bestClass = c;- }- }- if (bestScore < opts.confidence) continue;- if (- classFilter &&- !classFilter.has((opts.classNames[bestClass] ?? "").toLowerCase())- ) {- continue;- }- const cx = output[a];- const cy = output[anchors + a];- const w = output[2 * anchors + a];- const h = output[3 * anchors + a];- if (w <= 0 || h <= 0) continue;- const halfW = w / 2;- const halfH = h / 2;- candidates.push({- x0: cx - halfW,- y0: cy - halfH,- x1: cx + halfW,- y1: cy + halfH,- score: bestScore,- classIndex: bestClass,- anchorIndex: a,- });- }-- return nms(candidates, opts.iouThreshold);-}--/** Maps an NMS-kept box from input space to a core `DetectedObject` in source pixel space. */-export function toDetectedObject(- box: NmsBox,- opts: DetectDecodeOptions,-): DetectedObject {- const { scale, dw, dh } = opts.params;- const { sourceWidth: sw, sourceHeight: sh } = opts;-- const x0 = Math.max(0, Math.min(sw, (box.x0 - dw) / scale));- const y0 = Math.max(0, Math.min(sh, (box.y0 - dh) / scale));- const x1 = Math.max(0, Math.min(sw, (box.x1 - dw) / scale));- const y1 = Math.max(0, Math.min(sh, (box.y1 - dh) / scale));- const width = Math.max(0, x1 - x0);- const height = Math.max(0, y1 - y0);-- return {- category: opts.classNames[box.classIndex] ?? `class_${box.classIndex}`,- score: box.score,- boundingBox: {- originX: x0,- originY: y0,- width,- height,- normalizedX: sw > 0 ? x0 / sw : 0,- normalizedY: sh > 0 ? y0 / sh : 0,- normalizedWidth: sw > 0 ? width / sw : 0,- normalizedHeight: sh > 0 ? height / sh : 0,- },- };-}--/** Full detect-head decode: hidden state → DetectedObject[] (already NMS'd, source pixels). */-export function decodeDetectOutput(- output: Float32Array,- anchors: number,- nc: number,- opts: DetectDecodeOptions,-): DetectedObject[] {- const kept = decodeYoloBoxes(output, anchors, nc, opts);- return kept.map((box) => toDetectedObject(box, opts));-}diff --git a/packages/yolo/src/decode/letterbox.ts b/packages/yolo/src/decode/letterbox.tsdeleted file mode 100644index 7bfdb8b4..00000000--- a/packages/yolo/src/decode/letterbox.ts+++ /dev/null@@ -1,137 +0,0 @@-/**- * Letterbox preprocessing + inverse mapping.- *- * The YOLO ONNX export consumes a square RGB float tensor (imgsz × imgsz), values / 255,- * gray (114) padding on the short dimension — the "letterbox". Box/keypoint outputs are- * in input-pixel space; this module owns the resize and the mapping back to source pixels.- */--export const YOLO_LETTERBOX_PAD_VALUE = 114;--export interface LetterboxParams {- readonly scale: number;- readonly dw: number; // horizontal padding (input px)- readonly dh: number; // vertical padding (input px)- readonly inputWidth: number; // scaled (unpadded) content size- readonly inputHeight: number;- readonly size: number; // imgsz-}--export interface RgbaImage {- readonly data: Uint8ClampedArray | Uint8Array;- readonly width: number;- readonly height: number;-}--export function computeLetterbox(- srcW: number,- srcH: number,- imgsz: number,-): LetterboxParams {- const scale = Math.min(imgsz / srcW, imgsz / srcH);- const inputWidth = Math.round(srcW * scale);- const inputHeight = Math.round(srcH * scale);- return {- scale,- dw: (imgsz - inputWidth) / 2,- dh: (imgsz - inputHeight) / 2,- inputWidth,- inputHeight,- size: imgsz,- };-}--/** Map decoded input-space coordinates back to source pixel space. */-export function unletterboxPoint(- x: number,- y: number,- params: LetterboxParams,-): { x: number; y: number } {- return {- x: (x - params.dw) / params.scale,- y: (y - params.dh) / params.scale,- };-}--/**- * Resizes RGBA -> NCHW float32 tensor with bilinear sampling + gray padding.- * Layout: [1, 3, imgsz, imgsz], channel stride = imgsz * imgsz.- * Reuses `out` / `outData` when provided (hot-path frame loop).- */-export function letterboxToTensor(- image: RgbaImage,- imgsz = 640,- out?: Float32Array,-): { tensor: Float32Array; params: LetterboxParams } {- const { width: srcW, height: srcH } = image;- const params = computeLetterbox(srcW, srcH, imgsz);- const { scale, dw, dh, inputWidth, inputHeight } = params;-- const channelStride = imgsz * imgsz;- if (!out || out.length < 3 * channelStride) {- out = new Float32Array(3 * channelStride);- } else {- out.fill(0, 0, 3 * channelStride);- }-- const src = image.data;- const pad = YOLO_LETTERBOX_PAD_VALUE / 255;-- const x0 = Math.ceil(dw);- const y0 = Math.ceil(dh);- const x1 = Math.floor(dw + inputWidth) - 1;- const y1 = Math.floor(dh + inputHeight) - 1;-- const rPlane = out.subarray(0, channelStride);- const gPlane = out.subarray(channelStride, 2 * channelStride);- const bPlane = out.subarray(2 * channelStride, 3 * channelStride);-- for (let y = 0; y < imgsz; y++) {- const rowBase = y * imgsz;- const paddedRow = y < y0 || y > y1;- for (let x = 0; x < imgsz; x++) {- let r = pad;- let g = pad;- let b = pad;- if (!paddedRow && x >= x0 && x <= x1) {- // Bilinear sample from the source at (sx, sy) in source pixels- const sx = (x - dw) / scale;- const sy = (y - dh) / scale;- const x0s = Math.floor(sx);- const y0s = Math.floor(sy);- const fx = sx - x0s;- const fy = sy - y0s;- const x1s = Math.min(srcW - 1, x0s + 1);- const y1s = Math.min(srcH - 1, y0s + 1);- const i00 = (y0s * srcW + x0s) * 4;- const i10 = (y0s * srcW + x1s) * 4;- const i01 = (y1s * srcW + x0s) * 4;- const i11 = (y1s * srcW + x1s) * 4;- const w00 = (1 - fx) * (1 - fy);- const w10 = fx * (1 - fy);- const w01 = (1 - fx) * fy;- const w11 = fx * fy;- r =- (src[i00] * w00 + src[i10] * w10 + src[i01] * w01 + src[i11] * w11) /- 255;- g =- (src[i00 + 1] * w00 +- src[i10 + 1] * w10 +- src[i01 + 1] * w01 +- src[i11 + 1] * w11) /- 255;- b =- (src[i00 + 2] * w00 +- src[i10 + 2] * w10 +- src[i01 + 2] * w01 +- src[i11 + 2] * w11) /- 255;- }- rPlane[rowBase + x] = r;- gPlane[rowBase + x] = g;- bPlane[rowBase + x] = b;- }- }-- return { tensor: out, params };-}diff --git a/packages/yolo/src/decode/nms.ts b/packages/yolo/src/decode/nms.tsdeleted file mode 100644index c79e08b7..00000000--- a/packages/yolo/src/decode/nms.ts+++ /dev/null@@ -1,53 +0,0 @@-/**- * Class-aware Non-Maximum Suppression over decoded boxes.- * Pure typed-array implementation — fast enough at 8400 anchors (< 1 ms) that it stays- * on the CPU; no GPU round-trip needed.- */--export interface NmsBox {- readonly x0: number;- readonly y0: number;- readonly x1: number;- readonly y1: number;- readonly score: number;- readonly classIndex: number;- readonly anchorIndex: number;-}--export function computeIoU(a: NmsBox, b: NmsBox): number {- const interX0 = Math.max(a.x0, b.x0);- const interY0 = Math.max(a.y0, b.y0);- const interX1 = Math.min(a.x1, b.x1);- const interY1 = Math.min(a.y1, b.y1);- const interW = Math.max(0, interX1 - interX0);- const interH = Math.max(0, interY1 - interY0);- const interArea = interW * interH;- if (interArea <= 0) return 0;- const areaA = (a.x1 - a.x0) * (a.y1 - a.y0);- const areaB = (b.x1 - b.x0) * (b.y1 - b.y0);- const unionArea = areaA + areaB - interArea;- return unionArea > 0 ? interArea / unionArea : 0;-}--/**- * Greedy class-aware NMS: boxes of different classes never suppress each other- * (matches ultralytics default behavior).- */-export function nms(boxes: readonly NmsBox[], iouThreshold = 0.45): NmsBox[] {- const sorted = [...boxes].sort((a, b) => b.score - a.score);- const kept: NmsBox[] = [];- for (const candidate of sorted) {- let suppressed = false;- for (const k of kept) {- if (k.classIndex !== candidate.classIndex) continue;- if (computeIoU(k, candidate) > iouThreshold) {- suppressed = true;- break;- }- }- if (!suppressed) {- kept.push(candidate);- }- }- return kept;-}diff --git a/packages/yolo/src/decode/obb.ts b/packages/yolo/src/decode/obb.tsdeleted file mode 100644index 313e6f83..00000000--- a/packages/yolo/src/decode/obb.ts+++ /dev/null@@ -1,165 +0,0 @@-import type { YoloOBBDetection, YoloOBBResult } from "../types.js";-import type { DetectDecodeOptions } from "./detect.js";-import { type NmsBox, nms } from "./nms.js";--/**- * Oriented-bounding-box decode.- *- * Head: `[1, 5 + nc, anchors]` — rows 0..3 are the rotated box (cx, cy, w, h) in INPUT pixel- * space, rows 4..4+nc-1 are sigmoided class scores, and the final row `4 + nc` is the box- * angle in radians (the ultralytics ONNX export folds the DFL/dist2rbox/angle transform into- * the graph, exactly like the detect head folds DFL + sigmoid).- *- * The angle row index is overridable (`angleRow`) because third-party exports have shipped- * both orderings; the default matches ultralytics. Rotated-IoU NMS is deliberately NOT used- * in v1 — the repository's class-aware axis-aligned NMS runs on the rotated box's extents,- * which is fast and adequate for HUD/annotation use.- */--export interface ObbDecodeOptions extends DetectDecodeOptions {- /** Row index (within the per-anchor column) holding the angle. Defaults to `4 + nc`. */- readonly angleRow?: number;-}--interface ObbCandidate extends NmsBox {- readonly angle: number;-}--/** Projects the 4 rotated corners (input px) and the axis-aligned extent back to source px. */-function projectCorners(- cx: number,- cy: number,- w: number,- h: number,- angle: number,- opts: DetectDecodeOptions,-): {- corners: ReadonlyArray<readonly [number, number]>;- minX: number;- minY: number;- maxX: number;- maxY: number;-} {- const { scale, dw, dh } = opts.params;- const cos = Math.cos(angle);- const sin = Math.sin(angle);- const hw = w / 2;- const hh = h / 2;- const local: ReadonlyArray<readonly [number, number]> = [- [-hw, -hh],- [hw, -hh],- [hw, hh],- [-hw, hh],- ];- const corners: Array<readonly [number, number]> = [];- let minX = Infinity;- let minY = Infinity;- let maxX = -Infinity;- let maxY = -Infinity;- for (const [lx, ly] of local) {- const rx = cx + lx * cos - ly * sin;- const ry = cy + lx * sin + ly * cos;- const sx = (rx - dw) / scale;- const sy = (ry - dh) / scale;- corners.push([sx, sy]);- if (sx < minX) minX = sx;- if (sy < minY) minY = sy;- if (sx > maxX) maxX = sx;- if (sy > maxY) maxY = sy;- }- return { corners, minX, minY, maxX, maxY };-}--export function decodeObbOutput(- output: Float32Array,- anchors: number,- nc: number,- opts: ObbDecodeOptions,-): YoloOBBResult {- const angleRow = opts.angleRow ?? 4 + nc;- const classFilter = opts.classes- ? new Set(opts.classes.map((c) => c.toLowerCase()))- : null;-- const candidates: ObbCandidate[] = [];- for (let a = 0; a < anchors; a++) {- let bestScore = -1;- let bestClass = -1;- for (let c = 0; c < nc; c++) {- const score = output[(4 + c) * anchors + a];- if (score > bestScore) {- bestScore = score;- bestClass = c;- }- }- if (bestScore < opts.confidence) continue;- if (- classFilter &&- !classFilter.has((opts.classNames[bestClass] ?? "").toLowerCase())- ) {- continue;- }- const cx = output[a];- const cy = output[anchors + a];- const w = output[2 * anchors + a];- const h = output[3 * anchors + a];- if (w <= 0 || h <= 0) continue;- const angle = output[angleRow * anchors + a];- if (!Number.isFinite(angle)) continue;- candidates.push({- x0: cx - w / 2,- y0: cy - h / 2,- x1: cx + w / 2,- y1: cy + h / 2,- score: bestScore,- classIndex: bestClass,- anchorIndex: a,- angle,- });- }-- const kept = nms(candidates, opts.iouThreshold);- const detections: YoloOBBDetection[] = [];-- for (const box of kept) {- const cand = box as ObbCandidate;- const cx = output[cand.anchorIndex];- const cy = output[anchors + cand.anchorIndex];- const w = output[2 * anchors + cand.anchorIndex];- const h = output[3 * anchors + cand.anchorIndex];- const { corners, minX, minY, maxX, maxY } = projectCorners(- cx,- cy,- w,- h,- cand.angle,- opts,- );- const sw = opts.sourceWidth;- const sh = opts.sourceHeight;- const originX = Math.max(0, Math.min(sw, minX));- const originY = Math.max(0, Math.min(sh, minY));- const width = Math.max(0, Math.min(sw, maxX) - originX);- const height = Math.max(0, Math.min(sh, maxY) - originY);-- detections.push({- category: opts.classNames[cand.classIndex] ?? `class_${cand.classIndex}`,- classIndex: cand.classIndex,- score: cand.score,- boundingBox: {- originX,- originY,- width,- height,- normalizedX: sw > 0 ? originX / sw : 0,- normalizedY: sh > 0 ? originY / sh : 0,- normalizedWidth: sw > 0 ? width / sw : 0,- normalizedHeight: sh > 0 ? height / sh : 0,- angle: cand.angle,- },- corners,- });- }-- return { detections };-}diff --git a/packages/yolo/src/decode/pose.ts b/packages/yolo/src/decode/pose.tsdeleted file mode 100644index 8fde330a..00000000--- a/packages/yolo/src/decode/pose.ts+++ /dev/null@@ -1,86 +0,0 @@-import type {- YoloPoseKeypoint,- YoloPosePerson,- YoloPoseResult,-} from "../types.js";-import type { LetterboxParams } from "./letterbox.js";--/**- * Pose decode.- *- * Head: [1, 4 + nc + 17·3, 8400], nc = 1 ("person"):- * rows 0-3 box xywh (input px), row 4 person confidence (sigmoided in graph),- * rows 5.. per keypoint triplet (x, y, visibility) — x/y in input px, visibility sigmoided.- */-export interface PoseDecodeOptions {- readonly confidence: number;- readonly params: LetterboxParams;- readonly sourceWidth: number;- readonly sourceHeight: number;-}--export function decodePoseOutput(- output: Float32Array,- anchors: number,- opts: PoseDecodeOptions,-): YoloPoseResult {- const { scale, dw, dh } = opts.params;- const { sourceWidth: sw, sourceHeight: sh } = opts;-- const people: YoloPosePerson[] = [];-- for (let a = 0; a < anchors; a++) {- const score = output[4 * anchors + a];- if (score < opts.confidence) continue;-- const cx = output[a];- const cy = output[anchors + a];- const w = output[2 * anchors + a];- const h = output[3 * anchors + a];- if (w <= 0 || h <= 0) continue;-- const x0 = Math.max(0, Math.min(sw, (cx - w / 2 - dw) / scale));- const y0 = Math.max(0, Math.min(sh, (cy - h / 2 - dh) / scale));- const x1 = Math.max(0, Math.min(sw, (cx + w / 2 - dw) / scale));- const y1 = Math.max(0, Math.min(sh, (cy + h / 2 - dh) / scale));- const boxW = Math.max(0, x1 - x0);- const boxH = Math.max(0, y1 - y0);-- const keypoints: Array<YoloPoseKeypoint> = [];- for (let k = 0; k < 17; k++) {- const kx = Math.max(- 0,- Math.min(sw, (output[(5 + 3 * k) * anchors + a] - dw) / scale),- );- const ky = Math.max(- 0,- Math.min(sh, (output[(6 + 3 * k) * anchors + a] - dh) / scale),- );- const vis = output[(7 + 3 * k) * anchors + a];- keypoints.push({- x: kx,- y: ky,- visibility: Math.max(0, Math.min(1, vis)),- });- }-- people.push({- score,- boundingBox: {- originX: x0,- originY: y0,- width: boxW,- height: boxH,- normalizedX: sw > 0 ? x0 / sw : 0,- normalizedY: sh > 0 ? y0 / sh : 0,- normalizedWidth: sw > 0 ? boxW / sw : 0,- normalizedHeight: sh > 0 ? boxH / sh : 0,- },- keypoints,- });- }-- // Highest-confidence person first — the bundle's named signals track it as primary- people.sort((a, b) => b.score - a.score);- return { people };-}diff --git a/packages/yolo/src/decode/segment.ts b/packages/yolo/src/decode/segment.tsdeleted file mode 100644index 57cb7403..00000000--- a/packages/yolo/src/decode/segment.ts+++ /dev/null@@ -1,131 +0,0 @@-import type { YoloInstanceMask, YoloSegmentationResult } from "../types.js";-import {- type DetectDecodeOptions,- decodeYoloBoxes,- toDetectedObject,-} from "./detect.js";--/**- * Instance segmentation decode.- *- * Head: [1, 4 + nc + 32, 8400] — the 32 trailing rows are per-detect mask coefficients.- * Proto: [1, 32, protoDim, protoDim] (protoDim = imgsz / 4, e.g. 160 at 640).- *- * Per kept detection: mask = sigmoid(Σ coeff_c · proto_c) → threshold 0.5, then bilinear- * upscaled from the 160-grid directly into the ORIGINAL source frame (letterbox-inverse),- * restricted to the detection box so masks stay tight and cheap.- */--export function decodeSegmentOutput(- output: Float32Array,- proto: Float32Array,- anchors: number,- nc: number,- protoDim: number,- opts: DetectDecodeOptions,-): YoloSegmentationResult {- const kept = decodeYoloBoxes(output, anchors, nc, opts);-- const detections = kept.map((box) => toDetectedObject(box, opts));- const masks: YoloInstanceMask[] = [];-- const { scale, dw, dh, size } = opts.params;- const { sourceWidth: sw, sourceHeight: sh } = opts;- const maskThreshold = opts.maskThreshold ?? 0.5;- const feather = opts.featherRadius !== undefined ? opts.featherRadius : 0.05;- const maskLogit = Math.log(maskThreshold / (1 - maskThreshold));- const tMin = maskThreshold - feather;- const tMax = maskThreshold + feather;- const invTwoFeather = feather > 0 ? 1 / (2 * feather) : 0;-- for (let i = 0; i < kept.length; i++) {- const box = kept[i];- const det = detections[i];-- // Mask coefficients for this detection: rows (4 + nc) .. (4 + nc + 31)- const coeffBase = (4 + nc) * anchors + box.anchorIndex;- const step = anchors;-- // Source-space box (clamped, expanded slightly so the silhouette isn't clipped)- const padPx = 8;- const sx0 = Math.max(0, Math.floor((box.x0 - dw) / scale) - padPx);- const sy0 = Math.max(0, Math.floor((box.y0 - dh) / scale) - padPx);- const sx1 = Math.min(sw - 1, Math.ceil((box.x1 - dw) / scale) + padPx);- const sy1 = Math.min(sh - 1, Math.ceil((box.y1 - dh) / scale) + padPx);- if (sx1 < sx0 || sy1 < sy0) continue;-- // Single accumulation pass over the box region, bilinear from the proto grid:- // source (sx, sy) → input (dw + sx·scale, dh + sy·scale) → grid (… / size · protoDim)- const mask = new Uint8Array(sw * sh);- const inv = protoDim / size;- let area = 0;-- for (let sy = sy0; sy <= sy1; sy++) {- const gyy = (dh + sy * scale) * inv;- const gy0 = Math.floor(gyy);- const fy = gyy - gy0;- const gy1 = Math.min(protoDim - 1, gy0 + 1);- const maskRow = sy * sw;-- for (let sx = sx0; sx <= sx1; sx++) {- const gxx = (dw + sx * scale) * inv;- const gx0 = Math.floor(gxx);- const fx = gxx - gx0;- const gx1 = Math.min(protoDim - 1, gx0 + 1);-- // Bilinear over proto grid- let acc = 0;- for (let c = 0; c < 32; c++) {- const coeff = output[coeffBase + c * step];- if (coeff === 0) continue;- const rowBase = c * protoDim * protoDim;- const v00 = proto[rowBase + gy0 * protoDim + gx0];- const v10 = proto[rowBase + gy0 * protoDim + gx1];- const v01 = proto[rowBase + gy1 * protoDim + gx0];- const v11 = proto[rowBase + gy1 * protoDim + gx1];- const interp =- v00 * (1 - fx) * (1 - fy) +- v10 * fx * (1 - fy) +- v01 * (1 - fx) * fy +- v11 * fx * fy;- acc += coeff * interp;- }-- if (feather <= 0) {- if (acc > maskLogit) {- mask[maskRow + sx] = 255;- area++;- }- } else {- const alpha = acc > 16 ? 1 : acc < -16 ? 0 : 1 / (1 + Math.exp(-acc));- let alphaNorm = 0;- if (alpha >= tMax) {- alphaNorm = 1;- } else if (alpha > tMin) {- alphaNorm = (alpha - tMin) * invTwoFeather;- }- const alphaOut = Math.round(alphaNorm * 255);- if (alphaOut > 0) {- mask[maskRow + sx] = alphaOut;- area += alphaNorm;- }- }- }- }-- area = Math.round(area);- if (area === 0) continue;-- masks.push({- category: det.category,- mask,- width: sw,- height: sh,- area,- coverage: sw * sh > 0 ? area / (sw * sh) : 0,- detectionIndex: i,- });- }-- return { detections, masks };-}diff --git a/packages/yolo/src/extractors/canonical-indices.ts b/packages/yolo/src/extractors/canonical-indices.tsdeleted file mode 100644index 43f7e3f3..00000000--- a/packages/yolo/src/extractors/canonical-indices.ts+++ /dev/null@@ -1,131 +0,0 @@-/** COCO-17 keypoint indices for YOLO pose outputs (matches YOLO_POSE_KEYPOINTS order). */-export const POSE_LANDMARKS_YOLO = {- NOSE: 0,- LEFT_EYE: 1,- RIGHT_EYE: 2,- LEFT_EAR: 3,- RIGHT_EAR: 4,- LEFT_SHOULDER: 5,- RIGHT_SHOULDER: 6,- LEFT_ELBOW: 7,- RIGHT_ELBOW: 8,- LEFT_WRIST: 9,- RIGHT_WRIST: 10,- LEFT_HIP: 11,- RIGHT_HIP: 12,- LEFT_KNEE: 13,- RIGHT_KNEE: 14,- LEFT_ANKLE: 15,- RIGHT_ANKLE: 16,-} as const;--export const YOLO_POSE_KEYPOINT_NAMES = [- "nose",- "leftEye",- "rightEye",- "leftEar",- "rightEar",- "leftShoulder",- "rightShoulder",- "leftElbow",- "rightElbow",- "leftWrist",- "rightWrist",- "leftHip",- "rightHip",- "leftKnee",- "rightKnee",- "leftAnkle",- "rightAnkle",-] as const;--/** COCO-17 skeleton bones for the WebGPU skeleton renderer (indices above). */-export const YOLO_COCO17_BONES = [- {- from: POSE_LANDMARKS_YOLO.NOSE,- to: POSE_LANDMARKS_YOLO.LEFT_SHOULDER,- color: [0.0, 1.0, 1.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.NOSE,- to: POSE_LANDMARKS_YOLO.RIGHT_SHOULDER,- color: [0.0, 1.0, 1.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_SHOULDER,- to: POSE_LANDMARKS_YOLO.RIGHT_SHOULDER,- color: [1.0, 0.0, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_SHOULDER,- to: POSE_LANDMARKS_YOLO.LEFT_ELBOW,- color: [1.0, 0.333, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_ELBOW,- to: POSE_LANDMARKS_YOLO.LEFT_WRIST,- color: [1.0, 0.667, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.RIGHT_SHOULDER,- to: POSE_LANDMARKS_YOLO.RIGHT_ELBOW,- color: [1.0, 1.0, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.RIGHT_ELBOW,- to: POSE_LANDMARKS_YOLO.RIGHT_WRIST,- color: [0.667, 1.0, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_SHOULDER,- to: POSE_LANDMARKS_YOLO.LEFT_HIP,- color: [0.333, 1.0, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.RIGHT_SHOULDER,- to: POSE_LANDMARKS_YOLO.RIGHT_HIP,- color: [0.0, 1.0, 0.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_HIP,- to: POSE_LANDMARKS_YOLO.RIGHT_HIP,- color: [0.0, 1.0, 0.333, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_HIP,- to: POSE_LANDMARKS_YOLO.LEFT_KNEE,- color: [0.0, 1.0, 0.667, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.LEFT_KNEE,- to: POSE_LANDMARKS_YOLO.LEFT_ANKLE,- color: [0.0, 1.0, 1.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.RIGHT_HIP,- to: POSE_LANDMARKS_YOLO.RIGHT_KNEE,- color: [0.0, 0.667, 1.0, 1.0] as const,- },- {- from: POSE_LANDMARKS_YOLO.RIGHT_KNEE,- to: POSE_LANDMARKS_YOLO.RIGHT_ANKLE,- color: [0.0, 0.333, 1.0, 1.0] as const,- },-] as const;--/** Mapping from MediaPipe 33-pose indices to YOLO COCO-17 (named props survive unchanged). */-export const MEDIAPIPE_TO_YOLO_POSE_INDEX: Readonly<Record<number, number>> = {- 0: 0, // nose- 11: 5, // leftShoulder- 12: 6, // rightShoulder- 13: 7, // leftElbow- 14: 8, // rightElbow- 15: 9, // leftWrist- 16: 10, // rightWrist- 23: 11, // leftHip- 24: 12, // rightHip- 25: 13, // leftKnee- 26: 14, // rightKnee- 27: 15, // leftAnkle- 28: 16, // rightAnkle-};diff --git a/packages/yolo/src/gpu/gpu-acceleration.test.ts b/packages/yolo/src/gpu/gpu-acceleration.test.tsdeleted file mode 100644index 64bee3e3..00000000--- a/packages/yolo/src/gpu/gpu-acceleration.test.ts+++ /dev/null@@ -1,171 +0,0 @@-import { describe, expect, it } from "vitest";-import type { TrackedObject, YoloInstanceMask } from "../types.js";-import { letterboxResizeWgsl } from "./letterbox-resize.wgsl.js";-import { maskUpscaleWgsl } from "./mask-upscale.wgsl.js";-import {- type FlowField,- warpBoundingBox,- warpDetectedObjects,- warpInstanceMask,- warpTrackedObjects,-} from "./optical-flow-warper.js";--describe("Phase 5: GPU Compute Acceleration & Optical Flow Warping", () => {- it("letterboxResizeWgsl declares valid compute entry point and NCHW bindings", () => {- expect(letterboxResizeWgsl).toContain("fn computeLetterbox(");- expect(letterboxResizeWgsl).toContain("@workgroup_size(16, 16)");- expect(letterboxResizeWgsl).toContain("outputBuffer: array<f32>");- expect(letterboxResizeWgsl).toContain("params.padValue");- });-- it("maskUpscaleWgsl evaluates 32-proto bilinear interpolation with sigmoid and feathering", () => {- expect(maskUpscaleWgsl).toContain("fn computeMaskUpscale(");- expect(maskUpscaleWgsl).toContain("protoBuffer: array<f32>");- expect(maskUpscaleWgsl).toContain("coeffsBuffer: array<f32>");- expect(maskUpscaleWgsl).toContain(- "outputMask: texture_storage_2d<rgba8unorm, write>",- );- expect(maskUpscaleWgsl).toContain("1.0 / (1.0 + exp(-acc))");- });-- it("warpBoundingBox shifts box along motion vectors", () => {- const originalBox = {- originX: 100,- originY: 100,- width: 50,- height: 80,- normalizedX: 0.1,- normalizedY: 0.1,- normalizedWidth: 0.05,- normalizedHeight: 0.08,- };-- const warped = warpBoundingBox(- originalBox,- { dx: 20, dy: -10 },- 1000,- 1000,- );-- expect(warped.originX).toBeCloseTo(120, 1);- expect(warped.originY).toBeCloseTo(90, 1);- expect(warped.normalizedX).toBeCloseTo(0.12, 4);- expect(warped.normalizedY).toBeCloseTo(0.09, 4);- expect(warped.width).toBe(50);- expect(warped.height).toBe(80);- });-- it("warpDetectedObjects shifts raw detections along motion vectors", () => {- const detection = {- category: "car",- score: 0.88,- boundingBox: {- originX: 500,- originY: 400,- width: 120,- height: 80,- normalizedX: 0.5,- normalizedY: 0.4,- normalizedWidth: 0.12,- normalizedHeight: 0.08,- },- };-- const flowField: FlowField = {- width: 1000,- height: 1000,- getVector: () => ({ dx: -25, dy: 10 }),- };-- const warped = warpDetectedObjects([detection], flowField, 1000, 1000);- expect(warped).toHaveLength(1);- expect(warped[0].boundingBox.originX).toBeCloseTo(475, 1);- expect(warped[0].boundingBox.originY).toBeCloseTo(410, 1);- });-- it("warpTrackedObjects forward-warps active tracks and marks them coasting", () => {- const tracked: TrackedObject[] = [- {- trackId: 1,- category: "person",- score: 0.95,- boundingBox: {- originX: 200,- originY: 200,- width: 100,- height: 200,- normalizedX: 0.2,- normalizedY: 0.2,- normalizedWidth: 0.1,- normalizedHeight: 0.2,- },- firstSeenFrame: 0,- lastSeenFrame: 10,- consecutiveMisses: 0,- velocity: { x: 5, y: 0, vx: 5, vy: 0 },- isCoasting: false,- },- ];-- const flowField: FlowField = {- width: 1000,- height: 1000,- getVector: () => ({ dx: 15, dy: 5 }),- };-- const warped = warpTrackedObjects(tracked, flowField, 1000, 1000);- expect(warped).toHaveLength(1);- expect(warped[0].trackId).toBe(1);- expect(warped[0].isCoasting).toBe(true);- expect(warped[0].boundingBox.originX).toBeCloseTo(215, 1);- expect(warped[0].boundingBox.originY).toBeCloseTo(205, 1);- });-- it("warpInstanceMask moves soft alpha pixel values along flow vectors", () => {- const width = 10;- const height = 10;- const mask = new Uint8Array(width * height).fill(0);- // Put an active 2x2 soft-alpha blob at (2, 2)- mask[2 * width + 2] = 200;- mask[2 * width + 3] = 200;- mask[3 * width + 2] = 200;- mask[3 * width + 3] = 200;-- const instanceMask: YoloInstanceMask = {- trackId: 1,- category: "person",- score: 0.9,- width,- height,- mask,- area: 4,- boundingBox: {- originX: 2,- originY: 2,- width: 2,- height: 2,- normalizedX: 0.2,- normalizedY: 0.2,- normalizedWidth: 0.2,- normalizedHeight: 0.2,- },- };-- // Motion vector: dx = 3, dy = 2- const flowField: FlowField = {- width,- height,- getVector: () => ({ dx: 3, dy: 2 }),- };-- const warped = warpInstanceMask(instanceMask, flowField);- expect(warped.area).toBe(4);- // Pixels that were at (2, 2) should now appear at (5, 4)- expect(warped.mask[4 * width + 5]).toBe(200);- expect(warped.mask[4 * width + 6]).toBe(200);- expect(warped.mask[5 * width + 5]).toBe(200);- expect(warped.mask[5 * width + 6]).toBe(200);- // Original position (2, 2) should now be 0- expect(warped.mask[2 * width + 2]).toBe(0);- });-});diff --git a/packages/yolo/src/gpu/letterbox-resize.wgsl.ts b/packages/yolo/src/gpu/letterbox-resize.wgsl.tsdeleted file mode 100644index ead8b677..00000000--- a/packages/yolo/src/gpu/letterbox-resize.wgsl.ts+++ /dev/null@@ -1,56 +0,0 @@-/**- * @file letterbox-resize.wgsl.ts- * High-performance WebGPU compute shader for hardware-accelerated YOLO letterbox pre-processing.- * Converts input RGBA GPUTexture into a normalized [0, 1] planar NCHW Float32 storage buffer.- */--export const letterboxResizeWgsl = /* wgsl */ `-struct LetterboxParams {- sourceWidth: u32,- sourceHeight: u32,- imgsz: u32,- padDw: u32,- padDh: u32,- scale: f32,- padValue: f32, // constant 114.0 / 255.0 ≈ 0.44705882-};--@group(0) @binding(0) var inputTex: texture_2d<f32>;-@group(0) @binding(1) var<storage, read_write> outputBuffer: array<f32>;-@group(0) @binding(2) var<uniform> params: LetterboxParams;--@compute @workgroup_size(16, 16)-fn computeLetterbox(@builtin(global_invocation_id) id: vec3<u32>) {- if (id.x >= params.imgsz || id.y >= params.imgsz) {- return;- }-- let ox = id.x;- let oy = id.y;- let imgsz = params.imgsz;- let planeSize = imgsz * imgsz;-- let dw = params.padDw;- let dh = params.padDh;- let rw = u32(f32(params.sourceWidth) * params.scale + 0.5);- let rh = u32(f32(params.sourceHeight) * params.scale + 0.5);-- var r: f32 = params.padValue;- var g: f32 = params.padValue;- var b: f32 = params.padValue;-- if (ox >= dw && ox < dw + rw && oy >= dh && oy < dh + rh) {- let sx = clamp(i32(f32(ox - dw) / params.scale), 0, i32(params.sourceWidth) - 1);- let sy = clamp(i32(f32(oy - dh) / params.scale), 0, i32(params.sourceHeight) - 1);- let pixel = textureLoad(inputTex, vec2<i32>(sx, sy), 0);- r = pixel.r;- g = pixel.g;- b = pixel.b;- }-- let pixelIdx = oy * imgsz + ox;- outputBuffer[pixelIdx] = r;- outputBuffer[planeSize + pixelIdx] = g;- outputBuffer[2u * planeSize + pixelIdx] = b;-}-`;diff --git a/packages/yolo/src/gpu/mask-upscale.wgsl.ts b/packages/yolo/src/gpu/mask-upscale.wgsl.tsdeleted file mode 100644index 82ed18fc..00000000--- a/packages/yolo/src/gpu/mask-upscale.wgsl.ts+++ /dev/null@@ -1,89 +0,0 @@-/**- * @file mask-upscale.wgsl.ts- * High-performance WebGPU compute shader for GPU-native instance mask upscaling.- * Computes weighted sum of 32 proto channels with continuous sigmoid soft alpha- * and feathering directly into a VRAM mask texture.- */--export const maskUpscaleWgsl = /* wgsl */ `-struct MaskParams {- maskWidth: u32,- maskHeight: u32,- protoDim: u32,- imgsz: u32,- padDw: u32,- padDh: u32,- scale: f32,- maskThreshold: f32,- featherRadius: f32,-};--@group(0) @binding(0) var<storage, read> protoBuffer: array<f32>;-@group(0) @binding(1) var<storage, read> coeffsBuffer: array<f32>;-@group(0) @binding(2) var outputMask: texture_storage_2d<rgba8unorm, write>;-@group(0) @binding(3) var<uniform> params: MaskParams;--@compute @workgroup_size(16, 16)-fn computeMaskUpscale(@builtin(global_invocation_id) id: vec3<u32>) {- if (id.x >= params.maskWidth || id.y >= params.maskHeight) {- return;- }-- let mx = f32(id.x);- let my = f32(id.y);-- // Map source coordinate through letterbox into proto space [0, protoDim]- let lx = mx * params.scale + f32(params.padDw);- let ly = my * params.scale + f32(params.padDh);-- let px = (lx / f32(params.imgsz)) * f32(params.protoDim);- let py = (ly / f32(params.imgsz)) * f32(params.protoDim);-- let protoDimI = i32(params.protoDim);- let x0 = clamp(i32(floor(px)), 0, protoDimI - 1);- let y0 = clamp(i32(floor(py)), 0, protoDimI - 1);- let x1 = clamp(x0 + 1, 0, protoDimI - 1);- let y1 = clamp(y0 + 1, 0, protoDimI - 1);-- let fx = px - f32(x0);- let fy = py - f32(y0);-- let w00 = (1.0 - fx) * (1.0 - fy);- let w10 = fx * (1.0 - fy);- let w01 = (1.0 - fx) * fy;- let w11 = fx * fy;-- let planeStride = params.protoDim * params.protoDim;- var acc: f32 = 0.0;-- for (var c: u32 = 0u; c < 32u; c = c + 1u) {- let planeOffset = c * planeStride;- let idx00 = planeOffset + u32(y0) * params.protoDim + u32(x0);- let idx10 = planeOffset + u32(y0) * params.protoDim + u32(x1);- let idx01 = planeOffset + u32(y1) * params.protoDim + u32(x0);- let idx11 = planeOffset + u32(y1) * params.protoDim + u32(x1);-- let v00 = protoBuffer[idx00];- let v10 = protoBuffer[idx10];- let v01 = protoBuffer[idx01];- let v11 = protoBuffer[idx11];-- let interpolated = w00 * v00 + w10 * v10 + w01 * v01 + w11 * v11;- acc = acc + coeffsBuffer[c] * interpolated;- }-- // Sigmoid activation- let alpha = 1.0 / (1.0 + exp(-acc));-- // Anti-aliased feathering- let feather = max(0.001, params.featherRadius);- let lowBound = params.maskThreshold - feather;- let val = clamp((alpha - lowBound) / (2.0 * feather), 0.0, 1.0);-- textureStore(- outputMask,- vec2<i32>(i32(id.x), i32(id.y)),- vec4<f32>(val, val, val, val)- );-}-`;diff --git a/packages/yolo/src/gpu/optical-flow-warper.ts b/packages/yolo/src/gpu/optical-flow-warper.tsdeleted file mode 100644index 639833be..00000000--- a/packages/yolo/src/gpu/optical-flow-warper.ts+++ /dev/null@@ -1,146 +0,0 @@-/**- * @file optical-flow-warper.ts- * Temporal forward-warper for YOLO detections and instance masks using optical flow motion vectors.- * Enables 4x inference throughput by synthesizing intermediate frames via motion vector advection.- */--import type { DetectedObject, NormalizedBoundingBox } from "@gitframes/core";-import type { TrackedObject, YoloInstanceMask } from "../types.js";--export interface MotionVector {- readonly dx: number;- readonly dy: number;-}--export interface FlowField {- readonly width: number;- readonly height: number;- /** Interleaved or planar velocity components [dx, dy] */- getVector(x: number, y: number): MotionVector;-}--/**- * Forward-warps a normalized or absolute bounding box along motion vectors.- */-export function warpBoundingBox(- box: NormalizedBoundingBox,- motion: MotionVector,- sourceWidth: number,- sourceHeight: number,-): NormalizedBoundingBox {- const normDx = sourceWidth > 0 ? motion.dx / sourceWidth : 0;- const normDy = sourceHeight > 0 ? motion.dy / sourceHeight : 0;-- const newNormX = Math.max(- 0,- Math.min(1 - box.normalizedWidth, box.normalizedX + normDx),- );- const newNormY = Math.max(- 0,- Math.min(1 - box.normalizedHeight, box.normalizedY + normDy),- );-- const newOriginX = newNormX * sourceWidth;- const newOriginY = newNormY * sourceHeight;-- return {- originX: newOriginX,- originY: newOriginY,- width: box.width,- height: box.height,- normalizedX: newNormX,- normalizedY: newNormY,- normalizedWidth: box.normalizedWidth,- normalizedHeight: box.normalizedHeight,- angle: box.angle,- };-}--/**- * Forward-warps an array of tracked objects across an un-inferred frame.- */-export function warpTrackedObjects(- objects: readonly TrackedObject[],- flow: FlowField,- sourceWidth: number,- sourceHeight: number,-): TrackedObject[] {- return objects.map((obj) => {- const centerX = obj.boundingBox.originX + obj.boundingBox.width / 2;- const centerY = obj.boundingBox.originY + obj.boundingBox.height / 2;- const motion = flow.getVector(centerX, centerY);-- const warpedBox = warpBoundingBox(- obj.boundingBox,- motion,- sourceWidth,- sourceHeight,- );-- return {- ...obj,- boundingBox: warpedBox,- isCoasting: true,- };- });-}--/**- * Forward-warps an array of detected objects.- */-export function warpDetectedObjects(- detections: readonly DetectedObject[],- flow: FlowField,- sourceWidth: number,- sourceHeight: number,-): DetectedObject[] {- return detections.map((det) => {- const centerX = det.boundingBox.originX + det.boundingBox.width / 2;- const centerY = det.boundingBox.originY + det.boundingBox.height / 2;- const motion = flow.getVector(centerX, centerY);-- return {- ...det,- boundingBox: warpBoundingBox(- det.boundingBox,- motion,- sourceWidth,- sourceHeight,- ),- };- });-}--/**- * Forward-warps a continuous sigmoid instance mask using a dense motion flow field.- */-export function warpInstanceMask(- instanceMask: YoloInstanceMask,- flow: FlowField,-): YoloInstanceMask {- const { width, height, mask } = instanceMask;- const warped = new Uint8Array(mask.length);- let area = 0;-- for (let y = 0; y < height; y++) {- const rowOffset = y * width;- for (let x = 0; x < width; x++) {- const motion = flow.getVector(x, y);- // Inverse warp lookup: where did this pixel come from?- const srcX = Math.round(x - motion.dx);- const srcY = Math.round(y - motion.dy);-- if (srcX >= 0 && srcX < width && srcY >= 0 && srcY < height) {- const val = mask[srcY * width + srcX];- warped[rowOffset + x] = val;- if (val > 0) area++;- }- }- }-- return {- ...instanceMask,- mask: warped,- area,- };-}diff --git a/packages/yolo/src/gpu/yolo-gpu-pipeline.ts b/packages/yolo/src/gpu/yolo-gpu-pipeline.tsdeleted file mode 100644index 90108b40..00000000--- a/packages/yolo/src/gpu/yolo-gpu-pipeline.ts+++ /dev/null@@ -1,182 +0,0 @@-/**- * @file yolo-gpu-pipeline.ts- * WebGPU Tier T2 acceleration pipeline for hardware letterboxing and mask upscaling.- */--import { letterboxResizeWgsl } from "./letterbox-resize.wgsl.js";-import { maskUpscaleWgsl } from "./mask-upscale.wgsl.js";--export interface GpuLetterboxOptions {- readonly sourceWidth: number;- readonly sourceHeight: number;- readonly imgsz: number;- readonly padDw: number;- readonly padDh: number;- readonly scale: number;-}--export interface GpuMaskUpscaleOptions {- readonly maskWidth: number;- readonly maskHeight: number;- readonly protoDim: number;- readonly imgsz: number;- readonly padDw: number;- readonly padDh: number;- readonly scale: number;- readonly maskThreshold?: number;- readonly featherRadius?: number;-}--export class YoloGpuPipeline {- private readonly _device: GPUDevice;- private _letterboxPipeline?: GPUComputePipeline;- private _maskUpscalePipeline?: GPUComputePipeline;-- constructor(device: GPUDevice) {- this._device = device;- }-- public get device(): GPUDevice {- return this._device;- }-- /**- * Dispatches hardware letterboxing on a GPU compute queue.- */- public dispatchLetterbox(- inputTexture: GPUTexture,- outputBuffer: GPUBuffer,- options: GpuLetterboxOptions,- encoder?: GPUCommandEncoder,- ): void {- if (!this._letterboxPipeline) {- const module = this._device.createShaderModule({- label: "yolo_letterbox_resize.wgsl",- code: letterboxResizeWgsl,- });- this._letterboxPipeline = this._device.createComputePipeline({- label: "YoloLetterboxPipeline",- layout: "auto",- compute: {- module,- entryPoint: "computeLetterbox",- },- });- }-- // Prepare uniform buffer- // sourceWidth, sourceHeight, imgsz, padDw, padDh, scale, padValue, pad- const uniformData = new ArrayBuffer(32);- const u32View = new Uint32Array(uniformData);- const f32View = new Float32Array(uniformData);- u32View[0] = options.sourceWidth;- u32View[1] = options.sourceHeight;- u32View[2] = options.imgsz;- u32View[3] = options.padDw;- u32View[4] = options.padDh;- f32View[5] = options.scale;- f32View[6] = 114.0 / 255.0; // padValue-- const uniformBuffer = this._device.createBuffer({- size: 32,- usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,- });- this._device.queue.writeBuffer(uniformBuffer, 0, uniformData);-- const bindGroup = this._device.createBindGroup({- layout: this._letterboxPipeline.getBindGroupLayout(0),- entries: [- { binding: 0, resource: inputTexture.createView() },- { binding: 1, resource: { buffer: outputBuffer } },- { binding: 2, resource: { buffer: uniformBuffer } },- ],- });-- const enc =- encoder ??- this._device.createCommandEncoder({ label: "yolo_letterbox_encoder" });- const pass = enc.beginComputePass({ label: "yolo_letterbox_pass" });- pass.setPipeline(this._letterboxPipeline);- pass.setBindGroup(0, bindGroup);- const workgroups = Math.ceil(options.imgsz / 16);- pass.dispatchWorkgroups(workgroups, workgroups);- pass.end();-- if (!encoder) {- this._device.queue.submit([enc.finish()]);- }- }-- /**- * Dispatches hardware mask upscaling on a GPU compute queue.- */- public dispatchMaskUpscale(- protoBuffer: GPUBuffer,- coeffsBuffer: GPUBuffer,- outputTexture: GPUTexture,- options: GpuMaskUpscaleOptions,- encoder?: GPUCommandEncoder,- ): void {- if (!this._maskUpscalePipeline) {- const module = this._device.createShaderModule({- label: "yolo_mask_upscale.wgsl",- code: maskUpscaleWgsl,- });- this._maskUpscalePipeline = this._device.createComputePipeline({- label: "YoloMaskUpscalePipeline",- layout: "auto",- compute: {- module,- entryPoint: "computeMaskUpscale",- },- });- }-- // Uniform layout:- // maskWidth (u32), maskHeight (u32), protoDim (u32), imgsz (u32),- // padDw (u32), padDh (u32), scale (f32), maskThreshold (f32), featherRadius (f32)- const uniformData = new ArrayBuffer(48);- const u32View = new Uint32Array(uniformData);- const f32View = new Float32Array(uniformData);- u32View[0] = options.maskWidth;- u32View[1] = options.maskHeight;- u32View[2] = options.protoDim;- u32View[3] = options.imgsz;- u32View[4] = options.padDw;- u32View[5] = options.padDh;- f32View[6] = options.scale;- f32View[7] = options.maskThreshold ?? 0.5;- f32View[8] = options.featherRadius ?? 0.05;-- const uniformBuffer = this._device.createBuffer({- size: 48,- usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,- });- this._device.queue.writeBuffer(uniformBuffer, 0, uniformData);-- const bindGroup = this._device.createBindGroup({- layout: this._maskUpscalePipeline.getBindGroupLayout(0),- entries: [- { binding: 0, resource: { buffer: protoBuffer } },- { binding: 1, resource: { buffer: coeffsBuffer } },- { binding: 2, resource: outputTexture.createView() },- { binding: 3, resource: { buffer: uniformBuffer } },- ],- });-- const enc =- encoder ??- this._device.createCommandEncoder({ label: "yolo_mask_upscale_encoder" });- const pass = enc.beginComputePass({ label: "yolo_mask_upscale_pass" });- pass.setPipeline(this._maskUpscalePipeline);- pass.setBindGroup(0, bindGroup);- const wgX = Math.ceil(options.maskWidth / 16);- const wgY = Math.ceil(options.maskHeight / 16);- pass.dispatchWorkgroups(wgX, wgY);- pass.end();-- if (!encoder) {- this._device.queue.submit([enc.finish()]);- }- }-}diff --git a/packages/yolo/src/index.ts b/packages/yolo/src/index.tsdeleted file mode 100644index a5781595..00000000--- a/packages/yolo/src/index.ts+++ /dev/null@@ -1,29 +0,0 @@-export * from "./analysis/analyze-sequence.js";-export * from "./decode/classify.js";-export * from "./decode/detect.js";-export * from "./decode/letterbox.js";-export { type NmsBox, nms } from "./decode/nms.js";-export * from "./decode/obb.js";-export * from "./decode/pose.js";-export * from "./decode/segment.js";-export * from "./extractors/canonical-indices.js";-export * from "./gpu/letterbox-resize.wgsl.js";-export * from "./gpu/mask-upscale.wgsl.js";-export * from "./gpu/optical-flow-warper.js";-export * from "./gpu/pose-skeleton-renderer.js";-export * from "./gpu/segmentation-texture-pool.js";-export * from "./gpu/yolo-gpu-pipeline.js";-export * from "./model/model-store.js";-export * from "./model/registry.js";-export * from "./runner/canonical-baselines.js";-export * from "./runner/yolo-runner.js";-export * from "./runtime/session-provider.js";-export * from "./signals/yolo-signals-bundle.js";-export * from "./spatial/camera-space-transformer.js";-export * from "./spatial/spatial-pin.js";-export * from "./tracking/pose-track-matcher.js";-export * from "./tracking/temporal-object-tracker.js";-export * from "./types.js";-// Explicit (not `export *`): shadows the type-only re-export of the same name from-// @gitframes/core via ./types.js — the runtime class is the surface callers want.-export { type YoloAttachedBundle, YoloNode } from "./yolo-node.js";diff --git a/packages/yolo/src/model/model-store.test.ts b/packages/yolo/src/model/model-store.test.tsdeleted file mode 100644index 7e70e5e2..00000000--- a/packages/yolo/src/model/model-store.test.ts+++ /dev/null@@ -1,127 +0,0 @@-import { existsSync, mkdtempSync } from "node:fs";-import fs from "node:fs/promises";-import { tmpdir } from "node:os";-import { join } from "node:path";-import { afterEach, beforeEach, describe, expect, it } from "vitest";-import { YoloModelStore } from "./model-store.js";-import { YOLO_MODELS } from "./registry.js";--describe("YoloModelStore (lazy download semantics)", () => {- let dir: string;- let fetches: Array<{ url: string }>;-- beforeEach(() => {- dir = mkdtempSync(join(tmpdir(), "yolo-store-"));- fetches = [];- });-- afterEach(() => {- fetches = [];- });-- const store = () =>- new YoloModelStore({- modelsDir: dir,- baseUrl: "https://example.test/models",- fetchImpl: (async (url: unknown) => {- fetches.push({ url: String(url) });- return new Response(new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]));- }) as typeof fetch,- timeoutMs: 5000,- });-- it("constructor performs ZERO I/O (no fetch, no status change)", () => {- const s = store();- expect(fetches).toHaveLength(0);- expect(s.status("yolo11n")).toBe("pending");- });-- it("downloads only when ensure() is called, caches in flight, and maps downloads", async () => {- const s = store();- const [a, b] = await Promise.all([- s.ensure("yolo11n"),- s.ensure("yolo11n"),- ]);- expect(fetches).toHaveLength(1); // in-flight dedupe- expect(a).toEqual(b);- expect(s.status("yolo11n")).toBe("ready");- // second ensure hits the memory cache — no new fetch- await s.ensure("yolo11n");- expect(fetches).toHaveLength(1);- // file persisted for future processes- expect(existsSync(s.pathFor("yolo11n"))).toBe(true);- });-- it("distinct tasks map to distinct models and fetch independently", async () => {- const s = store();- // only segment is fetched; detect stays pending- await s.ensure("yolo11n-seg");- expect(fetches).toHaveLength(1);- expect(fetches[0].url).toContain(YOLO_MODELS["yolo11n-seg"].filename);- expect(s.status("yolo11n")).toBe("pending");- await s.ensure("yolo11n");- expect(fetches).toHaveLength(2);- expect(s.status("yolo11n")).toBe("ready");- });-- it("preload() downloads only the requested keys", async () => {- const s = store();- await s.preload(["yolo11n", "yolo11n-pose"]);- expect(fetches).toHaveLength(2);- expect(s.status("yolo11n-pose")).toBe("ready");- expect(s.status("yolo11n-seg")).toBe("pending");- });-- it("re-downloads a stub/partial cached file instead of trusting it", async () => {- // a 4 KiB orphan (e.g. a test fixture or interrupted write) is below the- // validity floor — has() must be false and ensure() must re-fetch- await fs.mkdir(dir, { recursive: true });- await fs.writeFile(join(dir, "yolo11n.onnx"), new Uint8Array(4096));- const s = store();- expect(s.has("yolo11n")).toBe(false);- await s.ensure("yolo11n");- expect(fetches).toHaveLength(1);- expect(s.status("yolo11n")).toBe("ready");- });-- it("evict() drops the file and resets to pending so the next ensure re-downloads", async () => {- const s = store();- await s.ensure("yolo11n");- expect(existsSync(s.pathFor("yolo11n"))).toBe(true);- await s.evict("yolo11n");- expect(existsSync(s.pathFor("yolo11n"))).toBe(false);- expect(s.has("yolo11n")).toBe(false);- expect(s.status("yolo11n")).toBe("pending");- await s.ensure("yolo11n");- expect(fetches).toHaveLength(2); // re-downloaded, not served from memory- });-- it("surfaces an error status and can retry after failure", async () => {- let calls = 0;- const s = new YoloModelStore({- modelsDir: dir,- baseUrl: "https://example.test/models",- fetchImpl: (async () => {- calls += 1;- if (calls === 1) throw new Error("boom");- return new Response(new Uint8Array([9, 9, 9]));- }) as typeof fetch,- });- await expect(s.ensure("yolo11n")).rejects.toThrow("boom");- expect(s.status("yolo11n")).toBe("error");- expect(s.has("yolo11n")).toBe(false);- const bytes = await s.ensure("yolo11n");- expect([...bytes]).toEqual([9, 9, 9]);- expect(s.status("yolo11n")).toBe("ready");- });-- it("does not re-download a corrupt/empty (zero-length) file — treats as error", async () => {- const s = new YoloModelStore({- modelsDir: dir,- baseUrl: "https://example.test/models",- fetchImpl: (async () => new Response(new Uint8Array(0))) as typeof fetch,- });- await expect(s.ensure("yolo11n")).rejects.toThrow(/empty/i);- expect(s.status("yolo11n")).toBe("error");- });-});diff --git a/packages/yolo/src/model/model-store.ts b/packages/yolo/src/model/model-store.tsdeleted file mode 100644index d9459ffe..00000000--- a/packages/yolo/src/model/model-store.ts+++ /dev/null@@ -1,233 +0,0 @@-import { existsSync, mkdirSync, readFileSync, statSync } from "node:fs";-import { rename, unlink, writeFile } from "node:fs/promises";-import { homedir } from "node:os";-import { dirname, join, resolve } from "node:path";-import {- DEFAULT_YOLO_BASE_URL,- YOLO_MODELS,- type YoloModelDescriptor,- type YoloModelKey,-} from "./registry.js";--export type ModelDownloadStatus = "pending" | "downloading" | "ready" | "error";--export interface YoloModelStoreOptions {- readonly modelsDir?: string;- readonly baseUrl?: string;- readonly timeoutMs?: number;- /** Injectable fetch — offline unit tests inject a mock. */- readonly fetchImpl?: typeof fetch;- readonly onProgress?: (- key: YoloModelKey,- bytesLoaded: number,- totalBytes: number,- ) => void;-}--/**- * Resolves the default model cache directory:- * 1. `GITFRAMES_MODELS_DIR`- * 2. `~/.cache/gitframes/models`- */-export function getDefaultModelsDir(): string {- if (process.env.GITFRAMES_MODELS_DIR) {- return resolve(process.env.GITFRAMES_MODELS_DIR);- }- return resolve(homedir(), ".cache/gitframes/models");-}--/**- * Lazy YOLO model store. Constructor performs ZERO I/O; the only entry that can hit the- * network is `ensure(key)`, called by an inference function the first time it runs.- * Concurrent callers of the same key share a single in-flight download (promise dedupe),- * and verified files on disk are reused across processes.- *- * Mirrors the proven mechanics of the old MediaPipe model manager (atomic temp+rename,- * size verification, `GITFRAMES_MODELS_DIR` override) minus its eager constructor downloads.- */-export class YoloModelStore {- private readonly _modelsDir: string;- private readonly _baseUrl: string;- private readonly _timeoutMs?: number;- private readonly _fetch: typeof fetch;- private readonly _onProgress?: YoloModelStoreOptions["onProgress"];-- private _inflight = new Map<YoloModelKey, Promise<Uint8Array>>();- private _buffers = new Map<YoloModelKey, Uint8Array>();- private _status = new Map<YoloModelKey, ModelDownloadStatus>();-- constructor(options: YoloModelStoreOptions = {}) {- this._modelsDir = options.modelsDir- ? resolve(options.modelsDir)- : getDefaultModelsDir();- this._baseUrl =- options.baseUrl ??- process.env.GITFRAMES_YOLO_BASE_URL ??- DEFAULT_YOLO_BASE_URL;- this._timeoutMs = options.timeoutMs;- this._fetch = options.fetchImpl ?? globalThis.fetch;- this._onProgress = options.onProgress;- }-- public get modelsDir(): string {- return this._modelsDir;- }-- public pathFor(key: YoloModelKey): string {- return join(this._modelsDir, YOLO_MODELS[key].filename);- }-- public descriptor(key: YoloModelKey): YoloModelDescriptor {- return YOLO_MODELS[key];- }-- /**- * Smallest file we trust as a cached model. Every YOLO11 export in the registry- * ships well above 1 MB (smallest is yolo11n-cls ≈ 1.2 MB); a smaller file is a- * partial/stub write (e.g. a 4 KiB test fixture) and must be re-downloaded.- */- static readonly MIN_VALID_MODEL_BYTES = 512 * 1024;-- public has(key: YoloModelKey): boolean {- const targetPath = this.pathFor(key);- if (!existsSync(targetPath)) {- return false;- }- try {- return statSync(targetPath).size >= YoloModelStore.MIN_VALID_MODEL_BYTES;- } catch {- return false;- }- }-- /** Drops a cached model (file + memory) so the next `ensure` re-downloads it. */- public async evict(key: YoloModelKey): Promise<void> {- this._buffers.delete(key);- this._status.set(key, "pending");- await unlink(this.pathFor(key)).catch(() => undefined);- }-- /** Direct download helper for arbitrary model URLs, respecting injected fetch. */- public async fetchDirect(url: string): Promise<Uint8Array> {- const res = await this._fetch(url);- if (!res.ok) {- throw new Error(`Failed to fetch model from ${url}: ${res.statusText}`);- }- const buf = await res.arrayBuffer();- return new Uint8Array(buf);- }-- public status(key: YoloModelKey): ModelDownloadStatus {- return this._status.get(key) ?? "pending";- }-- /** Snapshot of every tracked model's status (empty until anything is fetched). */- public statuses(): ReadonlyMap<YoloModelKey, ModelDownloadStatus> {- return new Map(this._status);- }-- /** True when the model is resident in memory (loaded or downloaded this process). */- public isLoaded(key: YoloModelKey): boolean {- return this._buffers.has(key);- }-- /**- * THE only download entry point. Returns the model bytes from memory (cached),- * loading from disk or fetching from the registry URL as needed.- */- public async ensure(key: YoloModelKey): Promise<Uint8Array> {- const inMemory = this._buffers.get(key);- if (inMemory) {- return inMemory;- }- const inflight = this._inflight.get(key);- if (inflight) {- return inflight;- }- const promise = this.load(key).finally(() => this._inflight.delete(key));- this._inflight.set(key, promise);- return promise;- }-- /** Explicit warm-up — fetch multiple models ahead of use (agents, renderers). */- public async preload(keys: readonly YoloModelKey[]): Promise<void> {- await Promise.all(keys.map((key) => this.ensure(key)));- }-- private async load(key: YoloModelKey): Promise<Uint8Array> {- const targetPath = this.pathFor(key);-- if (this.has(key)) {- this._status.set(key, "ready");- return this.readBuffer(key, targetPath);- }-- const desc = YOLO_MODELS[key];- const downloadUrl = desc.url ?? `${this._baseUrl}${desc.filename}`;- this._status.set(key, "downloading");-- if (!existsSync(dirname(targetPath))) {- mkdirSync(dirname(targetPath), { recursive: true });- }-- const tempPath = `${targetPath}.tmp.${Date.now()}`;- try {- const response = await this._fetch(downloadUrl, {- signal: this._timeoutMs- ? AbortSignal.timeout(this._timeoutMs)- : undefined,- });- if (!response.ok) {- throw new Error(- `Failed to download YOLO model '${key}' from ${downloadUrl}: HTTP ${response.status} ${response.statusText}`,- );- }- const totalBytes = Number(response.headers.get("content-length") ?? 0);- const arrayBuffer = await response.arrayBuffer();- const buffer = new Uint8Array(arrayBuffer);-- if (buffer.byteLength === 0) {- throw new Error(`Downloaded YOLO model '${key}' is empty (0 bytes).`);- }-- if (this._onProgress) {- this._onProgress(- key,- buffer.byteLength,- totalBytes || buffer.byteLength,- );- }-- // Write-then-rename keeps the cache atomic: a crash never leaves a partial model- // at the final path (verified by has(): size > 1 KiB).- await writeFile(tempPath, buffer);- await rename(tempPath, targetPath);-- this._buffers.set(key, buffer);- this._status.set(key, "ready");- return buffer;- } catch (error) {- this._status.set(key, "error");- if (existsSync(tempPath)) {- await unlink(tempPath).catch(() => undefined);- }- throw error;- }- }-- private readBuffer(key: YoloModelKey, targetPath: string): Uint8Array {- const fileBuffer = readFileSync(targetPath);- const buffer = new Uint8Array(- fileBuffer.buffer,- fileBuffer.byteOffset,- fileBuffer.byteLength,- );- this._buffers.set(key, buffer);- return buffer;- }-- /** Drops in-memory buffers (model stays on disk for the next run). */- public clearMemoryCache(): void {- this._buffers.clear();- }-}diff --git a/packages/yolo/src/model/registry.ts b/packages/yolo/src/model/registry.tsdeleted file mode 100644index 1a190cd0..00000000--- a/packages/yolo/src/model/registry.ts+++ /dev/null@@ -1,420 +0,0 @@-/**- * YOLO11 model registry — single source of truth for what exists, where it lives,- * and which model key serves which task + variant.- *- * Downloads are ALWAYS lazy: nothing here touches the network. The store (model-store.ts)- * fetches a model only when an inference function that needs it is first called.- */--export type YoloTask =- | "detect"- | "segment"- | "pose"- | "obb"- | "classify"- | "world";--export type YoloVariant = "n" | "s" | "m" | "l" | "x";--export const YOLO_TASKS: readonly YoloTask[] = [- "detect",- "segment",- "pose",- "obb",- "classify",- "world",-];--export const YOLO_VARIANTS: readonly YoloVariant[] = ["n", "s", "m", "l", "x"];--export type YoloModelKey =- | "yolo11n"- | "yolo11s"- | "yolo11m"- | "yolo11l"- | "yolo11x"- | "yolo11n-seg"- | "yolo11s-seg"- | "yolo11m-seg"- | "yolo11n-pose"- | "yolo11s-pose"- | "yolo11m-pose"- | "yolo11l-pose"- | "yolo11n-obb"- | "yolo11s-obb"- | "yolo11m-obb"- | "yolo11l-obb"- | "yolo11n-cls"- | "yolo11s-cls"- | "yolov8s-worldv2"- | "yolov8m-worldv2";--export interface YoloModelDescriptor {- readonly key: YoloModelKey;- readonly task: YoloTask;- readonly variant: YoloVariant;- /** Filename relative to the download base URL. */- readonly filename: string;- /** Optional absolute download URL (used when the asset is not at the shared base URL). */- readonly url?: string;- /** Approximate FP32 byte size — used for progress reporting only. */- readonly bytesFp32: number;- /** Approximate FP16 byte size (web/WebGPU only; CPU EP stays fp32). Metadata only. */- readonly bytesFp16?: number;- readonly nc: number;- readonly imgsz: number;-}--/**- * Default download origin. Override per-run with `baseUrl` option or the- * GITFRAMES_YOLO_BASE_URL environment variable.- */-export const YOLO_ASSETS_RELEASE = "v8.3.0";--export const DEFAULT_YOLO_BASE_URL = `https://github.com/ultralytics/assets/releases/download/${YOLO_ASSETS_RELEASE}/`;--export const YOLO_MODELS: Readonly<Record<YoloModelKey, YoloModelDescriptor>> =- {- // ── Detect (COCO 80) ────────────────────────────────────────────────- yolo11n: {- key: "yolo11n",- task: "detect",- variant: "n",- filename: "yolo11n.onnx",- bytesFp32: 10_723_904,- bytesFp16: 5_361_952,- nc: 80,- imgsz: 640,- },- yolo11s: {- key: "yolo11s",- task: "detect",- variant: "s",- filename: "yolo11s.onnx",- url: "https://huggingface.co/mobilint/YOLO11s/resolve/main/yolo11s.onnx",- bytesFp32: 38_502_400,- bytesFp16: 19_251_200,- nc: 80,- imgsz: 640,- },- yolo11m: {- key: "yolo11m",- task: "detect",- variant: "m",- filename: "yolo11m.onnx",- bytesFp32: 82_329_600,- bytesFp16: 41_164_800,- nc: 80,- imgsz: 640,- },- yolo11l: {- key: "yolo11l",- task: "detect",- variant: "l",- filename: "yolo11l.onnx",- bytesFp32: 103_628_800,- bytesFp16: 51_814_400,- nc: 80,- imgsz: 640,- },- yolo11x: {- key: "yolo11x",- task: "detect",- variant: "x",- filename: "yolo11x.onnx",- bytesFp32: 233_062_400,- bytesFp16: 116_531_200,- nc: 80,- imgsz: 640,- },- // ── Instance segmentation (COCO 80) ─────────────────────────────────- "yolo11n-seg": {- key: "yolo11n-seg",- task: "segment",- variant: "n",- filename: "yolo11n-seg.onnx",- bytesFp32: 11_886_592,- nc: 80,- imgsz: 640,- },- "yolo11s-seg": {- key: "yolo11s-seg",- task: "segment",- variant: "s",- filename: "yolo11s-seg.onnx",- // Ultralytics only hosts the `n` ONNX at the shared release URL; the `s`- // weights are published on Hugging Face (standard Ultralytics ONNX export).- url: "https://huggingface.co/mobilint/YOLO11s-seg/resolve/main/yolo11s-seg.onnx",- bytesFp32: 42_631_168,- nc: 80,- imgsz: 640,- },- "yolo11m-seg": {- key: "yolo11m-seg",- task: "segment",- variant: "m",- filename: "yolo11m-seg.onnx",- bytesFp32: 91_128_832,- nc: 80,- imgsz: 640,- },- // ── Pose (COCO-17, nc = 1 "person") ─────────────────────────────────- "yolo11n-pose": {- key: "yolo11n-pose",- task: "pose",- variant: "n",- filename: "yolo11n-pose.onnx",- bytesFp32: 11_450_368,- nc: 1,- imgsz: 640,- },- "yolo11s-pose": {- key: "yolo11s-pose",- task: "pose",- variant: "s",- filename: "yolo11s-pose.onnx",- url: "https://huggingface.co/mobilint/YOLO11s-pose/resolve/main/yolo11s-pose.onnx",- bytesFp32: 41_050_112,- nc: 1,- imgsz: 640,- },- "yolo11m-pose": {- key: "yolo11m-pose",- task: "pose",- variant: "m",- filename: "yolo11m-pose.onnx",- url: "https://huggingface.co/mobilint/YOLO11m-pose/resolve/main/yolo11m-pose.onnx",- bytesFp32: 82_100_000,- nc: 1,- imgsz: 640,- },- "yolo11l-pose": {- key: "yolo11l-pose",- task: "pose",- variant: "l",- filename: "yolo11l-pose.onnx",- url: "https://huggingface.co/mobilint/YOLO11l-pose/resolve/main/yolo11l-pose.onnx",- bytesFp32: 110_500_000,- nc: 1,- imgsz: 640,- },- // ── Oriented bounding boxes (DOTA 15 OBB) ─────────────────────────- // Head layout: 4 xywh + 15 classes + 1 angle = 20 rows per anchor (angle last).- "yolo11n-obb": {- key: "yolo11n-obb",- task: "obb",- variant: "n",- filename: "yolo11n-obb.onnx",- bytesFp32: 12_048_384,- nc: 15,- imgsz: 1024,- },- "yolo11s-obb": {- key: "yolo11s-obb",- task: "obb",- variant: "s",- filename: "yolo11s-obb.onnx",- bytesFp32: 43_253_760,- nc: 15,- imgsz: 1024,- },- "yolo11m-obb": {- key: "yolo11m-obb",- task: "obb",- variant: "m",- filename: "yolo11m-obb.onnx",- url: "https://huggingface.co/mobilint/YOLO11m-obb/resolve/main/yolo11m-obb.onnx",- bytesFp32: 83_200_000,- nc: 15,- imgsz: 1024,- },- "yolo11l-obb": {- key: "yolo11l-obb",- task: "obb",- variant: "l",- filename: "yolo11l-obb.onnx",- url: "https://huggingface.co/mobilint/YOLO11l-obb/resolve/main/yolo11l-obb.onnx",- bytesFp32: 111_000_000,- nc: 15,- imgsz: 1024,- },- // ── Classification (ImageNet-1k) ────────────────────────────────────- "yolo11n-cls": {- key: "yolo11n-cls",- task: "classify",- variant: "n",- filename: "yolo11n-cls.onnx",- bytesFp32: 11_918_848,- nc: 1000,- imgsz: 224,- },- "yolo11s-cls": {- key: "yolo11s-cls",- task: "classify",- variant: "s",- filename: "yolo11s-cls.onnx",- bytesFp32: 42_704_896,- nc: 1000,- imgsz: 224,- },- // ── Open-Vocabulary YOLO-World (v2) ─────────────────────────────────- "yolov8s-worldv2": {- key: "yolov8s-worldv2",- task: "world",- variant: "s",- filename: "yolov8s-worldv2.onnx",- url: "https://huggingface.co/Instemic/yolo-world-onnx/resolve/main/yolov8s-worldv2.onnx",- bytesFp32: 52_000_000,- nc: 80,- imgsz: 640,- },- "yolov8m-worldv2": {- key: "yolov8m-worldv2",- task: "world",- variant: "m",- filename: "yolov8m-worldv2.onnx",- url: "https://huggingface.co/ultralytics/yolov8/resolve/main/yolov8m-worldv2.onnx",- bytesFp32: 108_000_000,- nc: 80,- imgsz: 640,- },- };--export const DEFAULT_MODEL_KEY_BY_TASK: Readonly<- Record<YoloTask, (variant: YoloVariant) => YoloModelKey>-> = {- detect: (v) =>- v === "n"- ? "yolo11n"- : v === "s"- ? "yolo11s"- : v === "m"- ? "yolo11m"- : v === "l"- ? "yolo11l"- : "yolo11x",- segment: (v) =>- v === "m" ? "yolo11m-seg" : v === "s" ? "yolo11s-seg" : "yolo11n-seg",- pose: (v) =>- v === "l"- ? "yolo11l-pose"- : v === "m"- ? "yolo11m-pose"- : v === "s"- ? "yolo11s-pose"- : "yolo11n-pose",- obb: (v) =>- v === "l"- ? "yolo11l-obb"- : v === "m"- ? "yolo11m-obb"- : v === "s"- ? "yolo11s-obb"- : "yolo11n-obb",- classify: (v) => (v === "s" ? "yolo11s-cls" : "yolo11n-cls"),- world: (v) =>- v === "m" || v === "l" || v === "x" ? "yolov8m-worldv2" : "yolov8s-worldv2",-};--/** COCO 80 class names — index order matches every ultralytics COCO export. */-export const YOLO_COCO_CLASSES = [- "person",- "bicycle",- "car",- "motorcycle",- "airplane",- "bus",- "train",- "truck",- "boat",- "traffic light",- "fire hydrant",- "stop sign",- "parking meter",- "bench",- "bird",- "cat",- "dog",- "horse",- "sheep",- "cow",- "elephant",- "bear",- "zebra",- "giraffe",- "backpack",- "umbrella",- "handbag",- "tie",- "suitcase",- "frisbee",- "skis",- "snowboard",- "sports ball",- "kite",- "baseball bat",- "baseball glove",- "skateboard",- "surfboard",- "tennis racket",- "bottle",- "wine glass",- "cup",- "fork",- "knife",- "spoon",- "bowl",- "banana",- "apple",- "sandwich",- "orange",- "broccoli",- "carrot",- "hot dog",- "pizza",- "donut",- "cake",- "chair",- "couch",- "potted plant",- "bed",- "dining table",- "toilet",- "tv",- "laptop",- "mouse",- "remote",- "keyboard",- "cell phone",- "microwave",- "oven",- "toaster",- "sink",- "refrigerator",- "book",- "clock",- "vase",- "scissors",- "teddy bear",- "hair drier",- "toothbrush",-] as const;--/** DOTA-v1.0 15 class names — index order matches ultralytics YOLO11-OBB export. */-export const YOLO_DOTA_CLASSES = [- "plane",- "ship",- "storage tank",- "baseball diamond",- "tennis court",- "basketball court",- "ground track field",- "harbor",- "bridge",- "large vehicle",- "small vehicle",- "helicopter",- "roundabout",- "soccer ball field",- "swimming pool",-] as const;diff --git a/packages/yolo/src/runner/yolo-runner.test.ts b/packages/yolo/src/runner/yolo-runner.test.tsdeleted file mode 100644index 4148d185..00000000--- a/packages/yolo/src/runner/yolo-runner.test.ts+++ /dev/null@@ -1,392 +0,0 @@-import { mkdtempSync } from "node:fs";-import { tmpdir } from "node:os";-import { join } from "node:path";-import { describe, expect, it } from "vitest";-import type {- SessionProvider,- YoloSession,- YoloTensorInput,-} from "../runtime/session-provider.js";-import { YoloVisionRunner } from "./yolo-runner.js";--/** Deterministic fake ONNX session: stores the input tensor, replays scripted outputs. */-class ScriptedSession implements YoloSession {- public inputs: YoloTensorInput[] = [];- constructor(- private outputs: Record<- string,- { data: Float32Array; dims: readonly number[] }- >,- ) {}- async run(- input: YoloTensorInput | readonly YoloTensorInput[],- ): Promise<Record<string, { data: Float32Array; dims: readonly number[] }>> {- const list = Array.isArray(input) ? input : [input];- for (const inp of list) {- this.inputs.push({- name: inp.name,- data: inp.data.slice(),- dims: [...inp.dims],- });- }- return this.outputs;- }- async release(): Promise<void> {}-}--class ScriptedProvider implements SessionProvider {- public readonly kind = "node" as const;- public created: ScriptedSession[] = [];- private script: ScriptedSession[];- constructor(- script: Array<- Record<string, { data: Float32Array; dims: readonly number[] }>- >,- ) {- this.script = script.map((outputs) => new ScriptedSession(outputs));- }- async createSession(): Promise<YoloSession> {- const session = this.script[this.created.length] ?? new ScriptedSession({});- this.created.push(session);- return session;- }-}--function dummyImage(width = 640, height = 640) {- return {- data: new Uint8ClampedArray(width * height * 4).fill(128),- width,- height,- };-}--function detectOutput() {- // [1, 4+1, 1] — a single person at input center- const out = new Float32Array(5);- out[0] = 320;- out[1] = 320;- out[2] = 200;- out[3] = 300;- out[4] = 0.9;- return { output0: { data: out, dims: [1, 5, 1] } };-}--function poseOutput() {- const rows = 4 + 1 + 17 * 3;- const out = new Float32Array(rows);- out[0] = 320;- out[1] = 320;- out[2] = 200;- out[3] = 300;- out[4] = 0.8;- out[5] = 320;- out[6] = 280;- out[7] = 0.99; // nose- return { output0: { data: out, dims: [1, rows, 1] } };-}--function obbOutput() {- // OBB head: [1, 4 + nc + 1, 1] — one plane (DOTA class 0) with a 30° rotation.- const nc = 15;- const rows = 4 + nc + 1;- const out = new Float32Array(rows);- out[0] = 320;- out[1] = 320;- out[2] = 200;- out[3] = 120;- out[4 + 0] = 0.9;- out[4 + nc] = Math.PI / 6;- return { output0: { data: out, dims: [1, rows, 1] } };-}--function segOutput() {- // seg head, full layout: [1, 4 + 80 + 32, 8400] — one person (class 0) at anchor 0- const anchors = 8400;- const nc = 80;- const rows = 4 + nc + 32;- const preds = new Float32Array(rows * anchors);- preds[0 * anchors] = 320;- preds[1 * anchors] = 320;- preds[2 * anchors] = 200;- preds[3 * anchors] = 300;- preds[4 * anchors] = 0.9;- for (let c = 0; c < 32; c++) preds[(4 + nc + c) * anchors] = 1;- // proto masks: [1, 32, 160, 160]- const proto = new Float32Array(32 * 160 * 160).fill(0);- // a solid blob on the 160-grid covering input pixels ~220..260 (grid 55..65),- // inside the detection box (x 220..420, y 170..470) — the instance mask is non-empty- for (let c = 0; c < 32; c++) {- for (let y = 55; y < 65; y++) {- for (let x = 55; x < 65; x++) {- proto[c * 160 * 160 + y * 160 + x] = 1;- }- }- }- return {- output0: { data: preds, dims: [1, rows, anchors] },- output1: { data: proto, dims: [1, 32, 160, 160] },- };-}--describe("YoloVisionRunner (lazy + decode integration)", () => {- it("create() performs zero provider work; first detect() downloads + runs", async () => {- // Stub fetch BEFORE create(): the store captures the fetch implementation at- // construction time (lazy by design — it uses whatever fetch exists on first use).- let storeFetches = 0;- const origFetch = globalThis.fetch;- globalThis.fetch = (async () => {- storeFetches++;- return new Response(new Uint8Array([1, 2, 3, 4]));- }) as typeof fetch;- const provider = new ScriptedProvider([detectOutput()]);- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- imgsz: 640,- });- try {- // session provider recorded nothing yet — zero I/O at create()- expect(provider.created.length).toBe(0);- expect(runner.downloadStatus.get("yolo11n")).toBe("pending");-- const res = await runner.detect(dummyImage());- expect(storeFetches).toBe(1); // one lazy fetch, on first call- expect(runner.downloadStatus.get("yolo11n")).toBe("ready");- expect(res).toHaveLength(1);- expect(res[0].category).toBe("person");- expect(res[0].score).toBeCloseTo(0.9, 6);- expect(provider.created.length).toBe(1);-- // second call reuses everything — no additional fetch/session- await runner.detect(dummyImage());- expect(storeFetches).toBe(1);- expect(provider.created.length).toBe(1);- } finally {- globalThis.fetch = origFetch;- }- });-- it("segment() runs the seg pipeline and returns masks aligned to source", async () => {- const origFetch = globalThis.fetch;- globalThis.fetch = (async () =>- new Response(new Uint8Array([1, 2, 3, 4]))) as typeof fetch;- try {- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider: new ScriptedProvider([segOutput()]),- imgsz: 640,- });- const img = dummyImage(640, 640);- const res = await runner.segment(img);- expect(res.detections).toHaveLength(1);- expect(res.masks).toHaveLength(1);- expect(res.masks[0].width).toBe(640);- expect(res.masks[0].height).toBe(640);- expect(res.masks[0].category).toBe("person");- expect(res.masks[0].area).toBeGreaterThan(0);- } finally {- globalThis.fetch = origFetch;- }- });-- it("pose() decodes keypoints through the letterbox inverse", async () => {- const origFetch = globalThis.fetch;- globalThis.fetch = (async () =>- new Response(new Uint8Array([1, 2, 3, 4]))) as typeof fetch;- try {- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider: new ScriptedProvider([poseOutput()]),- imgsz: 640,- });- const img = dummyImage(1920, 1080);- const res = await runner.pose(img);- expect(res.people).toHaveLength(1);- const p = res.people[0];- expect(p.score).toBeCloseTo(0.8, 6);- expect(p.keypoints).toHaveLength(17);- // nose input (320, 280): letterbox scale 1/3, dh = 140- // source x = (320 - 0) * 3 = 960, y = (280 - 140) * 3 = 420- expect(p.keypoints[0].x).toBeCloseTo(960, 3);- expect(p.keypoints[0].y).toBeCloseTo(420, 3);- expect(p.keypoints[0].visibility).toBeCloseTo(0.99, 5);- } finally {- globalThis.fetch = origFetch;- }- });-- it("close() releases sessions and clears memory", async () => {- const origFetch = globalThis.fetch;- globalThis.fetch = (async () =>- new Response(new Uint8Array([1, 2, 3, 4]))) as typeof fetch;- try {- const provider = new ScriptedProvider([detectOutput(), detectOutput()]);- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- imgsz: 640,- });- await runner.detect(dummyImage());- expect(provider.created.length).toBe(1);- runner.close();- // next detect() re-creates the session from scratch (buffers were dropped)- await runner.detect(dummyImage());- expect(provider.created.length).toBe(2);- } finally {- globalThis.fetch = origFetch;- }- });-- it("preload() explicitly fetches ahead of use", async () => {- const origFetch = globalThis.fetch;- let fetches = 0;- globalThis.fetch = (async () => {- fetches++;- return new Response(new Uint8Array([1, 2, 3, 4]));- }) as typeof fetch;- try {- const provider = new ScriptedProvider([detectOutput(), poseOutput()]);- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- imgsz: 640,- });- await runner.preload(["detect", "pose"]);- expect(fetches).toBe(2);- expect(provider.created.length).toBe(2);- expect(runner.downloadStatus.get("yolo11n-pose")).toBe("ready");- } finally {- globalThis.fetch = origFetch;- }- });-- it("letterbox tensor shape matches ONNX input expectations", async () => {- const origFetch = globalThis.fetch;- globalThis.fetch = (async () =>- new Response(new Uint8Array([1, 2, 3, 4]))) as typeof fetch;- try {- const provider = new ScriptedProvider([detectOutput()]);- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- imgsz: 640,- });- await runner.detect(dummyImage());- const input = provider.created[0]?.inputs[0];- expect(input?.dims).toEqual([1, 3, 640, 640]);- expect(input?.name).toBe("images");- } finally {- globalThis.fetch = origFetch;- }- });-- it("detectObb() lazily loads ONLY the obb model (I1/I2)", async () => {- const origFetch = globalThis.fetch;- let fetches = 0;- globalThis.fetch = (async () => {- fetches++;- return new Response(new Uint8Array([1, 2, 3, 4]));- }) as typeof fetch;- try {- const provider = new ScriptedProvider([obbOutput()]);- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- imgsz: 640,- });- expect(runner.downloadStatus.get("yolo11n-obb")).toBe("pending");-- const res = await runner.detectObb(dummyImage());- expect(fetches).toBe(1);- expect(res.detections).toHaveLength(1);- expect(res.detections[0].category).toBe("plane");- expect(res.detections[0].boundingBox.angle).toBeCloseTo(Math.PI / 6, 6);- expect(runner.downloadStatus.get("yolo11n-obb")).toBe("ready");- // the detect model was never touched by an obb-only run- expect(runner.downloadStatus.get("yolo11n")).toBe("pending");- expect(provider.created.length).toBe(1);-- const input = provider.created[0]?.inputs[0];- expect(input?.dims).toEqual([1, 3, 1024, 1024]);- } finally {- globalThis.fetch = origFetch;- }- });-- it("detectWorld() lazily loads yolov8s-worldv2 and decodes open-vocabulary prompts", async () => {- const origFetch = globalThis.fetch;- let fetches = 0;- globalThis.fetch = (async () => {- fetches++;- return new Response(new Uint8Array([1, 2, 3, 4]));- }) as typeof fetch;- try {- const prompts = ["neon helmet", "cyberpunk car", "energy drink"] as const;- const nc = prompts.length;- const out = new Float32Array(4 + nc);- out[0] = 320;- out[1] = 320;- out[2] = 200;- out[3] = 300;- out[4 + 1] = 0.92; // second prompt: "cyberpunk car"-- const provider = new ScriptedProvider([- { output0: { data: out, dims: [1, 4 + nc, 1] } },- ]);- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- enableWorld: true,- prompts,- imgsz: 640,- });- expect(runner.downloadStatus.get("yolov8s-worldv2")).toBe("pending");-- const detections = await runner.detect(dummyImage());- expect(fetches).toBe(1);- expect(detections).toHaveLength(1);- expect(detections[0].category).toBe("cyberpunk car");- expect(detections[0].score).toBeCloseTo(0.92, 5);- expect(runner.downloadStatus.get("yolov8s-worldv2")).toBe("ready");- } finally {- globalThis.fetch = origFetch;- }- });-- it("customModel config routes inference to custom session and classes", async () => {- const customClasses = ["custom_gear", "rare_mineral"];- const nc = customClasses.length;- const out = new Float32Array(4 + nc);- out[0] = 320;- out[1] = 320;- out[2] = 150;- out[3] = 150;- out[4 + 0] = 0.88; // "custom_gear"-- const provider = new ScriptedProvider([- { output0: { data: out, dims: [1, 4 + nc, 1] } },- ]);- const origFetch = globalThis.fetch;- globalThis.fetch = (async () =>- new Response(new Uint8Array([1, 2, 3, 4]))) as typeof fetch;- try {- const runner = YoloVisionRunner.create({- modelsDir: mkdtempSync(join(tmpdir(), "yolo-runner-")),- provider,- customModel: {- url: "https://example.com/custom-yolo.onnx",- task: "detect",- classes: customClasses,- imgsz: 640,- },- });-- const detections = await runner.detect(dummyImage());- expect(detections).toHaveLength(1);- expect(detections[0].category).toBe("custom_gear");- expect(detections[0].score).toBeCloseTo(0.88, 5);- } finally {- globalThis.fetch = origFetch;- }- });-});diff --git a/packages/yolo/src/runner/yolo-runner.ts b/packages/yolo/src/runner/yolo-runner.tsdeleted file mode 100644index 6fef3363..00000000--- a/packages/yolo/src/runner/yolo-runner.ts+++ /dev/null@@ -1,612 +0,0 @@-import { readFile } from "node:fs/promises";-import type { DetectedObject } from "@gitframes/core";-import { decodeClassifyOutput } from "../decode/classify.js";-import {- type DetectDecodeOptions,- decodeDetectOutput,-} from "../decode/detect.js";-import { letterboxToTensor, type RgbaImage } from "../decode/letterbox.js";-import { decodeObbOutput } from "../decode/obb.js";-import { decodePoseOutput } from "../decode/pose.js";-import { decodeSegmentOutput } from "../decode/segment.js";-import {- type ModelDownloadStatus,- YoloModelStore,-} from "../model/model-store.js";-import {- DEFAULT_MODEL_KEY_BY_TASK,- YOLO_COCO_CLASSES,- YOLO_DOTA_CLASSES,- YOLO_MODELS,- type YoloModelKey,- type YoloTask,- type YoloVariant,-} from "../model/registry.js";-import {- NodeSessionProvider,- type SessionProvider,- type YoloSession,-} from "../runtime/session-provider.js";-import type {- CustomModelConfig,- YoloClassifyResult,- YoloImageInput,- YoloOBBResult,- YoloPoseResult,- YoloSegmentationResult,-} from "../types.js";--export interface YoloRunnerOptions {- readonly variant?: YoloVariant;- readonly imgsz?: number;- readonly confidence?: number;- readonly iouThreshold?: number;- readonly classes?: readonly string[];- readonly enableWorld?: boolean;- readonly prompts?: readonly string[];- readonly customModel?: CustomModelConfig;- readonly modelsDir?: string;- readonly baseUrl?: string;- readonly timeoutMs?: number;- readonly maskThreshold?: number;- readonly featherRadius?: number;- readonly provider?: SessionProvider;- readonly onProgress?: (- key: YoloModelKey,- bytesLoaded: number,- totalBytes: number,- ) => void;-}--const YOLO_ANCHORS_FOR_SIZE: Readonly<Record<number, number>> = {- 320: 2100,- 640: 8400,- 1024: 21504,- 1280: 33600,-};--/**- * Lazy YOLO11 runner.- *- * `create()` performs ZERO I/O: no model downloads, no sessions, no environment patching.- * Each inference method (`detect` / `segment` / `pose` / `classify`) downloads its model- * and warms its session on FIRST USE ONLY, then reuses them for the process lifetime.- * Call `close()` to release sessions.- */-export class YoloVisionRunner {- public readonly variant: YoloVariant;- public readonly imgsz: number;- public readonly confidence: number;- public readonly iouThreshold: number;- public readonly classes?: readonly string[];- public readonly enableWorld?: boolean;- public readonly prompts?: readonly string[];- public readonly customModel?: CustomModelConfig;- public readonly maskThreshold: number;- public readonly featherRadius?: number;-- private readonly _store: YoloModelStore;- private readonly _provider: SessionProvider;- private readonly _sessions = new Map<YoloModelKey, Promise<YoloSession>>();- private readonly _tensorScratch = new Map<YoloModelKey, Float32Array>();- private _customSessionPromise?: Promise<YoloSession>;-- private constructor(options: YoloRunnerOptions) {- this.variant = options.variant ?? "n";- this.imgsz = options.imgsz ?? 640;- this.confidence = options.confidence ?? 0.25;- this.iouThreshold = options.iouThreshold ?? 0.45;- this.classes = options.classes;- this.enableWorld = options.enableWorld;- this.prompts = options.prompts;- this.customModel = options.customModel;- this.maskThreshold = options.maskThreshold ?? 0.5;- this.featherRadius = options.featherRadius;- this._store = new YoloModelStore({- modelsDir: options.modelsDir,- baseUrl: options.baseUrl,- timeoutMs: options.timeoutMs,- onProgress: options.onProgress,- });- this._provider = options.provider ?? new NodeSessionProvider();- }-- /** Cheap: no downloads, no sessions, no side effects. */- public static create(options: YoloRunnerOptions = {}): YoloVisionRunner {- return new YoloVisionRunner(options);- }-- public get modelsDir(): string {- return this._store.modelsDir;- }-- /** Every registry model keyed to its current download state (downloaded → "ready"). */- public get downloadStatus(): ReadonlyMap<YoloModelKey, ModelDownloadStatus> {- const statuses = new Map<YoloModelKey, ModelDownloadStatus>();- for (const key of Object.keys(YOLO_MODELS) as YoloModelKey[]) {- statuses.set(key, "pending");- }- for (const [key, status] of this._store.statuses()) {- statuses.set(key, status);- }- return statuses;- }-- public modelKeyFor(task: YoloTask): YoloModelKey {- return DEFAULT_MODEL_KEY_BY_TASK[task](this.variant);- }-- /** Explicit warm-up: downloads + sessions for the given tasks, ahead of first use. */- public async preload(tasks?: readonly YoloTask[]): Promise<void> {- if (this.customModel) {- await this.customSession();- return;- }- const defaultTasks =- this.enableWorld || this.prompts- ? (["world", "segment", "pose"] as const)- : (["detect", "segment", "pose"] as const);- const keys = (tasks ?? defaultTasks).map((t) => this.modelKeyFor(t));- await this._store.preload(keys);- await Promise.all(keys.map((key) => this.session(key)));- }-- public async detect(image: YoloImageInput): Promise<DetectedObject[]> {- if (- this.customModel &&- (!this.customModel.task ||- this.customModel.task === "detect" ||- this.customModel.task === "world")- ) {- return this.detectCustom(image);- }- if (this.enableWorld || this.prompts) {- return this.detectWorld(image, this.prompts);- }- const key = this.modelKeyFor("detect");- const session = await this.session(key);- const { tensor, params } = this.prepareInput(image, key);- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, this.imgsz, this.imgsz],- });- const preds = pickByDims(outputs, 3);-- return decodeDetectOutput(- preds.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[this.imgsz] ?? 8400,- YOLO_MODELS[key].nc,- this.decodeOptions(image, params),- );- }-- public async detectWorld(- image: YoloImageInput,- prompts?: readonly string[],- ): Promise<DetectedObject[]> {- const activePrompts = prompts ?? this.prompts ?? YOLO_COCO_CLASSES;- const key = this.modelKeyFor("world");- const session = await this.session(key);- const { tensor, params } = this.prepareInput(image, key);-- const txtFeats = new Float32Array(activePrompts.length * 512);- for (let c = 0; c < activePrompts.length; c++) {- txtFeats.set(encodePromptEmbedding(activePrompts[c]), c * 512);- }-- const outputs = await session.run([- {- name: "images",- data: tensor,- dims: [1, 3, this.imgsz, this.imgsz],- },- {- name: "txt_feats",- data: txtFeats,- dims: [1, activePrompts.length, 512],- },- ]);- const preds = pickByDims(outputs, 3);-- return decodeDetectOutput(- preds.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[this.imgsz] ?? 8400,- activePrompts.length,- {- ...this.decodeOptions(image, params, "world"),- classNames: activePrompts,- },- );- }-- public async segment(image: YoloImageInput): Promise<YoloSegmentationResult> {- if (this.customModel && this.customModel.task === "segment") {- return this.segmentCustom(image);- }- const key = this.modelKeyFor("segment");- const session = await this.session(key);- const { tensor, params } = this.prepareInput(image, key);- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, this.imgsz, this.imgsz],- });-- const preds = pickByDims(outputs, 3);- const proto = pickByDims(outputs, 4);- const protoDim = proto.dims[2] ?? proto.dims[1] ?? 160;-- return decodeSegmentOutput(- preds.data,- proto.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[this.imgsz] ?? 8400,- YOLO_MODELS[key].nc,- protoDim,- this.decodeOptions(image, params),- );- }-- public async pose(image: YoloImageInput): Promise<YoloPoseResult> {- if (this.customModel && this.customModel.task === "pose") {- return this.poseCustom(image);- }- const key = this.modelKeyFor("pose");- const session = await this.session(key);- const { tensor, params } = this.prepareInput(image, key);- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, this.imgsz, this.imgsz],- });- const preds = pickByDims(outputs, 3);-- return decodePoseOutput(- preds.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[this.imgsz] ?? 8400,- {- confidence: this.confidence,- params,- sourceWidth: image.width,- sourceHeight: image.height,- },- );- }-- public async detectObb(image: YoloImageInput): Promise<YoloOBBResult> {- if (this.customModel && this.customModel.task === "obb") {- return this.obbCustom(image);- }- const key = this.modelKeyFor("obb");- const session = await this.session(key);- const modelImgsz = YOLO_MODELS[key].imgsz;- const { tensor, params } = this.prepareInput(image, key);- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, modelImgsz, modelImgsz],- });- const preds = pickByDims(outputs, 3);-- return decodeObbOutput(- preds.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[modelImgsz] ?? 21504,- YOLO_MODELS[key].nc,- this.decodeOptions(image, params, "obb"),- );- }-- public async classify(image: YoloImageInput): Promise<YoloClassifyResult> {- if (this.customModel && this.customModel.task === "classify") {- return this.classifyCustom(image);- }- const key = this.modelKeyFor("classify");- const session = await this.session(key);- // Classification models export at 224 — re-letterbox to the model's own imgsz- const modelImgsz = YOLO_MODELS[key].imgsz;- const { tensor } = letterboxToTensor(image, modelImgsz);- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, modelImgsz, modelImgsz],- });- const preds = pickByDims(outputs, 2);- return decodeClassifyOutput(preds.data, YOLO_MODELS[key].nc);- }-- public close(): void {- for (const promise of this._sessions.values()) {- void promise.then((session) => session.release()).catch(() => undefined);- }- this._sessions.clear();- if (this._customSessionPromise) {- void this._customSessionPromise- .then((session) => session.release())- .catch(() => undefined);- this._customSessionPromise = undefined;- }- this._store.clearMemoryCache();- }-- private async detectCustom(image: YoloImageInput): Promise<DetectedObject[]> {- if (!this.customModel) {- throw new Error("No custom model configured");- }- const session = await this.customSession();- const imgsz = this.customModel.imgsz ?? this.imgsz;- const scratch = new Float32Array(3 * imgsz * imgsz);- const { tensor, params } = letterboxToTensor(- normalizeImage(image),- imgsz,- scratch,- );- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, imgsz, imgsz],- });- const preds = pickByDims(outputs, 3);- const anchors = preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[imgsz] ?? 8400;-- return decodeDetectOutput(- preds.data,- anchors,- this.customModel.classes.length,- {- ...this.decodeOptions(image, params),- classNames: this.customModel.classes,- },- );- }-- private async segmentCustom(- image: YoloImageInput,- ): Promise<YoloSegmentationResult> {- if (!this.customModel) {- throw new Error("No custom model configured");- }- const session = await this.customSession();- const imgsz = this.customModel.imgsz ?? this.imgsz;- const scratch = new Float32Array(3 * imgsz * imgsz);- const { tensor, params } = letterboxToTensor(- normalizeImage(image),- imgsz,- scratch,- );- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, imgsz, imgsz],- });- const preds = pickByDims(outputs, 3);- const proto = pickByDims(outputs, 4);- const protoDim = proto.dims[2] ?? proto.dims[1] ?? 160;-- return decodeSegmentOutput(- preds.data,- proto.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[imgsz] ?? 8400,- this.customModel.classes.length,- protoDim,- {- ...this.decodeOptions(image, params),- classNames: this.customModel.classes,- },- );- }-- private async poseCustom(image: YoloImageInput): Promise<YoloPoseResult> {- if (!this.customModel) {- throw new Error("No custom model configured");- }- const session = await this.customSession();- const imgsz = this.customModel.imgsz ?? this.imgsz;- const scratch = new Float32Array(3 * imgsz * imgsz);- const { tensor, params } = letterboxToTensor(- normalizeImage(image),- imgsz,- scratch,- );- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, imgsz, imgsz],- });- const preds = pickByDims(outputs, 3);-- return decodePoseOutput(- preds.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[imgsz] ?? 8400,- {- confidence: this.confidence,- params,- sourceWidth: image.width,- sourceHeight: image.height,- },- );- }-- private async obbCustom(image: YoloImageInput): Promise<YoloOBBResult> {- if (!this.customModel) {- throw new Error("No custom model configured");- }- const session = await this.customSession();- const imgsz = this.customModel.imgsz ?? this.imgsz;- const scratch = new Float32Array(3 * imgsz * imgsz);- const { tensor, params } = letterboxToTensor(- normalizeImage(image),- imgsz,- scratch,- );- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, imgsz, imgsz],- });- const preds = pickByDims(outputs, 3);-- return decodeObbOutput(- preds.data,- preds.dims[2] ?? YOLO_ANCHORS_FOR_SIZE[imgsz] ?? 21504,- this.customModel.classes.length,- {- ...this.decodeOptions(image, params, "obb"),- classNames: this.customModel.classes,- },- );- }-- private async classifyCustom(- image: YoloImageInput,- ): Promise<YoloClassifyResult> {- if (!this.customModel) {- throw new Error("No custom model configured");- }- const session = await this.customSession();- const imgsz = this.customModel.imgsz ?? 224;- const { tensor } = letterboxToTensor(image, imgsz);- const outputs = await session.run({- name: "images",- data: tensor,- dims: [1, 3, imgsz, imgsz],- });- const preds = pickByDims(outputs, 2);- return decodeClassifyOutput(preds.data, this.customModel.classes.length);- }-- private customSession(): Promise<YoloSession> {- if (!this.customModel) {- throw new Error("No customModel specified in YoloVisionRunner options");- }- if (!this._customSessionPromise) {- const cm = this.customModel;- const imgsz = cm.imgsz ?? this.imgsz;- this._customSessionPromise = this.loadCustomModelBytes(cm).then((bytes) =>- this._provider.createSession(bytes, { imgsz }),- );- }- return this._customSessionPromise;- }-- private async loadCustomModelBytes(- cm: CustomModelConfig,- ): Promise<Uint8Array> {- if (cm.path) {- return new Uint8Array(await readFile(cm.path));- }- if (cm.url) {- return this._store.fetchDirect(cm.url);- }- throw new Error("CustomModelConfig must specify either path or url");- }-- // ── internals ──────────────────────────────────────────────────────────-- private session(key: YoloModelKey): Promise<YoloSession> {- let promise = this._sessions.get(key);- if (!promise) {- const modelImgsz = YOLO_MODELS[key].imgsz ?? this.imgsz;- promise = this._store- .ensure(key)- .then((bytes) =>- this._provider.createSession(bytes, { imgsz: modelImgsz }),- )- .catch((err: unknown) => {- // Drop both the session cache entry AND the cached file: a parse- // failure means the on-disk bytes are corrupt — the next call must- // re-download instead of replaying the same bad file.- this._sessions.delete(key);- void this._store.evict(key);- throw err;- });- this._sessions.set(key, promise);- }- return promise;- }-- private prepareInput(- image: YoloImageInput,- key: YoloModelKey,- ): {- tensor: Float32Array;- params: ReturnType<typeof letterboxToTensor>["params"];- } {- const imgsz = YOLO_MODELS[key].imgsz ?? this.imgsz;- let scratch = this._tensorScratch.get(key);- if (!scratch || scratch.length < 3 * imgsz * imgsz) {- scratch = new Float32Array(3 * imgsz * imgsz);- this._tensorScratch.set(key, scratch);- }- return letterboxToTensor(normalizeImage(image), imgsz, scratch);- }-- private decodeOptions(- image: YoloImageInput,- params: ReturnType<typeof letterboxToTensor>["params"],- task: YoloTask = "detect",- ): DetectDecodeOptions {- return {- confidence: this.confidence,- iouThreshold: this.iouThreshold,- classes: this.classes,- classNames: task === "obb" ? YOLO_DOTA_CLASSES : YOLO_COCO_CLASSES,- params,- sourceWidth: image.width,- sourceHeight: image.height,- maskThreshold: this.maskThreshold,- featherRadius: this.featherRadius,- };- }-}--function normalizeImage(image: YoloImageInput): RgbaImage {- if (image.data instanceof Uint8ClampedArray) {- return image;- }- // Uint8Array may be a view into a larger buffer (WebGPU readback) — keep it as-is;- // the letterbox reads via indexing, so views are fine.- return image as RgbaImage;-}--function pickByDims(- outputs: Record<string, { data: Float32Array; dims: readonly number[] }>,- rank: number,-): { data: Float32Array; dims: readonly number[] } {- const hit = Object.values(outputs).find((t) => t.dims.length === rank);- if (!hit) {- throw new Error(- `YOLO model returned no ${rank}D output — got [${Object.values(outputs)- .map((t) => t.dims.join("x"))- .join(", ")}]. Check the exported model task head.`,- );- }- return hit;-}--function hashPromptString(str: string): number {- let h = 2166136261 >>> 0;- for (let i = 0; i < str.length; i++) {- h = Math.imul(h ^ str.charCodeAt(i), 16777619) >>> 0;- }- return h;-}--function encodePromptEmbedding(prompt: string): Float32Array {- let seed = hashPromptString(prompt);- const vec = new Float32Array(512);- let norm = 0;- for (let i = 0; i < 512; i++) {- seed = (seed + 0x6d2b79f5) >>> 0;- let t = seed;- t = Math.imul(t ^ (t >>> 15), t | 1);- t ^= t + Math.imul(t ^ (t >>> 7), t | 61);- const val = ((t ^ (t >>> 14)) >>> 0) / 4294967296 - 0.5;- vec[i] = val;- norm += val * val;- }- const invNorm = 1 / (Math.sqrt(norm) || 1);- for (let i = 0; i < 512; i++) {- vec[i] *= invNorm;- }- return vec;-}diff --git a/packages/yolo/src/schemas.test.ts b/packages/yolo/src/schemas.test.tsdeleted file mode 100644index d9041fa9..00000000--- a/packages/yolo/src/schemas.test.ts+++ /dev/null@@ -1,144 +0,0 @@-import { describe, expect, it } from "vitest";-import {- VisionAnalysisReportSchema,- YoloConfigSchema,- YoloOBBResultSchema,- YoloPoseResultSchema,- YoloSummarySchema,-} from "./schemas.js";--describe("yolo schemas", () => {- it("validates a frame summary", () => {- const summary = {- frame: 12,- objects: [- {- trackId: 1,- category: "person",- score: 0.91,- center: [540, 312] as [number, number],- speed: 128,- active: true,- },- ],- classes: ["person"],- masks: { person: 1 },- };- expect(YoloSummarySchema.safeParse(summary).success).toBe(true);- expect(YoloSummarySchema.safeParse({ ...summary, frame: -1 }).success).toBe(- false,- );- });-- it("validates the runtime config surface", () => {- expect(- YoloConfigSchema.safeParse({- enableObb: true,- variant: "s",- confidence: 0.4,- }).success,- ).toBe(true);- expect(YoloConfigSchema.safeParse({ confidence: 1.5 }).success).toBe(false);- expect(YoloConfigSchema.safeParse({ variant: "z" }).success).toBe(false);- });-- it("validates OBB detections with 4 corners and an angle", () => {- const ok = {- detections: [- {- category: "plane",- classIndex: 4,- score: 0.8,- boundingBox: {- originX: 1,- originY: 2,- width: 3,- height: 4,- normalizedX: 0.1,- normalizedY: 0.2,- normalizedWidth: 0.3,- normalizedHeight: 0.4,- angle: 0.5,- },- corners: [- [0, 0],- [1, 0],- [1, 1],- [0, 1],- ],- },- ],- };- expect(YoloOBBResultSchema.safeParse(ok).success).toBe(true);- expect(- YoloOBBResultSchema.safeParse({- detections: [{ ...ok.detections[0], corners: [[0, 0]] }],- }).success,- ).toBe(false);- });-- it("validates a pose result", () => {- const keypoints = Array.from({ length: 17 }, (_, i) => ({- x: i,- y: i,- visibility: 0.9,- }));- expect(- YoloPoseResultSchema.safeParse({- people: [- {- score: 0.9,- boundingBox: {- originX: 0,- originY: 0,- width: 10,- height: 10,- normalizedX: 0,- normalizedY: 0,- normalizedWidth: 0.1,- normalizedHeight: 0.1,- },- keypoints,- },- ],- }).success,- ).toBe(true);- expect(- YoloPoseResultSchema.safeParse({ people: [{ score: 1, keypoints: [] }] })- .success,- ).toBe(false);- });-- it("validates an analysis report", () => {- expect(- VisionAnalysisReportSchema.safeParse({- source: "clip.mp4",- totalFrames: 120,- fps: 24,- durationSec: 5,- tasks: ["detect"],- modelDownloads: { yolo11n: { bytes: 10_723_904, ms: 812 } },- inferenceMs: 4120,- trackCount: 1,- classes: {- person: {- framesPresent: 120,- totalFrames: 120,- maxConfidence: 0.94,- tracks: 1,- },- },- tracks: [- {- trackId: 1,- category: "person",- frames: [0, 119],- speedMeanPxS: 132,- centerPath: [[1, 2]],- },- ],- masks: { person: { meanCoverage: 0.18 } },- }).success,- ).toBe(true);- });-});diff --git a/packages/yolo/src/yolo-node.test.ts b/packages/yolo/src/yolo-node.test.tsdeleted file mode 100644index bedc072f..00000000--- a/packages/yolo/src/yolo-node.test.ts+++ /dev/null@@ -1,87 +0,0 @@-import { mkdtempSync } from "node:fs";-import { tmpdir } from "node:os";-import { join } from "node:path";-import { describe, expect, it } from "vitest";-import type {- SessionProvider,- YoloSession,- YoloTensorInput,-} from "./runtime/session-provider.js";-import { YoloNode } from "./yolo-node.js";--/** Minimal fake session provider — keeps the node test fully offline. */-class FakeProvider implements SessionProvider {- public readonly kind = "node" as const;- public created = 0;- async createSession(): Promise<YoloSession> {- this.created++;- return {- run: async (_input: YoloTensorInput) => ({- output0: { data: new Float32Array(0), dims: [1, 5, 0] },- }),- release: async () => {},- };- }-}--describe("YoloNode (lazy + composition AST)", () => {- it("constructor performs zero I/O and creates the signals bundle", () => {- const node = new YoloNode("https://example.test/video.mp4", {- enableDetection: true,- enablePose: true,- });- expect(node.kind).toBe("yolo");- expect(node.config.enableDetection).toBe(true);- expect(node.vision).toBeDefined();- expect(node.vision.objects).toBeDefined();- expect(node.vision.poseLandmarks).toBeDefined();- // runner not created until demanded- expect(node["_runner"]).toBeUndefined();- });-- it("toNode() + attach() produce a composition-ready AST with the same identity", () => {- const node = new YoloNode("clip.mp4", { enableSegmentation: true });- const ast = node.toNode();- expect(ast.kind).toBe("yolo");- expect(ast.source).toBe("clip.mp4");- expect(ast.config.enableSegmentation).toBe(true);-- const attached = YoloNode.attach("clip.mp4", { enableSegmentation: true });- expect(attached.node.kind).toBe("yolo");- expect(attached.node.source).toBe("clip.mp4");- // lazy helpers are exposed on the bundle- expect(typeof (attached as unknown as { ready?: unknown }).ready).toBe(- "function",- );- expect(typeof (attached as unknown as { runner?: unknown }).runner).toBe(- "function",- );- });-- it("ready() preloads only the enabled tasks' models", async () => {- const provider = new FakeProvider();- const node = new YoloNode(- "clip.mp4",- { enablePose: true }, // only pose- {},- { provider, modelsDir: mkdtempSync(join(tmpdir(), "yolo-node-")) },- );- let fetches = 0;- const origFetch = globalThis.fetch;- globalThis.fetch = (async () => {- fetches++;- return new Response(new Uint8Array(4096));- }) as typeof fetch;- try {- const runner = await node.ready();- expect(fetches).toBe(1); // only the pose model- expect(provider.created).toBe(1);- expect(runner.downloadStatus.get("yolo11n-pose")).toBe("ready");- expect(runner.downloadStatus.get("yolo11n")).toBe("pending");- node.close();- expect(node["_runner"]).toBeUndefined();- } finally {- globalThis.fetch = origFetch;- }- });-});diff --git a/packages/yolo/src/yolo-node.ts b/packages/yolo/src/yolo-node.tsdeleted file mode 100644index 7259e5de..00000000--- a/packages/yolo/src/yolo-node.ts+++ /dev/null@@ -1,166 +0,0 @@-import type { YoloConfig, YoloNode as YoloNodeAST } from "@gitframes/core";-import type { YoloTask } from "./model/registry.js";-import {- type YoloRunnerOptions,- YoloVisionRunner,-} from "./runner/yolo-runner.js";-import {- YoloVisionBundle,- type YoloVisionBundleOptions,-} from "./signals/yolo-signals-bundle.js";-import type { YoloInputSource } from "./types.js";--/**- * Bundle returned by `YoloNode.attach()`: the reactive signals surface plus the lazy lifecycle- * helpers (all zero-I/O until `runner()` really needs a model).- */-export interface YoloAttachedBundle extends YoloVisionBundle {- readonly node: YoloNodeAST;- /** Warm the enabled tasks' models/sessions ahead of first use. */- ready(): Promise<YoloVisionRunner>;- /** Lazily-created runner (downloads happen on first real inference). */- runner(): YoloVisionRunner;- /** Release this node's runner/sessions. */- close(): void;-}--/**- * YOLO vision node.- *- * LAZY BY CONSTRUCTION: creating a YoloNode performs ZERO I/O — no model- * downloads, no sessions, no file probes. The first inference call (or an- * explicit `await vision.ready()`) downloads the required models.- */-export class YoloNode {- public readonly id: string;- public readonly kind = "yolo" as const;- public readonly source: YoloInputSource;- public readonly config: YoloConfig;- public readonly vision: YoloVisionBundle;-- private _runner?: YoloVisionRunner;-- /** Runner construction options (modelsDir, provider, …) — injectable for tests/agents. */- private _runnerOptions: YoloRunnerOptions;-- constructor(- source: YoloInputSource,- config: YoloConfig = {},- bundleOptions: YoloVisionBundleOptions = {},- runnerOptions: YoloRunnerOptions = {},- ) {- this.id = "yolo-" + Math.random().toString(36).slice(2, 9);- this.source = source;- this.config = config;- this._runnerOptions = runnerOptions;- this.vision = new YoloVisionBundle({- ...bundleOptions,- config,- });- }-- /** Lazily-created runner; downloads happen here on first access. */- public runner(): YoloVisionRunner {- if (!this._runner) {- const {- variant,- imgsz,- confidence,- iouThreshold,- classes,- enableWorld,- prompts,- customModel,- modelsDir,- baseUrl,- maskThreshold,- featherRadius,- } = this.config;- // config values win over injected runner options, but only when set —- // an explicit undefined must not clobber an injected modelsDir/provider.- this._runner = YoloVisionRunner.create({- ...this._runnerOptions,- ...(variant !== undefined ? { variant } : {}),- ...(imgsz !== undefined ? { imgsz } : {}),- ...(confidence !== undefined ? { confidence } : {}),- ...(iouThreshold !== undefined ? { iouThreshold } : {}),- ...(classes !== undefined ? { classes } : {}),- ...(enableWorld !== undefined ? { enableWorld } : {}),- ...(prompts !== undefined ? { prompts } : {}),- ...(customModel !== undefined ? { customModel } : {}),- ...(modelsDir !== undefined ? { modelsDir } : {}),- ...(baseUrl !== undefined ? { baseUrl } : {}),- ...(maskThreshold !== undefined ? { maskThreshold } : {}),- ...(featherRadius !== undefined ? { featherRadius } : {}),- });- }- return this._runner;- }-- /** Explicit warm-up — downloads the models for the enabled tasks, ahead of first inference. */- public async ready(): Promise<YoloVisionRunner> {- const runner = this.runner();- const tasks: YoloTask[] = [];- if (this.config.enableDetection === true) tasks.push("detect");- if (this.config.enableSegmentation === true) tasks.push("segment");- if (this.config.enablePose === true) tasks.push("pose");- if (this.config.enableObb === true) tasks.push("obb");- if (this.config.enableClassification === true) tasks.push("classify");- if (this.config.enableWorld === true) tasks.push("world");- // No task explicitly enabled → default det or world task- const defaultTask = this.config.enableWorld === true ? "world" : "detect";- await runner.preload(tasks.length > 0 ? tasks : [defaultTask]);- return runner;- }-- public close(): void {- this._runner?.close();- this._runner = undefined;- }-- public static attach(- source: YoloInputSource,- config: YoloConfig = {},- bundleOptions: YoloVisionBundleOptions = {},- ): YoloAttachedBundle {- const nodeInstance = new YoloNode(source, config, bundleOptions);- const vision = nodeInstance.vision;-- // Attach AST node property to the vision bundle for composition graph inclusion- Object.defineProperty(vision, "node", {- value: {- id: nodeInstance.id,- kind: "yolo",- source: nodeInstance.source,- config: nodeInstance.config,- } as YoloNodeAST,- enumerable: true,- writable: false,- });-- // Surface the lazy ready()/runner() helpers on the bundle for agent DX- Object.defineProperty(vision, "ready", {- value: () => nodeInstance.ready(),- enumerable: false,- });- Object.defineProperty(vision, "runner", {- value: () => nodeInstance.runner(),- enumerable: false,- });- Object.defineProperty(vision, "close", {- value: () => nodeInstance.close(),- enumerable: false,- });-- return vision as YoloAttachedBundle;- }-- public toNode(): YoloNodeAST {- return {- id: this.id,- kind: "yolo",- source: this.source,- config: this.config,- };- }-}diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex ad07ab46..faf36bc7 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -1997,7 +1997,7 @@ importers:specifier: ^0.4.0version: 0.4.0- nodes/node-yolo:+ nodes/node-vision:dependencies:'@gitframes/core':specifier: workspace:*@@ -2005,9 +2005,9 @@ importers:'@gitframes/node-sdk':specifier: workspace:*version: link:../../packages/node-sdk- '@gitframes/yolo':+ '@gitframes/vision':specifier: workspace:*- version: link:../../packages/yolo+ version: link:../../packages/visionzod:specifier: ^4.3.6version: 4.3.6@@ -2316,9 +2316,9 @@ importers:'@gitframes/node-vignette':specifier: workspace:*version: link:../../nodes/node-vignette- '@gitframes/node-yolo':+ '@gitframes/node-vision':specifier: workspace:*- version: link:../../nodes/node-yolo+ version: link:../../nodes/node-vision'@gitframes/renderer':specifier: workspace:*version: link:../renderer@@ -2331,12 +2331,12 @@ importers:'@gitframes/tsconfig':specifier: workspace:*version: link:../tsconfig+ '@gitframes/vision':+ specifier: workspace:*+ version: link:../vision'@gitframes/webgpu-renderers':specifier: workspace:*version: link:../webgpu-renderers- '@gitframes/yolo':- specifier: workspace:*- version: link:../yolo'@types/fontkit':specifier: ^2.0.9version: 2.0.9@@ -2583,6 +2583,46 @@ importers:packages/tsconfig: {}+ packages/vision:+ dependencies:+ '@gitframes/core':+ specifier: workspace:*+ version: link:../core+ '@gitframes/tensor-webgpu':+ specifier: workspace:*+ version: link:../tensor-webgpu+ onnxruntime-node:+ specifier: ^1.24.0+ version: 1.30.0+ onnxruntime-web:+ specifier: ^1.24.0+ version: 1.30.0+ webgpu:+ specifier: ^0.4.0+ version: 0.4.0+ zod:+ specifier: ^4.3.6+ version: 4.3.6+ devDependencies:+ '@gitframes/tsconfig':+ specifier: workspace:*+ version: link:../tsconfig+ '@types/node':+ specifier: ^24.1.0+ version: 24.10.14+ skia-canvas:+ specifier: ^3.0.8+ version: 3.0.8+ tsdown:+ specifier: ^0.19.0+ version: 0.19.0([email protected])([email protected])+ typescript:+ specifier: ^5.8.3+ version: 5.9.3+ vitest:+ specifier: ^4.1.5+ version: 4.1.5(@opentelemetry/[email protected])(@types/[email protected])(@vitest/[email protected])([email protected](@noble/[email protected])([email protected]))([email protected](@types/[email protected])([email protected])([email protected])([email protected])([email protected])([email protected]))+packages/webgpu-renderers:dependencies:'@gitframes/client-utils':@@ -2653,43 +2693,6 @@ importers:specifier: ^4.0.0version: 4.0.0([email protected])([email protected])- packages/yolo:- dependencies:- '@gitframes/core':- specifier: workspace:*- version: link:../core- '@gitframes/tensor-webgpu':- specifier: workspace:*- version: link:../tensor-webgpu- onnxruntime-node:- specifier: ^1.24.0- version: 1.30.0- onnxruntime-web:- specifier: ^1.24.0- version: 1.30.0- webgpu:- specifier: ^0.4.0- version: 0.4.0- zod:- specifier: ^4.3.6- version: 4.3.6- devDependencies:- '@gitframes/tsconfig':- specifier: workspace:*- version: link:../tsconfig- '@types/node':- specifier: ^24.1.0- version: 24.10.14- tsdown:- specifier: ^0.19.0- version: 0.19.0([email protected])([email protected])- typescript:- specifier: ^5.8.3- version: 5.9.3- vitest:- specifier: ^4.1.5- version: 4.1.5(@opentelemetry/[email protected])(@types/[email protected])(@vitest/[email protected])([email protected](@noble/[email protected])([email protected]))([email protected](@types/[email protected])([email protected])([email protected])([email protected])([email protected])([email protected]))-packages:'@asamuzakjp/[email protected]':@@ -3052,9 +3055,6 @@ packages:'@emnapi/[email protected]':resolution: {integrity: sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==}- '@emnapi/[email protected]':- resolution: {integrity: sha512-mehfKSMWjjNol8659Z8KxEMrdSJDDot5SXMq00dM8BN4o+CLNXQ0xH2V7EchNHV4RmbZLmmPdEaXZc5H2FXmDg==}-'@emnapi/[email protected]':resolution: {integrity: sha512-WI0DdZ8xFSbgMjR1sFsKABJ/C5OnRrjT06JXbZKexJGrDuPTzZdDYfFlsgcCXCyf+suG5QU2e/y1Wo2V/OapLQ==}@@ -5981,18 +5981,10 @@ packages:resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}- resolution: {integrity: sha512-W/KYk+NFhkmsYpuHq5JykngiOCnxeVL8v8dFnqxSD8qEEdRfXk1SDM6JzNqcERbcGYj9tMrDQBYV9cjgnunFIg==}- engines: {node: '>=18'}-resolution: {integrity: sha512-QKAl9m8gWWGHV8jZcPeym6j+XULi6tOf1mT83WYJ4Lk2ytW/uwAWkrP0uFsdoYMdueVJ0qs26wZ+23xeB4ibNQ==}engines: {node: '>=18'}- resolution: {integrity: sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ==}- engines: {node: '>=12.0.0'}-resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==}engines: {node: '>=12.0.0'}@@ -6954,7 +6946,7 @@ snapshots:outdent: 0.5.0prettier: 2.8.8resolve-from: 5.0.0- semver: 7.7.4+ semver: 7.8.5'@changesets/[email protected]':dependencies:@@ -6963,7 +6955,7 @@ snapshots:'@changesets/should-skip-package': 0.1.2'@changesets/types': 6.1.0'@manypkg/get-packages': 1.1.3- semver: 7.7.4+ semver: 7.8.5'@changesets/[email protected]':dependencies:@@ -7021,7 +7013,7 @@ snapshots:'@changesets/types': 6.1.0'@manypkg/get-packages': 1.1.3picocolors: 1.1.1- semver: 7.7.4+ semver: 7.8.5'@changesets/[email protected]':dependencies:@@ -7119,11 +7111,6 @@ snapshots:tslib: 2.8.1optional: true- '@emnapi/[email protected]':- dependencies:- tslib: 2.8.1- optional: true-'@emnapi/[email protected]':dependencies:tslib: 2.8.1@@ -7477,7 +7464,7 @@ snapshots:'@napi-rs/[email protected]':dependencies:'@emnapi/core': 1.8.1- '@emnapi/runtime': 1.8.1+ '@emnapi/runtime': 1.11.3'@tybys/wasm-util': 0.10.1optional: true@@ -9934,15 +9921,8 @@ snapshots:- [email protected]: {}-- dependencies:- fdir: 6.5.0([email protected])- picomatch: 4.0.7-dependencies:fdir: 6.5.0([email protected])@@ -9994,13 +9974,13 @@ snapshots:empathic: 2.0.0hookable: 6.0.1import-without-cache: 0.2.5- obug: 2.1.1- picomatch: 4.0.5+ obug: 2.2.1+ picomatch: 4.0.7rolldown: 1.0.0-beta.59- semver: 7.7.4- tinyexec: 1.0.2- tinyglobby: 0.2.15+ semver: 7.8.5+ tinyexec: 1.3.0+ tinyglobby: 0.2.17tree-kill: 1.2.2unconfig-core: 7.5.0unrun: 0.2.28@@ -10021,13 +10001,13 @@ snapshots:empathic: 2.0.0hookable: 6.0.1import-without-cache: 0.2.5- obug: 2.1.1- picomatch: 4.0.5+ obug: 2.2.1+ picomatch: 4.0.7rolldown: 1.0.0-beta.59- semver: 7.7.4- tinyexec: 1.0.2- tinyglobby: 0.2.15+ semver: 7.8.5+ tinyexec: 1.3.0+ tinyglobby: 0.2.17tree-kill: 1.2.2unconfig-core: 7.5.0unrun: 0.2.28diff --git a/skills/gitframes-effects/SKILL.md b/skills/gitframes-effects/SKILL.mdindex 2faead5a..f7cd08bb 100644--- a/skills/gitframes-effects/SKILL.md+++ b/skills/gitframes-effects/SKILL.md@@ -1,6 +1,6 @@---name: gitframes-effects-description: Apply, configure, and modulate WebGPU post-processing shaders, cinematic color grading, tone mapping, procedural VFX, and YOLO neural vision conditioning in gitframes. Use when adding blur, grain, LUTs, relighting, curves, greenscreen color keying, or subject segmentation in gitframes compositions.+description: Apply, configure, and modulate WebGPU post-processing shaders, cinematic color grading, tone mapping, procedural VFX, and on-device vision (object tracking, instance segmentation, pose, person mattes) in gitframes. Use when adding blur, grain, LUTs, relighting, curves, greenscreen color keying, subject cutouts, or tracking-driven effects in gitframes compositions.---# gitframes-effects@@ -118,9 +118,61 @@ const pngBuffer = await clip.renderFrame({ atMs: 1500 });----## YOLO11 Neural Vision Conditioning+## On-Device Vision-`gitframes` integrates native WebGPU YOLO11 neural conditioning directly into the rendering pipeline. Models are downloaded and loaded lazily upon first invocation.+`gitframes` runs vision models on the rendered frame and exposes the results as reactive signals. Models download lazily on first use and are cached in `$GITFRAMES_MODELS_DIR` (default `~/.cache/gitframes/models`).++| Option | Model | Gives you |+| --- | --- | --- |+| `enableDetection` (default) | RTMDet-Ins | COCO-80 boxes, tracked over time → `vision.objects` |+| `enableSegmentation` | RTMDet-Ins (same pass) | Soft per-instance masks → `vision.masks`, `vision.segmentation` |+| `enablePose` | RTMO | 17 COCO keypoints per person → `vision.poseLandmarks`, `track.pose` |+| `enableMatte` | Selfie Segmenter | Fast person alpha → `vision.segmentation.matte` |++`variant: "t" | "s" | "m"` trades speed for accuracy (default `"s"`). `confidence` (default 0.3) and `classes: ["person", ...]` filter detections.++```typescript+// Whole composition: returns the reactive bundle+const vision = comp.withVision({ enableSegmentation: true, enablePose: true });++// One layer: renders through a node mode+Layer.video("assets/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });+```++Node modes: `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`. For `mask` / `matte` / `crop`, `matteSource: "instance"` (default; any COCO class, overlapping parts such as a dress merged into the subject) or `"selfie"` (people only, fastest).++### Choosing settings++Measured per 1280–2048 px frame on CPU (`onnxruntime-node`):++| Task | `t` | `s` (default) | `m` |+| --- | --- | --- | --- |+| detect / segment (one shared pass) | 150–300 ms | 200–340 ms | 420–700 ms |+| pose | ~50 ms | ~120 ms | ~280 ms |+| matte (Selfie Segmenter) | ~20 ms | ~20 ms | ~20 ms |++- Start with `"s"`. Use `"m"` for hero shots with fast motion blur or busy backgrounds; `"t"` mainly saves time on pose.+- Enabling both `enableDetection` and `enableSegmentation` costs one inference, not two.+- **Selfie matte only for close framing.** It is tuned for a person filling much of the frame: it misses distant figures and can report a "person" on close-ups with nobody in them. Keep `matteSource: "instance"` for anything else.+- Raise `confidence` (e.g. `0.5`) on abstract or stylized footage — at the default `0.3` the model will put loose labels ("teddy bear", "donut") on smoke, eyes and planets.+- Restrict `classes` when you only care about one thing; it also stops the subject from switching to another object when the person leaves frame.++### How it behaves at render time++- **One-frame delay.** Vision reads the layer's *previous* rendered frame, so the very first frame has no results (a cutout renders transparent) and masks trail the plate by one frame. This is invisible at normal playback speed.+- **Render vision frames in order.** `renderVideo` does this for you. For stills, render at least two consecutive frames with the same renderer and keep the last one. Jumping straight to a later frame (or using a frame grid) cuts out pixels from whatever frame was rendered before it, which shows up as a ghosted double of the subject.+- **Subject choice.** `mask` / `matte` / `crop` use the largest person; with no person, the most confident instance. Instances overlapping the subject and no more than twice its size are merged in (a dress, a held instrument) — large containers around it (a tunnel, a window frame) are not.+- Each vision layer tracks objects independently; track ids from two layers are unrelated.++### Troubleshooting++| Symptom | Fix |+| --- | --- |+| Part of a fast-moving garment drops out of the cutout | `variant: "m"`, or `keyBackground: true` to grow the subject into connected foreground |+| Cutout is the wrong object | Set `classes: ["person"]` (or the class you want) |+| Faint halo around the cutout on dark backgrounds | Lower `featherRadius`, or raise `maskThreshold` (e.g. `0.6`) |+| Renders offline / in CI | Pre-download with `runner.preload([...])` into `$GITFRAMES_MODELS_DIR`, or point `GITFRAMES_MODELS_BASE_URL` at a mirror |+| Check the models themselves | `pnpm --filter @gitframes/vision test:models` (downloads ~380 MB once) |### 1. Subject Sandwich ("Text Behind Subject")Cuts out the foreground subject from footage and sandwiches typography or graphics directly behind them:@@ -141,10 +193,12 @@ comp.addSubjectSandwich({```### 2. Smart Re-Framing (16:9 to 9:16 Auto-Crop)-Smoothly reframes landscape video into vertical shorts by tracking the focal subject with virtual camera damping:+Smoothly reframes landscape video into vertical shorts by following a tracked subject with virtual camera damping:```typescript+const vision = comp.withVision({ enableDetection: true });comp.addSmartFraming({source: "assets/action.mp4",+ target: vision.objects.primary,targetAspect: 9 / 16,damping: 0.15,leadHeadroom: 0.1@@ -154,7 +208,8 @@ comp.addSmartFraming({### 3. Subject Outline Glow & Neon PulseStrokes the segmented subject boundary with an audio-reactive contour glow:```typescript-comp.addSubjectOutline({+const vision = comp.withVision({ enableSegmentation: true });+comp.addSubjectOutline(vision.segmentation.subject, {source: "assets/character.mp4",color: "#FF5A1F",width: 6,@@ -163,16 +218,20 @@ comp.addSubjectOutline({```### 4. Tracked Region Blur-Automatically blurs faces, license plates, or specific detected classes:+Blurs faces, license plates, or any detected class by following a live track:```typescript-const runner = await comp.analyzeVisionSequence("assets/street.mp4", {- tasks: ["detect"],+const vision = comp.withVision({ classes: ["person"] });+layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });+```++### 5. Analyze Before Authoring+One-shot, ffmpeg-free report of what is in a clip (tracks, classes, mask coverage):+```typescript+const report = await comp.analyzeVisionSequence("assets/street.mp4", {+ tasks: ["detect", "pose"],categories: ["person"]});--// Primary detected track automatically blurred-const primaryTrack = runner.tracks[0];-layer.blurRegion(primaryTrack, { strength: 30 });+report.tracks; // [{ trackId, category, frames: [start, end], centerPath, ... }]```---diff --git a/skills/gitframes-effects/references/effects-catalog.md b/skills/gitframes-effects/references/effects-catalog.mdindex b05a41c5..c6d32c86 100644--- a/skills/gitframes-effects/references/effects-catalog.md+++ b/skills/gitframes-effects/references/effects-catalog.md@@ -42,7 +42,7 @@ Set `GITFRAMES_STRICT_PROPS=1` to make an unknown prop or an out-of-range value| `PBRGlass` | `PBRGlass` | `ior`, `roughness`, `transmission`, `dispersion`, `fresnelPower`, `tintColor` || `DepthOfField` | `DepthOfField` | `focusDistance`, `focalLength`, `aperture`, `maxBlur`, `quality` || `Crop` | `Crop` | `cropType`, `leftPercentage`, `topPercentage`, `widthPercentage`, `heightPercentage`, `roundness`, `pathPoints`, `mode` |-| `Yolo` | `Yolo` | `enableDetection`, `enableSegmentation`, `enablePose`, `enableClassification`, `enableObb`, `enableWorld`, `prompts`, `customModel`, `classes`, `confidence`, `iouThreshold`, `variant`, `imgsz`, `mode`, `delegate`, `modelsDir`, `baseUrl`, `maskThreshold`, `featherRadius`, `keyBackground`, `backgroundKeyThreshold`, `visionBundle` |+| `Vision` | `Vision` | `enableDetection`, `enableSegmentation`, `enablePose`, `enableMatte`, `classes`, `confidence`, `variant`, `mode`, `matteSource`, `maxMissedFrames`, `modelsDir`, `baseUrl`, `maskThreshold`, `featherRadius`, `keyBackground`, `backgroundKeyThreshold`, `visionBundle` || `Relight3D` | `Relight3D` | `lightType`, `intensity`, `lightPosX`, `lightPosY`, `lightPosZ`, `lightRadius`, `spotConeAngle`, `specularRoughness`, `specularStrength`, `metallic`, `ambientIntensity`, `volumetricDensity`, `depthScale`, `depthInvert`, `lightColor`, `ambientColor`, `normalMap` || `TemporalDeflicker` | `TemporalDeflicker` | `blendWeight`, `disocclusionThreshold`, `maxMotionPixels`, `scale`, `windowSize` || `Modulate` | `Modulate` | `hue`, `brightness`, `contrast`, `exposure`, `saturation`, `sepia` |@@ -418,12 +418,12 @@ new Crop({ cropType: "rect", leftPercentage: 0, topPercentage: 0 });| `pathPoints` | `{ x: number; y: number }[]` | — || `mode` | `"cropped" \| "rest"` | — |-### Yolo+### Vision```ts-import { Yolo } from "gitframes/effects";+import { Vision } from "gitframes/effects";-new Yolo({ confidence: 0.25, iouThreshold: 0.45, variant: "n" });+new Vision({ confidence: 0.3, variant: "s", mode: "passthrough" });```| Prop | Type | Default |@@ -431,18 +431,13 @@ new Yolo({ confidence: 0.25, iouThreshold: 0.45, variant: "n" });| `enableDetection` | `boolean` | — || `enableSegmentation` | `boolean` | — || `enablePose` | `boolean` | — |-| `enableClassification` | `boolean` | — |-| `enableObb` | `boolean` | — |-| `enableWorld` | `boolean` | — |-| `prompts` | `readonly string[]` | — |-| `customModel` | `CustomModelConfig` | — |+| `enableMatte` | `boolean` | — || `classes` | `readonly string[]` | — |-| `confidence` | `number` | `0.25` |-| `iouThreshold` | `number` | `0.45` |-| `variant` | `"n" \| "s" \| "m" \| "l" \| "x"` | `"n"` |-| `imgsz` | `number` | `640` |-| `mode` | `\| "passthrough" \| "mask" \| "matte" \| "crop" \| "skeleton" \| "boxes" \| "tracking" \| "obb"` | `"passthrough"` |-| `delegate` | `"CPU" \| "GPU"` | `"CPU"` |+| `confidence` | `number` | `0.3` |+| `variant` | `"t" \| "s" \| "m"` | `"s"` |+| `mode` | `\| "passthrough" \| "mask" \| "matte" \| "crop" \| "skeleton" \| "boxes" \| "tracking"` | `"passthrough"` |+| `matteSource` | `"instance" \| "selfie"` | `"instance"` |+| `maxMissedFrames` | `number` | — || `modelsDir` | `string` | — || `baseUrl` | `string` | — || `maskThreshold` | `number` | — |@@ -517,4 +512,4 @@ new Modulate({ hue: 0, brightness: 1.0, contrast: 1.0 });Beyond these classes, the engine ships node plugins under `nodes/` (audio DSP, generators,signal maths, vision). Each is a package named `@gitframes/node-<name>`:-`node-apply-lut`, `node-audio-compressor`, `node-audio-delay`, `node-audio-fade`, `node-audio-noise-gate`, `node-audio-parametric-eq`, `node-audio-reverb`, `node-audio-signal-extractor`, `node-blur`, `node-camera-parallax-3d`, `node-canvas-generator`, `node-caption-editor`, `node-caption-generator`, `node-channel-merger`, `node-channel-splitter`, `node-color-balance`, `node-colorkey`, `node-compositor`, `node-corner-pin`, `node-crop`, `node-curves`, `node-deflicker`, `node-displacement-map`, `node-extract-frame`, `node-extract-lut`, `node-film-grain`, `node-flip`, `node-gradient-map`, `node-halftone-screen`, `node-high-pass`, `node-kenburns`, `node-layer-style`, `node-levels`, `node-liquify`, `node-lottie`, `node-mask-math`, `node-mesh-warp`, `node-modulate`, `node-noise-generator`, `node-paint`, `node-patch-heal`, `node-procedural-vfx`, `node-refine-edge`, `node-refraction-caustics`, `node-relight-3d`, `node-resizer-scaler`, `node-selective-color`, `node-shadows-highlights`, `node-shape-generator`, `node-signal-gate`, `node-signal-math`, `node-signal`, `node-stereo-panning`, `node-svg`, `node-text`, `node-tile-offset`, `node-unsharp-mask`, `node-vignette`, `node-yolo`+`node-apply-lut`, `node-audio-compressor`, `node-audio-delay`, `node-audio-fade`, `node-audio-noise-gate`, `node-audio-parametric-eq`, `node-audio-reverb`, `node-audio-signal-extractor`, `node-blur`, `node-camera-parallax-3d`, `node-canvas-generator`, `node-caption-editor`, `node-caption-generator`, `node-channel-merger`, `node-channel-splitter`, `node-color-balance`, `node-colorkey`, `node-compositor`, `node-corner-pin`, `node-crop`, `node-curves`, `node-deflicker`, `node-displacement-map`, `node-extract-frame`, `node-extract-lut`, `node-film-grain`, `node-flip`, `node-gradient-map`, `node-halftone-screen`, `node-high-pass`, `node-kenburns`, `node-layer-style`, `node-levels`, `node-liquify`, `node-lottie`, `node-mask-math`, `node-mesh-warp`, `node-modulate`, `node-noise-generator`, `node-paint`, `node-patch-heal`, `node-procedural-vfx`, `node-refine-edge`, `node-refraction-caustics`, `node-relight-3d`, `node-resizer-scaler`, `node-selective-color`, `node-shadows-highlights`, `node-shape-generator`, `node-signal-gate`, `node-signal-math`, `node-signal`, `node-stereo-panning`, `node-svg`, `node-text`, `node-tile-offset`, `node-unsharp-mask`, `node-vignette`, `node-vision`diff --git a/skills/gitframes-render/SKILL.md b/skills/gitframes-render/SKILL.mdindex a1d20b68..d4acb7fe 100644--- a/skills/gitframes-render/SKILL.md+++ b/skills/gitframes-render/SKILL.md@@ -204,6 +204,8 @@ ffmpeg -i output/final-video.mp4 -af "volumedetect" -f null -| Entire frame renders black | Missing font registration or `NaN` in uniform | Ensure every font used in `Layer.text` is registered via `FontManager.register()` and listed in `comp.fonts`. Verify all signal outputs are finite numbers. || Media layer is invisible / blank | Bad input handle binding or zero opacity | Check that the media path exists and `opacity > 0`. If using an effect or section, ensure `fit: "cover"` and valid container dimensions. || Audio cuts out early | Missing `durationFrames` on audio layer | Explicitly assign `durationFrames: totalFrames` on `Layer.audio()`. |+| Vision cutout / mask is empty on a single `renderFrame` | Vision reads the previous rendered frame; frame 0 has none | Render two or more consecutive frames with the same `HeadlessMediaRenderer` and keep the last. `renderVideo` handles this automatically. |+| Ghosted double of the subject in a vision cutout | Frames rendered out of order (frame grid, or jumping from frame 3 to 48), so the cutout uses pixels from another frame | Verify vision layers with sequential frames or the exported MP4, not `renderFrameGrid`. || Script hangs after rendering | Background worker thread pool held open | Always include `process.exit(0)` at the end of render scripts. || Text looks misaligned or truncated | Layout node box smaller than text string | Set explicit wrapping `width` (e.g. `width: 1400`) or use `width: "fill"`. || Colors appear washed out / dark edges | Unpremultiplied alpha blending | The engine operates with premultiplied alpha. Ensure custom shaders use `srcFactor: "one"`. |