Skip to content

Code-first video & image, rendered natively on WebGPU.

4.7 KB
Raw Blame

Framefields

Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turbo monorepo, Node >=22, ESM, Biome (tabs, double quotes), tsdown builds, vitest tests. Rendering needs a GPU. Our aim is to improve the engine harness: no cheating or workarounds.

Working rules

  • Be brave; we can change anything. Aim for engineering elegance, not just goal completion.
  • Debug to the root cause. If a workaround is the only option, stop and ask.
  • No any. Named exports unless a default is required. Top-of-file imports (no await import unless required). Pass the cyclomatic complexity lint.
  • pnpm only; never hand-edit pnpm-lock.yaml.
  • Never run bare tsc (it emits into source). Use the package script (tsdown, or tsc --noEmit -p .).
  • Render scripts end with process.exit.
  • Plain console output, no colors. Don't use the user's name. Final answers concise.
  • Keep this file current when structure or dev patterns change but do not bloat it.

Repo map

  • packages/framefields: public SDK. Composition, Layer.*, LayerAnimation, Signal, effects (src/effects/classes.ts), FontManager, renderFrameGrid. src/index.ts is the export surface. CLI in src/cli/ (init, add, list, fonts add|list, check, api); check isolates builders in an IPC worker, --render adds GPU smoke checks, and api reads the build-generated type catalog. init never prompts or overwrites, and its file templates are strings in templates.ts (a test parses them).
  • packages/components: the component registry framefields add copies into user projects (shadcn-style). defineComponent/bind contract in src/core/contract.ts, shared helpers in src/core/ (keys, color, layout, surface, theme). Components import only framefields and ../core. Registry items declare font needs (e.g. emoji); CLI fonts are pinned by URL, size and SHA-256 in cli/fonts.ts. Prop docs in components.json are generated (drift test). Tree-shake size budgets are tested.
  • packages/compositions: Yoga layout (utils/layout/resolve-layout.ts), program schema (program/schema.ts), lint (program/lint.ts), animation/evaluator.
  • packages/webgpu-renderers: WGSL shaders, 2D/3D pipelines, Slug text (src/slug/), audio DSP.
  • packages/renderer: headless Node renderer (Dawn, WebCodecs/mediabunny).
  • packages/vision-sam: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), an optional peer of framefields. clip-vision.ts imports it only for comp.vision(..., { sam2 | sam3 }); keep runtime imports of it out of framefields (types only). Pins in src/models/registry.ts.
  • packages/core: shared types, Effect base, signals, emojiPresentation (the one UTS #51 rule used by layout, preload and lint).
  • Others: renderers, media, node-sdk, server-utils, client-utils, tensor-webgpu, vision, tsconfig.
  • nodes/node-*: one VFX/audio/layout node each (src/shared/, src/renderers/, src/index.ts). Only node-compositor has a SKILL.md; update it when Compositor behavior changes (guarded by skillmd-check.test.ts).
  • apps/renderer-service: HTTP render service (Docker, RunPod).
  • examples/NN_*: reference films. Study 19_framefields_film (structure), 22_framefields_launch (lyric sync), 28_physics_dancer (physics, vision, GPU particles) before writing one.
  • plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision,-cloud}: agent skills, symlinked from .agents/skills and .Codex/skills.
  • Package exports have a development condition pointing at src/; otherwise dist/, so pnpm build after changing a dependency.
  • Docs site is a separate repo: /Users/okanaslankan/framefields-dev/docs (Astro Starlight). docs/lab pins the published framefields version and builds the component demos; deploy with pnpm run deploy.

Commands

  • Build: pnpm build, or pnpm --filter <name> build.
  • Test: pnpm --filter <name> test (packages/renderer: test:vitest). Root pnpm test is slow.
  • Lint: pnpm exec biome check --write <files> (lefthook runs it on commit).
  • Example: pnpm --filter @framefields/example-19-framefields-film assets|render|test|typecheck.
  • Component gallery: pnpm --filter framefields gallery.
  • Skills: after engine API changes update the owning skill; pnpm --filter framefields test fails on stale generated effects or catalog. effects-catalog.md is generated by pnpm run sync:effects-catalog; never hand-edit.
  • Releases: changesets (.changeset/).

Local-only files

Gitignored, never commit: assets/fonts, *.mp4/mp3/wav/png/jpg, *.onnx, scratch*/, examples/output/, .hermes/, .env* (never print secrets). Put probes and logs under scratch/ or output/.