Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
Vendored Transformers.js WebGPU runtime
This directory packages the JavaScript and WASM runtime used by two local WebGPU paths in Chrome and by offline RAG's CPU/WASM semantic reranker:
- Apocalypse Mode / Settings -> local WebGPU chat offers Compass VLD
Tiny v5 (
webbrain-one/webbrain-compass-vld-tiny-v5-onnx, ~2.2 GB, the default), Compass Tiny v2.1 (webbrain-one/webbrain-compass-tiny-v2.1, ~1.87 GB) and Ternary Bonsai 2 2B / 4B in text and vision variants, used by the standalone-chat nuclear control and selectable as a normal chat provider. The underlying runtime also supports LFM2.5 and Nanbeige exports. An opt-in Bonsai 27B preset uses a separate vendored bitgpu worker, not this Transformers.js runtime; seesrc/chrome/vendor/bitgpu/README.webbrain.md. - Providers -> Vision -> LFM2.5-VL local fallback runs
LiquidAI/LFM2.5-VL-450M-ONNXas the dedicated screenshot sidecar. - Apocalypse Mode -> Offline RAG runs the explicitly downloaded, pinned
Xenova/multilingual-e5-smallq8 model in a separate CPU/WASM worker. Model weights remain optional and are never bundled or downloaded by a question.
Model weights are not bundled. Explicit download controls fetch the selected WebGPU model and store it in the browser cache; inference only uses cached models. The default preset is Compass VLD Tiny v5 (~2.2 GB), with underlying runtime support for LFM2.5 (2.6B, 1.2B Instruct, 1.2B Thinking, VL 1.6B, VL 3B) and Nanbeige4.2-3B. The selected preset is available through the nuclear control in standalone chat and can also be configured, downloaded, and selected directly in Settings -> Providers -> WebGPU. Reasoning presets supported by the runtime keep completed thinking out of visible answers. Current LFM2.5-VL layouts use:
embed_tokens: FP16vision_encoder: FP16decoder_model_merged: Q4
Ternary Bonsai 2 uses Q2/FP16 language graphs, the existing Qwen3.5 hybrid
attention runtime, and (for the Vision presets) an FP16 vision encoder. Its
reproducible export is in scripts/models/bonsai/. Danyalmalik's 2B/4B split ports
omit tokenizer and generation sidecars; the worker seeds the licensed, packaged
Qwen3.5 metadata from vendor/bonsai2/ for those exact repositories. It also
corrects their external-data declarations, chat stop tokens, and Q2 cache
precision. The 4B text preset uses Mike0021's
single causal graph with its own complete metadata. The Vision presets use
webbrain-one/Ternary-Bonsai-2-{2B,4B}-ONNX-WebGPU-Vision, the Qwen processor's
text-first image API, and a bounded image resolution.
The Vision presets prefill long prompts in 512-token slices that carry the
recurrent, convolution and KV cache, then let generate() continue from that
cache. A single pass over a multi-thousand-token prompt can occupy a GPU shared
with the display (Apple silicon, a display-attached card) for seconds; slices
bound each pass at the cost of roughly one extra decoder run per slice. Greedy
answers match the single-pass path. chrome.storage.local key
webgpuPrefillSliceTokens overrides the slice size (64–8192 tokens; 0
disables slicing).
Compass VLD Tiny v5, a fine-tune of MiniCPM-V-4.7-1B, has no Transformers.js
model class or processor. src/chrome/src/offscreen/compass-vld-runtime.js
creates its three graphs (embed_tokens Q8, vision_encoder FP16 and a
chunked-GatedDeltaNet decoder_model_merged Q8 that returns only last-position
logits and decision_hidden) with this bundle's ort.webgpu.mjs, the same
ONNX Runtime Web build Transformers.js uses, and ports the processor:
MiniCPM-V slicing, Pillow-compatible bicubic resampling, the image
placeholders and the canvas M-RoPE positions. Transformers.js supplies only the
tokenizer class and the chat-template renderer; the runtime builds the
tokenizer from the cached files itself, because AutoTokenizer.from_pretrained
first looks for tokenizer_config.json at revision main, which the pinned
download does not cache. Its files are fetched at a pinned revision, one at
a time, and each is checked against its pinned size and SHA-256 before it is
used. Its own sessions keep outputs and caches on the GPU
(preferredOutputLocation: 'gpu-buffer') and read back only the last-position
logits; the advice below about that option is for the Transformers.js
generation path.
For the split Bonsai text and Vision models, the worker corrects Transformers.js 4.2's text-only rotary cache offset; image turns keep the native three-axis positions. Independent screenshots are processed separately before concatenating their patches and grids, because the bundled image processor treats an image array as video frames. These adaptations are scoped to the named Bonsai models.
Michionlion/Nanbeige4.2-3B-ONNX-WebGPU needs no bundle patch, but it does
need two things from this runtime. It publishes one WebGPU-fused graph named
onnx/model_webgpu_mlp_q4f16.onnx instead of the default model_q4f16.onnx,
so the worker passes model_file_name: 'model_webgpu_mlp'; the two external
data shards are declared by the repository's own
transformers.js_config.use_external_data_format. Its graph also uses the
com.microsoft::MatMulNBitsMlp WebGPU kernel, which is compiled into the
vendored ort-wasm-simd-threaded.asyncify.wasm. Do not replace the ONNX
Runtime artifacts with a build that drops that kernel; a test asserts it is
present because its absence only shows up as an opaque session-creation
failure at runtime.
The dedicated 450M vision sidecar uses the same component layout and is
approximately 810 MB. The VL 1.6B chat export instead names its vision and
decoder components embed_images and decoder; the compatibility patch below
maps those files into the standard image-text session names.
Packaged files
| File / directory | Source | Purpose |
|---|---|---|
transformers.web.js |
@huggingface/transformers 4.2.0 |
Browser ESM model/processor APIs |
ort.webgpu.mjs |
onnxruntime-web 1.27.0 |
WebGPU execution provider |
onnxruntime-common/ |
matching onnxruntime-common dependency |
Tensor and session types |
ort-wasm-simd-threaded.asyncify.* |
onnxruntime-web 1.27.0 |
The only WASM bridge; carries the WebGPU/JSEP runtime |
LICENSE.transformers.txt |
@huggingface/transformers 4.2.0 |
Apache-2.0 license |
LICENSE.onnxruntime.txt |
ONNX Runtime 1.27.0 | MIT license |
ThirdPartyNotices.onnxruntime.txt |
ONNX Runtime 1.27.0 | Notices for incorporated third-party software |
The readable, unminified browser builds are committed so a fresh checkout is a complete, Chrome Web Store-reviewable extension. Remote executable code is not allowed by Manifest V3 CSP; only model/config/tokenizer data is fetched from Hugging Face.
Only the asyncify variant is vendored. ort.webgpu.mjs resolves its WASM
filename from flags fixed at ONNX Runtime build time, and in this build the jsep
and jspi branches are compiled out, so ort-wasm-simd-threaded.asyncify.* is the
only bridge that can ever load. The WebGPU/JSEP execution provider lives inside
that artifact; it is not a separate download. Copying the jsep or jspi pair
adds tens of megabytes of unreachable binary to the reviewed package, so do not
restore them without first checking that filename branch.
The ONNX Runtime files are intentionally newer than the version pinned by Transformers.js 4.2.0. Stable 1.27.0 contains WebGPU buffer-pool and Qwen3/QMoE correctness fixes needed by Ling while retaining the same public JavaScript session API used by this Transformers.js release.
Browser bundle patches
The upstream browser bundle contains two bare module specifiers that an unbundled extension cannot resolve. After copying a new release, rewrite them:
sed -i 's|"onnxruntime-web/webgpu"|"./ort.webgpu.mjs"|' \
src/chrome/vendor/transformers/transformers.web.js
sed -i 's|"onnxruntime-common"|"./onnxruntime-common/index.js"|' \
src/chrome/vendor/transformers/transformers.web.js
Verify that no executable bare imports remain:
grep -E '(import|export)[^"]*from\s+"[a-zA-Z@]' \
src/chrome/vendor/transformers/transformers.web.js \
| grep -v '^\s*//' | grep -v '^\s*\*'
The LFM2.5-VL ONNX repositories need two compatibility hooks. The 1.6B export
predates the standard ImageTextToText component filenames, so keep the small
session_file_names alias hook in
MODEL_SESSION_CONFIG[MODEL_TYPES.ImageTextToText]; the worker supplies aliases
through config["transformers.js_config"]. Keep the corresponding getSession
logic resolving device and dtype by logical session_name, while using the
aliased filename only to fetch the physical ONNX graph. Both current VL exports
also use the Transformers v5 processor layout: image metadata is nested in
processor_config.json, and the chat template lives in chat_template.jinja.
Keep loadImageProcessorConfig, image_processor_config_file, and
chat_template_file support so the worker can opt into that layout without
changing older models. Reapply these patches after replacing
transformers.web.js, and mirror the resulting browser bundle into Firefox so
the packaged vendor files remain byte-identical.
Updating
Use a temporary dependency install; WebBrain does not need a runtime npm dependency because the reviewed browser assets are committed directly:
npm install --no-save @huggingface/transformers@latest
cp node_modules/@huggingface/transformers/dist/transformers.web.js \
src/chrome/vendor/transformers/
cp node_modules/onnxruntime-web/dist/ort.webgpu.mjs \
src/chrome/vendor/transformers/
cp node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded.asyncify.{mjs,wasm} \
src/chrome/vendor/transformers/
rm -rf src/chrome/vendor/transformers/onnxruntime-common
mkdir src/chrome/vendor/transformers/onnxruntime-common
cp node_modules/onnxruntime-common/dist/esm/*.js \
src/chrome/vendor/transformers/onnxruntime-common/
Copy the Transformers.js license and the ONNX Runtime license plus
ThirdPartyNotices.txt into this directory whenever the runtime is updated.
Reapply the specifier and ImageTextToText session-alias patches, update the
version table above, then verify:
node --checkpasses for the provider, host, and worker.- Use local fallback enables the option without downloading weights.
- Test Connection reads
WB7from the packaged vision probe image. - The second test reuses browser-cached model files.
Runtime architecture
ProviderManager._createProvider('webgpu') / getVisionProvider()
-> WebGPUProvider.chat() / WebGPUVisionProvider.chat()
-> MV3 offscreen document
-> dedicated module Worker
-> text-generation pipeline / AutoProcessor + AutoModelForImageTextToText
-> selected preset (Compass VLD Tiny v5 through its own ONNX Runtime sessions,
Compass Tiny v2.1 or a custom ONNX repo) / local vision sidecar over WebGPU
Keep inference in the Worker. The MV3 service worker has no WebGPU, while the
offscreen document's main thread has shown tighter WASM allocation limits for
large ONNX runs. Do not set preferredOutputLocation: 'gpu-buffer' on this
generation path: Transformers.js decodes the generated tensor on the CPU and
must be allowed to download that output normally.
LFM2.5-VL expects the image placeholder before the user's text in its chat template. The Worker normalizes incoming OpenAI-style multimodal messages to that order. For the connection test, it replaces the packaged generic OCR image with three large, unlabeled color panels. This proves the local model received pixels without relying on fine OCR, which is brittle for a 450M model. Normal screenshots are never replaced by this probe-only path.