Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

blog: remove runpod dollar cost details from LFM3 launch benchmark

Codex release-state test131ac8f0c645
diff --git a/web/blog/index.html b/web/blog/index.html
index 866f88c63..90b23a502 100644
--- a/web/blog/index.html
+++ b/web/blog/index.html
@@ -674,7 +674,7 @@
"headline": "Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark",
"url": "https://webbrain.one/blog/lfm3-vl-2b-vs-webbrain-compass-vld",
"datePublished": "2026-10-28",
- "description": "We ran Liquid's new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites. Total cost on RunPod: $0.34.",
+ "description": "We ran Liquid's new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites.",
"author": {
"@type": "Person",
"name": "Emre Sokullu",
@@ -1475,7 +1475,7 @@
</section>
<div class="post-list" id="post-list">
- <a href="/blog/lfm3-vl-2b-vs-webbrain-compass-vld" class="post-card" data-search="Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark LFM3-VL-2B beats Compass VLD Tiny v5 on six of nine general vision and text suites, ties three, and loses none. Compass is still the only one of the two that is also a decision model. Next: an LFM3-based VLD. We ran Liquid&#39;s new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites. Total cost on RunPod: $0.34. October 28, 2026 2026-10-28 Liquid LFM3 LFM3-VL-2B d1-2B WebBrain Compass VLD Tiny v5 MiniCPM-V-4.7-1B BenchKit vision language model decision model llama.cpp RunPod">
+ <a href="/blog/lfm3-vl-2b-vs-webbrain-compass-vld" class="post-card" data-search="Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark LFM3-VL-2B beats Compass VLD Tiny v5 on six of nine general vision and text suites, ties three, and loses none. Compass is still the only one of the two that is also a decision model. Next: an LFM3-based VLD. We ran Liquid&#39;s new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites. October 28, 2026 2026-10-28 Liquid LFM3 LFM3-VL-2B d1-2B WebBrain Compass VLD Tiny v5 MiniCPM-V-4.7-1B BenchKit vision language model decision model llama.cpp RunPod">
<div class="post-meta">October 28, 2026 &middot; 8 min read</div>
<div class="post-title">Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark</div>
<div class="post-excerpt">LFM3-VL-2B beats Compass VLD Tiny v5 on six of nine general vision and text suites, ties three, and loses none. Compass is still the only one of the two that is also a decision model. Next: an LFM3-based VLD.</div>
diff --git a/web/blog/lfm3-vl-2b-vs-webbrain-compass-vld/index.html b/web/blog/lfm3-vl-2b-vs-webbrain-compass-vld/index.html
index 59faf2aa0..4e606396e 100644
--- a/web/blog/lfm3-vl-2b-vs-webbrain-compass-vld/index.html
+++ b/web/blog/lfm3-vl-2b-vs-webbrain-compass-vld/index.html
@@ -4,7 +4,7 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>LFM3-VL-2B vs WebBrain Compass VLD Tiny v5 - WebBrain Blog</title>
- <meta name="description" content="We ran Liquid&#39;s new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites. Total cost on RunPod: $0.34.">
+ <meta name="description" content="We ran Liquid&#39;s new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites.">
<meta name="keywords" content="Liquid LFM3, LFM3-VL-2B, d1-2B, WebBrain Compass VLD Tiny v5, MiniCPM-V-4.7-1B, BenchKit, vision language model, decision model, llama.cpp, RunPod">
<meta property="og:title" content="Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark">
<meta property="og:description" content="Same tasks, same llama.cpp build, same rented GPU. LFM3-VL-2B wins six of nine BenchKit suites against our MiniCPM-V-based Compass. Compass keeps one edge: it is also a decision model.">
@@ -18,7 +18,7 @@
<meta property="og:image:alt" content="WebBrain — Open-source AI browser agent">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="LFM3-VL-2B vs WebBrain Compass VLD Tiny v5">
- <meta name="twitter:description" content="4,857 BenchKit tasks per model on launch day for $0.34. LFM3 wins on vision and text; Compass still folds decisions into the same model. We&#39;ll build an LFM3-based VLD next.">
+ <meta name="twitter:description" content="4,857 BenchKit tasks per model on launch day. LFM3 wins on vision and text; Compass still folds decisions into the same model. We&#39;ll build an LFM3-based VLD next.">
<meta name="twitter:image" content="https://webbrain.one/twitter-image.png">
<meta name="twitter:image:alt" content="WebBrain — Open-source AI browser agent">
<link rel="icon" type="image/png" href="/favicon.png">
@@ -475,7 +475,7 @@
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark",
- "description": "We ran Liquid's new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites. Total cost on RunPod: $0.34.",
+ "description": "We ran Liquid's new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites.",
"author": {
"@type": "Person",
"name": "Emre Sokullu",
@@ -525,7 +525,7 @@
<article>
<div class="meta">October 28, 2026 &middot; 8 min read &middot; <a href="/blog">&larr; All posts</a></div>
<h1>Liquid LFM3-VL-2B vs WebBrain Compass VLD: a launch-day benchmark</h1>
- <p class="lede"><strong>Liquid released LFM3 on October 27, and that night we ran LFM3-VL-2B against WebBrain Compass VLD Tiny v5, the small on-device model inside WebBrain.</strong> Same tasks, same engine, same GPU: 4,857 BenchKit tasks per model on a rented RTX A4000 for $0.34. To be frank, Compass loses: LFM3-VL-2B wins six of nine suites and Compass wins none. Compass is still the only one of the two that is also a decision model, and that shapes what we build next.</p>
+ <p class="lede"><strong>Liquid released LFM3 on October 27, and that night we ran LFM3-VL-2B against WebBrain Compass VLD Tiny v5, the small on-device model inside WebBrain.</strong> Same tasks, same engine, same GPU: 4,857 BenchKit tasks per model on a rented RTX A4000. To be frank, Compass loses: LFM3-VL-2B wins six of nine suites and Compass wins none. Compass is still the only one of the two that is also a decision model, and that shapes what we build next.</p>
<h2 id="what-compass-actually-is">What Compass actually is</h2>
<p>WebBrain Compass VLD Tiny v5 is not a model we trained from scratch. It is our fine-tune of <a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-1B" target="_blank" rel="noopener">OpenBMB's MiniCPM-V-4.7-1B</a>, trained in two rounds on one RTX 5090, about 20 hours in total:</p>
<ul><li><strong>Round 1:</strong> one pass over 344,139 samples: WebBrain's own screenshot data, MolmoPoint GUI grounding, WebChain trajectories and the Compass text release. LoRA (rank 64) on every linear layer of the language model, the vision-to-language merger trained in full, the vision tower frozen.</li><li><strong>Round 2:</strong> 51,811 more samples, adding d1 page-state decision data, a larger share of text, and a replay of the visual data.</li></ul>
@@ -535,7 +535,7 @@
<div class="table-wrap"><table><thead><tr><th></th><th>Compass VLD Tiny v5 (MiniCPM-V-4.7-1B)</th><th>LFM3-VL-2B</th></tr></thead><tbody><tr><td>Total parameters</td><td>1.55B</td><td>1.92B</td></tr><tr><td>Language model, layers</td><td>498M</td><td>1,467M</td></tr><tr><td>Language model, embeddings and output head</td><td>508M</td><td>262M</td></tr><tr><td>Vision encoder</td><td>522M</td><td>164M</td></tr><tr><td>Vision-to-language projector</td><td>26M</td><td>25M</td></tr><tr><td>Language model design</td><td>Qwen3.5-0.8B-style: 18 Gated DeltaNet + 6 attention layers</td><td>LFM3: 13 gated short convolutions, 5 sliding-window and 6 global attention layers</td></tr></tbody></table></div>
<p>LFM3-VL-2B is about a quarter larger overall. It puts nearly three times as many parameters into its language-model layers, while MiniCPM puts about three times as many into its vision encoder. Keep that in mind below: the text-heavy suites favour the bigger language model, and the smaller one decodes faster.</p>
<h2 id="how-we-ran-it">How we ran it</h2>
-<ul><li><strong>Harness:</strong> <a href="https://github.com/DogukanUrker/BenchKit" target="_blank" rel="noopener">BenchKit</a> v0.2.1, headless, temperature 0, 300 seconds per task. Nine suites: four vision (MMVP, V* Bench, OCRBench, ChartQA) and five text (BenchKit's sanity set, IFEval, GSM8K, MMLU-Pro, BANKING77). ChartQA, GSM8K, MMLU-Pro and BANKING77 ran on fixed slices, and BenchKit spreads vision slices across categories.</li><li><strong>Engine:</strong> on launch day no vLLM or SGLang release could serve LFM3-VL, and llama.cpp support was an open pull request, <a href="https://github.com/ggml-org/llama.cpp/pull/31252" target="_blank" rel="noopener">#31252</a>. We built that branch on the pod; it was merged into llama.cpp later the same day. Both models ran on that single build, BF16 weights with F16 vision encoders, 16 requests in flight. We converted Compass to GGUF from its merged weights with llama.cpp's own converter.</li><li><strong>Settings:</strong> thinking off for both. Compass has no thinking mode and LFM3 turns it on by default, so we switched it off to compare like with like. LFM3 kept Liquid's recommended repeat penalty of 1.1; Compass uses none.</li><li><strong>Hardware and cost:</strong> one RTX A4000 16 GB on RunPod community cloud at $0.17 an hour. The whole session took 121 minutes, including two llama.cpp builds, the conversion, the d1-2B run further down and some idle time at the end: about <strong>$0.34</strong>.</li><li><strong>A check on the port:</strong> LFM3's thinking-off scores land within about a point of Liquid's own model card: V* 79.6 vs 80.6, OCRBench 81.5 vs 82.2, ChartQA 82.0 vs 81.4. The pre-release llama.cpp support behaves.</li></ul>
+<ul><li><strong>Harness:</strong> <a href="https://github.com/DogukanUrker/BenchKit" target="_blank" rel="noopener">BenchKit</a> v0.2.1, headless, temperature 0, 300 seconds per task. Nine suites: four vision (MMVP, V* Bench, OCRBench, ChartQA) and five text (BenchKit's sanity set, IFEval, GSM8K, MMLU-Pro, BANKING77). ChartQA, GSM8K, MMLU-Pro and BANKING77 ran on fixed slices, and BenchKit spreads vision slices across categories.</li><li><strong>Engine:</strong> on launch day no vLLM or SGLang release could serve LFM3-VL, and llama.cpp support was an open pull request, <a href="https://github.com/ggml-org/llama.cpp/pull/31252" target="_blank" rel="noopener">#31252</a>. We built that branch on the pod; it was merged into llama.cpp later the same day. Both models ran on that single build, BF16 weights with F16 vision encoders, 16 requests in flight. We converted Compass to GGUF from its merged weights with llama.cpp's own converter.</li><li><strong>Settings:</strong> thinking off for both. Compass has no thinking mode and LFM3 turns it on by default, so we switched it off to compare like with like. LFM3 kept Liquid's recommended repeat penalty of 1.1; Compass uses none.</li><li><strong>Hardware:</strong> one RTX A4000 16 GB on RunPod community cloud. The whole session took 121 minutes, including two llama.cpp builds, the conversion, the d1-2B run further down and some idle time at the end.</li><li><strong>A check on the port:</strong> LFM3's thinking-off scores land within about a point of Liquid's own model card: V* 79.6 vs 80.6, OCRBench 81.5 vs 82.2, ChartQA 82.0 vs 81.4. The pre-release llama.cpp support behaves.</li></ul>
<p>Both models answered identical tasks, so we compare them task by task. The table gives the paired difference with its 95% interval, and a suite counts as a win only when that interval stays on one side of zero.</p>
<h2 id="results">Results</h2>
<div class="bench-style">
diff --git a/web/blog/posts/lfm3-vl-2b-vs-webbrain-compass-vld.md b/web/blog/posts/lfm3-vl-2b-vs-webbrain-compass-vld.md
index 9e77b166d..2b864050c 100644
--- a/web/blog/posts/lfm3-vl-2b-vs-webbrain-compass-vld.md
+++ b/web/blog/posts/lfm3-vl-2b-vs-webbrain-compass-vld.md
@@ -6,7 +6,7 @@ sortOrder: -420
date: 2026-10-28
readTime: 8 min read
description: >
- We ran Liquid's new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites. Total cost on RunPod: $0.34.
+ We ran Liquid's new LFM3-VL-2B against WebBrain Compass VLD Tiny v5, our MiniCPM-V-4.7-1B fine-tune, on 4,857 BenchKit tasks per model the night LFM3 launched. LFM3 wins six of nine suites.
excerpt: >
LFM3-VL-2B beats Compass VLD Tiny v5 on six of nine general vision and text suites, ties three, and loses none. Compass is still the only one of the two that is also a decision model. Next: an LFM3-based VLD.
titleTag: >
@@ -18,7 +18,7 @@ ogDescription: >
twitterTitle: >
LFM3-VL-2B vs WebBrain Compass VLD Tiny v5
twitterDescription: >
- 4,857 BenchKit tasks per model on launch day for $0.34. LFM3 wins on vision and text; Compass still folds decisions into the same model. We'll build an LFM3-based VLD next.
+ 4,857 BenchKit tasks per model on launch day. LFM3 wins on vision and text; Compass still folds decisions into the same model. We'll build an LFM3-based VLD next.
keywords:
- Liquid LFM3
- LFM3-VL-2B
@@ -34,7 +34,7 @@ author: Emre Sokullu
authorUrl: https://emresokullu.com
html: true
lede: >
- **Liquid released LFM3 on October 27, and that night we ran LFM3-VL-2B against WebBrain Compass VLD Tiny v5, the small on-device model inside WebBrain.** Same tasks, same engine, same GPU: 4,857 BenchKit tasks per model on a rented RTX A4000 for $0.34. To be frank, Compass loses: LFM3-VL-2B wins six of nine suites and Compass wins none. Compass is still the only one of the two that is also a decision model, and that shapes what we build next.
+ **Liquid released LFM3 on October 27, and that night we ran LFM3-VL-2B against WebBrain Compass VLD Tiny v5, the small on-device model inside WebBrain.** Same tasks, same engine, same GPU: 4,857 BenchKit tasks per model on a rented RTX A4000. To be frank, Compass loses: LFM3-VL-2B wins six of nine suites and Compass wins none. Compass is still the only one of the two that is also a decision model, and that shapes what we build next.
---
## What Compass actually is
@@ -66,7 +66,7 @@ LFM3-VL-2B is about a quarter larger overall. It puts nearly three times as many
- **Harness:** [BenchKit](https://github.com/DogukanUrker/BenchKit) v0.2.1, headless, temperature 0, 300 seconds per task. Nine suites: four vision (MMVP, V* Bench, OCRBench, ChartQA) and five text (BenchKit's sanity set, IFEval, GSM8K, MMLU-Pro, BANKING77). ChartQA, GSM8K, MMLU-Pro and BANKING77 ran on fixed slices, and BenchKit spreads vision slices across categories.
- **Engine:** on launch day no vLLM or SGLang release could serve LFM3-VL, and llama.cpp support was an open pull request, [#31252](https://github.com/ggml-org/llama.cpp/pull/31252). We built that branch on the pod; it was merged into llama.cpp later the same day. Both models ran on that single build, BF16 weights with F16 vision encoders, 16 requests in flight. We converted Compass to GGUF from its merged weights with llama.cpp's own converter.
- **Settings:** thinking off for both. Compass has no thinking mode and LFM3 turns it on by default, so we switched it off to compare like with like. LFM3 kept Liquid's recommended repeat penalty of 1.1; Compass uses none.
-- **Hardware and cost:** one RTX A4000 16 GB on RunPod community cloud at $0.17 an hour. The whole session took 121 minutes, including two llama.cpp builds, the conversion, the d1-2B run further down and some idle time at the end: about **$0.34**.
+- **Hardware:** one RTX A4000 16 GB on RunPod community cloud. The whole session took 121 minutes, including two llama.cpp builds, the conversion, the d1-2B run further down and some idle time at the end.
- **A check on the port:** LFM3's thinking-off scores land within about a point of Liquid's own model card: V* 79.6 vs 80.6, OCRBench 81.5 vs 82.2, ChartQA 82.0 vs 81.4. The pre-release llama.cpp support behaves.
Both models answered identical tasks, so we compare them task by task. The table gives the paired difference with its 95% interval, and a suite counts as a win only when that interval stays on one side of zero.