Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

perf(ai-gateway): revert background head gpt-5.4 → gemini-3.5-flash:flex (3.3x cheaper) (#4384)

Louis Beaumont16f5bc8e941a
diff --git a/packages/ai-gateway/src/handlers/chat.ts b/packages/ai-gateway/src/handlers/chat.ts
index 595c8bc34..5efd53419 100644
--- a/packages/ai-gateway/src/handlers/chat.ts
+++ b/packages/ai-gateway/src/handlers/chat.ts
@@ -35,19 +35,16 @@ export const AUTO_WATERFALL_VISION = [
];
// Background waterfall — for latency-tolerant traffic (pipes, daily summary,
-// suggestions) where no user is waiting. This is the lane to BUY intelligence:
-// latency is free, so lead with gpt-5.4 (AA ~54 at its default reasoning effort,
-// 0.1x cache discount) on the OpenAI credit pool — smarter pipes, and it bleeds
-// background load OFF the strained GCP/Vertex credits. gemini-3.5-flash (flex)
-// + glm-5 + gemini-3-flash are the cheaper fallbacks if OpenAI 429s/errors;
-// runChain cascades on transient failures, and flex applies to the gemini
-// entries only (gpt-5.4 has no Vertex flex tier — it runs standard OpenAI).
-// NOTE: gpt-5.4 is ~3-4x the per-token cost of gemini-flex, so watch the $30k
-// OpenAI credit burn — at full background volume it can run ~$1-2k/day.
+// suggestions) where no user is waiting. Leads with gemini-3.5-flash on the FLEX
+// tier: background is ~84% cache-reads, so flex's 0.1x cache discount makes it the
+// cheapest decent-quality option ($/Mtok ≈ glm-5 but with the cache discount glm-5
+// lacks). Measured (6/19): gpt-5.4 here cost $590/day vs $177 on flex for the SAME
+// traffic — 3.3x more, on the small OpenAI pool, for a marginal IQ gain (54 vs ~50-55)
+// that pipes don't need; reverted (#4285→). glm-5 + gemini-3-flash are standard-tier
+// fallbacks for when flex is throttled.
export const AUTO_WATERFALL_BACKGROUND = [
- 'gpt-5.4', // smart reasoning model on OpenAI credits — latency-tolerant lane
- 'gemini-3.5-flash', // flex fallback (cheap) if OpenAI throttles/errors
- 'glm-5', // free Vertex MaaS fallback
+ 'gemini-3.5-flash', // flex tier — cheapest on the cache-heavy background mix
+ 'glm-5', // free Vertex MaaS fallback, standard tier
'gemini-3-flash', // near-free safety net
];