Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

fix(ai-gateway): interactive auto → gemini-3.5-flash standard, keep background on flex (#4259)

Louis Beaumont2ce35d7c5f85
diff --git a/packages/ai-gateway/src/handlers/chat.ts b/packages/ai-gateway/src/handlers/chat.ts
index 603999912..f0122812c 100644
--- a/packages/ai-gateway/src/handlers/chat.ts
+++ b/packages/ai-gateway/src/handlers/chat.ts
@@ -8,12 +8,17 @@ import { logModelOutcome } from '../services/model-health';
import { isFlexEligible } from '../utils/latency';
import { captureException } from '@sentry/cloudflare';
-// Auto model waterfall — open-weight Vertex MaaS picks ordered by quality
-// (free for users, low GCP burn) with a cheap Gemini safety net at the tail.
+// Auto model waterfall (INTERACTIVE) — leads with gemini-3.5-flash on the
+// STANDARD tier. Interactive flex is OFF (GEMINI_FLEX_INTERACTIVE=false) because
+// flex is best-effort latency and made user-facing chat slow; standard tier
+// gives normal low latency at higher quality than glm-5 (AA 55 vs 50). glm-5 and
+// the other free Vertex MaaS picks stay as fallbacks. Latency-tolerant traffic
+// uses AUTO_WATERFALL_BACKGROUND on the cheaper flex tier instead.
// Exported so tests can pin that every chain entry has a MODEL_PRICING match
// (otherwise served-model cost rows fall into the unknown-model estimate).
export const AUTO_WATERFALL = [
- 'glm-5', // closest Vertex match to gemini-3.5-flash (AA index 50 vs 55) at ~1/3 the output cost
+ 'gemini-3.5-flash', // standard tier (interactive non-flex) — fast + higher quality than glm-5
+ 'glm-5', // free Vertex MaaS fallback
'kimi-k2.5',
'deepseek-v3.2',
'glm-4.7',
@@ -451,9 +456,11 @@ export async function handleChatCompletions(
// scopes it to Gemini attempts; a flex 429 cascades to a standard sibling.
const flexEligible = isFlexEligible(latency, env);
- // Chain selection stays keyed on latency: interactive 'auto' still leads with
- // glm-5 (free MaaS), background swaps to the flex-first Gemini chain. Flex is
- // applied to whichever Gemini entries the chosen chain hits, via flexEligible.
+ // Chain selection keyed on latency: interactive 'auto' leads with
+ // gemini-3.5-flash at STANDARD tier (interactive flex is off), background swaps
+ // to the flex-first Gemini chain. Flex is applied to Gemini entries only when
+ // flexEligible — background always, interactive only if GEMINI_FLEX_INTERACTIVE
+ // is left "true" (we set it "false" so interactive chat stays low-latency).
const useBackgroundChain = latency === 'background';
if (body.model === 'auto') {
diff --git a/packages/ai-gateway/wrangler.toml b/packages/ai-gateway/wrangler.toml
index cf83cdae1..c285dd1ea 100644
--- a/packages/ai-gateway/wrangler.toml
+++ b/packages/ai-gateway/wrangler.toml
@@ -47,6 +47,13 @@ WHISPER_URL = ""
# Legacy compat
SELF_HOSTED_TRANSCRIPTION_URL = "https://audio.screenpi.pe"
+# Interactive chat must stay low-latency, so interactive Gemini uses the STANDARD
+# tier, not best-effort flex (flex on the interactive auto lane made chat slow).
+# Background traffic still flexes (isFlexEligible returns true for background
+# regardless of this flag). Set here (not just dashboard) so it persists as the
+# deliberate default across redeploys; FLEX_TIER_ENABLED remains the master switch.
+GEMINI_FLEX_INTERACTIVE = "false"
+
# Cost / flex tuning knobs — read with a code default, so they live in the CF
# dashboard (Settings → Variables), not here, and take effect with no redeploy:
# - FLEX_TIER_ENABLED ("true"): master kill switch for the Vertex Gemini flex tier.