Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

evals: cover GPT-5 output-limit recovery

Louis Beaumont717e3ec0382d
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
index 96e1ad530..3bd4c4836 100644
--- a/evals/coding-agent/README.md
+++ b/evals/coding-agent/README.md
@@ -9,7 +9,7 @@ This is an agent eval suite, not a unit-test suite. Every case contains:
- saved prompt, transcript, candidate patch, grader output, runtime fingerprint, and result;
- repeated-trial reporting with success rate, `pass@k`, and `pass^k`.
-The current app corpus contains 34 git-mined regressions. See
+The current app corpus contains 35 git-mined regressions. See
[DESIGN.md](./DESIGN.md) for the Anthropic guidance, source contract, and
history-mining workflow. The companion website manifest contributes 20 more.
diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.json
index 9c347f217..0071dfbeb 100644
--- a/evals/coding-agent/cases.json
+++ b/evals/coding-agent/cases.json
@@ -1,7 +1,7 @@
{
"schema_version": 1,
"suite": "screenpipe-app-coding-agent-regressions",
- "dataset_version": "2026-09-04.1",
+ "dataset_version": "2026-09-05.1",
"suite_type": "regression",
"owner": "screenpipe engineering",
"policy": {
@@ -985,6 +985,42 @@
],
"command": "cd packages/ai-gateway && bun test src/test/eval-nested-json-schema.test.ts src/test/openai-models.unit.test.ts --timeout=10000"
}
+ },
+ {
+ "id": "ai-gateway-gpt5-output-limit-retry",
+ "name": "Recover from retryable GPT-5 output-limit rejections",
+ "source": {
+ "kind": "git_regression",
+ "fix_commit": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",
+ "pull_request": "https://github.com/screenpipe/screenpipe/pull/6904"
+ },
+ "base_ref": "d375badd50f99e194c251c8dbc6ba90dc476231e",
+ "oracle_ref": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",
+ "oracle_paths": [
+ "packages/ai-gateway/src/providers/openai.ts"
+ ],
+ "gate": "advisory",
+ "trigger_paths": [
+ "packages/ai-gateway/src/providers/openai.ts",
+ "packages/ai-gateway/src/handlers/chat.ts",
+ "packages/ai-gateway/src/types.ts"
+ ],
+ "tags": ["regression", "escaped-failure", "ai-gateway", "openai", "gpt-5", "output-limit", "retry", "bounded-recovery"],
+ "prompt": "GPT-5 requests can fail before producing a response when the provider explicitly says the configured output limit was reached and asks the caller to retry with a higher limit. Recover that request with a bounded higher-limit retry instead of letting the model chain fail. Preserve the caller's initial limit until this exact rejection occurs, keep the retry within the supported GPT-5 ceiling, avoid a futile retry when already at that ceiling, leave non-GPT-5 and unrelated errors untouched, and preserve the provider's existing parameter-repair behavior and loop bound.",
+ "agent_timeout_seconds": 900,
+ "grader_dependency_links": [
+ { "source_path": "packages/ai-gateway/node_modules", "destination_path": "packages/ai-gateway/node_modules" }
+ ],
+ "grader": {
+ "timeout_seconds": 300,
+ "fixtures": [
+ {
+ "source_ref": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",
+ "source_path": "packages/ai-gateway/src/test/openai-param-retry.unit.test.ts"
+ }
+ ],
+ "command": "cd packages/ai-gateway && bun test src/test/openai-param-retry.unit.test.ts --timeout=10000"
+ }
}
]
}