YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
evals: cover GPT-5 output-limit recovery
Louis Beaumont717e3ec0382d
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.mdindex 96e1ad530..3bd4c4836 100644--- a/evals/coding-agent/README.md+++ b/evals/coding-agent/README.md@@ -9,7 +9,7 @@ This is an agent eval suite, not a unit-test suite. Every case contains:- saved prompt, transcript, candidate patch, grader output, runtime fingerprint, and result;- repeated-trial reporting with success rate, `pass@k`, and `pass^k`.-The current app corpus contains 34 git-mined regressions. See+The current app corpus contains 35 git-mined regressions. See[DESIGN.md](./DESIGN.md) for the Anthropic guidance, source contract, andhistory-mining workflow. The companion website manifest contributes 20 more.diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.jsonindex 9c347f217..0071dfbeb 100644--- a/evals/coding-agent/cases.json+++ b/evals/coding-agent/cases.json@@ -1,7 +1,7 @@{"schema_version": 1,"suite": "screenpipe-app-coding-agent-regressions",- "dataset_version": "2026-09-04.1",+ "dataset_version": "2026-09-05.1","suite_type": "regression","owner": "screenpipe engineering","policy": {@@ -985,6 +985,42 @@],"command": "cd packages/ai-gateway && bun test src/test/eval-nested-json-schema.test.ts src/test/openai-models.unit.test.ts --timeout=10000"}+ },+ {+ "id": "ai-gateway-gpt5-output-limit-retry",+ "name": "Recover from retryable GPT-5 output-limit rejections",+ "source": {+ "kind": "git_regression",+ "fix_commit": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",+ "pull_request": "https://github.com/screenpipe/screenpipe/pull/6904"+ },+ "base_ref": "d375badd50f99e194c251c8dbc6ba90dc476231e",+ "oracle_ref": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",+ "oracle_paths": [+ "packages/ai-gateway/src/providers/openai.ts"+ ],+ "gate": "advisory",+ "trigger_paths": [+ "packages/ai-gateway/src/providers/openai.ts",+ "packages/ai-gateway/src/handlers/chat.ts",+ "packages/ai-gateway/src/types.ts"+ ],+ "tags": ["regression", "escaped-failure", "ai-gateway", "openai", "gpt-5", "output-limit", "retry", "bounded-recovery"],+ "prompt": "GPT-5 requests can fail before producing a response when the provider explicitly says the configured output limit was reached and asks the caller to retry with a higher limit. Recover that request with a bounded higher-limit retry instead of letting the model chain fail. Preserve the caller's initial limit until this exact rejection occurs, keep the retry within the supported GPT-5 ceiling, avoid a futile retry when already at that ceiling, leave non-GPT-5 and unrelated errors untouched, and preserve the provider's existing parameter-repair behavior and loop bound.",+ "agent_timeout_seconds": 900,+ "grader_dependency_links": [+ { "source_path": "packages/ai-gateway/node_modules", "destination_path": "packages/ai-gateway/node_modules" }+ ],+ "grader": {+ "timeout_seconds": 300,+ "fixtures": [+ {+ "source_ref": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",+ "source_path": "packages/ai-gateway/src/test/openai-param-retry.unit.test.ts"+ }+ ],+ "command": "cd packages/ai-gateway && bun test src/test/openai-param-retry.unit.test.ts --timeout=10000"+ }}]}