YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(evals): reject premature OpenAI stream terminators
Louis Beaumontcbf541aae46f
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.mdindex bde7a9305..5dcc0ca2e 100644--- a/evals/coding-agent/README.md+++ b/evals/coding-agent/README.md@@ -1528,7 +1528,7 @@ calibration or baseline/reference verification.`app-openai-stream-tool-choice` executes the historical OpenAI provider with a synthetic SDK transport. The broken parent loses explicit tool choices on four streams and the unsupported-usage retry; seven neighboring outcomes still pass. The provider-only historical repair and current source pass all twelve checks.-The grader checks caller-selected policy, tool schemas, model/messages, input preservation, retry behavior, native tool fragments, content, usage and stream termination. Eleven calibration controls include equivalent code, an unused repair, forced policies, lost schemas/fragments and a missing-source setup error.+The grader checks caller-selected policy, tool schemas, model/messages, input preservation, retry behavior, native tool fragments, content, usage and stream termination. Fourteen calibration controls include equivalent code, an unused repair, forced policies, lost schemas/fragments, premature completion before content/finish/usage and a missing-source setup error. The grader requires the completion terminator to follow all response data, so content after it cannot satisfy preserved-output checks.```shbun test evals/coding-agent/calibrate-openai-tool-choice.test.jsdiff --git a/evals/coding-agent/calibrate-openai-tool-choice.test.js b/evals/coding-agent/calibrate-openai-tool-choice.test.jsindex 666cf2e9b..a1c27ba62 100644--- a/evals/coding-agent/calibrate-openai-tool-choice.test.js+++ b/evals/coding-agent/calibrate-openai-tool-choice.test.js@@ -29,3 +29,19 @@ test('forcing required tool policy is rejected',()=>fail(grade('forced-required'test('losing schemas is rejected',()=>fail(grade('lost-tools',item.oracle_ref,d=>{const p=join(d,provider);writeFileSync(p,readFileSync(p,'utf8').replaceAll("tools: body.tools as ChatCompletionCreateParams['tools'],",'tools: [],'));})));test('losing native stream fragments is rejected',()=>fail(grade('lost-fragments',item.oracle_ref,d=>change(d,provider,'const toolCalls = choice?.delta?.tool_calls;','const toolCalls = undefined;'))));test('missing provider is setup failure',()=>{const r=grade('missing',item.oracle_ref,d=>rmSync(join(d,provider)));expect(r.status).toBe(1);expect(r.stderr).toContain('Cannot find module');expect(r.stderr).toContain('0 pass');});++function prematureDone(cwd, point) {+ const path = join(cwd, provider), source = readFileSync(path, 'utf8');+ const done = "controller.enqueue(new TextEncoder().encode('data: [DONE]\\n\\n'));";+ expect(source.split(done)).toHaveLength(3);+ const withoutDone = source.replace(done, '');+ let index;+ if (point === 'content') index = withoutDone.indexOf('let finishReason: string | null = null;');+ else if (point === 'finish') index = withoutDone.lastIndexOf('controller.enqueue(', withoutDone.indexOf("finish_reason: finishReason || 'stop'"));+ else index = withoutDone.indexOf('// Emit usage before [DONE]');+ expect(index).toBeGreaterThan(0);+ writeFileSync(path, withoutDone.slice(0, index) + done + '\n' + withoutDone.slice(index));+}+for (const point of ['content', 'finish', 'usage']) {+ test(`completion terminator before ${point} is rejected`, () => fail(grade(`early-done-${point}`, item.oracle_ref, cwd => prematureDone(cwd, point))));+}diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.jsonindex daf6a240d..a214f64a9 100644--- a/evals/coding-agent/cases.json+++ b/evals/coding-agent/cases.json@@ -4739,7 +4739,7 @@"preserved-behavior"],"agent_timeout_seconds": 1200,- "prompt": "Streaming requests through the OpenAI provider lose the caller's tool-selection policy, although ordinary completions preserve it. Repair the existing provider behavior so required, auto, none and named-function tool_choice values reach the provider unchanged on streaming and non-streaming requests. An unspecified choice must remain unspecified, including requests without tools. Preserve the selected model, messages, exact tool schemas and input object. If an OpenAI-compatible server rejects stream_options.include_usage, the supported retry must omit that option while retaining the original tools and tool choice. Preserve streamed content, native tool-call identity/name/arguments, tool_calls finish reason, prompt/completion/total/cached usage and exactly one completion terminator. Do not change routing/model policy or authentication. Use synthetic SDK transport without live API calls or executing tools; no specific implementation shape is required. This task concerns provider request/response behavior, not native Pi configuration, model adherence to the tool policy or full gateway authentication.",+ "prompt": "Streaming requests through the OpenAI provider lose the caller's tool-selection policy, although ordinary completions preserve it. Repair the existing provider behavior so required, auto, none and named-function tool_choice values reach the provider unchanged on streaming and non-streaming requests. An unspecified choice must remain unspecified, including requests without tools. Preserve the selected model, messages, exact tool schemas and input object. If an OpenAI-compatible server rejects stream_options.include_usage, the supported retry must omit that option while retaining the original tools and tool choice. Preserve streamed content, native tool-call identity/name/arguments, tool_calls finish reason, prompt/completion/total/cached usage and exactly one completion terminator. Do not change routing/model policy or authentication. Use synthetic SDK transport without live API calls or executing tools; no specific implementation shape is required. This task concerns provider request/response behavior, not native Pi configuration, model adherence to the tool policy or full gateway authentication. All streamed content, tool fragments, finish reason and usage must arrive before the single completion terminator; clients stop consuming at that terminator.","grader": {"timeout_seconds": 60,"fixtures": [diff --git a/evals/coding-agent/graders/openai-tool-choice.test.js b/evals/coding-agent/graders/openai-tool-choice.test.jsindex 9fcd26279..fd0a7cf52 100644--- a/evals/coding-agent/graders/openai-tool-choice.test.js+++ b/evals/coding-agent/graders/openai-tool-choice.test.js@@ -49,7 +49,9 @@ test('unsupported usage retry retains the exact tool policy and tools', async ()test('ordinary streams preserve content, native tool fragments, usage and one terminator', async () => {const output = await invoke(true, undefined, false); expect(requests).toHaveLength(1); expect(requests[0].tool_choice).toBeUndefined();expect(output.split('data: [DONE]').length - 1).toBe(1);- const events = output.split('\n').filter(line => line.startsWith('data: ') && line !== 'data: [DONE]').map(line => JSON.parse(line.slice(6)));+ const frames = output.split('\n').filter(line => line.startsWith('data: '));+ expect(frames.at(-1)).toBe('data: [DONE]');+ const events = frames.slice(0, -1).map(line => JSON.parse(line.slice(6)));expect(events.map(e => e.choices?.[0]?.delta?.content || '').join('')).toBe('plain answer');expect(events.flatMap(e => e.choices?.[0]?.delta?.tool_calls || [])).toEqual([fragment]);expect(events.find(e => e.choices?.[0]?.finish_reason)?.choices[0].finish_reason).toBe('tool_calls');