Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
feat: support saved TypeSafe and OpenRouter providers
David Zhang9496540ad5b5
diff --git a/.env.example b/.env.exampledeleted file mode 100644index a59c8af..0000000--- a/.env.example+++ /dev/null@@ -1 +0,0 @@-AI_GATEWAY_API_KEY=diff --git a/README.md b/README.mdindex 4b9d6c4..d502e4f 100644--- a/README.md+++ b/README.md@@ -22,9 +22,12 @@ jg skilljg "How are telemetry events recorded and sent?" ./my-project```-Requires **Node.js 22+**, **macOS or Linux**, and a **Vercel AI Gateway key**.+Requires **Node.js 22+**, **macOS or Linux**, and a key for **Vercel AI Gateway, TypeSafe, or OpenRouter**.No separate Python, Bun, or ripgrep installation is required to use `jg`.+**Provider selection is on main and awaits the next npm release.** The published+0.2.0 package uses Vercel AI Gateway; build from source to try TypeSafe/OpenRouter.+## Install the agent skill — required for agent setupInstalling the CLI alone does not teach your coding agent to use it. **Install@@ -41,7 +44,7 @@ others) and asks where to install. Add `--global` for a user-wide install, orreturned context, and when to fill gaps with its normal tools. It skips redundantretrieval when the needed context is already known. The current repository skillchecks for `jg` and installs the CLI if it is missing; authentication still needs-your Gateway key. The skill installer itself does not configure credentials.+your selected provider’s key. The skill installer itself does not configure credentials.`jg skill` delegates to the [skills CLI](https://github.com/vercel-labs/skills)and needs npm/npx plus network access. You can also run that installer directly,@@ -112,13 +115,16 @@ include exact costs, failed tasks, and separately observed Jev charges. See the## Source, credentials, and local state-Searches send eligible source content to Jev through Vercel AI Gateway. Default+Searches send eligible source content to Jev through the provider selected during auth. Defaultfilesystem filtering respects ignore files and excludes hidden, dependency/build,binary, and obvious credential files. These filters are not a guarantee that allsensitive information has been removed; choose a search root you intend to send.-`jg auth` saves your key in an owner-only config file. `AI_GATEWAY_API_KEY` takes-precedence when set; `jg doctor` checks the connection with synthetic input.+`jg auth` asks for your provider, then saves its key in an owner-only config file.+Re-running auth replaces that setup; searches always use the saved provider.+`jg doctor` checks it with synthetic input. Existing saved keys without a provider+remain Vercel keys. Environment-based credentials and endpoint overrides are not+used; run `jg auth` if you previously relied on them.Evaluation answers are cached locally by default. The CLI writes its output tostdout and does not create report files. Use `jg --help` for cache controls,search overrides, and incomplete-result behavior.diff --git a/apps/cli/README.md b/apps/cli/README.mdindex 6db4dcd..ae5d0f6 100644--- a/apps/cli/README.md+++ b/apps/cli/README.md@@ -13,10 +13,22 @@ jg doctorjg "How are telemetry events recorded and sent?" ./my-project```-`auth` saves your Vercel AI Gateway key; `doctor` verifies it with synthetic input.-You can instead supply `AI_GATEWAY_API_KEY`. Searches send eligible source to Jev-through AI Gateway. Credentials use an owner-only config file. Evaluation answers-are cached locally; `jg --help` describes overrides and cache commands.+`auth` asks for Vercel AI Gateway, TypeSafe, or OpenRouter, then saves its key in+an owner-only config file. Searches use that provider until you run auth again.+`doctor` verifies access with synthetic input and names the selected provider.+For unattended setup, pipe the key from your secret manager to:++```sh+jg auth --provider openrouter --stdin+```++Both options are required for piped setup. Credentials are saved under+`$XDG_CONFIG_HOME/jevgrep/credentials.json`, or `~/.config/jevgrep/credentials.json`.+Existing saved records without a provider still mean Vercel, without a migration.+API-key, endpoint, and model environment overrides are ignored; users who only+configured an environment key must run auth. There is no automatic fallback or+per-search provider override. Searches send eligible source to the saved service.+Evaluation answers are cached locally; `jg --help` describes cache controls.The summary comes first, followed by file and declaration locations and selectedsource. Locations are reading leads, not a checklist. Omitted excerpts are marked;diff --git a/apps/cli/src/args.ts b/apps/cli/src/args.tsindex 1debff3..f335288 100644--- a/apps/cli/src/args.ts+++ b/apps/cli/src/args.ts@@ -1,12 +1,13 @@import { parseArgs } from "node:util";import type { SearchInput } from "@repo/core";+import { isProviderId, type ProviderId } from "@repo/core/providers";import { CliError } from "./errors";import { DEFAULT_MAX_SOURCE_BYTES } from "./render";export type Command =| { kind: "help" | "version" | "doctor" | "cache-clear" }| { kind: "skill"; agents: string[]; global: boolean; yes: boolean }- | { kind: "auth"; fromStdin: boolean }+ | { kind: "auth"; provider?: ProviderId }| {kind: "search";query: string;@@ -27,6 +28,7 @@ export function parseCommand(args: string[]): Command {help: { type: "boolean", short: "h" },version: { type: "boolean" },stdin: { type: "boolean" },+ provider: { type: "string" },agent: { type: "string", multiple: true },global: { type: "boolean" },yes: { type: "boolean" },@@ -49,9 +51,12 @@ export function parseCommand(args: string[]): Command {if (values.help || values.version) throw new CliError("Use --help or --version alone.");const first = positionals[0];if (first === "auth") {- if (positionals.length !== 1 || keys.some((key) => key !== "stdin"))- throw new CliError("Usage: jg auth [--stdin]");- return { kind: "auth", fromStdin: values.stdin ?? false };+ if (positionals.length !== 1 || keys.some((key) => !["stdin", "provider"].includes(key)))+ throw new CliError("Usage: jg auth OR jg auth --provider NAME --stdin");+ if (!keys.length) return { kind: "auth" };+ if (!values.stdin || !isProviderId(values.provider))+ throw new CliError("Use auth --provider vercel|typesafe|openrouter --stdin for a piped key.");+ return { kind: "auth", provider: values.provider };}if (first === "skill") {const agents = values.agent ?? [];@@ -77,7 +82,7 @@ export function parseCommand(args: string[]): Command {!first?.trim() ||positionals.length > 2 ||values.stdin ||- keys.some((key) => ["agent", "global", "yes"].includes(key))+ keys.some((key) => ["agent", "global", "yes", "provider"].includes(key)))throw new CliError('Usage: jg "question" [root]. Run jg --help.');const rawBudget = values["max-source-bytes"];@@ -109,12 +114,17 @@ Usage: jg "question" [root]Root defaults to the current directory; use -- before a root beginning with -.Commands:- auth [--stdin] Save a Gateway key (hidden prompt, or an explicit pipe)+ auth Choose a provider, then save its key (hidden prompt)doctor Verify Jev access using a synthetic questionskill Install the agent skill via npx skills--help, -h Show usage--version Show the installed version+Auth automation:+ auth --provider vercel|typesafe|openrouter --stdin+ Save one provider/key from a pipe. Re-running auth replaces your setup.+ Saved credentials only; provider key/URL environment variables are ignored.+Skill installation options:--agent NAME Target an agent (repeat for multiple agents)--global Install for the current user instead of this projectdiff --git a/apps/cli/src/auth.ts b/apps/cli/src/auth.tsindex 0d7c2f4..a7d4aa4 100644--- a/apps/cli/src/auth.ts+++ b/apps/cli/src/auth.ts@@ -2,7 +2,8 @@ import { chmod, mkdir, readFile, rename, rm, writeFile } from "node:fs/promises"import { homedir } from "node:os";import { join } from "node:path";import { randomUUID } from "node:crypto";-import { isCancel, password } from "@clack/prompts";+import { isCancel, password, select } from "@clack/prompts";+import { providers, isProviderId, type ProviderId } from "@repo/core/providers";import { CliError } from "./errors";export function configDirectory() {@@ -17,9 +18,11 @@ function validateKey(raw: string): string {return key;}-export async function authenticate(fromStdin: boolean, signal: AbortSignal) {+export type Credentials = { provider: ProviderId; apiKey: string };++export async function authenticate(provider: ProviderId | undefined, signal: AbortSignal) {let key: string;- if (fromStdin) {+ if (provider !== undefined) {const chunks: Buffer[] = [];let bytes = 0;const abort = () => process.stdin.destroy(new DOMException("Interrupted", "AbortError"));@@ -36,9 +39,23 @@ export async function authenticate(fromStdin: boolean, signal: AbortSignal) {signal.removeEventListener("abort", abort);}} else {- if (!process.stdin.isTTY) throw new CliError("Use auth --stdin to read a piped key.");+ if (!process.stdin.isTTY)+ throw new CliError(+ "Use auth --provider vercel|typesafe|openrouter --stdin to read a piped key.",+ );+ const selected = await select<ProviderId>({+ message: "Choose your Jev provider",+ options: (Object.keys(providers) as ProviderId[]).map((value) => ({+ value,+ label: providers[value].label,+ })),+ output: process.stdout,+ signal,+ });+ if (isCancel(selected)) throw new DOMException("Interrupted", "AbortError");+ provider = selected;const answer = await password({- message: "Paste your Vercel AI Gateway API key",+ message: `Paste your ${providers[provider].label} API key`,output: process.stdout,signal,});@@ -53,21 +70,19 @@ export async function authenticate(fromStdin: boolean, signal: AbortSignal) {await chmod(directory, 0o700);const temporary = join(directory, `.credentials-${randomUUID()}.json`);try {- await writeFile(temporary, JSON.stringify({ apiKey: key }) + "\n", { mode: 0o600, flag: "wx" });+ await writeFile(temporary, JSON.stringify({ provider, apiKey: key }) + "\n", {+ mode: 0o600,+ flag: "wx",+ });+ signal.throwIfAborted();await rename(temporary, join(directory, "credentials.json"));} finally {await rm(temporary, { force: true });}- process.stdout.write("AI Gateway key saved. Run jg doctor to verify access.\n");+ process.stdout.write(`${providers[provider].label} key saved. Run jg doctor to verify access.\n`);}-export async function loadApiKey(): Promise<string> {- const fromEnvironment = process.env.AI_GATEWAY_API_KEY;- if (fromEnvironment !== undefined) {- if (!fromEnvironment.trim())- throw new CliError("AI_GATEWAY_API_KEY is empty; saved credentials are disabled.");- return validateKey(fromEnvironment);- }+export async function loadCredentials(): Promise<Credentials> {try {const credentials = JSON.parse(await readFile(join(configDirectory(), "credentials.json"), "utf8"),@@ -75,10 +90,12 @@ export async function loadApiKey(): Promise<string> {if (typeof credentials.apiKey !== "string" || !credentials.apiKey.trim()) {throw new CliError("Invalid credentials. Run jg auth again.");}- return validateKey(credentials.apiKey);+ const provider = Object.hasOwn(credentials, "provider") ? credentials.provider : "vercel";+ if (!isProviderId(provider)) throw new CliError("Invalid provider. Run jg auth again.");+ return { provider, apiKey: validateKey(credentials.apiKey) };} catch (error) {if ((error as NodeJS.ErrnoException).code === "ENOENT") {- throw new CliError("Run jg auth or set AI_GATEWAY_API_KEY.");+ throw new CliError("Run jg auth or use jg auth --provider NAME --stdin.");}throw new CliError("Could not read valid credentials. Run jg auth again.");}diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.tsindex 9657c90..0393439 100644--- a/apps/cli/src/index.ts+++ b/apps/cli/src/index.ts@@ -1,5 +1,6 @@#!/usr/bin/env node-import { authenticate, configDirectory, loadApiKey } from "./auth";+import { authenticate, configDirectory, loadCredentials } from "./auth";+import { providers } from "@repo/core/providers";import { help, parseCommand } from "./args";import { CliError } from "./errors";import { renderResult } from "./render";@@ -46,7 +47,7 @@ async function main() {process.exitCode = await installSkill(command, controller.signal);return;case "auth":- return authenticate(command.fromStdin, controller.signal);+ return authenticate(command.provider, controller.signal);case "cache-clear": {const { createEvaluationCache } = await import("@repo/core");const cache = createEvaluationCache({ directory: cacheDirectory() });@@ -56,12 +57,11 @@ async function main() {return write("Cache cleared.\n");}case "doctor": {- const apiKey = await loadApiKey();+ const credentials = await loadCredentials();const { createEvaluator } = await import("@repo/core");const evaluator = createEvaluator({- apiKey,+ ...credentials,signal: controller.signal,- baseURL: process.env.AI_GATEWAY_BASE_URL,});try {const answers = await evaluator.evaluate({@@ -74,18 +74,20 @@ async function main() {},});if (!(answers.relevant! > 0.5))- throw new CliError("Jev returned an unexpected answer to the connection check.");- await write("Jev connection verified.\n");+ throw new CliError(+ `${providers[credentials.provider].label} returned an unexpected answer to the connection check.`,+ );+ await write(`Jev connection verified through ${providers[credentials.provider].label}.\n`);} catch (error) {if (error instanceof CliError) throw error;throw new CliError(- "Jev connection check failed. Check your Gateway key, model access, and network.",+ `Jev connection check failed through ${providers[credentials.provider].label}. Check your saved key, model access, and network.`,);}return;}case "search": {- const apiKey = await loadApiKey();+ const credentials = await loadCredentials();const { retrieve, createEvaluator, createEvaluationCache } = await import("@repo/core");const cache = createEvaluationCache({directory: cacheDirectory(),@@ -94,9 +96,8 @@ async function main() {const evaluator = createEvaluator({cache,policyVersion: JSON.stringify(command.policy),- apiKey,+ ...credentials,signal: controller.signal,- baseURL: process.env.AI_GATEWAY_BASE_URL,});const result = await retrieve({diff --git a/apps/cli/test/args.test.ts b/apps/cli/test/args.test.tsindex 455f377..389a470 100644--- a/apps/cli/test/args.test.ts+++ b/apps/cli/test/args.test.ts@@ -48,3 +48,21 @@ test("source allocation is independent of the filesystem policy used for retrievexpect(unlimited.policy).toEqual({ hidden: true, noIgnore: true });expect(bounded.policy).toEqual(unlimited.policy);});++test("auth requires explicit provider for stdin and keeps provider selection out of search", () => {+ expect(parseCommand(["auth"])).toEqual({ kind: "auth" });+ expect(parseCommand(["auth", "--provider", "openrouter", "--stdin"])).toEqual({+ kind: "auth",+ provider: "openrouter",+ });+ for (const args of [+ ["auth", "--stdin"],+ ["auth", "--provider", "vercel"],+ ["auth", "--provider", "unknown", "--stdin"],+ ["query", "--provider", "typesafe"],+ ["doctor", "--provider", "vercel"],+ ["skill", "--provider", "vercel"],+ ["cache", "clear", "--provider", "vercel"],+ ])+ expect(() => parseCommand(args)).toThrow();+});diff --git a/apps/cli/test/auth.test.ts b/apps/cli/test/auth.test.tsindex 6bb709e..3419e30 100644--- a/apps/cli/test/auth.test.ts+++ b/apps/cli/test/auth.test.ts@@ -5,27 +5,29 @@ import { testInDocker, withCli } from "./cli";testInDocker("installed auth saves privately, bounds stdin, and never echoes keys", async () => {await withCli(async ({ home, run }) => {- const saved = await run(["auth", "--stdin"], "test-gateway-secret\n");+ const saved = await run(["auth", "--provider", "vercel", "--stdin"], "test-gateway-secret\n");expect(saved.code).toBe(0);expect(saved.stderr).toBe("");expect(saved.stdout).toContain("key saved");expect(saved.stdout).not.toContain("test-gateway-secret");const file = join(home, "jevgrep", "credentials.json");- expect(JSON.parse(await readFile(file, "utf8")).apiKey).toBe("test-gateway-secret");+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({+ provider: "vercel",+ apiKey: "test-gateway-secret",+ });expect((await stat(file)).mode & 0o777).toBe(0o600);expect((await stat(join(home, "jevgrep"))).mode & 0o777).toBe(0o700);- const disabled = await run(["doctor"]);- expect(disabled.code).toBe(1);- expect(disabled.stderr).toBe("");- expect(disabled.stdout).toContain("saved credentials are disabled");- const whitespace = await run(["auth", "--stdin"], "two secrets\n");+ const whitespace = await run(["auth", "--provider", "vercel", "--stdin"], "two secrets\n");expect(whitespace.code).toBe(1);expect(whitespace.stderr).toBe("");expect(whitespace.stdout).not.toContain("two secrets");- const huge = await run(["auth", "--stdin"], "x".repeat(8193));+ const huge = await run(["auth", "--provider", "vercel", "--stdin"], "x".repeat(8193));expect(huge.code).toBe(1);expect(huge.stderr).toBe("");expect(huge.stdout).toContain("exceeds");- expect(JSON.parse(await readFile(file, "utf8")).apiKey).toBe("test-gateway-secret");+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({+ provider: "vercel",+ apiKey: "test-gateway-secret",+ });});});diff --git a/apps/cli/test/cli.ts b/apps/cli/test/cli.tsindex f63f8f8..5959b03 100644--- a/apps/cli/test/cli.ts+++ b/apps/cli/test/cli.ts@@ -25,8 +25,6 @@ export async function withCli(HOME: home,XDG_CONFIG_HOME: home,XDG_CACHE_HOME: join(home, "cache"),- AI_GATEWAY_API_KEY: "",- AI_GATEWAY_BASE_URL: undefined,...env,},stdin: new Blob([input]),diff --git a/apps/cli/test/process.test.ts b/apps/cli/test/process.test.tsindex 6eb0e0c..d412095 100644--- a/apps/cli/test/process.test.ts+++ b/apps/cli/test/process.test.ts@@ -1,4 +1,6 @@import { expect } from "bun:test";+import { readFile, readdir } from "node:fs/promises";+import { join } from "node:path";import { execFile } from "node:child_process";import { promisify } from "node:util";import { fileURLToPath } from "node:url";@@ -30,7 +32,7 @@ finally:`,cli,],- { env: { ...process.env, HOME: home, AI_GATEWAY_API_KEY: "" } },+ { env: { ...process.env, HOME: home } },);expect(JSON.parse(stdout)).toEqual({ code: 0, stderr: "" });});@@ -38,7 +40,15 @@ finally:testInDocker("interactive auth hides input and exits 130 on interruption", async () => {await withCli(async ({ home }) => {- for (const mode of ["save", "interrupt"]) {+ for (const mode of [+ "save",+ "interrupt-provider",+ "interrupt-key",+ "cancel-provider",+ "cancel-key",+ ]) {+ const file = join(home, "jevgrep", "credentials.json");+ const previous = mode === "save" ? undefined : await readFile(file, "utf8");const { stdout } = await execute("python3",[@@ -50,14 +60,25 @@ process = subprocess.Popen(["node", sys.argv[1], "auth"], stdin=slave, stdout=suos.close(slave)output = b""try:- deadline = time.monotonic() + 5- while b"API key" not in output:- if time.monotonic() >= deadline:- raise RuntimeError("Auth prompt did not appear")- if select.select([process.stdout], [], [], 0.1)[0]:- output += os.read(process.stdout.fileno(), 4096)- if sys.argv[2] == "save":+ def wait_for(marker):+ global output+ deadline = time.monotonic() + 5+ while marker not in output:+ if time.monotonic() >= deadline:+ raise RuntimeError("Auth prompt did not appear")+ if select.select([process.stdout], [], [], 0.1)[0]:+ part = os.read(process.stdout.fileno(), 4096)+ if not part: raise RuntimeError("Auth exited before prompt")+ output += part+ wait_for(b"Choose your Jev provider")+ mode = sys.argv[2]+ if not mode.endswith("provider"):+ os.write(master, b"\\x1b[B\\x1b[B\\r")+ wait_for(b"API key")+ if mode == "save":os.write(master, b"pty-fixture-secret\\r")+ elif mode.startswith("cancel"):+ os.write(master, b"\\x03")else:process.send_signal(signal.SIGINT)rest, errors = process.communicate(timeout=5)@@ -79,13 +100,24 @@ finally:cli,mode,],- { env: { ...process.env, HOME: home, XDG_CONFIG_HOME: home, AI_GATEWAY_API_KEY: "" } },+ { env: { ...process.env, HOME: home, XDG_CONFIG_HOME: home } },);const result = JSON.parse(stdout);expect(result.code).toBe(mode === "save" ? 0 : 130);expect(result.stderr).toBe("");expect(result.stdout + result.echo).not.toContain("pty-fixture-secret");expect(result.stdout).toContain(mode === "save" ? "key saved" : "Interrupted");+ if (mode === "save") {+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({+ provider: "openrouter",+ apiKey: "pty-fixture-secret",+ });+ expect(result.stdout.indexOf("Vercel AI Gateway")).toBeLessThan(+ result.stdout.indexOf("TypeSafe"),+ );+ expect(result.stdout.indexOf("TypeSafe")).toBeLessThan(result.stdout.indexOf("OpenRouter"));+ } else expect(await readFile(file, "utf8")).toBe(previous);+ expect(await readdir(join(home, "jevgrep"))).toEqual(["credentials.json"]);}});});diff --git a/docs/architecture.md b/docs/architecture.mdindex 9a6a911..e17811a 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -95,11 +95,14 @@ does not guarantee that the coding agent tests the right behavioral boundary.## Integration and acceptance-Use TypeScript and the AI SDK evaluation interface through AI Gateway. Pass native+Use TypeScript and the AI SDK evaluation interface through one TypeSafe-compatible+adapter, with fixed Vercel, TypeSafe and OpenRouter presets. Pass native`state` and `questions` objects; source is a string field within that data. TheSDK handles HTTP serialization. Keep provider authentication and model access atthe core boundary so future providers do not reshape traversal or stdout.-The CLI authentication module handles the Gateway key; provider expansion is deferred.+The CLI auth module owns one saved provider/key. The core preset owner resolves+its endpoint/model; the evaluator owns retries and answer-cache identity. Provider+selection never changes traversal or source rendering.Acceptance measures the entire downstream Sol task: an official solve at a lowerfull task cost than its fixed baseline, while preserving baseline solves. Jevdiff --git a/evals/implementation/swebench/gateway_broker.py b/evals/implementation/swebench/gateway_broker.pyindex fba46ca..ea9e950 100644--- a/evals/implementation/swebench/gateway_broker.py+++ b/evals/implementation/swebench/gateway_broker.py@@ -74,13 +74,19 @@ class Gateway(http.server.BaseHTTPRequestHandler):codex=self.path=='/codex/v1/responses'engine='codex' if codex else 'claude' if claude else 'jev'if (codex or claude) and config.get('agent_engine')!=engine:self.send_error(403);return- jev=self.path=='/evaluation-model' and self.headers.get('ai-model-id')=='typesafe-ai/jev'+ jev=self.path=='/typesafe/v1/systemone' and self.headers.get('x-jevgrep-original-url')=='https://ai-gateway.vercel.sh/typesafe/v1/systemone'if jev and config.get('allow_jev') is not True:self.send_error(403);returnif not (claude or codex or jev):self.send_error(403);returnsize=int(self.headers.get('Content-Length','0'))if size<1 or size>2000000:self.send_error(413);returndata=self.rfile.read(size)- if jev:response_trace=capture_jev(request_id,data)+ if jev:+ try:body=json.loads(data)+ except ValueError:self.send_error(403);return+ if (not isinstance(body,dict) or body.get('model')!='typesafe-ai/jev' or not isinstance(body.get('state'),dict) or not isinstance(body.get('questions'),dict) or+ any(not isinstance(question,dict) or question.get('type')!='noul' or not isinstance(question.get('instructions'),str) for question in body['questions'].values())):+ self.send_error(403);return+ response_trace=capture_jev(request_id,data)headers={'Authorization':'Bearer '+config['key'],'Content-Type':'application/json'}if codex:body=json.loads(data)@@ -95,8 +101,7 @@ class Gateway(http.server.BaseHTTPRequestHandler):if self.headers.get(header):headers[header]=self.headers[header]target=GATEWAY_ORIGIN+self.pathelse:- target=GATEWAY_ORIGIN+'/v4/ai/evaluation-model'- headers.update({'ai-model-id':'typesafe-ai/jev','ai-evaluation-model-specification-version':'4','ai-gateway-protocol-version':'0.0.1','ai-gateway-auth-method':'api-key'})+ target=GATEWAY_ORIGIN+'/typesafe/v1/systemone'with LOG_LOCK:print(json.dumps({'kind':engine+'-request-start','requestId':request_id,'startedAt':time.time(),'operation':'count_tokens' if route.path.endswith('/count_tokens') else 'generation','streamed':bool(body.get('stream')) if claude or codex else False}),flush=True)request=urllib.request.Request(target,data=data,headers=headers)try:response=urllib.request.urlopen(request,timeout=90)diff --git a/evals/implementation/swebench/installed.py b/evals/implementation/swebench/installed.pyindex b03c17a..001f6a4 100644--- a/evals/implementation/swebench/installed.py+++ b/evals/implementation/swebench/installed.py@@ -19,6 +19,7 @@ import uuidHERE = Path(__file__).resolve().parentROOT = HERE.parents[2]+PROVIDER_ROUTE = HERE / 'provider-route.mjs' if (HERE / 'provider-route.mjs').exists() else ROOT / 'test/fixtures/provider-route.mjs'REGISTRY = json.loads((HERE / 'fixed-baselines.json').read_text())HARNESS_COMMIT = '02e7a74ffd0b707aab73d203fe87bdc7c76afc8e'TIMING_POLICY = {'clock': 'work_excluding_standalone_jg_search_wait', 'work_seconds': 900,@@ -171,13 +172,13 @@ tar -C /opt -cf /tmp/installed-prefix.tar jg-installwrite_json(out / 'agent-inputs.json', [public for _, _, public in pairs])frozen = out / 'runner'frozen.mkdir(mode=0o700)- for source in [Path(__file__).resolve(), HERE / 'gateway_broker.py', HERE / 'fixed-baselines.json']:+ for source in [Path(__file__).resolve(), HERE / 'gateway_broker.py', HERE / 'fixed-baselines.json', PROVIDER_ROUTE]:shutil.copyfile(source, frozen / source.name)artifacts = [*frozen.iterdir(), package, out / 'installed-prefix.tar', out / 'skill.md', out / 'agent-inputs.json', root / REGISTRY['dataset']]- plan = {'schema': 3, 'timing_policy': dict(TIMING_POLICY), 'status': 'frozen', 'cells': cells,+ plan = {'schema': 4, 'timing_policy': dict(TIMING_POLICY), 'status': 'frozen', 'cells': cells,'purpose': 'One prospective package cohort; reuse fixed baselines without execution.','inputs': str(out / 'agent-inputs.json'), 'skill': str(out / 'skill.md'), 'package': str(package),- 'runner': str(frozen / 'installed.py'), 'broker': str(frozen / 'gateway_broker.py'),+ 'runner': str(frozen / 'installed.py'), 'broker': str(frozen / 'gateway_broker.py'), 'provider_preload': str(frozen / 'provider-route.mjs'),'registry': str(frozen / 'fixed-baselines.json'), 'dataset': str(root / REGISTRY['dataset']),'tooling': str(root / REGISTRY['tooling']),'artifacts': {str(path): digest(path) for path in artifacts}}@@ -189,16 +190,18 @@ tar -C /opt -cf /tmp/installed-prefix.tar jg-installdef load_cohort(path):plan = json.loads(path.read_text())- if plan.get('schema') != 3 or plan.get('status') != 'frozen':- raise ValueError('Expected a frozen schema-3 cohort; use the archived runner for older studies')+ if plan.get('schema') != 4 or plan.get('status') != 'frozen':+ raise ValueError('Expected a frozen schema-4 cohort; use the archived runner for older studies')if plan.get('timing_policy') != TIMING_POLICY:raise ValueError('Frozen treatment timing policy changed')for artifact, expected in plan['artifacts'].items():if digest(artifact) != expected:raise ValueError('Frozen artifact changed: ' + artifact)- for key in ['runner', 'broker', 'registry', 'inputs', 'skill', 'package', 'dataset']:- if plan[key] not in plan['artifacts']:+ for key in ['runner', 'broker', 'provider_preload', 'registry', 'inputs', 'skill', 'package', 'dataset']:+ if plan.get(key) not in plan['artifacts']:raise ValueError('Plan omits required artifact: ' + key)+ if digest(PROVIDER_ROUTE) != digest(plan['provider_preload']):+ raise ValueError('Use the frozen provider preload')if digest(__file__) != digest(plan['runner']) or digest(HERE / 'gateway_broker.py') != digest(plan['broker']) or json.loads(Path(plan['registry']).read_text()) != REGISTRY:raise ValueError('Use the frozen runner and registry')prefixes = {item['cell']['candidate'] for item in plan['cells']}@@ -458,6 +461,12 @@ def run(args):receipt['prompt_sha256'] = digest(out / 'prompt.txt')token = uuid.uuid4().hexput(proxy, '/run/gateway.json', json.dumps({'key': os.environ['AI_GATEWAY_API_KEY'], 'token': token, 'agent_engine': 'codex', 'allow_jev': True}).encode(), 'root:root')+ dx(['mkdir', '-p', '/home/agent/.config/jevgrep'], 'agent')+ dx(['chmod', '700', '/home/agent/.config/jevgrep'], 'agent')+ dx(['mkdir', '-p', '/opt/jg-harness'])+ put(tag, '/home/agent/.config/jevgrep/credentials.json', json.dumps({'provider': 'vercel', 'apiKey': token}).encode())+ put(tag, '/opt/jg-harness/provider-route.mjs', Path(plan['provider_preload']).read_bytes(), 'root:root', '444')+ receipt['provider_preload_sha256'] = digest(plan['provider_preload'])config = ('model = "openai/gpt-5.6-sol"\nmodel_provider = "vercel"\nmodel_reasoning_effort = "medium"\n''[model_providers.vercel]\nname = "Vercel AI Gateway"\n''base_url = "http://model-egress:3129/codex/v1"\nenv_key = "JEVGREP_MODEL_TOKEN"\nwire_api = "responses"\n')@@ -467,7 +476,9 @@ def run(args):path = '/opt/jg-install/bin:/opt/node-v24.14.0-linux-x64/bin:/opt/miniconda3/envs/testbed/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'invocation = ['docker', 'exec', '-u', 'agent', '-e', 'HOME=/home/agent', '-e', 'PATH=' + path,'-e', 'CONDA_PREFIX=/opt/miniconda3/envs/testbed', '-e', 'JEVGREP_MODEL_TOKEN=' + token,- '-e', 'AI_GATEWAY_API_KEY=' + token, '-e', 'AI_GATEWAY_BASE_URL=http://model-egress:3129', '-w', '/testbed', tag, *native]+ '-e', 'XDG_CONFIG_HOME=/home/agent/.config',+ '-e', 'NODE_OPTIONS=--import=/opt/jg-harness/provider-route.mjs',+ '-e', 'JEVGREP_TEST_PROVIDER_ORIGIN=http://model-egress:3129', '-w', '/testbed', tag, *native]receipt['status'] = 'running'receipt['agent_started_at'] = time.time()write_json(out / 'receipt.json', receipt)@@ -578,19 +589,21 @@ def observed_jev(out, events, log_valid, traces_copied):try:response = responses[identifier]body = json.loads(response.read_text())- gateway = body['providerMetadata']['gateway']- value = gateway.get('cost')- if type(value) not in (str, int, float):- raise ValueError('Missing or invalid response cost')- cost = Decimal(str(value))- if not cost.is_finite() or cost < 0 or not math.isfinite(float(cost)):- raise ValueError('Invalid response cost')- costs.append(cost)usage = body.get('usage', {})- if count(usage.get('inputTokens')): inputs.append(usage['inputTokens'])- if count(usage.get('outputTokens')): outputs.append(usage['outputTokens'])+ if count(usage.get('input_tokens', usage.get('inputTokens'))): inputs.append(usage.get('input_tokens', usage.get('inputTokens')))+ if count(usage.get('output_tokens', usage.get('outputTokens'))): outputs.append(usage.get('output_tokens', usage.get('outputTokens')))+ gateway = body.get('provider_metadata', body.get('providerMetadata', {})).get('gateway', {})provider_attempts = gateway.get('routing', {}).get('totalProviderAttemptCount')if count(provider_attempts): attempts.append(provider_attempts)+ value = gateway.get('cost')+ # Billing can be absent while usage and transport coverage remain known.+ try:+ if type(value) not in (str, int, float): raise ValueError('Missing cost')+ cost = Decimal(str(value))+ if not cost.is_finite() or cost < 0 or not math.isfinite(float(cost)): raise ValueError('Invalid cost')+ costs.append(cost)+ except (ValueError, InvalidOperation):+ passmatching = [event for event in ends if event.get('requestId') == identifier]if len(matching) != 1:coverage = False@@ -606,7 +619,7 @@ def observed_jev(out, events, log_valid, traces_copied):if not math.isfinite(known):known = Nonecomplete = False- return {'basis': 'Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation',+ return {'basis': 'Retained response provider_metadata.gateway.cost or historical providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation','included_in_scored_task_cost': False, 'client_calls': len(ids), 'responses_with_cost': len(costs),'complete': complete, 'observed_cost_usd': known if complete else None, 'known_cost_usd': known,'input_tokens': sum(inputs) if coverage and len(inputs) == len(ids) else None,diff --git a/evals/implementation/swebench/test_installed.py b/evals/implementation/swebench/test_installed.pyindex 2652e95..a21fb8f 100644--- a/evals/implementation/swebench/test_installed.py+++ b/evals/implementation/swebench/test_installed.py@@ -170,9 +170,9 @@ class InstalledTests(unittest.TestCase):patcher=patch.object(runner,'REGISTRY',registry);patcher.start();self.addCleanup(patcher.stop)inputs = root / 'agent-inputs.json'; runner.write_json(inputs,rows)dataset=root/'dataset.json';dataset.write_text('fixture dataset')- paths = [dataset,Path(runner.__file__),HERE/'gateway_broker.py',registry_path,inputs,skill,package,prefix]- plan = {'schema':3,'timing_policy':dict(runner.TIMING_POLICY),'status':'frozen','cells':cells,'runner':str(Path(runner.__file__)),'broker':str(HERE/'gateway_broker.py'),- 'registry':str(registry_path),'tooling':str(root/'tooling'),'dataset':str(root/'dataset.json'),+ paths = [runner.PROVIDER_ROUTE,dataset,Path(runner.__file__),HERE/'gateway_broker.py',registry_path,inputs,skill,package,prefix]+ plan = {'schema':4,'timing_policy':dict(runner.TIMING_POLICY),'status':'frozen','cells':cells,'runner':str(Path(runner.__file__)),'broker':str(HERE/'gateway_broker.py'),+ 'provider_preload':str(runner.PROVIDER_ROUTE),'registry':str(registry_path),'tooling':str(root/'tooling'),'dataset':str(root/'dataset.json'),'inputs':str(inputs),'skill':str(skill),'package':str(package),'artifacts':{str(p):runner.digest(p) for p in paths}}plan_path=root/'plan.json';runner.write_json(plan_path,plan)return plan_path,{**plan,**cells[0]}@@ -186,9 +186,10 @@ class InstalledTests(unittest.TestCase):def test_timeout_with_zero_exit_retains_patch_but_cannot_complete_treatment(self):with tempfile.TemporaryDirectory() as temporary:path, plan = self.fixture(Path(temporary))- calls, elapsed, lifetime = [], [0], [float('inf')]+ calls, writes, elapsed, lifetime = [], {}, [0], [float('inf')]def docker(argv, checked=True, **kwargs):calls.append(argv)+ if 'input' in kwargs: writes[argv[-3]] = kwargs['input']if argv[:3] == ['docker', 'run', '-d'] and argv[-2] == 'sleep':lifetime[0] = float(argv[-1])if argv[:2] == ['docker', 'exec'] and elapsed[0] >= lifetime[0]:@@ -208,6 +209,13 @@ class InstalledTests(unittest.TestCase):output = b'fixture patch'return subprocess.CompletedProcess(argv, 0, output, b'')def expired(invocation, stdout, stderr, policy):+ self.assertIn('NODE_OPTIONS=--import=/opt/jg-harness/provider-route.mjs', invocation)+ self.assertIn('JEVGREP_TEST_PROVIDER_ORIGIN=http://model-egress:3129', invocation)+ self.assertFalse(any(arg.startswith('AI_GATEWAY_') for arg in invocation))+ saved = json.loads(writes['/home/agent/.config/jevgrep/credentials.json'])+ self.assertEqual(saved['provider'], 'vercel')+ self.assertNotEqual(saved['apiKey'], 'fixture-only')+ self.assertEqual(writes['/opt/jg-harness/provider-route.mjs'], runner.PROVIDER_ROUTE.read_bytes())# Native completion can already be in the pipe when the work# deadline is noticed. A zero OS exit cannot override that clock.# Simulate an hour of retrieval plus expired work. Docker's@@ -402,21 +410,86 @@ class InstalledTests(unittest.TestCase):self.assertEqual(result['gateway_cost_usd'],0.1);self.assertTrue(result['successful_cost_win'])self.assertEqual(result['jev']['observed_cost_usd'],100.02)+ def test_native_usage_survives_missing_cost(self):+ for cost in ['0.04', None, 'NaN', True, '0']:+ with self.subTest(cost=cost), tempfile.TemporaryDirectory() as temporary:+ root=Path(temporary);events=self.jev_fixture(root)+ for index in range(2):+ response=root/'jev-traces'/(str(index)*32+'.response.json')+ runner.write_json(response,{'usage':{'input_tokens':11,'output_tokens':3},'provider_metadata':{'gateway':{'cost':cost,'generationId':'fixture-generation'}}})+ events[index*2+1]['responseBytes']=response.stat().st_size+ result=runner.observed_jev(root,events,True,True)+ self.assertEqual(result['input_tokens'],22);self.assertEqual(result['output_tokens'],6)+ self.assertEqual(result['observed_cost_usd'],float(cost)*2 if cost in ('0.04','0') else None)+ self.assertEqual(result['complete'],cost in ('0.04','0'))++ def test_preload_is_required_and_identity_checked(self):+ with tempfile.TemporaryDirectory() as temporary:+ root=Path(temporary);path,plan=self.fixture(root)+ runner.load_cohort(path)+ plan=json.loads(path.read_text())+ plan['schema']=3;runner.write_json(path,plan)+ with self.assertRaisesRegex(ValueError,'schema-4'):runner.load_cohort(path)+ plan['schema']=4+ preload=root/'provider-route.mjs';preload.write_bytes(runner.PROVIDER_ROUTE.read_bytes())+ plan['provider_preload']=str(preload)+ runner.write_json(path,plan)+ with self.assertRaisesRegex(ValueError,'required artifact'):runner.load_cohort(path)+ plan['artifacts'][str(preload)]=runner.digest(preload);runner.write_json(path,plan)+ runner.load_cohort(path)+ preload.write_text('modified')+ with self.assertRaisesRegex(ValueError,'Frozen artifact changed'):runner.load_cohort(path)+ plan['artifacts'][str(preload)]=runner.digest(preload);runner.write_json(path,plan)+ with self.assertRaisesRegex(ValueError,'frozen provider preload'):runner.load_cohort(path)++ def test_broker_rejects_wrong_target_auth_and_body_before_capture(self):+ with tempfile.TemporaryDirectory() as temporary:+ root=Path(temporary);config=root/'gateway.json'+ config.write_text(json.dumps({'key':'real-key','token':'token','allow_jev':True}))+ route='/typesafe/v1/systemone'+ headers={'Authorization':'Bearer token','x-jevgrep-original-url':'https://ai-gateway.vercel.sh'+route}+ body={'model':'typesafe-ai/jev','state':{},'questions':{'q':{'type':'noul','instructions':'matches?'}}}+ cases=[('/evaluation-model',headers,body),(route+'?target=evil',headers,body),+ (route,{**headers,'Authorization':'Bearer wrong'},body),+ (route,{**headers,'x-jevgrep-original-url':'https://evil.example'+route},body),+ (route,headers,{**body,'model':'other'}),(route,headers,{**body,'state':None}),+ (route,headers,{**body,'questions':{'q':{'type':'boolean'}}}),(route,headers,[]),(route,headers,'invalid-json'),(route,{'Authorization':'Bearer token'},body)]+ with patch.object(broker,'CONFIG_PATH',str(config)),patch.object(broker,'TRACE_DIR',str(root/'traces')),patch.object(broker.urllib.request,'urlopen',side_effect=AssertionError('No forwarding')),contextlib.redirect_stdout(io.StringIO()):+ server=http.server.ThreadingHTTPServer(('127.0.0.1',0),broker.Gateway)+ thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()+ try:+ for path,auth,value in cases:+ client=http.client.HTTPConnection('127.0.0.1',server.server_port)+ client.request('POST',path,value if isinstance(value,str) else json.dumps(value),auth)+ response=client.getresponse();self.assertEqual(response.status,403);response.read();client.close()+ finally:server.shutdown();server.server_close();thread.join()+ self.assertFalse((root/'traces').exists())+def test_broker_captures_exact_jev_bodies_without_auth_headers(self):with tempfile.TemporaryDirectory() as temporary:root=Path(temporary);config=root/'gateway.json';config.write_text(json.dumps({'key':'REAL_KEY_SENTINEL','token':'BROKER_TOKEN_SENTINEL','allow_jev':True,'agent_engine':'codex'}))- request_body=b'{"state":{"source":"public fixture"},"questions":{}}';response_body=b'{"answers":{}}'- class Response(io.BytesIO):- status=200;headers={'Content-Type':'application/json'}- def read1(self,size):return self.read(size)- with patch.object(broker,'CONFIG_PATH',str(config)),patch.object(broker,'TRACE_DIR',str(root/'traces')),patch.object(broker.urllib.request,'urlopen',return_value=Response(response_body)),contextlib.redirect_stdout(io.StringIO()):+ request_body=b'{"model":"typesafe-ai/jev","state":{"source":"public fixture"},"questions":{"q":{"type":"noul","instructions":"matches?"}}}';response_body=b'{"answers":{},"usage":{"input_tokens":11,"output_tokens":3},"provider_metadata":{"gateway":{"cost":"0.04","generationId":"gen_fixture"}}}'+ forwarded={}+ class Upstream(http.server.BaseHTTPRequestHandler):+ def log_message(self,*args):pass+ def do_POST(self):+ forwarded.update(path=self.path,body=self.rfile.read(int(self.headers['Content-Length'])),headers=dict(self.headers))+ self.send_response(200);self.send_header('Content-Type','application/json');self.end_headers();self.wfile.write(response_body)+ upstream=http.server.ThreadingHTTPServer(('127.0.0.1',0),Upstream)+ upstream_thread=threading.Thread(target=upstream.serve_forever,daemon=True);upstream_thread.start()+ self.addCleanup(upstream_thread.join);self.addCleanup(upstream.server_close);self.addCleanup(upstream.shutdown)+ with patch.object(broker,'CONFIG_PATH',str(config)),patch.object(broker,'TRACE_DIR',str(root/'traces')),patch.object(broker,'GATEWAY_ORIGIN','http://127.0.0.1:'+str(upstream.server_port)),contextlib.redirect_stdout(io.StringIO()):server=http.server.ThreadingHTTPServer(('127.0.0.1',0),broker.Gateway)thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()try:client=http.client.HTTPConnection('127.0.0.1',server.server_port)- client.request('POST','/evaluation-model',request_body,{'Authorization':'Bearer BROKER_TOKEN_SENTINEL','ai-model-id':'typesafe-ai/jev'})+ client.request('POST','/typesafe/v1/systemone',request_body,{'Authorization':'Bearer BROKER_TOKEN_SENTINEL','x-jevgrep-original-url':'https://ai-gateway.vercel.sh/typesafe/v1/systemone'})response=client.getresponse();self.assertEqual(response.status,200);self.assertEqual(response.read(),response_body);client.close()finally:server.shutdown();server.server_close();thread.join()+ self.assertEqual(forwarded['path'],'/typesafe/v1/systemone')+ self.assertEqual(forwarded['body'],request_body)+ self.assertEqual(forwarded['headers']['Authorization'],'Bearer REAL_KEY_SENTINEL')+ self.assertNotIn('X-Jevgrep-Original-Url',forwarded['headers'])requests=list((root/'traces').glob('*.request.json'));responses=list((root/'traces').glob('*.response.json'))self.assertEqual(requests[0].read_bytes(),request_body);self.assertEqual(responses[0].read_bytes(),response_body)for file in requests+responses:diff --git a/packages/core/package.json b/packages/core/package.jsonindex 80d7f39..03890b7 100644--- a/packages/core/package.json+++ b/packages/core/package.json@@ -5,6 +5,7 @@"type": "module","exports": {".": "./src/index.ts",+ "./providers": "./src/providers.ts","./source": "./src/source.ts"},"scripts": {diff --git a/packages/core/src/gateway.ts b/packages/core/src/evaluator.tssimilarity index 88%rename from packages/core/src/gateway.tsrename to packages/core/src/evaluator.tsindex 419130b..ff4475b 100644--- a/packages/core/src/gateway.ts+++ b/packages/core/src/evaluator.ts@@ -1,4 +1,6 @@-import { createGateway, experimental_evaluate as evaluate } from "ai";+import { APICallError, experimental_evaluate as evaluate } from "ai";+import { createTypeSafeAi } from "@ai-sdk/typesafe-ai";+import { providers, type ProviderId } from "./providers";import { type createEvaluationCache, type CacheInput } from "./cache";import { setTimeout as delay } from "node:timers/promises";@@ -23,14 +25,16 @@ export class EvaluationFailure extends Error {}export function createEvaluator(options: {+ provider: ProviderId;apiKey: string;cache?: ReturnType<typeof createEvaluationCache>;policyVersion?: string;- baseURL?: string;+ fetch?: typeof fetch;signal: AbortSignal;requestLimit?: number;timeoutMs?: number;}) {+ const preset = providers[options.provider];let requests = 0;let cacheHits = 0;let cooldownUntil = 0;@@ -39,15 +43,15 @@ export function createEvaluator(options: {if (options.signal.aborted) throw new EvaluationFailure("cancelled");if (authenticationFailure.signal.aborted) throw new EvaluationFailure("authentication");}- const gateway = createGateway({+ const provider = createTypeSafeAi({apiKey: options.apiKey,- baseURL: options.baseURL,+ baseURL: preset.baseURL,fetch: async (input, init) => {assertActive();if (requests >= (options.requestLimit ?? 50_000))throw new EvaluationFailure("request-limit");requests++;- const response = await fetch(input, init);+ const response = await (options.fetch ?? fetch)(input, init);if (response.status === 429) {const raw = response.headers.get("retry-after");const seconds = raw === null ? NaN : Number(raw);@@ -81,8 +85,10 @@ export function createEvaluator(options: {const cacheInput: CacheInput = {request,namespace: {- model: "typesafe-ai/jev",- provider: options.baseURL ?? "vercel-ai-gateway",+ model: preset.model,+ provider: options.provider,+ endpoint: preset.baseURL,+ protocol: "typesafe-ai-3.0.8",policyVersion: options.policyVersion ?? "1",parserVersion: "cpython-3.11.3-pyodide-0.25.1-ts-5.9.3",promptVersion: "unit-locators-1",@@ -121,7 +127,7 @@ export function createEvaluator(options: {assertActive();try {const result = await evaluate({- model: gateway.evaluationModel("typesafe-ai/jev"),+ model: provider.evaluationModel(preset.model),...request,maxRetries: 0,abortSignal: AbortSignal.any([@@ -163,7 +169,8 @@ export function createEvaluator(options: {status === 408 ||status === 429 ||(typeof status === "number" && status >= 500 && status <= 599) ||- ["GatewayInternalServerError", "GatewayTimeoutError", "TimeoutError"].includes(name);+ name === "TimeoutError" ||+ (APICallError.isInstance(error) && error.statusCode === undefined && error.isRetryable);if (navigation && status === 429) attemptLimit = Math.max(attemptLimit, 2);if ((navigation && !transient) || attempt + 1 === attemptLimit)throw new EvaluationFailure(diff --git a/packages/core/src/index.ts b/packages/core/src/index.tsindex 771fd6a..97e0dfe 100644--- a/packages/core/src/index.ts+++ b/packages/core/src/index.ts@@ -1,4 +1,4 @@export { retrieve } from "./retrieve";-export { createEvaluator, EvaluationFailure } from "./gateway";+export { createEvaluator, EvaluationFailure } from "./evaluator";export { createEvaluationCache } from "./cache";export type { SearchInput, RetrievalResult, FileEvidence } from "./types";diff --git a/packages/core/src/providers.ts b/packages/core/src/providers.tsnew file mode 100644index 0000000..afd7a55--- /dev/null+++ b/packages/core/src/providers.ts@@ -0,0 +1,23 @@+export const providers = {+ vercel: {+ label: "Vercel AI Gateway",+ baseURL: "https://ai-gateway.vercel.sh/typesafe/v1",+ model: "typesafe-ai/jev",+ },+ typesafe: {+ label: "TypeSafe",+ baseURL: "https://api.typesafe.ai/v1",+ model: "jev-1.13.0",+ },+ openrouter: {+ label: "OpenRouter",+ baseURL: "https://openrouter.ai/api/v1",+ model: "jev-1.13",+ },+} as const;++export type ProviderId = keyof typeof providers;++export function isProviderId(value: unknown): value is ProviderId {+ return typeof value === "string" && Object.hasOwn(providers, value);+}diff --git a/packages/core/src/retrieve.ts b/packages/core/src/retrieve.tsindex 8d8d027..adff0b1 100644--- a/packages/core/src/retrieve.ts+++ b/packages/core/src/retrieve.ts@@ -1,6 +1,6 @@import { basename, extname } from "node:path";import { createFilesystem, type Snapshot, type DirectoryEntry } from "./filesystem";-import { type EvaluationRequest, EvaluationFailure } from "./gateway";+import { type EvaluationRequest, EvaluationFailure } from "./evaluator";import { inspect, pythonPreview, sourceForUnit, splitSource } from "./source";import {navigationRequest,diff --git a/packages/core/src/selection.ts b/packages/core/src/selection.tsindex 04f24d6..4c90464 100644--- a/packages/core/src/selection.ts+++ b/packages/core/src/selection.ts@@ -1,4 +1,4 @@-import { EvaluationFailure } from "./gateway";+import { EvaluationFailure } from "./evaluator";import { evidenceRequest, type Evidence } from "./requests";import {inspect,diff --git a/packages/core/src/types.ts b/packages/core/src/types.tsindex bbd14ab..31a3c1a 100644--- a/packages/core/src/types.ts+++ b/packages/core/src/types.ts@@ -1,5 +1,5 @@import type { FilesystemPolicy } from "./filesystem";-import type { EvaluationRequest } from "./gateway";+import type { EvaluationRequest } from "./evaluator";import type { Range } from "./source";export type { Range } from "./source";diff --git a/packages/core/test-node/provider-protocol.test.mjs b/packages/core/test-node/provider-protocol.mjssimilarity index 100%rename from packages/core/test-node/provider-protocol.test.mjsrename to packages/core/test-node/provider-protocol.mjsdiff --git a/packages/core/test/cache.test.ts b/packages/core/test/cache.test.tsindex 4fbddce..101316e 100644--- a/packages/core/test/cache.test.ts+++ b/packages/core/test/cache.test.ts@@ -75,10 +75,12 @@ test("source, path, query, ordered questions and namespace changes cannot reuse},{ ...input, request: { ...input.request, questions: ["question2", "question1"] } },{ ...input, request: { ...input.request, query: "different question" } },- ...["model", "provider", "policyVersion", "promptVersion"].map((name) => ({- ...input,- namespace: { ...input.namespace, [name]: "changed" },- })),+ ...["model", "provider", "endpoint", "protocol", "policyVersion", "promptVersion"].map(+ (name) => ({+ ...input,+ namespace: { ...input.namespace, [name]: "changed" },+ }),+ ),];for (const variant of variants) expect(await cache.get(variant)).toBeUndefined();expect(await cache.get(input)).toEqual({ question1: 0.8 });diff --git a/packages/core/test/selection.test.ts b/packages/core/test/selection.test.tsindex 0d1a055..a311699 100644--- a/packages/core/test/selection.test.ts+++ b/packages/core/test/selection.test.ts@@ -1,6 +1,6 @@import { expect, test } from "bun:test";import { selectFile } from "../src/selection";-import { EvaluationFailure, type EvaluationRequest } from "../src/gateway";+import { EvaluationFailure, type EvaluationRequest } from "../src/evaluator";import type { Evaluator } from "../src/types";import type { Declaration } from "../src/requests";diff --git a/scripts/licenses/README.md b/scripts/licenses/README.mdindex 2b20449..8d01e4f 100644--- a/scripts/licenses/README.md+++ b/scripts/licenses/README.md@@ -1,9 +1,10 @@# License sources omitted or abbreviated upstream-`provider-utils-5.0.45.LICENSE` is the Vercel AI repository license at-[the exact provider-utils 5.0.45 tag revision](https://github.com/vercel/ai/blob/08ae5ad05bc12496dd1ffcf64e34419e0831300d/LICENSE).-That npm package declares Apache-2.0 but omits its license file. The override is-version-specific so an upgrade cannot silently reuse an unchecked notice.+`vercel-ai.LICENSE` is the Vercel AI repository license, verified byte-for-byte at+[provider-utils 5.0.45](https://github.com/vercel/ai/blob/08ae5ad05bc12496dd1ffcf64e34419e0831300d/LICENSE)+and [provider-utils 5.0.49](https://github.com/vercel/ai/blob/ee3169b3c4880e2abe4d0d7c781243bb81822ec4/LICENSE).+Both npm packages declare Apache-2.0 but omit their license file. The notice+owner allows only these verified versions; a new version requires inspection.`Apache-2.0.txt` contains the [Apache Software Foundation's full license terms](https://www.apache.org/licenses/LICENSE-2.0.txt).The notice generator retains package copyright notices and includes these termsdiff --git a/scripts/licenses/provider-utils-5.0.45.LICENSE b/scripts/licenses/vercel-ai.LICENSEsimilarity index 100%rename from scripts/licenses/provider-utils-5.0.45.LICENSErename to scripts/licenses/vercel-ai.LICENSEdiff --git a/scripts/package-notices.mjs b/scripts/package-notices.mjsindex 7d44550..529d6c8 100644--- a/scripts/package-notices.mjs+++ b/scripts/package-notices.mjs@@ -4,6 +4,10 @@ import { dirname, resolve } from "node:path";/** Use emitted inputs, so the notice follows the actual bundled dependency graph. */export async function bundledNotices(metafile, root) {const packages = new Map();+ const retainedLicenses = {+ "@ai-sdk/[email protected]": "vercel-ai.LICENSE",+ "@ai-sdk/[email protected]": "vercel-ai.LICENSE",+ };let needsApacheTerms = false;for (const output of Object.values(metafile.outputs)) {for (const [input, contribution] of Object.entries(output.inputs)) {@@ -26,7 +30,7 @@ export async function bundledNotices(metafile, root) {).map((entry) => entry.name).sort();- if (!files.length && key !== "@ai-sdk/[email protected]")+ if (!files.length && !retainedLicenses[key])throw new Error(`Bundled dependency ${key} has no license file`);if (metadata.license === "Apache-2.0") needsApacheTerms = true;const texts = await Promise.all(@@ -39,7 +43,7 @@ export async function bundledNotices(metafile, root) {if (!files.length)texts.push(await readFile(- new URL("./licenses/provider-utils-5.0.45.LICENSE", import.meta.url),+ new URL(`./licenses/${retainedLicenses[key]}`, import.meta.url),"utf8",),);diff --git a/scripts/test-installed.sh b/scripts/test-installed.shindex 20be0db..db79a8d 100755--- a/scripts/test-installed.sh+++ b/scripts/test-installed.sh@@ -42,7 +42,7 @@ if [[ -n "$package_input" ]]; thencp "${JEVGREP_CANONICAL_SKILL:-$repo_root/skills/jevgrep/SKILL.md}" "$scratch/.package-input/canonical-skill.md"cp "$repo_root/test/installed.test.mjs" "$repo_root/test/runtime.Dockerfile" "$scratch/test/"mkdir -p "$scratch/test/fixtures"- cp "$repo_root/test/fixtures/skill-installer.mjs" "$scratch/test/fixtures/"+ cp "$repo_root/test/fixtures/skill-installer.mjs" "$repo_root/test/fixtures/provider-route.mjs" "$scratch/test/fixtures/"context_dir="$scratch"build_args=(--build-arg PACKAGE_STAGE=prebuilt)fidiff --git a/scripts/test-native.mjs b/scripts/test-native.mjsindex 37a7122..3dfe2b4 100644--- a/scripts/test-native.mjs+++ b/scripts/test-native.mjs@@ -5,7 +5,7 @@ import { createHash } from "node:crypto";import { cp, mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises";import { release, tmpdir } from "node:os";import { dirname, join, resolve } from "node:path";-import { fileURLToPath } from "node:url";+import { fileURLToPath, pathToFileURL } from "node:url";import { promisify } from "node:util";const execute = promisify(execFile);const root = fileURLToPath(new URL("../", import.meta.url));@@ -100,6 +100,8 @@ try {);const skill = join(scratch, "canonical-skill.md");await cp(process.env.JEVGREP_CANONICAL_SKILL || join(root, "skills/jevgrep/SKILL.md"), skill);+ const preload = join(scratch, "provider-route.mjs");+ await cp(join(root, "test/fixtures/provider-route.mjs"), preload);await symlink(process.execPath, join(runtimeBin, "node"));const runtimeEnv = {PATH: runtimeBin,@@ -107,8 +109,7 @@ try {XDG_CONFIG_HOME: config,XDG_CACHE_HOME: cache,TMPDIR: scratch,- AI_GATEWAY_API_KEY: "",- AI_GATEWAY_BASE_URL: "http://127.0.0.1:1/unused",+ JEVGREP_TEST_PROVIDER_PRELOAD: pathToFileURL(preload).href,JEVGREP_INSTALLED_BINARY: binary,JEVGREP_INSTALLED_PACKAGE: packageDirectory,JEVGREP_EXPECTED_SKILL: skill,@@ -130,7 +131,7 @@ try {"--test-reporter=tap","--test-concurrency=1","--test-name-pattern",- "^(installed local commands match the package without credentials|skill command delegates installation to npx without Gateway credentials|actual installed search parses Python and returns every relevant hierarchy branch)$",+ "^(installed local commands match the package without credentials|skill command delegates installation to npx without credentials|actual installed search parses Python and returns every relevant hierarchy branch|installed saved-provider journey: (vercel|typesafe|openrouter) auth doctor search cache replacement|installed legacy credentials use Vercel without rewriting saved bytes|installed saved credentials defeat conflicting environment and environment-only auth fails|installed invalid saved providers fail before HTTP without rewriting credentials|installed invalid auth preserves saved bytes and leaves no temporary credentials)$",join(root, "test/installed.test.mjs"),],{ cwd: scratch, env: runtimeEnv, timeout: 120_000, maxBuffer: 8_000_000 },@@ -138,7 +139,7 @@ try {assert.equal(tested.stderr, "");assert.match(tested.stdout,- /^# pass 3$/m,+ /^# pass 10$/m,"All native smoke journeys must execute; renamed selectors cannot silently pass",);console.log(tested.stdout.trimEnd());diff --git a/scripts/test-suite.sh b/scripts/test-suite.shindex 92ad1ec..dfa42ac 100644--- a/scripts/test-suite.sh+++ b/scripts/test-suite.sh@@ -1,7 +1,7 @@#!/usr/bin/env bashset -euo pipefail-bun test apps/cli/test packages/core/test test/reference test/gateway.test.ts test/retrieval.test.ts test/retrieval-freshness.test.ts-node --test packages/core/test-node/*.test.mjs+bun test apps/cli/test packages/core/test test/reference test/evaluator.test.ts test/retrieval.test.ts test/retrieval-freshness.test.ts+node --test packages/core/test-node/provider-protocol.mjsnode --experimental-strip-types --test test/parser/*.test.tsnode --test test/release.test.mjspython3 -B -m unittest discover -s evals/implementation/swebench -p test_installed.pydiff --git a/skills/jevgrep/SKILL.md b/skills/jevgrep/SKILL.mdindex 8105107..5c73bf6 100644--- a/skills/jevgrep/SKILL.md+++ b/skills/jevgrep/SKILL.md@@ -14,8 +14,8 @@ If prerequisites are missing or installation fails, report the specific blockerand use ordinary discovery; do not retry indefinitely or use sudo.If a search reports missing credentials, have the user run `jg auth` in their-terminal or configure `AI_GATEWAY_API_KEY` through their secret manager. Never ask-them to paste a key into chat. Do not launch the interactive auth prompt in a+terminal, select their provider, and enter its key. Saved credentials are required;+environment keys do not configure the CLI. Never ask them to paste a key into chat. Do not launch the interactive auth prompt in anoninteractive agent shell. Continue with ordinary discovery until configured.## Scopediff --git a/specs/provider-support/README.md b/specs/provider-support/README.mdindex 8db4b60..ba0beb0 100644--- a/specs/provider-support/README.md+++ b/specs/provider-support/README.md@@ -6,11 +6,11 @@ ordinary searches keep the same interface and retrieval behavior.## Next Agent Prompt-You are implementing this plan, last updated 2026-09-26. Slice 1 is complete:-18 Node protocol tests pass locally and in Docker, and all 5 frozen replay tests-remain green. Start [slice 2](slices/02-evaluator.md): replace the evaluator while-preserving production parity. Read [protocol evidence](assets/protocol-verification.md)-and [preservation gates](parity.md). No live provider calls have occurred.+Implementation is complete, last updated 2026-09-26. All four slices and required+checks pass; see [final verification](assets/final-verification.md). Close and+archive this spec after the final claim audit. TypeSafe/OpenRouter live access+remains unverified because credentials were unavailable; offline support is+verified and Vercel passed a live doctor check. No npm publication is selected.You have no unresolved product question. Published-package behavior and liveprovider access need verification, not invented compatibility code. Missing@@ -19,9 +19,9 @@ honestly. Do not collect secrets in chat or overwrite the user's own saved setupfor tests. Use isolated HOME/config/cache locations.- [x] [1. Protocol reproduction and controlled routing](slices/01-protocol.md)-- [ ] [2. Shared evaluator and production parity](slices/02-evaluator.md)-- [ ] [3. Saved-provider auth, installed journeys, and onboarding](slices/03-auth.md)-- [ ] [4. Maintained eval routing and final verification](slices/04-tooling.md)+- [x] [2. Shared evaluator and production parity](slices/02-evaluator.md)+- [x] [3. Saved-provider auth, installed journeys, and onboarding](slices/03-auth.md)+- [x] [4. Maintained eval routing and final verification](slices/04-tooling.md)Update this prompt and the owning slice with evidence, remaining work and anydeviations before ending each pass. Review each implementation slice beforediff --git a/specs/provider-support/assets/auth-verification.md b/specs/provider-support/assets/auth-verification.mdnew file mode 100644index 0000000..e0a2cf5--- /dev/null+++ b/specs/provider-support/assets/auth-verification.md@@ -0,0 +1,80 @@+# Saved-provider verification++2026-09-26. Installed Docker journeys cover Vercel, TypeSafe and OpenRouter:+auth from stdin, provider-specific doctor, search, cache reuse, replacement and+no fallback. They check private storage, ignored environment keys/URLs, invalid+providers/keys, legacy bytes and file identity remaining unchanged. The Docker+PTY test selects OpenRouter, checks hidden entry, and cancels at both prompts;+prior credentials survive and no temporary credential file remains.++All 32 installed Docker tests and all 10 native macOS smoke journeys passed on+the same candidate archive. The native run used Node v24.14.0, darwin/arm64 25.6.0.++## Live check++The actual built CLI ran this sequence with isolated config/cache directories:++```text+jg auth --provider vercel --stdin [key supplied privately; exit 0]+jg doctor+Jev connection verified through Vercel AI Gateway.+```++Doctor exited 0 with empty stderr. The key came from the authorized local Duet+environment, was never printed, and the temporary credentials were removed.+TypeSafe/OpenRouter keys were unavailable there; live service access for those+providers remains unverified. No repository source was used in the live check.++A fresh smaller-model skill check correctly directed a missing-credential user+to terminal `jg auth`, ignored an OpenRouter environment key, avoided asking for+secrets, and continued ordinary discovery. The initial consultation was invalid:+its instruction prohibited shell commands needed to read the skill. That was+corrected before the fresh check; no product behavior was changed to fit it.++## Exact verified reference packet++This is the complete frozen packet matched byte-for-byte by each provider's+production replay; it is not a sample invented for documentation.++```text+Jevgrep: 3 relevant files.+AGENTS.md lookup (root and returned-file ancestors): none found.+- "src/backend/events.ts" — implementation, caller, test, fixture, helper; selected source and structural context below+ Reading lead source: lines 1-1+ Reading lead BackendTelemetry.recordEvent: lines 4-6+- "src/telemetry.ts" — implementation, caller, test, fixture, helper; selected source and structural context below+ Reading lead Telemetry.recordEvent: lines 3-5+- "tests/telemetry.test.ts" — implementation, caller, test, fixture, helper; selected source and structural context below+ Reading lead source: lines 1-1+ Reading lead testEventName: lines 3-5+End file list.++Source block "src/backend/events.ts" lines 1-8:+1: import { Telemetry } from '../telemetry';+2: // Backends preserve the same event contract.+3: export class BackendTelemetry extends Telemetry {+4: recordEvent(name: string) {+5: return super.recordEvent(name);+6: }+7: }+8:++Source block "src/telemetry.ts" lines 1-7:+1: // Events preserve the caller's name.+2: export class Telemetry {+3: recordEvent(name: string) {+4: return { name, recorded: true };+5: }+6: }+7:++Source block "tests/telemetry.test.ts" lines 1-6:+1: import { Telemetry } from '../src/telemetry';+2: // Regression example: do not rename the caller's event.+3: export function testEventName() {+4: return new Telemetry().recordEvent('opened').name === 'opened';+5: }+6:++End context.+```diff --git a/specs/provider-support/assets/evaluator-verification.md b/specs/provider-support/assets/evaluator-verification.mdnew file mode 100644index 0000000..d9657f8--- /dev/null+++ b/specs/provider-support/assets/evaluator-verification.md@@ -0,0 +1,27 @@+# Evaluator preservation evidence++2026-09-26. All three fixed presets reproduce the frozen request multiset and+complete stdout through the built production CLI under controlled probabilities.+Missing and invalid responses remain incomplete with exit 2. The historical+oracle, manifest, corpus and skill research body were not changed.++The application still owns actual HTTP attempt counting, the 50,000 guard,+shared cooldown, retries, navigation splitting and cancellation. SDK retries are+disabled. The Node disconnect comparison matches the historical recovery path;+409/422 do not acquire a navigation retry or split; 503 remains split-eligible.+Numeric and HTTP-date Retry-After checks pass. Auth failure aborts siblings and+future requests. Existing retry-time source exclusion/mutation checks pass.++The cache keeps its existing answer-only storage. Provider replacement causes a+miss; old protocol, endpoint, model, policy, prompt and changed source cannot+share an answer. Warm identical requests reuse answers. The installed artifact+continues to hide excluded source and reject stale donors.++The broad Docker pass completed 125 core/CLI/reference tests, 18 Node protocol+checks, 23 parser tests and 4 release tests. Independent code review found no+remaining actionable defect. These are deterministic preservation checks, not a+new task-solve or savings benchmark.++The evaluator and auth were integrated as one cutover, rather than temporarily+publishing provider selection before transport routing. This changed commit+sequencing only; all planned behavior and gates remained.diff --git a/specs/provider-support/assets/final-verification.md b/specs/provider-support/assets/final-verification.mdnew file mode 100644index 0000000..62afb05--- /dev/null+++ b/specs/provider-support/assets/final-verification.md@@ -0,0 +1,41 @@+# Final provider-support verification++2026-09-26. Implementation gates passed; no npm release or paid task benchmark+was run. Existing baselines, historical study artifacts, the reference manifest,+covered source files, corpus and frozen research skill remain unchanged.++| Gate | Result |+| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |+| `bun run verify` | Exit 0: types, lint, 125 core/CLI/reference, 18 Node protocol, 23 parser, 4 release, 27 maintained harness and 32 installed Docker tests |+| `bun run format:check` | Exit 0 |+| Latest schema-4 harness, local Python and Docker | 27/27 in each, exit 0; rerun after the final plan-version adjustment |+| Exact candidate, installed Docker `--prebuilt` | 32/32, exit 0 |+| Same candidate, native `--prebuilt` | 10/10, exit 0; Node v24.14.0, darwin/arm64 25.6.0 |+| Independent whole-diff Codex review | No actionable defects; pending suite completion was not assumed by the reviewer |+| Live built CLI `jg doctor`, isolated saved Vercel setup | Exit 0, expected success text, empty stderr |+| Live TypeSafe/OpenRouter | Unverified: credentials unavailable in the authorized local environment |++Candidate: `@dzhng/jevgrep` source version 0.2.0, prepared with the existing package+validator but **not published**. SHA-256:++```text+d8d7d0012da3163f2f615fddc01e07795251af28296950b5f248d990322bd6b2+```++The packaged file allowlist excludes the routing preload and fixtures; runtime+requirements remain Node only. Docker runtime tests use network isolation and+synthetic local HTTP; dependency downloads used Docker Desktop's existing proxy+through a temporary Docker client config, without changing the user's settings.++The broker's real local-upstream test verifies auth replacement, exact native+request/response capture, and rejection before forwarding for wrong targets,+credentials or bodies. Synthetic accounting retains 22 input tokens when cost+is absent while the full Jev total remains unknown. Jev observations remain+excluded from scored coding-agent cost. This proves maintained tooling behavior,+not a newly measured solve rate, cost saving or latency.++Review and cleanup preserved the one evaluator/one preset owner, removed the old+module and environment setup path, retained exact-version license validation,+and avoided a public transport override. The sole new data-format choice is+recorded in [the choices ledger](../choices.md). No new benchmark cohort was+prepared or executed.diff --git a/specs/provider-support/assets/protocol-verification.md b/specs/provider-support/assets/protocol-verification.mdindex a143323..15d8236 100644--- a/specs/provider-support/assets/protocol-verification.md+++ b/specs/provider-support/assets/protocol-verification.md@@ -38,7 +38,7 @@ omits a license, like 5.0.45. The upstream repository LICENSE at tag revision5.0.45 license. Slice 2 must extend the exact-version notice mapping before theadapter becomes part of the product bundle.-Reproduce with `node --test packages/core/test-node/provider-protocol.test.mjs`;+Reproduce with `node --test packages/core/test-node/provider-protocol.mjs`;the same command runs inside `scripts/test-reference.sh`, and the normal suitenow includes it. No credentials or live model calls were used.diff --git a/specs/provider-support/choices.md b/specs/provider-support/choices.mdindex 4e8969e..0fd5c43 100644--- a/specs/provider-support/choices.md+++ b/specs/provider-support/choices.md@@ -1,10 +1,25 @@# Provider support choices-No unplanned product choice has been made in slice 1. The SDK pin, endpoint/model-presets, external routing seam and immutable reference were specified. Probe file-names and test layout are delegated implementation discretion.--The routing fixture preserves the SDK's original body instead of converting it-to a stream. This is a correctness fix within the specified transport contract,-not an additional product behavior. Further passes append only decisions not-already made or delegated by the plan.+## Sound — high confidence++### New benchmark plans use schema 4++When a new benchmark cohort is frozen, its plan now includes the exact routing+preload used by the installed CLI. A plan without that file cannot reproduce the+new transport. The plan did not specify how to version this required addition.+New plans use schema 4 so a reader can distinguish them from old schema-3 plans;+old studies continue using their original frozen runners and are never rewritten.+The alternative was to reuse schema 3 while changing its required fields, which+would conceal a real format change. Future tooling must honor this distinction.++This is a benchmark-harness format decision, not a credential migration: existing+saved providerless keys still work as Vercel, exactly as requested.++## Within the specification++Provider presets, single-record authentication, environment removal, legacy-key+handling, cache isolation, retry policy and the external test transport were+already decided by the spec. Internal names and probe layout were delegated.+The shared Node routing fixture preserves the SDK's original body after testing+showed that converting it to a stream obscures 401 errors; that implements the+specified transport contract rather than adding product behavior.diff --git a/specs/provider-support/slices/02-evaluator.md b/specs/provider-support/slices/02-evaluator.mdindex 185bd37..3020460 100644--- a/specs/provider-support/slices/02-evaluator.md+++ b/specs/provider-support/slices/02-evaluator.md@@ -1,7 +1,7 @@# 2. Replace the evaluator and prove production parity**Unlock:** one provider-neutral evaluator preserves the existing search contract.-Depends on slice 1. Status: not started.+Depends on slice 1. Status: complete; see [verification](../assets/evaluator-verification.md).## API seam and ownershipdiff --git a/specs/provider-support/slices/03-auth.md b/specs/provider-support/slices/03-auth.mdindex 96c380b..1529721 100644--- a/specs/provider-support/slices/03-auth.md+++ b/specs/provider-support/slices/03-auth.md@@ -1,7 +1,7 @@# 3. Save a provider and complete the installed journey**Unlock:** users can choose any supported service once and use ordinary `jg`.-Depends on slice 2. Status: not started.+Depends on slice 2. Status: complete; see [verification](../assets/auth-verification.md).## API and interactiondiff --git a/specs/provider-support/slices/04-tooling.md b/specs/provider-support/slices/04-tooling.mdindex fa458e8..9853fe3 100644--- a/specs/provider-support/slices/04-tooling.md+++ b/specs/provider-support/slices/04-tooling.md@@ -1,7 +1,7 @@# 4. Preserve maintained eval routing and close verification**Unlock:** the maintained harness can observe the actual installed product after-environment-based auth is removed. Depends on slice 3. Status: not started.+environment-based auth is removed. Depends on slice 3. Status: complete; see [verification](../assets/final-verification.md).## Seam and scopediff --git a/test/Dockerfile b/test/Dockerfileindex b8b05bc..118cfd7 100644--- a/test/Dockerfile+++ b/test/Dockerfile@@ -21,5 +21,5 @@ COPY evals/implementation/swebench/hierarchy-unit-locators-spike.ts evals/implemCOPY evals/implementation/swebench/installed.py evals/implementation/swebench/gateway_broker.py evals/implementation/swebench/test_installed.py evals/implementation/swebench/COPY evals/implementation/swebench/fixed-baselines.json evals/implementation/swebench/RUN bun build evals/implementation/swebench/hierarchy-unit-locators-spike.ts --target=node --outfile=/opt/jevgrep-reference.mjs-ENV JEVGREP_TEST_IN_DOCKER=1 HOME=/tmp/jevgrep-home XDG_CONFIG_HOME=/tmp/jevgrep-config XDG_CACHE_HOME=/tmp/jevgrep-cache AI_GATEWAY_API_KEY=""+ENV JEVGREP_TEST_IN_DOCKER=1 HOME=/tmp/jevgrep-home XDG_CONFIG_HOME=/tmp/jevgrep-config XDG_CACHE_HOME=/tmp/jevgrep-cacheCMD ["bun", "test", "test/reference/replay.test.ts"]diff --git a/test/gateway.test.ts b/test/evaluator.test.tssimilarity index 65%rename from test/gateway.test.tsrename to test/evaluator.test.tsindex 801a304..e5c609b 100644--- a/test/gateway.test.ts+++ b/test/evaluator.test.ts@@ -1,5 +1,6 @@+import { routeProviderFetch } from "./fixtures/provider-route.mjs";import { expect, test } from "bun:test";-import { createEvaluator } from "../packages/core/src/gateway";+import { createEvaluator } from "../packages/core/src/evaluator";test("Jev uses native state and validated boolean probabilities through real HTTP", async () => {const state = {@@ -10,17 +11,18 @@ test("Jev uses native state and validated boolean probabilities through real HTTport: 0,async fetch(request) {expect(await request.json()).toEqual({+ model: "typesafe-ai/jev",state,- questions: { useful: { type: "boolean", instructions: "Is the source useful?" } },- providerOptions: {},+ questions: { useful: { type: "noul", instructions: "Is the source useful?" } },});- return Response.json({ answers: { useful: { type: "boolean", probability: 0.8 } } });+ return Response.json({ answers: { useful: { type: "noul", noul: 0.8 } } });},});try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: new AbortController().signal,});const result = await evaluator.evaluate({@@ -42,13 +44,14 @@ test("transient failures retry within the shared request guard and never becomecalls++;return calls === 1? Response.json({ error: "retry" }, { status: 503 })- : Response.json({ answers: { q: { type: "boolean", probability: 0.2 } } });+ : Response.json({ answers: { q: { type: "noul", noul: 0.2 } } });},});try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: new AbortController().signal,requestLimit: 2,});@@ -86,7 +89,8 @@ test("cancellation of a rate-limited request prevents further attempts", async (try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: controller.signal,});const result = evaluator.evaluate({@@ -115,13 +119,14 @@ test("Retry-After delays a retry before the provider can recover", async () => {{ error: "rate limited" },{ status: 429, headers: { "retry-after": "0.1" } },)- : Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });+ : Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });},});try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: new AbortController().signal,});expect(@@ -150,7 +155,8 @@ test("stalled HTTP attempts time out without exceeding the evaluator attempt limtry {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: new AbortController().signal,timeoutMs: 100,});@@ -175,13 +181,14 @@ test("authentication failure stops other in-flight and subsequent query requestscalls++;if (calls === 1) return new Promise<Response>(() => {});if (calls === 2) return Response.json({ error: "unauthorized" }, { status: 401 });- return Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });+ return Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });},});try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: new AbortController().signal,timeoutMs: 1000,});@@ -225,7 +232,8 @@ test("authentication failure interrupts a sibling Retry-After wait", async () =>try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: new AbortController().signal,});const request = {@@ -245,3 +253,75 @@ test("authentication failure interrupts a sibling Retry-After wait", async () =>server.stop(true);}}, 2000);++for (const status of [409, 422, 503]) {+ test(`navigation HTTP ${status} preserves retry and split policy`, async () => {+ let calls = 0;+ const server = Bun.serve({+ port: 0,+ fetch() {+ calls++;+ return Response.json({ message: "synthetic failure" }, { status });+ },+ });+ try {+ const evaluator = createEvaluator({+ provider: "openrouter",+ apiKey: "fixture",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),+ signal: new AbortController().signal,+ });+ await expect(+ evaluator.evaluate(+ {+ state: { source: "code" },+ questions: {+ q0: { type: "boolean", instructions: "Relevant?" },+ q1: { type: "boolean", instructions: "Related?" },+ },+ },+ { navigation: true },+ ),+ ).rejects.toMatchObject({ kind: "provider", splitEligible: status === 503 });+ expect(calls).toBe(1);+ expect(evaluator.requests).toBe(1);+ } finally {+ server.stop(true);+ }+ });+}++test("native HTTP-date Retry-After is honored before another attempt", async () => {+ const retryAt = new Date(Math.ceil(Date.now() / 1000) * 1000 + 1000);+ const arrivals: number[] = [];+ const server = Bun.serve({+ port: 0,+ fetch() {+ arrivals.push(Date.now());+ return arrivals.length === 1+ ? Response.json(+ { message: "rate limited" },+ { status: 429, headers: { "retry-after": retryAt.toUTCString() } },+ )+ : Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });+ },+ });+ try {+ const evaluator = createEvaluator({+ provider: "typesafe",+ apiKey: "fixture",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),+ signal: new AbortController().signal,+ });+ expect(+ await evaluator.evaluate({+ state: "code",+ questions: { q: { type: "boolean", instructions: "Relevant?" } },+ }),+ ).toEqual({ q: 0.8 });+ expect(arrivals.length).toBe(2);+ expect(arrivals[1]!).toBeGreaterThanOrEqual(retryAt.getTime());+ } finally {+ server.stop(true);+ }+}, 5000);diff --git a/test/installed.test.mjs b/test/installed.test.mjsindex fe58b21..77484f7 100644--- a/test/installed.test.mjs+++ b/test/installed.test.mjs@@ -9,6 +9,7 @@ import {mkdir,mkdtemp,readFile,+ readdir,realpath,rm,stat,@@ -27,6 +28,20 @@ assert.ok(binary && packageDirectory && expectedSkill,"Run scripts/test-installed.sh in the Node-only container",);+// These independent expectations pin the public routes of the installed artifact.+const providers = {+ vercel: {+ label: "Vercel AI Gateway",+ url: "https://ai-gateway.vercel.sh/typesafe/v1/systemone",+ model: "typesafe-ai/jev",+ },+ typesafe: { label: "TypeSafe", url: "https://api.typesafe.ai/v1/systemone", model: "jev-1.13.0" },+ openrouter: {+ label: "OpenRouter",+ url: "https://openrouter.ai/api/v1/systemone",+ model: "jev-1.13",+ },+};const fixtureKey = "installed-http-fixture-key";const forbidden = "INSTALLED_FIXTURE_IGNORED_CONTENT_MUST_NEVER_UPLOAD";const query = "Find event recording implementations across the nested packages.";@@ -40,6 +55,9 @@ const source = (branch, name) =>async function context(t, mode = "healthy", executable = binary) {let expectedQuery = query;+ let expectedProvider = "vercel";+ let expectedKey = fixtureKey;+ const secrets = new Set([fixtureKey]);const scratch = await mkdtemp(join(tmpdir(), "jg-installed-"));const tree = join(scratch, "repository");const home = join(scratch, "home");@@ -83,14 +101,22 @@ async function context(t, mode = "healthy", executable = binary) {await writeFile(join(tree, "hidden.skip.py"), `value = "${forbidden}"\n`);await writeFile(join(tree, ".env"), `PRIVATE_VALUE=${forbidden}\n`);await writeFile(join(tree, "unrelated.md"), "This is an unrelated gardening document.\n");+ const credentialDirectory = join(config, "jevgrep");+ const credentials = join(credentialDirectory, "credentials.json");+ await mkdir(credentialDirectory, { mode: 0o700 });+ await writeFile(credentials, JSON.stringify({ provider: "vercel", apiKey: fixtureKey }) + "\n", {+ mode: 0o600,+ });server = createServer((request, response) => {void (async () => {assert.equal(request.method, "POST");- assert.equal(request.url, "/v4/ai/evaluation-model");- assert.equal(request.headers.authorization, `Bearer ${fixtureKey}`);- assert.equal(request.headers["ai-model-id"], "typesafe-ai/jev");- assert.equal(request.headers["ai-evaluation-model-specification-version"], "4");- assert.equal(request.headers["ai-gateway-protocol-version"], "0.0.1");+ const preset = providers[expectedProvider];+ assert.equal(request.url, new URL(preset.url).pathname);+ assert.equal(request.headers["x-jevgrep-original-url"], preset.url);+ assert.ok(+ request.headers.authorization === `Bearer ${expectedKey}`,+ "Saved key must authenticate the request",+ );assert.match(request.headers["content-type"] ?? "", /^application\/json/);const chunks = [];let bytes = 0;@@ -102,7 +128,8 @@ async function context(t, mode = "healthy", executable = binary) {const raw = Buffer.concat(chunks).toString("utf8");assert.ok(!raw.includes(forbidden), "Ignored/hidden source reached the provider");const body = JSON.parse(raw);- assert.deepEqual(Object.keys(body), ["state", "questions", "providerOptions"]);+ assert.deepEqual(Object.keys(body).sort(), ["model", "questions", "state"]);+ assert.equal(body.model, preset.model);assert.equal(typeof body.state,"object",@@ -111,11 +138,11 @@ async function context(t, mode = "healthy", executable = binary) {assert.ok(body.state !== null && !Array.isArray(body.state));assert.ok(Object.keys(body.questions).length > 0);for (const question of Object.values(body.questions)) {- assert.equal(question.type, "boolean");+ assert.equal(question.type, "noul");assert.ok(typeof question.instructions === "string" && question.instructions.length > 0);}assert.ok(requests.length < 256, "Synthetic search stopped making bounded forward progress");- requests.push({ body, raw, receivedAt: performance.now() });+ requests.push({ body, raw, provider: expectedProvider, receivedAt: performance.now() });if (typeof mode === "function" && (await mode({ body, raw, tree, response }))) return;if (mode === "rate-limit" && requests.length === 1) {response.writeHead(429, { "content-type": "application/json", "retry-after": "1" });@@ -187,9 +214,9 @@ async function context(t, mode = "healthy", executable = binary) {response.end(JSON.stringify({answers: Object.fromEntries(- ids.map((id, index) => [id, { type: "boolean", probability: probabilities[index] }]),+ ids.map((id, index) => [id, { type: "noul", noul: probabilities[index] }]),),- usage: { inputTokens: 1, outputTokens: 1 },+ usage: { input_tokens: 1, output_tokens: 1 },warnings: [{ type: "other", message: "installed-fixture-warning" }],}),);@@ -209,10 +236,11 @@ async function context(t, mode = "healthy", executable = binary) {XDG_CONFIG_HOME: config,XDG_CACHE_HOME: cache,TMPDIR: scratch,- AI_GATEWAY_API_KEY: fixtureKey,- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.address().port}/v4/ai`,+ NODE_OPTIONS: `--import=${process.env.JEVGREP_TEST_PROVIDER_PRELOAD ?? new URL("./fixtures/provider-route.mjs", import.meta.url).href}`,+ JEVGREP_TEST_PROVIDER_ORIGIN: `http://127.0.0.1:${server.address().port}`,};- const run = async (args, overrides = {}, head = false) => {+ const run = async (args, overrides = {}, head = false, input) => {+ if (input?.trim()) secrets.add(input.trim());const result = await new Promise((resolve, reject) => {const child = spawn(head ? "bash" : executable === binary ? binary : process.execPath,@@ -225,10 +253,16 @@ async function context(t, mode = "healthy", executable = binary) {cwd: tree,detached: true,env: { ...env, ...overrides },- stdio: ["ignore", "pipe", "pipe"],+ stdio: [input === undefined ? "ignore" : "pipe", "pipe", "pipe"],},);children.add(child);+ if (input !== undefined) {+ child.stdin.on("error", (error) => {+ if (error.code !== "EPIPE") reject(error);+ });+ child.stdin.end(input);+ }const stdout = [],stderr = [];let outputBytes = 0;@@ -267,7 +301,8 @@ async function context(t, mode = "healthy", executable = binary) {"","All application output, including SDK warnings, belongs on stdout",);- assert.ok(!result.stdout.includes(fixtureKey));+ for (const secret of secrets)+ assert.ok(!result.stdout.includes(secret), "Credential leaked in stdout");assert.ok(!result.stdout.includes("installed-fixture-warning"));assert.ok(!result.stdout.includes(forbidden));return result;@@ -275,7 +310,18 @@ async function context(t, mode = "healthy", executable = binary) {return {run,tree,+ credentials,+ config,+ credentialDirectory,requests,+ expectProvider(provider, key = fixtureKey) {+ expectedProvider = provider;+ expectedKey = key;+ secrets.add(key);+ },+ async removeCredentials() {+ await rm(credentials, { force: true });+ },set query(value) {expectedQuery = value;},@@ -352,7 +398,8 @@ test("installed local commands match the package without credentials", async (t)t.diagnostic(`Runtime ${process.version} ${process.platform}/${process.arch}; installed ${metadata.name}@${metadata.version}`,);- const noCredentials = { AI_GATEWAY_API_KEY: "" };+ await fixture.removeCredentials();+ const noCredentials = {};const help = await fixture.run(["--help"], noCredentials);assert.equal(help.code, 0, help.stdout);assert.match(help.stdout, /Usage: jg /);@@ -363,11 +410,12 @@ test("installed local commands match the package without credentials", async (t)await readFile(join(packageDirectory, "dist/skills/jevgrep/SKILL.md"), "utf8"),await readFile(expectedSkill, "utf8"),);- assert.equal(fixture.requests.length, 0, "Local commands must not contact Gateway");+ assert.equal(fixture.requests.length, 0, "Local commands must not contact a provider");});-test("skill command delegates installation to npx without Gateway credentials", async (t) => {+test("skill command delegates installation to npx without credentials", async (t) => {const fixture = await context(t);+ await fixture.removeCredentials();const bin = join(fixture.tree, "installer-bin");await mkdir(bin);await symlink(process.execPath, join(bin, "node"));@@ -375,7 +423,7 @@ test("skill command delegates installation to npx without Gateway credentials",await symlink(new URL("./fixtures/skill-installer.mjs", import.meta.url), npx);const result = await fixture.run(["skill", "--agent", "codex", "--agent", "claude-code", "--global", "--yes"],- { PATH: bin, AI_GATEWAY_API_KEY: "" },+ { PATH: bin },);assert.equal(result.code, 0, result.stdout);assert.match(result.stdout, /Installer completed/);@@ -396,7 +444,6 @@ test("skill command delegates installation to npx without Gateway credentials",assert.equal(fixture.requests.length, 0);const failed = await fixture.run(["skill"], {PATH: bin,- AI_GATEWAY_API_KEY: "",JEVGREP_INSTALLER_EXIT: "7",});assert.equal(failed.code, 7, failed.stdout);@@ -409,7 +456,7 @@ test("skill command delegates installation to npx without Gateway credentials","jevgrep",]);await rm(npx);- const unavailable = await fixture.run(["skill"], { PATH: bin, AI_GATEWAY_API_KEY: "" });+ const unavailable = await fixture.run(["skill"], { PATH: bin });assert.equal(unavailable.code, 1);assert.match(unavailable.stdout, /requires npx/);});@@ -494,12 +541,15 @@ test("warm cache reuses identical requests, no-cache bypasses reuse, and edits iassert.ok(!changed.stdout.includes("py-evidence-alpha:"));assert.ok(fixture.requests.slice(before).some(({ raw }) => raw.includes("edited-alpha-evidence:")),- "Changed source must reach Gateway instead of stale cache evidence",+ "Changed source must reach the provider instead of stale cache evidence",);before = fixture.requests.length;assert.equal((await fixture.run([query, fixture.tree])).stdout, changed.stdout);assertCachedRequestsAreReused(fixture.requests, before);- const cleared = await fixture.run(["cache", "clear"], { AI_GATEWAY_API_KEY: "" });+ const saved = await readFile(fixture.credentials);+ await fixture.removeCredentials();+ const cleared = await fixture.run(["cache", "clear"]);+ await writeFile(fixture.credentials, saved, { mode: 0o600 });assert.equal(cleared.code, 0, cleared.stdout);before = fixture.requests.length;assert.equal((await fixture.run([query, fixture.tree])).code, 0);@@ -512,7 +562,8 @@ test("doctor uses the installed SDK while missing credentials fail cleanly", asyassert.equal(doctor.code, 0, doctor.stdout);assert.ok(fixture.requests.length > 0);const before = fixture.requests.length;- const missing = await fixture.run([query, fixture.tree], { AI_GATEWAY_API_KEY: "" });+ await fixture.removeCredentials();+ const missing = await fixture.run([query, fixture.tree]);assert.equal(missing.code, 1, missing.stdout);assert.ok(missing.stdout.trim().length > 0);assert.equal(fixture.requests.length, before);@@ -766,7 +817,8 @@ test("missing or corrupt packaged Python assets fail closed without downloads",!fixture.requests.some(({ body }) => body.state.declarations),"Unavailable parser assets must not fabricate declaration evidence",);- assert.equal((await fixture.run(["--help"], { AI_GATEWAY_API_KEY: "" })).code, 0);+ await fixture.removeCredentials();+ assert.equal((await fixture.run(["--help"])).code, 0);await rm(copy, { recursive: true, force: true });}});@@ -786,7 +838,7 @@ for (const mutation of ["changed", "ignored"])response.end(JSON.stringify({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),),}),);@@ -826,7 +878,7 @@ test("installed queued freshness withholds excluded source uploads", async (t) =response.end(JSON.stringify({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.1 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.1 }]),),}),);@@ -845,3 +897,166 @@ test("installed queued freshness withholds excluded source uploads", async (t) =assert.match(result.stdout, /incomplete/);assert.ok(!result.stdout.includes("QUEUED_INSTALLED_SENTINEL"));});++for (const [provider, preset] of Object.entries(providers))+ test(`installed saved-provider journey: ${provider} auth doctor search cache replacement`, async (t) => {+ const fixture = await context(t);+ await fixture.removeCredentials();+ const key = `installed-${provider}-saved-key`;+ fixture.expectProvider(provider, key);+ const auth = await fixture.run(+ ["auth", "--provider", provider, "--stdin"],+ {},+ false,+ key + "\n",+ );+ assert.equal(auth.code, 0, auth.stdout);+ assert.ok(auth.stdout.includes(preset.label));+ assert.match(auth.stdout, /doctor/);+ assert.equal(fixture.requests.length, 0, "Auth must not contact any provider");+ assert.deepEqual(JSON.parse(await readFile(fixture.credentials, "utf8")), {+ provider,+ apiKey: key,+ });+ assert.equal((await stat(fixture.credentials)).mode & 0o777, 0o600);+ assert.equal((await stat(fixture.credentialDirectory)).mode & 0o777, 0o700);+ const doctor = await fixture.run(["doctor"]);+ assert.equal(doctor.code, 0, doctor.stdout);+ assert.ok(doctor.stdout.includes(preset.label));+ assert.equal(fixture.requests.length, 1);+ assert.deepEqual(Object.keys(fixture.requests[0].body.questions), ["relevant"]);+ assert.ok(+ !fixture.requests[0].raw.includes("py-evidence-"),+ "Doctor must use only synthetic source",+ );+ const cold = await fixture.run([query]);+ complete(cold);+ const before = fixture.requests.length;+ const warm = await fixture.run([query]);+ complete(warm);+ assert.equal(warm.stdout, cold.stdout);+ assertCachedRequestsAreReused(fixture.requests, before);++ const replacement = provider === "vercel" ? "typesafe" : "vercel";+ const replacementKey = `installed-${replacement}-replacement-key`;+ const beforeAuth = fixture.requests.length;+ const replaced = await fixture.run(+ ["auth", "--provider", replacement, "--stdin"],+ {},+ false,+ replacementKey + "\n",+ );+ assert.equal(replaced.code, 0, replaced.stdout);+ assert.equal(fixture.requests.length, beforeAuth);+ assert.deepEqual(JSON.parse(await readFile(fixture.credentials, "utf8")), {+ provider: replacement,+ apiKey: replacementKey,+ });+ fixture.expectProvider(replacement, replacementKey);+ const search = await fixture.run([query]);+ complete(search);+ assert.equal(search.stdout, cold.stdout);+ assert.ok(+ fixture.requests.length > beforeAuth,+ "Provider replacement must not reuse another provider's cache",+ );+ assert.ok(+ fixture.requests.slice(beforeAuth).every((request) => request.provider === replacement),+ );+ fixture.mode = "unauthorized";+ const beforeFailure = fixture.requests.length;+ const failed = await fixture.run(["doctor"]);+ assert.equal(failed.code, 1, failed.stdout);+ assert.equal(+ fixture.requests.length,+ beforeFailure + 1,+ "Authentication failure must not retry or fall back",+ );+ assert.ok(failed.stdout.includes(providers[replacement].label));+ assert.equal(fixture.requests.at(-1).provider, replacement);+ });++test("installed legacy credentials use Vercel without rewriting saved bytes", async (t) => {+ const fixture = await context(t);+ const original = ' { "apiKey": "' + fixtureKey + '" }\n';+ await writeFile(fixture.credentials, original);+ const before = await stat(fixture.credentials);+ const doctor = await fixture.run(["doctor"]);+ assert.equal(doctor.code, 0, doctor.stdout);+ assert.ok(doctor.stdout.includes(providers.vercel.label));+ complete(await fixture.run([query]));+ assert.ok(fixture.requests.every((request) => request.provider === "vercel"));+ assert.equal(await readFile(fixture.credentials, "utf8"), original);+ const after = await stat(fixture.credentials);+ assert.equal(after.mtimeMs, before.mtimeMs);+ assert.equal(after.ino, before.ino);+});++test("installed saved credentials defeat conflicting environment and environment-only auth fails", async (t) => {+ const fixture = await context(t);+ const conflicts = {+ AI_GATEWAY_API_KEY: "environment-gateway-key",+ TYPESAFE_API_KEY: "environment-typesafe-key",+ TYPESAFE_AI_API_KEY: "environment-sdk-typesafe-key",+ OPENROUTER_API_KEY: "environment-openrouter-key",+ AI_GATEWAY_BASE_URL: "http://127.0.0.1:1/forbidden",+ TYPESAFE_BASE_URL: "http://127.0.0.1:1/forbidden",+ OPENROUTER_BASE_URL: "http://127.0.0.1:1/forbidden",+ AI_GATEWAY_MODEL: "environment-model",+ TYPESAFE_MODEL: "environment-model",+ OPENROUTER_MODEL: "environment-model",+ };+ for (const provider of Object.keys(providers)) {+ await writeFile(fixture.credentials, JSON.stringify({ provider, apiKey: fixtureKey }));+ fixture.expectProvider(provider);+ const doctor = await fixture.run(["doctor"], conflicts);+ assert.equal(doctor.code, 0, doctor.stdout);+ assert.ok(doctor.stdout.includes(providers[provider].label));+ complete(await fixture.run([query, "--no-cache"], conflicts));+ }+ await fixture.removeCredentials();+ const before = fixture.requests.length;+ for (const args of [["doctor"], [query]]) {+ const result = await fixture.run(args, conflicts);+ assert.equal(result.code, 1, result.stdout);+ assert.match(result.stdout, /jg auth/);+ assert.ok(!result.stdout.includes("environment-"));+ }+ assert.equal(fixture.requests.length, before);+});++test("installed invalid saved providers fail before HTTP without rewriting credentials", async (t) => {+ const fixture = await context(t);+ for (const provider of [null, "", "unknown", false, 0, {}, []]) {+ const original = JSON.stringify({ provider, apiKey: fixtureKey }) + "\n";+ await writeFile(fixture.credentials, original);+ for (const args of [["doctor"], [query]]) {+ const result = await fixture.run(args);+ assert.equal(result.code, 1, result.stdout);+ assert.match(result.stdout, /auth/);+ assert.equal(await readFile(fixture.credentials, "utf8"), original);+ }+ }+ assert.equal(fixture.requests.length, 0);+});++test("installed invalid auth preserves saved bytes and leaves no temporary credentials", async (t) => {+ const fixture = await context(t);+ const original = await readFile(fixture.credentials, "utf8");+ for (const [args, input] of [+ [["auth", "--stdin"], "invalid-provider-key\n"],+ [["auth", "--provider", "typesafe"], undefined],+ [["auth", "--provider", "unknown", "--stdin"], "invalid-provider-key\n"],+ [["auth", "--provider", "vercel", "--stdin"], "\n"],+ [["auth", "--provider", "typesafe", "--stdin"], "key with spaces\n"],+ [["auth", "--provider", "openrouter", "--stdin"], "x".repeat(8193)],+ [["doctor", "--provider", "vercel"], undefined],+ [[query, "--provider", "typesafe"], undefined],+ ]) {+ const result = await fixture.run(args, {}, false, input);+ assert.equal(result.code, 1, result.stdout);+ assert.equal(await readFile(fixture.credentials, "utf8"), original);+ assert.deepEqual(await readdir(fixture.credentialDirectory), ["credentials.json"]);+ }+ assert.equal(fixture.requests.length, 0);+});diff --git a/test/reference/disconnect.test.ts b/test/reference/disconnect.test.tsindex c9ee48a..c3663be 100644--- a/test/reference/disconnect.test.ts+++ b/test/reference/disconnect.test.ts@@ -1,3 +1,4 @@+import { referenceTransport, wireResponse } from "./transport";import { expect } from "bun:test";import { createServer } from "node:http";import { resolve } from "node:path";@@ -5,10 +6,19 @@ import { testIfDocker as test } from "../helpers/docker";async function disconnected(reference: boolean) {const bodies: string[] = [];+ const transport = await referenceTransport();const server = createServer(async (request, response) => {const chunks: Buffer[] = [];for await (const chunk of request) chunks.push(Buffer.from(chunk));- const body = JSON.parse(Buffer.concat(chunks).toString());+ const body = transport.decode(+ {+ url: `http://localhost${request.url}`,+ method: request.method,+ headers: new Headers(request.headers as Record<string, string>),+ },+ JSON.parse(Buffer.concat(chunks).toString()),+ !reference,+ );bodies.push(JSON.stringify(body));if (bodies.length === 1) {response.destroy();@@ -16,11 +26,16 @@ async function disconnected(reference: boolean) {}response.writeHead(200, { "content-type": "application/json" });response.end(- JSON.stringify({- answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.05 }]),+ JSON.stringify(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.05 }]),+ ),+ },+ !reference,),- }),+ ),);});await new Promise<void>((done) => server.listen(0, "127.0.0.1", done));@@ -34,11 +49,7 @@ async function disconnected(reference: boolean) {? ["node", "/opt/jevgrep-reference.mjs", "--query", query, "--root", root]: ["node", resolve("apps/cli/dist/bin/index.js"), query, root, "--no-cache"],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${address.port}/v4/ai`,- },+ env: transport.env(!reference, `http://127.0.0.1:${address.port}`),stdout: "pipe",stderr: "pipe",timeout: 20000,@@ -51,6 +62,7 @@ async function disconnected(reference: boolean) {]);return { code, stdout, stderr, bodies: bodies.sort() };} finally {+ await transport.cleanup();server.closeAllConnections();await new Promise<void>((done) => server.close(() => done()));}diff --git a/test/reference/discovery-parity.test.ts b/test/reference/discovery-parity.test.tsindex 3e709db..fbca4df 100644--- a/test/reference/discovery-parity.test.ts+++ b/test/reference/discovery-parity.test.ts@@ -1,10 +1,12 @@+import { referenceTransport, wireResponse } from "./transport";+import { routeProviderFetch } from "../fixtures/provider-route.mjs";import { expect } from "bun:test";import { mkdtemp, mkdir, writeFile, rm, chmod } from "node:fs/promises";import { join } from "node:path";import { tmpdir } from "node:os";import { testIfDocker } from "../helpers/docker";import { retrieve } from "../../packages/core/src/retrieve";-import { createEvaluator, EvaluationFailure } from "../../packages/core/src/gateway";+import { createEvaluator, EvaluationFailure } from "../../packages/core/src/evaluator";type Body = {state: {@@ -38,12 +40,11 @@ async function trajectory([];const requests: Body[] = [];let failedGroup = false;+ const transport = await referenceTransport();const server = Bun.serve({port: 0,async fetch(request) {- expect(new URL(request.url).pathname).toBe("/v4/ai/evaluation-model");- expect(request.headers.get("ai-model-id")).toBe("typesafe-ai/jev");- const body = (await request.json()) as Body;+ const body = transport.decode(request, await request.json(), !reference) as Body;if (body.state.items || body.state.preview) requests.push(body);if (body.state.items && fault) {if (fault === "rate-limit")@@ -62,25 +63,30 @@ async function trajectory();}}- const response = Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id, i) => {- const item = body.state.items?.[i];- const probability =- item?.kind === "directory"- ? body.state.relationAnchor &&- item.path.split("/").some((segment) => segment.startsWith("related")) &&- !item.path.includes("-cap") &&- !item.path.includes("-escaped")- ? 0.9- : 0.5- : item?.path.startsWith("Anchor.")- ? 0.9- : 0.25;- return [id, { type: "boolean", probability }];- }),+ const response = Response.json(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id, i) => {+ const item = body.state.items?.[i];+ const probability =+ item?.kind === "directory"+ ? body.state.relationAnchor &&+ item.path.split("/").some((segment) => segment.startsWith("related")) &&+ !item.path.includes("-cap") &&+ !item.path.includes("-escaped")+ ? 0.9+ : 0.5+ : item?.path.startsWith("Anchor.")+ ? 0.9+ : 0.25;+ return [id, { type: "boolean", probability }];+ }),+ ),+ },+ !reference,),- });+ );if (reverseRelationCompletion &&body.state.relationAnchor &&@@ -104,11 +110,7 @@ async function trajectory(const child = Bun.spawn(["node", "/opt/jevgrep-reference.mjs", "--root", root, "--query", query],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,- },+ env: transport.env(!reference, `http://127.0.0.1:${server.port}`),stdout: "pipe",stderr: "pipe",},@@ -126,8 +128,9 @@ async function trajectory(const result = await retrieve({ root, query, signal },createEvaluator({+ provider: "vercel",apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}/v4/ai`,+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal,}),);@@ -136,6 +139,7 @@ async function trajectory(return requests;} finally {server.stop(true);+ await transport.cleanup();}}diff --git a/test/reference/evidence-order.test.ts b/test/reference/evidence-order.test.tsindex ce64fdb..528d6fd 100644--- a/test/reference/evidence-order.test.ts+++ b/test/reference/evidence-order.test.ts@@ -1,3 +1,4 @@+import { referenceTransport, wireResponse } from "./transport";import { expect } from "bun:test";import { mkdtemp, writeFile, rm } from "node:fs/promises";import { join, resolve } from "node:path";@@ -20,10 +21,11 @@ async function run(root: string, production: boolean) {const requests: Record<string, RequestBody> = {};const pending = new Map<string, () => void>();const released: string[] = [];+ const transport = await referenceTransport();const server = Bun.serve({port: 0,async fetch(request) {- const body = (await request.json()) as RequestBody;+ const body = transport.decode(request, await request.json(), production) as RequestBody;const phase = body.state.items? "discovery": body.state.declarations@@ -49,11 +51,16 @@ async function run(root: string, production: boolean) {}});}- return Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ return Response.json(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ ),+ },+ production,),- });+ );},});try {@@ -75,11 +82,7 @@ async function run(root: string, production: boolean) {"trace event recording",],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,- },+ env: transport.env(production, `http://127.0.0.1:${server.port}`),stdout: "pipe",stderr: "pipe",},@@ -95,6 +98,7 @@ async function run(root: string, production: boolean) {return { requests, stdout };} finally {server.stop(true);+ await transport.cleanup();}}diff --git a/test/reference/fallback-comments.test.ts b/test/reference/fallback-comments.test.tsindex 45a1730..2769f28 100644--- a/test/reference/fallback-comments.test.ts+++ b/test/reference/fallback-comments.test.ts@@ -1,3 +1,4 @@+import { referenceTransport, wireResponse } from "./transport";import { expect } from "bun:test";import { mkdtemp, rm, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";@@ -13,29 +14,36 @@ testIfDocker(const query = "Find source behavior";const requests: string[][] = [[], []];let arm = 0;+ const transport = await referenceTransport();const server = Bun.serve({port: 0,async fetch(request) {- const body = (await request.json()) as {+ const body = transport.decode(request, await request.json(), arm !== 0) as {state: { declarations?: Array<{ startLine: number }>; selectedEvidence?: unknown };questions: Record<string, unknown>;};requests[arm]!.push(JSON.stringify(body));- return Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id, index) => [- id,- {- type: "boolean",- probability: body.state.declarations- ? !body.state.selectedEvidence && body.state.declarations[index]?.startLine === 1- ? 0.9- : 0- : 0.9,- },- ]),+ return Response.json(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id, index) => [+ id,+ {+ type: "boolean",+ probability: body.state.declarations+ ? !body.state.selectedEvidence &&+ body.state.declarations[index]?.startLine === 1+ ? 0.9+ : 0+ : 0.9,+ },+ ]),+ ),+ },+ arm !== 0,),- });+ );},});try {@@ -53,11 +61,7 @@ testIfDocker("--no-cache",],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,- },+ env: transport.env(arm !== 0, `http://127.0.0.1:${server.port}`),stdout: "pipe",stderr: "pipe",},@@ -77,6 +81,7 @@ testIfDocker(expect(outputs[1]).toBe(outputs[0]);} finally {server.stop(true);+ await transport.cleanup();await rm(root, { recursive: true, force: true });}},diff --git a/test/reference/provider-parity.test.ts b/test/reference/provider-parity.test.tsnew file mode 100644index 0000000..cef281e--- /dev/null+++ b/test/reference/provider-parity.test.ts@@ -0,0 +1,32 @@+import { expect } from "bun:test";+import { readFile } from "node:fs/promises";+import { testIfDocker } from "../helpers/docker";+import { replay } from "./replay";+import { providers, type Provider } from "./transport";++for (const provider of Object.keys(providers) as Provider[]) {+ testIfDocker(+ `${provider} production preserves the frozen request multiset and complete packet`,+ async () => {+ const expected = JSON.parse(await readFile("test/reference/corpus.json", "utf8"));+ const result = await replay("healthy", true, provider);+ expect(result.code).toBe(0);+ expect(result.stderr).toBe("");+ expect(result.requests).toEqual(expected.requests);+ expect(result.stdout).toBe(expected.stdout);+ },+ 120_000,+ );+ for (const mode of ["missing", "invalid"] as const) {+ testIfDocker(+ `${provider} ${mode} answers preserve incomplete discovery`,+ async () => {+ const result = await replay(mode, true, provider);+ expect(result.code).toBe(2);+ expect(result.stderr).toBe("");+ expect(result.stdout).toContain("Jevgrep: 0 relevant files; discovery incomplete.");+ },+ 120_000,+ );+ }+}diff --git a/test/reference/python-parity.test.ts b/test/reference/python-parity.test.tsindex 0231514..5e6437f 100644--- a/test/reference/python-parity.test.ts+++ b/test/reference/python-parity.test.ts@@ -1,3 +1,4 @@+import { referenceTransport, wireResponse } from "./transport";import { expect } from "bun:test";import { mkdtemp, rm, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";@@ -40,29 +41,35 @@ for (const [name, source] of [const query = "Find target behavior";const requests: string[][] = [[], []];let arm = 0;+ const transport = await referenceTransport();const server = Bun.serve({port: 0,async fetch(request) {- const body = (await request.json()) as {+ const body = transport.decode(request, await request.json(), arm !== 0) as {questions: Record<string, unknown>;state?: { declarations?: { name: string }[] };};requests[arm]!.push(JSON.stringify(body));- return Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id, index) => [- id,- {- type: "boolean",- probability:- name === "CR-only method selected" &&- body.state?.declarations?.[index]?.name.endsWith(".context")- ? 0.1- : 0.9,- },- ]),+ return Response.json(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id, index) => [+ id,+ {+ type: "boolean",+ probability:+ name === "CR-only method selected" &&+ body.state?.declarations?.[index]?.name.endsWith(".context")+ ? 0.1+ : 0.9,+ },+ ]),+ ),+ },+ arm !== 0,),- });+ );},});try {@@ -80,11 +87,7 @@ for (const [name, source] of ["--no-cache",],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,- },+ env: transport.env(arm !== 0, `http://127.0.0.1:${server.port}`),stdout: "pipe",stderr: "pipe",},@@ -102,6 +105,7 @@ for (const [name, source] of [expect(outputs[1]).toBe(outputs[0]);} finally {server.stop(true);+ await transport.cleanup();await rm(root, { recursive: true, force: true });}},diff --git a/test/reference/replay.ts b/test/reference/replay.tsindex f3c2d3c..05b6a68 100644--- a/test/reference/replay.ts+++ b/test/reference/replay.ts@@ -1,17 +1,16 @@-import { expect } from "bun:test";+import { referenceTransport, wireResponse, type Provider } from "./transport";import { resolve } from "node:path";export async function replay(mode: "healthy" | "missing" | "invalid" = "healthy",production = false,+ provider: Provider = "vercel",) {const requests: unknown[] = [];+ const transport = await referenceTransport(provider, "reference-fixture");const server = Bun.serve({port: 0,async fetch(request) {- expect(new URL(request.url).pathname).toBe("/v4/ai/evaluation-model");- expect(request.headers.get("ai-model-id")).toBe("typesafe-ai/jev");- expect(request.headers.get("authorization")).toBe("Bearer reference-fixture");- const body = (await request.json()) as {+ const body = transport.decode(request, await request.json(), production) as {state: {items?: Array<{ path: string }>;path?: string;@@ -22,34 +21,44 @@ export async function replay(questions: Record<string, unknown>;};requests.push(body);- if (mode === "missing") return Response.json({ answers: {} });+ if (mode === "missing") return Response.json(wireResponse({ answers: {} }, production));if (mode === "invalid")- return Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 2 }]),- ),- });- return Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id, i) => [- id,+ return Response.json(+ wireResponse({- type: "boolean",- probability:- body.state.items?.[i]?.path === "unrelated.md"- ? 0.05- : body.state.items?.[i]?.path === "src/backend" && !body.state.relationAnchor- ? 0.1- : body.state.declarations &&- !body.state.selectedEvidence &&- body.state.path !== "src/telemetry.ts"- ? 0.4- : 0.9,+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 2 }]),+ ),},- ]),+ production,+ ),+ );+ return Response.json(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id, i) => [+ id,+ {+ type: "boolean",+ probability:+ body.state.items?.[i]?.path === "unrelated.md"+ ? 0.05+ : body.state.items?.[i]?.path === "src/backend" && !body.state.relationAnchor+ ? 0.1+ : body.state.declarations &&+ !body.state.selectedEvidence &&+ body.state.path !== "src/telemetry.ts"+ ? 0.4+ : 0.9,+ },+ ]),+ ),+ warnings: [{ type: "other", message: "fixture warning" }],+ },+ production,),- warnings: [{ type: "other", message: "fixture warning" }],- });+ );},});try {@@ -71,11 +80,7 @@ export async function replay("research how telemetry records event names",],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "reference-fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,- },+ env: transport.env(production, `http://127.0.0.1:${server.port}`),stdout: "pipe",stderr: "pipe",},@@ -93,5 +98,6 @@ export async function replay(};} finally {server.stop(true);+ await transport.cleanup();}}diff --git a/test/reference/repository-context-parity.test.ts b/test/reference/repository-context-parity.test.tsindex 354682b..db4d331 100644--- a/test/reference/repository-context-parity.test.ts+++ b/test/reference/repository-context-parity.test.ts@@ -1,3 +1,4 @@+import { referenceTransport, wireResponse } from "./transport";import { expect } from "bun:test";import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";import { join, resolve } from "node:path";@@ -11,11 +12,11 @@ async function run(largeAnchor = false,) {const requests: unknown[] = [];+ const transport = await referenceTransport();const server = Bun.serve({port: 0,async fetch(request) {- expect(new URL(request.url).pathname).toBe("/v4/ai/evaluation-model");- const body = (await request.json()) as {+ const body = transport.decode(request, await request.json(), production) as {state: {items?: Array<{ path: string; kind: string }>;relationAnchor?: unknown;@@ -25,35 +26,40 @@ async function run(questions: Record<string, unknown>;};requests.push(body);- return Response.json({- answers: Object.fromEntries(- Object.keys(body.questions).map((id, i) => [- id,- {- type: "boolean",- probability: body.state.items- ? largeAnchor- ? body.state.items[i]?.kind === "directory"- ? body.state.relationAnchor- ? 0.9- : 0.1- : body.state.items[i]?.path === "Anchor.ts"- ? 0.9- : 0.1- : 0.9- : body.state.declarations- ? !body.state.selectedEvidence &&- body.state.declarations[i]?.name === name &&- body.state.declarations[i]?.startLine === startLine- ? 0.9- : 0.1- : id === "test"- ? 0.9- : 0.1,- },- ]),+ return Response.json(+ wireResponse(+ {+ answers: Object.fromEntries(+ Object.keys(body.questions).map((id, i) => [+ id,+ {+ type: "boolean",+ probability: body.state.items+ ? largeAnchor+ ? body.state.items[i]?.kind === "directory"+ ? body.state.relationAnchor+ ? 0.9+ : 0.1+ : body.state.items[i]?.path === "Anchor.ts"+ ? 0.9+ : 0.1+ : 0.9+ : body.state.declarations+ ? !body.state.selectedEvidence &&+ body.state.declarations[i]?.name === name &&+ body.state.declarations[i]?.startLine === startLine+ ? 0.9+ : 0.1+ : id === "test"+ ? 0.9+ : 0.1,+ },+ ]),+ ),+ },+ production,),- });+ );},});try {@@ -62,11 +68,7 @@ async function run(? ["node", resolve("apps/cli/dist/bin/index.js"), "large regression", root, "--no-cache"]: ["node", "/opt/jevgrep-reference.mjs", "--root", root, "--query", "large regression"],{- env: {- PATH: process.env.PATH,- AI_GATEWAY_API_KEY: "fixture",- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,- },+ env: transport.env(production, `http://127.0.0.1:${server.port}`),stdout: "pipe",stderr: "pipe",},@@ -82,6 +84,7 @@ async function run(return { requests: requests.map((body) => JSON.stringify(body)).sort(), stdout };} finally {server.stop(true);+ await transport.cleanup();}}testIfDocker(diff --git a/test/reference/selection-replay.ts b/test/reference/selection-replay.tsindex ce678b7..abe282c 100644--- a/test/reference/selection-replay.ts+++ b/test/reference/selection-replay.ts@@ -7,7 +7,7 @@ import {contextWindows,} from "../../evals/implementation/swebench/source-method-windows-spike";import { completeSourceFragments } from "../../evals/implementation/swebench/complete-source-v64-spike";-import type { EvaluationRequest } from "../../packages/core/src/gateway";+import type { EvaluationRequest } from "../../packages/core/src/evaluator";import type { Declaration } from "../../packages/core/src/requests";import type { Range } from "../../packages/core/src/source";diff --git a/test/reference/selection.test.ts b/test/reference/selection.test.tsindex 0190c77..bb09ec0 100644--- a/test/reference/selection.test.ts+++ b/test/reference/selection.test.ts@@ -1,6 +1,6 @@import { expect, test } from "bun:test";import { selectFile } from "../../packages/core/src/selection";-import type { EvaluationRequest } from "../../packages/core/src/gateway";+import type { EvaluationRequest } from "../../packages/core/src/evaluator";import type { Declaration } from "../../packages/core/src/requests";import { replaySelection } from "./selection-replay";diff --git a/test/reference/transport.test.ts b/test/reference/transport.test.tsnew file mode 100644index 0000000..d0ea332--- /dev/null+++ b/test/reference/transport.test.ts@@ -0,0 +1,54 @@+import { expect, test } from "bun:test";+import { readFile } from "node:fs/promises";+import { referenceTransport, providers, type Provider } from "./transport";++test("wire normalization retains every corpus field and rejects unknown envelope and question fields", async () => {+ const corpus = JSON.parse(await readFile("test/reference/corpus.json", "utf8"));+ for (const provider of Object.keys(providers) as Provider[]) {+ const transport = await referenceTransport(provider);+ try {+ const preset = providers[provider];+ const request = new Request(preset.url, {+ method: "POST",+ headers: {+ authorization: "Bearer fixture",+ "x-jevgrep-original-url": preset.url,+ },+ });+ for (const serialized of corpus.requests) {+ const original = JSON.parse(serialized);+ const historical = new Request("http://localhost/v4/ai/evaluation-model", {+ method: "POST",+ headers: { authorization: "Bearer fixture", "ai-model-id": "typesafe-ai/jev" },+ });+ expect(JSON.stringify(transport.decode(historical, original, false))).toBe(serialized);+ const wire = {+ model: preset.model,+ state: original.state,+ questions: Object.fromEntries(+ Object.entries(original.questions).map(([id, question]) => [+ id,+ { ...(question as object), type: "noul" },+ ]),+ ),+ };+ expect(JSON.stringify(transport.decode(request, wire, true))).toBe(serialized);+ expect(() => transport.decode(request, { ...wire, extra: true }, true)).toThrow();+ const id = Object.keys(wire.questions)[0]!;+ expect(() =>+ transport.decode(+ request,+ {+ ...wire,+ questions: { ...wire.questions, [id]: { ...wire.questions[id], extra: true } },+ },+ true,+ ),+ ).toThrow();+ expect(() => transport.decode(request, { ...wire, model: "wrong-model" }, true)).toThrow();+ }+ } finally {+ await transport.cleanup();+ }+ }+});diff --git a/test/reference/transport.ts b/test/reference/transport.tsnew file mode 100644index 0000000..96a58b4--- /dev/null+++ b/test/reference/transport.ts@@ -0,0 +1,93 @@+import { expect } from "bun:test";+import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join, resolve } from "node:path";++// Independent expectations: deriving these from production would hide preset drift.+export const providers = {+ vercel: { url: "https://ai-gateway.vercel.sh/typesafe/v1/systemone", model: "typesafe-ai/jev" },+ typesafe: { url: "https://api.typesafe.ai/v1/systemone", model: "jev-1.13.0" },+ openrouter: { url: "https://openrouter.ai/api/v1/systemone", model: "jev-1.13" },+} as const;+export type Provider = keyof typeof providers;++export async function referenceTransport(provider: Provider = "vercel", key = "fixture") {+ const home = await mkdtemp(join(tmpdir(), "jg-reference-auth-"));+ await mkdir(join(home, "jevgrep"));+ await writeFile(+ join(home, "jevgrep/credentials.json"),+ JSON.stringify({ provider, apiKey: key }),+ );+ return {+ env(production: boolean, origin: string) {+ return production+ ? {+ PATH: process.env.PATH,+ XDG_CONFIG_HOME: home,+ NODE_OPTIONS: `--import=${resolve("test/fixtures/provider-route.mjs")}`,+ JEVGREP_TEST_PROVIDER_ORIGIN: origin,+ }+ : {+ PATH: process.env.PATH,+ AI_GATEWAY_API_KEY: key,+ AI_GATEWAY_BASE_URL: `${origin}/v4/ai`,+ };+ },+ decode(+ request: { url: string; method?: string; headers: Headers },+ raw: unknown,+ production: boolean,+ ) {+ expect(request.method).toBe("POST");+ expect(request.headers.get("authorization")).toBe(`Bearer ${key}`);+ const body = raw as {+ model?: string;+ providerOptions?: Record<string, unknown>;+ state: unknown;+ questions: Record<string, { type: string; instructions: string }>;+ };+ expect(Object.keys(body).sort()).toEqual(+ production ? ["model", "questions", "state"] : ["providerOptions", "questions", "state"],+ );+ expect(new URL(request.url).pathname).toBe(+ production ? new URL(providers[provider].url).pathname : "/v4/ai/evaluation-model",+ );+ if (production) {+ expect(request.headers.get("x-jevgrep-original-url")).toBe(providers[provider].url);+ expect(body.model).toBe(providers[provider].model);+ } else {+ expect(request.headers.get("ai-model-id")).toBe("typesafe-ai/jev");+ expect(body.providerOptions).toEqual({});+ }+ expect(body.state).toBeDefined();+ expect(body.questions).toBeObject();+ const questions = Object.fromEntries(+ Object.entries(body.questions).map(([id, question]) => {+ expect(Object.keys(question).sort()).toEqual(["instructions", "type"]);+ expect(question.type).toBe(production ? "noul" : "boolean");+ expect(typeof question.instructions).toBe("string");+ return [id, { ...question, type: "boolean" }];+ }),+ );+ // Only the validated transport envelope/types differ; state and question order stay exact.+ return { state: body.state, questions, providerOptions: {} };+ },+ cleanup: () => rm(home, { recursive: true, force: true }),+ };+}++export function wireResponse<+ T extends { answers: Record<string, { type: string; probability: number }> },+>(body: T, production: boolean) {+ return production+ ? {+ ...body,+ answers: Object.fromEntries(+ Object.entries(body.answers).map(([id, answer]) => [+ id,+ { type: "noul", noul: answer.probability },+ ]),+ ),+ }+ : body;+}diff --git a/test/retrieval-freshness.test.ts b/test/retrieval-freshness.test.tsindex aa3a626..f6007b0 100644--- a/test/retrieval-freshness.test.ts+++ b/test/retrieval-freshness.test.ts@@ -1,10 +1,11 @@+import { routeProviderFetch } from "./fixtures/provider-route.mjs";import { expect } from "bun:test";import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";import { join } from "node:path";import { tmpdir } from "node:os";import { testIfDocker } from "./helpers/docker";import { retrieve } from "../packages/core/src/retrieve";-import { createEvaluator } from "../packages/core/src/gateway";+import { createEvaluator } from "../packages/core/src/evaluator";testIfDocker("queued navigation never uploads source excluded after the first wave",@@ -29,7 +30,7 @@ testIfDocker(}return Response.json({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.1 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.1 }]),),});},@@ -39,7 +40,12 @@ testIfDocker(const signal = new AbortController().signal;const result = await retrieve({ root, query: "sentinel", signal },- createEvaluator({ apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}`, signal }),+ createEvaluator({+ apiKey: "fixture",+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),+ signal,+ }),);expect(uploads).toBe(8);expect(result.status).toBe("incomplete");@@ -71,7 +77,7 @@ for (const mutation of ["ignored", "changed"] as const)}return Response.json({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),),});},@@ -86,7 +92,8 @@ for (const mutation of ["ignored", "changed"] as const){ root, query: "selected", signal },createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal,}),);@@ -124,7 +131,12 @@ testIfDocker(const signal = new AbortController().signal;const result = await retrieve({ root, query: "retry", signal },- createEvaluator({ apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}`, signal }),+ createEvaluator({+ apiKey: "fixture",+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),+ signal,+ }),);expect(uploads).toBe(1);expect(result.status).toBe("incomplete");@@ -155,7 +167,7 @@ for (const phase of ["selection", "role"] as const)}return Response.json({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),),});},@@ -170,7 +182,8 @@ for (const phase of ["selection", "role"] as const){ root, query: "selected", signal },createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal,}),);@@ -202,7 +215,7 @@ for (const size of [2, 4])}return Response.json({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.1 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.1 }]),),});},@@ -216,7 +229,8 @@ for (const size of [2, 4]){ root, query: "selected", signal },createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal,}),);@@ -258,8 +272,8 @@ for (const donor of ["sample", "anchor"] as const)Object.keys(body.questions).map((id, index) => [id,{- type: "boolean",- probability: body.state.items?.[index]?.kind === "file" ? 0.9 : 0.1,+ type: "noul",+ noul: body.state.items?.[index]?.kind === "file" ? 0.9 : 0.1,},]),),@@ -281,7 +295,8 @@ for (const donor of ["sample", "anchor"] as const){ root, query: "selected", signal },createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal,}),);@@ -314,7 +329,7 @@ testIfDocker(}return Response.json({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),),});},@@ -325,7 +340,12 @@ testIfDocker(const signal = new AbortController().signal;const result = await retrieve({ root, query: "selected", signal },- createEvaluator({ apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}`, signal }),+ createEvaluator({+ apiKey: "fixture",+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),+ signal,+ }),);expect(attempts).toBe(1);expect(result.status).toBe("incomplete");diff --git a/test/retrieval.test.ts b/test/retrieval.test.tsindex 91fa7f0..535ffbe 100644--- a/test/retrieval.test.ts+++ b/test/retrieval.test.ts@@ -1,10 +1,11 @@+import { routeProviderFetch } from "./fixtures/provider-route.mjs";import { expect } from "bun:test";import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";import { testIfDocker } from "./helpers/docker";import { retrieve } from "../packages/core/src/retrieve";-import { createEvaluator } from "../packages/core/src/gateway";+import { createEvaluator } from "../packages/core/src/evaluator";testIfDocker("hierarchical retrieval returns source from multiple files without uploading excluded data",@@ -21,7 +22,7 @@ testIfDocker(sent.push(JSON.stringify(body));return Response.json({answers: Object.fromEntries(- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),),});},@@ -43,7 +44,8 @@ testIfDocker(const signal = new AbortController().signal;const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal,});const result = await retrieve({ root, query: "research event recording", signal }, evaluator);@@ -211,13 +213,14 @@ testIfDocker("an aborted evaluator cannot return a previously cached answer", asport: 0,fetch() {calls++;- return Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });+ return Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });},});try {const evaluator = createEvaluator({apiKey: "fixture",- baseURL: `http://127.0.0.1:${server.port}`,+ provider: "vercel",+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),signal: controller.signal,cache: createEvaluationCache({ directory }),});diff --git a/test/runtime.Dockerfile b/test/runtime.Dockerfileindex d60892f..e0feb55 100644--- a/test/runtime.Dockerfile+++ b/test/runtime.Dockerfile@@ -36,6 +36,7 @@ FROM node:22-bookworm-slim AS runtimeCOPY --from=install /opt/jevgrep /opt/jevgrepCOPY --from=package /artifacts/canonical-skill.md /test/canonical-skill.mdCOPY test/installed.test.mjs /test/installed.test.mjs+COPY test/fixtures/provider-route.mjs /test/fixtures/provider-route.mjsCOPY --chmod=755 test/fixtures/skill-installer.mjs /test/fixtures/skill-installer.mjsENV PATH=/opt/jevgrep/bin:/usr/local/bin:/usr/bin:/bin \JEVGREP_INSTALLED_BINARY=/opt/jevgrep/bin/jg \