Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

feat: support saved TypeSafe and OpenRouter providers

David Zhang9496540ad5b5
diff --git a/.env.example b/.env.example
deleted file mode 100644
index a59c8af..0000000
--- a/.env.example
+++ /dev/null
@@ -1 +0,0 @@
-AI_GATEWAY_API_KEY=
diff --git a/README.md b/README.md
index 4b9d6c4..d502e4f 100644
--- a/README.md
+++ b/README.md
@@ -22,9 +22,12 @@ jg skill
jg "How are telemetry events recorded and sent?" ./my-project
```
-Requires **Node.js 22+**, **macOS or Linux**, and a **Vercel AI Gateway key**.
+Requires **Node.js 22+**, **macOS or Linux**, and a key for **Vercel AI Gateway, TypeSafe, or OpenRouter**.
No separate Python, Bun, or ripgrep installation is required to use `jg`.
+**Provider selection is on main and awaits the next npm release.** The published
+0.2.0 package uses Vercel AI Gateway; build from source to try TypeSafe/OpenRouter.
+
## Install the agent skill — required for agent setup
Installing the CLI alone does not teach your coding agent to use it. **Install
@@ -41,7 +44,7 @@ others) and asks where to install. Add `--global` for a user-wide install, or
returned context, and when to fill gaps with its normal tools. It skips redundant
retrieval when the needed context is already known. The current repository skill
checks for `jg` and installs the CLI if it is missing; authentication still needs
-your Gateway key. The skill installer itself does not configure credentials.
+your selected provider’s key. The skill installer itself does not configure credentials.
`jg skill` delegates to the [skills CLI](https://github.com/vercel-labs/skills)
and needs npm/npx plus network access. You can also run that installer directly,
@@ -112,13 +115,16 @@ include exact costs, failed tasks, and separately observed Jev charges. See the
## Source, credentials, and local state
-Searches send eligible source content to Jev through Vercel AI Gateway. Default
+Searches send eligible source content to Jev through the provider selected during auth. Default
filesystem filtering respects ignore files and excludes hidden, dependency/build,
binary, and obvious credential files. These filters are not a guarantee that all
sensitive information has been removed; choose a search root you intend to send.
-`jg auth` saves your key in an owner-only config file. `AI_GATEWAY_API_KEY` takes
-precedence when set; `jg doctor` checks the connection with synthetic input.
+`jg auth` asks for your provider, then saves its key in an owner-only config file.
+Re-running auth replaces that setup; searches always use the saved provider.
+`jg doctor` checks it with synthetic input. Existing saved keys without a provider
+remain Vercel keys. Environment-based credentials and endpoint overrides are not
+used; run `jg auth` if you previously relied on them.
Evaluation answers are cached locally by default. The CLI writes its output to
stdout and does not create report files. Use `jg --help` for cache controls,
search overrides, and incomplete-result behavior.
diff --git a/apps/cli/README.md b/apps/cli/README.md
index 6db4dcd..ae5d0f6 100644
--- a/apps/cli/README.md
+++ b/apps/cli/README.md
@@ -13,10 +13,22 @@ jg doctor
jg "How are telemetry events recorded and sent?" ./my-project
```
-`auth` saves your Vercel AI Gateway key; `doctor` verifies it with synthetic input.
-You can instead supply `AI_GATEWAY_API_KEY`. Searches send eligible source to Jev
-through AI Gateway. Credentials use an owner-only config file. Evaluation answers
-are cached locally; `jg --help` describes overrides and cache commands.
+`auth` asks for Vercel AI Gateway, TypeSafe, or OpenRouter, then saves its key in
+an owner-only config file. Searches use that provider until you run auth again.
+`doctor` verifies access with synthetic input and names the selected provider.
+For unattended setup, pipe the key from your secret manager to:
+
+```sh
+jg auth --provider openrouter --stdin
+```
+
+Both options are required for piped setup. Credentials are saved under
+`$XDG_CONFIG_HOME/jevgrep/credentials.json`, or `~/.config/jevgrep/credentials.json`.
+Existing saved records without a provider still mean Vercel, without a migration.
+API-key, endpoint, and model environment overrides are ignored; users who only
+configured an environment key must run auth. There is no automatic fallback or
+per-search provider override. Searches send eligible source to the saved service.
+Evaluation answers are cached locally; `jg --help` describes cache controls.
The summary comes first, followed by file and declaration locations and selected
source. Locations are reading leads, not a checklist. Omitted excerpts are marked;
diff --git a/apps/cli/src/args.ts b/apps/cli/src/args.ts
index 1debff3..f335288 100644
--- a/apps/cli/src/args.ts
+++ b/apps/cli/src/args.ts
@@ -1,12 +1,13 @@
import { parseArgs } from "node:util";
import type { SearchInput } from "@repo/core";
+import { isProviderId, type ProviderId } from "@repo/core/providers";
import { CliError } from "./errors";
import { DEFAULT_MAX_SOURCE_BYTES } from "./render";
export type Command =
| { kind: "help" | "version" | "doctor" | "cache-clear" }
| { kind: "skill"; agents: string[]; global: boolean; yes: boolean }
- | { kind: "auth"; fromStdin: boolean }
+ | { kind: "auth"; provider?: ProviderId }
| {
kind: "search";
query: string;
@@ -27,6 +28,7 @@ export function parseCommand(args: string[]): Command {
help: { type: "boolean", short: "h" },
version: { type: "boolean" },
stdin: { type: "boolean" },
+ provider: { type: "string" },
agent: { type: "string", multiple: true },
global: { type: "boolean" },
yes: { type: "boolean" },
@@ -49,9 +51,12 @@ export function parseCommand(args: string[]): Command {
if (values.help || values.version) throw new CliError("Use --help or --version alone.");
const first = positionals[0];
if (first === "auth") {
- if (positionals.length !== 1 || keys.some((key) => key !== "stdin"))
- throw new CliError("Usage: jg auth [--stdin]");
- return { kind: "auth", fromStdin: values.stdin ?? false };
+ if (positionals.length !== 1 || keys.some((key) => !["stdin", "provider"].includes(key)))
+ throw new CliError("Usage: jg auth OR jg auth --provider NAME --stdin");
+ if (!keys.length) return { kind: "auth" };
+ if (!values.stdin || !isProviderId(values.provider))
+ throw new CliError("Use auth --provider vercel|typesafe|openrouter --stdin for a piped key.");
+ return { kind: "auth", provider: values.provider };
}
if (first === "skill") {
const agents = values.agent ?? [];
@@ -77,7 +82,7 @@ export function parseCommand(args: string[]): Command {
!first?.trim() ||
positionals.length > 2 ||
values.stdin ||
- keys.some((key) => ["agent", "global", "yes"].includes(key))
+ keys.some((key) => ["agent", "global", "yes", "provider"].includes(key))
)
throw new CliError('Usage: jg "question" [root]. Run jg --help.');
const rawBudget = values["max-source-bytes"];
@@ -109,12 +114,17 @@ Usage: jg "question" [root]
Root defaults to the current directory; use -- before a root beginning with -.
Commands:
- auth [--stdin] Save a Gateway key (hidden prompt, or an explicit pipe)
+ auth Choose a provider, then save its key (hidden prompt)
doctor Verify Jev access using a synthetic question
skill Install the agent skill via npx skills
--help, -h Show usage
--version Show the installed version
+Auth automation:
+ auth --provider vercel|typesafe|openrouter --stdin
+ Save one provider/key from a pipe. Re-running auth replaces your setup.
+ Saved credentials only; provider key/URL environment variables are ignored.
+
Skill installation options:
--agent NAME Target an agent (repeat for multiple agents)
--global Install for the current user instead of this project
diff --git a/apps/cli/src/auth.ts b/apps/cli/src/auth.ts
index 0d7c2f4..a7d4aa4 100644
--- a/apps/cli/src/auth.ts
+++ b/apps/cli/src/auth.ts
@@ -2,7 +2,8 @@ import { chmod, mkdir, readFile, rename, rm, writeFile } from "node:fs/promises"
import { homedir } from "node:os";
import { join } from "node:path";
import { randomUUID } from "node:crypto";
-import { isCancel, password } from "@clack/prompts";
+import { isCancel, password, select } from "@clack/prompts";
+import { providers, isProviderId, type ProviderId } from "@repo/core/providers";
import { CliError } from "./errors";
export function configDirectory() {
@@ -17,9 +18,11 @@ function validateKey(raw: string): string {
return key;
}
-export async function authenticate(fromStdin: boolean, signal: AbortSignal) {
+export type Credentials = { provider: ProviderId; apiKey: string };
+
+export async function authenticate(provider: ProviderId | undefined, signal: AbortSignal) {
let key: string;
- if (fromStdin) {
+ if (provider !== undefined) {
const chunks: Buffer[] = [];
let bytes = 0;
const abort = () => process.stdin.destroy(new DOMException("Interrupted", "AbortError"));
@@ -36,9 +39,23 @@ export async function authenticate(fromStdin: boolean, signal: AbortSignal) {
signal.removeEventListener("abort", abort);
}
} else {
- if (!process.stdin.isTTY) throw new CliError("Use auth --stdin to read a piped key.");
+ if (!process.stdin.isTTY)
+ throw new CliError(
+ "Use auth --provider vercel|typesafe|openrouter --stdin to read a piped key.",
+ );
+ const selected = await select<ProviderId>({
+ message: "Choose your Jev provider",
+ options: (Object.keys(providers) as ProviderId[]).map((value) => ({
+ value,
+ label: providers[value].label,
+ })),
+ output: process.stdout,
+ signal,
+ });
+ if (isCancel(selected)) throw new DOMException("Interrupted", "AbortError");
+ provider = selected;
const answer = await password({
- message: "Paste your Vercel AI Gateway API key",
+ message: `Paste your ${providers[provider].label} API key`,
output: process.stdout,
signal,
});
@@ -53,21 +70,19 @@ export async function authenticate(fromStdin: boolean, signal: AbortSignal) {
await chmod(directory, 0o700);
const temporary = join(directory, `.credentials-${randomUUID()}.json`);
try {
- await writeFile(temporary, JSON.stringify({ apiKey: key }) + "\n", { mode: 0o600, flag: "wx" });
+ await writeFile(temporary, JSON.stringify({ provider, apiKey: key }) + "\n", {
+ mode: 0o600,
+ flag: "wx",
+ });
+ signal.throwIfAborted();
await rename(temporary, join(directory, "credentials.json"));
} finally {
await rm(temporary, { force: true });
}
- process.stdout.write("AI Gateway key saved. Run jg doctor to verify access.\n");
+ process.stdout.write(`${providers[provider].label} key saved. Run jg doctor to verify access.\n`);
}
-export async function loadApiKey(): Promise<string> {
- const fromEnvironment = process.env.AI_GATEWAY_API_KEY;
- if (fromEnvironment !== undefined) {
- if (!fromEnvironment.trim())
- throw new CliError("AI_GATEWAY_API_KEY is empty; saved credentials are disabled.");
- return validateKey(fromEnvironment);
- }
+export async function loadCredentials(): Promise<Credentials> {
try {
const credentials = JSON.parse(
await readFile(join(configDirectory(), "credentials.json"), "utf8"),
@@ -75,10 +90,12 @@ export async function loadApiKey(): Promise<string> {
if (typeof credentials.apiKey !== "string" || !credentials.apiKey.trim()) {
throw new CliError("Invalid credentials. Run jg auth again.");
}
- return validateKey(credentials.apiKey);
+ const provider = Object.hasOwn(credentials, "provider") ? credentials.provider : "vercel";
+ if (!isProviderId(provider)) throw new CliError("Invalid provider. Run jg auth again.");
+ return { provider, apiKey: validateKey(credentials.apiKey) };
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
- throw new CliError("Run jg auth or set AI_GATEWAY_API_KEY.");
+ throw new CliError("Run jg auth or use jg auth --provider NAME --stdin.");
}
throw new CliError("Could not read valid credentials. Run jg auth again.");
}
diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.ts
index 9657c90..0393439 100644
--- a/apps/cli/src/index.ts
+++ b/apps/cli/src/index.ts
@@ -1,5 +1,6 @@
#!/usr/bin/env node
-import { authenticate, configDirectory, loadApiKey } from "./auth";
+import { authenticate, configDirectory, loadCredentials } from "./auth";
+import { providers } from "@repo/core/providers";
import { help, parseCommand } from "./args";
import { CliError } from "./errors";
import { renderResult } from "./render";
@@ -46,7 +47,7 @@ async function main() {
process.exitCode = await installSkill(command, controller.signal);
return;
case "auth":
- return authenticate(command.fromStdin, controller.signal);
+ return authenticate(command.provider, controller.signal);
case "cache-clear": {
const { createEvaluationCache } = await import("@repo/core");
const cache = createEvaluationCache({ directory: cacheDirectory() });
@@ -56,12 +57,11 @@ async function main() {
return write("Cache cleared.\n");
}
case "doctor": {
- const apiKey = await loadApiKey();
+ const credentials = await loadCredentials();
const { createEvaluator } = await import("@repo/core");
const evaluator = createEvaluator({
- apiKey,
+ ...credentials,
signal: controller.signal,
- baseURL: process.env.AI_GATEWAY_BASE_URL,
});
try {
const answers = await evaluator.evaluate({
@@ -74,18 +74,20 @@ async function main() {
},
});
if (!(answers.relevant! > 0.5))
- throw new CliError("Jev returned an unexpected answer to the connection check.");
- await write("Jev connection verified.\n");
+ throw new CliError(
+ `${providers[credentials.provider].label} returned an unexpected answer to the connection check.`,
+ );
+ await write(`Jev connection verified through ${providers[credentials.provider].label}.\n`);
} catch (error) {
if (error instanceof CliError) throw error;
throw new CliError(
- "Jev connection check failed. Check your Gateway key, model access, and network.",
+ `Jev connection check failed through ${providers[credentials.provider].label}. Check your saved key, model access, and network.`,
);
}
return;
}
case "search": {
- const apiKey = await loadApiKey();
+ const credentials = await loadCredentials();
const { retrieve, createEvaluator, createEvaluationCache } = await import("@repo/core");
const cache = createEvaluationCache({
directory: cacheDirectory(),
@@ -94,9 +96,8 @@ async function main() {
const evaluator = createEvaluator({
cache,
policyVersion: JSON.stringify(command.policy),
- apiKey,
+ ...credentials,
signal: controller.signal,
- baseURL: process.env.AI_GATEWAY_BASE_URL,
});
const result = await retrieve(
{
diff --git a/apps/cli/test/args.test.ts b/apps/cli/test/args.test.ts
index 455f377..389a470 100644
--- a/apps/cli/test/args.test.ts
+++ b/apps/cli/test/args.test.ts
@@ -48,3 +48,21 @@ test("source allocation is independent of the filesystem policy used for retriev
expect(unlimited.policy).toEqual({ hidden: true, noIgnore: true });
expect(bounded.policy).toEqual(unlimited.policy);
});
+
+test("auth requires explicit provider for stdin and keeps provider selection out of search", () => {
+ expect(parseCommand(["auth"])).toEqual({ kind: "auth" });
+ expect(parseCommand(["auth", "--provider", "openrouter", "--stdin"])).toEqual({
+ kind: "auth",
+ provider: "openrouter",
+ });
+ for (const args of [
+ ["auth", "--stdin"],
+ ["auth", "--provider", "vercel"],
+ ["auth", "--provider", "unknown", "--stdin"],
+ ["query", "--provider", "typesafe"],
+ ["doctor", "--provider", "vercel"],
+ ["skill", "--provider", "vercel"],
+ ["cache", "clear", "--provider", "vercel"],
+ ])
+ expect(() => parseCommand(args)).toThrow();
+});
diff --git a/apps/cli/test/auth.test.ts b/apps/cli/test/auth.test.ts
index 6bb709e..3419e30 100644
--- a/apps/cli/test/auth.test.ts
+++ b/apps/cli/test/auth.test.ts
@@ -5,27 +5,29 @@ import { testInDocker, withCli } from "./cli";
testInDocker("installed auth saves privately, bounds stdin, and never echoes keys", async () => {
await withCli(async ({ home, run }) => {
- const saved = await run(["auth", "--stdin"], "test-gateway-secret\n");
+ const saved = await run(["auth", "--provider", "vercel", "--stdin"], "test-gateway-secret\n");
expect(saved.code).toBe(0);
expect(saved.stderr).toBe("");
expect(saved.stdout).toContain("key saved");
expect(saved.stdout).not.toContain("test-gateway-secret");
const file = join(home, "jevgrep", "credentials.json");
- expect(JSON.parse(await readFile(file, "utf8")).apiKey).toBe("test-gateway-secret");
+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({
+ provider: "vercel",
+ apiKey: "test-gateway-secret",
+ });
expect((await stat(file)).mode & 0o777).toBe(0o600);
expect((await stat(join(home, "jevgrep"))).mode & 0o777).toBe(0o700);
- const disabled = await run(["doctor"]);
- expect(disabled.code).toBe(1);
- expect(disabled.stderr).toBe("");
- expect(disabled.stdout).toContain("saved credentials are disabled");
- const whitespace = await run(["auth", "--stdin"], "two secrets\n");
+ const whitespace = await run(["auth", "--provider", "vercel", "--stdin"], "two secrets\n");
expect(whitespace.code).toBe(1);
expect(whitespace.stderr).toBe("");
expect(whitespace.stdout).not.toContain("two secrets");
- const huge = await run(["auth", "--stdin"], "x".repeat(8193));
+ const huge = await run(["auth", "--provider", "vercel", "--stdin"], "x".repeat(8193));
expect(huge.code).toBe(1);
expect(huge.stderr).toBe("");
expect(huge.stdout).toContain("exceeds");
- expect(JSON.parse(await readFile(file, "utf8")).apiKey).toBe("test-gateway-secret");
+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({
+ provider: "vercel",
+ apiKey: "test-gateway-secret",
+ });
});
});
diff --git a/apps/cli/test/cli.ts b/apps/cli/test/cli.ts
index f63f8f8..5959b03 100644
--- a/apps/cli/test/cli.ts
+++ b/apps/cli/test/cli.ts
@@ -25,8 +25,6 @@ export async function withCli(
HOME: home,
XDG_CONFIG_HOME: home,
XDG_CACHE_HOME: join(home, "cache"),
- AI_GATEWAY_API_KEY: "",
- AI_GATEWAY_BASE_URL: undefined,
...env,
},
stdin: new Blob([input]),
diff --git a/apps/cli/test/process.test.ts b/apps/cli/test/process.test.ts
index 6eb0e0c..d412095 100644
--- a/apps/cli/test/process.test.ts
+++ b/apps/cli/test/process.test.ts
@@ -1,4 +1,6 @@
import { expect } from "bun:test";
+import { readFile, readdir } from "node:fs/promises";
+import { join } from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { fileURLToPath } from "node:url";
@@ -30,7 +32,7 @@ finally:
`,
cli,
],
- { env: { ...process.env, HOME: home, AI_GATEWAY_API_KEY: "" } },
+ { env: { ...process.env, HOME: home } },
);
expect(JSON.parse(stdout)).toEqual({ code: 0, stderr: "" });
});
@@ -38,7 +40,15 @@ finally:
testInDocker("interactive auth hides input and exits 130 on interruption", async () => {
await withCli(async ({ home }) => {
- for (const mode of ["save", "interrupt"]) {
+ for (const mode of [
+ "save",
+ "interrupt-provider",
+ "interrupt-key",
+ "cancel-provider",
+ "cancel-key",
+ ]) {
+ const file = join(home, "jevgrep", "credentials.json");
+ const previous = mode === "save" ? undefined : await readFile(file, "utf8");
const { stdout } = await execute(
"python3",
[
@@ -50,14 +60,25 @@ process = subprocess.Popen(["node", sys.argv[1], "auth"], stdin=slave, stdout=su
os.close(slave)
output = b""
try:
- deadline = time.monotonic() + 5
- while b"API key" not in output:
- if time.monotonic() >= deadline:
- raise RuntimeError("Auth prompt did not appear")
- if select.select([process.stdout], [], [], 0.1)[0]:
- output += os.read(process.stdout.fileno(), 4096)
- if sys.argv[2] == "save":
+ def wait_for(marker):
+ global output
+ deadline = time.monotonic() + 5
+ while marker not in output:
+ if time.monotonic() >= deadline:
+ raise RuntimeError("Auth prompt did not appear")
+ if select.select([process.stdout], [], [], 0.1)[0]:
+ part = os.read(process.stdout.fileno(), 4096)
+ if not part: raise RuntimeError("Auth exited before prompt")
+ output += part
+ wait_for(b"Choose your Jev provider")
+ mode = sys.argv[2]
+ if not mode.endswith("provider"):
+ os.write(master, b"\\x1b[B\\x1b[B\\r")
+ wait_for(b"API key")
+ if mode == "save":
os.write(master, b"pty-fixture-secret\\r")
+ elif mode.startswith("cancel"):
+ os.write(master, b"\\x03")
else:
process.send_signal(signal.SIGINT)
rest, errors = process.communicate(timeout=5)
@@ -79,13 +100,24 @@ finally:
cli,
mode,
],
- { env: { ...process.env, HOME: home, XDG_CONFIG_HOME: home, AI_GATEWAY_API_KEY: "" } },
+ { env: { ...process.env, HOME: home, XDG_CONFIG_HOME: home } },
);
const result = JSON.parse(stdout);
expect(result.code).toBe(mode === "save" ? 0 : 130);
expect(result.stderr).toBe("");
expect(result.stdout + result.echo).not.toContain("pty-fixture-secret");
expect(result.stdout).toContain(mode === "save" ? "key saved" : "Interrupted");
+ if (mode === "save") {
+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({
+ provider: "openrouter",
+ apiKey: "pty-fixture-secret",
+ });
+ expect(result.stdout.indexOf("Vercel AI Gateway")).toBeLessThan(
+ result.stdout.indexOf("TypeSafe"),
+ );
+ expect(result.stdout.indexOf("TypeSafe")).toBeLessThan(result.stdout.indexOf("OpenRouter"));
+ } else expect(await readFile(file, "utf8")).toBe(previous);
+ expect(await readdir(join(home, "jevgrep"))).toEqual(["credentials.json"]);
}
});
});
diff --git a/docs/architecture.md b/docs/architecture.md
index 9a6a911..e17811a 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -95,11 +95,14 @@ does not guarantee that the coding agent tests the right behavioral boundary.
## Integration and acceptance
-Use TypeScript and the AI SDK evaluation interface through AI Gateway. Pass native
+Use TypeScript and the AI SDK evaluation interface through one TypeSafe-compatible
+adapter, with fixed Vercel, TypeSafe and OpenRouter presets. Pass native
`state` and `questions` objects; source is a string field within that data. The
SDK handles HTTP serialization. Keep provider authentication and model access at
the core boundary so future providers do not reshape traversal or stdout.
-The CLI authentication module handles the Gateway key; provider expansion is deferred.
+The CLI auth module owns one saved provider/key. The core preset owner resolves
+its endpoint/model; the evaluator owns retries and answer-cache identity. Provider
+selection never changes traversal or source rendering.
Acceptance measures the entire downstream Sol task: an official solve at a lower
full task cost than its fixed baseline, while preserving baseline solves. Jev
diff --git a/evals/implementation/swebench/gateway_broker.py b/evals/implementation/swebench/gateway_broker.py
index fba46ca..ea9e950 100644
--- a/evals/implementation/swebench/gateway_broker.py
+++ b/evals/implementation/swebench/gateway_broker.py
@@ -74,13 +74,19 @@ class Gateway(http.server.BaseHTTPRequestHandler):
codex=self.path=='/codex/v1/responses'
engine='codex' if codex else 'claude' if claude else 'jev'
if (codex or claude) and config.get('agent_engine')!=engine:self.send_error(403);return
- jev=self.path=='/evaluation-model' and self.headers.get('ai-model-id')=='typesafe-ai/jev'
+ jev=self.path=='/typesafe/v1/systemone' and self.headers.get('x-jevgrep-original-url')=='https://ai-gateway.vercel.sh/typesafe/v1/systemone'
if jev and config.get('allow_jev') is not True:self.send_error(403);return
if not (claude or codex or jev):self.send_error(403);return
size=int(self.headers.get('Content-Length','0'))
if size<1 or size>2000000:self.send_error(413);return
data=self.rfile.read(size)
- if jev:response_trace=capture_jev(request_id,data)
+ if jev:
+ try:body=json.loads(data)
+ except ValueError:self.send_error(403);return
+ if (not isinstance(body,dict) or body.get('model')!='typesafe-ai/jev' or not isinstance(body.get('state'),dict) or not isinstance(body.get('questions'),dict) or
+ any(not isinstance(question,dict) or question.get('type')!='noul' or not isinstance(question.get('instructions'),str) for question in body['questions'].values())):
+ self.send_error(403);return
+ response_trace=capture_jev(request_id,data)
headers={'Authorization':'Bearer '+config['key'],'Content-Type':'application/json'}
if codex:
body=json.loads(data)
@@ -95,8 +101,7 @@ class Gateway(http.server.BaseHTTPRequestHandler):
if self.headers.get(header):headers[header]=self.headers[header]
target=GATEWAY_ORIGIN+self.path
else:
- target=GATEWAY_ORIGIN+'/v4/ai/evaluation-model'
- headers.update({'ai-model-id':'typesafe-ai/jev','ai-evaluation-model-specification-version':'4','ai-gateway-protocol-version':'0.0.1','ai-gateway-auth-method':'api-key'})
+ target=GATEWAY_ORIGIN+'/typesafe/v1/systemone'
with LOG_LOCK:print(json.dumps({'kind':engine+'-request-start','requestId':request_id,'startedAt':time.time(),'operation':'count_tokens' if route.path.endswith('/count_tokens') else 'generation','streamed':bool(body.get('stream')) if claude or codex else False}),flush=True)
request=urllib.request.Request(target,data=data,headers=headers)
try:response=urllib.request.urlopen(request,timeout=90)
diff --git a/evals/implementation/swebench/installed.py b/evals/implementation/swebench/installed.py
index b03c17a..001f6a4 100644
--- a/evals/implementation/swebench/installed.py
+++ b/evals/implementation/swebench/installed.py
@@ -19,6 +19,7 @@ import uuid
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[2]
+PROVIDER_ROUTE = HERE / 'provider-route.mjs' if (HERE / 'provider-route.mjs').exists() else ROOT / 'test/fixtures/provider-route.mjs'
REGISTRY = json.loads((HERE / 'fixed-baselines.json').read_text())
HARNESS_COMMIT = '02e7a74ffd0b707aab73d203fe87bdc7c76afc8e'
TIMING_POLICY = {'clock': 'work_excluding_standalone_jg_search_wait', 'work_seconds': 900,
@@ -171,13 +172,13 @@ tar -C /opt -cf /tmp/installed-prefix.tar jg-install
write_json(out / 'agent-inputs.json', [public for _, _, public in pairs])
frozen = out / 'runner'
frozen.mkdir(mode=0o700)
- for source in [Path(__file__).resolve(), HERE / 'gateway_broker.py', HERE / 'fixed-baselines.json']:
+ for source in [Path(__file__).resolve(), HERE / 'gateway_broker.py', HERE / 'fixed-baselines.json', PROVIDER_ROUTE]:
shutil.copyfile(source, frozen / source.name)
artifacts = [*frozen.iterdir(), package, out / 'installed-prefix.tar', out / 'skill.md', out / 'agent-inputs.json', root / REGISTRY['dataset']]
- plan = {'schema': 3, 'timing_policy': dict(TIMING_POLICY), 'status': 'frozen', 'cells': cells,
+ plan = {'schema': 4, 'timing_policy': dict(TIMING_POLICY), 'status': 'frozen', 'cells': cells,
'purpose': 'One prospective package cohort; reuse fixed baselines without execution.',
'inputs': str(out / 'agent-inputs.json'), 'skill': str(out / 'skill.md'), 'package': str(package),
- 'runner': str(frozen / 'installed.py'), 'broker': str(frozen / 'gateway_broker.py'),
+ 'runner': str(frozen / 'installed.py'), 'broker': str(frozen / 'gateway_broker.py'), 'provider_preload': str(frozen / 'provider-route.mjs'),
'registry': str(frozen / 'fixed-baselines.json'), 'dataset': str(root / REGISTRY['dataset']),
'tooling': str(root / REGISTRY['tooling']),
'artifacts': {str(path): digest(path) for path in artifacts}}
@@ -189,16 +190,18 @@ tar -C /opt -cf /tmp/installed-prefix.tar jg-install
def load_cohort(path):
plan = json.loads(path.read_text())
- if plan.get('schema') != 3 or plan.get('status') != 'frozen':
- raise ValueError('Expected a frozen schema-3 cohort; use the archived runner for older studies')
+ if plan.get('schema') != 4 or plan.get('status') != 'frozen':
+ raise ValueError('Expected a frozen schema-4 cohort; use the archived runner for older studies')
if plan.get('timing_policy') != TIMING_POLICY:
raise ValueError('Frozen treatment timing policy changed')
for artifact, expected in plan['artifacts'].items():
if digest(artifact) != expected:
raise ValueError('Frozen artifact changed: ' + artifact)
- for key in ['runner', 'broker', 'registry', 'inputs', 'skill', 'package', 'dataset']:
- if plan[key] not in plan['artifacts']:
+ for key in ['runner', 'broker', 'provider_preload', 'registry', 'inputs', 'skill', 'package', 'dataset']:
+ if plan.get(key) not in plan['artifacts']:
raise ValueError('Plan omits required artifact: ' + key)
+ if digest(PROVIDER_ROUTE) != digest(plan['provider_preload']):
+ raise ValueError('Use the frozen provider preload')
if digest(__file__) != digest(plan['runner']) or digest(HERE / 'gateway_broker.py') != digest(plan['broker']) or json.loads(Path(plan['registry']).read_text()) != REGISTRY:
raise ValueError('Use the frozen runner and registry')
prefixes = {item['cell']['candidate'] for item in plan['cells']}
@@ -458,6 +461,12 @@ def run(args):
receipt['prompt_sha256'] = digest(out / 'prompt.txt')
token = uuid.uuid4().hex
put(proxy, '/run/gateway.json', json.dumps({'key': os.environ['AI_GATEWAY_API_KEY'], 'token': token, 'agent_engine': 'codex', 'allow_jev': True}).encode(), 'root:root')
+ dx(['mkdir', '-p', '/home/agent/.config/jevgrep'], 'agent')
+ dx(['chmod', '700', '/home/agent/.config/jevgrep'], 'agent')
+ dx(['mkdir', '-p', '/opt/jg-harness'])
+ put(tag, '/home/agent/.config/jevgrep/credentials.json', json.dumps({'provider': 'vercel', 'apiKey': token}).encode())
+ put(tag, '/opt/jg-harness/provider-route.mjs', Path(plan['provider_preload']).read_bytes(), 'root:root', '444')
+ receipt['provider_preload_sha256'] = digest(plan['provider_preload'])
config = ('model = "openai/gpt-5.6-sol"\nmodel_provider = "vercel"\nmodel_reasoning_effort = "medium"\n'
'[model_providers.vercel]\nname = "Vercel AI Gateway"\n'
'base_url = "http://model-egress:3129/codex/v1"\nenv_key = "JEVGREP_MODEL_TOKEN"\nwire_api = "responses"\n')
@@ -467,7 +476,9 @@ def run(args):
path = '/opt/jg-install/bin:/opt/node-v24.14.0-linux-x64/bin:/opt/miniconda3/envs/testbed/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
invocation = ['docker', 'exec', '-u', 'agent', '-e', 'HOME=/home/agent', '-e', 'PATH=' + path,
'-e', 'CONDA_PREFIX=/opt/miniconda3/envs/testbed', '-e', 'JEVGREP_MODEL_TOKEN=' + token,
- '-e', 'AI_GATEWAY_API_KEY=' + token, '-e', 'AI_GATEWAY_BASE_URL=http://model-egress:3129', '-w', '/testbed', tag, *native]
+ '-e', 'XDG_CONFIG_HOME=/home/agent/.config',
+ '-e', 'NODE_OPTIONS=--import=/opt/jg-harness/provider-route.mjs',
+ '-e', 'JEVGREP_TEST_PROVIDER_ORIGIN=http://model-egress:3129', '-w', '/testbed', tag, *native]
receipt['status'] = 'running'
receipt['agent_started_at'] = time.time()
write_json(out / 'receipt.json', receipt)
@@ -578,19 +589,21 @@ def observed_jev(out, events, log_valid, traces_copied):
try:
response = responses[identifier]
body = json.loads(response.read_text())
- gateway = body['providerMetadata']['gateway']
- value = gateway.get('cost')
- if type(value) not in (str, int, float):
- raise ValueError('Missing or invalid response cost')
- cost = Decimal(str(value))
- if not cost.is_finite() or cost < 0 or not math.isfinite(float(cost)):
- raise ValueError('Invalid response cost')
- costs.append(cost)
usage = body.get('usage', {})
- if count(usage.get('inputTokens')): inputs.append(usage['inputTokens'])
- if count(usage.get('outputTokens')): outputs.append(usage['outputTokens'])
+ if count(usage.get('input_tokens', usage.get('inputTokens'))): inputs.append(usage.get('input_tokens', usage.get('inputTokens')))
+ if count(usage.get('output_tokens', usage.get('outputTokens'))): outputs.append(usage.get('output_tokens', usage.get('outputTokens')))
+ gateway = body.get('provider_metadata', body.get('providerMetadata', {})).get('gateway', {})
provider_attempts = gateway.get('routing', {}).get('totalProviderAttemptCount')
if count(provider_attempts): attempts.append(provider_attempts)
+ value = gateway.get('cost')
+ # Billing can be absent while usage and transport coverage remain known.
+ try:
+ if type(value) not in (str, int, float): raise ValueError('Missing cost')
+ cost = Decimal(str(value))
+ if not cost.is_finite() or cost < 0 or not math.isfinite(float(cost)): raise ValueError('Invalid cost')
+ costs.append(cost)
+ except (ValueError, InvalidOperation):
+ pass
matching = [event for event in ends if event.get('requestId') == identifier]
if len(matching) != 1:
coverage = False
@@ -606,7 +619,7 @@ def observed_jev(out, events, log_valid, traces_copied):
if not math.isfinite(known):
known = None
complete = False
- return {'basis': 'Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation',
+ return {'basis': 'Retained response provider_metadata.gateway.cost or historical providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation',
'included_in_scored_task_cost': False, 'client_calls': len(ids), 'responses_with_cost': len(costs),
'complete': complete, 'observed_cost_usd': known if complete else None, 'known_cost_usd': known,
'input_tokens': sum(inputs) if coverage and len(inputs) == len(ids) else None,
diff --git a/evals/implementation/swebench/test_installed.py b/evals/implementation/swebench/test_installed.py
index 2652e95..a21fb8f 100644
--- a/evals/implementation/swebench/test_installed.py
+++ b/evals/implementation/swebench/test_installed.py
@@ -170,9 +170,9 @@ class InstalledTests(unittest.TestCase):
patcher=patch.object(runner,'REGISTRY',registry);patcher.start();self.addCleanup(patcher.stop)
inputs = root / 'agent-inputs.json'; runner.write_json(inputs,rows)
dataset=root/'dataset.json';dataset.write_text('fixture dataset')
- paths = [dataset,Path(runner.__file__),HERE/'gateway_broker.py',registry_path,inputs,skill,package,prefix]
- plan = {'schema':3,'timing_policy':dict(runner.TIMING_POLICY),'status':'frozen','cells':cells,'runner':str(Path(runner.__file__)),'broker':str(HERE/'gateway_broker.py'),
- 'registry':str(registry_path),'tooling':str(root/'tooling'),'dataset':str(root/'dataset.json'),
+ paths = [runner.PROVIDER_ROUTE,dataset,Path(runner.__file__),HERE/'gateway_broker.py',registry_path,inputs,skill,package,prefix]
+ plan = {'schema':4,'timing_policy':dict(runner.TIMING_POLICY),'status':'frozen','cells':cells,'runner':str(Path(runner.__file__)),'broker':str(HERE/'gateway_broker.py'),
+ 'provider_preload':str(runner.PROVIDER_ROUTE),'registry':str(registry_path),'tooling':str(root/'tooling'),'dataset':str(root/'dataset.json'),
'inputs':str(inputs),'skill':str(skill),'package':str(package),'artifacts':{str(p):runner.digest(p) for p in paths}}
plan_path=root/'plan.json';runner.write_json(plan_path,plan)
return plan_path,{**plan,**cells[0]}
@@ -186,9 +186,10 @@ class InstalledTests(unittest.TestCase):
def test_timeout_with_zero_exit_retains_patch_but_cannot_complete_treatment(self):
with tempfile.TemporaryDirectory() as temporary:
path, plan = self.fixture(Path(temporary))
- calls, elapsed, lifetime = [], [0], [float('inf')]
+ calls, writes, elapsed, lifetime = [], {}, [0], [float('inf')]
def docker(argv, checked=True, **kwargs):
calls.append(argv)
+ if 'input' in kwargs: writes[argv[-3]] = kwargs['input']
if argv[:3] == ['docker', 'run', '-d'] and argv[-2] == 'sleep':
lifetime[0] = float(argv[-1])
if argv[:2] == ['docker', 'exec'] and elapsed[0] >= lifetime[0]:
@@ -208,6 +209,13 @@ class InstalledTests(unittest.TestCase):
output = b'fixture patch'
return subprocess.CompletedProcess(argv, 0, output, b'')
def expired(invocation, stdout, stderr, policy):
+ self.assertIn('NODE_OPTIONS=--import=/opt/jg-harness/provider-route.mjs', invocation)
+ self.assertIn('JEVGREP_TEST_PROVIDER_ORIGIN=http://model-egress:3129', invocation)
+ self.assertFalse(any(arg.startswith('AI_GATEWAY_') for arg in invocation))
+ saved = json.loads(writes['/home/agent/.config/jevgrep/credentials.json'])
+ self.assertEqual(saved['provider'], 'vercel')
+ self.assertNotEqual(saved['apiKey'], 'fixture-only')
+ self.assertEqual(writes['/opt/jg-harness/provider-route.mjs'], runner.PROVIDER_ROUTE.read_bytes())
# Native completion can already be in the pipe when the work
# deadline is noticed. A zero OS exit cannot override that clock.
# Simulate an hour of retrieval plus expired work. Docker's
@@ -402,21 +410,86 @@ class InstalledTests(unittest.TestCase):
self.assertEqual(result['gateway_cost_usd'],0.1);self.assertTrue(result['successful_cost_win'])
self.assertEqual(result['jev']['observed_cost_usd'],100.02)
+ def test_native_usage_survives_missing_cost(self):
+ for cost in ['0.04', None, 'NaN', True, '0']:
+ with self.subTest(cost=cost), tempfile.TemporaryDirectory() as temporary:
+ root=Path(temporary);events=self.jev_fixture(root)
+ for index in range(2):
+ response=root/'jev-traces'/(str(index)*32+'.response.json')
+ runner.write_json(response,{'usage':{'input_tokens':11,'output_tokens':3},'provider_metadata':{'gateway':{'cost':cost,'generationId':'fixture-generation'}}})
+ events[index*2+1]['responseBytes']=response.stat().st_size
+ result=runner.observed_jev(root,events,True,True)
+ self.assertEqual(result['input_tokens'],22);self.assertEqual(result['output_tokens'],6)
+ self.assertEqual(result['observed_cost_usd'],float(cost)*2 if cost in ('0.04','0') else None)
+ self.assertEqual(result['complete'],cost in ('0.04','0'))
+
+ def test_preload_is_required_and_identity_checked(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root=Path(temporary);path,plan=self.fixture(root)
+ runner.load_cohort(path)
+ plan=json.loads(path.read_text())
+ plan['schema']=3;runner.write_json(path,plan)
+ with self.assertRaisesRegex(ValueError,'schema-4'):runner.load_cohort(path)
+ plan['schema']=4
+ preload=root/'provider-route.mjs';preload.write_bytes(runner.PROVIDER_ROUTE.read_bytes())
+ plan['provider_preload']=str(preload)
+ runner.write_json(path,plan)
+ with self.assertRaisesRegex(ValueError,'required artifact'):runner.load_cohort(path)
+ plan['artifacts'][str(preload)]=runner.digest(preload);runner.write_json(path,plan)
+ runner.load_cohort(path)
+ preload.write_text('modified')
+ with self.assertRaisesRegex(ValueError,'Frozen artifact changed'):runner.load_cohort(path)
+ plan['artifacts'][str(preload)]=runner.digest(preload);runner.write_json(path,plan)
+ with self.assertRaisesRegex(ValueError,'frozen provider preload'):runner.load_cohort(path)
+
+ def test_broker_rejects_wrong_target_auth_and_body_before_capture(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root=Path(temporary);config=root/'gateway.json'
+ config.write_text(json.dumps({'key':'real-key','token':'token','allow_jev':True}))
+ route='/typesafe/v1/systemone'
+ headers={'Authorization':'Bearer token','x-jevgrep-original-url':'https://ai-gateway.vercel.sh'+route}
+ body={'model':'typesafe-ai/jev','state':{},'questions':{'q':{'type':'noul','instructions':'matches?'}}}
+ cases=[('/evaluation-model',headers,body),(route+'?target=evil',headers,body),
+ (route,{**headers,'Authorization':'Bearer wrong'},body),
+ (route,{**headers,'x-jevgrep-original-url':'https://evil.example'+route},body),
+ (route,headers,{**body,'model':'other'}),(route,headers,{**body,'state':None}),
+ (route,headers,{**body,'questions':{'q':{'type':'boolean'}}}),(route,headers,[]),(route,headers,'invalid-json'),(route,{'Authorization':'Bearer token'},body)]
+ with patch.object(broker,'CONFIG_PATH',str(config)),patch.object(broker,'TRACE_DIR',str(root/'traces')),patch.object(broker.urllib.request,'urlopen',side_effect=AssertionError('No forwarding')),contextlib.redirect_stdout(io.StringIO()):
+ server=http.server.ThreadingHTTPServer(('127.0.0.1',0),broker.Gateway)
+ thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
+ try:
+ for path,auth,value in cases:
+ client=http.client.HTTPConnection('127.0.0.1',server.server_port)
+ client.request('POST',path,value if isinstance(value,str) else json.dumps(value),auth)
+ response=client.getresponse();self.assertEqual(response.status,403);response.read();client.close()
+ finally:server.shutdown();server.server_close();thread.join()
+ self.assertFalse((root/'traces').exists())
+
def test_broker_captures_exact_jev_bodies_without_auth_headers(self):
with tempfile.TemporaryDirectory() as temporary:
root=Path(temporary);config=root/'gateway.json';config.write_text(json.dumps({'key':'REAL_KEY_SENTINEL','token':'BROKER_TOKEN_SENTINEL','allow_jev':True,'agent_engine':'codex'}))
- request_body=b'{"state":{"source":"public fixture"},"questions":{}}';response_body=b'{"answers":{}}'
- class Response(io.BytesIO):
- status=200;headers={'Content-Type':'application/json'}
- def read1(self,size):return self.read(size)
- with patch.object(broker,'CONFIG_PATH',str(config)),patch.object(broker,'TRACE_DIR',str(root/'traces')),patch.object(broker.urllib.request,'urlopen',return_value=Response(response_body)),contextlib.redirect_stdout(io.StringIO()):
+ request_body=b'{"model":"typesafe-ai/jev","state":{"source":"public fixture"},"questions":{"q":{"type":"noul","instructions":"matches?"}}}';response_body=b'{"answers":{},"usage":{"input_tokens":11,"output_tokens":3},"provider_metadata":{"gateway":{"cost":"0.04","generationId":"gen_fixture"}}}'
+ forwarded={}
+ class Upstream(http.server.BaseHTTPRequestHandler):
+ def log_message(self,*args):pass
+ def do_POST(self):
+ forwarded.update(path=self.path,body=self.rfile.read(int(self.headers['Content-Length'])),headers=dict(self.headers))
+ self.send_response(200);self.send_header('Content-Type','application/json');self.end_headers();self.wfile.write(response_body)
+ upstream=http.server.ThreadingHTTPServer(('127.0.0.1',0),Upstream)
+ upstream_thread=threading.Thread(target=upstream.serve_forever,daemon=True);upstream_thread.start()
+ self.addCleanup(upstream_thread.join);self.addCleanup(upstream.server_close);self.addCleanup(upstream.shutdown)
+ with patch.object(broker,'CONFIG_PATH',str(config)),patch.object(broker,'TRACE_DIR',str(root/'traces')),patch.object(broker,'GATEWAY_ORIGIN','http://127.0.0.1:'+str(upstream.server_port)),contextlib.redirect_stdout(io.StringIO()):
server=http.server.ThreadingHTTPServer(('127.0.0.1',0),broker.Gateway)
thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
try:
client=http.client.HTTPConnection('127.0.0.1',server.server_port)
- client.request('POST','/evaluation-model',request_body,{'Authorization':'Bearer BROKER_TOKEN_SENTINEL','ai-model-id':'typesafe-ai/jev'})
+ client.request('POST','/typesafe/v1/systemone',request_body,{'Authorization':'Bearer BROKER_TOKEN_SENTINEL','x-jevgrep-original-url':'https://ai-gateway.vercel.sh/typesafe/v1/systemone'})
response=client.getresponse();self.assertEqual(response.status,200);self.assertEqual(response.read(),response_body);client.close()
finally:server.shutdown();server.server_close();thread.join()
+ self.assertEqual(forwarded['path'],'/typesafe/v1/systemone')
+ self.assertEqual(forwarded['body'],request_body)
+ self.assertEqual(forwarded['headers']['Authorization'],'Bearer REAL_KEY_SENTINEL')
+ self.assertNotIn('X-Jevgrep-Original-Url',forwarded['headers'])
requests=list((root/'traces').glob('*.request.json'));responses=list((root/'traces').glob('*.response.json'))
self.assertEqual(requests[0].read_bytes(),request_body);self.assertEqual(responses[0].read_bytes(),response_body)
for file in requests+responses:
diff --git a/packages/core/package.json b/packages/core/package.json
index 80d7f39..03890b7 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -5,6 +5,7 @@
"type": "module",
"exports": {
".": "./src/index.ts",
+ "./providers": "./src/providers.ts",
"./source": "./src/source.ts"
},
"scripts": {
diff --git a/packages/core/src/gateway.ts b/packages/core/src/evaluator.ts
similarity index 88%
rename from packages/core/src/gateway.ts
rename to packages/core/src/evaluator.ts
index 419130b..ff4475b 100644
--- a/packages/core/src/gateway.ts
+++ b/packages/core/src/evaluator.ts
@@ -1,4 +1,6 @@
-import { createGateway, experimental_evaluate as evaluate } from "ai";
+import { APICallError, experimental_evaluate as evaluate } from "ai";
+import { createTypeSafeAi } from "@ai-sdk/typesafe-ai";
+import { providers, type ProviderId } from "./providers";
import { type createEvaluationCache, type CacheInput } from "./cache";
import { setTimeout as delay } from "node:timers/promises";
@@ -23,14 +25,16 @@ export class EvaluationFailure extends Error {
}
export function createEvaluator(options: {
+ provider: ProviderId;
apiKey: string;
cache?: ReturnType<typeof createEvaluationCache>;
policyVersion?: string;
- baseURL?: string;
+ fetch?: typeof fetch;
signal: AbortSignal;
requestLimit?: number;
timeoutMs?: number;
}) {
+ const preset = providers[options.provider];
let requests = 0;
let cacheHits = 0;
let cooldownUntil = 0;
@@ -39,15 +43,15 @@ export function createEvaluator(options: {
if (options.signal.aborted) throw new EvaluationFailure("cancelled");
if (authenticationFailure.signal.aborted) throw new EvaluationFailure("authentication");
}
- const gateway = createGateway({
+ const provider = createTypeSafeAi({
apiKey: options.apiKey,
- baseURL: options.baseURL,
+ baseURL: preset.baseURL,
fetch: async (input, init) => {
assertActive();
if (requests >= (options.requestLimit ?? 50_000))
throw new EvaluationFailure("request-limit");
requests++;
- const response = await fetch(input, init);
+ const response = await (options.fetch ?? fetch)(input, init);
if (response.status === 429) {
const raw = response.headers.get("retry-after");
const seconds = raw === null ? NaN : Number(raw);
@@ -81,8 +85,10 @@ export function createEvaluator(options: {
const cacheInput: CacheInput = {
request,
namespace: {
- model: "typesafe-ai/jev",
- provider: options.baseURL ?? "vercel-ai-gateway",
+ model: preset.model,
+ provider: options.provider,
+ endpoint: preset.baseURL,
+ protocol: "typesafe-ai-3.0.8",
policyVersion: options.policyVersion ?? "1",
parserVersion: "cpython-3.11.3-pyodide-0.25.1-ts-5.9.3",
promptVersion: "unit-locators-1",
@@ -121,7 +127,7 @@ export function createEvaluator(options: {
assertActive();
try {
const result = await evaluate({
- model: gateway.evaluationModel("typesafe-ai/jev"),
+ model: provider.evaluationModel(preset.model),
...request,
maxRetries: 0,
abortSignal: AbortSignal.any([
@@ -163,7 +169,8 @@ export function createEvaluator(options: {
status === 408 ||
status === 429 ||
(typeof status === "number" && status >= 500 && status <= 599) ||
- ["GatewayInternalServerError", "GatewayTimeoutError", "TimeoutError"].includes(name);
+ name === "TimeoutError" ||
+ (APICallError.isInstance(error) && error.statusCode === undefined && error.isRetryable);
if (navigation && status === 429) attemptLimit = Math.max(attemptLimit, 2);
if ((navigation && !transient) || attempt + 1 === attemptLimit)
throw new EvaluationFailure(
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 771fd6a..97e0dfe 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -1,4 +1,4 @@
export { retrieve } from "./retrieve";
-export { createEvaluator, EvaluationFailure } from "./gateway";
+export { createEvaluator, EvaluationFailure } from "./evaluator";
export { createEvaluationCache } from "./cache";
export type { SearchInput, RetrievalResult, FileEvidence } from "./types";
diff --git a/packages/core/src/providers.ts b/packages/core/src/providers.ts
new file mode 100644
index 0000000..afd7a55
--- /dev/null
+++ b/packages/core/src/providers.ts
@@ -0,0 +1,23 @@
+export const providers = {
+ vercel: {
+ label: "Vercel AI Gateway",
+ baseURL: "https://ai-gateway.vercel.sh/typesafe/v1",
+ model: "typesafe-ai/jev",
+ },
+ typesafe: {
+ label: "TypeSafe",
+ baseURL: "https://api.typesafe.ai/v1",
+ model: "jev-1.13.0",
+ },
+ openrouter: {
+ label: "OpenRouter",
+ baseURL: "https://openrouter.ai/api/v1",
+ model: "jev-1.13",
+ },
+} as const;
+
+export type ProviderId = keyof typeof providers;
+
+export function isProviderId(value: unknown): value is ProviderId {
+ return typeof value === "string" && Object.hasOwn(providers, value);
+}
diff --git a/packages/core/src/retrieve.ts b/packages/core/src/retrieve.ts
index 8d8d027..adff0b1 100644
--- a/packages/core/src/retrieve.ts
+++ b/packages/core/src/retrieve.ts
@@ -1,6 +1,6 @@
import { basename, extname } from "node:path";
import { createFilesystem, type Snapshot, type DirectoryEntry } from "./filesystem";
-import { type EvaluationRequest, EvaluationFailure } from "./gateway";
+import { type EvaluationRequest, EvaluationFailure } from "./evaluator";
import { inspect, pythonPreview, sourceForUnit, splitSource } from "./source";
import {
navigationRequest,
diff --git a/packages/core/src/selection.ts b/packages/core/src/selection.ts
index 04f24d6..4c90464 100644
--- a/packages/core/src/selection.ts
+++ b/packages/core/src/selection.ts
@@ -1,4 +1,4 @@
-import { EvaluationFailure } from "./gateway";
+import { EvaluationFailure } from "./evaluator";
import { evidenceRequest, type Evidence } from "./requests";
import {
inspect,
diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts
index bbd14ab..31a3c1a 100644
--- a/packages/core/src/types.ts
+++ b/packages/core/src/types.ts
@@ -1,5 +1,5 @@
import type { FilesystemPolicy } from "./filesystem";
-import type { EvaluationRequest } from "./gateway";
+import type { EvaluationRequest } from "./evaluator";
import type { Range } from "./source";
export type { Range } from "./source";
diff --git a/packages/core/test-node/provider-protocol.test.mjs b/packages/core/test-node/provider-protocol.mjs
similarity index 100%
rename from packages/core/test-node/provider-protocol.test.mjs
rename to packages/core/test-node/provider-protocol.mjs
diff --git a/packages/core/test/cache.test.ts b/packages/core/test/cache.test.ts
index 4fbddce..101316e 100644
--- a/packages/core/test/cache.test.ts
+++ b/packages/core/test/cache.test.ts
@@ -75,10 +75,12 @@ test("source, path, query, ordered questions and namespace changes cannot reuse
},
{ ...input, request: { ...input.request, questions: ["question2", "question1"] } },
{ ...input, request: { ...input.request, query: "different question" } },
- ...["model", "provider", "policyVersion", "promptVersion"].map((name) => ({
- ...input,
- namespace: { ...input.namespace, [name]: "changed" },
- })),
+ ...["model", "provider", "endpoint", "protocol", "policyVersion", "promptVersion"].map(
+ (name) => ({
+ ...input,
+ namespace: { ...input.namespace, [name]: "changed" },
+ }),
+ ),
];
for (const variant of variants) expect(await cache.get(variant)).toBeUndefined();
expect(await cache.get(input)).toEqual({ question1: 0.8 });
diff --git a/packages/core/test/selection.test.ts b/packages/core/test/selection.test.ts
index 0d1a055..a311699 100644
--- a/packages/core/test/selection.test.ts
+++ b/packages/core/test/selection.test.ts
@@ -1,6 +1,6 @@
import { expect, test } from "bun:test";
import { selectFile } from "../src/selection";
-import { EvaluationFailure, type EvaluationRequest } from "../src/gateway";
+import { EvaluationFailure, type EvaluationRequest } from "../src/evaluator";
import type { Evaluator } from "../src/types";
import type { Declaration } from "../src/requests";
diff --git a/scripts/licenses/README.md b/scripts/licenses/README.md
index 2b20449..8d01e4f 100644
--- a/scripts/licenses/README.md
+++ b/scripts/licenses/README.md
@@ -1,9 +1,10 @@
# License sources omitted or abbreviated upstream
-`provider-utils-5.0.45.LICENSE` is the Vercel AI repository license at
-[the exact provider-utils 5.0.45 tag revision](https://github.com/vercel/ai/blob/08ae5ad05bc12496dd1ffcf64e34419e0831300d/LICENSE).
-That npm package declares Apache-2.0 but omits its license file. The override is
-version-specific so an upgrade cannot silently reuse an unchecked notice.
+`vercel-ai.LICENSE` is the Vercel AI repository license, verified byte-for-byte at
+[provider-utils 5.0.45](https://github.com/vercel/ai/blob/08ae5ad05bc12496dd1ffcf64e34419e0831300d/LICENSE)
+and [provider-utils 5.0.49](https://github.com/vercel/ai/blob/ee3169b3c4880e2abe4d0d7c781243bb81822ec4/LICENSE).
+Both npm packages declare Apache-2.0 but omit their license file. The notice
+owner allows only these verified versions; a new version requires inspection.
`Apache-2.0.txt` contains the [Apache Software Foundation's full license terms](https://www.apache.org/licenses/LICENSE-2.0.txt).
The notice generator retains package copyright notices and includes these terms
diff --git a/scripts/licenses/provider-utils-5.0.45.LICENSE b/scripts/licenses/vercel-ai.LICENSE
similarity index 100%
rename from scripts/licenses/provider-utils-5.0.45.LICENSE
rename to scripts/licenses/vercel-ai.LICENSE
diff --git a/scripts/package-notices.mjs b/scripts/package-notices.mjs
index 7d44550..529d6c8 100644
--- a/scripts/package-notices.mjs
+++ b/scripts/package-notices.mjs
@@ -4,6 +4,10 @@ import { dirname, resolve } from "node:path";
/** Use emitted inputs, so the notice follows the actual bundled dependency graph. */
export async function bundledNotices(metafile, root) {
const packages = new Map();
+ const retainedLicenses = {
+ "@ai-sdk/[email protected]": "vercel-ai.LICENSE",
+ "@ai-sdk/[email protected]": "vercel-ai.LICENSE",
+ };
let needsApacheTerms = false;
for (const output of Object.values(metafile.outputs)) {
for (const [input, contribution] of Object.entries(output.inputs)) {
@@ -26,7 +30,7 @@ export async function bundledNotices(metafile, root) {
)
.map((entry) => entry.name)
.sort();
- if (!files.length && key !== "@ai-sdk/[email protected]")
+ if (!files.length && !retainedLicenses[key])
throw new Error(`Bundled dependency ${key} has no license file`);
if (metadata.license === "Apache-2.0") needsApacheTerms = true;
const texts = await Promise.all(
@@ -39,7 +43,7 @@ export async function bundledNotices(metafile, root) {
if (!files.length)
texts.push(
await readFile(
- new URL("./licenses/provider-utils-5.0.45.LICENSE", import.meta.url),
+ new URL(`./licenses/${retainedLicenses[key]}`, import.meta.url),
"utf8",
),
);
diff --git a/scripts/test-installed.sh b/scripts/test-installed.sh
index 20be0db..db79a8d 100755
--- a/scripts/test-installed.sh
+++ b/scripts/test-installed.sh
@@ -42,7 +42,7 @@ if [[ -n "$package_input" ]]; then
cp "${JEVGREP_CANONICAL_SKILL:-$repo_root/skills/jevgrep/SKILL.md}" "$scratch/.package-input/canonical-skill.md"
cp "$repo_root/test/installed.test.mjs" "$repo_root/test/runtime.Dockerfile" "$scratch/test/"
mkdir -p "$scratch/test/fixtures"
- cp "$repo_root/test/fixtures/skill-installer.mjs" "$scratch/test/fixtures/"
+ cp "$repo_root/test/fixtures/skill-installer.mjs" "$repo_root/test/fixtures/provider-route.mjs" "$scratch/test/fixtures/"
context_dir="$scratch"
build_args=(--build-arg PACKAGE_STAGE=prebuilt)
fi
diff --git a/scripts/test-native.mjs b/scripts/test-native.mjs
index 37a7122..3dfe2b4 100644
--- a/scripts/test-native.mjs
+++ b/scripts/test-native.mjs
@@ -5,7 +5,7 @@ import { createHash } from "node:crypto";
import { cp, mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises";
import { release, tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
-import { fileURLToPath } from "node:url";
+import { fileURLToPath, pathToFileURL } from "node:url";
import { promisify } from "node:util";
const execute = promisify(execFile);
const root = fileURLToPath(new URL("../", import.meta.url));
@@ -100,6 +100,8 @@ try {
);
const skill = join(scratch, "canonical-skill.md");
await cp(process.env.JEVGREP_CANONICAL_SKILL || join(root, "skills/jevgrep/SKILL.md"), skill);
+ const preload = join(scratch, "provider-route.mjs");
+ await cp(join(root, "test/fixtures/provider-route.mjs"), preload);
await symlink(process.execPath, join(runtimeBin, "node"));
const runtimeEnv = {
PATH: runtimeBin,
@@ -107,8 +109,7 @@ try {
XDG_CONFIG_HOME: config,
XDG_CACHE_HOME: cache,
TMPDIR: scratch,
- AI_GATEWAY_API_KEY: "",
- AI_GATEWAY_BASE_URL: "http://127.0.0.1:1/unused",
+ JEVGREP_TEST_PROVIDER_PRELOAD: pathToFileURL(preload).href,
JEVGREP_INSTALLED_BINARY: binary,
JEVGREP_INSTALLED_PACKAGE: packageDirectory,
JEVGREP_EXPECTED_SKILL: skill,
@@ -130,7 +131,7 @@ try {
"--test-reporter=tap",
"--test-concurrency=1",
"--test-name-pattern",
- "^(installed local commands match the package without credentials|skill command delegates installation to npx without Gateway credentials|actual installed search parses Python and returns every relevant hierarchy branch)$",
+ "^(installed local commands match the package without credentials|skill command delegates installation to npx without credentials|actual installed search parses Python and returns every relevant hierarchy branch|installed saved-provider journey: (vercel|typesafe|openrouter) auth doctor search cache replacement|installed legacy credentials use Vercel without rewriting saved bytes|installed saved credentials defeat conflicting environment and environment-only auth fails|installed invalid saved providers fail before HTTP without rewriting credentials|installed invalid auth preserves saved bytes and leaves no temporary credentials)$",
join(root, "test/installed.test.mjs"),
],
{ cwd: scratch, env: runtimeEnv, timeout: 120_000, maxBuffer: 8_000_000 },
@@ -138,7 +139,7 @@ try {
assert.equal(tested.stderr, "");
assert.match(
tested.stdout,
- /^# pass 3$/m,
+ /^# pass 10$/m,
"All native smoke journeys must execute; renamed selectors cannot silently pass",
);
console.log(tested.stdout.trimEnd());
diff --git a/scripts/test-suite.sh b/scripts/test-suite.sh
index 92ad1ec..dfa42ac 100644
--- a/scripts/test-suite.sh
+++ b/scripts/test-suite.sh
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
-bun test apps/cli/test packages/core/test test/reference test/gateway.test.ts test/retrieval.test.ts test/retrieval-freshness.test.ts
-node --test packages/core/test-node/*.test.mjs
+bun test apps/cli/test packages/core/test test/reference test/evaluator.test.ts test/retrieval.test.ts test/retrieval-freshness.test.ts
+node --test packages/core/test-node/provider-protocol.mjs
node --experimental-strip-types --test test/parser/*.test.ts
node --test test/release.test.mjs
python3 -B -m unittest discover -s evals/implementation/swebench -p test_installed.py
diff --git a/skills/jevgrep/SKILL.md b/skills/jevgrep/SKILL.md
index 8105107..5c73bf6 100644
--- a/skills/jevgrep/SKILL.md
+++ b/skills/jevgrep/SKILL.md
@@ -14,8 +14,8 @@ If prerequisites are missing or installation fails, report the specific blocker
and use ordinary discovery; do not retry indefinitely or use sudo.
If a search reports missing credentials, have the user run `jg auth` in their
-terminal or configure `AI_GATEWAY_API_KEY` through their secret manager. Never ask
-them to paste a key into chat. Do not launch the interactive auth prompt in a
+terminal, select their provider, and enter its key. Saved credentials are required;
+environment keys do not configure the CLI. Never ask them to paste a key into chat. Do not launch the interactive auth prompt in a
noninteractive agent shell. Continue with ordinary discovery until configured.
## Scope
diff --git a/specs/provider-support/README.md b/specs/provider-support/README.md
index 8db4b60..ba0beb0 100644
--- a/specs/provider-support/README.md
+++ b/specs/provider-support/README.md
@@ -6,11 +6,11 @@ ordinary searches keep the same interface and retrieval behavior.
## Next Agent Prompt
-You are implementing this plan, last updated 2026-09-26. Slice 1 is complete:
-18 Node protocol tests pass locally and in Docker, and all 5 frozen replay tests
-remain green. Start [slice 2](slices/02-evaluator.md): replace the evaluator while
-preserving production parity. Read [protocol evidence](assets/protocol-verification.md)
-and [preservation gates](parity.md). No live provider calls have occurred.
+Implementation is complete, last updated 2026-09-26. All four slices and required
+checks pass; see [final verification](assets/final-verification.md). Close and
+archive this spec after the final claim audit. TypeSafe/OpenRouter live access
+remains unverified because credentials were unavailable; offline support is
+verified and Vercel passed a live doctor check. No npm publication is selected.
You have no unresolved product question. Published-package behavior and live
provider access need verification, not invented compatibility code. Missing
@@ -19,9 +19,9 @@ honestly. Do not collect secrets in chat or overwrite the user's own saved setup
for tests. Use isolated HOME/config/cache locations.
- [x] [1. Protocol reproduction and controlled routing](slices/01-protocol.md)
-- [ ] [2. Shared evaluator and production parity](slices/02-evaluator.md)
-- [ ] [3. Saved-provider auth, installed journeys, and onboarding](slices/03-auth.md)
-- [ ] [4. Maintained eval routing and final verification](slices/04-tooling.md)
+- [x] [2. Shared evaluator and production parity](slices/02-evaluator.md)
+- [x] [3. Saved-provider auth, installed journeys, and onboarding](slices/03-auth.md)
+- [x] [4. Maintained eval routing and final verification](slices/04-tooling.md)
Update this prompt and the owning slice with evidence, remaining work and any
deviations before ending each pass. Review each implementation slice before
diff --git a/specs/provider-support/assets/auth-verification.md b/specs/provider-support/assets/auth-verification.md
new file mode 100644
index 0000000..e0a2cf5
--- /dev/null
+++ b/specs/provider-support/assets/auth-verification.md
@@ -0,0 +1,80 @@
+# Saved-provider verification
+
+2026-09-26. Installed Docker journeys cover Vercel, TypeSafe and OpenRouter:
+auth from stdin, provider-specific doctor, search, cache reuse, replacement and
+no fallback. They check private storage, ignored environment keys/URLs, invalid
+providers/keys, legacy bytes and file identity remaining unchanged. The Docker
+PTY test selects OpenRouter, checks hidden entry, and cancels at both prompts;
+prior credentials survive and no temporary credential file remains.
+
+All 32 installed Docker tests and all 10 native macOS smoke journeys passed on
+the same candidate archive. The native run used Node v24.14.0, darwin/arm64 25.6.0.
+
+## Live check
+
+The actual built CLI ran this sequence with isolated config/cache directories:
+
+```text
+jg auth --provider vercel --stdin [key supplied privately; exit 0]
+jg doctor
+Jev connection verified through Vercel AI Gateway.
+```
+
+Doctor exited 0 with empty stderr. The key came from the authorized local Duet
+environment, was never printed, and the temporary credentials were removed.
+TypeSafe/OpenRouter keys were unavailable there; live service access for those
+providers remains unverified. No repository source was used in the live check.
+
+A fresh smaller-model skill check correctly directed a missing-credential user
+to terminal `jg auth`, ignored an OpenRouter environment key, avoided asking for
+secrets, and continued ordinary discovery. The initial consultation was invalid:
+its instruction prohibited shell commands needed to read the skill. That was
+corrected before the fresh check; no product behavior was changed to fit it.
+
+## Exact verified reference packet
+
+This is the complete frozen packet matched byte-for-byte by each provider's
+production replay; it is not a sample invented for documentation.
+
+```text
+Jevgrep: 3 relevant files.
+AGENTS.md lookup (root and returned-file ancestors): none found.
+- "src/backend/events.ts" — implementation, caller, test, fixture, helper; selected source and structural context below
+ Reading lead source: lines 1-1
+ Reading lead BackendTelemetry.recordEvent: lines 4-6
+- "src/telemetry.ts" — implementation, caller, test, fixture, helper; selected source and structural context below
+ Reading lead Telemetry.recordEvent: lines 3-5
+- "tests/telemetry.test.ts" — implementation, caller, test, fixture, helper; selected source and structural context below
+ Reading lead source: lines 1-1
+ Reading lead testEventName: lines 3-5
+End file list.
+
+Source block "src/backend/events.ts" lines 1-8:
+1: import { Telemetry } from '../telemetry';
+2: // Backends preserve the same event contract.
+3: export class BackendTelemetry extends Telemetry {
+4: recordEvent(name: string) {
+5: return super.recordEvent(name);
+6: }
+7: }
+8:
+
+Source block "src/telemetry.ts" lines 1-7:
+1: // Events preserve the caller's name.
+2: export class Telemetry {
+3: recordEvent(name: string) {
+4: return { name, recorded: true };
+5: }
+6: }
+7:
+
+Source block "tests/telemetry.test.ts" lines 1-6:
+1: import { Telemetry } from '../src/telemetry';
+2: // Regression example: do not rename the caller's event.
+3: export function testEventName() {
+4: return new Telemetry().recordEvent('opened').name === 'opened';
+5: }
+6:
+
+End context.
+```
diff --git a/specs/provider-support/assets/evaluator-verification.md b/specs/provider-support/assets/evaluator-verification.md
new file mode 100644
index 0000000..d9657f8
--- /dev/null
+++ b/specs/provider-support/assets/evaluator-verification.md
@@ -0,0 +1,27 @@
+# Evaluator preservation evidence
+
+2026-09-26. All three fixed presets reproduce the frozen request multiset and
+complete stdout through the built production CLI under controlled probabilities.
+Missing and invalid responses remain incomplete with exit 2. The historical
+oracle, manifest, corpus and skill research body were not changed.
+
+The application still owns actual HTTP attempt counting, the 50,000 guard,
+shared cooldown, retries, navigation splitting and cancellation. SDK retries are
+disabled. The Node disconnect comparison matches the historical recovery path;
+409/422 do not acquire a navigation retry or split; 503 remains split-eligible.
+Numeric and HTTP-date Retry-After checks pass. Auth failure aborts siblings and
+future requests. Existing retry-time source exclusion/mutation checks pass.
+
+The cache keeps its existing answer-only storage. Provider replacement causes a
+miss; old protocol, endpoint, model, policy, prompt and changed source cannot
+share an answer. Warm identical requests reuse answers. The installed artifact
+continues to hide excluded source and reject stale donors.
+
+The broad Docker pass completed 125 core/CLI/reference tests, 18 Node protocol
+checks, 23 parser tests and 4 release tests. Independent code review found no
+remaining actionable defect. These are deterministic preservation checks, not a
+new task-solve or savings benchmark.
+
+The evaluator and auth were integrated as one cutover, rather than temporarily
+publishing provider selection before transport routing. This changed commit
+sequencing only; all planned behavior and gates remained.
diff --git a/specs/provider-support/assets/final-verification.md b/specs/provider-support/assets/final-verification.md
new file mode 100644
index 0000000..62afb05
--- /dev/null
+++ b/specs/provider-support/assets/final-verification.md
@@ -0,0 +1,41 @@
+# Final provider-support verification
+
+2026-09-26. Implementation gates passed; no npm release or paid task benchmark
+was run. Existing baselines, historical study artifacts, the reference manifest,
+covered source files, corpus and frozen research skill remain unchanged.
+
+| Gate | Result |
+| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| `bun run verify` | Exit 0: types, lint, 125 core/CLI/reference, 18 Node protocol, 23 parser, 4 release, 27 maintained harness and 32 installed Docker tests |
+| `bun run format:check` | Exit 0 |
+| Latest schema-4 harness, local Python and Docker | 27/27 in each, exit 0; rerun after the final plan-version adjustment |
+| Exact candidate, installed Docker `--prebuilt` | 32/32, exit 0 |
+| Same candidate, native `--prebuilt` | 10/10, exit 0; Node v24.14.0, darwin/arm64 25.6.0 |
+| Independent whole-diff Codex review | No actionable defects; pending suite completion was not assumed by the reviewer |
+| Live built CLI `jg doctor`, isolated saved Vercel setup | Exit 0, expected success text, empty stderr |
+| Live TypeSafe/OpenRouter | Unverified: credentials unavailable in the authorized local environment |
+
+Candidate: `@dzhng/jevgrep` source version 0.2.0, prepared with the existing package
+validator but **not published**. SHA-256:
+
+```text
+d8d7d0012da3163f2f615fddc01e07795251af28296950b5f248d990322bd6b2
+```
+
+The packaged file allowlist excludes the routing preload and fixtures; runtime
+requirements remain Node only. Docker runtime tests use network isolation and
+synthetic local HTTP; dependency downloads used Docker Desktop's existing proxy
+through a temporary Docker client config, without changing the user's settings.
+
+The broker's real local-upstream test verifies auth replacement, exact native
+request/response capture, and rejection before forwarding for wrong targets,
+credentials or bodies. Synthetic accounting retains 22 input tokens when cost
+is absent while the full Jev total remains unknown. Jev observations remain
+excluded from scored coding-agent cost. This proves maintained tooling behavior,
+not a newly measured solve rate, cost saving or latency.
+
+Review and cleanup preserved the one evaluator/one preset owner, removed the old
+module and environment setup path, retained exact-version license validation,
+and avoided a public transport override. The sole new data-format choice is
+recorded in [the choices ledger](../choices.md). No new benchmark cohort was
+prepared or executed.
diff --git a/specs/provider-support/assets/protocol-verification.md b/specs/provider-support/assets/protocol-verification.md
index a143323..15d8236 100644
--- a/specs/provider-support/assets/protocol-verification.md
+++ b/specs/provider-support/assets/protocol-verification.md
@@ -38,7 +38,7 @@ omits a license, like 5.0.45. The upstream repository LICENSE at tag revision
5.0.45 license. Slice 2 must extend the exact-version notice mapping before the
adapter becomes part of the product bundle.
-Reproduce with `node --test packages/core/test-node/provider-protocol.test.mjs`;
+Reproduce with `node --test packages/core/test-node/provider-protocol.mjs`;
the same command runs inside `scripts/test-reference.sh`, and the normal suite
now includes it. No credentials or live model calls were used.
diff --git a/specs/provider-support/choices.md b/specs/provider-support/choices.md
index 4e8969e..0fd5c43 100644
--- a/specs/provider-support/choices.md
+++ b/specs/provider-support/choices.md
@@ -1,10 +1,25 @@
# Provider support choices
-No unplanned product choice has been made in slice 1. The SDK pin, endpoint/model
-presets, external routing seam and immutable reference were specified. Probe file
-names and test layout are delegated implementation discretion.
-
-The routing fixture preserves the SDK's original body instead of converting it
-to a stream. This is a correctness fix within the specified transport contract,
-not an additional product behavior. Further passes append only decisions not
-already made or delegated by the plan.
+## Sound — high confidence
+
+### New benchmark plans use schema 4
+
+When a new benchmark cohort is frozen, its plan now includes the exact routing
+preload used by the installed CLI. A plan without that file cannot reproduce the
+new transport. The plan did not specify how to version this required addition.
+New plans use schema 4 so a reader can distinguish them from old schema-3 plans;
+old studies continue using their original frozen runners and are never rewritten.
+The alternative was to reuse schema 3 while changing its required fields, which
+would conceal a real format change. Future tooling must honor this distinction.
+
+This is a benchmark-harness format decision, not a credential migration: existing
+saved providerless keys still work as Vercel, exactly as requested.
+
+## Within the specification
+
+Provider presets, single-record authentication, environment removal, legacy-key
+handling, cache isolation, retry policy and the external test transport were
+already decided by the spec. Internal names and probe layout were delegated.
+The shared Node routing fixture preserves the SDK's original body after testing
+showed that converting it to a stream obscures 401 errors; that implements the
+specified transport contract rather than adding product behavior.
diff --git a/specs/provider-support/slices/02-evaluator.md b/specs/provider-support/slices/02-evaluator.md
index 185bd37..3020460 100644
--- a/specs/provider-support/slices/02-evaluator.md
+++ b/specs/provider-support/slices/02-evaluator.md
@@ -1,7 +1,7 @@
# 2. Replace the evaluator and prove production parity
**Unlock:** one provider-neutral evaluator preserves the existing search contract.
-Depends on slice 1. Status: not started.
+Depends on slice 1. Status: complete; see [verification](../assets/evaluator-verification.md).
## API seam and ownership
diff --git a/specs/provider-support/slices/03-auth.md b/specs/provider-support/slices/03-auth.md
index 96c380b..1529721 100644
--- a/specs/provider-support/slices/03-auth.md
+++ b/specs/provider-support/slices/03-auth.md
@@ -1,7 +1,7 @@
# 3. Save a provider and complete the installed journey
**Unlock:** users can choose any supported service once and use ordinary `jg`.
-Depends on slice 2. Status: not started.
+Depends on slice 2. Status: complete; see [verification](../assets/auth-verification.md).
## API and interaction
diff --git a/specs/provider-support/slices/04-tooling.md b/specs/provider-support/slices/04-tooling.md
index fa458e8..9853fe3 100644
--- a/specs/provider-support/slices/04-tooling.md
+++ b/specs/provider-support/slices/04-tooling.md
@@ -1,7 +1,7 @@
# 4. Preserve maintained eval routing and close verification
**Unlock:** the maintained harness can observe the actual installed product after
-environment-based auth is removed. Depends on slice 3. Status: not started.
+environment-based auth is removed. Depends on slice 3. Status: complete; see [verification](../assets/final-verification.md).
## Seam and scope
diff --git a/test/Dockerfile b/test/Dockerfile
index b8b05bc..118cfd7 100644
--- a/test/Dockerfile
+++ b/test/Dockerfile
@@ -21,5 +21,5 @@ COPY evals/implementation/swebench/hierarchy-unit-locators-spike.ts evals/implem
COPY evals/implementation/swebench/installed.py evals/implementation/swebench/gateway_broker.py evals/implementation/swebench/test_installed.py evals/implementation/swebench/
COPY evals/implementation/swebench/fixed-baselines.json evals/implementation/swebench/
RUN bun build evals/implementation/swebench/hierarchy-unit-locators-spike.ts --target=node --outfile=/opt/jevgrep-reference.mjs
-ENV JEVGREP_TEST_IN_DOCKER=1 HOME=/tmp/jevgrep-home XDG_CONFIG_HOME=/tmp/jevgrep-config XDG_CACHE_HOME=/tmp/jevgrep-cache AI_GATEWAY_API_KEY=""
+ENV JEVGREP_TEST_IN_DOCKER=1 HOME=/tmp/jevgrep-home XDG_CONFIG_HOME=/tmp/jevgrep-config XDG_CACHE_HOME=/tmp/jevgrep-cache
CMD ["bun", "test", "test/reference/replay.test.ts"]
diff --git a/test/gateway.test.ts b/test/evaluator.test.ts
similarity index 65%
rename from test/gateway.test.ts
rename to test/evaluator.test.ts
index 801a304..e5c609b 100644
--- a/test/gateway.test.ts
+++ b/test/evaluator.test.ts
@@ -1,5 +1,6 @@
+import { routeProviderFetch } from "./fixtures/provider-route.mjs";
import { expect, test } from "bun:test";
-import { createEvaluator } from "../packages/core/src/gateway";
+import { createEvaluator } from "../packages/core/src/evaluator";
test("Jev uses native state and validated boolean probabilities through real HTTP", async () => {
const state = {
@@ -10,17 +11,18 @@ test("Jev uses native state and validated boolean probabilities through real HTT
port: 0,
async fetch(request) {
expect(await request.json()).toEqual({
+ model: "typesafe-ai/jev",
state,
- questions: { useful: { type: "boolean", instructions: "Is the source useful?" } },
- providerOptions: {},
+ questions: { useful: { type: "noul", instructions: "Is the source useful?" } },
});
- return Response.json({ answers: { useful: { type: "boolean", probability: 0.8 } } });
+ return Response.json({ answers: { useful: { type: "noul", noul: 0.8 } } });
},
});
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: new AbortController().signal,
});
const result = await evaluator.evaluate({
@@ -42,13 +44,14 @@ test("transient failures retry within the shared request guard and never become
calls++;
return calls === 1
? Response.json({ error: "retry" }, { status: 503 })
- : Response.json({ answers: { q: { type: "boolean", probability: 0.2 } } });
+ : Response.json({ answers: { q: { type: "noul", noul: 0.2 } } });
},
});
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: new AbortController().signal,
requestLimit: 2,
});
@@ -86,7 +89,8 @@ test("cancellation of a rate-limited request prevents further attempts", async (
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: controller.signal,
});
const result = evaluator.evaluate({
@@ -115,13 +119,14 @@ test("Retry-After delays a retry before the provider can recover", async () => {
{ error: "rate limited" },
{ status: 429, headers: { "retry-after": "0.1" } },
)
- : Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });
+ : Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });
},
});
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: new AbortController().signal,
});
expect(
@@ -150,7 +155,8 @@ test("stalled HTTP attempts time out without exceeding the evaluator attempt lim
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: new AbortController().signal,
timeoutMs: 100,
});
@@ -175,13 +181,14 @@ test("authentication failure stops other in-flight and subsequent query requests
calls++;
if (calls === 1) return new Promise<Response>(() => {});
if (calls === 2) return Response.json({ error: "unauthorized" }, { status: 401 });
- return Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });
+ return Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });
},
});
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: new AbortController().signal,
timeoutMs: 1000,
});
@@ -225,7 +232,8 @@ test("authentication failure interrupts a sibling Retry-After wait", async () =>
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: new AbortController().signal,
});
const request = {
@@ -245,3 +253,75 @@ test("authentication failure interrupts a sibling Retry-After wait", async () =>
server.stop(true);
}
}, 2000);
+
+for (const status of [409, 422, 503]) {
+ test(`navigation HTTP ${status} preserves retry and split policy`, async () => {
+ let calls = 0;
+ const server = Bun.serve({
+ port: 0,
+ fetch() {
+ calls++;
+ return Response.json({ message: "synthetic failure" }, { status });
+ },
+ });
+ try {
+ const evaluator = createEvaluator({
+ provider: "openrouter",
+ apiKey: "fixture",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
+ signal: new AbortController().signal,
+ });
+ await expect(
+ evaluator.evaluate(
+ {
+ state: { source: "code" },
+ questions: {
+ q0: { type: "boolean", instructions: "Relevant?" },
+ q1: { type: "boolean", instructions: "Related?" },
+ },
+ },
+ { navigation: true },
+ ),
+ ).rejects.toMatchObject({ kind: "provider", splitEligible: status === 503 });
+ expect(calls).toBe(1);
+ expect(evaluator.requests).toBe(1);
+ } finally {
+ server.stop(true);
+ }
+ });
+}
+
+test("native HTTP-date Retry-After is honored before another attempt", async () => {
+ const retryAt = new Date(Math.ceil(Date.now() / 1000) * 1000 + 1000);
+ const arrivals: number[] = [];
+ const server = Bun.serve({
+ port: 0,
+ fetch() {
+ arrivals.push(Date.now());
+ return arrivals.length === 1
+ ? Response.json(
+ { message: "rate limited" },
+ { status: 429, headers: { "retry-after": retryAt.toUTCString() } },
+ )
+ : Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });
+ },
+ });
+ try {
+ const evaluator = createEvaluator({
+ provider: "typesafe",
+ apiKey: "fixture",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
+ signal: new AbortController().signal,
+ });
+ expect(
+ await evaluator.evaluate({
+ state: "code",
+ questions: { q: { type: "boolean", instructions: "Relevant?" } },
+ }),
+ ).toEqual({ q: 0.8 });
+ expect(arrivals.length).toBe(2);
+ expect(arrivals[1]!).toBeGreaterThanOrEqual(retryAt.getTime());
+ } finally {
+ server.stop(true);
+ }
+}, 5000);
diff --git a/test/installed.test.mjs b/test/installed.test.mjs
index fe58b21..77484f7 100644
--- a/test/installed.test.mjs
+++ b/test/installed.test.mjs
@@ -9,6 +9,7 @@ import {
mkdir,
mkdtemp,
readFile,
+ readdir,
realpath,
rm,
stat,
@@ -27,6 +28,20 @@ assert.ok(
binary && packageDirectory && expectedSkill,
"Run scripts/test-installed.sh in the Node-only container",
);
+// These independent expectations pin the public routes of the installed artifact.
+const providers = {
+ vercel: {
+ label: "Vercel AI Gateway",
+ url: "https://ai-gateway.vercel.sh/typesafe/v1/systemone",
+ model: "typesafe-ai/jev",
+ },
+ typesafe: { label: "TypeSafe", url: "https://api.typesafe.ai/v1/systemone", model: "jev-1.13.0" },
+ openrouter: {
+ label: "OpenRouter",
+ url: "https://openrouter.ai/api/v1/systemone",
+ model: "jev-1.13",
+ },
+};
const fixtureKey = "installed-http-fixture-key";
const forbidden = "INSTALLED_FIXTURE_IGNORED_CONTENT_MUST_NEVER_UPLOAD";
const query = "Find event recording implementations across the nested packages.";
@@ -40,6 +55,9 @@ const source = (branch, name) =>
async function context(t, mode = "healthy", executable = binary) {
let expectedQuery = query;
+ let expectedProvider = "vercel";
+ let expectedKey = fixtureKey;
+ const secrets = new Set([fixtureKey]);
const scratch = await mkdtemp(join(tmpdir(), "jg-installed-"));
const tree = join(scratch, "repository");
const home = join(scratch, "home");
@@ -83,14 +101,22 @@ async function context(t, mode = "healthy", executable = binary) {
await writeFile(join(tree, "hidden.skip.py"), `value = "${forbidden}"\n`);
await writeFile(join(tree, ".env"), `PRIVATE_VALUE=${forbidden}\n`);
await writeFile(join(tree, "unrelated.md"), "This is an unrelated gardening document.\n");
+ const credentialDirectory = join(config, "jevgrep");
+ const credentials = join(credentialDirectory, "credentials.json");
+ await mkdir(credentialDirectory, { mode: 0o700 });
+ await writeFile(credentials, JSON.stringify({ provider: "vercel", apiKey: fixtureKey }) + "\n", {
+ mode: 0o600,
+ });
server = createServer((request, response) => {
void (async () => {
assert.equal(request.method, "POST");
- assert.equal(request.url, "/v4/ai/evaluation-model");
- assert.equal(request.headers.authorization, `Bearer ${fixtureKey}`);
- assert.equal(request.headers["ai-model-id"], "typesafe-ai/jev");
- assert.equal(request.headers["ai-evaluation-model-specification-version"], "4");
- assert.equal(request.headers["ai-gateway-protocol-version"], "0.0.1");
+ const preset = providers[expectedProvider];
+ assert.equal(request.url, new URL(preset.url).pathname);
+ assert.equal(request.headers["x-jevgrep-original-url"], preset.url);
+ assert.ok(
+ request.headers.authorization === `Bearer ${expectedKey}`,
+ "Saved key must authenticate the request",
+ );
assert.match(request.headers["content-type"] ?? "", /^application\/json/);
const chunks = [];
let bytes = 0;
@@ -102,7 +128,8 @@ async function context(t, mode = "healthy", executable = binary) {
const raw = Buffer.concat(chunks).toString("utf8");
assert.ok(!raw.includes(forbidden), "Ignored/hidden source reached the provider");
const body = JSON.parse(raw);
- assert.deepEqual(Object.keys(body), ["state", "questions", "providerOptions"]);
+ assert.deepEqual(Object.keys(body).sort(), ["model", "questions", "state"]);
+ assert.equal(body.model, preset.model);
assert.equal(
typeof body.state,
"object",
@@ -111,11 +138,11 @@ async function context(t, mode = "healthy", executable = binary) {
assert.ok(body.state !== null && !Array.isArray(body.state));
assert.ok(Object.keys(body.questions).length > 0);
for (const question of Object.values(body.questions)) {
- assert.equal(question.type, "boolean");
+ assert.equal(question.type, "noul");
assert.ok(typeof question.instructions === "string" && question.instructions.length > 0);
}
assert.ok(requests.length < 256, "Synthetic search stopped making bounded forward progress");
- requests.push({ body, raw, receivedAt: performance.now() });
+ requests.push({ body, raw, provider: expectedProvider, receivedAt: performance.now() });
if (typeof mode === "function" && (await mode({ body, raw, tree, response }))) return;
if (mode === "rate-limit" && requests.length === 1) {
response.writeHead(429, { "content-type": "application/json", "retry-after": "1" });
@@ -187,9 +214,9 @@ async function context(t, mode = "healthy", executable = binary) {
response.end(
JSON.stringify({
answers: Object.fromEntries(
- ids.map((id, index) => [id, { type: "boolean", probability: probabilities[index] }]),
+ ids.map((id, index) => [id, { type: "noul", noul: probabilities[index] }]),
),
- usage: { inputTokens: 1, outputTokens: 1 },
+ usage: { input_tokens: 1, output_tokens: 1 },
warnings: [{ type: "other", message: "installed-fixture-warning" }],
}),
);
@@ -209,10 +236,11 @@ async function context(t, mode = "healthy", executable = binary) {
XDG_CONFIG_HOME: config,
XDG_CACHE_HOME: cache,
TMPDIR: scratch,
- AI_GATEWAY_API_KEY: fixtureKey,
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.address().port}/v4/ai`,
+ NODE_OPTIONS: `--import=${process.env.JEVGREP_TEST_PROVIDER_PRELOAD ?? new URL("./fixtures/provider-route.mjs", import.meta.url).href}`,
+ JEVGREP_TEST_PROVIDER_ORIGIN: `http://127.0.0.1:${server.address().port}`,
};
- const run = async (args, overrides = {}, head = false) => {
+ const run = async (args, overrides = {}, head = false, input) => {
+ if (input?.trim()) secrets.add(input.trim());
const result = await new Promise((resolve, reject) => {
const child = spawn(
head ? "bash" : executable === binary ? binary : process.execPath,
@@ -225,10 +253,16 @@ async function context(t, mode = "healthy", executable = binary) {
cwd: tree,
detached: true,
env: { ...env, ...overrides },
- stdio: ["ignore", "pipe", "pipe"],
+ stdio: [input === undefined ? "ignore" : "pipe", "pipe", "pipe"],
},
);
children.add(child);
+ if (input !== undefined) {
+ child.stdin.on("error", (error) => {
+ if (error.code !== "EPIPE") reject(error);
+ });
+ child.stdin.end(input);
+ }
const stdout = [],
stderr = [];
let outputBytes = 0;
@@ -267,7 +301,8 @@ async function context(t, mode = "healthy", executable = binary) {
"",
"All application output, including SDK warnings, belongs on stdout",
);
- assert.ok(!result.stdout.includes(fixtureKey));
+ for (const secret of secrets)
+ assert.ok(!result.stdout.includes(secret), "Credential leaked in stdout");
assert.ok(!result.stdout.includes("installed-fixture-warning"));
assert.ok(!result.stdout.includes(forbidden));
return result;
@@ -275,7 +310,18 @@ async function context(t, mode = "healthy", executable = binary) {
return {
run,
tree,
+ credentials,
+ config,
+ credentialDirectory,
requests,
+ expectProvider(provider, key = fixtureKey) {
+ expectedProvider = provider;
+ expectedKey = key;
+ secrets.add(key);
+ },
+ async removeCredentials() {
+ await rm(credentials, { force: true });
+ },
set query(value) {
expectedQuery = value;
},
@@ -352,7 +398,8 @@ test("installed local commands match the package without credentials", async (t)
t.diagnostic(
`Runtime ${process.version} ${process.platform}/${process.arch}; installed ${metadata.name}@${metadata.version}`,
);
- const noCredentials = { AI_GATEWAY_API_KEY: "" };
+ await fixture.removeCredentials();
+ const noCredentials = {};
const help = await fixture.run(["--help"], noCredentials);
assert.equal(help.code, 0, help.stdout);
assert.match(help.stdout, /Usage: jg /);
@@ -363,11 +410,12 @@ test("installed local commands match the package without credentials", async (t)
await readFile(join(packageDirectory, "dist/skills/jevgrep/SKILL.md"), "utf8"),
await readFile(expectedSkill, "utf8"),
);
- assert.equal(fixture.requests.length, 0, "Local commands must not contact Gateway");
+ assert.equal(fixture.requests.length, 0, "Local commands must not contact a provider");
});
-test("skill command delegates installation to npx without Gateway credentials", async (t) => {
+test("skill command delegates installation to npx without credentials", async (t) => {
const fixture = await context(t);
+ await fixture.removeCredentials();
const bin = join(fixture.tree, "installer-bin");
await mkdir(bin);
await symlink(process.execPath, join(bin, "node"));
@@ -375,7 +423,7 @@ test("skill command delegates installation to npx without Gateway credentials",
await symlink(new URL("./fixtures/skill-installer.mjs", import.meta.url), npx);
const result = await fixture.run(
["skill", "--agent", "codex", "--agent", "claude-code", "--global", "--yes"],
- { PATH: bin, AI_GATEWAY_API_KEY: "" },
+ { PATH: bin },
);
assert.equal(result.code, 0, result.stdout);
assert.match(result.stdout, /Installer completed/);
@@ -396,7 +444,6 @@ test("skill command delegates installation to npx without Gateway credentials",
assert.equal(fixture.requests.length, 0);
const failed = await fixture.run(["skill"], {
PATH: bin,
- AI_GATEWAY_API_KEY: "",
JEVGREP_INSTALLER_EXIT: "7",
});
assert.equal(failed.code, 7, failed.stdout);
@@ -409,7 +456,7 @@ test("skill command delegates installation to npx without Gateway credentials",
"jevgrep",
]);
await rm(npx);
- const unavailable = await fixture.run(["skill"], { PATH: bin, AI_GATEWAY_API_KEY: "" });
+ const unavailable = await fixture.run(["skill"], { PATH: bin });
assert.equal(unavailable.code, 1);
assert.match(unavailable.stdout, /requires npx/);
});
@@ -494,12 +541,15 @@ test("warm cache reuses identical requests, no-cache bypasses reuse, and edits i
assert.ok(!changed.stdout.includes("py-evidence-alpha:"));
assert.ok(
fixture.requests.slice(before).some(({ raw }) => raw.includes("edited-alpha-evidence:")),
- "Changed source must reach Gateway instead of stale cache evidence",
+ "Changed source must reach the provider instead of stale cache evidence",
);
before = fixture.requests.length;
assert.equal((await fixture.run([query, fixture.tree])).stdout, changed.stdout);
assertCachedRequestsAreReused(fixture.requests, before);
- const cleared = await fixture.run(["cache", "clear"], { AI_GATEWAY_API_KEY: "" });
+ const saved = await readFile(fixture.credentials);
+ await fixture.removeCredentials();
+ const cleared = await fixture.run(["cache", "clear"]);
+ await writeFile(fixture.credentials, saved, { mode: 0o600 });
assert.equal(cleared.code, 0, cleared.stdout);
before = fixture.requests.length;
assert.equal((await fixture.run([query, fixture.tree])).code, 0);
@@ -512,7 +562,8 @@ test("doctor uses the installed SDK while missing credentials fail cleanly", asy
assert.equal(doctor.code, 0, doctor.stdout);
assert.ok(fixture.requests.length > 0);
const before = fixture.requests.length;
- const missing = await fixture.run([query, fixture.tree], { AI_GATEWAY_API_KEY: "" });
+ await fixture.removeCredentials();
+ const missing = await fixture.run([query, fixture.tree]);
assert.equal(missing.code, 1, missing.stdout);
assert.ok(missing.stdout.trim().length > 0);
assert.equal(fixture.requests.length, before);
@@ -766,7 +817,8 @@ test("missing or corrupt packaged Python assets fail closed without downloads",
!fixture.requests.some(({ body }) => body.state.declarations),
"Unavailable parser assets must not fabricate declaration evidence",
);
- assert.equal((await fixture.run(["--help"], { AI_GATEWAY_API_KEY: "" })).code, 0);
+ await fixture.removeCredentials();
+ assert.equal((await fixture.run(["--help"])).code, 0);
await rm(copy, { recursive: true, force: true });
}
});
@@ -786,7 +838,7 @@ for (const mutation of ["changed", "ignored"])
response.end(
JSON.stringify({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),
),
}),
);
@@ -826,7 +878,7 @@ test("installed queued freshness withholds excluded source uploads", async (t) =
response.end(
JSON.stringify({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.1 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.1 }]),
),
}),
);
@@ -845,3 +897,166 @@ test("installed queued freshness withholds excluded source uploads", async (t) =
assert.match(result.stdout, /incomplete/);
assert.ok(!result.stdout.includes("QUEUED_INSTALLED_SENTINEL"));
});
+
+for (const [provider, preset] of Object.entries(providers))
+ test(`installed saved-provider journey: ${provider} auth doctor search cache replacement`, async (t) => {
+ const fixture = await context(t);
+ await fixture.removeCredentials();
+ const key = `installed-${provider}-saved-key`;
+ fixture.expectProvider(provider, key);
+ const auth = await fixture.run(
+ ["auth", "--provider", provider, "--stdin"],
+ {},
+ false,
+ key + "\n",
+ );
+ assert.equal(auth.code, 0, auth.stdout);
+ assert.ok(auth.stdout.includes(preset.label));
+ assert.match(auth.stdout, /doctor/);
+ assert.equal(fixture.requests.length, 0, "Auth must not contact any provider");
+ assert.deepEqual(JSON.parse(await readFile(fixture.credentials, "utf8")), {
+ provider,
+ apiKey: key,
+ });
+ assert.equal((await stat(fixture.credentials)).mode & 0o777, 0o600);
+ assert.equal((await stat(fixture.credentialDirectory)).mode & 0o777, 0o700);
+ const doctor = await fixture.run(["doctor"]);
+ assert.equal(doctor.code, 0, doctor.stdout);
+ assert.ok(doctor.stdout.includes(preset.label));
+ assert.equal(fixture.requests.length, 1);
+ assert.deepEqual(Object.keys(fixture.requests[0].body.questions), ["relevant"]);
+ assert.ok(
+ !fixture.requests[0].raw.includes("py-evidence-"),
+ "Doctor must use only synthetic source",
+ );
+ const cold = await fixture.run([query]);
+ complete(cold);
+ const before = fixture.requests.length;
+ const warm = await fixture.run([query]);
+ complete(warm);
+ assert.equal(warm.stdout, cold.stdout);
+ assertCachedRequestsAreReused(fixture.requests, before);
+
+ const replacement = provider === "vercel" ? "typesafe" : "vercel";
+ const replacementKey = `installed-${replacement}-replacement-key`;
+ const beforeAuth = fixture.requests.length;
+ const replaced = await fixture.run(
+ ["auth", "--provider", replacement, "--stdin"],
+ {},
+ false,
+ replacementKey + "\n",
+ );
+ assert.equal(replaced.code, 0, replaced.stdout);
+ assert.equal(fixture.requests.length, beforeAuth);
+ assert.deepEqual(JSON.parse(await readFile(fixture.credentials, "utf8")), {
+ provider: replacement,
+ apiKey: replacementKey,
+ });
+ fixture.expectProvider(replacement, replacementKey);
+ const search = await fixture.run([query]);
+ complete(search);
+ assert.equal(search.stdout, cold.stdout);
+ assert.ok(
+ fixture.requests.length > beforeAuth,
+ "Provider replacement must not reuse another provider's cache",
+ );
+ assert.ok(
+ fixture.requests.slice(beforeAuth).every((request) => request.provider === replacement),
+ );
+ fixture.mode = "unauthorized";
+ const beforeFailure = fixture.requests.length;
+ const failed = await fixture.run(["doctor"]);
+ assert.equal(failed.code, 1, failed.stdout);
+ assert.equal(
+ fixture.requests.length,
+ beforeFailure + 1,
+ "Authentication failure must not retry or fall back",
+ );
+ assert.ok(failed.stdout.includes(providers[replacement].label));
+ assert.equal(fixture.requests.at(-1).provider, replacement);
+ });
+
+test("installed legacy credentials use Vercel without rewriting saved bytes", async (t) => {
+ const fixture = await context(t);
+ const original = ' { "apiKey": "' + fixtureKey + '" }\n';
+ await writeFile(fixture.credentials, original);
+ const before = await stat(fixture.credentials);
+ const doctor = await fixture.run(["doctor"]);
+ assert.equal(doctor.code, 0, doctor.stdout);
+ assert.ok(doctor.stdout.includes(providers.vercel.label));
+ complete(await fixture.run([query]));
+ assert.ok(fixture.requests.every((request) => request.provider === "vercel"));
+ assert.equal(await readFile(fixture.credentials, "utf8"), original);
+ const after = await stat(fixture.credentials);
+ assert.equal(after.mtimeMs, before.mtimeMs);
+ assert.equal(after.ino, before.ino);
+});
+
+test("installed saved credentials defeat conflicting environment and environment-only auth fails", async (t) => {
+ const fixture = await context(t);
+ const conflicts = {
+ AI_GATEWAY_API_KEY: "environment-gateway-key",
+ TYPESAFE_API_KEY: "environment-typesafe-key",
+ TYPESAFE_AI_API_KEY: "environment-sdk-typesafe-key",
+ OPENROUTER_API_KEY: "environment-openrouter-key",
+ AI_GATEWAY_BASE_URL: "http://127.0.0.1:1/forbidden",
+ TYPESAFE_BASE_URL: "http://127.0.0.1:1/forbidden",
+ OPENROUTER_BASE_URL: "http://127.0.0.1:1/forbidden",
+ AI_GATEWAY_MODEL: "environment-model",
+ TYPESAFE_MODEL: "environment-model",
+ OPENROUTER_MODEL: "environment-model",
+ };
+ for (const provider of Object.keys(providers)) {
+ await writeFile(fixture.credentials, JSON.stringify({ provider, apiKey: fixtureKey }));
+ fixture.expectProvider(provider);
+ const doctor = await fixture.run(["doctor"], conflicts);
+ assert.equal(doctor.code, 0, doctor.stdout);
+ assert.ok(doctor.stdout.includes(providers[provider].label));
+ complete(await fixture.run([query, "--no-cache"], conflicts));
+ }
+ await fixture.removeCredentials();
+ const before = fixture.requests.length;
+ for (const args of [["doctor"], [query]]) {
+ const result = await fixture.run(args, conflicts);
+ assert.equal(result.code, 1, result.stdout);
+ assert.match(result.stdout, /jg auth/);
+ assert.ok(!result.stdout.includes("environment-"));
+ }
+ assert.equal(fixture.requests.length, before);
+});
+
+test("installed invalid saved providers fail before HTTP without rewriting credentials", async (t) => {
+ const fixture = await context(t);
+ for (const provider of [null, "", "unknown", false, 0, {}, []]) {
+ const original = JSON.stringify({ provider, apiKey: fixtureKey }) + "\n";
+ await writeFile(fixture.credentials, original);
+ for (const args of [["doctor"], [query]]) {
+ const result = await fixture.run(args);
+ assert.equal(result.code, 1, result.stdout);
+ assert.match(result.stdout, /auth/);
+ assert.equal(await readFile(fixture.credentials, "utf8"), original);
+ }
+ }
+ assert.equal(fixture.requests.length, 0);
+});
+
+test("installed invalid auth preserves saved bytes and leaves no temporary credentials", async (t) => {
+ const fixture = await context(t);
+ const original = await readFile(fixture.credentials, "utf8");
+ for (const [args, input] of [
+ [["auth", "--stdin"], "invalid-provider-key\n"],
+ [["auth", "--provider", "typesafe"], undefined],
+ [["auth", "--provider", "unknown", "--stdin"], "invalid-provider-key\n"],
+ [["auth", "--provider", "vercel", "--stdin"], "\n"],
+ [["auth", "--provider", "typesafe", "--stdin"], "key with spaces\n"],
+ [["auth", "--provider", "openrouter", "--stdin"], "x".repeat(8193)],
+ [["doctor", "--provider", "vercel"], undefined],
+ [[query, "--provider", "typesafe"], undefined],
+ ]) {
+ const result = await fixture.run(args, {}, false, input);
+ assert.equal(result.code, 1, result.stdout);
+ assert.equal(await readFile(fixture.credentials, "utf8"), original);
+ assert.deepEqual(await readdir(fixture.credentialDirectory), ["credentials.json"]);
+ }
+ assert.equal(fixture.requests.length, 0);
+});
diff --git a/test/reference/disconnect.test.ts b/test/reference/disconnect.test.ts
index c9ee48a..c3663be 100644
--- a/test/reference/disconnect.test.ts
+++ b/test/reference/disconnect.test.ts
@@ -1,3 +1,4 @@
+import { referenceTransport, wireResponse } from "./transport";
import { expect } from "bun:test";
import { createServer } from "node:http";
import { resolve } from "node:path";
@@ -5,10 +6,19 @@ import { testIfDocker as test } from "../helpers/docker";
async function disconnected(reference: boolean) {
const bodies: string[] = [];
+ const transport = await referenceTransport();
const server = createServer(async (request, response) => {
const chunks: Buffer[] = [];
for await (const chunk of request) chunks.push(Buffer.from(chunk));
- const body = JSON.parse(Buffer.concat(chunks).toString());
+ const body = transport.decode(
+ {
+ url: `http://localhost${request.url}`,
+ method: request.method,
+ headers: new Headers(request.headers as Record<string, string>),
+ },
+ JSON.parse(Buffer.concat(chunks).toString()),
+ !reference,
+ );
bodies.push(JSON.stringify(body));
if (bodies.length === 1) {
response.destroy();
@@ -16,11 +26,16 @@ async function disconnected(reference: boolean) {
}
response.writeHead(200, { "content-type": "application/json" });
response.end(
- JSON.stringify({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.05 }]),
+ JSON.stringify(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.05 }]),
+ ),
+ },
+ !reference,
),
- }),
+ ),
);
});
await new Promise<void>((done) => server.listen(0, "127.0.0.1", done));
@@ -34,11 +49,7 @@ async function disconnected(reference: boolean) {
? ["node", "/opt/jevgrep-reference.mjs", "--query", query, "--root", root]
: ["node", resolve("apps/cli/dist/bin/index.js"), query, root, "--no-cache"],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${address.port}/v4/ai`,
- },
+ env: transport.env(!reference, `http://127.0.0.1:${address.port}`),
stdout: "pipe",
stderr: "pipe",
timeout: 20000,
@@ -51,6 +62,7 @@ async function disconnected(reference: boolean) {
]);
return { code, stdout, stderr, bodies: bodies.sort() };
} finally {
+ await transport.cleanup();
server.closeAllConnections();
await new Promise<void>((done) => server.close(() => done()));
}
diff --git a/test/reference/discovery-parity.test.ts b/test/reference/discovery-parity.test.ts
index 3e709db..fbca4df 100644
--- a/test/reference/discovery-parity.test.ts
+++ b/test/reference/discovery-parity.test.ts
@@ -1,10 +1,12 @@
+import { referenceTransport, wireResponse } from "./transport";
+import { routeProviderFetch } from "../fixtures/provider-route.mjs";
import { expect } from "bun:test";
import { mkdtemp, mkdir, writeFile, rm, chmod } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { testIfDocker } from "../helpers/docker";
import { retrieve } from "../../packages/core/src/retrieve";
-import { createEvaluator, EvaluationFailure } from "../../packages/core/src/gateway";
+import { createEvaluator, EvaluationFailure } from "../../packages/core/src/evaluator";
type Body = {
state: {
@@ -38,12 +40,11 @@ async function trajectory(
[];
const requests: Body[] = [];
let failedGroup = false;
+ const transport = await referenceTransport();
const server = Bun.serve({
port: 0,
async fetch(request) {
- expect(new URL(request.url).pathname).toBe("/v4/ai/evaluation-model");
- expect(request.headers.get("ai-model-id")).toBe("typesafe-ai/jev");
- const body = (await request.json()) as Body;
+ const body = transport.decode(request, await request.json(), !reference) as Body;
if (body.state.items || body.state.preview) requests.push(body);
if (body.state.items && fault) {
if (fault === "rate-limit")
@@ -62,25 +63,30 @@ async function trajectory(
);
}
}
- const response = Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id, i) => {
- const item = body.state.items?.[i];
- const probability =
- item?.kind === "directory"
- ? body.state.relationAnchor &&
- item.path.split("/").some((segment) => segment.startsWith("related")) &&
- !item.path.includes("-cap") &&
- !item.path.includes("-escaped")
- ? 0.9
- : 0.5
- : item?.path.startsWith("Anchor.")
- ? 0.9
- : 0.25;
- return [id, { type: "boolean", probability }];
- }),
+ const response = Response.json(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id, i) => {
+ const item = body.state.items?.[i];
+ const probability =
+ item?.kind === "directory"
+ ? body.state.relationAnchor &&
+ item.path.split("/").some((segment) => segment.startsWith("related")) &&
+ !item.path.includes("-cap") &&
+ !item.path.includes("-escaped")
+ ? 0.9
+ : 0.5
+ : item?.path.startsWith("Anchor.")
+ ? 0.9
+ : 0.25;
+ return [id, { type: "boolean", probability }];
+ }),
+ ),
+ },
+ !reference,
),
- });
+ );
if (
reverseRelationCompletion &&
body.state.relationAnchor &&
@@ -104,11 +110,7 @@ async function trajectory(
const child = Bun.spawn(
["node", "/opt/jevgrep-reference.mjs", "--root", root, "--query", query],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,
- },
+ env: transport.env(!reference, `http://127.0.0.1:${server.port}`),
stdout: "pipe",
stderr: "pipe",
},
@@ -126,8 +128,9 @@ async function trajectory(
const result = await retrieve(
{ root, query, signal },
createEvaluator({
+ provider: "vercel",
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}/v4/ai`,
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal,
}),
);
@@ -136,6 +139,7 @@ async function trajectory(
return requests;
} finally {
server.stop(true);
+ await transport.cleanup();
}
}
diff --git a/test/reference/evidence-order.test.ts b/test/reference/evidence-order.test.ts
index ce64fdb..528d6fd 100644
--- a/test/reference/evidence-order.test.ts
+++ b/test/reference/evidence-order.test.ts
@@ -1,3 +1,4 @@
+import { referenceTransport, wireResponse } from "./transport";
import { expect } from "bun:test";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { join, resolve } from "node:path";
@@ -20,10 +21,11 @@ async function run(root: string, production: boolean) {
const requests: Record<string, RequestBody> = {};
const pending = new Map<string, () => void>();
const released: string[] = [];
+ const transport = await referenceTransport();
const server = Bun.serve({
port: 0,
async fetch(request) {
- const body = (await request.json()) as RequestBody;
+ const body = transport.decode(request, await request.json(), production) as RequestBody;
const phase = body.state.items
? "discovery"
: body.state.declarations
@@ -49,11 +51,16 @@ async function run(root: string, production: boolean) {
}
});
}
- return Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ return Response.json(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ ),
+ },
+ production,
),
- });
+ );
},
});
try {
@@ -75,11 +82,7 @@ async function run(root: string, production: boolean) {
"trace event recording",
],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,
- },
+ env: transport.env(production, `http://127.0.0.1:${server.port}`),
stdout: "pipe",
stderr: "pipe",
},
@@ -95,6 +98,7 @@ async function run(root: string, production: boolean) {
return { requests, stdout };
} finally {
server.stop(true);
+ await transport.cleanup();
}
}
diff --git a/test/reference/fallback-comments.test.ts b/test/reference/fallback-comments.test.ts
index 45a1730..2769f28 100644
--- a/test/reference/fallback-comments.test.ts
+++ b/test/reference/fallback-comments.test.ts
@@ -1,3 +1,4 @@
+import { referenceTransport, wireResponse } from "./transport";
import { expect } from "bun:test";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
@@ -13,29 +14,36 @@ testIfDocker(
const query = "Find source behavior";
const requests: string[][] = [[], []];
let arm = 0;
+ const transport = await referenceTransport();
const server = Bun.serve({
port: 0,
async fetch(request) {
- const body = (await request.json()) as {
+ const body = transport.decode(request, await request.json(), arm !== 0) as {
state: { declarations?: Array<{ startLine: number }>; selectedEvidence?: unknown };
questions: Record<string, unknown>;
};
requests[arm]!.push(JSON.stringify(body));
- return Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id, index) => [
- id,
- {
- type: "boolean",
- probability: body.state.declarations
- ? !body.state.selectedEvidence && body.state.declarations[index]?.startLine === 1
- ? 0.9
- : 0
- : 0.9,
- },
- ]),
+ return Response.json(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id, index) => [
+ id,
+ {
+ type: "boolean",
+ probability: body.state.declarations
+ ? !body.state.selectedEvidence &&
+ body.state.declarations[index]?.startLine === 1
+ ? 0.9
+ : 0
+ : 0.9,
+ },
+ ]),
+ ),
+ },
+ arm !== 0,
),
- });
+ );
},
});
try {
@@ -53,11 +61,7 @@ testIfDocker(
"--no-cache",
],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,
- },
+ env: transport.env(arm !== 0, `http://127.0.0.1:${server.port}`),
stdout: "pipe",
stderr: "pipe",
},
@@ -77,6 +81,7 @@ testIfDocker(
expect(outputs[1]).toBe(outputs[0]);
} finally {
server.stop(true);
+ await transport.cleanup();
await rm(root, { recursive: true, force: true });
}
},
diff --git a/test/reference/provider-parity.test.ts b/test/reference/provider-parity.test.ts
new file mode 100644
index 0000000..cef281e
--- /dev/null
+++ b/test/reference/provider-parity.test.ts
@@ -0,0 +1,32 @@
+import { expect } from "bun:test";
+import { readFile } from "node:fs/promises";
+import { testIfDocker } from "../helpers/docker";
+import { replay } from "./replay";
+import { providers, type Provider } from "./transport";
+
+for (const provider of Object.keys(providers) as Provider[]) {
+ testIfDocker(
+ `${provider} production preserves the frozen request multiset and complete packet`,
+ async () => {
+ const expected = JSON.parse(await readFile("test/reference/corpus.json", "utf8"));
+ const result = await replay("healthy", true, provider);
+ expect(result.code).toBe(0);
+ expect(result.stderr).toBe("");
+ expect(result.requests).toEqual(expected.requests);
+ expect(result.stdout).toBe(expected.stdout);
+ },
+ 120_000,
+ );
+ for (const mode of ["missing", "invalid"] as const) {
+ testIfDocker(
+ `${provider} ${mode} answers preserve incomplete discovery`,
+ async () => {
+ const result = await replay(mode, true, provider);
+ expect(result.code).toBe(2);
+ expect(result.stderr).toBe("");
+ expect(result.stdout).toContain("Jevgrep: 0 relevant files; discovery incomplete.");
+ },
+ 120_000,
+ );
+ }
+}
diff --git a/test/reference/python-parity.test.ts b/test/reference/python-parity.test.ts
index 0231514..5e6437f 100644
--- a/test/reference/python-parity.test.ts
+++ b/test/reference/python-parity.test.ts
@@ -1,3 +1,4 @@
+import { referenceTransport, wireResponse } from "./transport";
import { expect } from "bun:test";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
@@ -40,29 +41,35 @@ for (const [name, source] of [
const query = "Find target behavior";
const requests: string[][] = [[], []];
let arm = 0;
+ const transport = await referenceTransport();
const server = Bun.serve({
port: 0,
async fetch(request) {
- const body = (await request.json()) as {
+ const body = transport.decode(request, await request.json(), arm !== 0) as {
questions: Record<string, unknown>;
state?: { declarations?: { name: string }[] };
};
requests[arm]!.push(JSON.stringify(body));
- return Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id, index) => [
- id,
- {
- type: "boolean",
- probability:
- name === "CR-only method selected" &&
- body.state?.declarations?.[index]?.name.endsWith(".context")
- ? 0.1
- : 0.9,
- },
- ]),
+ return Response.json(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id, index) => [
+ id,
+ {
+ type: "boolean",
+ probability:
+ name === "CR-only method selected" &&
+ body.state?.declarations?.[index]?.name.endsWith(".context")
+ ? 0.1
+ : 0.9,
+ },
+ ]),
+ ),
+ },
+ arm !== 0,
),
- });
+ );
},
});
try {
@@ -80,11 +87,7 @@ for (const [name, source] of [
"--no-cache",
],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,
- },
+ env: transport.env(arm !== 0, `http://127.0.0.1:${server.port}`),
stdout: "pipe",
stderr: "pipe",
},
@@ -102,6 +105,7 @@ for (const [name, source] of [
expect(outputs[1]).toBe(outputs[0]);
} finally {
server.stop(true);
+ await transport.cleanup();
await rm(root, { recursive: true, force: true });
}
},
diff --git a/test/reference/replay.ts b/test/reference/replay.ts
index f3c2d3c..05b6a68 100644
--- a/test/reference/replay.ts
+++ b/test/reference/replay.ts
@@ -1,17 +1,16 @@
-import { expect } from "bun:test";
+import { referenceTransport, wireResponse, type Provider } from "./transport";
import { resolve } from "node:path";
export async function replay(
mode: "healthy" | "missing" | "invalid" = "healthy",
production = false,
+ provider: Provider = "vercel",
) {
const requests: unknown[] = [];
+ const transport = await referenceTransport(provider, "reference-fixture");
const server = Bun.serve({
port: 0,
async fetch(request) {
- expect(new URL(request.url).pathname).toBe("/v4/ai/evaluation-model");
- expect(request.headers.get("ai-model-id")).toBe("typesafe-ai/jev");
- expect(request.headers.get("authorization")).toBe("Bearer reference-fixture");
- const body = (await request.json()) as {
+ const body = transport.decode(request, await request.json(), production) as {
state: {
items?: Array<{ path: string }>;
path?: string;
@@ -22,34 +21,44 @@ export async function replay(
questions: Record<string, unknown>;
};
requests.push(body);
- if (mode === "missing") return Response.json({ answers: {} });
+ if (mode === "missing") return Response.json(wireResponse({ answers: {} }, production));
if (mode === "invalid")
- return Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 2 }]),
- ),
- });
- return Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id, i) => [
- id,
+ return Response.json(
+ wireResponse(
{
- type: "boolean",
- probability:
- body.state.items?.[i]?.path === "unrelated.md"
- ? 0.05
- : body.state.items?.[i]?.path === "src/backend" && !body.state.relationAnchor
- ? 0.1
- : body.state.declarations &&
- !body.state.selectedEvidence &&
- body.state.path !== "src/telemetry.ts"
- ? 0.4
- : 0.9,
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 2 }]),
+ ),
},
- ]),
+ production,
+ ),
+ );
+ return Response.json(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id, i) => [
+ id,
+ {
+ type: "boolean",
+ probability:
+ body.state.items?.[i]?.path === "unrelated.md"
+ ? 0.05
+ : body.state.items?.[i]?.path === "src/backend" && !body.state.relationAnchor
+ ? 0.1
+ : body.state.declarations &&
+ !body.state.selectedEvidence &&
+ body.state.path !== "src/telemetry.ts"
+ ? 0.4
+ : 0.9,
+ },
+ ]),
+ ),
+ warnings: [{ type: "other", message: "fixture warning" }],
+ },
+ production,
),
- warnings: [{ type: "other", message: "fixture warning" }],
- });
+ );
},
});
try {
@@ -71,11 +80,7 @@ export async function replay(
"research how telemetry records event names",
],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "reference-fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,
- },
+ env: transport.env(production, `http://127.0.0.1:${server.port}`),
stdout: "pipe",
stderr: "pipe",
},
@@ -93,5 +98,6 @@ export async function replay(
};
} finally {
server.stop(true);
+ await transport.cleanup();
}
}
diff --git a/test/reference/repository-context-parity.test.ts b/test/reference/repository-context-parity.test.ts
index 354682b..db4d331 100644
--- a/test/reference/repository-context-parity.test.ts
+++ b/test/reference/repository-context-parity.test.ts
@@ -1,3 +1,4 @@
+import { referenceTransport, wireResponse } from "./transport";
import { expect } from "bun:test";
import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";
import { join, resolve } from "node:path";
@@ -11,11 +12,11 @@ async function run(
largeAnchor = false,
) {
const requests: unknown[] = [];
+ const transport = await referenceTransport();
const server = Bun.serve({
port: 0,
async fetch(request) {
- expect(new URL(request.url).pathname).toBe("/v4/ai/evaluation-model");
- const body = (await request.json()) as {
+ const body = transport.decode(request, await request.json(), production) as {
state: {
items?: Array<{ path: string; kind: string }>;
relationAnchor?: unknown;
@@ -25,35 +26,40 @@ async function run(
questions: Record<string, unknown>;
};
requests.push(body);
- return Response.json({
- answers: Object.fromEntries(
- Object.keys(body.questions).map((id, i) => [
- id,
- {
- type: "boolean",
- probability: body.state.items
- ? largeAnchor
- ? body.state.items[i]?.kind === "directory"
- ? body.state.relationAnchor
- ? 0.9
- : 0.1
- : body.state.items[i]?.path === "Anchor.ts"
- ? 0.9
- : 0.1
- : 0.9
- : body.state.declarations
- ? !body.state.selectedEvidence &&
- body.state.declarations[i]?.name === name &&
- body.state.declarations[i]?.startLine === startLine
- ? 0.9
- : 0.1
- : id === "test"
- ? 0.9
- : 0.1,
- },
- ]),
+ return Response.json(
+ wireResponse(
+ {
+ answers: Object.fromEntries(
+ Object.keys(body.questions).map((id, i) => [
+ id,
+ {
+ type: "boolean",
+ probability: body.state.items
+ ? largeAnchor
+ ? body.state.items[i]?.kind === "directory"
+ ? body.state.relationAnchor
+ ? 0.9
+ : 0.1
+ : body.state.items[i]?.path === "Anchor.ts"
+ ? 0.9
+ : 0.1
+ : 0.9
+ : body.state.declarations
+ ? !body.state.selectedEvidence &&
+ body.state.declarations[i]?.name === name &&
+ body.state.declarations[i]?.startLine === startLine
+ ? 0.9
+ : 0.1
+ : id === "test"
+ ? 0.9
+ : 0.1,
+ },
+ ]),
+ ),
+ },
+ production,
),
- });
+ );
},
});
try {
@@ -62,11 +68,7 @@ async function run(
? ["node", resolve("apps/cli/dist/bin/index.js"), "large regression", root, "--no-cache"]
: ["node", "/opt/jevgrep-reference.mjs", "--root", root, "--query", "large regression"],
{
- env: {
- PATH: process.env.PATH,
- AI_GATEWAY_API_KEY: "fixture",
- AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.port}/v4/ai`,
- },
+ env: transport.env(production, `http://127.0.0.1:${server.port}`),
stdout: "pipe",
stderr: "pipe",
},
@@ -82,6 +84,7 @@ async function run(
return { requests: requests.map((body) => JSON.stringify(body)).sort(), stdout };
} finally {
server.stop(true);
+ await transport.cleanup();
}
}
testIfDocker(
diff --git a/test/reference/selection-replay.ts b/test/reference/selection-replay.ts
index ce678b7..abe282c 100644
--- a/test/reference/selection-replay.ts
+++ b/test/reference/selection-replay.ts
@@ -7,7 +7,7 @@ import {
contextWindows,
} from "../../evals/implementation/swebench/source-method-windows-spike";
import { completeSourceFragments } from "../../evals/implementation/swebench/complete-source-v64-spike";
-import type { EvaluationRequest } from "../../packages/core/src/gateway";
+import type { EvaluationRequest } from "../../packages/core/src/evaluator";
import type { Declaration } from "../../packages/core/src/requests";
import type { Range } from "../../packages/core/src/source";
diff --git a/test/reference/selection.test.ts b/test/reference/selection.test.ts
index 0190c77..bb09ec0 100644
--- a/test/reference/selection.test.ts
+++ b/test/reference/selection.test.ts
@@ -1,6 +1,6 @@
import { expect, test } from "bun:test";
import { selectFile } from "../../packages/core/src/selection";
-import type { EvaluationRequest } from "../../packages/core/src/gateway";
+import type { EvaluationRequest } from "../../packages/core/src/evaluator";
import type { Declaration } from "../../packages/core/src/requests";
import { replaySelection } from "./selection-replay";
diff --git a/test/reference/transport.test.ts b/test/reference/transport.test.ts
new file mode 100644
index 0000000..d0ea332
--- /dev/null
+++ b/test/reference/transport.test.ts
@@ -0,0 +1,54 @@
+import { expect, test } from "bun:test";
+import { readFile } from "node:fs/promises";
+import { referenceTransport, providers, type Provider } from "./transport";
+
+test("wire normalization retains every corpus field and rejects unknown envelope and question fields", async () => {
+ const corpus = JSON.parse(await readFile("test/reference/corpus.json", "utf8"));
+ for (const provider of Object.keys(providers) as Provider[]) {
+ const transport = await referenceTransport(provider);
+ try {
+ const preset = providers[provider];
+ const request = new Request(preset.url, {
+ method: "POST",
+ headers: {
+ authorization: "Bearer fixture",
+ "x-jevgrep-original-url": preset.url,
+ },
+ });
+ for (const serialized of corpus.requests) {
+ const original = JSON.parse(serialized);
+ const historical = new Request("http://localhost/v4/ai/evaluation-model", {
+ method: "POST",
+ headers: { authorization: "Bearer fixture", "ai-model-id": "typesafe-ai/jev" },
+ });
+ expect(JSON.stringify(transport.decode(historical, original, false))).toBe(serialized);
+ const wire = {
+ model: preset.model,
+ state: original.state,
+ questions: Object.fromEntries(
+ Object.entries(original.questions).map(([id, question]) => [
+ id,
+ { ...(question as object), type: "noul" },
+ ]),
+ ),
+ };
+ expect(JSON.stringify(transport.decode(request, wire, true))).toBe(serialized);
+ expect(() => transport.decode(request, { ...wire, extra: true }, true)).toThrow();
+ const id = Object.keys(wire.questions)[0]!;
+ expect(() =>
+ transport.decode(
+ request,
+ {
+ ...wire,
+ questions: { ...wire.questions, [id]: { ...wire.questions[id], extra: true } },
+ },
+ true,
+ ),
+ ).toThrow();
+ expect(() => transport.decode(request, { ...wire, model: "wrong-model" }, true)).toThrow();
+ }
+ } finally {
+ await transport.cleanup();
+ }
+ }
+});
diff --git a/test/reference/transport.ts b/test/reference/transport.ts
new file mode 100644
index 0000000..96a58b4
--- /dev/null
+++ b/test/reference/transport.ts
@@ -0,0 +1,93 @@
+import { expect } from "bun:test";
+import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+
+// Independent expectations: deriving these from production would hide preset drift.
+export const providers = {
+ vercel: { url: "https://ai-gateway.vercel.sh/typesafe/v1/systemone", model: "typesafe-ai/jev" },
+ typesafe: { url: "https://api.typesafe.ai/v1/systemone", model: "jev-1.13.0" },
+ openrouter: { url: "https://openrouter.ai/api/v1/systemone", model: "jev-1.13" },
+} as const;
+export type Provider = keyof typeof providers;
+
+export async function referenceTransport(provider: Provider = "vercel", key = "fixture") {
+ const home = await mkdtemp(join(tmpdir(), "jg-reference-auth-"));
+ await mkdir(join(home, "jevgrep"));
+ await writeFile(
+ join(home, "jevgrep/credentials.json"),
+ JSON.stringify({ provider, apiKey: key }),
+ );
+ return {
+ env(production: boolean, origin: string) {
+ return production
+ ? {
+ PATH: process.env.PATH,
+ XDG_CONFIG_HOME: home,
+ NODE_OPTIONS: `--import=${resolve("test/fixtures/provider-route.mjs")}`,
+ JEVGREP_TEST_PROVIDER_ORIGIN: origin,
+ }
+ : {
+ PATH: process.env.PATH,
+ AI_GATEWAY_API_KEY: key,
+ AI_GATEWAY_BASE_URL: `${origin}/v4/ai`,
+ };
+ },
+ decode(
+ request: { url: string; method?: string; headers: Headers },
+ raw: unknown,
+ production: boolean,
+ ) {
+ expect(request.method).toBe("POST");
+ expect(request.headers.get("authorization")).toBe(`Bearer ${key}`);
+ const body = raw as {
+ model?: string;
+ providerOptions?: Record<string, unknown>;
+ state: unknown;
+ questions: Record<string, { type: string; instructions: string }>;
+ };
+ expect(Object.keys(body).sort()).toEqual(
+ production ? ["model", "questions", "state"] : ["providerOptions", "questions", "state"],
+ );
+ expect(new URL(request.url).pathname).toBe(
+ production ? new URL(providers[provider].url).pathname : "/v4/ai/evaluation-model",
+ );
+ if (production) {
+ expect(request.headers.get("x-jevgrep-original-url")).toBe(providers[provider].url);
+ expect(body.model).toBe(providers[provider].model);
+ } else {
+ expect(request.headers.get("ai-model-id")).toBe("typesafe-ai/jev");
+ expect(body.providerOptions).toEqual({});
+ }
+ expect(body.state).toBeDefined();
+ expect(body.questions).toBeObject();
+ const questions = Object.fromEntries(
+ Object.entries(body.questions).map(([id, question]) => {
+ expect(Object.keys(question).sort()).toEqual(["instructions", "type"]);
+ expect(question.type).toBe(production ? "noul" : "boolean");
+ expect(typeof question.instructions).toBe("string");
+ return [id, { ...question, type: "boolean" }];
+ }),
+ );
+ // Only the validated transport envelope/types differ; state and question order stay exact.
+ return { state: body.state, questions, providerOptions: {} };
+ },
+ cleanup: () => rm(home, { recursive: true, force: true }),
+ };
+}
+
+export function wireResponse<
+ T extends { answers: Record<string, { type: string; probability: number }> },
+>(body: T, production: boolean) {
+ return production
+ ? {
+ ...body,
+ answers: Object.fromEntries(
+ Object.entries(body.answers).map(([id, answer]) => [
+ id,
+ { type: "noul", noul: answer.probability },
+ ]),
+ ),
+ }
+ : body;
+}
diff --git a/test/retrieval-freshness.test.ts b/test/retrieval-freshness.test.ts
index aa3a626..f6007b0 100644
--- a/test/retrieval-freshness.test.ts
+++ b/test/retrieval-freshness.test.ts
@@ -1,10 +1,11 @@
+import { routeProviderFetch } from "./fixtures/provider-route.mjs";
import { expect } from "bun:test";
import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { testIfDocker } from "./helpers/docker";
import { retrieve } from "../packages/core/src/retrieve";
-import { createEvaluator } from "../packages/core/src/gateway";
+import { createEvaluator } from "../packages/core/src/evaluator";
testIfDocker(
"queued navigation never uploads source excluded after the first wave",
@@ -29,7 +30,7 @@ testIfDocker(
}
return Response.json({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.1 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.1 }]),
),
});
},
@@ -39,7 +40,12 @@ testIfDocker(
const signal = new AbortController().signal;
const result = await retrieve(
{ root, query: "sentinel", signal },
- createEvaluator({ apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}`, signal }),
+ createEvaluator({
+ apiKey: "fixture",
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
+ signal,
+ }),
);
expect(uploads).toBe(8);
expect(result.status).toBe("incomplete");
@@ -71,7 +77,7 @@ for (const mutation of ["ignored", "changed"] as const)
}
return Response.json({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),
),
});
},
@@ -86,7 +92,8 @@ for (const mutation of ["ignored", "changed"] as const)
{ root, query: "selected", signal },
createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal,
}),
);
@@ -124,7 +131,12 @@ testIfDocker(
const signal = new AbortController().signal;
const result = await retrieve(
{ root, query: "retry", signal },
- createEvaluator({ apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}`, signal }),
+ createEvaluator({
+ apiKey: "fixture",
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
+ signal,
+ }),
);
expect(uploads).toBe(1);
expect(result.status).toBe("incomplete");
@@ -155,7 +167,7 @@ for (const phase of ["selection", "role"] as const)
}
return Response.json({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),
),
});
},
@@ -170,7 +182,8 @@ for (const phase of ["selection", "role"] as const)
{ root, query: "selected", signal },
createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal,
}),
);
@@ -202,7 +215,7 @@ for (const size of [2, 4])
}
return Response.json({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.1 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.1 }]),
),
});
},
@@ -216,7 +229,8 @@ for (const size of [2, 4])
{ root, query: "selected", signal },
createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal,
}),
);
@@ -258,8 +272,8 @@ for (const donor of ["sample", "anchor"] as const)
Object.keys(body.questions).map((id, index) => [
id,
{
- type: "boolean",
- probability: body.state.items?.[index]?.kind === "file" ? 0.9 : 0.1,
+ type: "noul",
+ noul: body.state.items?.[index]?.kind === "file" ? 0.9 : 0.1,
},
]),
),
@@ -281,7 +295,8 @@ for (const donor of ["sample", "anchor"] as const)
{ root, query: "selected", signal },
createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal,
}),
);
@@ -314,7 +329,7 @@ testIfDocker(
}
return Response.json({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),
),
});
},
@@ -325,7 +340,12 @@ testIfDocker(
const signal = new AbortController().signal;
const result = await retrieve(
{ root, query: "selected", signal },
- createEvaluator({ apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}`, signal }),
+ createEvaluator({
+ apiKey: "fixture",
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
+ signal,
+ }),
);
expect(attempts).toBe(1);
expect(result.status).toBe("incomplete");
diff --git a/test/retrieval.test.ts b/test/retrieval.test.ts
index 91fa7f0..535ffbe 100644
--- a/test/retrieval.test.ts
+++ b/test/retrieval.test.ts
@@ -1,10 +1,11 @@
+import { routeProviderFetch } from "./fixtures/provider-route.mjs";
import { expect } from "bun:test";
import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { testIfDocker } from "./helpers/docker";
import { retrieve } from "../packages/core/src/retrieve";
-import { createEvaluator } from "../packages/core/src/gateway";
+import { createEvaluator } from "../packages/core/src/evaluator";
testIfDocker(
"hierarchical retrieval returns source from multiple files without uploading excluded data",
@@ -21,7 +22,7 @@ testIfDocker(
sent.push(JSON.stringify(body));
return Response.json({
answers: Object.fromEntries(
- Object.keys(body.questions).map((id) => [id, { type: "boolean", probability: 0.9 }]),
+ Object.keys(body.questions).map((id) => [id, { type: "noul", noul: 0.9 }]),
),
});
},
@@ -43,7 +44,8 @@ testIfDocker(
const signal = new AbortController().signal;
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal,
});
const result = await retrieve({ root, query: "research event recording", signal }, evaluator);
@@ -211,13 +213,14 @@ testIfDocker("an aborted evaluator cannot return a previously cached answer", as
port: 0,
fetch() {
calls++;
- return Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });
+ return Response.json({ answers: { q: { type: "noul", noul: 0.8 } } });
},
});
try {
const evaluator = createEvaluator({
apiKey: "fixture",
- baseURL: `http://127.0.0.1:${server.port}`,
+ provider: "vercel",
+ fetch: routeProviderFetch(fetch, `http://127.0.0.1:${server.port}`),
signal: controller.signal,
cache: createEvaluationCache({ directory }),
});
diff --git a/test/runtime.Dockerfile b/test/runtime.Dockerfile
index d60892f..e0feb55 100644
--- a/test/runtime.Dockerfile
+++ b/test/runtime.Dockerfile
@@ -36,6 +36,7 @@ FROM node:22-bookworm-slim AS runtime
COPY --from=install /opt/jevgrep /opt/jevgrep
COPY --from=package /artifacts/canonical-skill.md /test/canonical-skill.md
COPY test/installed.test.mjs /test/installed.test.mjs
+COPY test/fixtures/provider-route.mjs /test/fixtures/provider-route.mjs
COPY --chmod=755 test/fixtures/skill-installer.mjs /test/fixtures/skill-installer.mjs
ENV PATH=/opt/jevgrep/bin:/usr/local/bin:/usr/bin:/bin \
JEVGREP_INSTALLED_BINARY=/opt/jevgrep/bin/jg \