Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

Add self-modification registry evals (#3661)

prhaa56912599972
diff --git a/e2e/fixtures/agent-self-modification/README.md b/e2e/fixtures/agent-self-modification/README.md
index 2b30a190c..e9804afa1 100644
--- a/e2e/fixtures/agent-self-modification/README.md
+++ b/e2e/fixtures/agent-self-modification/README.md
@@ -16,13 +16,14 @@ This exercises the current standard scaffold without duplicating it in the fixtu
## Cases
+- `add-agent-browser.eval.ts` asks for browser automation without naming a registry address. It checks the child's registry search and approved install, and compares the resulting extension with the checkout's official registry source. `add-slack-channel.eval.ts` similarly checks discovery, then verifies that the setup-bearing Slack item is handed off to `eve add channel/slack` without installation. Neither case needs service credentials or an external account.
- `create-background-replication-check.eval.ts` creates a ten-second background workflow, calls it on the next turn of the authoring session, and checks its immediate receipt, delayed completion, task identity, typed input, and structured positive result.
- `create-incident-triage.eval.ts` creates an incident-triage tool and checks precedence and threshold rules across typed inputs.
- `create-shipping-quote.eval.ts` creates a quote calculator and checks destination, started-kilogram, free-shipping, and expedited pricing boundaries.
- `offer-repair.eval.ts` reproduces an incorrect reorder recommendation, checks that the parent offers but does not start a repair, confirms it, and verifies the repaired tool.
- `repair-order-total.eval.ts` first reproduces quantity and discount errors in the fixture's existing tool, asks self-mod to investigate the incorrect invoice, and verifies the fix, cent rounding, and single-item/empty-order regressions.
-Each case checks actual tool inputs and outputs, not the assistant's claim that the work succeeded. Cases also reject source changes outside their specified tool file. The arithmetic cases use synthetic data and do not require external services. These checks cover the specified behavior; they are not a general security audit of generated code.
+Each case checks actual tool inputs and outputs, not the assistant's claim that the work succeeded. Cases also reject source changes outside their specified target. The registry cases serve only the selected checkout-owned manifests on loopback via `EVE_DEV_OFFICIAL_REGISTRY_URL`; this exercises `registry_add` and `eve add` without depending on eve.dev. They verify agent-browser installation and the Slack setup handoff, not browser automation or channel activation. The arithmetic cases use synthetic data and do not require external services. These checks cover the specified behavior; they are not a general security audit of generated code.
## Adding a case
@@ -33,7 +34,7 @@ Wrap source-mutating cases in `withSelfModification(t, async (selfMod) => { ...
3. Use `selfMod.assertOnlyChanged(paths)` to check the permitted edit scope, then `selfMod.apply()` to force a runtime rebuild.
4. Use `selfMod.verify(prompt)` to invoke the tool in a fresh session. Assert the call's input and structured output with `requireToolCall`, including boundary cases and previously working behavior.
-The harness snapshots the complete `agent/` tree, tracks sessions, and retires them before restoring source. Restoration removes unexpected files and restores deleted files, binary contents, and file modes. Source watching is suspended during restoration. If session retirement or restoration fails, later mutation cases fail instead of continuing against uncertain state; failed retirement retains the backup path in the error.
+The harness snapshots the complete `agent/` tree plus installer-mutated project files, tracks sessions, and retires them before restoring source. Restoration removes unexpected files and restores deleted files, binary contents, and file modes. Source watching is suspended during restoration. If session retirement or restoration fails, later mutation cases fail instead of continuing against uncertain state; failed retirement retains the backup path in the error.
Keep `maxConcurrency: 1`. The harness also serializes cleanup that continues after an eval timeout and acquires a checkout lock before snapshotting source. A concurrent `eve eval` process fails before mutation. If cleanup cannot safely restore source, the lock remains with owner diagnostics; remove it only after inspecting the retained backup and confirming no eval or development server is mutating the fixture.
diff --git a/e2e/fixtures/agent-self-modification/evals/evals.config.ts b/e2e/fixtures/agent-self-modification/evals/evals.config.ts
index 6c07ee96d..be63a38aa 100644
--- a/e2e/fixtures/agent-self-modification/evals/evals.config.ts
+++ b/e2e/fixtures/agent-self-modification/evals/evals.config.ts
@@ -1,8 +1,14 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
+import { startRegistryServer } from "./self-modification/registry-server";
+
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
maxConcurrency: 1,
timeoutMs: 240_000,
+ setup: startRegistryServer,
+ async teardown(stop) {
+ await stop?.();
+ },
});
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/add-agent-browser.eval.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/add-agent-browser.eval.ts
new file mode 100644
index 000000000..15d7d9fc3
--- /dev/null
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/add-agent-browser.eval.ts
@@ -0,0 +1,22 @@
+import { defineEval } from "eve/evals";
+
+import { withSelfModification } from "./harness";
+import { verifyRegistryInstall } from "./registry-install";
+
+export default defineEval({
+ tags: ["real-model"],
+ description:
+ "Self-mod installs the official agent-browser extension instead of authoring browser tools.",
+ async test(t) {
+ await withSelfModification(t, async (selfMod) => {
+ const run = await selfMod.request("Can you add browser automation?");
+ await verifyRegistryInstall({
+ address: "extension/agent-browser",
+ source: "registry/extensions/agent-browser.ts",
+ target: "extensions/browser.ts",
+ run,
+ selfMod,
+ });
+ });
+ },
+});
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/add-slack-channel.eval.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/add-slack-channel.eval.ts
new file mode 100644
index 000000000..ef54ad31d
--- /dev/null
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/add-slack-channel.eval.ts
@@ -0,0 +1,16 @@
+import { defineEval } from "eve/evals";
+
+import { withSelfModification } from "./harness";
+import { verifyRegistryHandoff } from "./registry-install";
+
+export default defineEval({
+ tags: ["real-model"],
+ description:
+ "Self-mod discovers the official Slack channel and hands its required setup to the terminal.",
+ async test(t) {
+ await withSelfModification(t, async (selfMod) => {
+ const run = await selfMod.request("Can you add the slack channel?");
+ await verifyRegistryHandoff({ address: "channel/slack", run, selfMod });
+ });
+ },
+});
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/create-background-replication-check.eval.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/create-background-replication-check.eval.ts
index 6a0abd5dc..13bb04e46 100644
--- a/e2e/fixtures/agent-self-modification/evals/self-modification/create-background-replication-check.eval.ts
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/create-background-replication-check.eval.ts
@@ -14,6 +14,7 @@ export default defineEval({
async test(t) {
await withSelfModification(t, async (selfMod) => {
+ const previousRevision = await selfMod.runtimeRevision();
const authored = await selfMod.request(
[
`Alice needs a reusable local background workflow named ${TOOL_NAME} for checking a replicated import in future conversations.`,
@@ -25,7 +26,7 @@ export default defineEval({
);
await selfMod.readSource(TOOL_PATH);
await selfMod.assertOnlyChanged([TOOL_PATH]);
- // Do not force a rebuild: this case covers automatic next-turn availability.
+ await selfMod.waitForRebuild(previousRevision);
const batchId = `batch-${randomUUID()}`;
const expectedRecords = 137;
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/harness.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/harness.ts
index d469afca1..b6bb8db1d 100644
--- a/e2e/fixtures/agent-self-modification/evals/self-modification/harness.ts
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/harness.ts
@@ -11,12 +11,15 @@ import {
writeFile,
} from "node:fs/promises";
import { tmpdir } from "node:os";
-import { dirname, join } from "node:path";
+import { dirname, join, resolve } from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
import type { EveEvalContext, EveEvalLiveTurn, EveEvalSession, EveEvalTurn } from "eve/evals";
const SELF_MODIFICATION_AGENT = "self-modification__agent";
const CLEANUP_TIMEOUT_MS = 30_000;
+const REBUILD_TIMEOUT_MS = 30_000;
+const REBUILD_POLL_INTERVAL_MS = 100;
const LOCK_DIRECTORY = ".eve-self-modification-eval.lock";
// Each eval entry bundles its relative imports separately. Share the lock across those copies.
const shared = globalThis as typeof globalThis & {
@@ -93,12 +96,19 @@ export class SelfModificationHarness {
readonly #t: EveEvalContext;
readonly #sourceRoot: string;
readonly #backupRoot: string;
+ readonly #projectFiles: ReadonlyMap<string, Buffer | undefined>;
readonly #turns = new Set<EveEvalLiveTurn>();
- private constructor(t: EveEvalContext, sourceRoot: string, backupRoot: string) {
+ private constructor(
+ t: EveEvalContext,
+ sourceRoot: string,
+ backupRoot: string,
+ projectFiles: ReadonlyMap<string, Buffer | undefined>,
+ ) {
this.#t = t;
this.#sourceRoot = sourceRoot;
this.#backupRoot = backupRoot;
+ this.#projectFiles = projectFiles;
}
static async create(
@@ -119,7 +129,23 @@ export class SelfModificationHarness {
try {
await options.onBackupCreated?.(backupRoot);
await cp(sourceRoot, join(backupRoot, "agent"), { recursive: true, verbatimSymlinks: true });
- return new SelfModificationHarness(t, sourceRoot, backupRoot);
+ const projectRoot = dirname(sourceRoot);
+ const projectFiles = new Map<string, Buffer | undefined>();
+ for (const path of [
+ join(projectRoot, ".env.local"),
+ join(projectRoot, ".env.example"),
+ join(projectRoot, "package.json"),
+ resolve(projectRoot, "../../../pnpm-lock.yaml"),
+ ]) {
+ projectFiles.set(
+ path,
+ await readFile(path).catch((error: NodeJS.ErrnoException) => {
+ if (error.code === "ENOENT") return undefined;
+ throw error;
+ }),
+ );
+ }
+ return new SelfModificationHarness(t, sourceRoot, backupRoot, projectFiles);
} catch (error) {
await rm(backupRoot, { recursive: true, force: true });
throw error;
@@ -168,10 +194,50 @@ export class SelfModificationHarness {
),
continuation?.result().then((turn) => turn.expectOk()),
]);
- this.#t.calledSubagent(SELF_MODIFICATION_AGENT);
+ this.#t.calledSubagent(SELF_MODIFICATION_AGENT, { status: "working" });
return { child, parent, session: liveParent.session };
}
+ /** Approves the registry install on the parent and observes the resumed child turn. */
+ async approveRegistry(run: SelfModificationRun): Promise<EveEvalTurn> {
+ const toolName = "registry_add";
+ const liveParent = this.#t.target.watchTurn(run.session.sessionId, {
+ startIndex: run.session.state.streamIndex,
+ });
+ this.#turns.add(liveParent);
+ const approval = await liveParent.waitForEvent("input.requested", {
+ data: { requests: [{ action: { kind: "tool-call", toolName } }] },
+ });
+ const requests = approval.data.requests.filter(
+ (request) => request.action.kind === "tool-call" && request.action.toolName === toolName,
+ );
+ if (requests.length !== 1) {
+ throw new Error(`Expected one pending ${toolName} approval, found ${requests.length}.`);
+ }
+ (await liveParent.result()).expectOk();
+ const session = liveParent.session;
+ session.requireInputRequest({ toolName });
+ const childSessionId = run.child.sessionId;
+ const previousChild = [...this.#turns]
+ .reverse()
+ .find((turn) => turn.sessionId === childSessionId);
+ if (previousChild === undefined) throw new Error("Missing registry child session cursor.");
+ const child = this.#t.target.watchTurn(childSessionId, {
+ startIndex: previousChild.session.state.streamIndex,
+ });
+ this.#turns.add(child);
+ const responses = session.pendingInputRequests.map((request) => ({
+ optionId: "approve",
+ requestId: request.requestId,
+ }));
+ const parent = await session.startRespond(responses);
+ this.#turns.add(parent);
+ const [childTurn, parentTurn] = await Promise.all([child.result(), parent.result()]);
+ childTurn.expectOk();
+ parentTurn.expectOk();
+ return childTurn;
+ }
+
followUp(session: EveEvalSession, prompt: string): Promise<EveEvalTurn> {
return this.#runTurn(session, prompt);
}
@@ -190,6 +256,43 @@ export class SelfModificationHarness {
this.#t.log(`Self-modification runtime revision: ${body.revision}`);
}
+ async runtimeRevision(): Promise<string> {
+ const revision = await this.#readRuntimeRevision();
+ if (revision === undefined)
+ throw new Error("Could not read the current authored runtime revision.");
+ return revision;
+ }
+
+ async waitForRebuild(previousRevision: string): Promise<string> {
+ const deadline = Date.now() + REBUILD_TIMEOUT_MS;
+ while (Date.now() < deadline) {
+ this.#t.signal.throwIfAborted();
+ const revision = await this.#readRuntimeRevision();
+ if (revision !== undefined && revision !== previousRevision) {
+ this.#t.log(`Self-modification runtime revision: ${revision}`);
+ return revision;
+ }
+ await delay(REBUILD_POLL_INTERVAL_MS, undefined, { signal: this.#t.signal });
+ }
+ throw new Error(`Authored runtime did not rebuild within ${REBUILD_TIMEOUT_MS}ms.`);
+ }
+
+ async #readRuntimeRevision(): Promise<string | undefined> {
+ try {
+ const response = await this.#t.target.fetch("/eve/v1/dev/runtime-artifacts", {
+ signal: this.#t.signal,
+ });
+ if (!response.ok) return undefined;
+ const body = (await response.json()) as { revision?: unknown };
+ return typeof body.revision === "string" && body.revision.length > 0
+ ? body.revision
+ : undefined;
+ } catch {
+ this.#t.signal.throwIfAborted();
+ return undefined;
+ }
+ }
+
async assertOnlyChanged(sourcePaths: readonly string[]): Promise<void> {
const original = await sourceFiles(join(this.#backupRoot, "agent"));
const current = await sourceFiles(this.#sourceRoot);
@@ -257,6 +360,10 @@ export class SelfModificationHarness {
recursive: true,
verbatimSymlinks: true,
});
+ for (const [path, content] of this.#projectFiles) {
+ if (content === undefined) await rm(path, { force: true });
+ else await writeFile(path, content);
+ }
} finally {
await this.#post(`resume?lease=${lease}`, signal);
}
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/registry-install.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/registry-install.ts
new file mode 100644
index 000000000..921cc0648
--- /dev/null
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/registry-install.ts
@@ -0,0 +1,66 @@
+import { readFile } from "node:fs/promises";
+import { resolve } from "node:path";
+
+import type { SelfModificationHarness, SelfModificationRun } from "./harness";
+
+export async function verifyRegistryHandoff(input: {
+ readonly address: string;
+ readonly run: SelfModificationRun;
+ readonly selfMod: SelfModificationHarness;
+}): Promise<void> {
+ const { address, run, selfMod } = input;
+ assertDiscovered(run, address);
+ const pending = run.child.toolCalls.filter((call) => call.name === "registry_add");
+ if (pending.length !== 1 || pending[0]?.input.address !== address) {
+ throw new Error(`Self-modification did not request ${address} exactly once.`);
+ }
+ const handoff = await selfMod.approveRegistry(run);
+ const result = handoff.toolCalls.find((call) => call.name === "registry_add");
+ if (
+ result?.status !== "completed" ||
+ (result.output as { nextCommand?: unknown; status?: unknown } | undefined)?.status !==
+ "needs-terminal" ||
+ (result.output as { nextCommand?: unknown } | undefined)?.nextCommand !== `eve add ${address}`
+ ) {
+ throw new Error(`Self-modification did not hand ${address} off to the terminal.`);
+ }
+ await selfMod.assertOnlyChanged([]);
+}
+
+export async function verifyRegistryInstall(input: {
+ readonly address: string;
+ readonly source: string;
+ readonly target: string;
+ readonly run: SelfModificationRun;
+ readonly selfMod: SelfModificationHarness;
+}): Promise<void> {
+ const { address, source, target, run, selfMod } = input;
+ assertDiscovered(run, address);
+ const pending = run.child.toolCalls.filter((call) => call.name === "registry_add");
+ if (pending.length !== 1 || pending[0]?.input.address !== address) {
+ throw new Error(`Self-modification did not request ${address} exactly once.`);
+ }
+ const installed = await selfMod.approveRegistry(run);
+ const result = installed.toolCalls.find((call) => call.name === "registry_add");
+ if (
+ result?.status !== "completed" ||
+ (result.output as { status?: unknown } | undefined)?.status !== "installed"
+ ) {
+ throw new Error(`Self-modification did not install ${address}.`);
+ }
+ await selfMod.assertOnlyChanged([target]);
+ const expected = await readFile(resolve(process.cwd(), "../../../apps/docs", source), "utf8");
+ const actual = await selfMod.readSource(target);
+ if (actual !== expected) throw new Error(`${target} is not the official registry scaffold.`);
+}
+
+function assertDiscovered(run: SelfModificationRun, address: string): void {
+ run.child.calledTool("search_registry");
+ const discovered = run.child.toolCalls.some((call) => {
+ if (call.name !== "search_registry" || call.status !== "completed") return false;
+ const items = (call.output as { items?: unknown } | undefined)?.items;
+ return Array.isArray(items) && items.some((item) => item.address === address);
+ });
+ if (!discovered)
+ throw new Error(`Self-modification did not discover ${address} in the registry.`);
+}
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/registry-server.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/registry-server.ts
new file mode 100644
index 000000000..5ff25379d
--- /dev/null
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/registry-server.ts
@@ -0,0 +1,62 @@
+import { createServer } from "node:http";
+import { readFile } from "node:fs/promises";
+import { resolve } from "node:path";
+
+const docsRoot = resolve(process.cwd(), "../../../apps/docs");
+const addresses = new Set(["extension/agent-browser", "channel/slack"]);
+
+/** Serve checkout-owned official manifests without depending on eve.dev or a deployed registry. */
+export async function startRegistryServer(): Promise<() => Promise<void>> {
+ const registry = JSON.parse(await readFile(resolve(docsRoot, "registry.json"), "utf8")) as {
+ items: { name: string; files?: { path: string }[] }[];
+ };
+ const items = registry.items.filter((item) => addresses.has(item.name));
+ if (items.length !== addresses.size) throw new Error("Registry eval item missing from checkout.");
+ const manifests = new Map<string, unknown>(
+ await Promise.all(
+ items.map(
+ async (item) =>
+ [
+ `/r/${item.name}.json`,
+ {
+ ...item,
+ files:
+ item.files === undefined
+ ? undefined
+ : await Promise.all(
+ item.files.map(async (file) => ({
+ ...file,
+ content: await readFile(resolve(docsRoot, file.path), "utf8"),
+ })),
+ ),
+ },
+ ] as const,
+ ),
+ ),
+ );
+ const server = createServer((request, response) => {
+ const body =
+ request.url === "/r/registry.json"
+ ? { ...registry, items }
+ : manifests.get(request.url ?? "");
+ response.writeHead(body === undefined ? 404 : 200, { "content-type": "application/json" });
+ response.end(JSON.stringify(body ?? { error: "Not found" }));
+ });
+ try {
+ await new Promise<void>((done) => server.listen(0, "127.0.0.1", done));
+ } catch (error) {
+ server.close();
+ throw error;
+ }
+ const address = server.address();
+ if (address === null || typeof address === "string") throw new Error("No registry port.");
+ const previous = process.env.EVE_DEV_OFFICIAL_REGISTRY_URL;
+ process.env.EVE_DEV_OFFICIAL_REGISTRY_URL = `http://127.0.0.1:${address.port}/r`;
+ return async () => {
+ if (previous === undefined) delete process.env.EVE_DEV_OFFICIAL_REGISTRY_URL;
+ else process.env.EVE_DEV_OFFICIAL_REGISTRY_URL = previous;
+ await new Promise<void>((done, reject) =>
+ server.close((error) => (error ? reject(error) : done())),
+ );
+ };
+}
diff --git a/e2e/fixtures/agent-self-modification/package.json b/e2e/fixtures/agent-self-modification/package.json
index 88f842047..109e12d03 100644
--- a/e2e/fixtures/agent-self-modification/package.json
+++ b/e2e/fixtures/agent-self-modification/package.json
@@ -34,6 +34,13 @@
"self-modification/repair-order-total"
]
},
+ {
+ "name": "registry",
+ "evals": [
+ "self-modification/add-agent-browser",
+ "self-modification/add-slack-channel"
+ ]
+ },
{
"name": "create",
"evals": [
diff --git a/e2e/fixtures/agent-self-modification/test/harness.scenario.test.mjs b/e2e/fixtures/agent-self-modification/test/harness.scenario.test.mjs
index bf920dc2e..d6c0822e2 100644
--- a/e2e/fixtures/agent-self-modification/test/harness.scenario.test.mjs
+++ b/e2e/fixtures/agent-self-modification/test/harness.scenario.test.mjs
@@ -108,6 +108,25 @@ async function assertTreeRestored(root) {
await assert.rejects(readFile(join(root, "unexpected.txt")));
}
+test("close restores registry installer project files", async () => {
+ const root = await mkdtemp(join(tmpdir(), "eve-selfmod-project-"));
+ await mkdir(join(root, "agent"));
+ await writeFile(join(root, ".env.example"), "EXISTING=original\n");
+ await writeFile(join(root, "package.json"), '{"name":"original"}\n');
+ const harness = await SelfModificationHarness.create(context(targetFor()), join(root, "agent"));
+ try {
+ await writeFile(join(root, ".env.example"), "EXISTING=changed\n");
+ await writeFile(join(root, ".env.local"), "BROWSER_USE_API_KEY=\n");
+ await writeFile(join(root, "package.json"), '{"name":"changed"}\n');
+ await harness.close();
+ assert.equal(await readFile(join(root, ".env.example"), "utf8"), "EXISTING=original\n");
+ assert.equal(await readFile(join(root, "package.json"), "utf8"), '{"name":"original"}\n');
+ await assert.rejects(readFile(join(root, ".env.local")), { code: "ENOENT" });
+ } finally {
+ await rm(root, { recursive: true, force: true });
+ }
+});
+
test("close retires every session before restoring the complete source tree", async () => {
const child = liveTurn("child");
const verification = liveTurn("verification");
@@ -336,6 +355,25 @@ test("a failed eval restores source, releases its checkout lock, and preserves t
await assert.rejects(readFile(join(root, ".eve-self-modification-eval.lock", "owner.json")));
});
+test("waitForRebuild polls until the authored runtime revision changes", async () => {
+ await withHarness(async ({ harness, target, close }) => {
+ let reads = 0;
+ const fetch = target.fetch;
+ target.fetch = async (path, options) => {
+ if (path === "/eve/v1/dev/runtime-artifacts") {
+ reads += 1;
+ return response({ revision: reads < 3 ? "before" : "after" });
+ }
+ return fetch(path, options);
+ };
+
+ const previous = await harness.runtimeRevision();
+ assert.equal(await harness.waitForRebuild(previous), "after");
+ assert.equal(reads, 3);
+ await close();
+ });
+});
+
test("apply rejects a rebuild response without a runtime revision", async () => {
await withHarness(async ({ harness, target, close }) => {
let apply = true;
diff --git a/e2e/fixtures/agent-self-modification/test/registry-server.scenario.test.mjs b/e2e/fixtures/agent-self-modification/test/registry-server.scenario.test.mjs
new file mode 100644
index 000000000..54de46f23
--- /dev/null
+++ b/e2e/fixtures/agent-self-modification/test/registry-server.scenario.test.mjs
@@ -0,0 +1,36 @@
+import assert from "node:assert/strict";
+import { readFile } from "node:fs/promises";
+import { resolve } from "node:path";
+import { test } from "node:test";
+
+import { startRegistryServer } from "../evals/self-modification/registry-server.ts";
+
+test("serves only checkout-owned registry items and restores the override", async () => {
+ const previous = process.env.EVE_DEV_OFFICIAL_REGISTRY_URL;
+ const stop = await startRegistryServer();
+ try {
+ const base = process.env.EVE_DEV_OFFICIAL_REGISTRY_URL;
+ assert.match(base, /^http:\/\/127\.0\.0\.1:\d+\/r$/);
+ const catalog = await (await fetch(`${base}/registry.json`)).json();
+ assert.deepEqual(
+ catalog.items.map((item) => item.name),
+ ["extension/agent-browser", "channel/slack"],
+ );
+ for (const item of catalog.items) {
+ const response = await fetch(`${base}/${item.name}.json`);
+ assert.equal(response.status, 200);
+ const manifest = await response.json();
+ assert.equal(manifest.name, item.name);
+ for (const file of manifest.files ?? []) {
+ assert.equal(
+ file.content,
+ await readFile(resolve("../../../apps/docs", file.path), "utf8"),
+ );
+ }
+ }
+ assert.equal((await fetch(`${base}/channel/telegram.json`)).status, 404);
+ } finally {
+ await stop();
+ }
+ assert.equal(process.env.EVE_DEV_OFFICIAL_REGISTRY_URL, previous);
+});