Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

7.4 KB
Raw Blame

eve/extensions/code

eve/extensions/code is an eve extension for coding work. It contributes apply_patch, grep, investigation and review skills, a read-only worker subagent, sandbox tooling, and shared PR-watch primitives. Vercel credentials are brokered before each turn. GitHub instructions, the pr skill, and the authenticated gh tool live in eve/extensions/git. Because eve workflow directives are application-only, consumers own their prwatch / prwatch_delete workflow tools.

It ships inside the eve package. This private @eve/code workspace package is its source of truth: eve's build copies extension/ into packages/eve/src/extensions/code/extension and publishes it with these entry points:

  • eve/extensions/code: the extension
  • eve/extensions/code/sandbox: sandbox bootstrap and credential helpers
  • eve/extensions/code/tools: apply_patch and grep (gh is a deprecated re-export)
  • eve/extensions/code/prwatch: PR-watch primitives for consumer-owned workflow tools

Mount

// agent/extensions/code.ts
import code from "eve/extensions/code";

export default code({
  // Optional; omit to mount without Connect-backed Vercel authentication.
  vercel: { connector: "vercel/acme-bot" },
});

Both connectors are optional; code({}) mounts without either. The Vercel connector requests an app-subject token. Firewall delivery is the default and keeps tokens outside sandbox processes; without a consumer broker, it requires a sandbox provider that exposes setNetworkPolicy() and fails otherwise. Use delivery: "command" on the Vercel connector for providers without mutable network policy. A top-level broker(sandbox, rules) callback lets the consumer merge Vercel credential rules into its own network policy.

The github option is deprecated; configure it on eve/extensions/git instead. While it is set here, this extension still contributes code__gh, its GitHub instructions, and the signed-commit code__pr skill. Without it, this extension contributes no GitHub tool, instructions, or pull request skill.

The read-only worker subagent defaults to openai/gpt-5.6-terra-fast with xhigh reasoning. Set worker: { model, reasoning, openaiReasoningEffort? } to choose another model; openaiReasoningEffort is passed to OpenAI models as reasoningEffort.

Sandbox bootstrap

Install CLI tooling in the environment's prepare callback:

// agent/sandbox.ts
import { defineSandbox } from "eve/sandbox";
import { VercelSandbox } from "eve/sandbox/vercel";
import { installCodeTooling } from "eve/extensions/code/sandbox";

export const environment = VercelSandbox.environment({
  prepare: async (sandbox) => {
    await installCodeTooling(sandbox, { vercel: true });
  },
});

export default defineSandbox(() => environment.open());

eve derives the prepared environment generation from the sandbox file and environment options, not from imported helpers, so upgrading eve alone does not rebuild an existing prepared artifact.

Computer use lives in eve/computer-use. Mount it next to this extension when the sandbox has a desktop. installComputerUse, startComputerUse, and COMPUTER_USE_REVALIDATION_KEY are still re-exported from eve/extensions/code/sandbox, and computer_use from eve/extensions/code/tools, but both are deprecated.

Preparation ensures gh, installs wrappers for gh, vc, and gh-signed-commit, and installs TypeScript diagnostics. With eve/extensions/git configured for github, repositories requiring verified signatures use gh-signed-commit through its gh tool.

Non-Connect escape hatch

Consumers with a PAT, benchmark token, or another credential provider can omit the corresponding connector and call authenticateGitHub or authenticateVercel from eve/extensions/code/sandbox in their own sandbox lifecycle. These helpers support firewall, command-delivery, and broker options for consumer-owned commands; they do not configure the extension's gh tool.

Develop in this workspace

Rebuild eve after editing extension/; the local agent under agent/ mounts the built eve/extensions/code:

pnpm --filter eve build
pnpm --filter @eve/code typecheck
pnpm --filter @eve/code test
pnpm --filter @eve/code test:scenario
pnpm exec oxlint packages/eve-code
pnpm exec oxfmt --check packages/eve-code

Tests live under test/, outside the extension distribution. Unit and integration tests run through the workspace's matching test tasks. The package integration task depends only on eve's build. The root integration command runs the framework suite before the other packages because that suite rebuilds the runtime files they import. Scenario tests exercise temporary files, local Git repositories, and subprocesses; they need Node.js 24 or newer, Git, and Bash, but no model or service credentials.

The typescript-compiler development alias supplies the JavaScript compiler API used by the diagnostics worker test. The workspace's TypeScript 7 CLI remains the package typechecker. prepack builds the extension; installation does not run the extension CLI before the local framework has been built.

Benchmarks

eve-code is benchmarked with eve-bench on the SWE-lean dataset. eve-bench owns datasets, execution, comparisons, and reports; this package keeps no benchmark runner of its own.

In CI

The eve-code > Benchmark harness workflow runs whenever packages/eve-code/** changes. It benchmarks the PR head's eve-code, opencode, and pi together, using the model in the EVE_CODE_BENCH_MODEL repository variable. Every trial runs in its own Vercel Sandbox, and all of them start at once, so a run takes about as long as its slowest trial. Each harness is compared with its own latest result from main. The report covers resolved tasks, latency, and token usage. It goes to the job summary and a PR comment.

Comment /benchmark to re-run the default harnesses, or /benchmark <harness>[,<harness>...] to choose, for example /benchmark codex,eve-code. You need write access to the repository. Every push to main that touches eve-code publishes fresh results to the eve-bench result store, which is where later PRs get their baselines.

Locally

With eve-bench linked (npm link in its checkout), commit and push this checkout, then run:

eve-bench -a eve-code --agent-dir packages/eve-code --model openai/gpt-6-luna --reasoning low \
  --scope <vercel-team> --execution vercel-sandbox

Pass several harnesses to compare them in one run, for example -a eve-code,opencode,pi. Use --task <name> to run a single task. Local runs never publish to the result store.

Investigating a failed trial

Trial sandboxes are deleted as soon as each trial finishes. Before deletion, eve-bench pulls each trial's logs and traces out of the sandbox and uploads them with the workflow artifact. To read them, point eve-bench at the CI run:

eve-bench trials logs https://github.com/vercel/eve/actions/runs/<id>
eve-bench trials logs https://github.com/vercel/eve/actions/runs/<id> --harness eve-code --task <task>

The report's Diagnostics section prints this command with the run URL filled in. The download uses the GitHub CLI and needs read access to this repository. The raw files (agent.log, events.ndjson, observability.ndjson, verifier.log) are cached under ~/.cache/eve-bench/runs/.