Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
Preserve partial evidence and verify installed failure and freshness paths
David Zhang9633676a67f3
diff --git a/evals/README.md b/evals/README.mdindex ec1c739..6e78ad1 100644--- a/evals/README.md+++ b/evals/README.md@@ -1,68 +1,28 @@# Official benchmark evaluation-The [accepted architecture](../docs/architecture.md) records the final spike decision.-The [research ledger](runs/swebench/auto-research-80/research.json) retains experiments-and the explicit revision from an 80% to a 70% target.--Evaluate Jevgrep discovery strategies on a pinned subset of SWE-bench using its official-behavioral grader. Current architecture spikes use Codex Sol with and without Jevgrep, with the-same source, model settings and execution limits within each pair. Claude-evaluation is deferred to full implementation. The [SWE-bench workflow](implementation/swebench/) owns dataset-selection, reproducibility and grading. An earlier [discovery mechanism spike](implementation/swebench/strategy-v42-notes.md) compares hierarchical and broader retrieval.--The [matched Sol research-trace findings](implementation/swebench/paired-sol-research-findings.md)-connect baseline reading behavior to the next architectural probes.--An earlier [completed native comparison](implementation/swebench/batch-native-v81-results.md)-tests batched source inspection. Solve rates match baseline, but both engines take-more time and cost more overall. All timing and accounting checks pass.-The [earlier complete-source comparison](implementation/swebench/open-native-v70-results.md)-remains separate; results are not pooled across candidates.-The [content-informed folder comparison](implementation/swebench/folder-content-v85-results.md)-tests descendant code samples before folder pruning; it does not establish a task-level improvement.-The [two-level discovery comparison](implementation/swebench/lookahead-v86-results.md)-tests speculative enumeration and proceeds to a separate whole-task evaluation.-The [Gateway error diagnostic](implementation/swebench/gateway-errors-v84-results.md)-captures intermittent upstream failures without treating them as relevance judgments.--The [navigation question-scope diagnostic](implementation/swebench/navigation-scope-v82-results.md)-records a wording change that did not recover the observed navigation gap.-The [independent-question diagnostic](implementation/swebench/dual-navigation-v87-results.md)-tests a separate related-implementation judgment and preserves partial valid answers.-The [source-opening diagnostics](implementation/swebench/source-opening-v64-results.md)-separate that coverage gap from query-sensitive relevance judgments; provider failures-limit the evidence and no new native improvement is established.-The [broader relevance diagnostic](implementation/swebench/behavior-question-v67-results.md)-finds both useful additions and excess context, and shows the existing output cap-can discard a recovered chunk before the agent receives it.-The [full-repository batching comparison](implementation/swebench/batch-retrieval-v80-results.md)-measures retrieval throughput; the native comparison above tests its downstream effect.--The [source-batching diagnostic](implementation/swebench/source-batch-v78-results.md)-separates request savings from small-group latency and changed source judgments.--The [full-repository recovery comparison](implementation/swebench/recovery-retrieval-v74-results.md)-shows why retrieval latency must be read alongside unresolved judgments and returned source.--The [navigation recovery diagnostic](implementation/swebench/navigation-recovery-v72-results.md)-tests faster recovery from provider failures while preserving explicit unknowns;-full-repository and native outcomes remain separate acceptance gates.--The [current acceptance policy](cost-quality-policy.md) requires baseline-or-better-official solve rate and agent cost per task for Sol. Elapsed-time is diagnostic only during these architecture spikes. Keep failures and infrastructure problems visible. Report-per-task outcomes and aggregate spend per solved task; do not average only the-successful or fastest attempts. Small subsets provide initial evidence, not a-general performance guarantee.--The frozen [accounting policy](accounting.md) defines billing and measurement;-the current acceptance policy above supersedes its speed requirement. Jev is free for-this evaluation, and its tokens are not coding-agent tokens. Normal agent search-remains available after Jev returns context and discovery signals.--Own-repository fixtures, custom evidence rubrics, previous DeepSWE experiments,-and their results and harnesses are deprecated. They remain local reference-material and are excluded from Git. They are not acceptance evidence for the new-benchmark workflow. Generated datasets, images, workspaces, predictions and raw-traces also stay outside version control; commit only reproducible tooling,-selection manifests and concise official-benchmark reports.+Use SWE-bench's official behavioral grader to compare the same task, source,+model and harness with and without Jevgrep. [Accepted architecture](../docs/architecture.md)+records the spike decision and its evidence limits. The production implementation+must earn its own results; historical wins do not transfer automatically.++The maintained [installed-package checkpoint](implementation/swebench/installed.md)+drives the actual `jg` executable with Sol and reuses the retained Requests+baseline. It preserves raw Jev requests/responses, coding-agent actions, patches,+official grades and complete generation bills. This single-task checkpoint is+not the final ten-task confirmation.++A baseline is immutable per task/model/harness and is never rerun to improve a+comparison. The benchmark requires initial Jevgrep use to isolate retrieval's+effect, while the production skill remains selective. Agents may fill remaining+context gaps with ordinary tools.++[Cost and quality policy](cost-quality-policy.md) owns acceptance. Count the full+coding-agent task, including implementation, verification and failed attempts.+Jev cost and tokens are excluded. Unknown bills cannot count as cost wins. Timing+is diagnostic; preserve baseline solves before claiming lower cost. The tuned+Python cohort does not prove untouched-task or language-wide generalization.++Raw traces, repository snapshots and generated study artifacts stay in ignored+run storage. Commit reproducible official tooling and concise evidence reports.+Personal-repository fixtures and custom-rubric evaluations are deprecated and+stay outside Git; they are not product acceptance evidence.diff --git a/package.json b/package.jsonindex 8e8d185..4974785 100644--- a/package.json+++ b/package.json@@ -23,10 +23,11 @@"format:check": "oxfmt --check .","test:reference": "bash scripts/test-reference.sh","test:parser": "bash scripts/test-reference.sh node --experimental-strip-types --test test/parser/source.test.ts && bash scripts/test-installed.sh --test-name-pattern \"runtime has|parses Python\"",- "test:filesystem": "bash scripts/test-reference.sh bun test packages/core/test/filesystem.test.ts test/retrieval.test.ts",+ "test:filesystem": "bash scripts/test-reference.sh bun test packages/core/test/filesystem.test.ts test/retrieval.test.ts && bash scripts/test-installed.sh --test-name-pattern \"filesystem policy\"","test:installed": "bash scripts/test-installed.sh","test:e2e": "bun run test:installed",- "verify": "bun run check-types && bun run lint && bun run test && bun run test:installed"+ "verify": "bun run check-types && bun run lint && bun run test && bun run test:installed",+ "eval:swebench": "python3 evals/implementation/swebench/installed.py"},"devDependencies": {"oxfmt": "0.57.0",diff --git a/packages/core/src/gateway.ts b/packages/core/src/gateway.tsindex 4ca4fd3..5dd29e1 100644--- a/packages/core/src/gateway.ts+++ b/packages/core/src/gateway.ts@@ -27,10 +27,16 @@ export function createEvaluator(options: {let requests = 0;let cacheHits = 0;let cooldownUntil = 0;+ const authenticationFailure = new AbortController();+ function assertActive() {+ if (options.signal.aborted) throw new EvaluationFailure("cancelled");+ if (authenticationFailure.signal.aborted) throw new EvaluationFailure("authentication");+ }const gateway = createGateway({apiKey: options.apiKey,baseURL: options.baseURL,fetch: async (input, init) => {+ assertActive();if (requests >= (options.requestLimit ?? 50_000))throw new EvaluationFailure("request-limit");requests++;@@ -62,7 +68,7 @@ export function createEvaluator(options: {return requests;},async evaluate(request: EvaluationRequest): Promise<Record<string, number>> {- if (options.signal.aborted) throw new EvaluationFailure("cancelled");+ assertActive();const cacheInput: CacheInput = {request,namespace: {@@ -74,7 +80,7 @@ export function createEvaluator(options: {},};const cached = await options.cache?.get(cacheInput);- if (options.signal.aborted) throw new EvaluationFailure("cancelled");+ assertActive();if (cached &&Object.keys(cached).length === Object.keys(request.questions).length &&@@ -86,7 +92,7 @@ export function createEvaluator(options: {return cached;}for (let attempt = 0; attempt < 3; attempt++) {- if (options.signal.aborted) throw new EvaluationFailure("cancelled");+ assertActive();if (requests >= (options.requestLimit ?? 50_000))throw new EvaluationFailure("request-limit");const wait = Math.max(@@ -96,8 +102,11 @@ export function createEvaluator(options: {);if (wait) {try {- await delay(wait, undefined, { signal: options.signal });+ await delay(wait, undefined, {+ signal: AbortSignal.any([options.signal, authenticationFailure.signal]),+ });} catch {+ assertActive();throw new EvaluationFailure("cancelled");}}@@ -108,6 +117,7 @@ export function createEvaluator(options: {maxRetries: 0,abortSignal: AbortSignal.any([options.signal,+ authenticationFailure.signal,AbortSignal.timeout(options.timeoutMs ?? 30_000),]),});@@ -117,14 +127,17 @@ export function createEvaluator(options: {await options.cache?.put(cacheInput, scores);return scores;} catch (error) {- if (options.signal.aborted) throw new EvaluationFailure("cancelled");+ assertActive();if (requests >= (options.requestLimit ?? 50_000))throw new EvaluationFailure("request-limit");const status =error && typeof error === "object" && "statusCode" in error? error.statusCode: undefined;- if (status === 401 || status === 403) throw new EvaluationFailure("authentication");+ if (status === 401 || status === 403) {+ authenticationFailure.abort();+ throw new EvaluationFailure("authentication");+ }if (typeof status === "number" &&status >= 400 &&diff --git a/packages/core/src/retrieve.ts b/packages/core/src/retrieve.tsindex adaf1e4..c772ec5 100644--- a/packages/core/src/retrieve.ts+++ b/packages/core/src/retrieve.ts@@ -225,6 +225,7 @@ export async function retrieve(input: SearchInput, evaluator: Evaluator): Promisreturn result.snapshot;}issue(result.status === "issue" ? result.issue.kind : "changed");+ if (result.status === "issue" && result.issue.kind === "interrupted") return;const prior = files.get(candidate.path);if (prior)files.set(candidate.path, {@@ -243,7 +244,7 @@ export async function retrieve(input: SearchInput, evaluator: Evaluator): Promisconst results = await Promise.allSettled(Array.from({ length: Math.min(8, items.length) }, async () => {try {- while (next < items.length && !stop) await work(items[next++]!);+ while (next < items.length && !stop && !input.signal.aborted) await work(items[next++]!);} catch (error) {stop = true;throw error;@@ -306,7 +307,10 @@ export async function retrieve(input: SearchInput, evaluator: Evaluator): Promiscandidate.score,evaluator,async () => {- if (!(await unchanged(candidate))) return null;+ if (input.signal.aborted) throw new EvaluationFailure("cancelled");+ const current = await unchanged(candidate);+ if (input.signal.aborted) throw new EvaluationFailure("cancelled");+ if (!current) return null;return { evidence: await evidence?.() };},files.get(candidate.path),@@ -317,6 +321,7 @@ export async function retrieve(input: SearchInput, evaluator: Evaluator): Promisawait select();const evidence: Evidence[] = [];for (const candidate of ordered) {+ if (stop || input.signal.aborted) break;if (!files.get(candidate.path)!.excerpts.length) continue;// Context donors obey the same current eligibility/hash check as target files.if (!(await unchanged(candidate))) continue;diff --git a/packages/core/src/selection.ts b/packages/core/src/selection.tsindex d3c04e1..34ded87 100644--- a/packages/core/src/selection.ts+++ b/packages/core/src/selection.ts@@ -103,38 +103,38 @@ export async function selectFile(if (pending.length) groups.push(pending);let invalidated = false;for (const group of groups) {- const prepared = prepare ? await prepare() : {};- if (prepared === null) {- invalidated = true;- selected.length = 0;- leads.clear();- break;- }- const first = Math.max(1, group[0]!.range.startLine - 8),- last = Math.min(lines.length, group.at(-1)!.range.endLine + 8);- const oversizedContext = [...lines.slice(0, 20), ...lines.slice(first - 1, last)].some(- (line) => Buffer.byteLength(line) > sourceUnitBytes,- );- // Line-only windows cannot describe a partial giant line; send only the parser's bounded byte spans.- const context =- group.some(partialLine) || oversizedContext- ? group- .map(- (unit) =>- `Source lines ${unit.range.startLine}-${unit.range.endLine}; source bytes ${unit.sourceByteStart}-${unit.sourceByteEnd}:\n${sourceForUnit(snapshot, unit)}`,- )- .join("\n")- : bytes.length <= 16000- ? snapshot.source- : `Opening context:\n${lines.slice(0, 20).join("\n")}\nSource lines ${first}-${last}:\n${lines.slice(first - 1, last).join("\n")}`;- const request = evidenceRequest(- query,- snapshot.path,- context,- group.map((unit) => ({ name: unit.name, ...unit.range })),- prepared.evidence,- );try {+ const prepared = prepare ? await prepare() : {};+ if (prepared === null) {+ invalidated = true;+ selected.length = 0;+ leads.clear();+ break;+ }+ const first = Math.max(1, group[0]!.range.startLine - 8),+ last = Math.min(lines.length, group.at(-1)!.range.endLine + 8);+ const oversizedContext = [...lines.slice(0, 20), ...lines.slice(first - 1, last)].some(+ (line) => Buffer.byteLength(line) > sourceUnitBytes,+ );+ // Line-only windows cannot describe a partial giant line; send only the parser's bounded byte spans.+ const context =+ group.some(partialLine) || oversizedContext+ ? group+ .map(+ (unit) =>+ `Source lines ${unit.range.startLine}-${unit.range.endLine}; source bytes ${unit.sourceByteStart}-${unit.sourceByteEnd}:\n${sourceForUnit(snapshot, unit)}`,+ )+ .join("\n")+ : bytes.length <= 16000+ ? snapshot.source+ : `Opening context:\n${lines.slice(0, 20).join("\n")}\nSource lines ${first}-${last}:\n${lines.slice(first - 1, last).join("\n")}`;+ const request = evidenceRequest(+ query,+ snapshot.path,+ context,+ group.map((unit) => ({ name: unit.name, ...unit.range })),+ prepared.evidence,+ );const answers = await evaluator.evaluate(request);const values = group.map((unit, index) => {const value = answers[`q${index}`];diff --git a/scripts/test-suite.sh b/scripts/test-suite.shindex edc41af..c895300 100644--- a/scripts/test-suite.sh+++ b/scripts/test-suite.sh@@ -3,3 +3,4 @@ set -euo pipefailbun test apps/cli/test packages/core/test test/reference test/gateway.test.ts test/retrieval.test.tsnode --experimental-strip-types --test test/parser/source.test.tsnode --test test/release.test.mjs+python3 -B -m unittest discover -s evals/implementation/swebench -p test_installed.pydiff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.mdindex 51f69ba..731fd40 100644--- a/specs/jevgrep/README.md+++ b/specs/jevgrep/README.md@@ -7,20 +7,23 @@ Jev classification through Vercel AI Gateway. This is the build plan; production## Next Agent PromptYou are implementing this spec. Status: **implementing**, updated **2026-09-25**.-Next: run the installed `jg` on the-retained official Requests task with Sol. Parser, filesystem, evaluator, selection,-cache and CLI components are integrated; slice acceptance remains open until its-full evidence exists. Reference HTTP replay and the Node-only installed fixture-journey pass. Two mutation regressions now verify that invalidated evidence is-removed and reported incomplete. The default test runner passes under non-root, read-only Docker isolation; see+Next: finish the reviewed failure fixes, then integrate and measure the one+bounded source-output candidate on the saved Requests task. The installed Requests+checkpoint is complete: official solve passed, but Sol cost exceeded baseline.+See [the paired trace and billing report](assets/installed-requests-checkpoint.md).+The result proves integration, not the final quality gate.++Parser and filesystem gates pass. Installed cache mutation, failure recovery,+real timeout and head-pipe journeys have passed; the latest authentication-wait+and pipeline-cleanup corrections are undergoing focused verification. See[integrated verification](assets/integration-verification.md).-Priority after that checkpoint: fault/cache coverage, measured output policy,-frozen ten-task confirmation, and release verification. Never rerun saved-baselines. The current source allocation is uncapped: the accepted reference-validated its budget flag but did not enforce it. A cap needs slice 07 evidence.-See [verification evidence](assets/reference-verification.md),-[contracts](contracts.md), and the [decision map](map.md).+Priority: finish fault/cache acceptance, measure source allocation, freeze one+policy for all ten tasks, then complete supported-platform and release gates.+Never rerun saved baselines. Production source remains uncapped until slice 07+has official grade and full cost evidence. Jev API cost is reported separately+and excluded from scored Sol cost. See [contracts](contracts.md) and the+[decision map](map.md).No user decision blocks starting. External verification dependencies are a workingDocker daemon, explicit Gateway credentials for live tests, retained official task@@ -36,8 +39,8 @@ not claims about the current scaffold.- [x] [01 — Reference and real HTTP fixture](slices/01-reference.md)- [x] [02 — Bundled parser parity](slices/02-parser.md)-- [ ] [03 — Filesystem eligibility and snapshots](slices/03-filesystem.md)-- [ ] [04 — Installed CLI + skill + first real task](slices/04-checkpoint.md)+- [x] [03 — Filesystem eligibility and snapshots](slices/03-filesystem.md)+- [x] [04 — Installed CLI + skill + first real task](slices/04-checkpoint.md)- [ ] [05 — Faults, cancellation and partial results](slices/05-failures.md)- [ ] [06 — Default-on fresh cache](slices/06-cache.md)- [ ] [07 — Measured source budget](slices/07-output-policy.md)diff --git a/specs/jevgrep/assets/installed-requests-checkpoint.md b/specs/jevgrep/assets/installed-requests-checkpoint.mdnew file mode 100644index 0000000..ce93dc8--- /dev/null+++ b/specs/jevgrep/assets/installed-requests-checkpoint.md@@ -0,0 +1,76 @@+# Installed Requests checkpoint++The frozen installed candidate solved `psf__requests-1142` under the official+SWE-bench grader. This is an integration/quality pass and a cost loss, not a cohort+acceptance result. The saved baseline was reused without execution.++| | Installed jg | Fixed baseline |+| --- | ---: | ---: |+| Officially resolved | yes | yes |+| Full Sol cost | $0.3491436 | $0.2685004 |+| Sol generations | 13 | 11 |++All treatment generations have matching billing metadata. Cost increased+$0.0806432 (30.03%). Jev cost and tokens are excluded. The agent invoked `jg` once;+its 85 Jev HTTP attempts all returned 200. The CLI packet was 2,852 UTF-8 bytes.+The implementation patch is identical to baseline; treatment added broader tests.++Both arms use `openai/gpt-5.6-sol`, medium effort, the same 900-second deadline,+Codex 0.153.4 and the retained runtime/source identities. The treatment's mandatory+initial retrieval instruction and installed canonical skill are explicit arm+changes. Candidate package SHA256:+`aebb98108817285b21b0dced87786e4060dd28a1509a2ac3236c3d3689217be7`.++## Exact observed retrieval++Sol's query was:++> requests.get always adds Content-Length header; expected GET requests with no body to omit automatically generated Content-Length, while preserving body/header behavior. Find request preparation, header calculation helpers, callers, and tests.++The packet supplied `requests/models.py:231–256,385–414` and+`test_requests.py:18–26`, plus six file leads. It exposed the unconditional header+assignment and authentication recalculation immediately. `prepare_body` appeared+as a lead rather than source. The test excerpt contained HTTPBIN/helper/class+scaffolding, not behavioral assertions.++Sol recognized the defect, then read broader models/test source and inspected+adapters, utilities, structures and setup. Its first bundled inspection failed+before producing output; it recovered with separate commands. It ran verification+twice, with a test edit between the successful runs. This was not unchanged-test+repetition. Treatment had 24 command executions and 48,127 output bytes versus+baseline's 9 and 42,168; some commands were parallel, so these are not model-call+counts.++Thin surrounding/test context and broad optional leads are hypotheses for future+presentation work. Additional guidance searches, recovery from the failed command,+source reads and test elaboration are observed agent behavior. One pair cannot+assign the cost increase causally or eliminate ordinary model variance.++## Retained evidence++Local ignored study: `evals/runs/swebench/installed-jg-requests-checkpoint-v3/`.+It contains the frozen plan, npm package, installed prefix and safe input export.+Its `attempt/` retains raw native rollout/events, `jg-stdout.txt`, raw Jev bodies,+patch, official grading console/receipt and generation lookups/accounting.+Baseline: `evals/runs/swebench/lookahead-native-v88/psf__requests-1142/codex-baseline/`.+Official report run: `jg-installed-psf__requests-1142-8cd1b55070bf`.++A separate live installed query completed first in+`evals/runs/swebench/installed-jg-live-query-v1/`. Its Docker launch stderr contains+only the initial Node image pull; the application output is the saved stdout.+The maintained runner and this compact report are versioned; raw task artifacts+remain local. No task, baseline or failure was dropped from the result.++### Separately observed Jev API cost++All 85 retained Jev responses include Gateway cost metadata: summed reported cost+is **$0.011889360**, with 283,080 input tokens and 8,963 output tokens. Sol cost is+$0.3491436; including Jev gives $0.361032960. This is response-reported API cost,+not an invoice reconciliation. The raw sum is retained in the attempt's+`jev-accounting.json`. Jev stays excluded from the scored task-cost metric per+user policy. The earlier accounting field `jev_cost_usd: 0` represented that+scoring exclusion; it was not a measured zero API charge.++The 85 client calls also contain 100 internal Gateway provider attempts, so the+previous statement that all calls returned HTTP 200 must not be read as saying+there were no internal provider failures or fallbacks.diff --git a/specs/jevgrep/assets/integration-verification.md b/specs/jevgrep/assets/integration-verification.mdindex ed9d067..3a516d6 100644--- a/specs/jevgrep/assets/integration-verification.md+++ b/specs/jevgrep/assets/integration-verification.md@@ -65,3 +65,27 @@ duration and confirms three attempts, then a classified provider failure.`bun run test:parser` is now the combined named gate: conformance in Node pluspack/install and actual Python units in a runtime without development prerequisites.It passed all 15 conformance tests and both selected installed journeys.++## Installed failure and cache checkpoint++The installed failure selector passed all ten selected journeys, including the+actual three 30-second stalled HTTP attempts, malformed answers/JSON, retry+recovery, rate limits, SIGINT, and `head -200`. Separate installed journeys cover+cold/warm identical output, no-cache, changed query, add/delete/ignore changes,+same-size edits with restored mtime, failed-answer recovery and cache clear.+The named filesystem gate also passed the installed wide/deep plain-root and+nested-repository fixture with excluded sentinels and unreadable source.++Independent Codex review found two confirmed remaining failure-path defects:+authentication did not abort a sibling retry delay, and a timeout of the shell+pipeline could leave descendants holding pipes. The evaluator now interrupts both+HTTP work and retry sleeps with the shared authentication signal. Test processes+have their own process group, so bounded cleanup covers the whole pipeline.+A subprocess probe verified descendant pipes close on group termination.+The reviewer could not bind localhost in its sandbox; its in-memory/subprocess+probes were reproduced by the parent, whose Docker HTTP gate passed.++The focused Docker gateway/retrieval/selection gate passes 22 tests and 118+assertions, including preserving already-selected source when cancellation occurs+between declaration groups. Typecheck and lint pass. These failure fixes preserve+the healthy request construction used by the retained Requests checkpoint.diff --git a/specs/jevgrep/choices.md b/specs/jevgrep/choices.mdindex 900f72d..490055d 100644--- a/specs/jevgrep/choices.md+++ b/specs/jevgrep/choices.md@@ -114,3 +114,27 @@ selected MIT for Jevgrep itself.Gap: the spec required licenses but left collection mechanics open. Reach: anew dependency can require a verified notice update, while ordinary builds needno network license lookup. Verdict: sound. Confidence: high.++### Share fatal authentication cancellation across a search++When: installed failure acceptance. If one request is rejected for its API key,+other requests using the same key stop, including requests sleeping before a retry.+The evaluator owns one shared cancellation signal; the caller's cancellation+remains distinguishable. Otherwise a sibling could spend its entire retry delay+waiting after the search already knows the key cannot work.++Gap: stopping on global authentication failure was required; the shared signal+and its scope were implementation choices. Reach: future retry paths must obey+the same evaluator cancellation. Verdict: sound. Confidence: high.++### Keep verified partial source when the caller interrupts selection++When: installed failure acceptance. A user can interrupt after some declaration+groups have returned useful source. That interruption ends further work and marks+the result incomplete; it does not itself mean the source changed. Actual changed+or newly excluded snapshots still discard their evidence.++Gap: the filesystem interruption and invalidation paths shared a preparation+boundary, so the implementation had to distinguish their effect on retained data.+Reach: partial output remains useful without claiming complete discovery or a+filesystem lock. Verdict: sound. Confidence: high.diff --git a/specs/jevgrep/slices/03-filesystem.md b/specs/jevgrep/slices/03-filesystem.mdindex ef18805..2215761 100644--- a/specs/jevgrep/slices/03-filesystem.md+++ b/specs/jevgrep/slices/03-filesystem.md@@ -1,6 +1,6 @@# Make all retrieval reads eligible and consistent-Status: planned. Depends on: 01. Read [contracts](../contracts.md) first.+Status: verified. Depends on: 01. Read [contracts](../contracts.md) first.## Contract and owner@@ -35,3 +35,5 @@ Show the artifact and summarize deviations. This is a non-blocking review checkpcontinue on the evidence if the user does not respond. Feedback that changes thepublic contract or acceptance measure requires updating this slice before broadeningimplementation. Record new choices and update the README handoff before ending.++The named Docker filesystem gate passed, including the installed wide/deep plain-root and nested-repository journey. See [integration evidence](../assets/integration-verification.md).diff --git a/specs/jevgrep/slices/04-checkpoint.md b/specs/jevgrep/slices/04-checkpoint.mdindex 07add7f..86a6836 100644--- a/specs/jevgrep/slices/04-checkpoint.md+++ b/specs/jevgrep/slices/04-checkpoint.md@@ -1,6 +1,6 @@# Deliver the installed CLI and agent workflow-Status: planned. Depends on: 01, 02, 03. Read [contracts](../contracts.md) first.+Status: verified (integration only). Depends on: 01, 02, 03. Read [contracts](../contracts.md) first.## Contract and owner@@ -8,7 +8,7 @@ Compose core `retrieve` and the CLI contract. Port hierarchy, file admission, soCLI owns credentials, rendering, stdout-only errors, cancellation and exit codes. Core result carries selected versus rendered ranges and completeness. Ship `jg skill` and the canonical `skills/jevgrep/SKILL.md` following write-skills. Production invocation is selective, benchmark wrapper forces first retrieval. Implement initial finite evaluator attempts and 50k counter now; the later fault slice expands conformance rather than allowing runaway work here.-Build/pack/install outside checkout. First checkpoint deliberately has no persistent cache yet; do not advertise final cache behavior until slice 06. This is feature staging, not a temporary second retrieval implementation. Start with the accepted source allocation, retaining all qualifying paths.+Build/pack/install outside checkout. Cache was integrated through the single evaluator before this checkpoint; slice 06 owns its full freshness acceptance. Start with the accepted source allocation, retaining all qualifying paths.## Human-runnable artifact@@ -29,7 +29,7 @@ Internal naming and command implementation, packaging mechanics; public contract## Keep green-Parser/filesystem/reference checks; no automatic report file. Cache is the only intentionally unshipped final capability at this checkpoint.+Parser/filesystem/reference checks; no automatic report file. Later slices own remaining fault, cache, output and cohort acceptance.## Review@@ -37,3 +37,5 @@ Show the artifact and summarize deviations. This is a non-blocking review checkpcontinue on the evidence if the user does not respond. Feedback that changes thepublic contract or acceptance measure requires updating this slice before broadeningimplementation. Record new choices and update the README handoff before ending.++The installed fixture, live query and official Requests task completed. The task solved but cost more than its saved baseline. Exact query, output, follow-up behavior, grade and billing are in the [checkpoint report](../assets/installed-requests-checkpoint.md). Reproduction uses the maintained [runner workflow](../../../evals/implementation/swebench/installed.md); the frozen checkpoint retains its original runner.diff --git a/test/Dockerfile b/test/Dockerfileindex 446e0f2..71fbc3e 100644--- a/test/Dockerfile+++ b/test/Dockerfile@@ -18,6 +18,7 @@ RUN bun scripts/build-cli.tsRUN mkdir -p /opt/npm-cache && cd apps/cli && npm pack --ignore-scripts --pack-destination /opt && npm install --prefix /opt/npm-warm --cache /opt/npm-cache --ignore-scripts --no-audit --no-fund /opt/dzhng-jevgrep-*.tgz && chmod -R a+rX /opt/npm-cacheENV npm_config_cache=/opt/npm-cache npm_config_offline=trueCOPY evals/implementation/swebench/hierarchy-unit-locators-spike.ts evals/implementation/swebench/source-method-windows-spike.ts evals/implementation/swebench/source-method-declarations-spike.py evals/implementation/swebench/content-handoff-v47-spike.ts evals/implementation/swebench/content-handoff-v47-spike.py evals/implementation/swebench/complete-source-v64-spike.ts evals/implementation/swebench/preview-fragments-v51-spike.ts evals/implementation/swebench/source-neighborhood-spike.py evals/implementation/swebench/+COPY evals/implementation/swebench/installed.py evals/implementation/swebench/gateway_broker.py evals/implementation/swebench/test_installed.py evals/implementation/swebench/RUN bun build evals/implementation/swebench/hierarchy-unit-locators-spike.ts --target=node --outfile=/opt/jevgrep-reference.mjsENV JEVGREP_TEST_IN_DOCKER=1 HOME=/tmp/jevgrep-home XDG_CONFIG_HOME=/tmp/jevgrep-config XDG_CACHE_HOME=/tmp/jevgrep-cache AI_GATEWAY_API_KEY=""CMD ["bun", "test", "test/reference/replay.test.ts"]diff --git a/test/gateway.test.ts b/test/gateway.test.tsindex 8a36e1d..d27fc37 100644--- a/test/gateway.test.ts+++ b/test/gateway.test.ts@@ -155,7 +155,7 @@ test("stalled HTTP attempts time out without exceeding the evaluator attempt limapiKey: "fixture",baseURL: `http://127.0.0.1:${server.port}`,signal: new AbortController().signal,- timeoutMs: 20,+ timeoutMs: 100,retryDelayMs: 0,});await expect(@@ -170,3 +170,84 @@ test("stalled HTTP attempts time out without exceeding the evaluator attempt limserver.stop(true);}}, 2000);++test("authentication failure stops other in-flight and subsequent query requests", async () => {+ let calls = 0;+ const server = Bun.serve({+ port: 0,+ fetch() {+ calls++;+ if (calls === 1) return new Promise<Response>(() => {});+ if (calls === 2) return Response.json({ error: "unauthorized" }, { status: 401 });+ return Response.json({ answers: { q: { type: "boolean", probability: 0.8 } } });+ },+ });+ try {+ const evaluator = createEvaluator({+ apiKey: "fixture",+ baseURL: `http://127.0.0.1:${server.port}`,+ signal: new AbortController().signal,+ timeoutMs: 1000,+ retryDelayMs: 0,+ });+ const request = {+ state: "test",+ questions: { q: { type: "boolean" as const, instructions: "Relevant?" } },+ };+ const outcomes = await Promise.allSettled([+ evaluator.evaluate(request),+ evaluator.evaluate(request),+ ]);+ expect(outcomes).toMatchObject([+ { status: "rejected", reason: { kind: "authentication" } },+ { status: "rejected", reason: { kind: "authentication" } },+ ]);+ await expect(evaluator.evaluate(request)).rejects.toMatchObject({ kind: "authentication" });+ expect(calls).toBe(2);+ } finally {+ server.stop(true);+ }+}, 2000);++test("authentication failure interrupts a sibling Retry-After wait", async () => {+ let calls = 0;+ let releaseAuthentication!: () => void;+ const authenticationReady = new Promise<void>((resolve) => {+ releaseAuthentication = resolve;+ });+ const server = Bun.serve({+ port: 0,+ async fetch() {+ calls++;+ if (calls === 1) {+ await authenticationReady;+ return Response.json({ error: "unauthorized" }, { status: 401 });+ }+ setTimeout(releaseAuthentication, 50);+ return Response.json({ error: "limited" }, { status: 429, headers: { "retry-after": "30" } });+ },+ });+ try {+ const evaluator = createEvaluator({+ apiKey: "fixture",+ baseURL: `http://127.0.0.1:${server.port}`,+ signal: new AbortController().signal,+ retryDelayMs: 0,+ });+ const request = {+ state: "test",+ questions: { q: { type: "boolean" as const, instructions: "Relevant?" } },+ };+ const outcomes = await Promise.allSettled([+ evaluator.evaluate(request),+ evaluator.evaluate(request),+ ]);+ expect(outcomes).toMatchObject([+ { status: "rejected", reason: { kind: "authentication" } },+ { status: "rejected", reason: { kind: "authentication" } },+ ]);+ expect(calls).toBe(2);+ } finally {+ server.stop(true);+ }+}, 2000);diff --git a/test/installed.test.mjs b/test/installed.test.mjsindex 726e4f1..7c795ba 100644--- a/test/installed.test.mjs+++ b/test/installed.test.mjs@@ -2,7 +2,19 @@ import { test } from "node:test";import assert from "node:assert/strict";import { spawn, spawnSync } from "node:child_process";import { createServer } from "node:http";-import { access, mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";+import {+ access,+ chmod,+ mkdir,+ mkdtemp,+ readFile,+ realpath,+ rm,+ stat,+ symlink,+ utimes,+ writeFile,+} from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";@@ -26,18 +38,27 @@ const source = (branch, name) =>`# Synthetic installed-package fixture\nclass ${name}:\n """Records an event in the ${branch} package."""\n @staticmethod\n def record_event(value):\n return "py-evidence-${branch}:" + value\n\n def unrelated():\n return "unrelated"\n`;async function context(t, mode = "healthy") {+ let expectedQuery = query;const scratch = await mkdtemp(join(tmpdir(), "jg-installed-"));const tree = join(scratch, "repository");const home = join(scratch, "home");const config = join(scratch, "config");const cache = join(scratch, "cache");const children = new Set();+ const signalChild = (child, signal) => {+ if (!child.pid) return;+ try {+ process.kill(-child.pid, signal);+ } catch (error) {+ if (error.code !== "ESRCH") throw error;+ }+ };const requests = [];const protocolErrors = [];let malformedResponses = 0;let server;t.after(async () => {- for (const child of children) child.kill("SIGKILL");+ for (const child of children) signalChild(child, "SIGKILL");if (server) {server.closeAllConnections();await new Promise((resolve) => server.close(resolve));@@ -93,7 +114,12 @@ async function context(t, mode = "healthy") {assert.ok(typeof question.instructions === "string" && question.instructions.length > 0);}assert.ok(requests.length < 256, "Synthetic search stopped making bounded forward progress");- requests.push({ body, raw });+ requests.push({ body, raw, receivedAt: performance.now() });+ if (mode === "rate-limit" && requests.length === 1) {+ response.writeHead(429, { "content-type": "application/json", "retry-after": "1" });+ response.end(JSON.stringify({ error: "fixture rate limit" }));+ return;+ }if (mode === "disconnect" && requests.length === 1) {response.destroy();return;@@ -103,8 +129,9 @@ async function context(t, mode = "healthy") {response.end("{broken");return;}+ if (mode === "stalled") return;if (mode === "interrupt") {- for (const child of children) child.kill("SIGINT");+ for (const child of children) signalChild(child, "SIGINT");return;}if (mode === "transient" && requests.length === 1) {@@ -119,7 +146,7 @@ async function context(t, mode = "healthy") {}let probabilities;if (Array.isArray(body.state.items)) {- assert.equal(body.state.query, query);+ assert.equal(body.state.query, expectedQuery);probabilities = body.state.items.map((item) =>mode === "negative"? 0.05@@ -183,25 +210,33 @@ async function context(t, mode = "healthy") {AI_GATEWAY_API_KEY: fixtureKey,AI_GATEWAY_BASE_URL: `http://127.0.0.1:${server.address().port}/v4/ai`,};- const run = async (args, overrides = {}) => {+ const run = async (args, overrides = {}, head = false) => {const result = await new Promise((resolve, reject) => {- const child = spawn(binary, args, {- cwd: tree,- env: { ...env, ...overrides },- stdio: ["ignore", "pipe", "pipe"],- });+ const child = spawn(+ head ? "bash" : binary,+ head ? ["-o", "pipefail", "-c", '"$@" | head -200', "jg-pipe", binary, ...args] : args,+ {+ cwd: tree,+ detached: true,+ env: { ...env, ...overrides },+ stdio: ["ignore", "pipe", "pipe"],+ },+ );children.add(child);const stdout = [],stderr = [];let outputBytes = 0;- const timer = setTimeout(() => child.kill("SIGKILL"), 45_000);+ const timer = setTimeout(+ () => signalChild(child, "SIGKILL"),+ mode === "stalled" ? 120_000 : 45_000,+ );for (const [stream, chunks] of [[child.stdout, stdout],[child.stderr, stderr],])stream.on("data", (chunk) => {outputBytes += chunk.length;- if (outputBytes > 2_000_000) child.kill("SIGKILL");+ if (outputBytes > 2_000_000) signalChild(child, "SIGKILL");else chunks.push(chunk);});child.once("error", (error) => {@@ -235,6 +270,12 @@ async function context(t, mode = "healthy") {run,tree,requests,+ set query(value) {+ expectedQuery = value;+ },+ set mode(value) {+ mode = value;+ },get malformedResponses() {return malformedResponses;},@@ -353,6 +394,9 @@ test("warm cache avoids HTTP, no-cache bypasses reuse, and edited content invalibefore,"Warm search must reuse every identical successful evaluation",);+ t.diagnostic(+ `Cold HTTP requests: ${before}; warm additional requests: ${fixture.requests.length - before}; stdout identical.`,+ );complete(await fixture.run([query, fixture.tree, "--no-cache"]));assert.ok(fixture.requests.length > before, "--no-cache must reach the HTTP fixture");const edited = join(fixture.tree, "alpha/nested/first.py");@@ -437,3 +481,153 @@ test("a disconnected provider request retries without losing installed source evassert.deepEqual(fixture.requests[0].body, fixture.requests[1].body);for (const [branch] of branches) assert.ok(result.stdout.includes(`py-evidence-${branch}:`));});++test("failed provider answers are retried after recovery rather than reused from cache", async (t) => {+ const fixture = await context(t, "invalid-json");+ assert.equal((await fixture.run([query])).code, 2);+ const failedRequests = fixture.requests.length;+ fixture.mode = "healthy";+ const recovered = await fixture.run([query]);+ assert.equal(recovered.code, 0);+ assert.ok(fixture.requests.length > failedRequests);+ for (const [branch] of branches) assert.ok(recovered.stdout.includes(`py-evidence-${branch}:`));+});++test("cache observes additions, deletions, ignore changes, and same-size edits with restored mtime", async (t) => {+ const fixture = await context(t);+ complete(await fixture.run([query]));+ const added = "delta/nested/fourth.py";+ await mkdir(join(fixture.tree, "delta/nested"), { recursive: true });+ await writeFile(join(fixture.tree, added), source("delta", "CollectorDelta"));+ let before = fixture.requests.length;+ const addition = await fixture.run([query]);+ assert.equal(addition.code, 0);+ assert.ok(addition.stdout.includes("py-evidence-delta:"));+ assert.ok(fixture.requests.slice(before).some(({ raw }) => raw.includes("py-evidence-delta:")));++ await rm(join(fixture.tree, "alpha/nested/first.py"));+ const deletion = await fixture.run([query]);+ assert.equal(deletion.code, 0);+ assert.ok(!deletion.stdout.includes("alpha/nested/first.py"));+ assert.ok(!deletion.stdout.includes("py-evidence-alpha:"));++ await writeFile(join(fixture.tree, ".ignore"), "ignored/\n*.skip.py\nbeta/\n");+ before = fixture.requests.length;+ const ignored = await fixture.run([query]);+ assert.equal(ignored.code, 0);+ assert.ok(!ignored.stdout.includes("py-evidence-beta:"));+ assert.ok(!fixture.requests.slice(before).some(({ raw }) => raw.includes("py-evidence-beta:")));++ const path = join(fixture.tree, "gamma/nested/third.py");+ const original = await stat(path);+ await writeFile(+ path,+ source("gamma", "CollectorGamma").replace("py-evidence-gamma:", "py-evidence-GAMMA:"),+ );+ await utimes(path, original.atime, original.mtime);+ before = fixture.requests.length;+ const edited = await fixture.run([query]);+ assert.equal(edited.code, 0);+ assert.ok(edited.stdout.includes("py-evidence-GAMMA:"));+ assert.ok(!edited.stdout.includes("py-evidence-gamma:"));+ assert.ok(fixture.requests.slice(before).some(({ raw }) => raw.includes("py-evidence-GAMMA:")));+});++test("filesystem policy survives wide and deep installed traversal with excluded sentinels", async (t) => {+ const fixture = await context(t);+ const wide = join(fixture.tree, "wide");+ await mkdir(wide);+ for (let index = 0; index < 129; index++) {+ await writeFile(join(wide, `${index}.skip.py`), forbidden);+ }+ const relative = ["wide", ...Array.from({ length: 12 }, (_, i) => `d${i}`), "odd\nname.py"].join(+ "/",+ );+ const parent = join(fixture.tree, relative.slice(0, relative.lastIndexOf("/")));+ await mkdir(parent, { recursive: true });+ await writeFile(join(fixture.tree, relative), source("wide", "CollectorWide"));+ await writeFile(join(wide, ".hidden.py"), forbidden);+ await writeFile(join(wide, "binary.py"), Buffer.from(`\0${forbidden}`));+ await writeFile(join(wide, "key.txt"), `-----BEGIN PRIVATE KEY-----\n${forbidden}`);+ const outside = join(fixture.tree, "..", "outside.py");+ await writeFile(outside, forbidden);+ await symlink(outside, join(wide, "escape.py"));+ await symlink(wide, join(wide, "cycle"));+ assert.equal(spawnSync("mkfifo", [join(wide, "pipe")]).status, 0);+ await writeFile(join(wide, "unreadable.py"), forbidden);+ await chmod(join(wide, "unreadable.py"), 0);++ const nested = join(fixture.tree, "nested-repo");+ await mkdir(nested);+ await writeFile(join(fixture.tree, ".gitignore"), "nested-repo/*.py\n");+ await writeFile(join(nested, ".git"), "gitdir: elsewhere\n");+ await writeFile(join(nested, ".gitignore"), "blocked_local.py\n");+ await writeFile(+ join(fixture.tree, ".ignore"),+ "ignored/\n*.skip.py\nnested-repo/blocked_parent.py\n",+ );+ await writeFile(join(nested, "allowed.py"), source("nested", "CollectorNested"));+ await writeFile(join(nested, "blocked_local.py"), forbidden);+ await writeFile(join(nested, "blocked_parent.py"), forbidden);++ const result = await fixture.run([query]);+ assert.equal(result.code, 2);+ assert.match(result.stdout, /Issue: "unreadable"/);+ assert.ok(result.stdout.includes("py-evidence-nested:"));+ assert.ok(result.stdout.includes(JSON.stringify(relative)));+ assert.ok(result.stdout.includes("py-evidence-wide:"));+ for (const [branch] of branches) assert.ok(result.stdout.includes(`py-evidence-${branch}:`));+ assert.ok(fixture.requests.some(({ raw }) => raw.includes("py-evidence-wide:")));+});++test("a changed query cannot reuse another query's cached evaluations", async (t) => {+ const fixture = await context(t);+ complete(await fixture.run([query]));+ const before = fixture.requests.length;+ const changedQuery = "Find recording methods and their event return values.";+ fixture.query = changedQuery;+ complete(await fixture.run([changedQuery]));+ assert.ok(fixture.requests.length > before);+ assert.ok(fixture.requests.slice(before).every(({ body }) => body.state.query === changedQuery));+});++test(+ "a stalled provider response exhausts bounded installed timeouts",+ { timeout: 130_000 },+ async (t) => {+ const fixture = await context(t, "stalled");+ await rm(fixture.tree, { recursive: true });+ await mkdir(fixture.tree);+ await writeFile(join(fixture.tree, "only.py"), source("only", "CollectorOnly"));+ const result = await fixture.run([query]);+ assert.equal(result.code, 2);+ assert.match(result.stdout, /Status: incomplete/);+ assert.equal(fixture.requests.length, 3);+ },+);++test("a head -200 consumer closes the stdout pipe without leaving jg running", async (t) => {+ const fixture = await context(t);+ await rm(fixture.tree, { recursive: true });+ await mkdir(fixture.tree);+ const large = Array.from(+ { length: 500 },+ (_, i) =>+ `class Collector${i}${"A".repeat(96)}:\n def record_event(self):\n return "py-evidence-pipe"\n`,+ ).join("\n");+ await writeFile(join(fixture.tree, "only.py"), large);+ const result = await fixture.run([query], {}, true);+ assert.equal(result.code, 0);+ assert.match(result.stdout, /Status: complete/);+ assert.equal(result.stdout.trimEnd().split("\n").length, 200);+ assert.ok(!result.stdout.includes("End context."));+});++test("a rate-limited provider retry waits and recovers the installed evidence", async (t) => {+ const fixture = await context(t, "rate-limit");+ const result = await fixture.run([query]);+ complete(result);+ assert.deepEqual(fixture.requests[0].body, fixture.requests[1].body);+ assert.ok(fixture.requests[1].receivedAt - fixture.requests[0].receivedAt >= 950);+ assert.equal(fixture.requests.filter(({ raw }) => raw === fixture.requests[0].raw).length, 2);+});diff --git a/test/retrieval.test.ts b/test/retrieval.test.tsindex 1dae3e7..5226a60 100644--- a/test/retrieval.test.ts+++ b/test/retrieval.test.ts@@ -272,3 +272,42 @@ testIfDocker(}},);++testIfDocker(+ "cancellation between declaration groups preserves already selected source",+ async () => {+ const root = await mkdtemp(join(tmpdir(), "jg-partial-cancel-"));+ const controller = new AbortController();+ let groups = 0;+ try {+ await writeFile(+ join(root, "events.ts"),+ Array.from({ length: 20 }, (_, i) => `export function event${i}() {return ${i};}\n`).join(+ "",+ ),+ );+ const result = await retrieve(+ { root, query: "event behavior", signal: controller.signal },+ {+ requests: 0,+ async evaluate(request) {+ if ((request.state as { declarations?: unknown[] }).declarations) {+ groups++;+ controller.abort();+ }+ return Object.fromEntries(Object.keys(request.questions).map((id) => [id, 0.9]));+ },+ },+ );+ expect(result.status).toBe("interrupted");+ expect(groups).toBe(1);+ const source = result.files+ .flatMap((file) => file.excerpts.map((excerpt) => excerpt.source))+ .join("\n");+ expect(source).toContain("function event0");+ expect(source).not.toContain("function event19");+ } finally {+ await rm(root, { recursive: true, force: true });+ }+ },+);