Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

Record platform and full verification with unresolved spike parity

David Zhangf9854bfddec0
diff --git a/package.json b/package.json
index 4974785..eb7388d 100644
--- a/package.json
+++ b/package.json
@@ -22,12 +22,13 @@
"format": "oxfmt --write .",
"format:check": "oxfmt --check .",
"test:reference": "bash scripts/test-reference.sh",
- "test:parser": "bash scripts/test-reference.sh node --experimental-strip-types --test test/parser/source.test.ts && bash scripts/test-installed.sh --test-name-pattern \"runtime has|parses Python\"",
+ "test:parser": "bash scripts/test-reference.sh node --experimental-strip-types --test test/parser/source.test.ts && bash scripts/test-installed.sh --test-name-pattern \"runtime has|local commands|parses Python\"",
"test:filesystem": "bash scripts/test-reference.sh bun test packages/core/test/filesystem.test.ts test/retrieval.test.ts && bash scripts/test-installed.sh --test-name-pattern \"filesystem policy\"",
"test:installed": "bash scripts/test-installed.sh",
"test:e2e": "bun run test:installed",
"verify": "bun run check-types && bun run lint && bun run test && bun run test:installed",
- "eval:swebench": "python3 evals/implementation/swebench/installed.py"
+ "eval:swebench": "python3 evals/implementation/swebench/installed.py",
+ "test:native": "node scripts/test-native.mjs"
},
"devDependencies": {
"oxfmt": "0.57.0",
diff --git a/scripts/test-installed.sh b/scripts/test-installed.sh
index 026911b..7eada81 100755
--- a/scripts/test-installed.sh
+++ b/scripts/test-installed.sh
@@ -48,4 +48,6 @@ node_args=(node --test --test-concurrency=1)
if [[ -n "$pattern" ]]; then node_args+=(--test-name-pattern "$pattern"); fi
node_args+=(/test/installed.test.mjs)
docker build "${platform[@]}" "${build_args[@]}" -f "$context_dir/test/runtime.Dockerfile" -t "$image" "$context_dir"
-docker run --rm --network none --read-only --tmpfs /tmp:rw,nosuid,nodev,size=256m "${platform[@]}" "$image" "${node_args[@]}"
+docker run --rm --network none --read-only --cpus 2 --memory 2g --pids-limit 256 \
+ --cap-drop ALL --security-opt no-new-privileges \
+ --tmpfs /tmp:rw,nosuid,nodev,size=256m "${platform[@]}" "$image" "${node_args[@]}"
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.md
index 87fa50e..ae0b6db 100644
--- a/specs/jevgrep/README.md
+++ b/specs/jevgrep/README.md
@@ -15,9 +15,13 @@ Requests trial officially solved and reduced Sol cost against uncapped, but stil
cost more than baseline. See [source-budget decision](assets/source-budget-decision.md)
and [the uncapped trace report](assets/installed-requests-checkpoint.md).
-Slices 01–07 have their named evidence. Native macOS installed verification is
-being completed independently. Remaining work: frozen cohort acceptance, supported
-Linux architecture checks, release closeout, whole-spec review and close-spec.
+Slices 01–07 have their named evidence. Native macOS and both Linux architectures have
+focused [installed-runtime evidence](assets/runtime-verification.md). The full deterministic `verify` gate passes. Remaining work: diagnose the
+spike-versus-production expansion difference, finish frozen cohort acceptance,
+release closeout, whole-spec review and close-spec.
+The spike expands already expanded context on its second pass; production does
+not. Current helper/template tests do not establish full-pipeline parity. Diagnose
+this with identical-source/answer replay while the frozen cohort stays unchanged.
Never rerun saved baselines or pool candidates. Report Jev API cost separately;
only full Sol task cost is scored. See [integrated verification](assets/integration-verification.md),
[contracts](contracts.md), and [decision map](map.md).
diff --git a/specs/jevgrep/assets/integration-verification.md b/specs/jevgrep/assets/integration-verification.md
index f566494..bd54806 100644
--- a/specs/jevgrep/assets/integration-verification.md
+++ b/specs/jevgrep/assets/integration-verification.md
@@ -98,3 +98,17 @@ tarball additionally passes installed runtime, source-budget and head checks.
Maintained cohort accounting passes 14 Docker tests, including retained response
costs when transport logs are missing. Missing coverage leaves total Jev cost
unknown while preserving its known subtotal; Sol scoring is unchanged.
+
+## Full deterministic closeout gate
+
+`bun run verify` passed on the integrated source-budget/native-harness branch:
+62 Bun tests (402 assertions), 15 Node parser tests, three release-validator tests,
+14 maintained runner tests, and all 20 installed journeys. Typecheck and lint also
+passed. Installed containers now have CPU/memory/process ceilings and dropped
+capabilities, matching the spec's bounded test-isolation requirement. The native
+smoke uses the same portable journeys; the parser selector retains the split
+local-command case so its earlier coverage is not lost.
+
+This is deterministic correctness evidence. It does not resolve the known
+spike-versus-production context-expansion difference or substitute for the live
+cohort quality gate. See [runtime evidence](runtime-verification.md).
diff --git a/specs/jevgrep/assets/runtime-verification.md b/specs/jevgrep/assets/runtime-verification.md
new file mode 100644
index 0000000..971f72e
--- /dev/null
+++ b/specs/jevgrep/assets/runtime-verification.md
@@ -0,0 +1,31 @@
+# Installed runtime verification
+
+The frozen cohort archive has SHA-256
+`98bcf47e7de30ad82c8b452a86b958505c3befe6512d00f98486101c6a9fc802`.
+It contains the 1,500-byte source policy and canonical skill used by every
+prospective treatment. Its production source and skill were byte-compared with
+the committed implementation before removing the candidate worktree.
+
+Native macOS Apple Silicon: Darwin arm64 25.6.0, Node 24.14.0. The exact archive
+passed the two reused installed journeys: help/version/skill without credentials,
+and a real Python search over three hierarchy branches through a loopback HTTP
+Gateway fixture. npm installed outside the checkout with temporary HOME/XDG and
+only Node available on the runtime PATH. Removing the packaged grammar made search
+fail with exit 1 while local commands still passed. Independent Codex review found
+no actionable issue. This proves the printed Node/macOS combination, not every
+Node version on macOS.
+
+Linux amd64: the exact same archive passed runtime isolation/keyless commands,
+Python search and bounded-source preservation in the Node 22 Docker runtime.
+The installed image has no checkout or Python/Bun/compiler prerequisites. The
+corresponding Linux arm64 candidate journeys also passed during slice 07.
+
+Reproduce native evidence with `node scripts/test-native.mjs --prebuilt PACKAGE`.
+For Linux, set `JEVGREP_TEST_PLATFORM=linux/amd64` or `linux/arm64` and run
+`scripts/test-installed.sh --prebuilt PACKAGE`. The default installed suite is
+broader than this focused architecture check; its full closeout remains separate.
+
+The npm-facing README was added after the cohort archive was frozen. It documents
+setup and explicit skill installation; it does not change executable, grammar,
+dependencies or skill. Final packaging validation must include this documentation
+and exercise its exact archive before a later user-requested publication.
diff --git a/specs/jevgrep/assets/source-budget-decision.md b/specs/jevgrep/assets/source-budget-decision.md
index 2bbc797..765834d 100644
--- a/specs/jevgrep/assets/source-budget-decision.md
+++ b/specs/jevgrep/assets/source-budget-decision.md
@@ -66,3 +66,26 @@ and $0.006980 respectively. Fewer turns and less generated text explain much of
the saving against uncapped, but their causal connection to the byte cap remains
uncertain. Better test evidence is a possible future hypothesis, not an untested
change folded into this frozen policy.
+
+## Accepted spike comparison prompted by user review
+
+The accepted spike's Requests treatment cost $0.2597596 versus $0.2685004
+baseline (3.26% lower). Its stdout was 7,059 UTF-8 bytes, compared with 2,852
+uncapped production and 2,183 capped production. It returned five relevant files,
+reported incomplete discovery and a scoped `AGENTS.md` lookup, and supplied more
+request construction, header preparation and surrounding method context. The
+production packets supplied six paths, less source and no scoped guidance lookup.
+These are actual packet differences, not proof of their causal contribution.
+
+The production skill was also rewritten from the accepted skill: detailed
+symbol-first reads, lookup reuse and explicit polling instructions were shortened.
+Production traces include repeated guidance searches and broader follow-up reads.
+All runs retain the faulty content-length method. The source-cap comparison
+therefore does not establish parity with the winning spike; one can reduce cost
+against the uncapped production run while still regress against the spike.
+
+The causal split between changed representation, changed agent instructions,
+live classification/query variation and Sol behavior is unresolved. This needs
+an exact spike-versus-production trace/request comparison before attributing the
+regression to normal variance or calling the production policy a reproduced win.
+The active frozen cohort remains unchanged and must be reported in full.
diff --git a/specs/jevgrep/choices.md b/specs/jevgrep/choices.md
index c91ef79..bea8e9f 100644
--- a/specs/jevgrep/choices.md
+++ b/specs/jevgrep/choices.md
@@ -165,3 +165,15 @@ cannot satisfy the cohort gate, and the runner has no baseline execution path.
Gap: artifact layout and execution mechanics were delegated. Reach: results from
different candidates cannot be pooled into acceptance; failed attempts and unknown
bills remain visible. Verdict: sound. Confidence: high.
+
+### Reuse installed command journeys for native macOS verification
+
+When: supported-runtime verification. Install a tarball into a temporary npm
+prefix, give its process a PATH containing only Node, and drive the same keyless
+commands and Python HTTP search used in Docker. Filesystem and failure suites
+remain Docker-only. The native result records its actual Node/macOS version.
+
+Gap: the native smoke mechanism was delegated. Reach: portable assertions stay
+shared while Linux tool-absence assertions retain their own Docker scope; a Mac
+with Python installed cannot accidentally satisfy a runtime dependency through
+PATH. Verdict: sound. Confidence: high.