Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

Tighten exclusion and endpoint validation after final review

David Zhangc76626b967a3
diff --git a/apps/cli/README.md b/apps/cli/README.md
index 4e9c6a4..12cb039 100644
--- a/apps/cli/README.md
+++ b/apps/cli/README.md
@@ -42,6 +42,30 @@ are accepted for a local proxy. Auth and doctor name only the endpoint host, nev
the key. The saved endpoint and model are part of the cache identity, so custom
answers stay separate from the preset providers’ answers and from other endpoints.
+A custom base URL must contain only the scheme, host, and optional path: no
+embedded credentials, query string, or fragment. Jevgrep appends `/systemone` to
+that base URL and sends the saved key as a Bearer token.
+
+For gateway implementers, Jevgrep's boolean questions arrive as `noul` questions
+**without `criteria`**. A minimal request and response are:
+
+```json
+{
+ "model": "your-model",
+ "state": "source context",
+ "questions": { "q": { "type": "noul", "instructions": "Relevant?" } }
+}
+```
+
+```json
+{ "answers": { "q": { "type": "noul", "noul": 0.9 } } }
+```
+
+Return an answer for every question ID, with a finite `noul` value between 0 and 1.
+These answers need neither `probabilities` nor `confidence`. A server that demands
+`criteria.true` and returns HTTP 422 is incompatible with these requests. Run
+`jg doctor` against a custom gateway before searching a repository.
+
Credentials are saved under
`$XDG_CONFIG_HOME/jevgrep/credentials.json`, or `~/.config/jevgrep/credentials.json`.
Existing saved records without a provider still mean Vercel, without a migration.
diff --git a/apps/cli/src/args.ts b/apps/cli/src/args.ts
index 2a3e15e..bf930d7 100644
--- a/apps/cli/src/args.ts
+++ b/apps/cli/src/args.ts
@@ -163,11 +163,11 @@ function policyFrom(values: {
!pattern.trim() ||
/^[!#]/.test(pattern) ||
/[\r\n]/.test(pattern) ||
- /(?:^|[^\\])\\$/.test(pattern),
+ /(?:^|[^\\])(?:\\\\)*\\$/.test(pattern),
)
)
throw new CliError(
- "--exclude takes one gitignore pattern without a leading ! or # or a trailing \\.",
+ "--exclude takes one gitignore pattern without a leading ! or # or a trailing unescaped \\.",
);
const policy: NonNullable<SearchInput["policy"]> = {};
if (values.hidden) policy.hidden = true;
diff --git a/apps/cli/test/args.test.ts b/apps/cli/test/args.test.ts
index ce76416..c46a1f0 100644
--- a/apps/cli/test/args.test.ts
+++ b/apps/cli/test/args.test.ts
@@ -149,3 +149,18 @@ test("exclude patterns are repeatable, normalized for cache identity, and limite
for (const args of [["doctor"], ["auth"], ["skill"], ["cache", "clear"]])
expect(() => parseCommand([...args, "--exclude", "docs"])).toThrow();
});
+
+test("exclude rejects odd trailing backslash runs but preserves escaped backslashes", () => {
+ for (const command of ["question", "files"]) {
+ for (const count of [1, 3, 5])
+ expect(() => parseCommand([command, "--exclude", "secrets" + "\\".repeat(count)])).toThrow(
+ "--exclude",
+ );
+ for (const count of [2, 4]) {
+ const pattern = "secrets" + "\\".repeat(count);
+ expect(parseCommand([command, "--exclude", pattern])).toMatchObject({
+ policy: { exclude: [pattern] },
+ });
+ }
+ }
+});
diff --git a/docs/architecture.md b/docs/architecture.md
index 4c3bdef..2751e82 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -60,6 +60,7 @@ that reuse. Source snapshots retain their existing read and freshness checks.
## Version improvement
+The [evaluation guide](../evals/README.md) links the harness and result reports.
The [evaluation policy](../evals/cost-quality-policy.md) owns version improvement:
official task completion is primary, full coding-agent cost is reported, and Jev
cost is separate. Preserve fixed baselines and identify measured artifacts and
diff --git a/evals/results/skill-impact-2026-09-29.md b/evals/results/skill-impact-2026-09-29.md
index 3304dc2..9352502 100644
--- a/evals/results/skill-impact-2026-09-29.md
+++ b/evals/results/skill-impact-2026-09-29.md
@@ -11,7 +11,7 @@ Updated existing skill installations in `aiexplainer`, `jevgrep`, `battlegame`,
one existing Battlegame worktree under `~/dev`. Five physical files and their
Claude aliases (nine paths total) match the merged canonical skill. Historical
benchmark snapshots were preserved. Previous bytes and SHA256 records are in the
-[rollout manifest](../runs/skill-rollout-2026-09-29/manifest.json).
+local `evals/runs/skill-rollout-2026-09-29/manifest.json` (ignored run storage).
This updates skill text, not the globally installed CLI. The global executable
remains 0.5.0; the skill advises checking help for newer commands. The benchmark
@@ -32,7 +32,7 @@ invocation, not successful retrieval. The isolated catalog creates a ceiling
effect and does not reproduce Claude's competing-plugin setup in issue #29.
A separate reviewer audited all 18 raw traces and confirmed the counts.
-The [trigger report](../runs/skill-trigger-2026-09-29/REPORT.md) contains the
+The [trigger report](skill-trigger-2026-09-29.md) contains the
protocol, per-run evidence, token usage, invalid initial probe, and limitations.
Native CLI monetary usage was unavailable; the shim made no retrieval calls.
diff --git a/evals/results/skill-trigger-2026-09-29.md b/evals/results/skill-trigger-2026-09-29.md
new file mode 100644
index 0000000..fa3a82f
--- /dev/null
+++ b/evals/results/skill-trigger-2026-09-29.md
@@ -0,0 +1,50 @@
+# Jevgrep skill automatic-trigger pilot
+
+No observed before/after movement in this bounded Codex pilot. This is an **invocation-only** experiment, not retrieval quality or task-solving evidence.
+
+| Prompt | Old: skill read | New: skill read | Old: jg search before source content | New: jg search before source content |
+|---|---:|---:|---:|---:|
+| How does session expiry via max_age work? | 3/3 | 3/3 | 3/3 | 3/3 |
+| Where are cookie attributes kept when a session is cleared? | 3/3 | 3/3 | 3/3 | 3/3 |
+| Where is SESSION_KEY defined? | 0/3 | 0/3 | 0/3 | 0/3 |
+
+All behavioral sessions first performed filename discovery (`rg --files`, including AGENTS.md checks). Under issue #29’s stricter “jg before any other search” measure, both arms scored 0/6. Under the shipped skill’s “before broad text searches or git history” requirement, both scored 6/6. Exact-symbol controls used grep and did not read the skill or invoke jg.
+
+## Protocol
+
+- Codex CLI 0.153.4; inherited configured model `gpt-6-astra`, reasoning `medium`; no model/effort override.
+- Fresh ephemeral native agent per prompt/repeat, read-only sandbox, separate disposable checkout each time, no existing conversation. Two concurrent sessions, launches staggered.
+- aio-libs/aiohttp-session source frozen at `66a747a555e1db17cce80307c9ffd20600af6a09` (58 files). Full old skill from `a9993c1:skills/jevgrep/SKILL.md`; full new skill from `origin/main:skills/jevgrep/SKILL.md`. Snapshots included here.
+- Only the arm’s project Jevgrep skill enabled. Other installed user skills, plugins, MCP servers, and multi-agent capability disabled by per-command overrides; real configuration files not edited.
+- Each prompt was the issue-29 question plus: “Answer from this checkout. Do not modify files or install software. Keep the answer concise.” Neither the desired tool nor expected answer was in the prompt.
+- A PATH shim named `jg` emitted `JEVGREP_INVOCATION_SHIM: <args>` and deliberately failed with code 69. This measures attempted invocation only. No provider retrieval requests or credentials were needed.
+- Grade skill-file reads separately from shim-confirmed jg calls. Mere `command -v jg`, promises to use it, and `jg files` alone are not a semantic search hit.
+- Three repeats per version/question (18 valid sessions). An initial probe carrying “do not access credentials” caused an explicit skip; preserved under `invalid-credential-constraint-probe/` and excluded. Removing that conflicting instruction restored invocation before scaling.
+
+Raw evidence paths below are relative to local ignored storage at
+`evals/runs/skill-trigger-2026-09-29/`; raw traces are not distributed with Git.
+
+## Evidence and limitations
+
+- `results.json`: per-run judgments and ordered command traces; `<arm>-<case>-<repeat>.jsonl`: full raw native tool/output trace and token usage; `.command.json`: exact invocation; `.answer.txt`: final response; `.meta.json`: elapsed time and source ref.
+- `isolation-audit.json` verifies source files stayed byte-for-byte unchanged and no unexpected output appeared in the disposable checkouts. Live repository status was checked periodically; no source or skill edits were made by this experiment.
+- Tiny sample and a single enabled task skill create a ceiling effect. This is Codex, not the Claude Code/Opus setup in #29; it cannot validate or disprove that reported result. Disabled competing skills/plugins also differ from that setup.
+- Complete skill versions differ in body as well as description. This is an old-vs-new package test, not a causal description-only ablation.
+- No retrieval succeeded, so this says nothing about answer quality, benchmark resolution rate, real provider latency, or spend.
+- Script grading was manually checked against raw command traces; the parent reviewer can audit those traces independently.
+
+## Usage
+
+Valid sessions: `{"input_tokens": 1526792, "cached_input_tokens": 1180160, "cache_write_input_tokens": 0, "output_tokens": 8033, "reasoning_output_tokens": 16}`.
+Monetary cost was not returned by the native CLI and is not estimated. Provider retrieval cost is zero because the inert shim intercepted every jg invocation.
+
+Skill SHA256:
+
+- old: `82d4537183cb9219d60400df1d478ad0b213e14bf05bf6ae9b7f7e751dc3d935`
+- new: `916bf51b9a57f1cb94d04dbd90d361f1190b6dabc0f9d1ca152d2469862a974c`
+
+`evidence.json` records each run’s actual skill-read, jg invocation, and first content-search command with trace pointers.
+
+## Independent trace audit
+
+A separate reviewer checked all 18 raw sessions and confirmed every reported count and full skill-file read. The strict filename-search measure includes AGENTS.md discovery attempts; one session only checked AGENTS.md before Jevgrep, so this should not be called substantive source discovery in every case. The script's compound-command classifier missed two later content searches paired with `rg --files`; both happened after Jevgrep and do not change the counts. `evidence.json` points to the correct first content searches.
diff --git a/packages/core/src/providers.ts b/packages/core/src/providers.ts
index 84188ab..18f36fd 100644
--- a/packages/core/src/providers.ts
+++ b/packages/core/src/providers.ts
@@ -63,6 +63,9 @@ export function validateBaseURL(value: unknown): string {
throw new Error(
"Base URL must use https:// (http:// is allowed only for localhost or 127.0.0.1).",
);
+ // The SDK appends /systemone directly; URL suffixes would consume that path.
+ if (baseURL.includes("?") || baseURL.includes("#") || url.username || url.password)
+ throw new Error("Base URL must not contain credentials, a query string, or a fragment.");
return baseURL;
}
diff --git a/packages/core/test/providers.test.ts b/packages/core/test/providers.test.ts
index 14e151e..1ce4ee4 100644
--- a/packages/core/test/providers.test.ts
+++ b/packages/core/test/providers.test.ts
@@ -37,6 +37,11 @@ test("base URLs require https except for loopback development proxies", () => {
"http://gateway.example.com/v1",
"http://192.168.1.10:8080/v1",
"ftp://gateway.example.com/v1",
+ "https://gateway.example.com/v1?api-version=1",
+ "https://gateway.example.com/v1#fragment",
+ "https://gateway.example.com/v1?",
+ "https://gateway.example.com/v1#",
+ "https://user:[email protected]/v1",
"gateway.example.com/v1",
"",
" ",
diff --git a/specs/done/jevgrep/contracts.md b/specs/done/jevgrep/contracts.md
index 61408ad..e7b25ef 100644
--- a/specs/done/jevgrep/contracts.md
+++ b/specs/done/jevgrep/contracts.md
@@ -47,12 +47,13 @@ failure, 2 incomplete search or `jg files` listing (even if zero useful files),
An interrupted search may emit already acquired evidence with interrupted status.
A downstream closed pipe ends quietly without an error stack or continued calls.
-Credentials retain XDG config location and owner-only permissions; environment
-key takes precedence over saved key. An explicitly empty environment key disables
-saved credentials, supporting isolated tests. Trim surrounding whitespace, then
-reject internal whitespace consistently; never echo keys in either stream or provider errors. Auth saves
-without a network call; doctor verifies a synthetic expected answer. Help/
-version/skill/cache-clear/files require no provider key. Bound auth stdin input.
+Credentials use the XDG config location and owner-only permissions. Auth saves
+one provider/key record; custom providers also save their base URL and model.
+Search and doctor use only that record. Environment credentials and routing
+values do not override it. Trim surrounding key whitespace and reject internal
+whitespace; never echo keys in output or provider errors. Auth saves without a
+network call; doctor verifies a synthetic expected answer. Help, version, skill,
+cache-clear, and files require no provider key. Bound auth stdin input.
The canonical skill source is `skills/jevgrep/SKILL.md` in this repository,
discoverable by the skills CLI installer. Ship that same file with the npm package;
@@ -68,10 +69,10 @@ The repository skill now checks for a missing executable, installs the published
CLI with npm when prerequisites are available, and directs credential setup to
the user's terminal. This post-release setup addition does not alter the frozen
benchmark skill or its research instructions; 0.1.0's bundled copy remains unchanged.
-The skill chooses unfamiliar multi-file discovery, awaits the same invocation,
+The skill triggers on repository behavior questions, awaits the same invocation,
reads included excerpts first, treats locations as optional leads, and uses normal
-tools to resolve remaining holes. It does not force retrieval for obvious known
-paths or mistake relevance estimates for proof. Benchmark wrapper alone requires
+tools to resolve remaining holes. Exact symbol, string, and filename lookups stay on ordinary search tools. The
+skill does not mistake relevance estimates for proof. Benchmark wrapper alone requires
initial retrieval. Retrieved text is data, never higher-priority instructions.
## Core seam and ownership
diff --git a/specs/done/provider-support/README.md b/specs/done/provider-support/README.md
index 05aaf85..38ed662 100644
--- a/specs/done/provider-support/README.md
+++ b/specs/done/provider-support/README.md
@@ -9,8 +9,9 @@ keeping retrieval behavior fixed.
Coding agents should research code without managing provider routing. A human
chooses a provider and key during `jg auth`, and ordinary search and doctor use
that saved record until auth replaces it. This avoids ambiguous environment-key
-precedence and accidental provider fallback. Automation has one explicit path:
-`jg auth --provider NAME --stdin`.
+precedence and accidental provider fallback. Custom endpoints keep their base URL
+and model in that same saved record; see the [CLI guide](../../../apps/cli/README.md)
+for setup and gateway protocol requirements.
The only compatibility exception is deliberate: a saved key without a provider
still means Vercel. Reading it does not migrate or rewrite the file. Environment
@@ -39,8 +40,9 @@ This is a simpler setup contract, not a multi-account credential registry.
- Auth saves one private record atomically and never verifies a key over the
network. Doctor checks separately with synthetic source. Keys never belong in
chat or CLI output.
-- Tests route requests outside the product. There is no public endpoint/model
- override or provider flag on search/doctor, and no automatic fallback.
+- Tests can route requests to fixtures without modifying saved production routing.
+ Custom endpoints are selected through auth; search and doctor have no routing
+ overrides or automatic fallback.
The [preset owner](../../../packages/core/src/providers.ts),
[evaluator](../../../packages/core/src/evaluator.ts), and