Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
fix sensitive content filtering review gaps
David Zhanga9f1ca6b5a2c
diff --git a/README.md b/README.mdindex 63f3422..21f83b1 100644--- a/README.md+++ b/README.md@@ -130,8 +130,16 @@ Current protocols and subsequent experiments are in theSearches send eligible source content to Jev through the provider selected during auth. Defaultfilesystem filtering respects ignore files and excludes hidden, dependency/build,-binary, and obvious credential files. These filters are not a guarantee that all+binary, and obvious credential files, including `.env` variants and any path+component whose name ends in `.env`.+It also excludes a whole file as `sensitive_content` when its text matches a+conservative credential heuristic: a known provider token prefix (such as `npm_`+or `glpat-`), a Slack webhook URL, or a credential-shaped assignment with a+long, high-entropy value. UUID-shaped identifiers are excluded from that entropy+signal. This is best-effort detection, not redaction or a guarantee that allsensitive information has been removed; choose a search root you intend to send.+Use `--include-sensitive` when a search must include sensitive names, private-key+markers, and content-level matches.`jg files [root]` counts the files a search under that root may read, grouped bytop-level directory, with no provider key or network request. It takes the samefiltering flags as search.diff --git a/packages/core/src/filesystem.ts b/packages/core/src/filesystem.tsindex 1ab5d32..bd8db4c 100644--- a/packages/core/src/filesystem.ts+++ b/packages/core/src/filesystem.ts@@ -63,7 +63,7 @@ export const filesystemDefaults = Object.freeze({".npmrc",".pypirc",] as readonly string[],- sensitiveSuffixes: [".pem", ".key", ".p12", ".pfx"] as readonly string[],+ sensitiveSuffixes: [".env", ".pem", ".key", ".p12", ".pfx"] as readonly string[],pageSize: 128,maxOpenDirectories: 64,maxFileBytes: 16 * 1024 * 1024,diff --git a/packages/core/src/secrets.ts b/packages/core/src/secrets.tsindex a931fb5..35ac640 100644--- a/packages/core/src/secrets.ts+++ b/packages/core/src/secrets.ts@@ -19,9 +19,8 @@* `private_key` exclusion semantics.* - Length >= 16 and entropy >= 3.5 bits/char: tuned so* `password: "SuperSecret123!@#"` (17 chars, ~3.57 bits) is caught while- * `password123` is not. UUID-shaped values (~3.4 bits) are knowingly missed;- * raising the bar would trade fewer false positives for more leaks — the- * wrong direction for this filter.+ * `password123` is not. UUID-shaped values are explicitly exempted because+ * they are common identifiers and do not provide enough evidence of a secret.* - bcrypt/argon2 password hashes are flagged too: not directly usable, but* offline-crackable, so conservative exclusion is acceptable.* - An assignment whose value opens a quote that never closes is treated as@@ -50,6 +49,7 @@ const KNOWN_PREFIXES = ["ghu_","ghs_","github_pat_",+ "glpat-","xoxb-","xoxp-","xoxa-",@@ -72,10 +72,7 @@ const TOKEN_TAIL = "[A-Za-z0-9_\\-]{16,}";*/const KNOWN_PREFIX_PATTERNS = [...KNOWN_PREFIXES].sort((a, b) => b.length - a.length)- .map(- (prefix) =>- new RegExp(`(?<![A-Za-z0-9_])${escapeRegExp(prefix)}(${TOKEN_TAIL})`, "g"),- );+ .map((prefix) => new RegExp(`(?<![A-Za-z0-9_])${escapeRegExp(prefix)}(${TOKEN_TAIL})`, "g"));/** Slack webhook URLs carry the secret in the path, not in a token prefix. */const SLACK_WEBHOOK_PATTERN = /hooks\.slack\.com\/services\/[A-Za-z0-9_\-/]{24,}/;@@ -90,10 +87,13 @@ const KEY_NAME_PATTERN =/** Assignment shapes across .env / YAML / JSON / TOML / INI and source files, with optional quotes and declaration keywords. */const ASSIGNMENT_PATTERN =- /^\s*(?:(?:const|let|var|export|default|public|private|protected|static|readonly|final)\s+)*["']?([^"'=\s:]+)["']?\s*[:=]\s*([\s\S]+?)\s*$/;+ /^\s*(?:(?:const|let|var|export|default|public|private|protected|static|readonly|final)\s+)*["']?([^"'=\s:]+)["']?\s*[:=]\s*/;const PLACEHOLDER_PATTERN =/^(?:your|example|test|fake|dummy|sample|changeme|placeholder|redacted|masked|todo|fixme|none|null|undefined|nil|tbd)[\w-]*$|^x+$|^<.*>$|^\*+$|^\$\{[^}]*\}$|^\{\{[^}]*\}\}$/i;+const UUID_PATTERN = /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/i;+const REFERENCE_PATTERN =+ /^(?:process|import\.meta|env|config|settings|secrets)(?:\.[A-Za-z_$][\w$]*)+$/;const MIN_SECRET_LENGTH = 16;const MIN_SECRET_ENTROPY = 3.5;@@ -117,9 +117,7 @@ function shannonEntropy(value: string): number {/** True when the tail of a prefix-matched token looks like redacted/example material. */function isPlaceholderTail(tail: string): boolean {const lower = tail.toLowerCase();- return (- /xxx|example|placeholder|redacted|masked/.test(lower) || new Set(lower).size === 1- );+ return /xxx|example|placeholder|redacted|masked/.test(lower) || new Set(lower).size === 1;}/** Returns the matched secret token, or undefined when no known prefix fires. */@@ -131,6 +129,9 @@ function findSecretToken(line: string): string | undefined {pattern.lastIndex = 0;let match: RegExpExecArray | null;while ((match = pattern.exec(line)) !== null) {+ // npm exposes package metadata as long `npm_package_*` environment+ // names; those identifiers are not npm access tokens.+ if (match[0].toLowerCase().startsWith("npm_package_")) continue;if (!placeholderAt.has(match.index)) {if (match[1] !== undefined && !isPlaceholderTail(match[1])) return match[0];placeholderAt.add(match.index);@@ -142,7 +143,12 @@ function findSecretToken(line: string): string | undefined {}function cleanValue(raw: string): string {- const value = raw.trim().replace(/[,;]$/, "").trim();+ let value = raw+ .trim()+ .replace(/[,;\s]+$/, "")+ .trim();+ if (value.endsWith("}") && !value.startsWith("${") && !value.startsWith("{{"))+ value = value.slice(0, -1).trim();const quote = value[0];if (value.length < 2 || (quote !== '"' && quote !== "'" && quote !== "`")) return value;for (let i = 1; i < value.length; i++) {@@ -155,6 +161,31 @@ function cleanValue(raw: string): string {return value;}+/** Find assignment-shaped expressions outside quoted strings on one logical line. */+function assignments(line: string): Array<{ key: string; rawValue: string }> {+ const matches: Array<{ key: string; rawValue: string }> = [];+ let quote: string | null = null;+ for (let index = 0; index < line.length; index++) {+ const char = line[index]!;+ if (quote !== null) {+ if (char === "\\") index++;+ else if (char === quote) quote = null;+ continue;+ }+ if (char === '"' || char === "'" || char === "`") {+ quote = char;+ continue;+ }+ if (index !== 0 && !"{,;".includes(char)) continue;+ const candidate = line.slice(index === 0 ? index : index + 1);+ const match = ASSIGNMENT_PATTERN.exec(candidate);+ if (match?.[1] !== undefined) {+ matches.push({ key: match[1], rawValue: candidate.slice(match[0].length) });+ }+ }+ return matches;+}+function isPlaceholder(value: string): boolean {return value.length === 0 || PLACEHOLDER_PATTERN.test(value);}@@ -176,6 +207,8 @@ function hasUnclosedQuote(raw: string): boolean {function looksLikeSecretValue(value: string): boolean {if (value.length < MIN_SECRET_LENGTH || isPlaceholder(value)) return false;+ if (REFERENCE_PATTERN.test(value)) return false;+ if (UUID_PATTERN.test(value)) return false;if (findSecretToken(value) !== undefined) return true;if (LONG_TOKEN_RUN.test(value)) return true;return shannonEntropy(value) >= MIN_SECRET_ENTROPY;@@ -236,13 +269,9 @@ export function containsLikelySecret(source: string): boolean {if (!trimmed || trimmed.startsWith("#") || trimmed.startsWith("//")) continue;if (findSecretToken(trimmed) !== undefined) return true;if (SLACK_WEBHOOK_PATTERN.test(trimmed)) return true;- const assignment = ASSIGNMENT_PATTERN.exec(trimmed);- if (- assignment?.[1] !== undefined &&- assignment[2] !== undefined &&- KEY_NAME_PATTERN.test(assignment[1])- ) {- const rawValue = assignment[2];+ for (const assignment of assignments(trimmed)) {+ if (!KEY_NAME_PATTERN.test(assignment.key)) continue;+ const rawValue = assignment.rawValue;if (hasUnclosedQuote(rawValue)) return true;if (looksLikeSecretValue(cleanValue(rawValue))) return true;}diff --git a/packages/core/test/filesystem.test.ts b/packages/core/test/filesystem.test.tsindex 43c5eac..e31b17e 100644--- a/packages/core/test/filesystem.test.ts+++ b/packages/core/test/filesystem.test.ts@@ -501,3 +501,41 @@ test("includeSensitive bypasses content-level secret filtering", async () => {await reader.close();}});++test("sensitive suffix policy excludes .env files at any path depth", async () => {+ const root = await fixture({+ "config/application.env": "PRIVATE_VALUE=not-a-token\n",+ "config.env/application.ts": "export const value = 1;\n",+ });+ const reader = await createFilesystem({ root });+ try {+ expect(await reader.readSnapshot("config/application.env")).toMatchObject({+ status: "excluded",+ reason: "sensitive_name",+ });+ expect(await reader.readSnapshot("config.env/application.ts")).toMatchObject({+ status: "excluded",+ reason: "sensitive_name",+ });+ } finally {+ await reader.close();+ }+});++test("includeSensitive also bypasses GitLab token content filtering", async () => {+ const root = await fixture({+ "gitlab.json": '{"token":"glpat-abcDEF1234567890abcdef1234567890ab"}\n',+ });+ const defaultReader = await createFilesystem({ root });+ const broadReader = await createFilesystem({ root, policy: { includeSensitive: true } });+ try {+ expect(await defaultReader.readSnapshot("gitlab.json")).toMatchObject({+ status: "excluded",+ reason: "sensitive_content",+ });+ expect(await broadReader.readSnapshot("gitlab.json")).toMatchObject({ status: "ok" });+ } finally {+ await defaultReader.close();+ await broadReader.close();+ }+});diff --git a/packages/core/test/secrets.test.ts b/packages/core/test/secrets.test.tsindex ae3ba10..f9eb426 100644--- a/packages/core/test/secrets.test.ts+++ b/packages/core/test/secrets.test.ts@@ -25,9 +25,7 @@ test("flags the four secret shapes from jevgrep#51", () => {test("flags yaml-style secrets and bearer tokens", () => {expect(containsLikelySecret(`database:\n password: "${FAKE_PASSWORD}"\n`)).toBe(true);- expect(- containsLikelySecret(`headers:\n authorization: Bearer ${FAKE_GITHUB}\n`),- ).toBe(true);+ expect(containsLikelySecret(`headers:\n authorization: Bearer ${FAKE_GITHUB}\n`)).toBe(true);});test("ignores placeholders, short and low-entropy values", () => {@@ -41,14 +39,14 @@ test("ignores placeholders, short and low-entropy values", () => {test("ignores ordinary code that merely mentions key-like words", () => {expect(- containsLikelySecret(`const token = getToken();\nfunction authenticate(token) {\n return token;\n}\n`),- ).toBe(false);- expect(- containsLikelySecret(`// Set your api_key in the config file before running.\n`),- ).toBe(false);- expect(- containsLikelySecret(`# The password field is required by the login form.\n`),+ containsLikelySecret(+ `const token = getToken();\nfunction authenticate(token) {\n return token;\n}\n`,+ ),).toBe(false);+ expect(containsLikelySecret(`// Set your api_key in the config file before running.\n`)).toBe(+ false,+ );+ expect(containsLikelySecret(`# The password field is required by the login form.\n`)).toBe(false);expect(containsLikelySecret(`export function loadSettings() {\n return readFileSync("settings.json", "utf8");\n}\n`,@@ -60,9 +58,7 @@ test("regression: review blockers #1-#3", () => {// #1: `sk-` inside a longer word must not flag (`disk-` contains `sk-`).expect(containsLikelySecret(`backup: disk-abcdef1234567890ABCDEF\n`)).toBe(false);// #1 control: a real `sk-` token still flags.- expect(- containsLikelySecret(`key: "sk-abcdef1234567890ABCDEF"\n`),- ).toBe(true);+ expect(containsLikelySecret(`key: "sk-abcdef1234567890ABCDEF"\n`)).toBe(true);// #2: multi-line quoted values are joined before detection.expect(containsLikelySecret(@@ -83,35 +79,29 @@ test("regression: review blockers #1-#3", () => {test("regression: review majors #4-#7", () => {// #4: more provider prefixes.- expect(- containsLikelySecret(`NPM_TOKEN=npm_abcDEF1234567890abcdef1234567890ab\n`),- ).toBe(true);- expect(- containsLikelySecret(`pypi_token = "pypi-AgEIcHlwaS5vcmcCJGU5Y2YxMjM0NTY3ODkw"\n`),- ).toBe(true);+ expect(containsLikelySecret(`NPM_TOKEN=npm_abcDEF1234567890abcdef1234567890ab\n`)).toBe(true);+ expect(containsLikelySecret(`pypi_token = "pypi-AgEIcHlwaS5vcmcCJGU5Y2YxMjM0NTY3ODkw"\n`)).toBe(+ true,+ );// #5: bearer_token key name.- expect(- containsLikelySecret(`bearer_token = "7fK8qN2mT6vX9yZ3bD4hJ1pL5sW8aC0eG2i"\n`),- ).toBe(true);+ expect(containsLikelySecret(`bearer_token = "7fK8qN2mT6vX9yZ3bD4hJ1pL5sW8aC0eG2i"\n`)).toBe(true);// #6: inline comments are stripped quote-aware; they must not decide detection.expect(containsLikelySecret(`API_KEY=abc # a very long production comment adding entropy here\n`),).toBe(false);expect(- containsLikelySecret(- `API_KEY=sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v # production\n`,- ),+ containsLikelySecret(`API_KEY=sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v # production\n`),).toBe(true);// `#` inside a quoted value is preserved, not treated as a comment.expect(containsLikelySecret(`password = "xQ9#vL2$mK8@nB4!pT6&wZ3*"\n`)).toBe(true);// #7: short-but-strong values (16+) are caught.expect(containsLikelySecret(`password: "SuperSecret123!@#"\n`)).toBe(true);// Declaration keywords in source files are handled.+ expect(containsLikelySecret(`const password = "SuperSecret123!@#";\n`)).toBe(true);+ // CRLF line endings work.expect(- containsLikelySecret(`const password = "SuperSecret123!@#";\n`),+ containsLikelySecret(`"OPENAI_API_KEY": "sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v"\r\n`),).toBe(true);- // CRLF line endings work.- expect(containsLikelySecret(`"OPENAI_API_KEY": "sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v"\r\n`)).toBe(true);// Slack webhook URLs flag.expect(containsLikelySecret(@@ -119,3 +109,26 @@ test("regression: review majors #4-#7", () => {),).toBe(true);});++test("does not treat npm package environment names as npm tokens", () => {+ expect(containsLikelySecret("process.env.npm_package_scripts_test\n")).toBe(false);+ expect(containsLikelySecret("NPM_TOKEN=npm_abcDEF1234567890abcdef1234567890ab\n")).toBe(true);+});++test("flags credential assignments inside inline objects", () => {+ expect(containsLikelySecret('const cfg = { api_key: "xQ9#vL2$mK8@nB4!pT6&wZ3*" };\n')).toBe(true);+});++test("ignores inline credential assignments that reference environment values", () => {+ expect(containsLikelySecret("const cfg = { api_key: process.env.API_KEY };\n")).toBe(false);+});++test("flags GitLab personal access token prefixes", () => {+ expect(containsLikelySecret("GITLAB_TOKEN=glpat-abcDEF1234567890abcdef1234567890ab\n")).toBe(+ true,+ );+});++test("does not classify UUID-shaped identifiers as secrets", () => {+ expect(containsLikelySecret('api_key: "550e8400-e29b-41d4-a716-446655440000"\n')).toBe(false);+});diff --git a/specs/done/jevgrep/contracts.md b/specs/done/jevgrep/contracts.mdindex e7b25ef..944c2ec 100644--- a/specs/done/jevgrep/contracts.md+++ b/specs/done/jevgrep/contracts.md@@ -163,11 +163,21 @@ At a nested repository boundary reset inherited `.gitignore` scope; ancestor`.ignore` remains applicable. Read parent rules only within the explicit root.Skip dot paths, dependency/build directories, binary/invalid UTF-8 content, obvious-credential filenames and private-key markers by default. Final exact name lists-are delegated to one documented policy module with fixtures, not scattered regexes.+credential filenames (including `.env` variants and any path component whose+name ends in `.env`) and private-key markers by default. After an eligible file is decoded, exclude the whole file as+`sensitive_content` when its content matches the conservative detector in+`packages/core/src/secrets.ts`: known provider prefixes (including `npm_` and+`glpat-`), Slack webhook URLs, or credential-shaped assignments whose values are+at least 16 characters with at least 3.5 bits/character of Shannon entropy.+Placeholder values and UUID-shaped identifiers are excluded from the entropy+signal. This heuristic is best effort and does not redact or guarantee that+sensitive content is absent. `includeSensitive` / `--include-sensitive` bypasses+the sensitive-name, private-key, and sensitive-content checks while leaving the+other filesystem policy and hard exclusions in force. Final exact name lists are+delegated to one documented policy module with fixtures, not scattered regexes.Overrides do not make binary files parseable. Hard-exclude Jevgrep's own credentialand cache storage and Git metadata. Read eligibility metadata without uploading-excluded source. No secret scanning guarantee beyond the documented policy.+excluded source.Resolve the requested root once. Do not follow descendant symlinks in v1; skipsockets, devices and FIFOs. Normal policy exclusions are not incomplete searches.