Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

fix sensitive content filtering review gaps

David Zhanga9f1ca6b5a2c
diff --git a/README.md b/README.md
index 63f3422..21f83b1 100644
--- a/README.md
+++ b/README.md
@@ -130,8 +130,16 @@ Current protocols and subsequent experiments are in the
Searches send eligible source content to Jev through the provider selected during auth. Default
filesystem filtering respects ignore files and excludes hidden, dependency/build,
-binary, and obvious credential files. These filters are not a guarantee that all
+binary, and obvious credential files, including `.env` variants and any path
+component whose name ends in `.env`.
+It also excludes a whole file as `sensitive_content` when its text matches a
+conservative credential heuristic: a known provider token prefix (such as `npm_`
+or `glpat-`), a Slack webhook URL, or a credential-shaped assignment with a
+long, high-entropy value. UUID-shaped identifiers are excluded from that entropy
+signal. This is best-effort detection, not redaction or a guarantee that all
sensitive information has been removed; choose a search root you intend to send.
+Use `--include-sensitive` when a search must include sensitive names, private-key
+markers, and content-level matches.
`jg files [root]` counts the files a search under that root may read, grouped by
top-level directory, with no provider key or network request. It takes the same
filtering flags as search.
diff --git a/packages/core/src/filesystem.ts b/packages/core/src/filesystem.ts
index 1ab5d32..bd8db4c 100644
--- a/packages/core/src/filesystem.ts
+++ b/packages/core/src/filesystem.ts
@@ -63,7 +63,7 @@ export const filesystemDefaults = Object.freeze({
".npmrc",
".pypirc",
] as readonly string[],
- sensitiveSuffixes: [".pem", ".key", ".p12", ".pfx"] as readonly string[],
+ sensitiveSuffixes: [".env", ".pem", ".key", ".p12", ".pfx"] as readonly string[],
pageSize: 128,
maxOpenDirectories: 64,
maxFileBytes: 16 * 1024 * 1024,
diff --git a/packages/core/src/secrets.ts b/packages/core/src/secrets.ts
index a931fb5..35ac640 100644
--- a/packages/core/src/secrets.ts
+++ b/packages/core/src/secrets.ts
@@ -19,9 +19,8 @@
* `private_key` exclusion semantics.
* - Length >= 16 and entropy >= 3.5 bits/char: tuned so
* `password: "SuperSecret123!@#"` (17 chars, ~3.57 bits) is caught while
- * `password123` is not. UUID-shaped values (~3.4 bits) are knowingly missed;
- * raising the bar would trade fewer false positives for more leaks — the
- * wrong direction for this filter.
+ * `password123` is not. UUID-shaped values are explicitly exempted because
+ * they are common identifiers and do not provide enough evidence of a secret.
* - bcrypt/argon2 password hashes are flagged too: not directly usable, but
* offline-crackable, so conservative exclusion is acceptable.
* - An assignment whose value opens a quote that never closes is treated as
@@ -50,6 +49,7 @@ const KNOWN_PREFIXES = [
"ghu_",
"ghs_",
"github_pat_",
+ "glpat-",
"xoxb-",
"xoxp-",
"xoxa-",
@@ -72,10 +72,7 @@ const TOKEN_TAIL = "[A-Za-z0-9_\\-]{16,}";
*/
const KNOWN_PREFIX_PATTERNS = [...KNOWN_PREFIXES]
.sort((a, b) => b.length - a.length)
- .map(
- (prefix) =>
- new RegExp(`(?<![A-Za-z0-9_])${escapeRegExp(prefix)}(${TOKEN_TAIL})`, "g"),
- );
+ .map((prefix) => new RegExp(`(?<![A-Za-z0-9_])${escapeRegExp(prefix)}(${TOKEN_TAIL})`, "g"));
/** Slack webhook URLs carry the secret in the path, not in a token prefix. */
const SLACK_WEBHOOK_PATTERN = /hooks\.slack\.com\/services\/[A-Za-z0-9_\-/]{24,}/;
@@ -90,10 +87,13 @@ const KEY_NAME_PATTERN =
/** Assignment shapes across .env / YAML / JSON / TOML / INI and source files, with optional quotes and declaration keywords. */
const ASSIGNMENT_PATTERN =
- /^\s*(?:(?:const|let|var|export|default|public|private|protected|static|readonly|final)\s+)*["']?([^"'=\s:]+)["']?\s*[:=]\s*([\s\S]+?)\s*$/;
+ /^\s*(?:(?:const|let|var|export|default|public|private|protected|static|readonly|final)\s+)*["']?([^"'=\s:]+)["']?\s*[:=]\s*/;
const PLACEHOLDER_PATTERN =
/^(?:your|example|test|fake|dummy|sample|changeme|placeholder|redacted|masked|todo|fixme|none|null|undefined|nil|tbd)[\w-]*$|^x+$|^<.*>$|^\*+$|^\$\{[^}]*\}$|^\{\{[^}]*\}\}$/i;
+const UUID_PATTERN = /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/i;
+const REFERENCE_PATTERN =
+ /^(?:process|import\.meta|env|config|settings|secrets)(?:\.[A-Za-z_$][\w$]*)+$/;
const MIN_SECRET_LENGTH = 16;
const MIN_SECRET_ENTROPY = 3.5;
@@ -117,9 +117,7 @@ function shannonEntropy(value: string): number {
/** True when the tail of a prefix-matched token looks like redacted/example material. */
function isPlaceholderTail(tail: string): boolean {
const lower = tail.toLowerCase();
- return (
- /xxx|example|placeholder|redacted|masked/.test(lower) || new Set(lower).size === 1
- );
+ return /xxx|example|placeholder|redacted|masked/.test(lower) || new Set(lower).size === 1;
}
/** Returns the matched secret token, or undefined when no known prefix fires. */
@@ -131,6 +129,9 @@ function findSecretToken(line: string): string | undefined {
pattern.lastIndex = 0;
let match: RegExpExecArray | null;
while ((match = pattern.exec(line)) !== null) {
+ // npm exposes package metadata as long `npm_package_*` environment
+ // names; those identifiers are not npm access tokens.
+ if (match[0].toLowerCase().startsWith("npm_package_")) continue;
if (!placeholderAt.has(match.index)) {
if (match[1] !== undefined && !isPlaceholderTail(match[1])) return match[0];
placeholderAt.add(match.index);
@@ -142,7 +143,12 @@ function findSecretToken(line: string): string | undefined {
}
function cleanValue(raw: string): string {
- const value = raw.trim().replace(/[,;]$/, "").trim();
+ let value = raw
+ .trim()
+ .replace(/[,;\s]+$/, "")
+ .trim();
+ if (value.endsWith("}") && !value.startsWith("${") && !value.startsWith("{{"))
+ value = value.slice(0, -1).trim();
const quote = value[0];
if (value.length < 2 || (quote !== '"' && quote !== "'" && quote !== "`")) return value;
for (let i = 1; i < value.length; i++) {
@@ -155,6 +161,31 @@ function cleanValue(raw: string): string {
return value;
}
+/** Find assignment-shaped expressions outside quoted strings on one logical line. */
+function assignments(line: string): Array<{ key: string; rawValue: string }> {
+ const matches: Array<{ key: string; rawValue: string }> = [];
+ let quote: string | null = null;
+ for (let index = 0; index < line.length; index++) {
+ const char = line[index]!;
+ if (quote !== null) {
+ if (char === "\\") index++;
+ else if (char === quote) quote = null;
+ continue;
+ }
+ if (char === '"' || char === "'" || char === "`") {
+ quote = char;
+ continue;
+ }
+ if (index !== 0 && !"{,;".includes(char)) continue;
+ const candidate = line.slice(index === 0 ? index : index + 1);
+ const match = ASSIGNMENT_PATTERN.exec(candidate);
+ if (match?.[1] !== undefined) {
+ matches.push({ key: match[1], rawValue: candidate.slice(match[0].length) });
+ }
+ }
+ return matches;
+}
+
function isPlaceholder(value: string): boolean {
return value.length === 0 || PLACEHOLDER_PATTERN.test(value);
}
@@ -176,6 +207,8 @@ function hasUnclosedQuote(raw: string): boolean {
function looksLikeSecretValue(value: string): boolean {
if (value.length < MIN_SECRET_LENGTH || isPlaceholder(value)) return false;
+ if (REFERENCE_PATTERN.test(value)) return false;
+ if (UUID_PATTERN.test(value)) return false;
if (findSecretToken(value) !== undefined) return true;
if (LONG_TOKEN_RUN.test(value)) return true;
return shannonEntropy(value) >= MIN_SECRET_ENTROPY;
@@ -236,13 +269,9 @@ export function containsLikelySecret(source: string): boolean {
if (!trimmed || trimmed.startsWith("#") || trimmed.startsWith("//")) continue;
if (findSecretToken(trimmed) !== undefined) return true;
if (SLACK_WEBHOOK_PATTERN.test(trimmed)) return true;
- const assignment = ASSIGNMENT_PATTERN.exec(trimmed);
- if (
- assignment?.[1] !== undefined &&
- assignment[2] !== undefined &&
- KEY_NAME_PATTERN.test(assignment[1])
- ) {
- const rawValue = assignment[2];
+ for (const assignment of assignments(trimmed)) {
+ if (!KEY_NAME_PATTERN.test(assignment.key)) continue;
+ const rawValue = assignment.rawValue;
if (hasUnclosedQuote(rawValue)) return true;
if (looksLikeSecretValue(cleanValue(rawValue))) return true;
}
diff --git a/packages/core/test/filesystem.test.ts b/packages/core/test/filesystem.test.ts
index 43c5eac..e31b17e 100644
--- a/packages/core/test/filesystem.test.ts
+++ b/packages/core/test/filesystem.test.ts
@@ -501,3 +501,41 @@ test("includeSensitive bypasses content-level secret filtering", async () => {
await reader.close();
}
});
+
+test("sensitive suffix policy excludes .env files at any path depth", async () => {
+ const root = await fixture({
+ "config/application.env": "PRIVATE_VALUE=not-a-token\n",
+ "config.env/application.ts": "export const value = 1;\n",
+ });
+ const reader = await createFilesystem({ root });
+ try {
+ expect(await reader.readSnapshot("config/application.env")).toMatchObject({
+ status: "excluded",
+ reason: "sensitive_name",
+ });
+ expect(await reader.readSnapshot("config.env/application.ts")).toMatchObject({
+ status: "excluded",
+ reason: "sensitive_name",
+ });
+ } finally {
+ await reader.close();
+ }
+});
+
+test("includeSensitive also bypasses GitLab token content filtering", async () => {
+ const root = await fixture({
+ "gitlab.json": '{"token":"glpat-abcDEF1234567890abcdef1234567890ab"}\n',
+ });
+ const defaultReader = await createFilesystem({ root });
+ const broadReader = await createFilesystem({ root, policy: { includeSensitive: true } });
+ try {
+ expect(await defaultReader.readSnapshot("gitlab.json")).toMatchObject({
+ status: "excluded",
+ reason: "sensitive_content",
+ });
+ expect(await broadReader.readSnapshot("gitlab.json")).toMatchObject({ status: "ok" });
+ } finally {
+ await defaultReader.close();
+ await broadReader.close();
+ }
+});
diff --git a/packages/core/test/secrets.test.ts b/packages/core/test/secrets.test.ts
index ae3ba10..f9eb426 100644
--- a/packages/core/test/secrets.test.ts
+++ b/packages/core/test/secrets.test.ts
@@ -25,9 +25,7 @@ test("flags the four secret shapes from jevgrep#51", () => {
test("flags yaml-style secrets and bearer tokens", () => {
expect(containsLikelySecret(`database:\n password: "${FAKE_PASSWORD}"\n`)).toBe(true);
- expect(
- containsLikelySecret(`headers:\n authorization: Bearer ${FAKE_GITHUB}\n`),
- ).toBe(true);
+ expect(containsLikelySecret(`headers:\n authorization: Bearer ${FAKE_GITHUB}\n`)).toBe(true);
});
test("ignores placeholders, short and low-entropy values", () => {
@@ -41,14 +39,14 @@ test("ignores placeholders, short and low-entropy values", () => {
test("ignores ordinary code that merely mentions key-like words", () => {
expect(
- containsLikelySecret(`const token = getToken();\nfunction authenticate(token) {\n return token;\n}\n`),
- ).toBe(false);
- expect(
- containsLikelySecret(`// Set your api_key in the config file before running.\n`),
- ).toBe(false);
- expect(
- containsLikelySecret(`# The password field is required by the login form.\n`),
+ containsLikelySecret(
+ `const token = getToken();\nfunction authenticate(token) {\n return token;\n}\n`,
+ ),
).toBe(false);
+ expect(containsLikelySecret(`// Set your api_key in the config file before running.\n`)).toBe(
+ false,
+ );
+ expect(containsLikelySecret(`# The password field is required by the login form.\n`)).toBe(false);
expect(
containsLikelySecret(
`export function loadSettings() {\n return readFileSync("settings.json", "utf8");\n}\n`,
@@ -60,9 +58,7 @@ test("regression: review blockers #1-#3", () => {
// #1: `sk-` inside a longer word must not flag (`disk-` contains `sk-`).
expect(containsLikelySecret(`backup: disk-abcdef1234567890ABCDEF\n`)).toBe(false);
// #1 control: a real `sk-` token still flags.
- expect(
- containsLikelySecret(`key: "sk-abcdef1234567890ABCDEF"\n`),
- ).toBe(true);
+ expect(containsLikelySecret(`key: "sk-abcdef1234567890ABCDEF"\n`)).toBe(true);
// #2: multi-line quoted values are joined before detection.
expect(
containsLikelySecret(
@@ -83,35 +79,29 @@ test("regression: review blockers #1-#3", () => {
test("regression: review majors #4-#7", () => {
// #4: more provider prefixes.
- expect(
- containsLikelySecret(`NPM_TOKEN=npm_abcDEF1234567890abcdef1234567890ab\n`),
- ).toBe(true);
- expect(
- containsLikelySecret(`pypi_token = "pypi-AgEIcHlwaS5vcmcCJGU5Y2YxMjM0NTY3ODkw"\n`),
- ).toBe(true);
+ expect(containsLikelySecret(`NPM_TOKEN=npm_abcDEF1234567890abcdef1234567890ab\n`)).toBe(true);
+ expect(containsLikelySecret(`pypi_token = "pypi-AgEIcHlwaS5vcmcCJGU5Y2YxMjM0NTY3ODkw"\n`)).toBe(
+ true,
+ );
// #5: bearer_token key name.
- expect(
- containsLikelySecret(`bearer_token = "7fK8qN2mT6vX9yZ3bD4hJ1pL5sW8aC0eG2i"\n`),
- ).toBe(true);
+ expect(containsLikelySecret(`bearer_token = "7fK8qN2mT6vX9yZ3bD4hJ1pL5sW8aC0eG2i"\n`)).toBe(true);
// #6: inline comments are stripped quote-aware; they must not decide detection.
expect(
containsLikelySecret(`API_KEY=abc # a very long production comment adding entropy here\n`),
).toBe(false);
expect(
- containsLikelySecret(
- `API_KEY=sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v # production\n`,
- ),
+ containsLikelySecret(`API_KEY=sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v # production\n`),
).toBe(true);
// `#` inside a quoted value is preserved, not treated as a comment.
expect(containsLikelySecret(`password = "xQ9#vL2$mK8@nB4!pT6&wZ3*"\n`)).toBe(true);
// #7: short-but-strong values (16+) are caught.
expect(containsLikelySecret(`password: "SuperSecret123!@#"\n`)).toBe(true);
// Declaration keywords in source files are handled.
+ expect(containsLikelySecret(`const password = "SuperSecret123!@#";\n`)).toBe(true);
+ // CRLF line endings work.
expect(
- containsLikelySecret(`const password = "SuperSecret123!@#";\n`),
+ containsLikelySecret(`"OPENAI_API_KEY": "sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v"\r\n`),
).toBe(true);
- // CRLF line endings work.
- expect(containsLikelySecret(`"OPENAI_API_KEY": "sk-K7q2Mv9xT4pL8nR3wY6bD1fH5jG0cS9aE2dZ7xU4v"\r\n`)).toBe(true);
// Slack webhook URLs flag.
expect(
containsLikelySecret(
@@ -119,3 +109,26 @@ test("regression: review majors #4-#7", () => {
),
).toBe(true);
});
+
+test("does not treat npm package environment names as npm tokens", () => {
+ expect(containsLikelySecret("process.env.npm_package_scripts_test\n")).toBe(false);
+ expect(containsLikelySecret("NPM_TOKEN=npm_abcDEF1234567890abcdef1234567890ab\n")).toBe(true);
+});
+
+test("flags credential assignments inside inline objects", () => {
+ expect(containsLikelySecret('const cfg = { api_key: "xQ9#vL2$mK8@nB4!pT6&wZ3*" };\n')).toBe(true);
+});
+
+test("ignores inline credential assignments that reference environment values", () => {
+ expect(containsLikelySecret("const cfg = { api_key: process.env.API_KEY };\n")).toBe(false);
+});
+
+test("flags GitLab personal access token prefixes", () => {
+ expect(containsLikelySecret("GITLAB_TOKEN=glpat-abcDEF1234567890abcdef1234567890ab\n")).toBe(
+ true,
+ );
+});
+
+test("does not classify UUID-shaped identifiers as secrets", () => {
+ expect(containsLikelySecret('api_key: "550e8400-e29b-41d4-a716-446655440000"\n')).toBe(false);
+});
diff --git a/specs/done/jevgrep/contracts.md b/specs/done/jevgrep/contracts.md
index e7b25ef..944c2ec 100644
--- a/specs/done/jevgrep/contracts.md
+++ b/specs/done/jevgrep/contracts.md
@@ -163,11 +163,21 @@ At a nested repository boundary reset inherited `.gitignore` scope; ancestor
`.ignore` remains applicable. Read parent rules only within the explicit root.
Skip dot paths, dependency/build directories, binary/invalid UTF-8 content, obvious
-credential filenames and private-key markers by default. Final exact name lists
-are delegated to one documented policy module with fixtures, not scattered regexes.
+credential filenames (including `.env` variants and any path component whose
+name ends in `.env`) and private-key markers by default. After an eligible file is decoded, exclude the whole file as
+`sensitive_content` when its content matches the conservative detector in
+`packages/core/src/secrets.ts`: known provider prefixes (including `npm_` and
+`glpat-`), Slack webhook URLs, or credential-shaped assignments whose values are
+at least 16 characters with at least 3.5 bits/character of Shannon entropy.
+Placeholder values and UUID-shaped identifiers are excluded from the entropy
+signal. This heuristic is best effort and does not redact or guarantee that
+sensitive content is absent. `includeSensitive` / `--include-sensitive` bypasses
+the sensitive-name, private-key, and sensitive-content checks while leaving the
+other filesystem policy and hard exclusions in force. Final exact name lists are
+delegated to one documented policy module with fixtures, not scattered regexes.
Overrides do not make binary files parseable. Hard-exclude Jevgrep's own credential
and cache storage and Git metadata. Read eligibility metadata without uploading
-excluded source. No secret scanning guarantee beyond the documented policy.
+excluded source.
Resolve the requested root once. Do not follow descendant symlinks in v1; skip
sockets, devices and FIFOs. Normal policy exclusions are not incomplete searches.