Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

feat(cli): --offset paging for inventory and text, deterministic ix text order (HR-02B)

KageBinary699271dcbc4d
diff --git a/docs/llm-format.md b/docs/llm-format.md
index 3c75d5c5..e8036ef8 100644
--- a/docs/llm-format.md
+++ b/docs/llm-format.md
@@ -122,7 +122,25 @@ record: `shown` is what follows, `total` is what there was before the cut.
```
callers target=verify_token shown=50 total=212 resolved=48 unresolved=2
-diagnostic code=results_truncated message="212 callers; showing 50. Raise --limit to see the rest."
+diagnostic code=results_truncated next_offset=50 message="212 callers; showing 50. next: ix callers --offset 50 verify_token"
+```
+
+A list that can be paged says where the next page starts: `next_offset=` on
+the `results_truncated` diagnostic, and a `next:` command in its message that
+repeats the caller's flags with `--offset`. A page past the first carries
+`offset=` on its header. A page that starts at or past the end is not an
+error: it prints a `diagnostic code=offset_past_end` and exits 0.
+
+The edge commands page over what one `/v1/expand` call returns, and the
+server stops at 1000 edges. When it reports a cut, or returns a full 1000,
+`total` is a floor: the list stays `results_truncated` on every page, its
+message says "At least N" and names the cap, no offset is called past the
+end, and the JSON summary carries `capped: true`. Past the cap, narrow the
+target (`--kind`, `--path`) instead of paging.
+
+```
+callers target=verify_token offset=50 shown=50 total=212 resolved=50
+diagnostic code=results_truncated next_offset=100 message="212 callers; showing 51-100. next: ix callers --offset 100 verify_token"
```
`shown < total` is the only signal that a list is partial, and it exists
@@ -134,10 +152,22 @@ Two commands cannot know a true total and say so rather than inventing one:
- `ix inventory` reports `shown=N truncated=true`. The backend applies the
limit and has no count endpoint, so the CLI only knows the window it asked
- for came back full.
+ for came back full. Its truncation message says "More than N" rather than
+ a total. `/v1/list` has no offset either, so `--offset N` fetches the N rows
+ before the page too: a deep page costs O(offset). The server returns at
+ most 5000 rows, so no page reaches past row 5000: a page at or beyond it
+ stays `truncated=true` (JSON: `capped: true`) and says to narrow with
+ `--path`, never `offset_past_end`.
- `ix text` reports `shown=N scanned=M`: ripgrep's traversal is bounded by a
scan window, and `scanned` is how many matches were ranked, not how many
- exist.
+ exist. The window holds the best `scanWindow(limit)` matches (code, then
+ tests, then prose; then path; then line) over every match ripgrep finds,
+ so the output is the same on every run. Pages are cut from that window, so
+ keep `--limit` fixed while paging. The page that reaches the end of a full
+ window has no `next_offset=`; its message says to narrow with `--path` or
+ `--language` instead, and so does a page past a full window (it is not
+ `offset_past_end`: more matches exist). `--format json` stays a bare array of matches with no
+ paging fields; the next offset is the offset plus its length.
## Rows an agent acts on
diff --git a/ix-cli/src/cli/__tests__/honest-truncation.test.ts b/ix-cli/src/cli/__tests__/honest-truncation.test.ts
index fb84af0f..eae80771 100644
--- a/ix-cli/src/cli/__tests__/honest-truncation.test.ts
+++ b/ix-cli/src/cli/__tests__/honest-truncation.test.ts
@@ -1,12 +1,20 @@
// Copyright 2026 Ix Infrastructure Inc.
import { Command } from "commander";
-import { describe, expect, it } from "vitest";
+import { afterEach, describe, expect, it, vi } from "vitest";
-import { renderEdgeResultsLlm, sliceEdgeResults, sliceRanked } from "../format.js";
+const listByKind = vi.hoisted(() => vi.fn());
+vi.mock("../../client/factory.js", () => ({ createClient: () => ({ listByKind }) }));
+vi.mock("../resolve.js", () => ({ resolveReadSystemId: async () => undefined }));
+vi.mock("../bootstrap.js", () => ({ resolveWorkspaceId: () => "ws" }));
+
+import {
+ formatTextResults, renderEdgeResultsLlm, renderTextResultsLlm, sliceEdgeResults, sliceRanked, truncationHint,
+ type TextResult,
+} from "../format.js";
import { nextPageCommand } from "../paging.js";
-import { renderInventoryLlm } from "../commands/inventory.js";
-import { scanWindow, textResultPriority } from "../commands/text.js";
+import { LIST_MAX_ROWS, registerInventoryCommand, renderInventoryLlm } from "../commands/inventory.js";
+import { compareTextResults, keepTopK, scanWindow, textResultPriority } from "../commands/text.js";
function node(name: string, path?: string) {
return { id: `${name}-0000-0000-0000-000000000000`, name, kind: "function", ...(path ? { provenance: { source_uri: path } } : {}) };
@@ -136,6 +144,16 @@ describe("paging with --offset", () => {
expect(lines.some((l) => l.includes("offset_past_end"))).toBe(false);
});
+ it("--limit 0 past the first page prints the count, not an inverted range", () => {
+ const s = sliceEdgeResults(mixed(), 0, 50);
+ expect(s).toMatchObject({ offset: 50, shown: 0, total: 212, truncated: true });
+ expect(truncationHint(s, "callers")).toBe("212 callers; showing 0. Raise --limit to see them.");
+ const lines = renderEdgeResultsLlm(s, "callers", "t", "graph");
+ expect(lines[1]).toBe('diagnostic code=results_truncated message="212 callers; showing 0. Raise --limit to see them."');
+ expect(lines.join("\n")).not.toContain("51-50");
+ expect(lines.join("\n")).not.toContain("next_offset");
+ });
+
it("leaves the header and rows alone without an offset", () => {
const lines = renderEdgeResultsLlm(sliceEdgeResults([node("a", "src/a.ts")], 50), "callers", "t", "graph");
expect(lines[0]).toBe("callers target=t shown=1 total=1 resolved=1");
@@ -180,6 +198,122 @@ describe("renderInventoryLlm", () => {
const nodes = [{ name: "Foo", kind: "class", provenance: { source_uri: "src/a.ts" } }];
expect(renderInventoryLlm("class", null, nodes)[0]).toBe("inventory kind=class shown=1");
});
+
+ it("a middle page names its offset and the next one", () => {
+ const nodes = Array.from({ length: 50 }, (_, i) => ({ name: `C${i}`, kind: "class", provenance: { source_uri: "src/a.ts" } }));
+ const lines = renderInventoryLlm("class", null, nodes, true, {
+ offset: 50, nextOffset: 100, next: "ix inventory --kind class --offset 100",
+ });
+ expect(lines.slice(0, 2)).toEqual([
+ "inventory kind=class offset=50 shown=50 truncated=true",
+ 'diagnostic code=results_truncated next_offset=100 message="More than 100 class entities exist. next: ix inventory --kind class --offset 100, or narrow with --path."',
+ ]);
+ });
+
+ it("keeps the first-page header as it was with offset 0", () => {
+ const nodes = [{ name: "Foo", kind: "class", provenance: { source_uri: "src/a.ts" } }];
+ expect(renderInventoryLlm("class", null, nodes, true, { offset: 0 })[0]).toBe("inventory kind=class shown=1 truncated=true");
+ });
+
+ it("past the end says so", () => {
+ const lines = renderInventoryLlm("class", null, [], false, { offset: 500 });
+ expect(lines).toEqual([
+ "inventory kind=class offset=500 shown=0",
+ 'diagnostic code=offset_past_end message="Fewer than 501 class entities exist; --offset 500 is past the end."',
+ ]);
+ });
+});
+
+describe("ix inventory --offset", () => {
+ afterEach(() => {
+ vi.restoreAllMocks();
+ listByKind.mockReset();
+ });
+
+ function rows(n: number) {
+ return Array.from({ length: n }, (_, i) => ({ name: `C${i + 1}`, kind: "class", provenance: { source_uri: "src/a.ts" } }));
+ }
+
+ async function run(argv: string[]): Promise<string[]> {
+ const log = vi.spyOn(console, "log").mockImplementation(() => {});
+ log.mockClear();
+ const program = new Command();
+ registerInventoryCommand(program);
+ await program.parseAsync(["inventory", "--kind", "class", ...argv], { from: "user" });
+ return log.mock.calls.map((c) => String(c[0]));
+ }
+
+ it("fetches offset+limit+1 rows and prints the page after the offset", async () => {
+ listByKind.mockResolvedValue(rows(5));
+ const out = await run(["--offset", "2", "--limit", "2", "--format", "json"]);
+ expect(listByKind).toHaveBeenCalledWith("class", expect.objectContaining({ limit: 5 }));
+ const json = JSON.parse(out[0]);
+ expect(json).toMatchObject({ offset: 2, shown: 2, truncated: true, nextOffset: 4 });
+ expect(json.byFile).toEqual([{ path: "src/a.ts", items: ["C3", "C4"] }]);
+ });
+
+ it("has no next offset when no row is left after the page", async () => {
+ listByKind.mockResolvedValue(rows(4));
+ const json = JSON.parse((await run(["--offset", "2", "--limit", "2", "--format", "json"]))[0]);
+ expect(json).toMatchObject({ offset: 2, shown: 2, truncated: false });
+ expect(json).not.toHaveProperty("nextOffset");
+ });
+
+ it("asks for no more than the server's row cap", async () => {
+ listByKind.mockResolvedValue(rows(10));
+ await run(["--offset", "4990", "--limit", "50", "--format", "json"]);
+ expect(listByKind).toHaveBeenCalledWith("class", expect.objectContaining({ limit: LIST_MAX_ROWS }));
+ });
+
+ it("an offset at or past the server cap is not past the end", async () => {
+ listByKind.mockResolvedValue(rows(LIST_MAX_ROWS));
+ for (const offset of ["5000", "6000"]) {
+ const out = await run(["--offset", offset, "--limit", "50", "--format", "llm"]);
+ expect(out[0]).toBe(`inventory kind=class offset=${offset} shown=0 truncated=true`);
+ expect(out[1]).toBe(
+ `diagnostic code=results_truncated message="The server returns at most 5000 class entities, so how many exist is unknown; --offset ${offset} is beyond the rows it returns. Narrow with --path (or a more specific --kind) to see the rest."`,
+ );
+ expect(out.join("\n")).not.toContain("offset_past_end");
+ }
+ });
+
+ it("the page that reaches the server cap stays truncated with no next page", async () => {
+ listByKind.mockResolvedValue(rows(LIST_MAX_ROWS));
+ const json = JSON.parse((await run(["--offset", "4990", "--limit", "50", "--format", "json"]))[0]);
+ expect(json).toMatchObject({ offset: 4990, shown: 10, truncated: true, capped: true });
+ expect(json).not.toHaveProperty("nextOffset");
+ const llm = await run(["--offset", "4990", "--limit", "50", "--format", "llm"]);
+ expect(llm[1]).toContain("this page ends at row 5000, the last it returns");
+ });
+
+ it("a --limit over the cap does not read a full server answer as the whole list", async () => {
+ listByKind.mockResolvedValue(rows(LIST_MAX_ROWS));
+ const json = JSON.parse((await run(["--limit", "10000", "--format", "json"]))[0]);
+ expect(json).toMatchObject({ shown: LIST_MAX_ROWS, truncated: true, capped: true });
+ });
+
+ it("a short answer to a request over the cap is the whole list", async () => {
+ listByKind.mockResolvedValue(rows(4995));
+ const out = await run(["--offset", "6000", "--limit", "50", "--format", "llm"]);
+ expect(out[1]).toContain("code=offset_past_end");
+ const json = JSON.parse((await run(["--offset", "4990", "--limit", "50", "--format", "json"]))[0]);
+ expect(json).toMatchObject({ shown: 5, truncated: false });
+ expect(json).not.toHaveProperty("capped");
+ });
+
+ it("text mode names the cap on an empty page past it", async () => {
+ listByKind.mockResolvedValue(rows(LIST_MAX_ROWS));
+ const out = await run(["--offset", "5000"]);
+ expect(out[0]).toContain("The server returns at most 5000 class entities");
+ });
+
+ it("names the next command in llm output, repeating --limit", async () => {
+ listByKind.mockResolvedValue(rows(5));
+ const out = await run(["--offset", "2", "--limit", "2", "--format", "llm"]);
+ expect(out[0]).toBe("inventory kind=class offset=2 shown=2 truncated=true");
+ expect(out[1]).toContain("next_offset=4");
+ expect(out[1]).toContain("next: ix inventory --kind class --limit 2 --format llm --offset 4");
+ });
});
describe("text result ranking", () => {
@@ -208,4 +342,92 @@ describe("text result ranking", () => {
expect(slice.rows.map((r) => r.path)).toEqual(["src/a.ts"]);
expect(slice.total).toBe(3);
});
+
+ it("orders code before tests before prose, then by path, then by line", () => {
+ const rows = [
+ text("README.md", 1), text("src/b.ts", 9), text("src/__tests__/a.test.ts", 1),
+ text("src/b.ts", 2), text("src/a.ts", 5), text("docs/a.md", 1),
+ ];
+ expect([...rows].sort(compareTextResults).map((r) => `${r.path}:${r.line_start}`)).toEqual([
+ "src/a.ts:5", "src/b.ts:2", "src/b.ts:9", "src/__tests__/a.test.ts:1", "README.md:1", "docs/a.md:1",
+ ]);
+ });
+
+ it("keepTopK keeps the best k, in order, with ties in arrival order", () => {
+ const rows = [text("docs/a.md", 1, "x"), text("src/a.ts", 1, "first"), text("src/a.ts", 1, "second")];
+ expect(keepTopK(rows, 2, compareTextResults).map((r) => r.snippet)).toEqual(["first", "second"]);
+ expect(keepTopK(rows, 0, compareTextResults)).toEqual([]);
+ });
+
+ it("pages of one ranked list are disjoint and add up to its head", () => {
+ const all = Array.from({ length: 100 }, (_, i) => text(`src/f${String(i).padStart(3, "0")}.ts`, 1)).sort(compareTextResults);
+ const pages = [0, 20, 40].map((offset) => sliceRanked(all, 20, undefined, offset).rows);
+ const union = pages.flat();
+ expect(new Set(union.map((r) => r.path)).size).toBe(60);
+ expect(union).toEqual(all.slice(0, 60));
+ });
+});
+
+function text(path: string, line: number, snippet = "needle"): TextResult {
+ return { path, line_start: line, line_end: line, snippet, engine: "ripgrep", score: 1 };
+}
+
+describe("renderTextResultsLlm paging", () => {
+ const all = Array.from({ length: 100 }, (_, i) => text(`src/f${String(i).padStart(3, "0")}.ts`, 1));
+
+ it("leaves the first-page header alone", () => {
+ const lines = renderTextResultsLlm(sliceRanked(all, 20));
+ expect(lines[0]).toBe("text shown=20 scanned=100 truncated=true");
+ expect(lines[1]).toBe('diagnostic code=results_truncated next_offset=20 message="100 matches scanned; showing 20. next: --offset 20"');
+ });
+
+ it("a middle page names its offset and the next command", () => {
+ const slice = { ...sliceRanked(all, 20, undefined, 20), next: "ix text --limit 20 --offset 40 needle" };
+ const lines = renderTextResultsLlm(slice, { windowFull: true });
+ expect(lines[0]).toBe("text offset=20 shown=20 scanned=100 truncated=true");
+ expect(lines[1]).toBe('diagnostic code=results_truncated next_offset=40 message="100 matches scanned; showing 21-40. next: ix text --limit 20 --offset 40 needle"');
+ });
+
+ it("the page that reaches the end of a full window says to narrow instead", () => {
+ const lines = renderTextResultsLlm(sliceRanked(all, 20, undefined, 80), { windowFull: true });
+ expect(lines[0]).toBe("text offset=80 shown=20 scanned=100 truncated=true");
+ expect(lines[1]).toBe('diagnostic code=results_truncated message="100 matches scanned; showing 81-100; the scan stops at 100. Narrow with --path or --language, or raise --limit."');
+ expect(lines.join("\n")).not.toContain("next_offset");
+ });
+
+ it("the last page of a window that held every match has no hint", () => {
+ const lines = renderTextResultsLlm(sliceRanked(all, 20, undefined, 80));
+ expect(lines[0]).toBe("text offset=80 shown=20 scanned=100");
+ expect(lines.some((l) => l.startsWith("diagnostic"))).toBe(false);
+ });
+
+ it("past the end of a window that held every match says so", () => {
+ const lines = renderTextResultsLlm(sliceRanked(all, 20, undefined, 120));
+ expect(lines).toEqual([
+ "text offset=120 shown=0 scanned=100",
+ 'diagnostic code=offset_past_end message="100 matches scanned; --offset 120 is past the end."',
+ ]);
+ });
+
+ it("past a full window is not past the end: it stays truncated and says to narrow", () => {
+ for (const offset of [100, 120]) {
+ const lines = renderTextResultsLlm(sliceRanked(all, 20, undefined, offset), { windowFull: true });
+ expect(lines).toEqual([
+ `text offset=${offset} shown=0 scanned=100 truncated=true`,
+ `diagnostic code=results_truncated message="100 matches scanned; --offset ${offset} is past the scan window, which stops at 100. Narrow with --path or --language, or raise --limit."`,
+ ]);
+ }
+ });
+
+ it("text mode prints the narrow hint, not past-the-end, past a full window", () => {
+ const log = vi.spyOn(console, "log").mockImplementation(() => {});
+ try {
+ formatTextResults(sliceRanked(all, 20, undefined, 100), "text", { windowFull: true });
+ const out = log.mock.calls.map((c) => String(c[0])).join("\n");
+ expect(out).toContain("is past the scan window, which stops at 100");
+ expect(out).not.toContain("is past the end");
+ } finally {
+ log.mockRestore();
+ }
+ });
});
diff --git a/ix-cli/src/cli/__tests__/mcp-structured-output.test.ts b/ix-cli/src/cli/__tests__/mcp-structured-output.test.ts
index a87aae3e..5d310353 100644
--- a/ix-cli/src/cli/__tests__/mcp-structured-output.test.ts
+++ b/ix-cli/src/cli/__tests__/mcp-structured-output.test.ts
@@ -174,7 +174,10 @@ describe("ix mcp structured output", () => {
// that number.
// The full catalog's descriptions now say when to use each tool and when
// not to, which is what a caller of `--tools=all` is choosing between.
- expect(total).toBeLessThan(17_000);
+ // Raised from 17,000 for the undescribed `offset` on ix_text and
+ // ix_inventory (~64 bytes each), which print next_offset= and so have to
+ // take it back. Neither is in the core set.
+ expect(total).toBeLessThan(17_200);
});
it("attaches structuredContent only where an outputSchema was declared", async () => {
diff --git a/ix-cli/src/cli/__tests__/mcp-tool-params.test.ts b/ix-cli/src/cli/__tests__/mcp-tool-params.test.ts
index f442413c..2078cc23 100644
--- a/ix-cli/src/cli/__tests__/mcp-tool-params.test.ts
+++ b/ix-cli/src/cli/__tests__/mcp-tool-params.test.ts
@@ -76,7 +76,7 @@ describe("resolving an ambiguous symbol over MCP", () => {
const tools = (await client.listTools()).tools;
const has = (name: string) =>
"offset" in (((tools.find((t) => t.name === name)!.inputSchema) as { properties?: Record<string, unknown> }).properties ?? {});
- for (const name of ["ix_neighbors", "ix_callers", "ix_callees", "ix_imports", "ix_imported_by"]) {
+ for (const name of ["ix_neighbors", "ix_callers", "ix_callees", "ix_imports", "ix_imported_by", "ix_text", "ix_inventory"]) {
expect(has(name), name).toBe(true);
}
for (const name of ["ix_impact", "ix_read", "ix_explain"]) {
@@ -84,6 +84,30 @@ describe("resolving an ambiguous symbol over MCP", () => {
}
});
+ it("forwards --offset on ix_text and ix_inventory, which print next_offset= too", async () => {
+ const { calls, runIx } = recorder();
+ const client = await connect(runIx);
+ await client.callTool({ name: "ix_text", arguments: { pattern: "needle", offset: 20 } });
+ await client.callTool({ name: "ix_inventory", arguments: { kind: "class", offset: 50 } });
+ await client.callTool({ name: "ix_text", arguments: { pattern: "needle" } });
+ await client.callTool({ name: "ix_inventory", arguments: { kind: "class" } });
+ expect(calls[0]).toContain("--offset=20");
+ expect(calls[1]).toContain("--offset=50");
+ expect(calls[2].some((a) => a.startsWith("--offset"))).toBe(false);
+ expect(calls[3].some((a) => a.startsWith("--offset"))).toBe(false);
+ });
+
+ it("rejects a negative offset on ix_text and ix_inventory", async () => {
+ const { calls, runIx } = recorder();
+ const client = await connect(runIx);
+ for (const [name, args] of [["ix_text", { pattern: "needle" }], ["ix_inventory", { kind: "class" }]] as const) {
+ const result = await client.callTool({ name, arguments: { ...args, offset: -1 } })
+ .then((r) => (r as CallToolResult).isError === true, () => true);
+ expect(result, name).toBe(true);
+ }
+ expect(calls).toEqual([]);
+ });
+
it("sends nothing for the flags the caller left out", async () => {
const { calls, runIx } = recorder();
const client = await connect(runIx);
diff --git a/ix-cli/src/cli/__tests__/text-stream.test.ts b/ix-cli/src/cli/__tests__/text-stream.test.ts
index ae64c7cb..2c574249 100644
--- a/ix-cli/src/cli/__tests__/text-stream.test.ts
+++ b/ix-cli/src/cli/__tests__/text-stream.test.ts
@@ -93,6 +93,92 @@ describe("text ripgrep stream lifecycle", () => {
await rejected;
expect(log).not.toHaveBeenCalled();
});
+ function matchAt(file: string, line: number) {
+ return JSON.stringify({ type: "match", data: {
+ path: { text: file }, line_number: line, lines: { text: `${file}:${line}` },
+ } }) + "\n";
+ }
+ async function runArgs(argv: string[], records: string[]): Promise<string[]> {
+ const program = new Command();
+ registerTextCommand(program);
+ const pending = program.parseAsync(["text", "--root", process.cwd(), ...argv], { from: "user" });
+ for (const r of records) child.stdout.write(r);
+ close(0);
+ await pending;
+ return log.mock.calls.map((c: unknown[]) => String(c[0]));
+ }
+ it("is byte-stable across arrival orders and keeps the best rows, not the first", async () => {
+ // --limit 1 scans a window of 5; twelve matches overflow it, and the one
+ // code match arrives last, after a full window of prose.
+ const records = [
+ ...Array.from({ length: 6 }, (_, i) => matchAt(`docs/d${i}.md`, i + 1)),
+ ...Array.from({ length: 5 }, (_, i) => matchAt(`tests/t${i}.ts`, 1)),
+ matchAt("src/code.ts", 7),
+ ];
+ const first = await runArgs(["needle", "--limit", "1", "--format", "json"], records);
+ log.mockClear();
+ child = Object.assign(new EventEmitter(), {
+ stdout: new PassThrough(), stderr: new PassThrough(), kill: vi.fn(() => true),
+ });
+ spawn.mockReturnValue(child);
+ const second = await runArgs(["needle", "--limit", "1", "--format", "json"], [...records].reverse());
+ expect(second).toEqual(first);
+ expect(JSON.parse(first[0]).map((r: { path: string }) => r.path)).toEqual(["src/code.ts"]);
+ });
+ it("asks ripgrep for one match per file past --limit, to tell a full file from a cut-off one", async () => {
+ await runArgs(["needle", "--limit", "5", "--format", "llm"], []);
+ const argv = spawn.mock.calls[0][1] as string[];
+ expect(argv.slice(0, argv.indexOf("--"))).toEqual(["--json", "--max-count", "6"]);
+ });
+ it("reports a file with more matches than ripgrep's per-file cap as a truncated scan", async () => {
+ // --limit 5 caps each file at 5; the sixth match is ripgrep's probe and is not shown.
+ const out = await runArgs(["needle", "--limit", "5", "--format", "llm"],
+ Array.from({ length: 6 }, (_, i) => matchAt("src/a.ts", i + 1)));
+ expect(out[0]).toBe("text shown=5 scanned=5 truncated=true");
+ expect(out[1]).toContain("Narrow with --path or --language");
+ expect(out.slice(2).map((l) => l.match(/line=(\d+)/)?.[1])).toEqual(["1", "2", "3", "4", "5"]);
+ });
+ it.each(["1", "5"])("does not call a file with exactly --limit %s matches truncated", async (limit) => {
+ const n = Number(limit);
+ const out = await runArgs(["needle", "--limit", limit, "--format", "llm"],
+ Array.from({ length: n }, (_, i) => matchAt("src/a.ts", i + 1)));
+ expect(out[0]).toBe(`text shown=${n} scanned=${n}`);
+ expect(out.some((l) => l.startsWith("diagnostic"))).toBe(false);
+ });
+ it("pages inside the scan window and repeats --limit in the next command", async () => {
+ const records = Array.from({ length: 5 }, (_, i) => matchAt(`src/f${i}.ts`, i + 1));
+ const out = await runArgs(["needle", "--limit", "2", "--offset", "2", "--format", "llm"], records);
+ expect(out[0]).toBe("text offset=2 shown=2 scanned=5 truncated=true");
+ expect(out[1]).toContain("next_offset=4");
+ expect(out[1]).toContain("next: ix text --limit 2 --format llm --root ");
+ expect(out[1]).toMatch(/--offset 4 needle"$/);
+ expect(out.slice(2).map((l) => l.match(/line=(\d+)/)?.[1])).toEqual(["3", "4"]);
+ });
+ it("ranks a match record whose fields come in another order the same way", async () => {
+ const reordered = JSON.stringify({ data: { line_number: 2, lines: { text: "late" }, path: { text: "src/a.ts" } }, type: "match" }) + "\n";
+ const out = await runArgs(["needle", "--limit", "5", "--format", "json"], [matchAt("src/b.ts", 1), reordered, matchAt("src/a.ts", 1)]);
+ expect(JSON.parse(out[0]).map((r: { path: string; line_start: number }) => `${r.path}:${r.line_start}`))
+ .toEqual(["src/a.ts:1", "src/a.ts:2", "src/b.ts:1"]);
+ });
+ it("counts and names files whose paths are not UTF-8 (ripgrep sends them as bytes)", async () => {
+ const bytesMatch = (raw: Buffer, line: number) => JSON.stringify({ type: "match", data: {
+ path: { bytes: raw.toString("base64") }, line_number: line, lines: { text: "foo" },
+ } }) + "\n";
+ const a = Buffer.concat([Buffer.from("a"), Buffer.from([0xff]), Buffer.from(".txt")]);
+ const b = Buffer.concat([Buffer.from("b"), Buffer.from([0xfe]), Buffer.from(".txt")]);
+ const records = [1, 2, 3].flatMap((n) => [bytesMatch(a, n), bytesMatch(b, n)]).concat(matchAt("c.txt", 1));
+ // Seven matches, at most three per file: under the per-file cap of 4, so nothing is cut.
+ const out = await runArgs(["needle", "--limit", "4", "--format", "llm"], records);
+ expect(out[0]).toBe("text shown=4 scanned=7 truncated=true");
+ expect(out[1]).toContain("next_offset=4");
+ expect(out.slice(2).map((l) => l.match(/path=(\S+) line=(\d+)/)?.slice(1).join(":")))
+ .toEqual(["a�.txt:1", "a�.txt:2", "a�.txt:3", "b�.txt:1"]);
+ });
+ it("guards a term starting with a dash in the next command", async () => {
+ const records = Array.from({ length: 5 }, (_, i) => matchAt(`src/f${i}.ts`, i + 1));
+ const out = await runArgs(["--limit", "2", "--format", "llm", "--", "-x"], records);
+ expect(out[1]).toMatch(/--offset 2 -- -x"$/);
+ });
it("preserves the missing-ripgrep error", async () => {
const pending = run();
const rejected = expect(pending).rejects.toThrow("exit 1");
diff --git a/ix-cli/src/cli/commands/inventory.ts b/ix-cli/src/cli/commands/inventory.ts
index e9423ee5..0cde96fd 100644
--- a/ix-cli/src/cli/commands/inventory.ts
+++ b/ix-cli/src/cli/commands/inventory.ts
@@ -9,6 +9,26 @@ import { relativePath, printJson } from "../format.js";
import { llmLine } from "../llm.js";
import { projectRow } from "../output-shape.js";
import { normalizePathSeparators } from "../path-match.js";
+import { nextPageCommand, OFFSET_FLAG, parseOffset } from "../paging.js";
+
+/**
+ * The most rows one /v1/list call returns. The backend clamps the request's
+ * `limit` to this (Ix-memory ListRoutes.scala: `ReadBounds.limit(body.limit,
+ * 200, 5000)`), and /v1/list has no offset, so no page can reach past it.
+ */
+export const LIST_MAX_ROWS = 5000;
+
+/** Where an `ix inventory` page starts, and the command for the next one. */
+export interface InventoryPaging {
+ offset?: number;
+ nextOffset?: number;
+ next?: string;
+ /**
+ * The server stopped at LIST_MAX_ROWS before the page's rows (plus the one
+ * past them) were in: how many entities exist is unknown past that row.
+ */
+ capped?: boolean;
+}
/**
* Render `ix inventory` as llm records: a header line then one `file` row per
@@ -21,10 +41,14 @@ import { normalizePathSeparators } from "../path-match.js";
* whether the window it asked for came back full. Printing the length of a
* capped list as a total is the claim that made `ix callers` report 50 callers
* for a symbol with 212.
+ *
+ * Paged (`--offset`), the header says `offset=` and the truncation diagnostic
+ * carries `next_offset=` with the command for the next page.
*/
export function renderInventoryLlm(
- kind: string, scope: string | null, nodes: any[], truncated = false,
+ kind: string, scope: string | null, nodes: any[], truncated = false, paging?: InventoryPaging,
): string[] {
+ const offset = paging?.offset ?? 0;
const byFile = new Map<string, string[]>();
const ungrouped: Array<{ name: string; kind: string }> = [];
for (const n of nodes) {
@@ -42,13 +66,21 @@ export function renderInventoryLlm(
const lines = [llmLine("inventory", [
["kind", kind],
["scope", scope ?? undefined],
+ ["offset", offset > 0 ? offset : undefined],
["shown", nodes.length],
["truncated", truncated || undefined],
])];
if (truncated) {
lines.push(llmLine("diagnostic", [
["code", "results_truncated"],
- ["message", `More than ${nodes.length} ${kind} entities exist. Raise --limit, or narrow with --path.`],
+ ["next_offset", paging?.nextOffset],
+ ["message", inventoryTruncationMessage(kind, offset, nodes.length, paging)],
+ ]));
+ }
+ if (nodes.length === 0 && offset > 0 && !truncated) {
+ lines.push(llmLine("diagnostic", [
+ ["code", "offset_past_end"],
+ ["message", inventoryPastEndMessage(kind, offset)],
]));
}
for (const [path, names] of byFile) {
@@ -60,6 +92,26 @@ export function renderInventoryLlm(
return lines;
}
+function inventoryTruncationMessage(kind: string, offset: number, shown: number, paging?: InventoryPaging): string {
+ if (paging?.capped && paging.nextOffset === undefined) return inventoryCapMessage(kind, offset, shown);
+ const more = `More than ${offset + shown} ${kind} entities exist.`;
+ if (paging?.nextOffset === undefined) return `${more} Raise --limit, or narrow with --path.`;
+ return `${more} next: ${paging.next ?? `--offset ${paging.nextOffset}`}, or narrow with --path.`;
+}
+
+/** A page at or past the server's row cap: the list goes on, but no offset reaches it. */
+function inventoryCapMessage(kind: string, offset: number, shown: number): string {
+ const head = `The server returns at most ${LIST_MAX_ROWS} ${kind} entities, so how many exist is unknown`;
+ const where = shown > 0
+ ? `this page ends at row ${offset + shown}, the last it returns`
+ : `--offset ${offset} is beyond the rows it returns`;
+ return `${head}; ${where}. Narrow with --path (or a more specific --kind) to see the rest.`;
+}
+
+function inventoryPastEndMessage(kind: string, offset: number): string {
+ return `Fewer than ${offset + 1} ${kind} entities exist; --offset ${offset} is past the end.`;
+}
+
export function registerInventoryCommand(program: Command): void {
program
.command("inventory")
@@ -67,6 +119,7 @@ export function registerInventoryCommand(program: Command): void {
.requiredOption("--kind <kind>", "Entity kind to list (class, method, function, file, module, etc.)")
.option("--path <path>", "Filter by source file path substring")
.option("--limit <n>", "Max results", "50")
+ .option("--offset <n>", OFFSET_FLAG, "0")
.option("--format <fmt>", "Output format (text|json|llm)", "text")
.addHelpText("after", `
Examples:
@@ -74,12 +127,14 @@ Examples:
ix inventory --kind class --path memory-layer
ix inventory --kind file --path ix-cli/src
ix inventory --kind method --limit 100
+ ix inventory --kind class --offset 50
ix inventory --format json --kind class`)
.action(async (opts: {
- kind: string; path?: string; limit: string; format: string;
- }) => {
+ kind: string; path?: string; limit: string; offset?: string; format: string;
+ }, cmd: Command) => {
const client = createClient();
const limit = parseInt(opts.limit, 10);
+ const offset = parseOffset(opts.offset);
// Pass --path as a server-side scope so the LIMIT is applied AFTER path
// filtering (otherwise a capped fetch can truncate away the target before
@@ -93,9 +148,18 @@ Examples:
const pathNeedle = opts.path ? normalizePathSeparators(opts.path) : undefined;
// One past the limit, so a full window is distinguishable from an exact
// fit. The backend has no count endpoint, so this is the only honest
- // answer available: "there is at least one more".
- let nodes = await client.listByKind(opts.kind, { limit: limit + 1, workspaceId: systemId ? undefined : resolveWorkspaceId(), scope: pathNeedle, systemId });
- const truncated = nodes.length > limit;
+ // answer available: "there is at least one more". /v1/list has no
+ // offset, so a page past the first fetches the rows before it too and
+ // drops them here; its order is stable, so pages line up.
+ //
+ // The server returns at most LIST_MAX_ROWS rows. A request over that
+ // which comes back full may have been cut, so the list is not known to
+ // end there: it stays truncated and no offset is called past the end.
+ const wanted = offset + limit + 1;
+ let nodes = await client.listByKind(opts.kind, { limit: Math.min(wanted, LIST_MAX_ROWS), workspaceId: systemId ? undefined : resolveWorkspaceId(), scope: pathNeedle, systemId });
+ const capped = wanted > LIST_MAX_ROWS && nodes.length >= LIST_MAX_ROWS;
+ const moreFetched = nodes.length > offset + limit;
+ const truncated = moreFetched || capped;
if (pathNeedle) {
nodes = nodes.filter((n) => {
@@ -109,8 +173,13 @@ Examples:
});
}
- nodes = nodes.slice(0, limit);
+ nodes = nodes.slice(offset, offset + limit);
const scope = opts.path ?? null;
+ const paging: InventoryPaging = { offset, ...(capped ? { capped } : {}) };
+ if (moreFetched && nodes.length > 0) {
+ paging.nextOffset = offset + nodes.length;
+ paging.next = nextPageCommand(cmd, [], paging.nextOffset);
+ }
if (opts.format === "json") {
// Group by file to avoid repeating the same path on every entry
@@ -135,8 +204,12 @@ Examples:
const output: any = {
kind: opts.kind,
scope: scope ?? undefined,
+ offset,
shown: nodes.length,
truncated,
+ ...(paging.nextOffset !== undefined ? { nextOffset: paging.nextOffset } : {}),
+ // The server's row cap was reached: the entity count is unknown.
+ ...(capped ? { capped: true } : {}),
byFile: grouped,
};
if (ungrouped.length > 0) output.ungrouped = ungrouped;
@@ -145,16 +218,24 @@ Examples:
}
if (opts.format === "llm") {
- for (const line of renderInventoryLlm(opts.kind, scope, nodes, truncated)) console.log(line);
+ for (const line of renderInventoryLlm(opts.kind, scope, nodes, truncated, paging)) console.log(line);
return;
}
if (nodes.length === 0) {
- console.log(`No ${opts.kind} entities found${scope ? ` in ${scope}` : ""}.`);
+ console.log(capped
+ ? inventoryCapMessage(opts.kind, offset, 0)
+ : offset > 0 && !truncated
+ ? inventoryPastEndMessage(opts.kind, offset)
+ : `No ${opts.kind} entities found${scope ? ` in ${scope}` : ""}.`);
return;
}
- const more = truncated ? " (more exist — raise --limit)" : "";
+ const more = truncated
+ ? paging.next ? ` (more exist — next: ${paging.next})`
+ : capped ? ` (${inventoryCapMessage(opts.kind, offset, nodes.length)})`
+ : " (more exist — raise --limit)"
+ : "";
console.log(`Inventory: ${nodes.length} ${opts.kind} entities${scope ? ` in ${scope}` : ""}${more}`);
for (const n of nodes) {
const name = n.name || n.attrs?.name || "(unnamed)";
diff --git a/ix-cli/src/cli/commands/text.ts b/ix-cli/src/cli/commands/text.ts
index 7dfd4d9a..5df6d560 100644
--- a/ix-cli/src/cli/commands/text.ts
+++ b/ix-cli/src/cli/commands/text.ts
@@ -5,19 +5,62 @@ import { createInterface } from "node:readline";
import path from "node:path";
import type { Command } from "commander";
import { formatTextResults, printJson, sliceRanked, type TextResult } from "../format.js";
+import { nextPageCommand, OFFSET_FLAG, parseOffset } from "../paging.js";
import { isPathInsideResolvedRoot, resolveWorkspaceRoot } from "../config.js";
import { stderr } from "../stderr.js";
import { llmError } from "../llm.js";
import { currentRunSignal } from "../../client/run-signal.js";
-type RunRipgrep = (args: string[], limit: number) => Promise<{ stdout: string }>;
+/**
+ * Runs ripgrep and returns at most `limit` match lines. `matched`, when given,
+ * counts every match record within the per-file `--max-count`, kept or not.
+ * `capped` says some file has more matches than `--max-count` let through, so
+ * the scan left matches out. The command sorts whatever comes back, so an
+ * injected executor's order does not matter.
+ */
+type RunRipgrep = (args: string[], limit: number) => Promise<{ stdout: string; matched?: number; capped?: boolean }>;
+
+/**
+ * Keep the best `k` of `items` under `compare`, in that order.
+ *
+ * The sort is stable, so items that compare equal keep the order they arrived
+ * in — which makes arrival the last tie-break without a sequence field.
+ */
+export function keepTopK<T>(items: T[], k: number, compare: (a: T, b: T) => number): T[] {
+ const sorted = [...items].sort(compare);
+ return sorted.length > k ? sorted.slice(0, Math.max(k, 0)) : sorted;
+}
-async function runRipgrep(args: string[], limit: number): Promise<{ stdout: string }> {
+/**
+ * The default executor. `root` is the workspace root matches are ranked
+ * against: the band regexes have to see workspace-relative paths, or a root
+ * under a `tests/` directory would put every match in the test band.
+ *
+ * ripgrep is asked for one match per file more than `--max-count` says. That
+ * extra match is never kept or counted; it only shows the file had more. A
+ * file with exactly `--max-count` matches looks the same as a cut-off one
+ * otherwise, and calling it cut off told every `--limit 1` search to narrow.
+ */
+async function runRipgrep(args: string[], limit: number, root: string): Promise<{ stdout: string; matched: number; capped: boolean }> {
+ const { argv, perFileCap } = probePerFileCap(args);
+ const perFile = new Map<string, number>();
+ let capped = false;
return new Promise((resolve, reject) => {
// Killed when an `ix mcp` tool call that runs it times out (run-signal.ts).
- const child = spawn("rg", args, { stdio: ["ignore", "pipe", "pipe"], signal: currentRunSignal() });
+ const child = spawn("rg", argv, { stdio: ["ignore", "pipe", "pipe"], signal: currentRunSignal() });
const lines = createInterface({ input: child.stdout, crlfDelay: Infinity });
- const matches: string[] = [];
+ // The best `limit` matches seen so far, not the first `limit` to arrive:
+ // ripgrep's walker is parallel, so which matches arrive first changes from
+ // run to run. Ranking every match keeps the output byte-stable. The buffer
+ // is cut back to `limit` whenever it reaches twice that, so memory stays
+ // bounded however many matches there are, and once it has been cut a
+ // match that ranks no better than the last row kept is dropped unseen
+ // (a later arrival loses a tie, as it would in the stable sort).
+ let kept: Array<{ key: RankKey; line: string }> = [];
+ let worst: RankKey | undefined;
+ const byKey = (a: { key: RankKey }, b: { key: RankKey }) => compareRankKeys(a.key, b.key);
+ const relative = new Map<string, { path: string; band: number }>();
+ let matched = 0;
let errorOutput = "";
child.stderr.setEncoding("utf8");
@@ -25,20 +68,36 @@ async function runRipgrep(args: string[], limit: number): Promise<{ stdout: stri
errorOutput = (errorOutput + chunk).slice(0, 64 * 1024);
});
lines.on("line", (line: string) => {
- if (matches.length >= limit) return;
- try {
- if (JSON.parse(line).type !== "match") return;
- } catch {
+ const found = matchRankFields(line);
+ if (!found) return;
+ const inFile = (perFile.get(found.file) ?? 0) + 1;
+ perFile.set(found.file, inFile);
+ if (inFile > perFileCap) {
+ capped = true;
return;
}
- matches.push(line);
+ matched++;
+ if (limit <= 0) return;
+ let file = relative.get(found.file);
+ if (!file) {
+ const rel = workspaceRelativePath(found.name, root);
+ file = { path: rel, band: textResultPriority(rel) };
+ relative.set(found.file, file);
+ }
+ const key: RankKey = { band: file.band, path: file.path, line: found.line, file: found.file };
+ if (worst && compareRankKeys(key, worst) >= 0) return;
+ kept.push({ key, line });
+ if (kept.length >= limit * 2) {
+ kept = keepTopK(kept, limit, byKey);
+ worst = kept[kept.length - 1]?.key;
+ }
});
child.on("error", reject);
child.on("close", (code, signal) => {
lines.close();
// Drain through exit so errors after the retained matches are not hidden.
if (code === 0 || code === 1) {
- resolve({ stdout: matches.join("\n") });
+ resolve({ stdout: keepTopK(kept, limit, byKey).map((m) => m.line).join("\n"), matched, capped });
} else {
reject(Object.assign(new Error("ripgrep failed"), {
code,
@@ -50,6 +109,118 @@ async function runRipgrep(args: string[], limit: number): Promise<{ stdout: stri
});
}
+/**
+ * `args` with its own `--max-count` (the one before `--`; anything after is
+ * the caller's term and path) raised by one, and the cap it had. A cap of 0
+ * or none at all is left as it is: there is nothing to tell apart.
+ */
+function probePerFileCap(args: string[]): { argv: string[]; perFileCap: number } {
+ const separator = args.indexOf("--");
+ const flags = separator >= 0 ? args.slice(0, separator) : args;
+ const capIndex = flags.indexOf("--max-count");
+ const cap = capIndex >= 0 ? Number(flags[capIndex + 1]) : NaN;
+ if (!Number.isSafeInteger(cap) || cap <= 0) {
+ return { argv: args, perFileCap: Number.isFinite(cap) ? cap : Infinity };
+ }
+ const argv = [...args];
+ argv[capIndex + 1] = String(cap + 1);
+ return { argv, perFileCap: cap };
+}
+
+/**
+ * What a match ranks by: band, then workspace-relative path, then line. `file`
+ * is ripgrep's own name for the file (see `ripgrepPath`) and only breaks ties
+ * between two files whose decoded paths are the same string.
+ */
+interface RankKey {
+ band: number;
+ path: string;
+ line: number;
+ file?: string;
+}
+
+function compareRankKeys(a: RankKey, b: RankKey): number {
+ if (a.band !== b.band) return a.band - b.band;
+ if (a.path !== b.path) return a.path < b.path ? -1 : 1;
+ if (a.line !== b.line) return a.line - b.line;
+ const fileA = a.file ?? "";
+ const fileB = b.file ?? "";
+ return fileA === fileB ? 0 : fileA < fileB ? -1 : 1;
+}
+
+/**
+ * A path from a ripgrep `--json` record. ripgrep sends a path that is not
+ * valid UTF-8 as `{"bytes": <base64>}` instead of `{"text": ...}`. `id` tells
+ * files apart exactly (it keys the per-file `--max-count` count); `name` is
+ * the path to show and rank by, with bytes that are not UTF-8 decoded to
+ * U+FFFD. Reading only `text` gave every such file the name "", so they all
+ * counted as one file and printed as the workspace root.
+ */
+function ripgrepPath(p: any): { id: string; name: string } {
+ if (typeof p?.text === "string") return { id: `t:${p.text}`, name: p.text };
+ if (typeof p?.bytes === "string") {
+ return { id: `b:${p.bytes}`, name: Buffer.from(p.bytes, "base64").toString("utf8") };
+ }
+ return { id: "", name: "" };
+}
+
+// ripgrep's `--json` match records start with these fields in this order. A
+// line that does not fit (a non-UTF-8 path arrives as `bytes`, say) falls back
+// to a full parse; nothing is ranked on a guess.
+const MATCH_HEAD = /^\{"type":"match","data":\{"path":\{"text":("(?:[^"\\]|\\.)*")\}/;
+const LINE_NUMBER = /"line_number":(\d+)/;
+
+/**
+ * The path and line of a ripgrep `match` record, or undefined for any other
+ * line. Most records are read with two regexes instead of a full JSON.parse:
+ * ripgrep is drained to exit and every match is ranked, so on a dense term
+ * the parse was most of the cost.
+ */
+function matchRankFields(line: string): { file: string; name: string; line: number } | undefined {
+ const head = MATCH_HEAD.exec(line);
+ const number = head ? LINE_NUMBER.exec(line) : null;
+ if (head && number) {
+ try {
+ const { id, name } = ripgrepPath({ text: JSON.parse(head[1]) });
+ return { file: id, name, line: Number(number[1]) };
+ } catch {
+ // fall through to the full parse
+ }
+ }
+ let parsed: any;
+ try {
+ parsed = JSON.parse(line);
+ } catch {
+ return undefined;
+ }
+ if (parsed?.type !== "match") return undefined;
+ const { id, name } = ripgrepPath(parsed.data?.path);
+ return { file: id, name, line: parsed.data?.line_number ?? 0 };
+}
+
+/** A path ripgrep printed, relative to `root` with forward slashes. */
+function workspaceRelativePath(filePath: string, root: string): string {
+ const absoluteFilePath = path.isAbsolute(filePath)
+ ? filePath
+ : path.resolve(process.cwd(), filePath);
+ return path.relative(root, absoluteFilePath).split(path.sep).join("/");
+}
+
+/** One ripgrep `match` record as a result, its path relative to `root`. */
+function textResultFromMatch(data: any, root: string): TextResult {
+ const filePath = ripgrepPath(data?.path).name;
+ const lineNum = data?.line_number ?? 0;
+ return {
+ path: workspaceRelativePath(filePath, root),
+ line_start: lineNum,
+ line_end: lineNum,
+ snippet: data?.lines?.text ?? "",
+ engine: "ripgrep",
+ score: 1.0,
+ language: inferLanguage(filePath),
+ };
+}
+
export function resolveTextSearchPath(root: string, searchPath: string): string {
return path.resolve(root, searchPath);
}
@@ -63,6 +234,11 @@ export function resolveTextSearchPath(root: string, searchPath: string): string
* Ranking the twenty rows it already cut cannot fix that; the scan has to see
* more than it prints. Bounded so a term matching everything is still one
* cheap pass.
+ *
+ * The window is the best `scanWindow(limit)` matches under
+ * `compareTextResults`, taken over every match ripgrep finds — not the first
+ * that many to arrive, which would differ between runs. `--offset` pages are
+ * cut from this one window, so they only line up while `--limit` stays fixed.
*/
const SCAN_WINDOW_FACTOR = 5;
const SCAN_WINDOW_MAX = 500;
@@ -84,9 +260,9 @@ const GENERATED_FILE = /(^|\/)(package-lock\.json|[^/]*\.lock|[^/]*\.sum)$/i;
* definition, or edited with any confidence. A test at least shows the symbol
* in use.
*
- * Only these three bands, and the sort is stable, so within a band ripgrep's
- * order survives. The alternative — scoring by line content — guesses at what
- * the caller meant from a single grep term.
+ * Only these three bands; within a band rows go by path and line (see
+ * `compareTextResults`). The alternative — scoring by line content — guesses
+ * at what the caller meant from a single grep term.
*/
export function textResultPriority(filePath: string): number {
const normalized = filePath.split(path.sep).join("/");
@@ -95,18 +271,36 @@ export function textResultPriority(filePath: string): number {
return 0;
}
-export function registerTextCommand(program: Command, executeRipgrep: RunRipgrep = runRipgrep): void {
+/**
+ * The total order `ix text` ranks by: band (`textResultPriority`), then path,
+ * then line. Paths compare by code unit, not locale, so the order is the same
+ * on every machine. Callers sort stably, so equal rows keep arrival order.
+ */
+export function compareTextResults(a: TextResult, b: TextResult): number {
+ return compareRankKeys(
+ { band: textResultPriority(a.path), path: a.path, line: a.line_start },
+ { band: textResultPriority(b.path), path: b.path, line: b.line_start },
+ );
+}
+
+export function registerTextCommand(program: Command, executeRipgrep?: RunRipgrep): void {
program
.command("text <term>")
.description("Fast lexical/text search across the codebase (uses ripgrep)")
.option("--limit <n>", "Max results", "20")
+ .option("--offset <n>", OFFSET_FLAG, "0")
.option("--path <dir>", "Restrict search to a workspace-relative directory", ".")
.option("--language <lang>", "Filter by language (python, typescript, scala, etc.)")
.option("--format <fmt>", "Output format (text|json|llm)", "text")
.option("--root <dir>", "Workspace root directory")
- .addHelpText("after", "\nExamples:\n ix text verify_token --language python\n ix text \"class.*Service\" --limit 10 --format json\n ix text TODO --path src/")
- .action(async (term: string, opts: { limit: string; path: string; format: string; language?: string; root?: string }) => {
+ .addHelpText("after", "\nExamples:\n ix text verify_token --language python\n ix text \"class.*Service\" --limit 10 --format json\n ix text TODO --path src/\n ix text TODO --offset 20")
+ .action(async (
+ term: string,
+ opts: { limit: string; offset?: string; path: string; format: string; language?: string; root?: string },
+ cmd: Command,
+ ) => {
const limit = parseInt(opts.limit, 10);
+ const offset = parseOffset(opts.offset);
const root = path.resolve(resolveWorkspaceRoot(opts.root));
const searchPath = resolveTextSearchPath(root, opts.path);
if (!isPathInsideResolvedRoot(root, searchPath)) {
@@ -147,39 +341,27 @@ export function registerTextCommand(program: Command, executeRipgrep: RunRipgrep
// other tool until the timeout and leaves the child process behind.
rgArgs.push("--", term, searchPath);
- const { stdout } = await executeRipgrep(rgArgs, window);
+ const run: RunRipgrep = executeRipgrep ?? ((args, k) => runRipgrep(args, k, root));
+ const { stdout, matched, capped } = await run(rgArgs, window);
const results: TextResult[] = [];
for (const line of stdout.split("\n")) {
if (!line.trim()) continue;
try {
const parsed = JSON.parse(line);
- if (parsed.type === "match") {
- const data = parsed.data;
- const filePath = data.path?.text ?? "";
- const absoluteFilePath = path.isAbsolute(filePath)
- ? filePath
- : path.resolve(process.cwd(), filePath);
- const lineNum = data.line_number ?? 0;
- results.push({
- path: path.relative(root, absoluteFilePath).split(path.sep).join("/"),
- line_start: lineNum,
- line_end: lineNum,
- snippet: data.lines?.text ?? "",
- engine: "ripgrep",
- score: 1.0,
- language: inferLanguage(filePath),
- });
- }
+ if (parsed.type === "match") results.push(textResultFromMatch(parsed.data, root));
} catch {
// skip non-JSON lines
}
}
- formatTextResults(
- sliceRanked(results, limit, (r) => textResultPriority(r.path)),
- opts.format,
- );
+ // Sorted here as well as in the executor, so the order never depends
+ // on which executor ran or how it delivered the rows. The window is
+ // then ranked, and every page is a plain cut of it.
+ results.sort(compareTextResults);
+ const slice = sliceRanked(results.slice(0, window), limit, undefined, offset);
+ if (slice.nextOffset !== undefined) slice.next = nextPageCommand(cmd, [term], slice.nextOffset);
+ formatTextResults(slice, opts.format, { windowFull: (matched ?? results.length) > window || capped === true });
} catch (err: any) {
if (err.code === "ENOENT") {
stderr("Error: ripgrep (rg) is not installed. Install it: https://github.com/BurntSushi/ripgrep#installation");
@@ -187,7 +369,7 @@ export function registerTextCommand(program: Command, executeRipgrep: RunRipgrep
}
// rg exit 1 = no matches; render as empty result set
if (err.code === 1 || err.status === 1) {
- formatTextResults(sliceRanked([] as TextResult[], limit), opts.format);
+ formatTextResults(sliceRanked([] as TextResult[], limit, undefined, offset), opts.format);
return;
}
// rg exit 2 = usage/regex error (e.g. literal "\n" in regex,
diff --git a/ix-cli/src/cli/format.ts b/ix-cli/src/cli/format.ts
index 879dc885..80e5917c 100644
--- a/ix-cli/src/cli/format.ts
+++ b/ix-cli/src/cli/format.ts
@@ -198,11 +198,20 @@ export function expandCapped(result: { nodes?: unknown[]; edges?: unknown[]; tru
|| (result.nodes?.length ?? 0) >= EXPAND_MAX_EDGES;
}
+/**
+ * Which rows a page printed: `51-100` past the first page, the bare count on
+ * it. A page that printed nothing is `0` — with `--limit 0 --offset 50` the
+ * range form would read `51-50`.
+ */
+function pageRange(slice: Slice<unknown>): string {
+ const offset = slice.offset ?? 0;
+ return offset > 0 && slice.shown > 0 ? `${offset + 1}-${offset + slice.shown}` : `${slice.shown}`;
+}
+
/** What was left out, and the one flag that gets it. */
export function truncationHint(slice: Slice<unknown>, relation: string): string {
- const offset = slice.offset ?? 0;
if (slice.capped) return cappedHint(slice, relation);
- const range = offset > 0 ? `${offset + 1}-${offset + slice.shown}` : `${slice.shown}`;
+ const range = pageRange(slice);
if (slice.nextOffset === undefined) return `${slice.total} ${relation}; showing ${range}. Raise --limit to see them.`;
const next = slice.next ?? `--offset ${slice.nextOffset}`;
return `${slice.total} ${relation}; showing ${range}. next: ${next}`;
@@ -216,11 +225,10 @@ function cappedHint(slice: Slice<unknown>, relation: string): string {
const offset = slice.offset ?? 0;
const head = `At least ${slice.total} ${relation} (the server returns at most ${EXPAND_MAX_EDGES} edges)`;
if (slice.nextOffset !== undefined) {
- const range = offset > 0 ? `${offset + 1}-${offset + slice.shown}` : `${slice.shown}`;
- return `${head}; showing ${range}. next: ${slice.next ?? `--offset ${slice.nextOffset}`}`;
+ return `${head}; showing ${pageRange(slice)}. next: ${slice.next ?? `--offset ${slice.nextOffset}`}`;
}
const where = slice.shown > 0
- ? `showing ${offset > 0 ? `${offset + 1}-${offset + slice.shown}` : slice.shown}, the end of what it returned`
+ ? `showing ${pageRange(slice)}, the end of what it returned`
: offset > 0 ? `--offset ${offset} is beyond the rows it returned` : "showing 0";
return `${head}; ${where}. Narrow the target with --kind or --path to see the rest.`;
}
@@ -661,22 +669,76 @@ export interface TextResult {
symbol_hint?: string;
}
+/** How `ix text` paged: whether the scan window filled before ripgrep ran out of matches. */
+export interface TextPaging {
+ /**
+ * More matches existed than the scan window kept. Pages are cut from that
+ * window, so the page that reaches its end has no next page even though the
+ * search does — the caller has to narrow the search instead.
+ */
+ windowFull?: boolean;
+}
+
+/**
+ * Whether a text page starts past the last match. Only when the scan window
+ * held every match: a full window is not the end of the search, so an offset
+ * beyond it is not past the end of anything the caller can know.
+ */
+function isTextPastEnd(slice: Slice<TextResult>, paging?: TextPaging): boolean {
+ return !paging?.windowFull && isPastEnd(slice);
+}
+
+/**
+ * What a text page left out, and how to get it: the next page while the scan
+ * window has rows left, a narrower search once the page reached the window's
+ * end. Undefined when nothing was left out.
+ */
+function textTruncationMessage(slice: Slice<TextResult>, paging?: TextPaging): string | undefined {
+ if (isTextPastEnd(slice, paging)) return undefined;
+ const range = pageRange(slice);
+ if (slice.nextOffset !== undefined) {
+ return `${slice.total} matches scanned; showing ${range}. next: ${slice.next ?? `--offset ${slice.nextOffset}`}`;
+ }
+ if (paging?.windowFull && slice.shown > 0) {
+ return `${slice.total} matches scanned; showing ${range}; the scan stops at ${slice.total}. Narrow with --path or --language, or raise --limit.`;
+ }
+ if (paging?.windowFull && (slice.offset ?? 0) >= slice.total) {
+ return `${slice.total} matches scanned; --offset ${slice.offset ?? 0} is past the scan window, which stops at ${slice.total}. Narrow with --path or --language, or raise --limit.`;
+ }
+ if (slice.truncated) {
+ return `${slice.total} matches scanned; showing ${range}. Narrow with --path or --language, or raise --limit.`;
+ }
+ return undefined;
+}
+
/**
* Render lexical search hits as llm `match` records (one per line).
*
* No `lang=`: the extension on `path` already says it, and on a 20-row answer
* the field was ~80 tokens of nothing. `--format json` keeps `language`.
+ *
+ * `scanned` is the size of the ranked scan window, not the number of matches
+ * in the repo; pages (`--offset`) are cut from that window.
*/
-export function renderTextResultsLlm(slice: Slice<TextResult>): string[] {
+export function renderTextResultsLlm(slice: Slice<TextResult>, paging?: TextPaging): string[] {
+ const message = textTruncationMessage(slice, paging);
const lines = [llmLine("text", [
+ ["offset", slice.offset ? slice.offset : undefined],
["shown", slice.shown],
["scanned", slice.total],
- ["truncated", slice.truncated || undefined],
+ ["truncated", slice.truncated || message !== undefined || undefined],
])];
- if (slice.truncated) {
+ if (message !== undefined) {
lines.push(llmLine("diagnostic", [
["code", "results_truncated"],
- ["message", `${slice.total} matches scanned; showing the ${slice.shown} highest-ranked. Narrow with --path or --language, or raise --limit.`],
+ ["next_offset", slice.nextOffset],
+ ["message", message],
+ ]));
+ }
+ if (slice.rows.length === 0 && isTextPastEnd(slice, paging)) {
+ lines.push(llmLine("diagnostic", [
+ ["code", "offset_past_end"],
+ ["message", pastEndMessage(slice, "matches scanned")],
]));
}
for (const r of slice.rows) {
@@ -690,9 +752,12 @@ export function renderTextResultsLlm(slice: Slice<TextResult>): string[] {
return lines;
}
-export function formatTextResults(slice: Slice<TextResult>, format: string): void {
+export function formatTextResults(slice: Slice<TextResult>, format: string, paging?: TextPaging): void {
const results = slice.rows;
if (format === "json") {
+ // A raw array on purpose: the plugin runners parse it as one. Paging state
+ // stays out of it; a JSON caller's next offset is its offset plus the
+ // array's length.
const compact = results.map(r => ({
...r,
path: relativePath(r.path) ?? r.path,
@@ -701,11 +766,13 @@ export function formatTextResults(slice: Slice<TextResult>, format: string): voi
return;
}
if (format === "llm") {
- for (const line of renderTextResultsLlm(slice)) console.log(line);
+ for (const line of renderTextResultsLlm(slice, paging)) console.log(line);
return;
}
if (results.length === 0) {
- console.log("No text matches found.");
+ console.log(isTextPastEnd(slice, paging)
+ ? pastEndMessage(slice, "matches scanned")
+ : textTruncationMessage(slice, paging) ?? "No text matches found.");
return;
}
for (const r of results) {
@@ -715,6 +782,8 @@ export function formatTextResults(slice: Slice<TextResult>, format: string): voi
` ${lang}${chalk.dim(r.path)}${chalk.cyan(":" + r.line_start)} ${sym}${r.snippet.trim()}`
);
}
+ const message = textTruncationMessage(slice, paging);
+ if (message !== undefined) console.log(chalk.dim(` ${message}`));
}
export interface LocateResult {
diff --git a/ix-cli/src/mcp/server.ts b/ix-cli/src/mcp/server.ts
index 5732d695..a47926cc 100644
--- a/ix-cli/src/mcp/server.ts
+++ b/ix-cli/src/mcp/server.ts
@@ -435,11 +435,14 @@ export function createIxMcpServer(options: CreateServerOptions = {}): McpServer
limit: z.number().int().min(1).max(100).default(20),
path: z.string().min(1).optional(),
language: z.string().min(1).optional(),
+ // Takes back next_offset= (see EDGE_TOOL_INPUT); no description, same budget.
+ offset: z.number().int().min(0).max(100000).optional(),
},
async (input) => {
const options = [`--limit=${numberArg(input, "limit")}`];
pushOption(options, "--path", input.path);
pushOption(options, "--language", input.language);
+ if (typeof input.offset === "number") options.push(`--offset=${numberArg(input, "offset")}`);
return runFormatted(runIx, "ix_text", ix("text", [stringArg(input, "pattern")], options));
},
);
@@ -610,10 +613,13 @@ export function createIxMcpServer(options: CreateServerOptions = {}): McpServer
kind: z.string().min(1).default("file"),
path: z.string().min(1).optional(),
limit: z.number().int().min(1).max(500).default(50),
+ // Takes back next_offset= (see EDGE_TOOL_INPUT); no description, same budget.
+ offset: z.number().int().min(0).max(100000).optional(),
},
async (input) => {
const options = [`--kind=${stringArg(input, "kind")}`, `--limit=${numberArg(input, "limit")}`];
pushOption(options, "--path", input.path);
+ if (typeof input.offset === "number") options.push(`--offset=${numberArg(input, "offset")}`);
return runFormatted(runIx, "ix_inventory", ix("inventory", [], options));
},
);
diff --git a/skills/ix/references/commands.md b/skills/ix/references/commands.md
index 33bd131c..cda004a6 100644
--- a/skills/ix/references/commands.md
+++ b/skills/ix/references/commands.md
@@ -189,8 +189,12 @@ ix inventory --kind function --path auth.py
- Always use `--limit` to cap result sets. Not on `ix diff --summary` or
`ix diff --full`, though: each already fixes the volume, so `--limit`
alongside either is refused rather than silently ignored.
-- Page `callers`, `callees`, `imports`, `imported-by` and `contains` with
- `--offset`, following the `next:` hint (or `next_offset=` in llm output).
+- Page `callers`, `callees`, `imports`, `imported-by`, `contains`,
+ `inventory` and `text` with `--offset`, following the `next:` hint (or
+ `next_offset=` in llm output). Keep `--limit` the same while paging. `text`
+ pages stop at its scan window; past that, narrow with `--path` or
+ `--language`. The server returns at most 1000 edges and 5000 `inventory`
+ rows; past those caps the output stays truncated and says to narrow.
- Use `--path` or `--language` to restrict text searches.
- Reuse exact entity IDs from previous JSON results when chaining commands.
- Decompose large questions into multiple targeted calls.
diff --git a/skills/ix/references/flags.md b/skills/ix/references/flags.md
index fef27c00..28db9096 100644
--- a/skills/ix/references/flags.md
+++ b/skills/ix/references/flags.md
@@ -510,6 +510,7 @@ List entities by kind with optional path scoping.
| `--kind` | `<kind>` | — | Entity kind to list (class, method, function, file, module, etc.) |
| `--path` | `<path>` | — | Filter by source file path substring |
| `--limit` | `<n>` | `50` | Max results |
+| `--offset` | `<n>` | `0` | Skip the first N results (use the next: hint to page) |
| `--format` | `text\|json\|llm` | `text` | Output format — see [output-formats.md](output-formats.md) |
| `--pretty` | — | off | Indent JSON output; the default only when stdout is a terminal |
| `--quiet` | — | off | Drop headers, section titles and advisory hints |
@@ -784,6 +785,7 @@ Fast lexical/text search across the codebase (uses ripgrep).
| Flag | Value | Default | Effect |
|---|---|---|---|
| `--limit` | `<n>` | `20` | Max results |
+| `--offset` | `<n>` | `0` | Skip the first N results (use the next: hint to page) |
| `--path` | `<dir>` | `.` | Restrict search to a workspace-relative directory |
| `--language` | `<lang>` | — | Filter by language (python, typescript, scala, etc.) |
| `--format` | `text\|json\|llm` | `text` | Output format — see [output-formats.md](output-formats.md) |