Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(code-graph): attribute calls in anonymous top-level code to the file's __module__ chunk (#5001)
garrytan46d76035d121
diff --git a/docs/architecture/key-files/core-services-1.md b/docs/architecture/key-files/core-services-1.mdindex 600da3e14e..b5cd8e3461 100644--- a/docs/architecture/key-files/core-services-1.md+++ b/docs/architecture/key-files/core-services-1.md@@ -33,9 +33,9 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/chronicle/{contract,eligibility,config,reasons,ledger,ledger-stats,execute,publish,event-identity,invite-projection,backfill,campaign,job,upgrade-notice,extract-events,proposal-diagnostics,ontology,narrative}.ts` + `src/core/cycle/chronicle.ts` + `src/eval/chronicle/harness.ts` + `src/commands/eval-chronicle.ts` — Life Chronicle: the temporal spine. `contract.ts` holds the ledger columns, the extractor version (part of the ledger key: a bump makes extracted content new to discovery and backfill), the phase bounds and the phase result details. `reasons.ts` is the one reason table (`CHRONICLE_REASONS`: stage, meaning, stored fix Action with argv/preview_argv/consent/actor, never a stored `next`; `auto_chronicle_off` has no fix because it is off by choice) and builds the `chronicle_backstop` receipt field (`{pending:'next_cycle', daily_remaining}` or `{skipped, stage, why, fix}`, omitted for pages that are not chronicle-shaped); `docs/guides/life-chronicle.md` renders its table from it. `eligibility.isChronicleEligible` is the one eligibility function (meeting/conversation/calendar-event + directory rescue; diary and event pages NEVER eligible — privacy + anti-loop); with a time policy it also applies the invite end (`not_yet_happened`), the recency window on the page's own date (an authored `effective_date`, else frontmatter `date`/`start`; a `fallback` date counts as undated) and the settle window. `config.ts` owns the switch and knobs: `autoChronicleSetting` reads `auto_chronicle` as on when unset, on/off for true/false words, and `invalid` (off) for any other word; `chronicleSettings` reads `chronicle.job_budget_usd`/`auto_daily_limit`/`auto_recent_days`/`auto_settle_seconds`/`judge_max_tokens`/`max_events_per_page` (#5329, default 25) with ranges (`CHRONICLE_NUMERIC_KEYS`, enforced by `config set`; a malformed stored row falls back and is listed in `invalid`) and `chronicle.activated_at` (stamped by the first write decision or phase run; earlier revisions are history); `autoChronicleNeedsAcknowledgement` is true while on and `chronicle.default_on_acknowledged` is unset (stamped by `config set auto_chronicle`). The opt-out is `gbrain config set auto_chronicle false` everywhere. `ledger.ts` owns `chronicle_page_state` (one row per source/page/content hash/extractor version: state, reason, trigger, deciding writer and request, `no_extract`, attempts, `next_attempt_at`, cost/unpriced, event slugs and hashes): `recordChronicleDecision` runs inside every coordinated publication that imports a page (from `queuePublicationEffects`), applies the shared writer-confinement predicate, `--no-extract`, the opt-out and eligibility, and puts `chronicle_backstop` on the outcome through `reasons.ts`; `reserveChronicleSlot` is the rolling 24 h reservation under a transaction advisory lock. `ledger-stats.ts` rolls up the ledger and reservations for doctor and the advisor. `upgrade-notice.ts` is the one-shot post-upgrade notice (`chronicle.default_on_notice_shown`). The global `chronicle` cycle phase (`cycle/chronicle.ts`) supersedes rows whose content is no longer live, discovers pages changed since activation with no row (unmanaged brains decide there; managed ones record `no_write_decision`), and executes settled rows round-robin across sources (≤50 judged items and a wall-time bound per run; automatic rows only while `auto_chronicle` is on, backfill rows always); no chat provider → no calls; the extract_atoms cost gate refuses an explicit cap on an unpriced model. `execute.ts` runs one row: live-content check, time rules, the reservation (automatic rows; attempts count), the judge in a `chronicle:<trigger>` BudgetTracker scope with the per-page cap and an explicit post-call cap check, failure classes (chat_error/budget → failed with backoff, no provider/no_pricing → failed until config changes, refused → skipped, only a parsed `[]` is `no_events`). `invite-projection.ts` stands in for the judge on an ended calendar invite (slug under `calendar/`/`cal/` or type `calendar-event`, with a title, `start` and `end`): one `Scheduled: <title>` event (`kind: meeting`, `captured_via: life-chronicle:invite`, not proof of attendance) with no chat call, no daily reservation and zero cost, through the same publication and ledger path. `publish.ts` publishes one generation against the pinned depth snapshot: each event write re-validates the depth revision, content hash, source incarnation, privacy and the deciding writer's grant (events also read as private when their `event.depth` origin is private by field or declared lineage, or missing, `search/private-visibility.ts`) (`managed_maintenance_chronicle_event`/`_retire` intents on managed brains, one transaction per event otherwise), stamps the depth's visibility, never overwrites an operator-edited event (live hash not in the ledger) or resurrects an operator-deleted one, and retires the previous generation's owned events it did not reproduce (`retired_by: life-chronicle`; restored when produced again) only when the caller passes `replaces`: `execute.ts` and `runChronicleExtract` pass it for a complete judged generation (`extract-events.replacesPreviousGeneration`: at least one event, no proposal dropped by the date screen or the per-page cap, and a body no longer than the JUDGE_BODY_CHARS the judge reads; a segmented page's truth lead is read one character past it to tell) and for an ineligible revision's empty generation, so an empty or partial generation publishes what it has, retires nothing and still records its ledger outcome (#7013). `event-identity.ts` assigns final slugs before publication: events sharing a base slug `life/events/<day>-<hash8>` (who, what, depth) are told apart by instant, `where` and `kind`; one keeps the base slug and the rest get `-<hash6>` (identical copies `-2`…); assignment reads the pages already at the candidate slugs, so it is order-independent, keeps a page's existing slug across corrections and re-extractions, and a page extracted before disambiguation keeps its base slug. `extract-events.runChronicleExtract` is the direct-caller body over the same snapshot and publication path: deterministic when/who, injectable judge (default = chat gateway; output cap 4000 tokens by default, operator override `chronicle.judge_max_tokens`), an ALL-or-nothing parse barrier (`isValidProposal` requires a real parseable date; a refused batch carries `proposal-diagnostics.diagnoseMalformedProposals`: a fixed code, the schema field and JSON type found, and counts, never a value, from at most 50 proposals, 50 `who` entries and 256 characters per string, on the direct result's `malformed`, the row outcome and the phase's warning line), then `screenChronicleProposals` drops, never writes, each proposal without a day (`date_imprecise`: `when` is a year or month) or dated after the page's own day (`future_dated`: `chronicleEventCutoff` is the latest of the page's own date, a meeting's `end` and a conversation's last parsed message, as a day in `chronicle.tz`, capped at today), then keeps at most `chronicle.max_events_per_page` of the survivors in the judge's order (#5329, the rest `over_cap`, no model call to count); drops are counted in `events_dropped` (phase details, row outcome, direct result) and a page whose every proposal was dropped records the drop reason on its extracted ledger row instead of `no_events`; content-addressed `life/events/` pages + a `timeline_entries` projection. An unusable judge response is never recorded as `no_events`: truncation, unparseable output, a provider error and a refusal surface as `judge_truncated`/`judge_parse_failed`/`judge_chat_error`/`judge_refused`. `backfill.ts` (`chronicle_backfill`) queues `trigger='backfill'` ledger rows, exempt from the daily limit and recency (unless `--recent`), across types and rescue prefixes under one global `--limit`; `--since` is the update date, `--dated-since` the page date; a dry run reports `estimated_usd` and skip reasons; a live run needs `--yes`; content already extracted or queued is never re-queued, even after `gbrain jobs prune`. `--max-usd` (#6199, `campaign.ts`) is a queue-time bound: a page queues only while queued × (per-attempt cap + `chronicleCallCeilingUsd`) × max attempts fits, rows are stamped (`campaign_id`, `attempt_cap_usd`, `max_attempts`, `pricing_policy`, `campaign_max_usd`; migration `chronicle_campaign_stamps`) and `queueCampaignRow` never takes a row another backfill queued or an executor holds; an unpriced model under the cap refuses with `no_pricing`. Executors obey the stamp: `claimChronicleRow` consumes a campaign row's attempt with the lease and refuses it past `max_attempts`, the tracker uses the stamped cap, `finish` sums spend per attempt (`cost_attempts`, so a replay adds nothing) and applies the outcome only while its attempt is the latest (a stale completion is fenced), and `failExhaustedCampaignRows` fails a crashed row whose attempts are spent as `campaign_exhausted`; any other ledger write clears the stamp. `job.ts` drains `chronicle_extract` jobs from older releases through the same executor. `ontology.ts` carries the deterministic pieces of the bi-temporal per-entity ontology that RIDES THE `facts` TABLE (migration v122 adds `dimension`/`value`/`value_hash`/`dim_status`): `valueHash` (normalized, timestamp-free; the v188 dedup index adds the stint's `valid_from`, so a crash-retry conflicts while a same-provenance return to an earlier value records a new interval), `normalizeDimension` (seed alias lexicon), `isNovelDimension` (novel → quarantined, excluded from resolution/context until confirmed). The engine methods (`mergeOntologyFact` — corroborate on same value (noop when the same provenance already observed it during the current stint), forward-supersede via `valid_until`+`superseded_by` on a new value, backdated conflicts kept + flagged; `getOntology` with `--asof` valid-time travel; `discoverOntologyDimensions`; `findOntologyConflicts` — currently-open rows only) live in BOTH engines; both engines are on the R8 `valid_until` write allow-list (engine-layer, `dimension IS NOT NULL` rows only). Chronicle reads (`getTimelineForDate`/`getSince`/`getLastSeen`/`getOnThisDay`) JOIN the depth page (`deleted_at IS NULL`), hide soft-deleted event projections at READ time, and order by event `effective_date` for intra-day sequence. Ops: `chronicle_day`/`chronicle_since`/`chronicle_last_seen`/`chronicle_on_this_day`/`ontology_*`/`volunteer_chronicle` (agent orientation via `src/core/context/chronicle-context.ts`)/`chronicle_backfill` (admin, localOnly). Diary privacy: fail-closed for `ctx.remote !== false` callers — diary-sourced ontology + conflict values are redacted, the four timeline read ops (`chronicle_day`/`chronicle_since`/`chronicle_on_this_day` + `volunteer_chronicle`'s `recent_timeline`) drop rows whose depth page, event page, or `source` provenance lives under `life/diary/`, and `chronicle_last_seen` answers the never-seen shape (`last_date`/`last_event_slug`/`days_ago` all null) when the latest sighting is diary-sourced or a diary page is queried as the entity. Remote callers read only `visibility = 'world'` observations, like recall: `getOntology`/`findOntologyConflicts` take a `visibility` filter applied before resolution and the conflict HAVING. `ontology_propose` commits through `coordinatedDatabaseWrite` on a managed brain. Page visibility rides the same policy: `ontology_get`/`ontology_conflicts`/`volunteer_chronicle`'s ontologies resolve through `readPolicyOpts`, and both engines' `getOntology`/`findOntologyConflicts` apply `privateProvenanceFilterFragment` (an observation whose provenance page, looked up in the fact's own source, is `visibility: private` is dropped BEFORE per-dimension resolution and before the conflict HAVING, so an untrusted caller resolves the newest value they may see rather than a private value or a hole). The chronicle reads' event-page LEFT JOIN carries the caller's source scope in BOTH engines, so a scoped read can't surface another source's event pages through the join. Search: `applyChronicleTypeBoost` in `search/hybrid.ts` (bounded [1.0,1.25], fires only inside the `recency !== 'off'` post-fusion branch → non-temporal search bit-for-bit unchanged). Advisor collector `collect-chronicle.ts` (conflicts; coverage gap with a scoped `chronicle-backfill --since <date> --limit 50 --dry-run`; `auto_chronicle_default_on` ask_user until acknowledged; `auto_chronicle_invalid`; `chronicle_chat_unavailable`; `chronicle_extraction_failing`; `chronicle_daily_limit`); doctor `chronicle_projection_health` and `auto_chronicle` (`checks/auto-chronicle.ts`; BRAIN category). Eval: `gbrain eval chronicle` — deterministic, own in-memory PGLite, 6 gold tasks (day order, last-seen, supersession, asof, conflict, isolation), exit 0 iff 6/6. Tests: `test/chronicle-*.test.ts` (automatic path: `chronicle-auto-{decision,phase,unmanaged}`, `chronicle-date-quality`, `chronicle-event-privacy`, `chronicle-event-identity`, `chronicle-invite-projection`, `chronicle-backfill`, `chronicle-backfill-max-usd`, `chronicle-reasons-5876`, `chronicle-max-events-5329`; Postgres arms in `test/e2e/chronicle-*-postgres.test.ts`), `test/eval-chronicle.test.ts`, `test/auto-chronicle-surfaces-5876.test.ts`.-- `src/core/chunkers/` — markdown chunking (`recursive.ts`) and code chunking. `code.ts` is a tree-sitter-based semantic chunker for 30 languages (plus SQL via DerekStride/tree-sitter-sql) with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` is folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases; `GRAMMAR_REVISIONS` (per-language chunker revisions: `lua: 1` after the tree-sitter-grammars v0.3.0 swap that ended the `gbrain serve` Lua-fence hang; `typescript`/`tsx`/`javascript`/`python`/`ruby: 1` for the #5001 `__module__` chunks) folds into that language's hash only, and `chunkerStamp()` (`8;lua=1;typescript=1;…`) is the `sources.chunker_version` gate value every sync path reads and writes, so a per-language change walks each source once and re-imports only that language's files while the cost gate (`grammarOnlyDrift`, `grammar_drift`) prices those files alone. Top-level statements that contain a call (`describe(...)`, `app.use(...)`, `print(f())`, Ruby `RSpec.describe ... do`) are chunked through `code-module-chunks.ts` (below) so their call edges have a chunk to hang on. Grammar provenance and the vendoring scripts: `src/assets/wasm/README.md`; pinned by `test/chunkers/code-lua.test.ts`. `extractSymbolName` has an inline SQL branch (`extractSqlSymbolName`) diving through DerekStride's `statement` wrapper into the inner DDL child (`create_table`/`create_function`/`create_view`/`create_index`/`create_procedure`/`create_type`/`create_schema`/`create_database`/`create_trigger`/`alter_table`/`alter_view`) and extracting the target identifier via the `name` field with identifier-shaped fallback; DML kinds (`select`/`insert`/`update`/`delete`/`merge`/`with`) deliberately return null so chunks emit unnamed (code-def is a DDL signal). `normalizeSymbolType` has parallel SQL branches mapping `create_table → 'table'`, `create_view → 'view'`, etc. `DEF_TYPES` (owned by `src/core/chunkers/def-types.ts`, re-exported by `src/commands/code-def.ts` — see that entry) carries the SQL kinds (`'table' | 'view' | 'index' | 'procedure' | 'schema' | 'database' | 'trigger'`) so the new chunks surface in `gbrain code-def <name>` queries.+- `src/core/chunkers/` — markdown chunking (`recursive.ts`) and code chunking. `code.ts` is a tree-sitter-based semantic chunker for 30 languages (plus SQL via DerekStride/tree-sitter-sql) with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` is folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases; `GRAMMAR_REVISIONS` (per-language chunker revisions: `lua: 1` after the tree-sitter-grammars v0.3.0 swap that ended the `gbrain serve` Lua-fence hang; `typescript`/`tsx`/`javascript`/`python`/`ruby: 2` for the #5001 `__module__` chunks) folds into that language's hash only, and `chunkerStamp()` (`8;lua=1;typescript=2;…`) is the `sources.chunker_version` gate value every sync path reads and writes, so a per-language change walks each source once and re-imports only that language's files while the cost gate (`grammarOnlyDrift`, `grammar_drift`) prices those files alone. Top-level statements that contain a call (`describe(...)`, `app.use(...)`, `print(f())`, Ruby `RSpec.describe ... do`) are chunked through `code-module-chunks.ts` (below) so their call edges have a chunk to hang on. Grammar provenance and the vendoring scripts: `src/assets/wasm/README.md`; pinned by `test/chunkers/code-lua.test.ts`. `extractSymbolName` has an inline SQL branch (`extractSqlSymbolName`) diving through DerekStride's `statement` wrapper into the inner DDL child (`create_table`/`create_function`/`create_view`/`create_index`/`create_procedure`/`create_type`/`create_schema`/`create_database`/`create_trigger`/`alter_table`/`alter_view`) and extracting the target identifier via the `name` field with identifier-shaped fallback; DML kinds (`select`/`insert`/`update`/`delete`/`merge`/`with`) deliberately return null so chunks emit unnamed (code-def is a DDL signal). `normalizeSymbolType` has parallel SQL branches mapping `create_table → 'table'`, `create_view → 'view'`, etc. `DEF_TYPES` (owned by `src/core/chunkers/def-types.ts`, re-exported by `src/commands/code-def.ts` — see that entry) carries the SQL kinds (`'table' | 'view' | 'index' | 'procedure' | 'schema' | 'database' | 'trigger'`) so the new chunks surface in `gbrain code-def <name>` queries.-- `src/core/chunkers/code-module-chunks.ts` — #5001 module-level statement chunks. `moduleStatementSymbol(node, language)` names a top-level node that is a statement containing a call (JS/TS/TSX and Python `expression_statement`, Ruby `call`) as `{ symbolName: '__module__', symbolType: 'module' }`; `code.ts` adds those nodes to its semantic set and emits them through the ordinary chunk path, with `moduleChunkQualifiedName` giving the edge identity `<file-basename>::__module__` (`vat-review.test::__module__`; a Ruby `module Foo` keeps its own name). `isModuleChunk` lets `mergeSmallSiblings` fold adjacent small statements (they are not definitions), and `mergedRunSymbol` keeps the `__module__` identity on a merged run that holds one, so `code-callers helper` lists the test body or route registration that calls `helper` instead of dropping the edge. A statement without a call (`x = 1`, a docstring) and every definition-only file are unchanged. The edge rows themselves come from `code-chunks.ts`'s `mapCodeEdges`, the one call-site → `code_edges_symbol` mapping shared by `importCodeFile` and the prepared apply. Pinned by `test/chunkers/code-module-level-calls.test.ts` (chunk inventory per language, merge identity, oversized split, PGLite import through `mapCodeEdges`, the per-language revision stamp).+- `src/core/chunkers/code-module-chunks.ts` — #5001 module-level statement chunks. `moduleStatementSymbol(node, language, named)` names a top-level node that contains a call and is not a named definition (JS/TS/TSX, Python, Ruby: a statement, a top-level `if`/`for`/`try`, Python's `if __name__ == "__main__":`, an anonymous `export default …`) as `{ symbolName: '__module__', symbolType: 'module' }`; `code.ts` adds those nodes to its semantic set and emits them through the ordinary chunk path, with `moduleChunkQualifiedName` giving the edge identity `<file-basename>::__module__` (`vat-review.test::__module__`; a Ruby `module Foo` keeps its own name). `isModuleChunk` lets `mergeSmallSiblings` fold adjacent small statements (they are not definitions), and `mergedRunSymbol` keeps the `__module__` identity on a merged run that holds one or a `holdsCall` declaration (`const app = createApp();`), so `code-callers helper` lists the test body or route registration that calls `helper` instead of dropping the edge. A statement without a call (`x = 1`, a docstring) and every definition-only file are unchanged. The edge rows themselves come from `code-chunks.ts`'s `mapCodeEdges`, the one call-site → `code_edges_symbol` mapping shared by `importCodeFile` and the prepared apply. Pinned by `test/chunkers/code-module-level-calls.test.ts` (chunk inventory per language, merge identity, oversized split, PGLite import through `mapCodeEdges`, the per-language revision stamp) and `test/chunkers/code-module-anonymous-5001.test.ts`.- `src/core/chunkers/def-types.ts` — `DEF_TYPES`: the ONE list of definition-shaped normalized (post-`normalizeSymbolType`) symbol types, shared by `gbrain code-def`'s lookup allowlist (`src/commands/code-def.ts` re-exports it) and the code chunker's small-sibling merge guard so the two can't drift — a symbol type code-def can resolve never has its `symbol_name` erased by merging a small definition into a neighbor. Includes the fallthrough forms `normalizeSymbolType` emits verbatim for tree-sitter node types it doesn't canonicalize (`'method declaration'`, `'struct specifier'`, `'record declaration'`, …) — most of an OO codebase's symbols arrive in those forms. Pinned by `test/chunkers/code-merge-defs.test.ts`.diff --git a/src/core/chunkers/code-module-chunks.ts b/src/core/chunkers/code-module-chunks.tsindex e78852d140..68f91fa494 100644--- a/src/core/chunkers/code-module-chunks.ts+++ b/src/core/chunkers/code-module-chunks.ts@@ -6,9 +6,11 @@* `print(helper())` or `RSpec.describe ... do` block is an expression statement, so no chunk covered it and every call* inside a test body or route registration was lost from `code-callers`.*- * A top-level statement that contains a call becomes a chunk with `symbolType: 'module'`, `symbolName: '__module__'`- * and the file-qualified identity `<file-basename>::__module__`, for JS/TS/TSX, Python and Ruby (the languages whose- * call edges get receiver resolution or RSpec-style top-level blocks). A run of small adjacent statements may merge+ * A top-level node that contains a call but names no definition becomes a chunk with `symbolType: 'module'`,+ * `symbolName: '__module__'` and the file-qualified identity `<file-basename>::__module__`, for JS/TS/TSX, Python and+ * Ruby (the languages whose call edges get receiver resolution or RSpec-style top-level blocks). That covers statements+ * (`describe(...)`, a top-level `if`/`for`/`try`, Python's `if __name__ == "__main__": main()`) and anonymous+ * definitions (`export default () => …`, `export default { fetch() { … } }`, `const { a } = useThing()`). A run of small adjacent statements may merge* (`mergeSmallSiblings`); the merged chunk keeps the `__module__` identity so its edges survive. Definition-only files* are unchanged. The chunk-inventory change is gated per language through `GRAMMAR_REVISIONS` (code.ts).*/@@ -17,22 +19,18 @@ import type { CodeChunk, SupportedCodeLanguage } from './code.ts';export const MODULE_SYMBOL_NAME = '__module__';export const MODULE_SYMBOL_TYPE = 'module';-interface ModuleStatementConfig {- /** Top-level node types that are statements rather than definitions. */- statementTypes: ReadonlySet<string>;- /** Call node types (the edge extractor's `CALL_CONFIG` for the language); a statement without one emits nothing. */- callTypes: ReadonlySet<string>;-}--const JS_CONFIG: ModuleStatementConfig = { statementTypes: new Set(['expression_statement']), callTypes: new Set(['call_expression']) };+const JS_CALL_TYPES: ReadonlySet<string> = new Set(['call_expression']);-const MODULE_STATEMENTS: Partial<Record<SupportedCodeLanguage, ModuleStatementConfig>> = {- typescript: JS_CONFIG,- tsx: JS_CONFIG,- javascript: JS_CONFIG,- python: { statementTypes: new Set(['expression_statement']), callTypes: new Set(['call']) },- // tree-sitter-ruby has no statement wrapper: a top-level call IS the node (`require 'x'`, `RSpec.describe Thing do … end`).- ruby: { statementTypes: new Set(['call', 'method_call']), callTypes: new Set(['call', 'method_call']) },+/**+ * Call node types per language (the edge extractor's `CALL_CONFIG`); a node without one emits nothing. tree-sitter-ruby+ * has no statement wrapper: a top-level call IS the node (`require 'x'`, `RSpec.describe Thing do … end`).+ */+const MODULE_CALL_TYPES: Partial<Record<SupportedCodeLanguage, ReadonlySet<string>>> = {+ typescript: JS_CALL_TYPES,+ tsx: JS_CALL_TYPES,+ javascript: JS_CALL_TYPES,+ python: new Set(['call']),+ ruby: new Set(['call', 'method_call']),};const MODULE_SYMBOL = { symbolName: MODULE_SYMBOL_NAME, symbolType: MODULE_SYMBOL_TYPE } as const;@@ -41,16 +39,16 @@ const isModuleSymbol = (symbol: { symbolName: string | null; symbolType: stringsymbol.symbolType === MODULE_SYMBOL_TYPE && symbol.symbolName === MODULE_SYMBOL_NAME;/**- * The `__module__` symbol for a top-level node that is a statement containing a call in `language`; null for every- * definition node and for statements with no call (`x = 1`, a docstring), which stay unchunked as before.+ * The `__module__` symbol for a top-level node that contains a call in `language` and is not a `named` definition;+ * null for a named definition and for nodes with no call (`x = 1`, a docstring, an import), which stay unchunked.*/-export function moduleStatementSymbol(node: any, language: SupportedCodeLanguage): typeof MODULE_SYMBOL | null {- const config = MODULE_STATEMENTS[language];- if (!config || !config.statementTypes.has(node.type)) return null;+export function moduleStatementSymbol(node: any, language: SupportedCodeLanguage, named: boolean): typeof MODULE_SYMBOL | null {+ const callTypes = MODULE_CALL_TYPES[language];+ if (!callTypes || named) return null;const stack: any[] = [node];while (stack.length) {const current = stack.pop();- if (config.callTypes.has(current.type)) return MODULE_SYMBOL;+ if (callTypes.has(current.type)) return MODULE_SYMBOL;for (const child of current.namedChildren) stack.push(child);}return null;@@ -73,11 +71,10 @@ export function isModuleChunk(chunk: CodeChunk): boolean {/*** Identity of a merged small-sibling run. A run never holds a named definition (#4511), so one that holds a module- * statement is module-level code: it keeps the `__module__` identity and its call edges. Any other run is the- * anonymous `merged` chunk it always was.+ * statement or a call (`const app = express();`, flagged `holdsCall`) is module-level code: it keeps the+ * `__module__` identity and its call edges. Any other run is the anonymous `merged` chunk it always was.*/export function mergedRunSymbol(group: CodeChunk[]): Pick<CodeChunk['metadata'], 'symbolName' | 'symbolType' | 'symbolNameQualified'> {- const module = group.find(isModuleChunk);- if (!module) return { symbolName: null, symbolType: 'merged' };- return { ...MODULE_SYMBOL, symbolNameQualified: module.metadata.symbolNameQualified };+ if (!group.some(c => isModuleChunk(c) || c.metadata.holdsCall)) return { symbolName: null, symbolType: 'merged' };+ return { ...MODULE_SYMBOL, symbolNameQualified: moduleChunkQualifiedName({ filePath: group[0]!.metadata.filePath, ...MODULE_SYMBOL }) };}diff --git a/src/core/chunkers/code.ts b/src/core/chunkers/code.tsindex d35de65e81..1eeb635460 100644--- a/src/core/chunkers/code.ts+++ b/src/core/chunkers/code.ts@@ -150,8 +150,9 @@ export const CHUNKER_VERSION = 8;* lua 1: tree-sitter-grammars v0.3.0 replaced a build whose scanner ran every* parse after the first on uninitialized state; Lua defs now become chunks.* typescript/tsx/javascript/python/ruby 1 (#5001): top-level call statements become `__module__` chunks (code-module-chunks.ts).+ * typescript/tsx/javascript/python/ruby 2 (#5001): so do top-level `if`/`for`/`try` blocks and anonymous definitions holding a call.*/-export const GRAMMAR_REVISIONS: Partial<Record<SupportedCodeLanguage, number>> = { lua: 1, typescript: 1, tsx: 1, javascript: 1, python: 1, ruby: 1 };+export const GRAMMAR_REVISIONS: Partial<Record<SupportedCodeLanguage, number>> = { lua: 1, typescript: 2, tsx: 2, javascript: 2, python: 2, ruby: 2 };/** The `sources.chunker_version` gate value: a change forces the next sync to walk every file (hash compare, no re-import of unchanged hashes). */export const chunkerStamp = (): string => [String(CHUNKER_VERSION), ...Object.entries(GRAMMAR_REVISIONS).map(([lang, rev]) => `${lang}=${rev}`)].join(';');@@ -199,6 +200,8 @@ export interface CodeChunkMetadata {symbolNameQualified?: string | null;/** N13-1: a const/let/var whose value is a function — merge-protected (def-types.ts). */definesFunction?: boolean;+ /** #5001: a named top-level node that contains a call; a merged run holding one keeps the `__module__` identity. */+ holdsCall?: boolean;}export interface CodeChunk {@@ -803,7 +806,7 @@ async function chunkParsedLanguage(const root = (tree as any).rootNode;const topLevelTypes = TOP_LEVEL_TYPES[language]!;- const semanticNodes = root.namedChildren.filter((n: any) => topLevelTypes.has(n.type) || moduleStatementSymbol(n, language) !== null);+ const semanticNodes = root.namedChildren.filter((n: any) => topLevelTypes.has(n.type) || moduleStatementSymbol(n, language, false) !== null);if (semanticNodes.length === 0) {return { chunks: fallbackChunks(source, filePath, language, opts), edges: [] };@@ -834,8 +837,10 @@ async function chunkParsedLanguage(// declaration; top-level chunk still uses the outer node's range// so the header shows the `export` keyword for completeness.const nestableNode = findNestableParent(node, nestedConfig);- const moduleSymbol = moduleStatementSymbol(node, language); // #5001: a top-level statement containing a call- const symbolName = moduleSymbol?.symbolName ?? extractSymbolName(nestableNode ?? node);+ const definedName = topLevelTypes.has(node.type) ? extractSymbolName(nestableNode ?? node) : null;+ // #5001: a top-level statement or anonymous definition containing a call is module-level code+ const moduleSymbol = moduleStatementSymbol(node, language, definedName !== null);+ const symbolName = moduleSymbol?.symbolName ?? definedName;// For SQL `statement` wrappers, the meaningful type lives on the inner// child. extractSymbolName already dives in for the name; mirror that// here so chunk headers say "table users" not "statement users".@@ -846,7 +851,7 @@ async function chunkParsedLanguage(? normalizeSymbolType(typeNode.namedChild(0).type): normalizeSymbolType(typeNode.type));- if (nestableNode && symbolName && nestedConfig) {+ if (nestableNode && !moduleSymbol && symbolName && nestedConfig) {const before = chunks.length;emitNestedScoped(nestableNode, [], source, filePath, language, nestedConfig, chunks);if (chunks.length > before) continue;@@ -861,6 +866,7 @@ async function chunkParsedLanguage(parentSymbolPath: [],});if (declaresFunctionValue(typeNode)) chunk.metadata.definesFunction = true;+ if (!moduleSymbol && moduleStatementSymbol(node, language, false)) chunk.metadata.holdsCall = true;chunks.push(chunk);continue;}diff --git a/test/chunkers/code-lua.test.ts b/test/chunkers/code-lua.test.tsindex 6c88572d60..4ad8bbff3d 100644--- a/test/chunkers/code-lua.test.ts+++ b/test/chunkers/code-lua.test.ts@@ -112,7 +112,7 @@ describe('the grammar swap re-chunks Lua files only', () => {expect(CHUNKER_VERSION).toBe(8);expect(GRAMMAR_REVISIONS.lua).toBe(1);// #5001 added per-language revisions for the `__module__` chunks after `lua=1`; the Lua entry is unchanged.- expect(chunkerStamp()).toBe('8;lua=1;typescript=1;tsx=1;javascript=1;python=1;ruby=1');+ expect(chunkerStamp()).toBe('8;lua=1;typescript=2;tsx=2;javascript=2;python=2;ruby=2');});test('grammarOnlyDrift names the languages whose revision moved, and nothing on a version bump or a fresh source', () => {diff --git a/test/chunkers/code-module-anonymous-5001.test.ts b/test/chunkers/code-module-anonymous-5001.test.tsnew file mode 100644index 0000000000..16a91cee2d--- /dev/null+++ b/test/chunkers/code-module-anonymous-5001.test.ts@@ -0,0 +1,110 @@+/**+ * #5001 residual: a call edge needs a chunk with a qualified symbol around its call site, and #7409 gave top-level+ * expression statements the file's `__module__` chunk. Calls in every other top-level shape that names no definition+ * were still dropped, and their text was never indexed (so code-refs missed it too):+ * - anonymous default exports: `export default function () {…}`, `export default async (req) => …`,+ * `export default { fetch(req) {…} }`, `export default defineConfig({…})`+ * - top-level control flow: a TS `if`/`for`/`try`, Python's `if __name__ == "__main__": main()`+ * - a merged run of small declarations whose values are calls (`const app = createApp(); const db = openDb();`)+ * Each now lands on `<file>::__module__`, through the real chunker, edge extractor, edge mapping and resolver.+ * Defect 2 (resolved:false for TS defs) is pinned here too: within-file resolution matches `export statement` and+ * `lexical declaration` definitions; only cross-file calls stay unresolved, and the CLI labels them so.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { PGLiteEngine } from '../../src/core/pglite-engine.ts';+import { importCodeFile } from '../../src/core/import-file.ts';+import { resolveSymbolEdgesIncremental } from '../../src/core/chunkers/symbol-resolver.ts';+import { chunkCodeTextFull } from '../../src/core/chunkers/code.ts';++let engine: PGLiteEngine;++const FILES: Record<string, string> = {+ 'src/handler-example.ts': [+ "export const helperExample = (n: number) => n + 1;",+ "export default async (req: { n: number }) => helperExample(req.n);",+ ].join('\n'),+ 'src/worker-example.ts': [+ "export function handleExample(req: unknown) { return req; }",+ "export default { fetch(req: unknown) { return handleExample(req); } };",+ ].join('\n'),+ 'src/boot-example.ts': [+ "export function startExample() { return 1; }",+ "export function reportExample(e: unknown) { return e; }",+ "if (process.env.EXAMPLE) { startExample(); }",+ "try { startExample(); } catch (e) { reportExample(e); }",+ ].join('\n'),+ 'src/app-example.ts': [+ "export function createAppExample() { return {}; }",+ "export function openDbExample() { return {}; }",+ "const app = createAppExample();",+ "const db = openDbExample();",+ ].join('\n'),+ 'tools/cli_example.py': [+ "def main_example():",+ " return 1",+ "",+ "if __name__ == '__main__':",+ " main_example()",+ ].join('\n'),+};++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+ for (const [path, content] of Object.entries(FILES)) await importCodeFile(engine, path, content, { noEmbed: true });+ await resolveSymbolEdgesIncremental(engine, { sourceId: 'default' });+}, 60_000);++afterAll(async () => { await engine.disconnect(); });++async function callers(symbol: string) {+ return (await engine.getCallersOf(symbol, { allSources: true }))+ .filter(e => e.edge_type === 'calls')+ .map(e => ({ from: e.from_symbol_qualified, resolved: e.resolved }));+}++describe('#5001 calls in top-level code that names no definition', () => {+ test('an anonymous default-export arrow attributes its calls to the module', async () => {+ expect(await callers('helperExample')).toEqual([{ from: 'handler-example::__module__', resolved: true }]);+ });++ test('a method of an anonymous default-export object attributes its calls to the module', async () => {+ expect(await callers('handleExample')).toEqual([{ from: 'worker-example::__module__', resolved: true }]);+ });++ test('top-level if and try blocks attribute their calls to the module', async () => {+ expect(await callers('startExample')).toEqual([{ from: 'boot-example::__module__', resolved: true }]);+ expect(await callers('reportExample')).toEqual([{ from: 'boot-example::__module__', resolved: true }]);+ });++ test('a merged run of call-valued declarations keeps the module identity', async () => {+ expect(await callers('createAppExample')).toEqual([{ from: 'app-example::__module__', resolved: true }]);+ expect(await callers('openDbExample')).toEqual([{ from: 'app-example::__module__', resolved: true }]);+ });++ test('Python main guard attributes main() to the module and indexes its text', async () => {+ expect(await callers('main_example')).toEqual([{ from: 'cli_example::__module__', resolved: true }]);+ const { chunks } = await chunkCodeTextFull(FILES['tools/cli_example.py']!, 'tools/cli_example.py');+ expect(chunks.some(c => c.metadata.symbolName === '__module__' && c.text.includes("if __name__ == '__main__':"))).toBe(true);+ });++ test('a named definition keeps its own chunk and name', async () => {+ const { chunks } = await chunkCodeTextFull(FILES['src/boot-example.ts']!, 'src/boot-example.ts');+ expect(chunks.find(c => c.metadata.symbolName === 'startExample')?.metadata.symbolType).toBe('export statement');+ });+});++describe('#5001 defect 2: within-file resolution of TS export-statement and lexical-declaration definitions', () => {+ test('a same-file call to an export-statement or lexical-declaration def is resolved', async () => {+ const source = [+ "const rowByKey = (rows: string[], k: string) => rows.find(r => r === k);",+ "export function breakdownExample(rows: string[]) { return rowByKey(rows, 'a'); }",+ "export function usesExample() { return breakdownExample([]); }",+ ].join('\n');+ await importCodeFile(engine, 'src/review-example.ts', source, { noEmbed: true });+ await resolveSymbolEdgesIncremental(engine, { sourceId: 'default' });+ expect(await callers('rowByKey')).toEqual([{ from: 'breakdownExample', resolved: true }]);+ expect(await callers('breakdownExample')).toEqual([{ from: 'usesExample', resolved: true }]);+ });+});diff --git a/test/chunkers/code-module-level-calls.test.ts b/test/chunkers/code-module-level-calls.test.tsindex 291d6256cf..4ea0d0860f 100644--- a/test/chunkers/code-module-level-calls.test.ts+++ b/test/chunkers/code-module-level-calls.test.ts@@ -166,9 +166,9 @@ export function shutdown() { return app.close(); }});describe('#5001 the five languages whose inventory grows re-chunk on their own, like the Lua grammar swap', () => {- test('CHUNKER_VERSION is unchanged; typescript, tsx, javascript, python and ruby carry revision 1', () => {+ test('CHUNKER_VERSION is unchanged; typescript, tsx, javascript, python and ruby carry revision 2', () => {expect(CHUNKER_VERSION).toBe(8);- for (const language of ['typescript', 'tsx', 'javascript', 'python', 'ruby'] as const) expect(GRAMMAR_REVISIONS[language]).toBe(1);+ for (const language of ['typescript', 'tsx', 'javascript', 'python', 'ruby'] as const) expect(GRAMMAR_REVISIONS[language]).toBe(2);expect(grammarOnlyDrift('8;lua=1', chunkerStamp())).toEqual(new Set(['typescript', 'tsx', 'javascript', 'python', 'ruby']));});});