AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): AST code extraction for 21 more languages (13 -> 34)
xerj-org8c392b7c07ac
diff --git a/ROADMAP.md b/ROADMAP.mdindex dee9ff451..6fa7a6eb6 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -22,7 +22,7 @@ These are implemented and exercised by real API requests / the test suite / benc**Exactness, precisely.** No probabilistic sketch sits in the metric path: `cardinality` is a true distinct count rather than an HLL estimate, and `terms` `doc_count` is precise. Two deliberate exceptions, stated the same way in `engine/README.md` and `llms-full.txt`: (1) the **sampling family** is a sample by definition — `run_sampler` sorts the matched documents by `_score` and keeps the first `shard_size` (default **200**), so every sub-aggregation under `sampler`, `random_sampler` or `diversified_sampler` is computed over that slice rather than the whole match set, `diversified_sampler` additionally caps documents per `field` value, and `random_sampler` shares the `sampler` implementation and **ignores ES's `probability`** (an accepted-and-ignored input, #204); (2) `percentiles` with the `hdr` option returns HdrHistogram-quantized values, deliberately, so ES's own outputs reproduce — the default `tdigest` path sorts every value and interpolates instead.- **Dense-vector kNN** (`knn` query and ES 8.x top-level `knn`): unfiltered kNN on a full-precision cosine field (≥1,024 docs) is served by a **persisted HNSW graph with exact rescoring** — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937); filtered/nested kNN, non-cosine similarity, SQ8 fields, and small indexes run the exact brute-force scan (cosine mapped to `(1+cos)/2`).- **Hybrid search** — BM25 + kNN combined in a single request via the `hybrid` **query type** with `rrf|linear|learned` fusion, verified live. (The ES-native top-level `{query, knn}` body does **not** fuse — see *Known partials*.)-- **Zero-config folder onboarding** — `xerj autoindex <folder>` sniffs files, infers datasets, and creates one index per dataset: tree-sitter AST extraction for 13 languages (symbols, defs, line numbers — expansion tracked in [#295](https://github.com/xerj-org/xerj/issues/295)), CSV/JSON/JSONL/XML/YAML/SQLite/PDF/DOCX/HTML/log formats, `.gitignore`/`.xerjignore` support, incremental re-runs, and a machine-parseable progress stream.+- **Zero-config folder onboarding** — `xerj autoindex <folder>` sniffs files, infers datasets, and creates one index per dataset: tree-sitter AST extraction for 34 languages (symbols, defs, line numbers — the [#295](https://github.com/xerj-org/xerj/issues/295) expansion; still open: Clojure and source-SQL wait on usable grammar crates, Nim/Crystal have none, fixed-form Fortran is deliberately unclaimed), CSV/JSON/JSONL/XML/YAML/SQLite/PDF/DOCX/HTML/log formats, `.gitignore`/`.xerjignore` support, incremental re-runs, and a machine-parseable progress stream.- **Agent-memory REST API** (`/_memory/*`), **second-brain knowledge graph** (`/_graph`), **anomaly detection** (`_ml` with continuous datafeeds), **auto-embed on ingest** (default embedder is deterministic **lexical** feature-hashing — never described as neural; `--embed-mode neural` runs the in-binary BERT encoder, `--embed-mode proxy` an external endpoint).- **Columnar storage** — the ZBS2 columnar block with 9 domain-aware encodings, ZSTD/LZ4 codecs, and SQ8 vector quantization, wired into the segment write path.- Bulk / scroll / delete-by-query, aliases, index templates, **executed** index-lifecycle policies (ISM-modeled, `_ilm/*` + `_plugins/_ism/*`, since rc.15), `_cat/*`, `_cluster/health`, `_count` / `_msearch` / `_mget`, `_update` / `_update_by_query` — all live-verified.diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 2c3305b90..a1fbce65c 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -5384,9 +5384,9 @@ dependencies = [[[package]]name = "tree-sitter"-version = "0.25.10"+version = "0.26.12"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "78f873475d258561b06f1c595d93308a7ed124d9977cb26b148c2084a4a3cc87"+checksum = "83c567a8e18ae93f20982c90370b16fd24023aeaf52f6052b96957ab253a0fec"dependencies = ["cc","regex",@@ -5436,6 +5436,56 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-dart"+version = "0.2.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "325dd1e24ee9ee21111e9c43680ae7d6010aaa9f282b048a99b9c7163c1cf553"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-elixir"+version = "0.3.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "66dd064a762ed95bfc29857fa3cb7403bb1e5cb88112de0f6341b7e47284ba40"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-erlang"+version = "0.20.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3107efc581d335488dc38d2f51ca8b7a6a98e36f07f95a8e963f031edda6f568"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-fortran"+version = "0.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ea5fea41363a9b59666001dc26c4b356d4e164da27c6ce31145e00aba6bf9b16"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-fsharp"+version = "0.3.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b069005841682d258b95e9ee8cb0c0a0f16b77318ff56000504169c80690c67b"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-go"version = "0.23.4"@@ -5446,6 +5496,26 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-groovy"+version = "0.1.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5a20016017f0865ba902ca50354f92429de5de8df994e64ab7fae087a13c40ed"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-haskell"+version = "0.23.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "977c51e504548cba13fc27cb5a2edab2124cf6716a1934915d07ab99523b05a4"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-java"version = "0.23.5"@@ -5466,12 +5536,93 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-julia"+version = "0.23.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4144731a178812ee867619b1e98b3b91e54c1652304b26e5ebe3175b701de323"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-kotlin-ng"+version = "1.1.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e800ebbda938acfbf224f4d2c34947a31994b1295ee6e819b65226c7b51b4450"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-language"version = "0.1.7"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "009994f150cc0cd50ff54917d5bc8bffe8cad10ca10d81c34da2ec421ae61782"+[[package]]+name = "tree-sitter-lua"+version = "0.5.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8daaf5f4235188a58603c39760d5fa5d4b920d36a299c934adddae757f32a10c"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-matlab"+version = "1.3.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8e8d8831a78547f54860dd8467166b1ab0c466d03d43c4bb447af55c024281f7"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-nix"+version = "0.3.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4952a9733f3a98f6683a0ccd1035d84ab7a52f7e84eeed58548d86765ad92de3"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-objc"+version = "3.0.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9ca8bb556423fc176f0535e79d525f783a6684d3c9da81bf9d905303c129e1d2"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-ocaml"+version = "0.25.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0b943275476bac8f73e08bc4050e21d89d61ff68e1751b789ba74917b9147a85"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-perl"+version = "1.1.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e79d4ae71b42e595c24c354b59905ade8162bd400ebc53ab916ea8aec54da92d"+dependencies = [+ "cc",+ "tree-sitter",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-php"version = "0.23.11"@@ -5482,6 +5633,16 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-powershell"+version = "0.26.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3faf304d44b9ddd4a7d97804bb8de7daf564336dd5a526dc6de5b39238243022"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-python"version = "0.23.6"@@ -5492,6 +5653,16 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-r"+version = "1.3.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ffc9954ec870dcad6cffdd302b405306c68cf031ed79a78cd9746f6740d9fe20"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-ruby"version = "0.23.1"@@ -5512,6 +5683,36 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-scala"+version = "0.26.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "24e0ab4505990bfe30051761d40a7bf4033ce5a81c9eda9e20e987a5cdc84826"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-solidity"+version = "1.2.13"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4eacf8875b70879f0cb670c60b233ad0b68752d9e1474e6c3ef168eea8a90b25"+dependencies = [+ "cc",+ "tree-sitter-language",+]++[[package]]+name = "tree-sitter-swift"+version = "0.7.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "fe36052155b9dd69ca82b3b8f1b4ccfb2d867125ac1a4db1dd7331829242668c"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "tree-sitter-typescript"version = "0.23.2"@@ -5522,6 +5723,16 @@ dependencies = ["tree-sitter-language",]+[[package]]+name = "tree-sitter-zig"+version = "1.1.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ab11fc124851b0db4dd5e55983bbd9631192e93238389dcd44521715e5d53e28"+dependencies = [+ "cc",+ "tree-sitter-language",+]+[[package]]name = "try-lock"version = "0.2.5"@@ -6488,14 +6699,35 @@ dependencies = ["tree-sitter-c","tree-sitter-c-sharp","tree-sitter-cpp",+ "tree-sitter-dart",+ "tree-sitter-elixir",+ "tree-sitter-erlang",+ "tree-sitter-fortran",+ "tree-sitter-fsharp","tree-sitter-go",+ "tree-sitter-groovy",+ "tree-sitter-haskell","tree-sitter-java","tree-sitter-javascript",+ "tree-sitter-julia",+ "tree-sitter-kotlin-ng",+ "tree-sitter-lua",+ "tree-sitter-matlab",+ "tree-sitter-nix",+ "tree-sitter-objc",+ "tree-sitter-ocaml",+ "tree-sitter-perl","tree-sitter-php",+ "tree-sitter-powershell","tree-sitter-python",+ "tree-sitter-r","tree-sitter-ruby","tree-sitter-rust",+ "tree-sitter-scala",+ "tree-sitter-solidity",+ "tree-sitter-swift","tree-sitter-typescript",+ "tree-sitter-zig","walkdir","xerj-common","xxhash-rust",diff --git a/engine/crates/xerj-autoindex/Cargo.toml b/engine/crates/xerj-autoindex/Cargo.tomlindex e3b4adad0..49f5587d5 100644--- a/engine/crates/xerj-autoindex/Cargo.toml+++ b/engine/crates/xerj-autoindex/Cargo.toml@@ -37,7 +37,10 @@ reqwest = { workspace = true, features = ["blocking"] }# --- AST code extraction (tree-sitter). Grammars expose a tree-sitter-language# LanguageFn, decoupled from the core version, so one core serves all. ----tree-sitter = "0.25"+# Core 0.26: forced by tree-sitter-perl 1.1.2, the only #295 grammar with a+# hard (`links`) core dep — same story as the c-sharp 0.24→0.25 bump. Every+# other grammar binds through tree-sitter-language and does not care.+tree-sitter = "0.26"tree-sitter-python = "0.23"tree-sitter-c = "0.23"tree-sitter-rust = "0.23"@@ -50,6 +53,44 @@ tree-sitter-ruby = "0.23"tree-sitter-php = "0.23"tree-sitter-c-sharp = "0.23"tree-sitter-bash = "0.23"+# --- #295 Tier 1: mainstream languages users expect. Versions are the ones+# checked against crates.io in the issue; each must expose a Language loadable+# by core 0.26 (ABI 13–15) — the all-grammars smoke test enforces that.+# NOT `tree-sitter-kotlin` (0.3.8): that crate hard-links core >=0.21,<0.23+# and cannot coexist with core 0.26. The `-ng` fork is the maintained+# tree-sitter-language binding of the same fwcd grammar.+tree-sitter-kotlin-ng = "1.1"+tree-sitter-swift = "0.7"+tree-sitter-scala = "0.26"+tree-sitter-dart = "0.2"+tree-sitter-lua = "0.5"+tree-sitter-perl = "1.1"+tree-sitter-r = "1.3"+tree-sitter-julia = "0.23"+tree-sitter-haskell = "0.23"+tree-sitter-elixir = "0.3"+# --- #295 Tier 2: common in ecosystems we already serve.+# NOT tree-sitter-clojure: its only release (0.1.0) hard-links core ^0.25.6,+# mutually exclusive with the core 0.26 that Tier 1's perl grammar forces —+# revisit when a clojure release against 0.26 exists.+# NOT tree-sitter-sql: 0.0.2 is immature (the issue itself says evaluate+# first or defer) and `.sql` routing must not steal sqldump files — deferred.+tree-sitter-erlang = "0.20"+tree-sitter-ocaml = "0.25"+tree-sitter-zig = "1.1"+tree-sitter-objc = "3.0"+tree-sitter-groovy = "0.1"+tree-sitter-powershell = "0.26"+tree-sitter-fsharp = "0.3"+tree-sitter-nix = "0.3"+# --- #295 Tier 3: long tail, only where the grammar proves solid.+# NOT tree-sitter-nim / tree-sitter-crystal: their only releases (0.1.0) are+# toy grammars (~17 node types, no module/struct/def nodes — verified in+# their node-types.json); they cannot parse real code in either language and+# no alternative crate exists on crates.io.+tree-sitter-fortran = "0.6"+tree-sitter-matlab = "1.3"+tree-sitter-solidity = "1.2"streaming-iterator = "0.1"diff --git a/engine/crates/xerj-autoindex/src/extract/code.rs b/engine/crates/xerj-autoindex/src/extract/code.rsindex 0bb934754..a44d9345d 100644--- a/engine/crates/xerj-autoindex/src/extract/code.rs+++ b/engine/crates/xerj-autoindex/src/extract/code.rs@@ -32,6 +32,12 @@ struct LangDef {exts: &'static [&'static str],language: Language,query: Query,+ /// Content probe for extensions claimed by more than one language (#295).+ /// `.m` is both Objective-C and MATLAB: when several registry rows claim+ /// an extension, the first row whose probe returns true wins; a row with+ /// no probe is the fallback owner. Rows with a unique extension leave this+ /// `None` and are never probed.+ probe: Option<fn(&str) -> bool>,}/// Is this extension a language we AST-parse? Cheap check used by the sniffer.@@ -42,14 +48,58 @@ pub fn is_code_ext(ext: &str) -> bool {fn def(name: &'static str, exts: &'static [&'static str], lang: Language, q: &str) -> LangDef {let query = Query::new(&lang, q).unwrap_or_else(|e| panic!("bad {name} query: {e}"));+ // Text predicates (#eq?/#any-of?/#match?) are NOT applied by the core+ // library — parse_symbols() evaluates them itself. Reject operators it+ // does not implement here, at registry build time, so a typo'd or+ // unsupported predicate fails `all_queries_compile` instead of silently+ // matching everything at extraction time.+ for i in 0..query.pattern_count() {+ for p in query.general_predicates(i) {+ assert!(+ matches!(&*p.operator, "eq?" | "not-eq?" | "any-of?" | "not-any-of?"),+ "{name} query uses unsupported predicate #{}",+ p.operator+ );+ }+ }LangDef {name,exts,language: lang,query,+ probe: None,}}+/// A `def(...)` whose extension is shared with another language — see+/// `LangDef::probe`.+fn def_probed(+ name: &'static str,+ exts: &'static [&'static str],+ lang: Language,+ q: &str,+ probe: fn(&str) -> bool,+) -> LangDef {+ let mut d = def(name, exts, lang, q);+ d.probe = Some(probe);+ d+}++/// `.m` probe: Objective-C vs MATLAB. Real Objective-C files near-universally+/// carry an `#import`/`@interface`/`@implementation`/`@protocol`/`@end`+/// marker; a `.m` with none of them is treated as MATLAB (the fallback row).+fn looks_like_objc(text: &str) -> bool {+ [+ "#import",+ "@interface",+ "@implementation",+ "@protocol",+ "@end",+ ]+ .iter()+ .any(|m| text.contains(m))+}+fn registry() -> &'static [LangDef] {static REG: OnceLock<Vec<LangDef>> = OnceLock::new();REG.get_or_init(|| {@@ -108,6 +158,139 @@ fn registry() -> &'static [LangDef] {tree_sitter_bash::LANGUAGE.into(),BASH_Q,),+ // ── #295 Tier 1 ─────────────────────────────────────────────+ def(+ "kotlin",+ &["kt", "kts"],+ tree_sitter_kotlin_ng::LANGUAGE.into(),+ KOTLIN_Q,+ ),+ def(+ "swift",+ &["swift"],+ tree_sitter_swift::LANGUAGE.into(),+ SWIFT_Q,+ ),+ // `.sc` is also SuperCollider; Scala is the statistically dominant+ // owner in the repos agents index, and no SuperCollider grammar is+ // registered, so no probe is needed (same policy as `.h` → C).+ def(+ "scala",+ &["scala", "sc"],+ tree_sitter_scala::LANGUAGE.into(),+ SCALA_Q,+ ),+ def("dart", &["dart"], tree_sitter_dart::LANGUAGE.into(), DART_Q),+ def("lua", &["lua"], tree_sitter_lua::LANGUAGE.into(), LUA_Q),+ // `.pl` is also Prolog. Perl owns it: overwhelmingly dominant in+ // real repos, and no Prolog grammar is registered to probe for.+ def(+ "perl",+ &["pl", "pm"],+ tree_sitter_perl::LANGUAGE.into(),+ PERL_Q,+ ),+ // `.r` is also Rebol; R owns it, same policy as `.pl`.+ def("r", &["r"], tree_sitter_r::LANGUAGE.into(), R_Q),+ def(+ "julia",+ &["jl"],+ tree_sitter_julia::LANGUAGE.into(),+ JULIA_Q,+ ),+ def(+ "haskell",+ &["hs"],+ tree_sitter_haskell::LANGUAGE.into(),+ HASKELL_Q,+ ),+ def(+ "elixir",+ &["ex", "exs"],+ tree_sitter_elixir::LANGUAGE.into(),+ ELIXIR_Q,+ ),+ // ── #295 Tier 2 ─────────────────────────────────────────────+ def(+ "erlang",+ &["erl", "hrl"],+ tree_sitter_erlang::LANGUAGE.into(),+ ERLANG_Q,+ ),+ // OCaml ships separate grammars for implementations and+ // interfaces; `.mli` needs its own row (and query) or interface+ // files would fail to parse and fall back to plain text.+ def(+ "ocaml",+ &["ml"],+ tree_sitter_ocaml::LANGUAGE_OCAML.into(),+ OCAML_Q,+ ),+ def(+ "ocaml_interface",+ &["mli"],+ tree_sitter_ocaml::LANGUAGE_OCAML_INTERFACE.into(),+ OCAML_MLI_Q,+ ),+ def("zig", &["zig"], tree_sitter_zig::LANGUAGE.into(), ZIG_Q),+ // `.m` collision (issue #295 decision): Objective-C vs MATLAB is+ // the one collision where both owners are registered, so it is+ // resolved by content probe — objc markers win, MATLAB is the+ // fallback row below. `.mm` (Objective-C++) is claimed too: the+ // objc grammar parses the Objective-C subset and a file it cannot+ // parse still indexes as plain text.+ def_probed(+ "objc",+ &["m", "mm"],+ tree_sitter_objc::LANGUAGE.into(),+ OBJC_Q,+ looks_like_objc,+ ),+ def(+ "groovy",+ &["groovy", "gradle"],+ tree_sitter_groovy::LANGUAGE.into(),+ GROOVY_Q,+ ),+ def(+ "powershell",+ &["ps1", "psm1", "psd1"],+ tree_sitter_powershell::LANGUAGE.into(),+ POWERSHELL_Q,+ ),+ def(+ "fsharp",+ &["fs", "fsx"],+ tree_sitter_fsharp::LANGUAGE_FSHARP.into(),+ FSHARP_Q,+ ),+ def("nix", &["nix"], tree_sitter_nix::LANGUAGE.into(), NIX_Q),+ // ── #295 Tier 3 ─────────────────────────────────────────────+ // Free-form Fortran only: `.f` (fixed-form) is deliberately NOT+ // claimed — the grammar is free-form, a fixed-form file would+ // parse to garbage, and an unclaimed extension still indexes as+ // text, which is strictly better than a wrong AST.+ def(+ "fortran",+ &["f90", "f95", "f03"],+ tree_sitter_fortran::LANGUAGE.into(),+ FORTRAN_Q,+ ),+ // MATLAB is the probe-less fallback for `.m` — must sort after+ // the objc row (first passing probe wins, probe-less row catches+ // the rest).+ def(+ "matlab",+ &["m"],+ tree_sitter_matlab::LANGUAGE.into(),+ MATLAB_Q,+ ),+ def(+ "solidity",+ &["sol"],+ tree_sitter_solidity::LANGUAGE.into(),+ SOLIDITY_Q,+ ),]})}@@ -124,15 +307,30 @@ pub fn extract(path: &Path, sn: &crate::sniff::Sniffed, sink: Sink) -> Result<Ex.and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase();- let Some(def) = registry().iter().find(|d| d.exts.contains(&ext.as_str())) else {+ let claimants: Vec<&LangDef> = registry()+ .iter()+ .filter(|d| d.exts.contains(&ext.as_str()))+ .collect();+ if claimants.is_empty() {stats.junk += 1;return Ok(stats);- };+ }let Some(bytes) = super::read_whole(path, false, CODE_CAP)? else {stats.junk += 1;return Ok(stats);};let (text, _) = crate::sniff::decode_text(&bytes);+ // Shared extensions (`.m`): first claimant whose content probe passes wins;+ // the probe-less row is the fallback. Single-claimant extensions skip the+ // probe entirely.+ let def = if claimants.len() == 1 {+ claimants[0]+ } else {+ claimants+ .iter()+ .find(|d| d.probe.map(|p| p(&text)).unwrap_or(true))+ .unwrap_or(&claimants[0])+ };// Parse + capture definitions. A parse failure (or an over-deep tree) is not// fatal — index the file as plain text so its content is still searchable.@@ -144,6 +342,48 @@ pub fn extract(path: &Path, sn: &crate::sniff::Sniffed, sink: Sink) -> Result<Ex/// (name, kind, 1-based line)type Symbol = (String, String, usize);+/// Evaluate a pattern's text predicates against one match. The core library+/// exposes `#eq?`/`#any-of?` (and their `not-` forms) as *general* predicates+/// and applies NONE of them itself — without this, the Elixir query would tag+/// every function call in the file as a definition. Operators are validated+/// in `def()`; an unknown one landing here rejects the match (fail closed).+fn predicates_hold(query: &Query, m: &tree_sitter::QueryMatch, text: &str) -> bool {+ use tree_sitter::QueryPredicateArg as Arg;+ let cap_text = |ix: u32| -> &str {+ m.captures+ .iter()+ .find(|c| c.index == ix)+ .and_then(|c| text.get(c.node.byte_range()))+ .unwrap_or("")+ };+ query.general_predicates(m.pattern_index).iter().all(|p| {+ let mut args = p.args.iter();+ let Some(Arg::Capture(first)) = args.next() else {+ return false;+ };+ let got = cap_text(*first);+ let strings: Vec<&str> = p+ .args+ .iter()+ .skip(1)+ .filter_map(|a| match a {+ Arg::String(s) => Some(&**s),+ Arg::Capture(c) => text.get(+ m.captures+ .iter()+ .find(|x| x.index == *c)+ .map(|x| x.node.byte_range())?,+ ),+ })+ .collect();+ match &*p.operator {+ "eq?" | "any-of?" => strings.iter().any(|s| *s == got),+ "not-eq?" | "not-any-of?" => !strings.iter().any(|s| *s == got),+ _ => false,+ }+ })+}+fn parse_symbols(def: &LangDef, text: &str) -> Option<Vec<Symbol>> {let mut parser = Parser::new();parser.set_language(&def.language).ok()?;@@ -153,8 +393,14 @@ fn parse_symbols(def: &LangDef, text: &str) -> Option<Vec<Symbol>> {let mut cursor = QueryCursor::new();let mut it = cursor.matches(&def.query, tree.root_node(), text.as_bytes());while let Some(m) = it.next() {+ if !predicates_hold(&def.query, m, text) {+ continue;+ }for cap in m.captures {let kind = names[cap.index as usize]; // capture name == symbol kind+ if kind.starts_with('_') {+ continue; // predicate-only capture (`@_kw`), not a symbol+ }let node = cap.node;let name = text.get(node.byte_range()).unwrap_or("").trim();if name.is_empty() || name.len() > 200 {@@ -519,6 +765,326 @@ const BASH_Q: &str = r#"(function_definition name: (word) @function)"#;+// ── #295 queries. Prior art: where the grammar crate ships an author-written+// `queries/tags.scm`, the definition patterns below are adapted from it and+// the source is cited; reference-capture patterns and doc-comment plumbing+// are dropped (this extractor only wants definitions), and `@definition.*`+// capture names are replaced by this file's capture-name==kind convention.+// Where no tags.scm ships, patterns are derived from the grammar's+// `src/node-types.json` and pinned by the fixture tests below. ────────────++// Kotlin has no tags.scm in tree-sitter-kotlin-ng 1.1.0; node-types.json:+// class/object/function declarations carry `name: (identifier)`. The grammar+// has no separate interface node — `interface Foo` is a class_declaration+// whose keyword token distinguishes it, so the "interface" pattern matches+// the anonymous token to keep `interface Foo` searchable as such. Kotlin has+// no `.h`-style split, and function-local `fun` is rare enough that the+// unanchored function pattern is signal (local helpers are still named+// definitions, as in Rust).+const KOTLIN_Q: &str = r#"+(class_declaration "interface" name: (identifier) @interface)+(class_declaration name: (identifier) @class)+(object_declaration name: (identifier) @object)+(function_declaration name: (identifier) @function)+(type_alias type: (identifier) @type)+"#;++// Adapted from tree-sitter-swift 0.7.3 queries/tags.scm (class_declaration /+// protocol_declaration / function_declaration name captures). The grammar+// folds class/struct/enum/actor/extension into one class_declaration node+// with a `declaration_kind` token field — matched here so a Swift `struct`+// is searchable as `struct Point`, not mislabelled a class. tags.scm's+// method/property captures are dropped: methods are the same+// function_declaration node captured below, and properties are the+// locals-trap shape (#170).+const SWIFT_Q: &str = r#"+(class_declaration declaration_kind: "class" name: (type_identifier) @class)+(class_declaration declaration_kind: "struct" name: (type_identifier) @struct)+(class_declaration declaration_kind: "enum" name: (type_identifier) @enum)+(class_declaration declaration_kind: "actor" name: (type_identifier) @class)+(class_declaration declaration_kind: "extension" name: (user_type) @class)+(protocol_declaration name: (type_identifier) @protocol)+(function_declaration name: (simple_identifier) @function)+"#;++// Adapted from tree-sitter-scala 0.26.2 queries/tags.scm. `val`/`var`+// captures are dropped: the same node appears inside method bodies, which is+// the 90%-locals trap measured for C in #170; tags.scm tags them because+// editors want local navigation — `defs` does not.+const SCALA_Q: &str = r#"+(class_definition name: (identifier) @class)+(object_definition name: (identifier) @object)+(trait_definition name: (identifier) @trait)+(enum_definition name: (identifier) @enum)+(full_enum_case name: (identifier) @const)+(simple_enum_case name: (identifier) @const)+(function_definition name: (identifier) @function)+(type_definition name: (type_identifier) @type)+"#;++// Adapted from tree-sitter-dart 0.2.0 queries/tags.scm (class / mixin / enum+// / function / method / typedef / enum-constant captures; getter/setter and+// constructor variants dropped as weight without retrieval surface).+const DART_Q: &str = r#"+(class_declaration name: (identifier) @class)+(mixin_declaration (identifier) @mixin)+(enum_declaration name: (identifier) @enum)+(enum_constant name: (identifier) @const)+(function_signature name: (identifier) @function)+(type_alias (type_identifier) @type)+"#;++// Adapted from tree-sitter-lua 0.5.0 queries/tags.scm: the three spellings a+// Lua module actually uses — `function M.add()`, `function M:method()`, and+// `local add = function()` / `M.add = function()` assignments, plus+// function-valued table fields.+const LUA_Q: &str = r#"+(function_declaration name: (identifier) @function)+(function_declaration name: (dot_index_expression field: (identifier) @function))+(function_declaration name: (method_index_expression method: (identifier) @method))+(assignment_statement (variable_list name: (identifier) @function) (expression_list value: (function_definition)))+(assignment_statement (variable_list name: (dot_index_expression field: (identifier) @function)) (expression_list value: (function_definition)))+(table_constructor (field name: (identifier) @function value: (function_definition)))+"#;++// No tags.scm in tree-sitter-perl 1.1.2; node-types.json: function_definition+// carries `name: (identifier)`, packages are a bare `package_name` child.+const PERL_Q: &str = r#"+(package_statement (package_name) @module)+(function_definition name: (identifier) @function)+"#;++// Adapted from tree-sitter-r 1.3.0 queries/tags.scm: R has no declaration+// keyword — a "definition" is `name <- function(...)` (or `=`), which is+// exactly what tags.scm matches. The string-lhs variants are dropped+// (assigning a function to a string name is vanishingly rare outside+// metaprogramming).+const R_Q: &str = r#"+(binary_operator lhs: (identifier) @function operator: "<-" rhs: (function_definition))+(binary_operator lhs: (identifier) @function operator: "=" rhs: (function_definition))+"#;++// No tags.scm in tree-sitter-julia 0.23.1; node-types.json: definitions wrap+// a `signature` (whose call_expression holds the name) or a `type_head`.+// The bare-identifier signature is the zero-arg `function f end` form; the+// binary_expression type_head is `struct Foo <: Bar`.+const JULIA_Q: &str = r#"+(function_definition (signature (call_expression (identifier) @function)))+(function_definition (signature (identifier) @function))+(macro_definition (signature (call_expression (identifier) @macro)))+(struct_definition (type_head (identifier) @struct))+(struct_definition (type_head (binary_expression (identifier) @struct)))+(abstract_definition (type_head (identifier) @type))+(abstract_definition (type_head (binary_expression (identifier) @type)))+(module_definition name: (identifier) @module)+"#;++// No tags.scm in tree-sitter-haskell 0.23.1; node-types.json: top level is+// `haskell > declarations > (function|signature|data_type|…)`. The function+// and signature patterns are anchored to `declarations` because the same+// `function` node appears in `where` blocks (via `local_binds`, which holds+// `decl` directly — so the anchor excludes exactly the locals, #170's trap).+// `signature` is captured too: every exported Haskell function has a type+// signature, and a file of signatures-only (class heads, foreign imports)+// must not extract zero symbols. Kinds follow the language's own keywords+// (`data`, `class`) so `data Maybe` matches what a Haskeller searches.+const HASKELL_Q: &str = r#"+(declarations (function name: (variable) @function))+(declarations (signature name: (variable) @function))+(data_type name: (name) @data)+(newtype name: (name) @data)+(class name: (name) @class)+(type_synomym name: (name) @type)+"#;++// Adapted from tree-sitter-elixir 0.3.5 queries/tags.scm — the module and+// function/macro definition patterns, including the guard-clause form. In+// Elixir `def` is itself a macro, so a definition is a `call` whose target+// spells a def-keyword: the `#any-of?` predicates (evaluated by+// predicates_hold — the core library applies none of them) are what keeps+// every other function call in the file out of `defs`. tags.scm's reference+// patterns and @ignore plumbing are dropped.+const ELIXIR_Q: &str = r#"+(call target: (identifier) @_kw (arguments (alias) @module) (#any-of? @_kw "defmodule" "defprotocol"))+(call target: (identifier) @_kw+ (arguments [+ (identifier) @function+ (call target: (identifier) @function)+ (binary_operator left: (call target: (identifier) @function) operator: "when")+ ])+ (#any-of? @_kw "def" "defp" "defdelegate" "defguard" "defguardp" "defmacro" "defmacrop" "defn" "defnp"))+"#;++// No tags.scm in tree-sitter-erlang 0.20.0 (the WhatsApp ELP grammar);+// node-types.json: fun_decl wraps function_clause (name: atom),+// module/record/define/type attributes carry their own name fields. `.hrl`+// headers are records + macros — without those two patterns a header+// extracts zero symbols, the same failure C had before #170. `-define`+// names are `var` (uppercase) or `atom` (lowercase); both are captured.+const ERLANG_Q: &str = r#"+(fun_decl (function_clause name: (atom) @function))+(module_attribute name: (atom) @module)+(record_decl name: (atom) @record)+(pp_define lhs: (macro_lhs name: (var) @const))+(pp_define lhs: (macro_lhs name: (atom) @const))+(type_alias name: (type_name name: (atom) @type))+"#;++// Adapted from tree-sitter-ocaml 0.25.0 queries/tags.scm: let-bindings with+// parameters (or a fun/function body) are functions — a bare `let x = 5` is+// deliberately not one; modules, module types, classes, methods, types and+// externals as tagged upstream. Doc-comment (#strip!) plumbing dropped.+const OCAML_Q: &str = r#"+(module_definition (module_binding (module_name) @module))+(module_type_definition (module_type_name) @interface)+(type_definition (type_binding name: (type_constructor) @type))+(value_definition (let_binding pattern: (value_name) @function (parameter)))+(value_definition (let_binding pattern: (value_name) @function body: (fun_expression)))+(value_definition (let_binding pattern: (value_name) @function body: (function_expression)))+(external (value_name) @function)+(method_definition (method_name) @method)+(class_definition (class_binding (class_name) @class))+"#;++// The interface grammar is separate (see the registry row). A `.mli` is a+// module's public surface: every `val` line IS the API, so+// value_specification is the load-bearing capture — the tags.scm above+// only covers `.ml` nodes and would extract zero symbols here.+const OCAML_MLI_Q: &str = r#"+(value_specification (value_name) @function)+(type_definition (type_binding name: (type_constructor) @type))+(module_definition (module_binding (module_name) @module))+(module_type_definition (module_type_name) @interface)+(external (value_name) @function)+"#;++// No tags.scm in tree-sitter-zig 1.1.2; node-types.json: functions carry+// `name: (identifier)`; types have NO name of their own — `const Point =+// struct {...}` is a variable_declaration whose value is the container+// declaration, so the type patterns match that shape. The bare top-level+// const pattern is anchored to source_file (function-local `const` in Zig is+// ordinary control flow, the locals trap); the container patterns need no+// anchor because the shape itself (a container child) cannot be a local.+const ZIG_Q: &str = r#"+(function_declaration name: (identifier) @function)+(variable_declaration (identifier) @struct (struct_declaration))+(variable_declaration (identifier) @enum (enum_declaration))+(variable_declaration (identifier) @union (union_declaration))+(source_file (variable_declaration (identifier) @const))+"#;++// No tags.scm in tree-sitter-objc 3.0.2. The C-family half (functions,+// structs, enums, typedefs, #defines) reuses C_Q's measured shapes — same+// nodes, same locals-trap reasoning (see C_Q). The Objective-C half from+// node-types.json: @interface/@implementation/@protocol carry their name as+// the FIRST identifier child (`superclass`/`category` are later fields, and+// a bare `(identifier)` pattern would capture those too — the `.` anchor is+// what keeps them out). Method selectors: unary selectors are a bare+// identifier child; each keyword segment is a keyword_declarator whose+// first child is the segment name (multi-segment selectors yield one symbol+// per segment; `defs` dedups).+const OBJC_Q: &str = r#"+(class_interface . (identifier) @class)+(class_implementation . (identifier) @class)+(protocol_declaration . (identifier) @protocol)+(method_definition (identifier) @method)+(method_definition (keyword_declarator . (identifier) @method))+(method_declaration (identifier) @method)+(method_declaration (keyword_declarator . (identifier) @method))+(function_definition declarator: (function_declarator declarator: (identifier) @function))+(function_definition declarator: (pointer_declarator declarator: (function_declarator declarator: (identifier) @function)))+(struct_specifier name: (type_identifier) @struct)+(enum_specifier name: (type_identifier) @enum)+(type_definition declarator: (type_identifier) @type)+(preproc_def name: (identifier) @const value: (preproc_arg))+"#;++// No tags.scm in tree-sitter-groovy 0.1.2; node-types.json mirrors the Java+// grammar's shapes (name: (identifier) throughout) plus a Groovy-specific+// function_definition for script-level `def foo() {}` — the form Gradle+// build scripts and Jenkinsfiles are made of.+const GROOVY_Q: &str = r#"+(class_declaration name: (identifier) @class)+(interface_declaration name: (identifier) @interface)+(enum_declaration name: (identifier) @enum)+(method_declaration name: (identifier) @method)+(constructor_declaration name: (identifier) @method)+(function_definition name: (identifier) @function)+"#;++// No tags.scm in tree-sitter-powershell 0.26.4; node-types.json: none of the+// definition nodes use fields — the name is the first named child+// (function_name for functions, simple_name for class/enum/method), so the+// class/enum patterns anchor to keep member names from matching.+const POWERSHELL_Q: &str = r#"+(function_statement (function_name) @function)+(class_statement . (simple_name) @class)+(class_method_definition (simple_name) @method)+(enum_statement . (simple_name) @enum)+"#;++// Adapted from tree-sitter-fsharp 0.3.11 queries/tags.scm. The function+// patterns keep upstream's anchoring to the four top-level contexts+// (file/named_module/module_defn/namespace via declaration_expression) —+// F# nests `let` inside functions constantly, and the same+// function_or_value_defn node models both, so the unanchored form is the+// locals trap (#170). The type pattern collapses upstream's nine-variant+// enumeration into a wildcard: every variant wraps the same type_name node.+const FSHARP_Q: &str = r#"+(named_module name: (long_identifier) @module)+(module_defn . (_) @module)+(type_definition (_ (type_name type_name: (_) @type)))+(file (declaration_expression (function_or_value_defn (function_declaration_left . (_) @function))))+(named_module (declaration_expression (function_or_value_defn (function_declaration_left . (_) @function))))+(module_defn (declaration_expression (function_or_value_defn (function_declaration_left . (_) @function))))+(namespace (declaration_expression (function_or_value_defn (function_declaration_left . (_) @function))))+(member_defn (method_or_prop_defn name: (property_or_ident) @method))+"#;++// Adapted from tree-sitter-nix 0.3.0 queries/tags.scm — its one definition+// pattern: a binding whose value is a function expression. The attrpath+// capture is narrowed to the attr identifier so `defs` carries `foo`, not+// `foo.bar.baz` punctuation.+const NIX_Q: &str = r#"+(binding attrpath: (attrpath attr: (identifier) @function) expression: (function_expression))+"#;++// Adapted from tree-sitter-fortran 0.6.0 queries/tags.scm: functions,+// subroutines, modules/programs/submodules, derived types. tags.scm maps+// program/submodule to module and derived types to class; derived types are+// @type here (`type :: point_t` is what a Fortran user searches).+const FORTRAN_Q: &str = r#"+(function_statement (name) @function)+(subroutine_statement (name) @function)+(module_statement (name) @module)+(submodule_statement (name) @module)+(program_statement (name) @module)+(derived_type_statement (type_name) @type)+"#;++// Adapted from tree-sitter-matlab 1.3.0 queries/neovim/tags.scm — the two+// definition patterns (function/class), name-fielded.+const MATLAB_Q: &str = r#"+(function_definition name: (identifier) @function)+(class_definition name: (identifier) @class)+"#;++// Adapted from tree-sitter-solidity 1.2.13 queries/tags.scm. Kinds keep the+// language's own keywords (contract/library/event) rather than tags.scm's+// class/interface mapping — `contract Token` is the search a Solidity user+// types. Functions inside contracts are the same function_definition node,+// captured once, unanchored (Solidity has no nested functions to trap on).+const SOLIDITY_Q: &str = r#"+(contract_declaration name: (identifier) @contract)+(interface_declaration name: (identifier) @interface)+(library_declaration name: (identifier) @library)+(function_definition name: (identifier) @function)+(struct_declaration name: (identifier) @struct)+(enum_declaration name: (identifier) @enum)+(event_definition name: (identifier) @event)+(modifier_definition name: (identifier) @function)+"#;+#[cfg(test)]mod tests {use super::*;@@ -558,6 +1124,134 @@ mod tests {assert_eq!(records[0].fields["title"], "app.py");}+ /// #295 acceptance criterion: every registered grammar must instantiate+ /// on the linked core. Core 0.26 accepts ABI 13–15; a grammar crate+ /// generated against a newer ABI would refuse to load — this makes that+ /// an immediate CI failure on every OS in the matrix rather than a+ /// runtime parse failure at a user's machine (the c-sharp crate already+ /// forced one core bump this way).+ #[test]+ fn all_languages_load() {+ for d in registry() {+ let mut p = Parser::new();+ assert!(+ p.set_language(&d.language).is_ok(),+ "{}: grammar ABI {} incompatible with linked tree-sitter core",+ d.name,+ d.language.abi_version()+ );+ }+ }++ /// The public docs (`ROADMAP.md`, `landing/index.html`) state a language+ /// count. Honest-claims rule: pin it to the registry so the number cannot+ /// drift. Registry ROWS exceed languages by one — OCaml needs two rows+ /// because `.ml` and `.mli` are separate grammars — while `tsx` has always+ /// been counted as its own language by these docs.+ #[test]+ fn documented_language_count() {+ assert_eq!(registry().len(), 35, "registry row count changed");+ let langs: std::collections::HashSet<&str> = registry()+ .iter()+ .map(|d| d.name.trim_end_matches("_interface"))+ .collect();+ assert_eq!(+ langs.len(),+ 34,+ "docs say 34 languages; update ROADMAP.md and landing/index.html"+ );+ }++ /// The #295 regression test: these languages fell through to the prose+ /// extractor at HEAD (verified live — `language`/`symbols`/`defs` all+ /// null after an `xerj autoindex` run over Kotlin/Swift/Lua/Elixir+ /// fixtures). An unclaimed extension is not code to the sniffer AT ALL,+ /// so those files were chunked into body-only records. This fails on the+ /// pre-#295 registry and pins every newly claimed extension to its+ /// language.+ #[test]+ fn issue_295_extensions_route() {+ for (ext, lang) in [+ ("kt", "kotlin"),+ ("kts", "kotlin"),+ ("swift", "swift"),+ ("scala", "scala"),+ ("sc", "scala"),+ ("dart", "dart"),+ ("lua", "lua"),+ ("pl", "perl"),+ ("pm", "perl"),+ ("r", "r"),+ ("jl", "julia"),+ ("hs", "haskell"),+ ("ex", "elixir"),+ ("exs", "elixir"),+ ("erl", "erlang"),+ ("hrl", "erlang"),+ ("ml", "ocaml"),+ ("mli", "ocaml_interface"),+ ("zig", "zig"),+ ("m", "objc"),+ ("mm", "objc"),+ ("groovy", "groovy"),+ ("gradle", "groovy"),+ ("ps1", "powershell"),+ ("psm1", "powershell"),+ ("psd1", "powershell"),+ ("fs", "fsharp"),+ ("fsx", "fsharp"),+ ("nix", "nix"),+ ("f90", "fortran"),+ ("f95", "fortran"),+ ("f03", "fortran"),+ ("sol", "solidity"),+ ] {+ assert!(is_code_ext(ext), "{ext} must be recognised as code");+ let d = registry().iter().find(|d| d.exts.contains(&ext));+ assert_eq!(+ d.map(|d| d.name),+ Some(lang),+ "{ext} must route to the {lang} grammar"+ );+ }+ // Deliberately NOT claimed, each for a stated reason: fixed-form+ // Fortran would mis-parse under the free-form grammar; nim/crystal+ // have no usable published grammar; sql routing is deferred; clojure+ // waits on a crate release against core 0.26.+ for ext in ["f", "nim", "cr", "sql", "clj"] {+ assert!(!is_code_ext(ext), "{ext} must stay unclaimed (see #295)");+ }+ }++ /// The `.m` collision decision from #295: content probe. Objective-C+ /// markers route to objc; a markerless `.m` is MATLAB. The probe only+ /// runs for extensions with more than one claimant.+ #[test]+ fn m_extension_content_probe() {+ assert!(looks_like_objc("#import <Foundation/Foundation.h>\n"));+ assert!(looks_like_objc("@interface Foo : NSObject\n@end\n"));+ assert!(!looks_like_objc(+ "function y = square(x)\n y = x^2;\nend\n"+ ));+ // Registry order is load-bearing: objc (probed) must sort before+ // matlab (fallback) among the `.m` claimants.+ let claimants: Vec<&str> = registry()+ .iter()+ .filter(|d| d.exts.contains(&"m"))+ .map(|d| d.name)+ .collect();+ assert_eq!(claimants, ["objc", "matlab"], "probe order broken");+ assert!(+ registry()+ .iter()+ .find(|d| d.name == "objc")+ .unwrap()+ .probe+ .is_some(),+ "objc row must carry the content probe"+ );+ }+#[test]fn python() {let s = syms(@@ -981,4 +1675,411 @@ mod tests {let s = syms("bash", "foo() { echo hi; }\nfunction bar { echo yo; }\n");assert!(has(&s, "foo", "function"));}++ // ── #295 fixtures: one real-world-shaped snippet per added language,+ // asserting at least one symbol of each kind its query captures. ─────++ #[test]+ fn kotlin() {+ let s = syms(+ "kotlin",+ "package demo\n\+ interface Greets { fun greet(): String }\n\+ class Greeter(val name: String) : Greets {\n\+ override fun greet(): String = \"hi $name\"\n\+ }\n\+ object Registry { fun lookup(id: Int) = id }\n\+ typealias Handler = (Int) -> Unit\n\+ fun topLevel(): Int = 42\n",+ );+ assert!(has(&s, "Greeter", "class"), "got {s:?}");+ assert!(has(&s, "Greets", "interface"), "got {s:?}");+ assert!(has(&s, "Registry", "object"), "got {s:?}");+ assert!(has(&s, "greet", "function"), "got {s:?}");+ assert!(has(&s, "topLevel", "function"), "got {s:?}");+ assert!(has(&s, "Handler", "type"), "got {s:?}");+ }++ #[test]+ fn swift() {+ let s = syms(+ "swift",+ "protocol Shape { func area() -> Double }\n\+ struct Point { var x: Double }\n\+ enum Direction { case north, south }\n\+ class Canvas {\n\+ func draw(_ p: Point) {}\n\+ }\n\+ func launch() {}\n",+ );+ assert!(has(&s, "Shape", "protocol"), "got {s:?}");+ assert!(has(&s, "Point", "struct"), "got {s:?}");+ assert!(has(&s, "Direction", "enum"), "got {s:?}");+ assert!(has(&s, "Canvas", "class"), "got {s:?}");+ assert!(has(&s, "draw", "function"), "got {s:?}");+ assert!(has(&s, "launch", "function"), "got {s:?}");+ }++ #[test]+ fn scala() {+ let s = syms(+ "scala",+ "trait Greeter { def greet(name: String): String }\n\+ class Impl extends Greeter { def greet(name: String) = s\"hi $name\" }\n\+ object Main { def run(): Unit = () }\n\+ enum Color { case Red, Green }\n\+ type Handler = String => Unit\n",+ );+ assert!(has(&s, "Greeter", "trait"), "got {s:?}");+ assert!(has(&s, "Impl", "class"), "got {s:?}");+ assert!(has(&s, "Main", "object"), "got {s:?}");+ assert!(has(&s, "greet", "function"), "got {s:?}");+ assert!(has(&s, "Color", "enum"), "got {s:?}");+ assert!(has(&s, "Red", "const"), "got {s:?}");+ assert!(has(&s, "Handler", "type"), "got {s:?}");+ }++ #[test]+ fn dart() {+ let s = syms(+ "dart",+ "class Greeter { String greet(String name) => 'hi'; }\n\+ mixin Musical { void play() {} }\n\+ enum Suit { hearts, spades }\n\+ typedef Handler = void Function(int);\n\+ int topLevel(int x) { return x; }\n",+ );+ assert!(has(&s, "Greeter", "class"), "got {s:?}");+ assert!(has(&s, "Musical", "mixin"), "got {s:?}");+ assert!(has(&s, "Suit", "enum"), "got {s:?}");+ assert!(has(&s, "hearts", "const"), "got {s:?}");+ assert!(has(&s, "Handler", "type"), "got {s:?}");+ assert!(has(&s, "topLevel", "function"), "got {s:?}");+ assert!(has(&s, "greet", "function"), "got {s:?}");+ }++ #[test]+ fn lua() {+ let s = syms(+ "lua",+ "local M = {}\n\+ function M.add(a, b) return a + b end\n\+ function M:reset() end\n\+ local helper = function(x) return x end\n\+ function standalone() end\n\+ local T = { handler = function() end }\n\+ return M\n",+ );+ assert!(has(&s, "add", "function"), "got {s:?}");+ assert!(has(&s, "reset", "method"), "got {s:?}");+ assert!(has(&s, "helper", "function"), "got {s:?}");+ assert!(has(&s, "standalone", "function"), "got {s:?}");+ assert!(has(&s, "handler", "function"), "got {s:?}");+ }++ #[test]+ fn perl() {+ let s = syms(+ "perl",+ "package My::Module;\n\+ sub new {\n my ($class) = @_;\n return bless {}, $class;\n}\n\+ sub greet { return 'hi'; }\n\+ 1;\n",+ );+ assert!(has(&s, "My::Module", "module"), "got {s:?}");+ assert!(has(&s, "new", "function"), "got {s:?}");+ assert!(has(&s, "greet", "function"), "got {s:?}");+ }++ #[test]+ fn r() {+ let s = syms(+ "r",+ "square <- function(x) {\n x^2\n}\ncube = function(x) x^3\nvalue <- 42\n",+ );+ assert!(has(&s, "square", "function"), "got {s:?}");+ assert!(has(&s, "cube", "function"), "got {s:?}");+ // Plain value assignment is not a definition.+ assert!(!has(&s, "value", "function"), "captured a value: {s:?}");+ }++ #[test]+ fn julia() {+ let s = syms(+ "julia",+ "module Geometry\n\+ struct Point\n x::Float64\nend\n\+ abstract type Shape end\n\+ struct Circle <: Shape\n r::Float64\nend\n\+ function area(c::Circle)\n 3.14 * c.r^2\nend\n\+ macro trace(ex)\n ex\nend\n\+ end\n",+ );+ assert!(has(&s, "Geometry", "module"), "got {s:?}");+ assert!(has(&s, "Point", "struct"), "got {s:?}");+ assert!(has(&s, "Circle", "struct"), "got {s:?}");+ assert!(has(&s, "Shape", "type"), "got {s:?}");+ assert!(has(&s, "area", "function"), "got {s:?}");+ assert!(has(&s, "trace", "macro"), "got {s:?}");+ }++ #[test]+ fn haskell() {+ // NOTE: single literal with explicit indentation — Haskell layout is+ // significant, and the `\`-continuation form strips leading spaces,+ // which silently promotes where-locals to top level.+ let s = syms(+ "haskell",+ "module Demo where\n\ndata Tree = Leaf | Node Tree Tree\nnewtype Wrapper = Wrapper Int\ntype Alias = Int\nclass Pretty a where\n pretty :: a -> String\ndepth :: Tree -> Int\ndepth t = go t\n where\n go Leaf = 0\n go (Node l r) = 1 + max (go l) (go r)\n",+ );+ assert!(has(&s, "Tree", "data"), "got {s:?}");+ assert!(has(&s, "Wrapper", "data"), "got {s:?}");+ assert!(has(&s, "Alias", "type"), "got {s:?}");+ assert!(has(&s, "Pretty", "class"), "got {s:?}");+ assert!(has(&s, "depth", "function"), "got {s:?}");+ // The where-bound helper is a local — the `declarations` anchor is+ // what keeps it out of `defs` (#170's trap, Haskell edition).+ assert!(!has(&s, "go", "function"), "captured a where-local: {s:?}");+ }++ #[test]+ fn elixir() {+ let s = syms(+ "elixir",+ "defmodule Server do\n\+ def start(port) do\n {:ok, port}\n end\n\+ def stop, do: :ok\n\+ defp validate(port) when is_integer(port), do: port\n\+ defmacro trace(ast), do: ast\n\+ def handle(:ping), do: :pong\n\+ IO.puts(\"not a definition\")\n\+ end\n",+ );+ assert!(has(&s, "Server", "module"), "got {s:?}");+ assert!(has(&s, "start", "function"), "got {s:?}");+ assert!(has(&s, "stop", "function"), "got {s:?}");+ assert!(has(&s, "validate", "function"), "got {s:?}");+ assert!(has(&s, "trace", "function"), "got {s:?}");+ assert!(has(&s, "handle", "function"), "got {s:?}");+ // The #any-of? predicate is what rejects ordinary calls — without+ // predicate evaluation every call in the file would be a "def".+ assert!(!has(&s, "puts", "function"), "captured a call: {s:?}");+ }++ #[test]+ fn erlang() {+ let s = syms(+ "erlang",+ "-module(server).\n\+ -record(state, {port, owner}).\n\+ -define(MAX_CONN, 100).\n\+ -define(default_port, 8080).\n\+ -type conn() :: pid().\n\+ start(Port) -> {ok, Port}.\n\+ stop() -> ok.\n",+ );+ assert!(has(&s, "server", "module"), "got {s:?}");+ assert!(has(&s, "state", "record"), "got {s:?}");+ assert!(has(&s, "MAX_CONN", "const"), "got {s:?}");+ assert!(has(&s, "default_port", "const"), "got {s:?}");+ assert!(has(&s, "conn", "type"), "got {s:?}");+ assert!(has(&s, "start", "function"), "got {s:?}");+ assert!(has(&s, "stop", "function"), "got {s:?}");+ }++ #[test]+ fn ocaml() {+ let s = syms(+ "ocaml",+ "module Geometry = struct\n let pi = 3.14\nend\n\+ type shape = Circle of float | Square of float\n\+ let area s = match s with Circle r -> r | Square w -> w\n\+ let double = fun x -> x * 2\n\+ let tau = 6.28\n\+ external now : unit -> float = \"caml_now\"\n",+ );+ assert!(has(&s, "Geometry", "module"), "got {s:?}");+ assert!(has(&s, "shape", "type"), "got {s:?}");+ assert!(has(&s, "area", "function"), "got {s:?}");+ assert!(has(&s, "double", "function"), "got {s:?}");+ assert!(has(&s, "now", "function"), "got {s:?}");+ // A bare value binding is not a function definition.+ assert!(!has(&s, "tau", "function"), "captured a value: {s:?}");+ }++ /// `.mli` parses under a DIFFERENT grammar; without its own registry row+ /// an interface file — the file that IS a module's public API — would+ /// fall back to plain text.+ #[test]+ fn ocaml_interface() {+ let s = syms(+ "ocaml_interface",+ "type shape = Circle of float\n\+ val area : shape -> float\n\+ val name : string\n\+ module type S = sig\n val id : int\nend\n",+ );+ assert!(has(&s, "shape", "type"), "got {s:?}");+ assert!(has(&s, "area", "function"), "got {s:?}");+ assert!(has(&s, "S", "interface"), "got {s:?}");+ }++ #[test]+ fn zig() {+ let s = syms(+ "zig",+ "const std = @import(\"std\");\n\+ const Point = struct {\n x: f32,\n pub fn norm(self: Point) f32 { return self.x; }\n};\n\+ const Direction = enum { north, south };\n\+ const Value = union { int: i32, float: f32 };\n\+ pub fn main() void {\n const local = 1;\n _ = local;\n}\n",+ );+ assert!(has(&s, "Point", "struct"), "got {s:?}");+ assert!(has(&s, "Direction", "enum"), "got {s:?}");+ assert!(has(&s, "Value", "union"), "got {s:?}");+ assert!(has(&s, "main", "function"), "got {s:?}");+ assert!(has(&s, "norm", "function"), "got {s:?}");+ assert!(has(&s, "std", "const"), "got {s:?}");+ // Function-local const must stay out (source_file anchor).+ assert!(!has(&s, "local", "const"), "captured a local: {s:?}");+ }++ #[test]+ fn objc() {+ let s = syms(+ "objc",+ "#import <Foundation/Foundation.h>\n\+ #define MAX_RETRIES 3\n\+ @protocol Drawable\n- (void)draw;\n@end\n\+ @interface Shape : NSObject\n- (double)area;\n@end\n\+ @implementation Shape\n\+ - (double)area { return 0; }\n\+ - (void)moveToX:(double)x y:(double)y { }\n\+ @end\n\+ static int helper(int v) { return v; }\n",+ );+ assert!(has(&s, "Shape", "class"), "got {s:?}");+ assert!(has(&s, "Drawable", "protocol"), "got {s:?}");+ assert!(has(&s, "area", "method"), "got {s:?}");+ assert!(has(&s, "moveToX", "method"), "got {s:?}");+ assert!(has(&s, "helper", "function"), "got {s:?}");+ assert!(has(&s, "MAX_RETRIES", "const"), "got {s:?}");+ }++ /// The other half of the `.m` probe: a markerless `.m` routes to MATLAB+ /// and extracts MATLAB symbols.+ #[test]+ fn matlab() {+ let s = syms("matlab", "function y = square(x)\n y = x^2;\nend\n");+ assert!(has(&s, "square", "function"), "got {s:?}");+ let c = syms(+ "matlab",+ "classdef Point\n methods\n function obj = Point()\n end\n end\nend\n",+ );+ assert!(has(&c, "Point", "class"), "got {c:?}");+ }++ #[test]+ fn groovy() {+ let s = syms(+ "groovy",+ "class Pipeline {\n\+ Pipeline() {}\n\+ def run(stage) { stage() }\n\+ }\n\+ interface Task { void execute() }\n\+ enum Phase { BUILD, TEST }\n\+ def deploy(env) { println env }\n",+ );+ assert!(has(&s, "Pipeline", "class"), "got {s:?}");+ assert!(has(&s, "Task", "interface"), "got {s:?}");+ assert!(has(&s, "Phase", "enum"), "got {s:?}");+ assert!(has(&s, "deploy", "function"), "got {s:?}");+ assert!(has(&s, "run", "method"), "got {s:?}");+ }++ #[test]+ fn powershell() {+ let s = syms(+ "powershell",+ "function Get-Widget {\n param($Name)\n $Name\n}\n\+ class Inventory {\n [int] $Count\n [void] Add([int] $n) { }\n}\n\+ enum Color { Red; Green }\n",+ );+ assert!(has(&s, "Get-Widget", "function"), "got {s:?}");+ assert!(has(&s, "Inventory", "class"), "got {s:?}");+ assert!(has(&s, "Add", "method"), "got {s:?}");+ assert!(has(&s, "Color", "enum"), "got {s:?}");+ }++ #[test]+ fn fsharp() {+ // NOTE: single literal with explicit indentation — F# layout is+ // significant (see the haskell fixture note).+ let s = syms(+ "fsharp",+ "module Demo\ntype Shape =\n | Circle of float\n | Square of float\ntype Point = { X: float; Y: float }\nlet area shape =\n let helper r = r * r\n helper 2.0\n",+ );+ assert!(has(&s, "Demo", "module"), "got {s:?}");+ assert!(has(&s, "Shape", "type"), "got {s:?}");+ assert!(has(&s, "Point", "type"), "got {s:?}");+ assert!(has(&s, "area", "function"), "got {s:?}");+ // Nested let is a local — the context anchoring keeps it out.+ assert!(!has(&s, "helper", "function"), "captured a local: {s:?}");+ }++ #[test]+ fn nix() {+ let s = syms(+ "nix",+ "{\n mkShell = pkgs: pkgs.mkShell {};\n version = \"1.0\";\n}\n",+ );+ assert!(has(&s, "mkShell", "function"), "got {s:?}");+ // A non-function binding is data, not a definition.+ assert!(!has(&s, "version", "function"), "captured data: {s:?}");+ }++ #[test]+ fn fortran() {+ let s = syms(+ "fortran",+ "module geometry\n\+ type :: point_t\n real :: x\n end type\n\+ contains\n\+ function area(r) result(a)\n real :: r, a\n a = r * r\n end function\n\+ subroutine reset(p)\n type(point_t) :: p\n end subroutine\n\+ end module\n\+ program main\n use geometry\nend program\n",+ );+ assert!(has(&s, "geometry", "module"), "got {s:?}");+ assert!(has(&s, "point_t", "type"), "got {s:?}");+ assert!(has(&s, "area", "function"), "got {s:?}");+ assert!(has(&s, "reset", "function"), "got {s:?}");+ assert!(has(&s, "main", "module"), "got {s:?}");+ }++ #[test]+ fn solidity() {+ let s = syms(+ "solidity",+ "contract Token {\n\+ struct Account { uint balance; }\n\+ enum Phase { Open, Closed }\n\+ event Transfer(address from, address to);\n\+ modifier onlyOwner() { _; }\n\+ function mint(uint amount) public { }\n\+ }\n\+ interface IToken { function total() external; }\n\+ library MathLib { function min(uint a, uint b) internal pure returns (uint) { return a; } }\n",+ );+ assert!(has(&s, "Token", "contract"), "got {s:?}");+ assert!(has(&s, "IToken", "interface"), "got {s:?}");+ assert!(has(&s, "MathLib", "library"), "got {s:?}");+ assert!(has(&s, "Account", "struct"), "got {s:?}");+ assert!(has(&s, "Phase", "enum"), "got {s:?}");+ assert!(has(&s, "Transfer", "event"), "got {s:?}");+ assert!(has(&s, "mint", "function"), "got {s:?}");+ assert!(has(&s, "onlyOwner", "function"), "got {s:?}");+ assert!(has(&s, "min", "function"), "got {s:?}");+ }}diff --git a/landing/index.html b/landing/index.htmlindex 59079ed4d..ed4cd0a99 100644--- a/landing/index.html+++ b/landing/index.html@@ -216,7 +216,7 @@ $ curl -sXPOST localhost:9200/_memory/agent-demo/_recall \above always resolve the latest. Every commit re-runs the<strong>1,365 / 1,368</strong> ES-YAML conformance suite; the changelogstates known defects instead of implying their absence. In the current- build: code arrives <strong>AST-parsed for 13 languages</strong> (symbols,+ build: code arrives <strong>AST-parsed for 34 languages</strong> (symbols,definitions, line numbers), index lifecycle policies <strong>execute</strong>(ILM / ISM), <code class="inline">.gitignore</code> is honoured onautoindex, and dense-vector kNN is HNSW-served with exact rescoring.@@ -289,7 +289,7 @@ $ curl -sXPOST localhost:9200/_memory/agent-demo/_recall \<div class="row"><div class="k">AST code search</div>- <div class="v">Source is parsed with tree-sitter (13 languages) into symbols with kind and line number plus a searchable definitions field — the agent retrieves a <em>function with its contract</em>, not a grep line that needs the whole file opened to judge</div>+ <div class="v">Source is parsed with tree-sitter (34 languages) into symbols with kind and line number plus a searchable definitions field — the agent retrieves a <em>function with its contract</em>, not a grep line that needs the whole file opened to judge</div></div><div class="row"><div class="k">Token economics</div>