Skip to content

Local code intelligence for developers and AI agents. Connect code, docs and infrastructure in a persistent graph, with CBM Lens, MCP tools, agent integrations and change-aware test selection.

fix(grammars): the WIT scanner's create function returns NULL

Martin Vogel02c0288ff309
diff --git a/internal/cbm/vendored/grammars/MANIFEST.md b/internal/cbm/vendored/grammars/MANIFEST.md
index 3ee1344a5..889306a51 100644
--- a/internal/cbm/vendored/grammars/MANIFEST.md
+++ b/internal/cbm/vendored/grammars/MANIFEST.md
@@ -111,6 +111,7 @@ row instead.
| purescript | `purescript/scanner.c`, serialize | guard `memcpy(buffer, indents->data, to_copy)` with `if (to_copy > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source when the indent vector is empty (formal UB, harmless) |
| plsql | `plsql/parser.c`, include | `#include <tree_sitter/parser.h>` → `#include "tree_sitter/parser.h"` | The older ABI-14 generator emits angle brackets; every other vendored grammar uses the quoted form, which resolves the per-grammar `tree_sitter/` header from the including file's directory |
| properties | `properties/scanner.c`, whole file | move the `reached_eof` flag out of a file-scope `static bool` and into a per-parser payload allocated by `..._external_scanner_create()` | **Data race, and the only one of its kind in 103 vendored scanners.** That flag is how the scanner refuses to emit a second end-of-input `FAKE_EOL`, and upstream keeps it in ONE process-wide object. With several worker threads indexing `.properties` files at once, whichever reached EOF first set the flag, so another thread's `!reached_eof` was false and its parse never received the `FAKE_EOL` it needed to finish — it sat at end-of-input asking for a token that would never arrive. Measured 2026-09-19: ~106 M parse operations on a 253-byte file before the per-file budget stopped it, after which the file was dropped from the graph and java/kotlin indexes differed run to run. Interleaved A/B under identical load, 40 runs each: unpatched 6 stalls, patched 0. The serialize/deserialize wire format is unchanged (the state is still the returned length). Upstream `6310671b24d4` still carries the static, so a re-vendor must re-apply this |
+| wit | `wit/scanner.c`, `..._external_scanner_create` | `{}` → `{ return NULL; }` | **Uninitialized read (MSan).** The function returns `void *` but had no `return`, and tree-sitter's runtime stores the result as the scanner payload, so every WIT parse used an indeterminate value (C11 6.9.1p12 makes that undefined). The scanner never reads its payload, so ordinary builds were unaffected, and the vendored build's `-w` hid `-Wreturn-type`. Under MSan the report of this read crashed and posed as a "stack-overflow", which is why five grammar/pipeline suites were excluded from the MSan lane. Upstream bytecodealliance/tree-sitter-wit still carries it at `059bfdd1af72` (2025-06-23, the last change to `src/scanner.c`) |
| swift | `swift/scanner.c`, `OP_SYMBOL_SUPPRESSOR` + `eat_operators` | `1UL <<` / `1 <<` → `1ULL <<` | UBSan: `1 << suppressor` shifts an `int` by up to `TOKEN_COUNT` bits, undefined once the index reaches 31, while the mask it feeds is `uint64_t`. `1UL << FAKE_TRY_BANG` is the same defect on Windows, where `unsigned long` is 32 bits and `FAKE_TRY_BANG` is 32; the CLANGARM64 leg runs UBSan in trap mode, so there it is an illegal instruction rather than a log line. Upstream already carries both changes: `fb63a7004f07` (2026-04-06, upstream #558) for `eat_operators`, `6ab8d1d74ebd` (2026-08-10) for the `OP_SYMBOL_SUPPRESSOR` entry. Our pin `8abb3e8b3325` (2026-03-20) predates both, so this is a backport rather than a local invention — a re-vendor past 2026-08-10 should delete this row, not re-apply it |
## Vendored from verified upstream
diff --git a/internal/cbm/vendored/grammars/wit/scanner.c b/internal/cbm/vendored/grammars/wit/scanner.c
index 829611a7b..3a98cbf67 100644
--- a/internal/cbm/vendored/grammars/wit/scanner.c
+++ b/internal/cbm/vendored/grammars/wit/scanner.c
@@ -10,7 +10,7 @@ enum TokenType {
LINE_DOC_CONTENT,
};
-void *tree_sitter_wit_external_scanner_create(void) {}
+void *tree_sitter_wit_external_scanner_create(void) { return NULL; }
void tree_sitter_wit_external_scanner_destroy(void *payload) {}
diff --git a/scripts/vendored-checksums.txt b/scripts/vendored-checksums.txt
index 224b6cd80..a34a5eacf 100644
--- a/scripts/vendored-checksums.txt
+++ b/scripts/vendored-checksums.txt
@@ -5,7 +5,7 @@ c5cfb43042b6b72045f4ba997834d0a7786d2793d91680868b5815b39f14fc78 internal/cbm/v
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/common/tree_sitter/alloc.h
31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c internal/cbm/vendored/common/tree_sitter/array.h
180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2 internal/cbm/vendored/common/tree_sitter/parser.h
-44bade69b14ac382b08835965eeec5f0fd6e54ccb7a80da21c0161128a10503f internal/cbm/vendored/grammars/MANIFEST.md
+57006171adb332d760ede4b413d900b2bbb57f55d3bf94ceecb6e74a720bc722 internal/cbm/vendored/grammars/MANIFEST.md
ad8425038de519f8c4e3e9339feebf99dfad8a6002dcf79227d91402780a32cc internal/cbm/vendored/grammars/ada/LICENSE
02805ec13939b749c891567be36bf024b09034e04c80683a1ae667272458d549 internal/cbm/vendored/grammars/ada/parser.c
115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db internal/cbm/vendored/grammars/ada/tree_sitter/alloc.h
@@ -899,7 +899,7 @@ d1b7f7ca8ec73d915cec4aa64d9e8b7f2d89b874225d4532e00d9c36c98528fe internal/cbm/v
df898ca6d3eb880b9fdf7d3179363d7657abcd4b0a0565d8fd6b47215d8cc149 internal/cbm/vendored/grammars/wgsl/tree_sitter/parser.h
3b26cd02b9d9bf324bce59edd88c02cb93fa9a908528b805264db8cde53295e6 internal/cbm/vendored/grammars/wit/LICENSE
d2c4cab0df0ba3cc692ae0731de4c644ad41114fcfd5b5a7a8ae669df39f8434 internal/cbm/vendored/grammars/wit/parser.c
-9749b186b3b7620995d9a9b747d34c39949fc497c20d7b79391799014bad9470 internal/cbm/vendored/grammars/wit/scanner.c
+0ec9697a583cf96af605af0f9a7a660e01e42c81cae847580bb2a44dbbbde8a7 internal/cbm/vendored/grammars/wit/scanner.c
115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db internal/cbm/vendored/grammars/wit/tree_sitter/alloc.h
db7d66e4305e3f37b5d31a3291acc01fed95054474a75b3c7cff72048ef9b797 internal/cbm/vendored/grammars/wit/tree_sitter/array.h
c4b9482069f61a2a26a590181baa34f08eb6dcc57a138def578e6f9720c35ea5 internal/cbm/vendored/grammars/wit/tree_sitter/parser.h