Local code intelligence for developers and AI agents. Connect code, docs and infrastructure in a persistent graph, with CBM Lens, MCP tools, agent integrations and change-aware test selection.
C aiderastclaude-codecode-analysiscode-intelligencecodexcursorcypherdeveloper-toolsgemini-cligraph-visualizationkilocodeknowledge-graphmcpmcp-servermodel-context-protocolopencodesqlitetree-sitterwindsurf
fix(grammar): give the properties scanner per-parser state
Martin Vogelc7ccb479b454
diff --git a/internal/cbm/vendored/grammars/MANIFEST.md b/internal/cbm/vendored/grammars/MANIFEST.mdindex 4025b5f50..74322763e 100644--- a/internal/cbm/vendored/grammars/MANIFEST.md+++ b/internal/cbm/vendored/grammars/MANIFEST.md@@ -87,6 +87,7 @@ row instead.| rescript | `rescript/scanner.c`, deserialize | guard `memcpy(state, buffer, n_bytes)` with `if (n_bytes > 0)` | UBSan: zero-length `memcpy` with a NULL `buffer` / `n_bytes == 0` on empty-state deserialize (formal UB, harmless). The sibling serialize copies a fixed `sizeof(ScannerState)` (always > 0, non-NULL src) and needs no guard. || purescript | `purescript/scanner.c`, serialize | guard `memcpy(buffer, indents->data, to_copy)` with `if (to_copy > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source when the indent vector is empty (formal UB, harmless) || plsql | `plsql/parser.c`, include | `#include <tree_sitter/parser.h>` → `#include "tree_sitter/parser.h"` | The older ABI-14 generator emits angle brackets; every other vendored grammar uses the quoted form, which resolves the per-grammar `tree_sitter/` header from the including file's directory |+| properties | `properties/scanner.c`, whole file | move the `reached_eof` flag out of a file-scope `static bool` and into a per-parser payload allocated by `..._external_scanner_create()` | **Data race, and the only one of its kind in 103 vendored scanners.** That flag is how the scanner refuses to emit a second end-of-input `FAKE_EOL`, and upstream keeps it in ONE process-wide object. With several worker threads indexing `.properties` files at once, whichever reached EOF first set the flag, so another thread's `!reached_eof` was false and its parse never received the `FAKE_EOL` it needed to finish — it sat at end-of-input asking for a token that would never arrive. Measured 2026-09-19: ~106 M parse operations on a 253-byte file before the per-file budget stopped it, after which the file was dropped from the graph and java/kotlin indexes differed run to run. Interleaved A/B under identical load, 40 runs each: unpatched 6 stalls, patched 0. The serialize/deserialize wire format is unchanged (the state is still the returned length). Upstream `6310671b24d4` still carries the static, so a re-vendor must re-apply this || swift | `swift/scanner.c`, `OP_SYMBOL_SUPPRESSOR` + `eat_operators` | `1UL <<` / `1 <<` → `1ULL <<` | UBSan: `1 << suppressor` shifts an `int` by up to `TOKEN_COUNT` bits, undefined once the index reaches 31, while the mask it feeds is `uint64_t`. `1UL << FAKE_TRY_BANG` is the same defect on Windows, where `unsigned long` is 32 bits and `FAKE_TRY_BANG` is 32; the CLANGARM64 leg runs UBSan in trap mode, so there it is an illegal instruction rather than a log line. Upstream already carries both changes: `fb63a7004f07` (2026-04-06, upstream #558) for `eat_operators`, `6ab8d1d74ebd` (2026-08-10) for the `OP_SYMBOL_SUPPRESSOR` entry. Our pin `8abb3e8b3325` (2026-03-20) predates both, so this is a backport rather than a local invention — a re-vendor past 2026-08-10 should delete this row, not re-apply it |## Vendored from verified upstreamdiff --git a/internal/cbm/vendored/grammars/properties/scanner.c b/internal/cbm/vendored/grammars/properties/scanner.cindex 583456268..5ef7b71b0 100644--- a/internal/cbm/vendored/grammars/properties/scanner.c+++ b/internal/cbm/vendored/grammars/properties/scanner.c@@ -1,5 +1,7 @@#include "tree_sitter/parser.h"+#include <stdlib.h>+#ifdef _MSC_VER#pragma warning(disable : 4100)#elif defined(__GNUC__) || defined(__clang__)@@ -8,21 +10,43 @@enum TokenType { FAKE_EOL };-static bool reached_eof = false;--bool tree_sitter_properties_external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {+/* The scanner's one bit of state -- whether THIS parse has already emitted the+ * end-of-input FAKE_EOL -- lives in the payload, one per parser, which is what+ * tree-sitter's create/destroy contract is for.+ *+ * It used to be a file-scope `static bool`: one object for the entire process,+ * shared by every thread. The grammar needs exactly one FAKE_EOL to close a+ * file, and `!reached_eof` is how the scanner refuses to emit a second one. So+ * when two threads indexed .properties files at the same time, whichever+ * reached EOF first set the flag, and the other thread's `!reached_eof` was+ * false -- it never received the FAKE_EOL its parse was waiting for and sat at+ * end-of-input asking for a token that would never arrive. Measured on+ * 2026-09-19: ~106 million parse operations on a 253-byte file before the+ * per-file budget stopped it, after which the file was dropped from the graph+ * and the index differed from run to run. */+typedef struct {+ bool reached_eof;+} Scanner;++bool tree_sitter_properties_external_scanner_scan(void *payload, TSLexer *lexer,+ const bool *valid_symbols) {+ Scanner *scanner = (Scanner *)payload;lexer->result_symbol = FAKE_EOL;- return reached_eof = !reached_eof && valid_symbols[FAKE_EOL] && lexer->eof(lexer);+ return scanner->reached_eof =+ !scanner->reached_eof && valid_symbols[FAKE_EOL] && lexer->eof(lexer);}+/* Wire format unchanged: the state IS the returned length (1 = already emitted,+ * 0 = not), and no bytes are written to the buffer. */unsigned tree_sitter_properties_external_scanner_serialize(void *payload, char *buffer) {- return reached_eof;+ return ((Scanner *)payload)->reached_eof;}-void tree_sitter_properties_external_scanner_deserialize(void *payload, const char *buffer, unsigned length) {- reached_eof = length;+void tree_sitter_properties_external_scanner_deserialize(void *payload, const char *buffer,+ unsigned length) {+ ((Scanner *)payload)->reached_eof = length;}-void *tree_sitter_properties_external_scanner_create() { return NULL; }+void *tree_sitter_properties_external_scanner_create() { return calloc(1, sizeof(Scanner)); }-void tree_sitter_properties_external_scanner_destroy(void *payload) { }\ No newline at end of file+void tree_sitter_properties_external_scanner_destroy(void *payload) { free(payload); }diff --git a/scripts/vendored-checksums.txt b/scripts/vendored-checksums.txtindex c4b809dc6..092f46f2e 100644--- a/scripts/vendored-checksums.txt+++ b/scripts/vendored-checksums.txt@@ -610,7 +610,7 @@ fbce36bb615a015ea545373da62c178fe9e7e3a7b7de656455738e6f671aa652 internal/cbm/vc4b9482069f61a2a26a590181baa34f08eb6dcc57a138def578e6f9720c35ea5 internal/cbm/vendored/grammars/prisma/tree_sitter/parser.h3297073719030b7261afd78b7267931ccc03a27f7a65ca6fc644316180f71bf6 internal/cbm/vendored/grammars/properties/LICENSE98e06c3a7b3043d7641ec0b2c28c9f66e54341e343e1f553f994ae5a6f0c0452 internal/cbm/vendored/grammars/properties/parser.c-3c00e7a8ec38662b05cb00858d2959b123b2b5c783eb3dc25c80b7577b4c27ac internal/cbm/vendored/grammars/properties/scanner.c+a313549615512cf06b2ee55b080bfee1559860aaa1dfcab56e9af6010323aad8 internal/cbm/vendored/grammars/properties/scanner.c115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db internal/cbm/vendored/grammars/properties/tree_sitter/alloc.h02d139d44303b0b0b85a899839dbc1346c478e7318d1f2971fad2b38fd57cb73 internal/cbm/vendored/grammars/properties/tree_sitter/array.hc4b9482069f61a2a26a590181baa34f08eb6dcc57a138def578e6f9720c35ea5 internal/cbm/vendored/grammars/properties/tree_sitter/parser.h