Local code intelligence for developers and AI agents. Connect code, docs and infrastructure in a persistent graph, with CBM Lens, MCP tools, agent integrations and change-aware test selection.
C aiderastclaude-codecode-analysiscode-intelligencecodexcursorcypherdeveloper-toolsgemini-cligraph-visualizationkilocodeknowledge-graphmcpmcp-servermodel-context-protocolopencodesqlitetree-sitterwindsurf
doc-links(pdf): notices for the PDF tables; the Unicode table generator
Martin Vogelc1da1924b2b9
diff --git a/THIRD_PARTY.md b/THIRD_PARTY.mdindex c23f0ec4d..be8e2fea1 100644--- a/THIRD_PARTY.md+++ b/THIRD_PARTY.md@@ -138,6 +138,109 @@ Semantic vector search uses static token embeddings derived from theSee `vendored/nomic/NOTICE` for the exact derivation procedure(per-token inference + int8 quantization via `scripts/extract_nomic_vectors.py`).+## Embedded Character Data (PDF text layer)++The PDF text-layer extractor (`internal/cbm/pdf/`) carries static tables+derived from the following sources. No upstream source code is included.++- **Adobe Glyph List** — the glyph-name subset in `internal/cbm/pdf/pdf_tables.c`+ (the names of the standard Latin encodings and common symbols, mapped to+ Unicode) follows [adobe-type-tools/agl-aglfn](https://github.com/adobe-type-tools/agl-aglfn)+ `glyphlist.txt`.+ - **License:** BSD-3-Clause+ - **Copyright:** Copyright 2002-2019 Adobe (http://www.adobe.com/).+- **Adobe core-14 font metrics** — the advance widths of the printable ASCII+ range of Helvetica and Times-Roman in `pdf_tables.c` (used only for a+ standard-14 font that omits its `/Widths`) are the values of Adobe's+ core-14 AFM files. The AFM files themselves are not distributed.+ - **Copyright:** Copyright (c) 1985, 1987, 1989, 1990, 1997 Adobe Systems+ Incorporated. All Rights Reserved.+- **Unicode Character Database** — `internal/cbm/pdf/pdf_unicode.c` (NFKC of+ each code point, canonical compositions, character classes) is generated+ by `scripts/gen-pdf-unicode.py` from CPython's `unicodedata` (Unicode+ 16.0.0).+ - **License:** Unicode License v3 (text below)+ - **Copyright:** Copyright © 1991-2026 Unicode, Inc.+- The base encodings (Standard, WinAnsi, MacRoman, Symbol) are those of+ ISO 32000-1, Annex D.++Adobe Glyph List license (BSD-3-Clause):++```+Copyright 2002-2019 Adobe (http://www.adobe.com/).++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are+met:++Redistributions of source code must retain the above copyright notice,+this list of conditions and the following disclaimer.++Redistributions in binary form must reproduce the above copyright+notice, this list of conditions and the following disclaimer in the+documentation and/or other materials provided with the distribution.++Neither the name of Adobe nor the names of its contributors may be+used to endorse or promote products derived from this software without+specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.+```++Unicode License v3:++```+UNICODE LICENSE V3++COPYRIGHT AND PERMISSION NOTICE++Copyright © 1991-2026 Unicode, Inc.++NOTICE TO USER: Carefully read the following legal agreement. BY+DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR+SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE+TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT+DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.++Permission is hereby granted, free of charge, to any person obtaining a+copy of data files and any associated documentation (the "Data Files") or+software and any associated documentation (the "Software") to deal in the+Data Files or Software without restriction, including without limitation+the rights to use, copy, modify, merge, publish, distribute, and/or sell+copies of the Data Files or Software, and to permit persons to whom the+Data Files or Software are furnished to do so, provided that either (a)+this copyright and permission notice appear with all copies of the Data+Files or Software, or (b) this copyright and permission notice appear in+associated Documentation.++THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY+KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF+THIRD PARTY RIGHTS.++IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE+BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,+OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,+WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,+ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA+FILES OR SOFTWARE.++Except as contained in this notice, the name of a copyright holder shall+not be used in advertising or otherwise to promote the sale, use or other+dealings in these Data Files or Software without prior written+authorization of the copyright holder.+```+## Hybrid LSP — Reference Language ServersThe Hybrid LSP layer (`internal/cbm/lsp/`) is an original C implementationdiff --git a/internal/cbm/pdf/pdf_tables.c b/internal/cbm/pdf/pdf_tables.cindex a892abf01..3d4256052 100644--- a/internal/cbm/pdf/pdf_tables.c+++ b/internal/cbm/pdf/pdf_tables.c@@ -4,8 +4,13 @@* Glyph names (the Adobe Glyph List subset every standard Latin encoding* uses, common TeX and symbol names, and the accented Latin letters),* the base encodings of ISO 32000-1 Annex D as code -> UTF-8, and- * approximate standard-14 widths. Generated from the field-tested tables;- * do not edit by hand.+ * approximate standard-14 widths. Generated once from the field-tested+ * prototype's tables; maintained data from here on (edit by hand, keep the+ * sources). Sources and notices (THIRD_PARTY.md): the Adobe Glyph List,+ * Copyright 2002-2019 Adobe, BSD-3-Clause; the printable-ASCII advance+ * widths of Helvetica and Times-Roman from Adobe's core-14 font metrics,+ * Copyright (c) 1985, 1987, 1989, 1990, 1997 Adobe Systems Incorporated;+ * ISO 32000-1 Annex D for the encodings.*/#include "pdf_internal.h"diff --git a/internal/cbm/pdf/pdf_unicode.c b/internal/cbm/pdf/pdf_unicode.cindex 4ae6af9b4..88ccaacb7 100644--- a/internal/cbm/pdf/pdf_unicode.c+++ b/internal/cbm/pdf/pdf_unicode.c@@ -5,7 +5,9 @@* CPython 3.14.6 answers: NFKC of each code point, the canonical compositions,* str.isspace, the re module's \w (str.isalnum or '_'), str.isalpha, and the* one-character str.isupper / str.islower / titlecase. Generated by- * gen_pdf_unicode.py; do not edit by hand.+ * scripts/gen-pdf-unicode.py; do not edit by hand. Data of the Unicode+ * Character Database, (c) Unicode, Inc., Unicode License v3+ * (THIRD_PARTY.md).*/#include "pdf_unicode.h"diff --git a/scripts/ci/generate-sbom.py b/scripts/ci/generate-sbom.pyindex 31f333c14..ff2e97efd 100644--- a/scripts/ci/generate-sbom.py+++ b/scripts/ci/generate-sbom.py@@ -49,6 +49,9 @@ sbom = {{"SPDXID": "SPDXRef-Package-simplecpp", "name": "simplecpp", "versionInfo": "1.x", "licenseDeclared": "0BSD", "downloadLocation": "https://github.com/danmar/simplecpp", "filesAnalyzed": False},{"SPDXID": "SPDXRef-Package-verstable", "name": "verstable", "versionInfo": "2.2.1", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/JacksonAllan/Verstable", "filesAnalyzed": False},{"SPDXID": "SPDXRef-Package-wyhash", "name": "wyhash", "versionInfo": "final-4.3", "licenseDeclared": "Unlicense", "downloadLocation": "https://github.com/wangyi-fudan/wyhash", "filesAnalyzed": False},+ {"SPDXID": "SPDXRef-Package-adobe-glyph-list", "name": "adobe-glyph-list", "versionInfo": "2.0", "licenseDeclared": "BSD-3-Clause", "downloadLocation": "https://github.com/adobe-type-tools/agl-aglfn", "filesAnalyzed": False, "comment": "Glyph-name subset in internal/cbm/pdf/pdf_tables.c (PDF text layer); see THIRD_PARTY.md"},+ {"SPDXID": "SPDXRef-Package-adobe-core14-metrics", "name": "adobe-core14-afm-metrics", "versionInfo": "1997", "licenseDeclared": "NOASSERTION", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": "Printable-ASCII advance widths of Helvetica and Times-Roman in internal/cbm/pdf/pdf_tables.c; (c) 1985-1997 Adobe Systems Incorporated; the AFM files are not distributed; see THIRD_PARTY.md"},+ {"SPDXID": "SPDXRef-Package-unicode-ucd", "name": "unicode-character-database", "versionInfo": "16.0.0", "licenseDeclared": "Unicode-3.0", "downloadLocation": "https://www.unicode.org/Public/16.0.0/ucd/", "filesAnalyzed": False, "comment": "NFKC, compositions and character classes in internal/cbm/pdf/pdf_unicode.c, generated by scripts/gen-pdf-unicode.py; see THIRD_PARTY.md"},{"SPDXID": "SPDXRef-Package-nomic-embed-code", "name": "nomic-embed-code-token-embeddings", "versionInfo": "1.0", "licenseDeclared": "Apache-2.0", "downloadLocation": "https://huggingface.co/nomic-ai/nomic-embed-code", "filesAnalyzed": False, "comment": "Derived int8 token embeddings; see vendored/nomic/NOTICE"},{"SPDXID": "SPDXRef-Package-tree-sitter-grammars", "name": "tree-sitter-grammars-aggregate", "versionInfo": f"{n_grammars}-grammars", "licenseDeclared": "MIT", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": f"Aggregate of {n_grammars} vendored tree-sitter grammars, compiled statically. Predominantly MIT; non-MIT families present include CC0-1.0 (clojure, fennel), Apache-2.0 (elixir, erlang, gleam, hcl, ini, jinja2, just, pkl, sway, wit), ISC (pine, templ), Unlicense (fish); first-party grammars (c) DeusData, MIT. This summary is non-exhaustive; the authoritative complete per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md, and full license texts ship in THIRD_PARTY_NOTICES.md inside each archive."},],diff --git a/scripts/gen-pdf-unicode.py b/scripts/gen-pdf-unicode.pynew file mode 100644index 000000000..d31417e2c--- /dev/null+++ b/scripts/gen-pdf-unicode.py@@ -0,0 +1,103 @@+#!/usr/bin/env python3+"""Generate internal/cbm/pdf/pdf_unicode.c: the Unicode facts the PDF mention+scanner needs, exactly as CPython sees them (the field test's scanner ran on+CPython's str and re).++usage: scripts/gen-pdf-unicode.py internal/cbm/pdf/pdf_unicode.c (then clang-format -i it)+The output depends on the CPython version (its Unicode data): the committed+tables were made with CPython 3.14.6 (Unicode 16.0.0).+"""+import sys+import unicodedata++MAXCP = 0x110000+++def is_surrogate(c):+ return 0xD800 <= c <= 0xDFFF+++def ranges(pred):+ out = []+ start = None+ for c in range(MAXCP):+ ok = not is_surrogate(c) and pred(chr(c))+ if ok and start is None:+ start = c+ elif not ok and start is not None:+ out.append((start, c - 1))+ start = None+ if start is not None:+ out.append((start, MAXCP - 1))+ return out+++def emit_ranges(f, name, rs):+ f.write("const pdf_urange_t %s[] = {\n" % name)+ for i in range(0, len(rs), 4):+ f.write(" " + " ".join("{0x%X, 0x%X}," % r for r in rs[i:i + 4]) + "\n")+ f.write("};\nconst int %s_COUNT = (int)(sizeof(%s) / sizeof(%s[0]));\n\n" % (name, name, name))+++def main():+ out = sys.argv[1]+ # per-character NFKC+ nfkc = []+ seq = []+ for c in range(MAXCP):+ if is_surrogate(c):+ continue+ ch = chr(c)+ m = unicodedata.normalize("NFKC", ch)+ if m != ch:+ nfkc.append((c, len(seq), len(m)))+ seq.extend(ord(x) for x in m)+ # canonical composition pairs: a + b -> c when NFC composes them+ comp = []+ for c in range(MAXCP):+ if is_surrogate(c):+ continue+ d = unicodedata.decomposition(chr(c))+ if not d or d.startswith("<"):+ continue+ parts = [int(x, 16) for x in d.split()]+ if len(parts) != 2:+ continue+ a, b = parts+ if unicodedata.normalize("NFC", chr(a) + chr(b)) == chr(c):+ comp.append((a, b, c))+ comp.sort()+ with open(out, "w", encoding="utf-8") as f:+ f.write("/*\n * pdf_unicode.c -- Unicode %s facts for the PDF mention scanner (generated).\n"+ " *\n * The field test's scanner ran on CPython's str and re; these tables are what\n"+ " * CPython %s answers: NFKC of each code point, the canonical compositions,\n"+ " * str.isspace, the re module's \\w (str.isalnum or '_'), str.isalpha, and the\n"+ " * one-character str.isupper / str.islower / titlecase. Generated by\n"+ " * scripts/gen-pdf-unicode.py; do not edit by hand. Data of the Unicode\n"+ " * Character Database, (c) Unicode, Inc., Unicode License v3\n"+ " * (THIRD_PARTY.md).\n */\n"+ % (unicodedata.unidata_version, sys.version.split()[0]))+ f.write("#include \"pdf_unicode.h\"\n\n")+ f.write("const pdf_nfkc_t PDF_NFKC[] = {\n")+ for i in range(0, len(nfkc), 4):+ f.write(" " + " ".join("{0x%X, %d, %d}," % e for e in nfkc[i:i + 4]) + "\n")+ f.write("};\nconst int PDF_NFKC_COUNT = (int)(sizeof(PDF_NFKC) / sizeof(PDF_NFKC[0]));\n\n")+ f.write("const uint32_t PDF_NFKC_SEQ[] = {\n")+ for i in range(0, len(seq), 10):+ f.write(" " + " ".join("0x%X," % x for x in seq[i:i + 10]) + "\n")+ f.write("};\n\n")+ f.write("const pdf_ucomp_t PDF_UCOMP[] = {\n")+ for i in range(0, len(comp), 3):+ f.write(" " + " ".join("{0x%X, 0x%X, 0x%X}," % e for e in comp[i:i + 3]) + "\n")+ f.write("};\nconst int PDF_UCOMP_COUNT = (int)(sizeof(PDF_UCOMP) / sizeof(PDF_UCOMP[0]));\n\n")+ emit_ranges(f, "PDF_U_SPACE", ranges(lambda c: c.isspace()))+ emit_ranges(f, "PDF_U_WORD", ranges(lambda c: c.isalnum() or c == "_"))+ emit_ranges(f, "PDF_U_ALPHA", ranges(lambda c: c.isalpha()))+ emit_ranges(f, "PDF_U_UPPER", ranges(lambda c: c.isupper()))+ emit_ranges(f, "PDF_U_LOWER", ranges(lambda c: c.islower()))+ emit_ranges(f, "PDF_U_TITLE", ranges(lambda c: c.istitle() and not c.isupper()))+ print("nfkc", len(nfkc), "seq", len(seq), "comp", len(comp))+++if __name__ == "__main__":+ main()