Skip to content

Local code intelligence for developers and AI agents. Connect code, docs and infrastructure in a persistent graph, with CBM Lens, MCP tools, agent integrations and change-aware test selection.

doc-links(pdf): notices for the PDF tables; the Unicode table generator

Martin Vogelc1da1924b2b9
diff --git a/THIRD_PARTY.md b/THIRD_PARTY.md
index c23f0ec4d..be8e2fea1 100644
--- a/THIRD_PARTY.md
+++ b/THIRD_PARTY.md
@@ -138,6 +138,109 @@ Semantic vector search uses static token embeddings derived from the
See `vendored/nomic/NOTICE` for the exact derivation procedure
(per-token inference + int8 quantization via `scripts/extract_nomic_vectors.py`).
+## Embedded Character Data (PDF text layer)
+
+The PDF text-layer extractor (`internal/cbm/pdf/`) carries static tables
+derived from the following sources. No upstream source code is included.
+
+- **Adobe Glyph List** — the glyph-name subset in `internal/cbm/pdf/pdf_tables.c`
+ (the names of the standard Latin encodings and common symbols, mapped to
+ Unicode) follows [adobe-type-tools/agl-aglfn](https://github.com/adobe-type-tools/agl-aglfn)
+ `glyphlist.txt`.
+ - **License:** BSD-3-Clause
+ - **Copyright:** Copyright 2002-2019 Adobe (http://www.adobe.com/).
+- **Adobe core-14 font metrics** — the advance widths of the printable ASCII
+ range of Helvetica and Times-Roman in `pdf_tables.c` (used only for a
+ standard-14 font that omits its `/Widths`) are the values of Adobe's
+ core-14 AFM files. The AFM files themselves are not distributed.
+ - **Copyright:** Copyright (c) 1985, 1987, 1989, 1990, 1997 Adobe Systems
+ Incorporated. All Rights Reserved.
+- **Unicode Character Database** — `internal/cbm/pdf/pdf_unicode.c` (NFKC of
+ each code point, canonical compositions, character classes) is generated
+ by `scripts/gen-pdf-unicode.py` from CPython's `unicodedata` (Unicode
+ 16.0.0).
+ - **License:** Unicode License v3 (text below)
+ - **Copyright:** Copyright © 1991-2026 Unicode, Inc.
+- The base encodings (Standard, WinAnsi, MacRoman, Symbol) are those of
+ ISO 32000-1, Annex D.
+
+Adobe Glyph List license (BSD-3-Clause):
+
+```
+Copyright 2002-2019 Adobe (http://www.adobe.com/).
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+Redistributions of source code must retain the above copyright notice,
+this list of conditions and the following disclaimer.
+
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+Neither the name of Adobe nor the names of its contributors may be
+used to endorse or promote products derived from this software without
+specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+```
+
+Unicode License v3:
+
+```
+UNICODE LICENSE V3
+
+COPYRIGHT AND PERMISSION NOTICE
+
+Copyright © 1991-2026 Unicode, Inc.
+
+NOTICE TO USER: Carefully read the following legal agreement. BY
+DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
+SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
+TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
+DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of data files and any associated documentation (the "Data Files") or
+software and any associated documentation (the "Software") to deal in the
+Data Files or Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, and/or sell
+copies of the Data Files or Software, and to permit persons to whom the
+Data Files or Software are furnished to do so, provided that either (a)
+this copyright and permission notice appear with all copies of the Data
+Files or Software, or (b) this copyright and permission notice appear in
+associated Documentation.
+
+THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+THIRD PARTY RIGHTS.
+
+IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
+BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
+OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
+WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
+ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
+FILES OR SOFTWARE.
+
+Except as contained in this notice, the name of a copyright holder shall
+not be used in advertising or otherwise to promote the sale, use or other
+dealings in these Data Files or Software without prior written
+authorization of the copyright holder.
+```
+
## Hybrid LSP — Reference Language Servers
The Hybrid LSP layer (`internal/cbm/lsp/`) is an original C implementation
diff --git a/internal/cbm/pdf/pdf_tables.c b/internal/cbm/pdf/pdf_tables.c
index a892abf01..3d4256052 100644
--- a/internal/cbm/pdf/pdf_tables.c
+++ b/internal/cbm/pdf/pdf_tables.c
@@ -4,8 +4,13 @@
* Glyph names (the Adobe Glyph List subset every standard Latin encoding
* uses, common TeX and symbol names, and the accented Latin letters),
* the base encodings of ISO 32000-1 Annex D as code -> UTF-8, and
- * approximate standard-14 widths. Generated from the field-tested tables;
- * do not edit by hand.
+ * approximate standard-14 widths. Generated once from the field-tested
+ * prototype's tables; maintained data from here on (edit by hand, keep the
+ * sources). Sources and notices (THIRD_PARTY.md): the Adobe Glyph List,
+ * Copyright 2002-2019 Adobe, BSD-3-Clause; the printable-ASCII advance
+ * widths of Helvetica and Times-Roman from Adobe's core-14 font metrics,
+ * Copyright (c) 1985, 1987, 1989, 1990, 1997 Adobe Systems Incorporated;
+ * ISO 32000-1 Annex D for the encodings.
*/
#include "pdf_internal.h"
diff --git a/internal/cbm/pdf/pdf_unicode.c b/internal/cbm/pdf/pdf_unicode.c
index 4ae6af9b4..88ccaacb7 100644
--- a/internal/cbm/pdf/pdf_unicode.c
+++ b/internal/cbm/pdf/pdf_unicode.c
@@ -5,7 +5,9 @@
* CPython 3.14.6 answers: NFKC of each code point, the canonical compositions,
* str.isspace, the re module's \w (str.isalnum or '_'), str.isalpha, and the
* one-character str.isupper / str.islower / titlecase. Generated by
- * gen_pdf_unicode.py; do not edit by hand.
+ * scripts/gen-pdf-unicode.py; do not edit by hand. Data of the Unicode
+ * Character Database, (c) Unicode, Inc., Unicode License v3
+ * (THIRD_PARTY.md).
*/
#include "pdf_unicode.h"
diff --git a/scripts/ci/generate-sbom.py b/scripts/ci/generate-sbom.py
index 31f333c14..ff2e97efd 100644
--- a/scripts/ci/generate-sbom.py
+++ b/scripts/ci/generate-sbom.py
@@ -49,6 +49,9 @@ sbom = {
{"SPDXID": "SPDXRef-Package-simplecpp", "name": "simplecpp", "versionInfo": "1.x", "licenseDeclared": "0BSD", "downloadLocation": "https://github.com/danmar/simplecpp", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-verstable", "name": "verstable", "versionInfo": "2.2.1", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/JacksonAllan/Verstable", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-wyhash", "name": "wyhash", "versionInfo": "final-4.3", "licenseDeclared": "Unlicense", "downloadLocation": "https://github.com/wangyi-fudan/wyhash", "filesAnalyzed": False},
+ {"SPDXID": "SPDXRef-Package-adobe-glyph-list", "name": "adobe-glyph-list", "versionInfo": "2.0", "licenseDeclared": "BSD-3-Clause", "downloadLocation": "https://github.com/adobe-type-tools/agl-aglfn", "filesAnalyzed": False, "comment": "Glyph-name subset in internal/cbm/pdf/pdf_tables.c (PDF text layer); see THIRD_PARTY.md"},
+ {"SPDXID": "SPDXRef-Package-adobe-core14-metrics", "name": "adobe-core14-afm-metrics", "versionInfo": "1997", "licenseDeclared": "NOASSERTION", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": "Printable-ASCII advance widths of Helvetica and Times-Roman in internal/cbm/pdf/pdf_tables.c; (c) 1985-1997 Adobe Systems Incorporated; the AFM files are not distributed; see THIRD_PARTY.md"},
+ {"SPDXID": "SPDXRef-Package-unicode-ucd", "name": "unicode-character-database", "versionInfo": "16.0.0", "licenseDeclared": "Unicode-3.0", "downloadLocation": "https://www.unicode.org/Public/16.0.0/ucd/", "filesAnalyzed": False, "comment": "NFKC, compositions and character classes in internal/cbm/pdf/pdf_unicode.c, generated by scripts/gen-pdf-unicode.py; see THIRD_PARTY.md"},
{"SPDXID": "SPDXRef-Package-nomic-embed-code", "name": "nomic-embed-code-token-embeddings", "versionInfo": "1.0", "licenseDeclared": "Apache-2.0", "downloadLocation": "https://huggingface.co/nomic-ai/nomic-embed-code", "filesAnalyzed": False, "comment": "Derived int8 token embeddings; see vendored/nomic/NOTICE"},
{"SPDXID": "SPDXRef-Package-tree-sitter-grammars", "name": "tree-sitter-grammars-aggregate", "versionInfo": f"{n_grammars}-grammars", "licenseDeclared": "MIT", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": f"Aggregate of {n_grammars} vendored tree-sitter grammars, compiled statically. Predominantly MIT; non-MIT families present include CC0-1.0 (clojure, fennel), Apache-2.0 (elixir, erlang, gleam, hcl, ini, jinja2, just, pkl, sway, wit), ISC (pine, templ), Unlicense (fish); first-party grammars (c) DeusData, MIT. This summary is non-exhaustive; the authoritative complete per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md, and full license texts ship in THIRD_PARTY_NOTICES.md inside each archive."},
],
diff --git a/scripts/gen-pdf-unicode.py b/scripts/gen-pdf-unicode.py
new file mode 100644
index 000000000..d31417e2c
--- /dev/null
+++ b/scripts/gen-pdf-unicode.py
@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""Generate internal/cbm/pdf/pdf_unicode.c: the Unicode facts the PDF mention
+scanner needs, exactly as CPython sees them (the field test's scanner ran on
+CPython's str and re).
+
+usage: scripts/gen-pdf-unicode.py internal/cbm/pdf/pdf_unicode.c (then clang-format -i it)
+The output depends on the CPython version (its Unicode data): the committed
+tables were made with CPython 3.14.6 (Unicode 16.0.0).
+"""
+import sys
+import unicodedata
+
+MAXCP = 0x110000
+
+
+def is_surrogate(c):
+ return 0xD800 <= c <= 0xDFFF
+
+
+def ranges(pred):
+ out = []
+ start = None
+ for c in range(MAXCP):
+ ok = not is_surrogate(c) and pred(chr(c))
+ if ok and start is None:
+ start = c
+ elif not ok and start is not None:
+ out.append((start, c - 1))
+ start = None
+ if start is not None:
+ out.append((start, MAXCP - 1))
+ return out
+
+
+def emit_ranges(f, name, rs):
+ f.write("const pdf_urange_t %s[] = {\n" % name)
+ for i in range(0, len(rs), 4):
+ f.write(" " + " ".join("{0x%X, 0x%X}," % r for r in rs[i:i + 4]) + "\n")
+ f.write("};\nconst int %s_COUNT = (int)(sizeof(%s) / sizeof(%s[0]));\n\n" % (name, name, name))
+
+
+def main():
+ out = sys.argv[1]
+ # per-character NFKC
+ nfkc = []
+ seq = []
+ for c in range(MAXCP):
+ if is_surrogate(c):
+ continue
+ ch = chr(c)
+ m = unicodedata.normalize("NFKC", ch)
+ if m != ch:
+ nfkc.append((c, len(seq), len(m)))
+ seq.extend(ord(x) for x in m)
+ # canonical composition pairs: a + b -> c when NFC composes them
+ comp = []
+ for c in range(MAXCP):
+ if is_surrogate(c):
+ continue
+ d = unicodedata.decomposition(chr(c))
+ if not d or d.startswith("<"):
+ continue
+ parts = [int(x, 16) for x in d.split()]
+ if len(parts) != 2:
+ continue
+ a, b = parts
+ if unicodedata.normalize("NFC", chr(a) + chr(b)) == chr(c):
+ comp.append((a, b, c))
+ comp.sort()
+ with open(out, "w", encoding="utf-8") as f:
+ f.write("/*\n * pdf_unicode.c -- Unicode %s facts for the PDF mention scanner (generated).\n"
+ " *\n * The field test's scanner ran on CPython's str and re; these tables are what\n"
+ " * CPython %s answers: NFKC of each code point, the canonical compositions,\n"
+ " * str.isspace, the re module's \\w (str.isalnum or '_'), str.isalpha, and the\n"
+ " * one-character str.isupper / str.islower / titlecase. Generated by\n"
+ " * scripts/gen-pdf-unicode.py; do not edit by hand. Data of the Unicode\n"
+ " * Character Database, (c) Unicode, Inc., Unicode License v3\n"
+ " * (THIRD_PARTY.md).\n */\n"
+ % (unicodedata.unidata_version, sys.version.split()[0]))
+ f.write("#include \"pdf_unicode.h\"\n\n")
+ f.write("const pdf_nfkc_t PDF_NFKC[] = {\n")
+ for i in range(0, len(nfkc), 4):
+ f.write(" " + " ".join("{0x%X, %d, %d}," % e for e in nfkc[i:i + 4]) + "\n")
+ f.write("};\nconst int PDF_NFKC_COUNT = (int)(sizeof(PDF_NFKC) / sizeof(PDF_NFKC[0]));\n\n")
+ f.write("const uint32_t PDF_NFKC_SEQ[] = {\n")
+ for i in range(0, len(seq), 10):
+ f.write(" " + " ".join("0x%X," % x for x in seq[i:i + 10]) + "\n")
+ f.write("};\n\n")
+ f.write("const pdf_ucomp_t PDF_UCOMP[] = {\n")
+ for i in range(0, len(comp), 3):
+ f.write(" " + " ".join("{0x%X, 0x%X, 0x%X}," % e for e in comp[i:i + 3]) + "\n")
+ f.write("};\nconst int PDF_UCOMP_COUNT = (int)(sizeof(PDF_UCOMP) / sizeof(PDF_UCOMP[0]));\n\n")
+ emit_ranges(f, "PDF_U_SPACE", ranges(lambda c: c.isspace()))
+ emit_ranges(f, "PDF_U_WORD", ranges(lambda c: c.isalnum() or c == "_"))
+ emit_ranges(f, "PDF_U_ALPHA", ranges(lambda c: c.isalpha()))
+ emit_ranges(f, "PDF_U_UPPER", ranges(lambda c: c.isupper()))
+ emit_ranges(f, "PDF_U_LOWER", ranges(lambda c: c.islower()))
+ emit_ranges(f, "PDF_U_TITLE", ranges(lambda c: c.istitle() and not c.isupper()))
+ print("nfkc", len(nfkc), "seq", len(seq), "comp", len(comp))
+
+
+if __name__ == "__main__":
+ main()