🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(web): ship the whole IBM Plex Mono family, not the latin subset (#18455)
Der Gerät42c72e36be3e
diff --git a/web/public/fonts/IBMPlexMono-Bold.woff2 b/web/public/fonts/IBMPlexMono-Bold.woff2index 262a671625..8ab5889115 100644Binary files a/web/public/fonts/IBMPlexMono-Bold.woff2 and b/web/public/fonts/IBMPlexMono-Bold.woff2 differdiff --git a/web/public/fonts/IBMPlexMono-OFL.txt b/web/public/fonts/IBMPlexMono-OFL.txtindex 046fc66490..01497cc60d 100644--- a/web/public/fonts/IBMPlexMono-OFL.txt+++ b/web/public/fonts/IBMPlexMono-OFL.txt@@ -1,8 +1,8 @@-Google Inc.+Copyright © 2017 IBM Corp. with Reserved Font Name "Plex"This Font Software is licensed under the SIL Open Font License, Version 1.1.-This license is copied below, and is also available with a FAQ at:-http://scripts.sil.org/OFL++This license is copied below, and is also available with a FAQ at: http://scripts.sil.org/OFL-----------------------------------------------------------@@ -18,7 +18,7 @@ with others.The OFL allows the licensed fonts to be used, studied, modified andredistributed freely as long as they are not sold by themselves. The-fonts, including any derivative works, can be bundled, embedded,+fonts, including any derivative works, can be bundled, embedded,redistributed and/or sold with any software provided that any reservednames are not used by derivative works. The fonts and derivatives,however, cannot be released under any other type of license. Thediff --git a/web/public/fonts/IBMPlexMono-Regular.woff2 b/web/public/fonts/IBMPlexMono-Regular.woff2index a6c77d653d..a491a95961 100644Binary files a/web/public/fonts/IBMPlexMono-Regular.woff2 and b/web/public/fonts/IBMPlexMono-Regular.woff2 differdiff --git a/web/public/fonts/README.md b/web/public/fonts/README.mdnew file mode 100644index 0000000000..b2307fcc20--- /dev/null+++ b/web/public/fonts/README.md@@ -0,0 +1,77 @@+# Self-hosted IBM Plex Mono++`web/src/styles/fonts.ts` loads these two faces through `next/font/local` and+`--font-mono` in `globals.css` points at them, so they back every `font-mono`+surface: CodeMirror editors (the prompt editor, JSON viewers), code blocks and+ids.++## Provenance++- **Family** — IBM Plex Mono, version 2.3 (`fontRevision` 2.003).+- **Source** — the static TTFs in+ [`google/fonts@633f320:ofl/ibmplexmono`](https://github.com/google/fonts/tree/633f3200539c52ee0aba2dfd7f46921417a81877/ofl/ibmplexmono),+ converted to woff2 by the command below. The commit is pinned so the command+ reproduces these exact bytes.+- **License** — SIL Open Font License 1.1. `IBMPlexMono-OFL.txt` is the copy+ from that same commit, with line endings normalised to LF; keep the two+ together.++`IBMPlexMono-Bold.woff2` is the real Bold face (`usWeightClass` 700). It is+declared at CSS `weight: 600` in `fonts.ts` because `--font-weight-bold` is 600+in this app, so that is the weight the browser asks for.++## Coverage contract++**Ship every glyph the upstream family has — 930 codepoints — and do not+subset by script.**++A script subset is what broke this before: the files were once the Google Fonts+`latin` subset (230 codepoints), so Latin Extended-A, Cyrillic, Vietnamese and+the box-drawing block had no glyphs and browsers fell back **per character** to+a system monospace. The visible result is one line of text rendered in two+typefaces.++Subsetting to Google Fonts' published subsets instead (`latin`, `latin-ext`,+`cyrillic`, `cyrillic-ext`, `vietnamese`) is also not enough: their union is 678+codepoints and omits characters the full family has, such as U+2074 (⁴).++`web/src/styles/fonts.clienttest.ts` asserts this contract against the shipped+binaries.++IBM Plex Mono 2.3 has no Greek. Greek text falls back and there is no fix for+it here.++## Regenerating++Needs [`fonttools`](https://github.com/fonttools/fonttools) with Brotli+(`pip install 'fonttools[woff]'`).++```sh+upstream=633f3200539c52ee0aba2dfd7f46921417a81877+for weight in Regular Bold; do+ curl -sSLO "https://raw.githubusercontent.com/google/fonts/$upstream/ofl/ibmplexmono/IBMPlexMono-$weight.ttf"+ pyftsubset "IBMPlexMono-$weight.ttf" \+ --unicodes='*' \+ --layout-features='*' \+ --flavor=woff2 \+ --output-file="IBMPlexMono-$weight.woff2"+done+```++`--unicodes='*'` keeps every character, `--layout-features='*'` keeps every+OpenType feature, and the pass still drops `DSIG` and `meta`, which mean+nothing in a web font.++Keep the TrueType hinting tables (`cvt`, `fpgm`, `prep`, `gasp`) that this+produces. `--no-hinting` saves about 12 KB per face and changes how the text+renders on Windows.++To take a newer upstream release, move `upstream` to the commit you want and+update `UPSTREAM_CODEPOINT_COUNT` in `fonts.clienttest.ts` and the version+above in the same commit. The test failing on an upgrade is the contract+working.++After regenerating, check against the files you replaced: codepoint coverage+must be a superset, and `unitsPerEm`, ascender, descender, cap height, x-height+and the advance widths must be unchanged — otherwise every monospace surface in+the app shifts.diff --git a/web/src/styles/fonts.clienttest.ts b/web/src/styles/fonts.clienttest.tsnew file mode 100644index 0000000000..ec943d8153--- /dev/null+++ b/web/src/styles/fonts.clienttest.ts@@ -0,0 +1,275 @@+import { readdirSync, readFileSync } from "node:fs";+import { join } from "node:path";+import { brotliDecompressSync } from "node:zlib";++/**+ * The self-hosted IBM Plex Mono faces back every `font-mono` surface. A face+ * that is missing a character does not fail loudly: the browser substitutes a+ * system monospace for that character alone, so one line of text renders in+ * two typefaces. These tests read the shipped binaries and assert the coverage+ * contract in `web/public/fonts/README.md`.+ */++const FONT_DIR = join(import.meta.dirname, "../../public/fonts");++/** Every character IBM Plex Mono 2.3 defines a glyph for. */+const UPSTREAM_CODEPOINT_COUNT = 930;++const SAMPLES = {+ "Latin-1": "àäçèéêîôùûÀÄÇÈÉÊÎÔÙÛ",+ "Latin Extended-A (Polish)": "ąćęłńóśźżĄĆĘŁŃÓŚŹŻ",+ "Latin Extended-A (Turkish)": "ğışİŞĞ",+ Vietnamese: "ăâđêôơưẠảấầẫậắằẵặ",+ Cyrillic: "абвгдеёжзийклмнопрстуфхцчшщъыьэюяАБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩ",+ "Cyrillic (Ukrainian)": "іїєґІЇЄҐ",+ "Box drawing": "─│┌┐└┘├┤┬┴┼═║╔╗╚╝╠╣╦╩╬",+ Arrows: "←→↑↓↔↕",+ Mathematical: "±×÷≈≠≤≥∑∏√∞∫",+} as const;++const WOFF2_SIGNATURE = 0x774f4632;+const WOFF2_HEADER_BYTES = 48;+/** Header field holding the byte length of the Brotli-compressed table block. */+const WOFF2_COMPRESSED_LENGTH_OFFSET = 20;+/** Table directory flag meaning "a four-byte tag follows" instead of a known-table index. */+const CUSTOM_TAG_INDEX = 0x3f;+/** The known-table indices this reader needs to recognise by name. */+const KNOWN_TAGS = new Map([+ [0, "cmap"],+ [10, "glyf"],+ [11, "loca"],+]);++const UINT16_VALUES = 0x10000;++/** Glyph ids are computed modulo 2^16, and the arithmetic can go negative. */+function wrapGlyphId(value: number): number {+ return ((value % UINT16_VALUES) + UINT16_VALUES) % UINT16_VALUES;+}++type Cursor = { offset: number };++/** Variable-length integer: seven bits per byte, high bit set to continue. */+function readUIntBase128(view: DataView, cursor: Cursor): number {+ let value = 0;+ for (let i = 0; i < 5; i++) {+ const byte = view.getUint8(cursor.offset++);+ value = value * 128 + (byte % 128);+ if (byte < 128) return value;+ }+ throw new Error("malformed UIntBase128");+}++/**+ * Extract the `cmap` table from a woff2 file. The table directory gives each+ * table's length in the Brotli-compressed block, which holds the tables+ * concatenated in directory order.+ */+function readCmapTable(file: Buffer): Buffer {+ const view = new DataView(file.buffer, file.byteOffset, file.byteLength);+ if (view.getUint32(0) !== WOFF2_SIGNATURE)+ throw new Error("not a woff2 file");++ const numTables = view.getUint16(12);+ const cursor: Cursor = { offset: WOFF2_HEADER_BYTES };+ let cmap: { offset: number; length: number } | undefined;+ let blockOffset = 0;++ for (let i = 0; i < numTables; i++) {+ const flags = view.getUint8(cursor.offset++);+ const knownTagIndex = flags % 64;+ const transformVersion = Math.floor(flags / 64);++ let tag: string | undefined;+ if (knownTagIndex === CUSTOM_TAG_INDEX) {+ tag = file.toString("latin1", cursor.offset, cursor.offset + 4);+ cursor.offset += 4;+ } else {+ tag = KNOWN_TAGS.get(knownTagIndex);+ }++ // `glyf` and `loca` are transformed at version 0 and stored as they are at+ // version 3; every other table is the other way round. A transformed table+ // carries a second length, which is the one it takes up in the block.+ const originalLength = readUIntBase128(view, cursor);+ const transformed =+ tag === "glyf" || tag === "loca"+ ? transformVersion === 0+ : transformVersion !== 0;+ const storedLength = transformed+ ? readUIntBase128(view, cursor)+ : originalLength;++ if (tag === "cmap") cmap = { offset: blockOffset, length: storedLength };+ blockOffset += storedLength;+ }++ if (!cmap) throw new Error("woff2 file has no cmap table");+ // Bounded by the header's compressed size: anything after the block is+ // padding, extended metadata or private data, none of it Brotli.+ const blockLength = view.getUint32(WOFF2_COMPRESSED_LENGTH_OFFSET);+ const tables = brotliDecompressSync(+ file.subarray(cursor.offset, cursor.offset + blockLength),+ );+ return tables.subarray(cmap.offset, cmap.offset + cmap.length);+}++/** cmap subtable format 4: segmented coverage of the Basic Multilingual Plane. */+function readFormat4(view: DataView, base: number, into: Set<number>): void {+ const segmentCount = view.getUint16(base + 6) / 2;+ const endCodeAt = base + 14;+ const startCodeAt = endCodeAt + segmentCount * 2 + 2;+ const idDeltaAt = startCodeAt + segmentCount * 2;+ const idRangeOffsetAt = idDeltaAt + segmentCount * 2;++ for (let segment = 0; segment < segmentCount; segment++) {+ const start = view.getUint16(startCodeAt + segment * 2);+ if (start === 0xffff) continue;+ const end = view.getUint16(endCodeAt + segment * 2);+ const delta = view.getInt16(idDeltaAt + segment * 2);+ const rangeOffset = view.getUint16(idRangeOffsetAt + segment * 2);++ for (+ let codepoint = start;+ codepoint <= end && codepoint !== 0xffff;+ codepoint+++ ) {+ let glyph: number;+ if (rangeOffset === 0) {+ glyph = wrapGlyphId(codepoint + delta);+ } else {+ const at =+ idRangeOffsetAt + segment * 2 + rangeOffset + (codepoint - start) * 2;+ glyph = view.getUint16(at);+ if (glyph !== 0) glyph = wrapGlyphId(glyph + delta);+ }+ // A segment is allowed to map a character to the "missing glyph".+ if (glyph !== 0) into.add(codepoint);+ }+ }+}++/** cmap subtable format 12: grouped coverage, used beyond the BMP. */+function readFormat12(view: DataView, base: number, into: Set<number>): void {+ const groupCount = view.getUint32(base + 12);+ for (let group = 0; group < groupCount; group++) {+ const at = base + 16 + group * 12;+ const start = view.getUint32(at);+ const end = view.getUint32(at + 4);+ const startGlyph = view.getUint32(at + 8);+ for (let codepoint = start; codepoint <= end; codepoint++) {+ // Glyph ids run consecutively from startGlyph, so at most the first+ // character of a group lands on the "missing glyph".+ if (startGlyph + (codepoint - start) !== 0) into.add(codepoint);+ }+ }+}++function readCodepoints(fileName: string): Set<number> {+ const cmap = readCmapTable(readFileSync(join(FONT_DIR, fileName)));+ const view = new DataView(cmap.buffer, cmap.byteOffset, cmap.byteLength);+ const codepoints = new Set<number>();++ const subtableCount = view.getUint16(2);+ for (let i = 0; i < subtableCount; i++) {+ const base = view.getUint32(4 + i * 8 + 4);+ const format = view.getUint16(base);+ if (format === 4) readFormat4(view, base, codepoints);+ else if (format === 12) readFormat12(view, base, codepoints);+ }+ return codepoints;+}++// Discovered rather than listed, so a face added later is held to the same+// contract instead of slipping in under a suite that stays green.+const faces = readdirSync(FONT_DIR)+ .filter((fileName) => fileName.endsWith(".woff2"))+ .sort()+ .map((fileName) => [fileName, readCodepoints(fileName)] as const);++it("finds the faces fonts.ts loads", () => {+ expect(faces.map(([fileName]) => fileName)).toEqual([+ "IBMPlexMono-Bold.woff2",+ "IBMPlexMono-Regular.woff2",+ ]);+});++describe.each(faces)("%s", (_fileName, codepoints) => {+ it("ships every glyph of the upstream family", () => {+ expect(codepoints.size).toBe(UPSTREAM_CODEPOINT_COUNT);+ });++ it.each(Object.entries(SAMPLES))("covers %s", (_script, sample) => {+ const missing = [...sample].filter(+ (character) => !codepoints.has(character.codePointAt(0)!),+ );+ expect(missing).toEqual([]);+ });+});++// The shipped faces carry only format 4 subtables, so these pin the format 12+// reader against the spec rather than against a binary we happen to have.+describe("cmap format 12 reader", () => {+ function subtable(+ groups: { start: number; end: number; startGlyph: number }[],+ ): DataView {+ const view = new DataView(new ArrayBuffer(16 + groups.length * 12));+ view.setUint16(0, 12);+ view.setUint32(12, groups.length);+ groups.forEach((group, index) => {+ const at = 16 + index * 12;+ view.setUint32(at, group.start);+ view.setUint32(at + 4, group.end);+ view.setUint32(at + 8, group.startGlyph);+ });+ return view;+ }++ function read(+ groups: { start: number; end: number; startGlyph: number }[],+ ): Set<number> {+ const covered = new Set<number>();+ readFormat12(subtable(groups), 0, covered);+ return covered;+ }++ it("keeps a group whose glyph ids happen to start at zero", () => {+ // Ids run consecutively, so only the first character is the missing glyph.+ const covered = read([{ start: 0, end: 0x24f, startGlyph: 0 }]);+ expect(covered.size).toBe(0x24f);+ expect(covered.has(0)).toBe(false);+ expect(covered.has(1)).toBe(true);+ expect(covered.has(0x24f)).toBe(true);+ });++ it("drops only the character that lands on the missing glyph", () => {+ const covered = read([{ start: 0x100, end: 0x10f, startGlyph: 0 }]);+ expect(covered.size).toBe(15);+ expect(covered.has(0x100)).toBe(false);+ expect(covered.has(0x101)).toBe(true);+ });++ it("keeps every character of a group mapped above the missing glyph", () => {+ const covered = read([{ start: 0x100, end: 0x10f, startGlyph: 5 }]);+ expect(covered.size).toBe(16);+ expect(covered.has(0x100)).toBe(true);+ });++ it("reads supplementary-plane groups", () => {+ const covered = read([{ start: 0x1f600, end: 0x1f602, startGlyph: 9 }]);+ expect([...covered]).toEqual([0x1f600, 0x1f601, 0x1f602]);+ });+});++it("declares the same characters in every weight", () => {+ // A character present in one weight only renders in a fallback font as soon+ // as it is bold, which is the same mixed-typeface bug one weight deep.+ const [[, reference]] = faces;+ const disagreements = faces.flatMap(([fileName, codepoints]) =>+ [+ ...[...reference].filter((cp) => !codepoints.has(cp)),+ ...[...codepoints].filter((cp) => !reference.has(cp)),+ ].map((cp) => `${fileName} U+${cp.toString(16).toUpperCase()}`),+ );+ expect(disagreements).toEqual([]);+});