Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(web): ship the whole IBM Plex Mono family, not the latin subset (#18455)

Der Gerät42c72e36be3e
diff --git a/web/public/fonts/IBMPlexMono-Bold.woff2 b/web/public/fonts/IBMPlexMono-Bold.woff2
index 262a671625..8ab5889115 100644
Binary files a/web/public/fonts/IBMPlexMono-Bold.woff2 and b/web/public/fonts/IBMPlexMono-Bold.woff2 differ
diff --git a/web/public/fonts/IBMPlexMono-OFL.txt b/web/public/fonts/IBMPlexMono-OFL.txt
index 046fc66490..01497cc60d 100644
--- a/web/public/fonts/IBMPlexMono-OFL.txt
+++ b/web/public/fonts/IBMPlexMono-OFL.txt
@@ -1,8 +1,8 @@
-Google Inc.
+Copyright © 2017 IBM Corp. with Reserved Font Name "Plex"
This Font Software is licensed under the SIL Open Font License, Version 1.1.
-This license is copied below, and is also available with a FAQ at:
-http://scripts.sil.org/OFL
+
+This license is copied below, and is also available with a FAQ at: http://scripts.sil.org/OFL
-----------------------------------------------------------
@@ -18,7 +18,7 @@ with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
-fonts, including any derivative works, can be bundled, embedded,
+fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
diff --git a/web/public/fonts/IBMPlexMono-Regular.woff2 b/web/public/fonts/IBMPlexMono-Regular.woff2
index a6c77d653d..a491a95961 100644
Binary files a/web/public/fonts/IBMPlexMono-Regular.woff2 and b/web/public/fonts/IBMPlexMono-Regular.woff2 differ
diff --git a/web/public/fonts/README.md b/web/public/fonts/README.md
new file mode 100644
index 0000000000..b2307fcc20
--- /dev/null
+++ b/web/public/fonts/README.md
@@ -0,0 +1,77 @@
+# Self-hosted IBM Plex Mono
+
+`web/src/styles/fonts.ts` loads these two faces through `next/font/local` and
+`--font-mono` in `globals.css` points at them, so they back every `font-mono`
+surface: CodeMirror editors (the prompt editor, JSON viewers), code blocks and
+ids.
+
+## Provenance
+
+- **Family** — IBM Plex Mono, version 2.3 (`fontRevision` 2.003).
+- **Source** — the static TTFs in
+ [`google/fonts@633f320:ofl/ibmplexmono`](https://github.com/google/fonts/tree/633f3200539c52ee0aba2dfd7f46921417a81877/ofl/ibmplexmono),
+ converted to woff2 by the command below. The commit is pinned so the command
+ reproduces these exact bytes.
+- **License** — SIL Open Font License 1.1. `IBMPlexMono-OFL.txt` is the copy
+ from that same commit, with line endings normalised to LF; keep the two
+ together.
+
+`IBMPlexMono-Bold.woff2` is the real Bold face (`usWeightClass` 700). It is
+declared at CSS `weight: 600` in `fonts.ts` because `--font-weight-bold` is 600
+in this app, so that is the weight the browser asks for.
+
+## Coverage contract
+
+**Ship every glyph the upstream family has — 930 codepoints — and do not
+subset by script.**
+
+A script subset is what broke this before: the files were once the Google Fonts
+`latin` subset (230 codepoints), so Latin Extended-A, Cyrillic, Vietnamese and
+the box-drawing block had no glyphs and browsers fell back **per character** to
+a system monospace. The visible result is one line of text rendered in two
+typefaces.
+
+Subsetting to Google Fonts' published subsets instead (`latin`, `latin-ext`,
+`cyrillic`, `cyrillic-ext`, `vietnamese`) is also not enough: their union is 678
+codepoints and omits characters the full family has, such as U+2074 (⁴).
+
+`web/src/styles/fonts.clienttest.ts` asserts this contract against the shipped
+binaries.
+
+IBM Plex Mono 2.3 has no Greek. Greek text falls back and there is no fix for
+it here.
+
+## Regenerating
+
+Needs [`fonttools`](https://github.com/fonttools/fonttools) with Brotli
+(`pip install 'fonttools[woff]'`).
+
+```sh
+upstream=633f3200539c52ee0aba2dfd7f46921417a81877
+for weight in Regular Bold; do
+ curl -sSLO "https://raw.githubusercontent.com/google/fonts/$upstream/ofl/ibmplexmono/IBMPlexMono-$weight.ttf"
+ pyftsubset "IBMPlexMono-$weight.ttf" \
+ --unicodes='*' \
+ --layout-features='*' \
+ --flavor=woff2 \
+ --output-file="IBMPlexMono-$weight.woff2"
+done
+```
+
+`--unicodes='*'` keeps every character, `--layout-features='*'` keeps every
+OpenType feature, and the pass still drops `DSIG` and `meta`, which mean
+nothing in a web font.
+
+Keep the TrueType hinting tables (`cvt`, `fpgm`, `prep`, `gasp`) that this
+produces. `--no-hinting` saves about 12 KB per face and changes how the text
+renders on Windows.
+
+To take a newer upstream release, move `upstream` to the commit you want and
+update `UPSTREAM_CODEPOINT_COUNT` in `fonts.clienttest.ts` and the version
+above in the same commit. The test failing on an upgrade is the contract
+working.
+
+After regenerating, check against the files you replaced: codepoint coverage
+must be a superset, and `unitsPerEm`, ascender, descender, cap height, x-height
+and the advance widths must be unchanged — otherwise every monospace surface in
+the app shifts.
diff --git a/web/src/styles/fonts.clienttest.ts b/web/src/styles/fonts.clienttest.ts
new file mode 100644
index 0000000000..ec943d8153
--- /dev/null
+++ b/web/src/styles/fonts.clienttest.ts
@@ -0,0 +1,275 @@
+import { readdirSync, readFileSync } from "node:fs";
+import { join } from "node:path";
+import { brotliDecompressSync } from "node:zlib";
+
+/**
+ * The self-hosted IBM Plex Mono faces back every `font-mono` surface. A face
+ * that is missing a character does not fail loudly: the browser substitutes a
+ * system monospace for that character alone, so one line of text renders in
+ * two typefaces. These tests read the shipped binaries and assert the coverage
+ * contract in `web/public/fonts/README.md`.
+ */
+
+const FONT_DIR = join(import.meta.dirname, "../../public/fonts");
+
+/** Every character IBM Plex Mono 2.3 defines a glyph for. */
+const UPSTREAM_CODEPOINT_COUNT = 930;
+
+const SAMPLES = {
+ "Latin-1": "àäçèéêîôùûÀÄÇÈÉÊÎÔÙÛ",
+ "Latin Extended-A (Polish)": "ąćęłńóśźżĄĆĘŁŃÓŚŹŻ",
+ "Latin Extended-A (Turkish)": "ğışİŞĞ",
+ Vietnamese: "ăâđêôơưẠảấầẫậắằẵặ",
+ Cyrillic: "абвгдеёжзийклмнопрстуфхцчшщъыьэюяАБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩ",
+ "Cyrillic (Ukrainian)": "іїєґІЇЄҐ",
+ "Box drawing": "─│┌┐└┘├┤┬┴┼═║╔╗╚╝╠╣╦╩╬",
+ Arrows: "←→↑↓↔↕",
+ Mathematical: "±×÷≈≠≤≥∑∏√∞∫",
+} as const;
+
+const WOFF2_SIGNATURE = 0x774f4632;
+const WOFF2_HEADER_BYTES = 48;
+/** Header field holding the byte length of the Brotli-compressed table block. */
+const WOFF2_COMPRESSED_LENGTH_OFFSET = 20;
+/** Table directory flag meaning "a four-byte tag follows" instead of a known-table index. */
+const CUSTOM_TAG_INDEX = 0x3f;
+/** The known-table indices this reader needs to recognise by name. */
+const KNOWN_TAGS = new Map([
+ [0, "cmap"],
+ [10, "glyf"],
+ [11, "loca"],
+]);
+
+const UINT16_VALUES = 0x10000;
+
+/** Glyph ids are computed modulo 2^16, and the arithmetic can go negative. */
+function wrapGlyphId(value: number): number {
+ return ((value % UINT16_VALUES) + UINT16_VALUES) % UINT16_VALUES;
+}
+
+type Cursor = { offset: number };
+
+/** Variable-length integer: seven bits per byte, high bit set to continue. */
+function readUIntBase128(view: DataView, cursor: Cursor): number {
+ let value = 0;
+ for (let i = 0; i < 5; i++) {
+ const byte = view.getUint8(cursor.offset++);
+ value = value * 128 + (byte % 128);
+ if (byte < 128) return value;
+ }
+ throw new Error("malformed UIntBase128");
+}
+
+/**
+ * Extract the `cmap` table from a woff2 file. The table directory gives each
+ * table's length in the Brotli-compressed block, which holds the tables
+ * concatenated in directory order.
+ */
+function readCmapTable(file: Buffer): Buffer {
+ const view = new DataView(file.buffer, file.byteOffset, file.byteLength);
+ if (view.getUint32(0) !== WOFF2_SIGNATURE)
+ throw new Error("not a woff2 file");
+
+ const numTables = view.getUint16(12);
+ const cursor: Cursor = { offset: WOFF2_HEADER_BYTES };
+ let cmap: { offset: number; length: number } | undefined;
+ let blockOffset = 0;
+
+ for (let i = 0; i < numTables; i++) {
+ const flags = view.getUint8(cursor.offset++);
+ const knownTagIndex = flags % 64;
+ const transformVersion = Math.floor(flags / 64);
+
+ let tag: string | undefined;
+ if (knownTagIndex === CUSTOM_TAG_INDEX) {
+ tag = file.toString("latin1", cursor.offset, cursor.offset + 4);
+ cursor.offset += 4;
+ } else {
+ tag = KNOWN_TAGS.get(knownTagIndex);
+ }
+
+ // `glyf` and `loca` are transformed at version 0 and stored as they are at
+ // version 3; every other table is the other way round. A transformed table
+ // carries a second length, which is the one it takes up in the block.
+ const originalLength = readUIntBase128(view, cursor);
+ const transformed =
+ tag === "glyf" || tag === "loca"
+ ? transformVersion === 0
+ : transformVersion !== 0;
+ const storedLength = transformed
+ ? readUIntBase128(view, cursor)
+ : originalLength;
+
+ if (tag === "cmap") cmap = { offset: blockOffset, length: storedLength };
+ blockOffset += storedLength;
+ }
+
+ if (!cmap) throw new Error("woff2 file has no cmap table");
+ // Bounded by the header's compressed size: anything after the block is
+ // padding, extended metadata or private data, none of it Brotli.
+ const blockLength = view.getUint32(WOFF2_COMPRESSED_LENGTH_OFFSET);
+ const tables = brotliDecompressSync(
+ file.subarray(cursor.offset, cursor.offset + blockLength),
+ );
+ return tables.subarray(cmap.offset, cmap.offset + cmap.length);
+}
+
+/** cmap subtable format 4: segmented coverage of the Basic Multilingual Plane. */
+function readFormat4(view: DataView, base: number, into: Set<number>): void {
+ const segmentCount = view.getUint16(base + 6) / 2;
+ const endCodeAt = base + 14;
+ const startCodeAt = endCodeAt + segmentCount * 2 + 2;
+ const idDeltaAt = startCodeAt + segmentCount * 2;
+ const idRangeOffsetAt = idDeltaAt + segmentCount * 2;
+
+ for (let segment = 0; segment < segmentCount; segment++) {
+ const start = view.getUint16(startCodeAt + segment * 2);
+ if (start === 0xffff) continue;
+ const end = view.getUint16(endCodeAt + segment * 2);
+ const delta = view.getInt16(idDeltaAt + segment * 2);
+ const rangeOffset = view.getUint16(idRangeOffsetAt + segment * 2);
+
+ for (
+ let codepoint = start;
+ codepoint <= end && codepoint !== 0xffff;
+ codepoint++
+ ) {
+ let glyph: number;
+ if (rangeOffset === 0) {
+ glyph = wrapGlyphId(codepoint + delta);
+ } else {
+ const at =
+ idRangeOffsetAt + segment * 2 + rangeOffset + (codepoint - start) * 2;
+ glyph = view.getUint16(at);
+ if (glyph !== 0) glyph = wrapGlyphId(glyph + delta);
+ }
+ // A segment is allowed to map a character to the "missing glyph".
+ if (glyph !== 0) into.add(codepoint);
+ }
+ }
+}
+
+/** cmap subtable format 12: grouped coverage, used beyond the BMP. */
+function readFormat12(view: DataView, base: number, into: Set<number>): void {
+ const groupCount = view.getUint32(base + 12);
+ for (let group = 0; group < groupCount; group++) {
+ const at = base + 16 + group * 12;
+ const start = view.getUint32(at);
+ const end = view.getUint32(at + 4);
+ const startGlyph = view.getUint32(at + 8);
+ for (let codepoint = start; codepoint <= end; codepoint++) {
+ // Glyph ids run consecutively from startGlyph, so at most the first
+ // character of a group lands on the "missing glyph".
+ if (startGlyph + (codepoint - start) !== 0) into.add(codepoint);
+ }
+ }
+}
+
+function readCodepoints(fileName: string): Set<number> {
+ const cmap = readCmapTable(readFileSync(join(FONT_DIR, fileName)));
+ const view = new DataView(cmap.buffer, cmap.byteOffset, cmap.byteLength);
+ const codepoints = new Set<number>();
+
+ const subtableCount = view.getUint16(2);
+ for (let i = 0; i < subtableCount; i++) {
+ const base = view.getUint32(4 + i * 8 + 4);
+ const format = view.getUint16(base);
+ if (format === 4) readFormat4(view, base, codepoints);
+ else if (format === 12) readFormat12(view, base, codepoints);
+ }
+ return codepoints;
+}
+
+// Discovered rather than listed, so a face added later is held to the same
+// contract instead of slipping in under a suite that stays green.
+const faces = readdirSync(FONT_DIR)
+ .filter((fileName) => fileName.endsWith(".woff2"))
+ .sort()
+ .map((fileName) => [fileName, readCodepoints(fileName)] as const);
+
+it("finds the faces fonts.ts loads", () => {
+ expect(faces.map(([fileName]) => fileName)).toEqual([
+ "IBMPlexMono-Bold.woff2",
+ "IBMPlexMono-Regular.woff2",
+ ]);
+});
+
+describe.each(faces)("%s", (_fileName, codepoints) => {
+ it("ships every glyph of the upstream family", () => {
+ expect(codepoints.size).toBe(UPSTREAM_CODEPOINT_COUNT);
+ });
+
+ it.each(Object.entries(SAMPLES))("covers %s", (_script, sample) => {
+ const missing = [...sample].filter(
+ (character) => !codepoints.has(character.codePointAt(0)!),
+ );
+ expect(missing).toEqual([]);
+ });
+});
+
+// The shipped faces carry only format 4 subtables, so these pin the format 12
+// reader against the spec rather than against a binary we happen to have.
+describe("cmap format 12 reader", () => {
+ function subtable(
+ groups: { start: number; end: number; startGlyph: number }[],
+ ): DataView {
+ const view = new DataView(new ArrayBuffer(16 + groups.length * 12));
+ view.setUint16(0, 12);
+ view.setUint32(12, groups.length);
+ groups.forEach((group, index) => {
+ const at = 16 + index * 12;
+ view.setUint32(at, group.start);
+ view.setUint32(at + 4, group.end);
+ view.setUint32(at + 8, group.startGlyph);
+ });
+ return view;
+ }
+
+ function read(
+ groups: { start: number; end: number; startGlyph: number }[],
+ ): Set<number> {
+ const covered = new Set<number>();
+ readFormat12(subtable(groups), 0, covered);
+ return covered;
+ }
+
+ it("keeps a group whose glyph ids happen to start at zero", () => {
+ // Ids run consecutively, so only the first character is the missing glyph.
+ const covered = read([{ start: 0, end: 0x24f, startGlyph: 0 }]);
+ expect(covered.size).toBe(0x24f);
+ expect(covered.has(0)).toBe(false);
+ expect(covered.has(1)).toBe(true);
+ expect(covered.has(0x24f)).toBe(true);
+ });
+
+ it("drops only the character that lands on the missing glyph", () => {
+ const covered = read([{ start: 0x100, end: 0x10f, startGlyph: 0 }]);
+ expect(covered.size).toBe(15);
+ expect(covered.has(0x100)).toBe(false);
+ expect(covered.has(0x101)).toBe(true);
+ });
+
+ it("keeps every character of a group mapped above the missing glyph", () => {
+ const covered = read([{ start: 0x100, end: 0x10f, startGlyph: 5 }]);
+ expect(covered.size).toBe(16);
+ expect(covered.has(0x100)).toBe(true);
+ });
+
+ it("reads supplementary-plane groups", () => {
+ const covered = read([{ start: 0x1f600, end: 0x1f602, startGlyph: 9 }]);
+ expect([...covered]).toEqual([0x1f600, 0x1f601, 0x1f602]);
+ });
+});
+
+it("declares the same characters in every weight", () => {
+ // A character present in one weight only renders in a fallback font as soon
+ // as it is bold, which is the same mixed-typeface bug one weight deep.
+ const [[, reference]] = faces;
+ const disagreements = faces.flatMap(([fileName, codepoints]) =>
+ [
+ ...[...reference].filter((cp) => !codepoints.has(cp)),
+ ...[...codepoints].filter((cp) => !reference.has(cp)),
+ ].map((cp) => `${fileName} U+${cp.toString(16).toUpperCase()}`),
+ );
+ expect(disagreements).toEqual([]);
+});