Skip to content

Claude's answers, rendered: Markdown, math, charts and diagrams drawn in the Claude Code terminal. Independent project, not affiliated with Anthropic.

TypeScript claude-code-mod

Picture select: highlight by Image blit; remove row strips

razorback16dc3c755cb225
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9e11714..1674355 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,7 +6,7 @@ All releases of Intelligent UI for Claude Code (`cc-intelligent-ui`). Claude Cod
### New
-- **Picture select.** Since Claude Code 2.1.305, the mouse selection of Claude Code skips pictures: no highlight and no text. In fullscreen mode, the mod now selects the text of a picture itself. Drag over the picture: the selected rows show on a blue background, and when you release the button, the mod copies their text and shows the toast "copied N chars of text". A click with no drag removes the selection. This needs Claude Code 2.1.305 or newer. `/iui select off` turns it off. Refer to [Copy text from a picture](docs/usage.md#copy-text-from-a-picture).
+- **Picture select.** Since Claude Code 2.1.305, the mouse selection of Claude Code skips pictures: no highlight and no text. In fullscreen mode, the mod now selects the text of a picture itself. Drag over the picture: the selected cells show on a blue background, and when you release the button, the mod copies their text and shows the toast "copied N chars of text". A click with no drag removes the selection. The highlight swaps the page's picture with `$.ui.blit` (a frame with the selected cells tinted), so the layout never changes and a clear leaves no old cells. This needs Claude Code 2.1.305 or newer. `/iui select off` turns it off. Refer to [Copy text from a picture](docs/usage.md#copy-text-from-a-picture).
### Changed
diff --git a/docs/development.md b/docs/development.md
index b087bf6..61cb408 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -20,11 +20,11 @@ claude plugin validate .
Set `IUI_DEBUG=1` before you start Claude Code. The mod then writes trace lines to `debug.log` in the cache folder (`${TMPDIR}/cc-intelligent-ui-<uid>/`, `/tmp` when `TMPDIR` is not set).
-Picture select writes one line for each post of a `Client` (lines that start with `select:`): the phase and the order, the drawing's id, the page, the anchor and the head (the `Client`'s cells), the size of the `Client`'s region, the page's size in cells, the rows of the message on screen (`onScreen`), the page's top row in the message, its rows above the screen (`above`), and the selected rows of each page.
+Picture select writes one line for each post of a `Client` (lines that start with `select:`): the phase and the order, the drawing's id, the page, the anchor and the head (the `Client`'s cells), the size of the `Client`'s region, the page's size in cells, the rows of the message on screen (`onScreen`), the page's top row in the message, its rows above the screen (`above`), and the selected rows of each page. Each blit of the highlight writes a line too (`select: blit`): the page, its selected rows (or `plain`), the times to decode the page (0 when it is decoded already), make the frame and write it, the frame's bytes, and the time of `$.ui.blit`. The first decode of a page writes `select: decoded` with its time.
## Module layout
-The hooks module is `hooks/register.tsx`. It connects the hooks to the modules in `hooks/lib/`. Each module has one topic, for example `markdown.ts` (blocks and segments), `mode.ts` (light or dark), `typst.ts` (compile), `fonts.ts` (the lean font set: the fonts that a compile of plain text loads), `draw.tsx` (the pages of an answer and the link rows under them), `links.ts` (the links, file paths and commands of a text), `answer.ts` (one text through the compile and cache steps), `plan.tsx` (the plan approval question and the plan rows), `frame.ts` (the terminal size and the watcher process, which also reports the clipboard changes), `clipboard.ts` (how the clipboard is read and watched: macOS, or the Linux tools; the clipboard shims, from the template `hooks/clip-shim.sh`, set up by `hooks/clip-shim-setup.sh`; and the reader of their asks), `copy.ts` (the text under a mouse selection of pictures), `probe.ts` (the drawn pages, and the words of a page that a copy reads, kept in memory) and `updates.ts` (the daily check for a new version). The values that change are in one object, `S`, in `state.ts`. Each `$` method has one call site, a wrapper in `api.ts`; the other modules call the wrappers. This keeps the `calls:` note of `claude plugin validate` short (the validator cuts a note at about 1,000 characters, and the catalogue scan then cannot read the footprint). The tests import from `hooks/register.tsx`, which exports the names that they use.
+The hooks module is `hooks/register.tsx`. It connects the hooks to the modules in `hooks/lib/`. Each module has one topic, for example `markdown.ts` (blocks and segments), `mode.ts` (light or dark), `typst.ts` (compile), `fonts.ts` (the lean font set: the fonts that a compile of plain text loads), `draw.tsx` (the pages of an answer and the link rows under them), `links.ts` (the links, file paths and commands of a text), `answer.ts` (one text through the compile and cache steps), `plan.tsx` (the plan approval question and the plan rows), `frame.ts` (the terminal size and the watcher process, which also reports the clipboard changes), `clipboard.ts` (how the clipboard is read and watched: macOS, or the Linux tools; the clipboard shims, from the template `hooks/clip-shim.sh`, set up by `hooks/clip-shim-setup.sh`; and the reader of their asks), `copy.ts` (the text under a mouse selection of pictures), `select.ts` and `blit.ts` (picture select and its highlight), `png.ts` (PNG decode and the frames' deflate pieces, in plain JavaScript), `probe.ts` (the drawn pages, and the words of a page that a copy reads, kept in memory) and `updates.ts` (the daily check for a new version). The values that change are in one object, `S`, in `state.ts`. Each `$` method has one call site, a wrapper in `api.ts`; the other modules call the wrappers. This keeps the `calls:` note of `claude plugin validate` short (the validator cuts a note at about 1,000 characters, and the catalogue scan then cannot read the footprint). The tests import from `hooks/register.tsx`, which exports the names that they use.
The skill for Claude is in `skills/intelligent-ui/`: `SKILL.md` (the contract, the workflow and an index), `references/` (one file for each topic: Typst, fletcher, CeTZ, plots and charts) and `scripts/check.sh` (the compile check, with the real template). `TYPST_PROMPT` in `hooks/lib/settings.ts` tells Claude when to load it.
diff --git a/docs/how-it-works.md b/docs/how-it-works.md
index 242592a..9d58ad9 100644
--- a/docs/how-it-works.md
+++ b/docs/how-it-works.md
@@ -25,13 +25,13 @@
From Claude Code 2.1.305, the mod's `Image` has `noSelect`: the selection painter and the copy on select of Claude Code skip picture cells. Thus the mod selects pictures itself (`hooks/lib/select.ts`):
1. Each page is in a `Box` of its size. The last child is a `Client` (`hooks/sel.tsx`) in a `Box` with a negative top margin (the page's rows), so it is over the page. The `Client` draws only an empty `Box` (anything that it draws hides the picture cells). It takes the pointer and posts the selection whole: the anchor, the head, the phase (`down`, `move`, `up` or `clear`) and an order. Posts of one frame merge, so a bare event could be lost.
-2. The `ui.message` hook keeps the newest selection and draws again. The selected rows are in stream order, in the rows of the whole drawing, so a drag can go on into the next page.
-3. The selected rows are in a `Box` over the page (also with a negative top margin). Each selected row is a `Box` of the selected columns, with `overflow: hidden` and the selection colour, that holds that row of the page as an `Image` of one row, moved left by the first column ("clip, do not crop"). The page is transparent, so the colour shows under the text. The engine paints an overlay again only when it changes, so the colour changes by one step at each redraw.
-4. The rows come from Typst (`render/strips.typ`, `--ppi 72`: one pixel for one pt): the first selection on a page cuts it into one PNG for each row, in a folder beside the page in its key folder (so the 2-day cleanup takes them). A PNG's rows cannot be cut from its bytes (its filters go across rows).
-5. The terminal fits an image into its cells with the aspect kept. A row PNG shows the same pixels as its row of the page only when both are at one scale with no space above or below. Thus a page has picture select only when its height divided by its rows is a whole number of pixels (`rowPx`), and its width is at most `rowPx × columns × cell width / cell height` (`stripPlan`). The template pads each page to whole rows at the cell height, so with whole-pixel cells (Ghostty, kitty) `rowPx` is the cell height.
+2. The `ui.message` hook keeps the newest selection. The selected rows are in stream order, in the rows of the whole drawing, so a drag can go on into the next page.
+3. The highlight does not change the layout. Each page stays one keyed `Image`, and the mod swaps its picture with `$.ui.blit` (`hooks/lib/blit.ts`). This is one small kitty command with the same image id (`a=T`, `t=f`, the file name), so the terminal replaces the picture in place. The frame is the page with the selected cells changed: in a selected cell, each pixel of the page (straight alpha) over the opaque selection colour; in the other cells, the page as it is. A clear swaps the page's own PNG back. Cell `i` is the pixels from `round(i × pixels / cells)` to `round((i + 1) × pixels / cells)`, so the cells cover the page with no gap at any ratio.
+4. The hooks runtime has no zlib, so `hooks/lib/png.ts` decodes the page in JavaScript (inflate and unfilter) at the first `down` on it, one time. The mod keeps the decoded pages in memory, 128 MB at most (about two pages of 3392 × 4284 pixels). For each row of cells, it makes a deflate piece one time (fixed Huffman codes, matches of one pixel to the left or one pixel up, ended on a byte). A row that is selected whole has a tinted piece, also made one time. A frame makes new pieces only for the first and last selected rows (partly selected), and joins the pieces into one PNG (about 1.5 MB for a page of 3392 × 4284 pixels). The hooks can write only text, so `perl` writes the frame (base64 on its stdin) to `<cache>/select-<id>/f-<n>.png`. The terminal reads the file after the blit. The mod deletes a frame file when 4 newer frames are written and it is 2 s old. The 2-day cleanup of the cache folder deletes the rest.
+5. A new frame starts at most once in 16 ms, and only after the last one is blitted. The posts between fold into one frame of the newest selection. If Claude Code refuses a blit, the mod writes the reason one time to the debug log and shows no highlight on that page until the next `down`. The copy still works.
6. At `up`, the mod reads the words under the selected cells the same way as for a copy of these cells (`textOfRun` on the words of the probe), and copies them with `$.ui.copy`.
-The `Client` and the rows are in the flow, not in absolute `Box`es. Claude Code (2.1.306) moves an absolute `Box` whose top is above the screen down to row 0, with its children. Then the pointer `y` of a `Client` in such a `Box` counts from row 0 of the screen, not from the top of the page, so the selection was too high by the rows that were scrolled away. Also, the selected rows above the screen showed on row 0, and the `Client` took the pointer under the page. A `Box` in the flow has its true position, also when its top is above the screen.
+The `Client` is in the flow, not in an absolute `Box`. Claude Code (2.1.306) moves an absolute `Box` whose top is above the screen down to row 0, with its children. Then the pointer `y` of a `Client` in such a `Box` counts from row 0 of the screen, not from the top of the page, so the selection was too high by the rows that were scrolled away. Also, the `Client` took the pointer under the page. A `Box` in the flow has its true position, also when its top is above the screen.
A `Client` over a picture also takes the selection from Claude Code before 2.1.305 (which selects and copies pictures itself), so the mod draws it only from 2.1.305 (`$.session.version()`), in fullscreen mode. A `Client` that fails (`ui.fault`) turns picture select off until a reload.
diff --git a/docs/limitations.md b/docs/limitations.md
index 849f5d2..5444209 100644
--- a/docs/limitations.md
+++ b/docs/limitations.md
@@ -27,9 +27,12 @@
- A selection is in one picture (an answer, a plan or a Markdown file, all its pages). A drag that starts on normal text and goes into a picture is the selection of Claude Code, which skips the picture. A drag that starts in a picture stays in that picture. Thus a selection of text and a picture together is not possible.
- A click on normal text does not remove a picture selection. A click in a picture, a new picture selection or your next prompt removes it.
- It has no double click (word) or triple click (line) selection, and no Shift+click to make a selection longer.
- - A page must fill whole cells: its height must be a whole number of rows, each row a whole number of pixels, at the cell shape of the terminal. The mod makes pages so in Ghostty and kitty. A page that is not so (a page drawn smaller after a resize until its new picture is ready, a page cut at 255 rows, or a cell size of a fraction of a pixel) has no picture select. There the selection of Claude Code skips it.
+ - The highlight shows a short time after the first `down` on a page: the mod decodes the page one time (approximately 50 ms for a page of 3392 × 756 pixels, 200 ms for 3392 × 4284). Each frame is a new PNG that the terminal decodes (1.5 MB at most for the tallest page).
+ - The highlight maps cells to pixels by the page's size in cells. If the terminal draws a page with space at its side or above (another aspect than its cells), the highlight can be off by part of a cell.
+ - When Claude Code draws the page again from the start (for example, a page that scrolled far away and comes back), the page shows without its highlight until the selection changes. The copy is not changed.
- The blue background is VS Code's selection colour, not the one of your terminal or Claude Code theme.
- - Each selected row is one more small image in the terminal (approximately 0.5 MB in Ghostty's image memory, whose default limit is 320 MB).
+ - A page whose PNG is over 4 MiB (the most that the mod can read) has no highlight.
+ - It needs `perl` (with `MIME::Base64`, a core module) to write the frames. With no `perl`, the selection and the copy work with no highlight.
- **Links in a picture:** a click on the picture does not open a link, because the mod cannot put a link on a picture. Use the link rows under the picture (refer to [Links, file paths and commands](usage.md#links-file-paths-and-commands)). A click on a command row copies the command only where Claude Code gets mouse clicks (fullscreen mode). A `file://` link opens in the app that the terminal chooses. A file row shows only when the file exists when the mod draws the answer.
- **PDF files** (refer to [What it draws](usage.md#what-it-draws)): a click on `‹`, `›` or the page number works only where Claude Code gets mouse clicks (fullscreen mode). The page field gets the keyboard only when the prompt is empty. If the prompt has text, the field opens without the keyboard: click the field to type. Then Esc goes to the prompt, not to the field. To close the field, click it and push Esc, or click its close mark. The mod draws one page at a time, and it does not show the number of pages. You cannot copy text from a PDF page.
- **The row grid** (`/iui grid on`, refer to [Selection and the row grid](usage.md#selection-and-the-row-grid)) has these limits:
diff --git a/hooks/lib/api.ts b/hooks/lib/api.ts
index e712f1a..407d291 100644
--- a/hooks/lib/api.ts
+++ b/hooks/lib/api.ts
@@ -4,7 +4,7 @@
// declaration that spells the call `$.noun.method(...)` (the validator
// follows `$` into a declaration only), with the types of EngineInterface.
import type {
- AskOptions, CommandSpec, EngineInterface, FsBytes, FsReadBytesOptions, FsReadOptions, FsStatOptions, HttpInit, InvalidatableEventName,
+ AskOptions, CommandSpec, EngineInterface, FsBytes, FsReadBytesOptions, FsReadOptions, FsStatOptions, HttpInit, ImageBlitArgs, InvalidatableEventName,
OpEventOf, PaneCloseArgs, PaneOpenArgs, ProcessRunInit, ProcessSpawnRequest, ResolveInput, ToastOptions, UiCopyArgs, UiLogOptions,
} from 'claude-code'
@@ -171,6 +171,10 @@ export function uiAsk($: EngineInterface, question: string, options?: readonly s
return $.ui.ask(question, options)
}
+export function uiBlit($: EngineInterface, args: ImageBlitArgs) {
+ return $.ui.blit(args)
+}
+
export function uiClose($: EngineInterface, pane: PaneCloseArgs) {
return $.ui.close(pane)
}
diff --git a/hooks/lib/blit.ts b/hooks/lib/blit.ts
new file mode 100644
index 0000000..e3105ad
--- /dev/null
+++ b/hooks/lib/blit.ts
@@ -0,0 +1,306 @@
+// Picture select's highlight (select.ts): each page stays one keyed Image
+// whose layout never changes, and a selection swaps its picture with
+// `$.ui.blit` (one small command to the terminal, the same image id). The
+// frame is the page with the selected cells over the selection colour: in
+// a selected cell each page pixel (straight alpha) over the opaque colour,
+// elsewhere the page as it is. A clear swaps the page's own PNG back.
+//
+// The page is decoded once, the first time a selection reaches it (png.ts:
+// the hooks have no zlib), and kept in memory (FRAMES_KEPT_BYTES at most).
+// Its rows are deflate pieces made once (plain, and tinted the first time a
+// whole row is selected); a frame makes only the selection's first and last
+// rows (partly selected) and joins the pieces: about 1.5 MB for a page of
+// 3392 x 4284. The hooks can write only text, so perl writes the frame
+// (base64 on its stdin) to a file of its own, which the terminal reads.
+import type { EngineInterface } from 'claude-code'
+
+import type { IuiTheme } from '../../types'
+import { drawnAnswers, S } from './state'
+import { chunk, concat, decodePng, deflatePiece, fromBase64, headOf, tailOf, toBase64, type Piece, type Rgba } from './png'
+import type { DrawnPage } from './copy'
+import type { SelRow } from './select'
+import { perlFor } from './host'
+import { traceDebug } from './log'
+import { clockAfter, clockNow, fsRead, processRun, uiBlit, uiLog } from './api'
+
+export type Rgb = readonly [number, number, number]
+
+// VS Code's selection colour of the mode.
+export const selectColor = (mode: IuiTheme) => (mode === 'light' ? '#add6ff' : '#264f78')
+export const rgbOf = (hex: string): Rgb => [1, 3, 5].map(i => parseInt(hex.slice(i, i + 2), 16)) as unknown as Rgb
+
+// The pixel edges of `cells` cells over `px` pixels: cell i is pixels
+// edges[i] to edges[i + 1] (exclusive). Each edge is rounded once, so the
+// cells tile the picture with no gap or overlap, whatever the ratio.
+export function edgesOf(px: number, cells: number): number[] {
+ return Array.from({ length: cells + 1 }, (_, i) => Math.round((i * px) / cells))
+}
+
+// One channel of a page pixel of alpha `a` over the opaque colour channel `c`.
+export const over = (p: number, a: number, c: number) => Math.floor((p * a + c * (255 - a) + 127) / 255)
+
+// Pixel rows y0 to y1 (exclusive) of `img` as PNG scanlines (filter 0),
+// pixels x0 to x1 (exclusive) over the colour.
+export function bandOf(img: Rgba, y0: number, y1: number, x0: number, x1: number, color: Rgb): Uint8Array {
+ const stride = img.width * 4
+ const out = new Uint8Array((y1 - y0) * (stride + 1))
+ const [r, g, b] = color
+ for (let y = y0; y < y1; y++) {
+ const o = (y - y0) * (stride + 1) + 1
+ out.set(img.data.subarray(y * stride, (y + 1) * stride), o)
+ for (let p = o + x0 * 4, end = o + x1 * 4; p < end; p += 4) {
+ const a = out[p + 3]!
+ if (a === 255) continue
+ out[p] = over(out[p]!, a, r)
+ out[p + 1] = over(out[p + 1]!, a, g)
+ out[p + 2] = over(out[p + 2]!, a, b)
+ out[p + 3] = 255
+ }
+ }
+ return out
+}
+
+// A piece with its IDAT chunk (the CRC is made once).
+type Chunked = { piece: Piece; chunk: Uint8Array }
+const chunked = (raw: Uint8Array, line: number): Chunked => {
+ const piece = deflatePiece(raw, 4, line)
+ return { piece, chunk: chunk('IDAT', piece.bytes) }
+}
+
+// A decoded page ready for frames: its pixels, the cells' pixel edges, the
+// plain row pieces, and the tinted ones made so far.
+export type Frames = { img: Rgba; xs: number[]; ys: number[]; color: Rgb; head: Uint8Array; plain: Chunked[]; tint: (Chunked | undefined)[] }
+
+export function framesOf(img: Rgba, box: { columns: number; rows: number }, color: Rgb): Frames {
+ const xs = edgesOf(img.width, box.columns)
+ const ys = edgesOf(img.height, box.rows)
+ const line = img.width * 4 + 1
+ const plain = Array.from({ length: box.rows }, (_, r) => chunked(bandOf(img, ys[r]!, ys[r + 1]!, 0, 0, color), line))
+ return { img, xs, ys, color, head: headOf(img.width, img.height), plain, tint: [] }
+}
+
+// The frame PNG of a page with `rows` selected (cells x0 to x1, inclusive).
+export function frameOf(f: Frames, rows: readonly SelRow[]): Uint8Array {
+ const line = f.img.width * 4 + 1
+ const columns = f.xs.length - 1
+ const at = new Map(rows.map(r => [r.row, r]))
+ const parts = f.plain.map((plain, r) => {
+ const sel = at.get(r)
+ if (!sel || sel.x1 < sel.x0) return plain
+ const [y0, y1] = [f.ys[r]!, f.ys[r + 1]!]
+ if (sel.x0 <= 0 && sel.x1 >= columns - 1) return (f.tint[r] ??= chunked(bandOf(f.img, y0, y1, 0, f.img.width, f.color), line))
+ return chunked(bandOf(f.img, y0, y1, f.xs[Math.max(0, sel.x0)]!, f.xs[Math.min(columns, sel.x1 + 1)]!, f.color), line)
+ })
+ return concat([f.head, ...parts.map(p => p.chunk), tailOf(parts.map(p => p.piece))])
+}
+
+// What a page shows, as a string: '' its own PNG, else the colour and rows.
+export const sigOf = (rows: readonly SelRow[], color: string) => (rows.length ? `${color}|${rows.map(r => `${r.row}:${r.x0}-${r.x1}`).join(',')}` : '')
+
+// ---- Throttle -------------------------------------------------------------------------
+
+// The fold of the blits: a run (a frame made and blitted) at most at once
+// and at most once a gap, so the posts between fold into one run that reads
+// the newest selection. `last` is the last run's start.
+export type Fold = { isRunning: boolean; isDirty: boolean; last: number }
+export const foldNew = (): Fold => ({ isRunning: false, isDirty: false, last: -Infinity })
+
+// A poke: `run` now, `wait` that long then run (the fold counts as running
+// meanwhile), or `fold` into the run that goes on (one more run after it).
+export function pokeFold(f: Fold, now: number, gapMs: number): { kind: 'run' } | { kind: 'wait'; ms: number } | { kind: 'fold' } {
+ if (f.isRunning) {
+ f.isDirty = true
+ return { kind: 'fold' }
+ }
+ f.isRunning = true
+ const ms = f.last + gapMs - now
+ return ms > 0 ? { kind: 'wait', ms } : { kind: 'run' }
+}
+
+// A run starts (at `now`), and ends: whether a poke came while it ran.
+export function startFold(f: Fold, now: number) {
+ f.isRunning = true
+ f.isDirty = false
+ f.last = now
+}
+export function endFold(f: Fold): boolean {
+ f.isRunning = false
+ return f.isDirty
+}
+
+// ---- The blits -------------------------------------------------------------------------
+
+// One blit frame a screen frame at most.
+export const FRAME_MS = 16
+// The decoded pages kept (their pixels), by bytes: about two of the tallest.
+export const FRAMES_KEPT_BYTES = 128 * 1024 * 1024
+// The frame files kept: the terminal reads each after its blit, at the
+// engine's next frame; older ones go once they are this old.
+const FILES_KEPT = 4
+const FILE_KEEP_MS = 2000
+
+// Writes stdin's base64 to the file (its folder made, a temporary name,
+// then renamed), and deletes the older frames named after it.
+export const WRITE_PL = 'use MIME::Base64; local $/; my $b = <STDIN>; my ($f, @old) = @ARGV; my ($d) = $f =~ m{^(.*)/}; mkdir($d, 0700) unless -d $d; open(my $o, ">:raw", "$f.tmp") or die "$!\\n"; print $o decode_base64($b); close($o) or die "$!\\n"; rename("$f.tmp", $f) or die "$!\\n"; unlink(@old)'
+
+// Each drawing's site (the requestId its Images are drawn under) and its
+// Images' key prefix, noted by drawPages; what each page shows now (sigOf,
+// by `<id>#<page>`); the decoded pages; the frame files written; the
+// reasons logged; pages whose blit was refused (none until the next down).
+export const blits = {
+ sites: new Map<string, { requestId: string; keys: string }>(),
+ shown: new Map<string, string>(),
+ frames: new Map<string, Frames | null>(),
+ files: [] as { path: string; at: number }[],
+ logged: new Set<string>(),
+ refused: new Set<string>(),
+ seq: 0,
+ dir: '',
+}
+
+// A drawing's site, at each draw of its pages.
+export function noteSite(id: string, requestId: string, keys: string) {
+ const was = blits.sites.get(id)
+ if (was?.requestId === requestId && was.keys === keys) return
+ blits.sites.set(id, { requestId, keys })
+ for (const k of [...blits.shown.keys()]) if (k.startsWith(`${id}#`)) blits.shown.delete(k)
+}
+
+// Forgets everything (for tests).
+export function forgetBlits() {
+ for (const m of [blits.sites, blits.shown, blits.frames, blits.logged, blits.refused, decoding]) m.clear()
+ blits.files = []
+ blits.seq = 0
+ blits.dir = ''
+}
+
+const ms = (t: number) => `${Math.round(t)}ms`
+
+// Logs a reason once (the debug log, and the trace).
+function logOnce($: EngineInterface, text: string) {
+ traceDebug($, `select: ${text}`)
+ if (blits.logged.has(text)) return
+ blits.logged.add(text)
+ void Promise.resolve(uiLog($, `Intelligent UI: picture select: ${text}`, { to: 'debug' })).catch(() => undefined)
+}
+
+// A page's frames: decoded once (null: it can not be, not tried again); a
+// decode that runs now is shared.
+const decoding = new Map<string, Promise<{ frames: Frames | null; ms: number }>>()
+function framesFor($: EngineInterface, page: DrawnPage, color: string): Promise<{ frames: Frames | null; ms: number }> {
+ const key = `${page.file}|${page.columns}x${page.rows}|${color}`
+ const kept = blits.frames.get(key)
+ if (kept !== undefined) {
+ blits.frames.delete(key)
+ blits.frames.set(key, kept)
+ return Promise.resolve({ frames: kept, ms: 0 })
+ }
+ const running = decoding.get(key)
+ if (running) return running
+ const run = decodeFrames($, page, color, key).finally(() => decoding.delete(key))
+ decoding.set(key, run)
+ return run
+}
+
+async function decodeFrames($: EngineInterface, page: DrawnPage, color: string, key: string): Promise<{ frames: Frames | null; ms: number }> {
+ const t = performance.now()
+ let frames: Frames | null = null
+ try {
+ const { base64 } = await fsRead($, page.file, { as: 'bytes' })
+ frames = framesOf(decodePng(fromBase64(base64)), page, rgbOf(color))
+ } catch (err) {
+ logOnce($, `no highlight for ${page.file}: ${err instanceof Error ? err.message : String(err)}`)
+ }
+ blits.frames.set(key, frames)
+ let bytes = [...blits.frames.values()].reduce((n, f) => n + (f?.img.data.length ?? 0), 0)
+ for (const [k, f] of blits.frames) {
+ if (bytes <= FRAMES_KEPT_BYTES || k === key) continue
+ bytes -= f?.img.data.length ?? 0
+ blits.frames.delete(k)
+ }
+ const took = performance.now() - t
+ if (frames) traceDebug($, `select: decoded ${page.file} ${frames.img.width}x${frames.img.height} in ${ms(took)}`)
+ return { frames, ms: took }
+}
+
+// Decodes a page before its first frame (at a down), in the background.
+export function warmPage($: EngineInterface, page: DrawnPage | undefined) {
+ if (page && S.env) clockAfter($, 0, () => void framesFor($, page, selectColor(S.theme)).catch(() => undefined))
+}
+
+// The newest frame file's name, and the old ones that can go.
+function nextFile(cache: string, now: number): { path: string; old: string[] } {
+ blits.dir ||= `${cache}/select-${now.toString(36)}${Math.floor(Math.random() * 1e6).toString(36)}`
+ const path = `${blits.dir}/f-${++blits.seq}.png`
+ const keep = blits.files.slice(-FILES_KEPT)
+ const old = blits.files.filter(f => !keep.includes(f) && now - f.at >= FILE_KEEP_MS)
+ blits.files = blits.files.filter(f => !old.includes(f))
+ blits.files.push({ path, at: now })
+ return { path, old: old.map(f => f.path) }
+}
+
+// Shows one page: its own PNG back, or a frame of `rows`.
+async function showPage($: EngineInterface, id: string, i: number, page: DrawnPage, rows: readonly SelRow[], sig: string): Promise<void> {
+ const k = `${id}#${i}`
+ const site = blits.sites.get(id)
+ const env = S.env
+ if (!site || !env) return
+ const key = `${site.keys}p${i}`
+ const rowsText = rows.length ? `rows ${rows[0]!.row}..${rows[rows.length - 1]!.row} (${rows.length})` : 'plain'
+ let source: { file: string; format: 'png' } = { file: page.file, format: 'png' }
+ let timing = ''
+ if (rows.length) {
+ const got = await framesFor($, page, selectColor(S.theme))
+ if (!got.frames) {
+ blits.shown.set(k, sig)
+ return
+ }
+ const now = await clockNow($).catch(() => Date.now())
+ const t1 = performance.now()
+ const png = frameOf(got.frames, rows)
+ const t2 = performance.now()
+ const { path, old } = nextFile(env.cache, now)
+ const ran = await processRun($, [perlFor(env.isMac), '-e', WRITE_PL, path, ...old], { stdin: toBase64(png) }).catch(err => ({ exitCode: 1, stderr: String(err) }))
+ const t3 = performance.now()
+ if (ran.exitCode !== 0) {
+ logOnce($, `the frame could not be written: ${ran.stderr.trim()}`)
+ return
+ }
+ source = { file: path, format: 'png' }
+ timing = `decode ${ms(got.ms)} frame ${ms(t2 - t1)} write ${ms(t3 - t2)} ${png.length} bytes`
+ }
+ const t4 = performance.now()
+ const res = await uiBlit($, { requestId: site.requestId, key, source }).catch(err => ({ deny: String(err) }))
+ const blit = `blit ${ms(performance.now() - t4)}`
+ if (res.deny) {
+ blits.shown.delete(k)
+ blits.refused.add(k)
+ logOnce($, `the engine refused the highlight's blit (${res.deny})`)
+ return
+ }
+ blits.shown.set(k, sig)
+ traceDebug($, `select: blit ${id} page ${i} ${rowsText} ${timing ? `${timing} ` : ''}${blit}`)
+}
+
+// Brings every page to the selection now: the selected ones to their
+// frames, the ones no longer selected back to their PNG. `selected` gives
+// a drawing's pages and rows (or none).
+export async function showSelection($: EngineInterface, selected: { id: string; pages: readonly DrawnPage[]; rows: readonly (readonly SelRow[])[] } | null) {
+ const color = selectColor(S.theme)
+ const want = new Map<string, { id: string; i: number; page: DrawnPage; rows: readonly SelRow[] }>()
+ if (selected) selected.pages.forEach((page, i) => want.set(`${selected.id}#${i}`, { id: selected.id, i, page, rows: selected.rows[i] ?? [] }))
+ // Pages shown with a highlight before, of this drawing or another.
+ for (const k of blits.shown.keys()) {
+ if (want.has(k) || !blits.shown.get(k)) continue
+ const [id, n] = [k.slice(0, k.lastIndexOf('#')), Number(k.slice(k.lastIndexOf('#') + 1))]
+ const page = drawnAnswers.get(id)?.pages[n]
+ if (page) want.set(k, { id, i: n, page, rows: [] })
+ else blits.shown.delete(k)
+ }
+ for (const [k, w] of want) {
+ const sig = sigOf(w.rows, color)
+ if ((blits.shown.get(k) ?? '') === sig || blits.refused.has(k)) continue
+ await showPage($, w.id, w.i, w.page, w.rows, sig)
+ }
+}
+
diff --git a/hooks/lib/draw.tsx b/hooks/lib/draw.tsx
index f35cbad..dbc3054 100644
--- a/hooks/lib/draw.tsx
+++ b/hooks/lib/draw.tsx
@@ -11,8 +11,8 @@ import { DEFAULT_GRID, gridFirstRow, type GridMetrics } from './grid'
import { FENCE, imagePathOf } from './markdown'
import { linksOf, type LinkItem } from './links'
import { fitBox, fitsInline, roomColumns } from './terminal'
-import { overlayRows, picked, redrawSelection, selectColor, selectedRows, selectOn, stripFile, stripPlan, type SelRow } from './select'
-import { cutStrips, hasStrips } from './strips'
+import { selectOn } from './select'
+import { noteSite } from './blit'
import { fsExists, fsRead, uiCopy, uiResolve, uiToast } from './api'
// FNV-1a over the parts, twice with different offsets: 16 hex digits.
@@ -181,7 +181,8 @@ export async function inlineOf($: EngineInterface, result: Result): Promise<stri
// first line's row), `gutter` its width (0: none), `marginTop` the rows above,
// `keys` a prefix for its Images' and Buttons' keys (several sets in one
// tree), `links` the link rows under the pages (one row each), `id` the
-// drawing's id in drawnAnswers (its pages take picture select).
+// drawing's id in drawnAnswers (its pages take picture select, its Images
+// swapped by key under the render's requestId: blit.ts).
export type PagesLayout = { bullet?: boolean; gutter?: number; marginTop?: number; keys?: string; links?: readonly ShownLink[]; id?: string }
// The bullet's row when the page is drawn `factor` as wide.
@@ -195,7 +196,7 @@ export function scaledBulletRow(bulletRow: number, factor: number): number {
// the room now (an old one in an open epoch) draws smaller to fit, the
// bullet scaled too: the engine shifts a wider Image, it does not clip it.
export function drawPages(
- $: EngineInterface, e: ResolveInput<RenderComponent, 'terminal'> & { viewport?: RenderViewport }, result: Result, inline?: readonly string[], layout: PagesLayout = {},
+ $: EngineInterface, e: ResolveInput<RenderComponent, 'terminal'> & { viewport?: RenderViewport; requestId?: string }, result: Result, inline?: readonly string[], layout: PagesLayout = {},
) {
const table = uiResolve($, e)
const { Box, Text, Image } = table
@@ -207,12 +208,14 @@ export function drawPages(
const factor = first && first.columns > room ? room / first.columns : 1
const keys = layout.keys ?? ''
const boxes = result.pages.map(page => fitBox(page, room))
- // Picture select: each page in a Box of its size, the selected rows over
- // it, and last the Client that takes the pointer (select.ts). Both in the
- // flow, pulled up over the page (marginTop), never absolute (overlayRows).
+ // Picture select: each page in a Box of its size with the Client that
+ // takes the pointer over it, in the flow, pulled up over the page
+ // (marginTop): never absolute (the engine moves an absolute Box whose top
+ // is above the screen down to row 0, its pointer too). The highlight
+ // swaps the Image's picture (blit.ts); the layout stays.
const id = layout.id
- const cell = id && !inline ? selectOn({ hasClient: !!Client, isFullscreen: e.viewport?.isFullscreen }) : null
- const sel = id && cell && picked.now?.id === id ? selectedRows(picked.now, boxes) : null
+ const isSelect = !!id && !inline && !!Client && selectOn({ hasClient: true, isFullscreen: e.viewport?.isFullscreen })
+ if (isSelect && e.requestId) noteSite(id, e.requestId, keys)
return (
<Box flexDirection="row" marginTop={layout.marginTop ?? MARGIN_TOP}>
{gutter > 0 ? (
@@ -233,14 +236,10 @@ export function drawPages(
alt={page.alt}
/>
)
- const plan = cell ? stripPlan(page, box, cell) : null
- if (!plan || !Client || !id) return image
+ if (!isSelect || !Client || !id) return image
return (
<Box key={`${keys}w${i}`} flexDirection="column" width={box.columns} height={box.rows} flexShrink={0}>
{image}
- <Box key={`${keys}o${i}`} flexDirection="column" marginTop={-box.rows} width={box.columns} height={box.rows} flexShrink={0}>
- {drawSelected($, e, page, { ...plan, columns: box.columns }, sel?.[i] ?? [], `${keys}${i}`)}
- </Box>
<Box key={`${keys}c${i}`} marginTop={-box.rows} width={box.columns} height={box.rows} flexShrink={0}>
<Client key={`${keys}sel${i}`} module="../sel.tsx" width={box.columns} height={box.rows} props={{ id, page: i }} />
</Box>
@@ -254,31 +253,6 @@ export function drawPages(
)
}
-// The selected rows of a page ("clip, do not crop"): each row's strip (the
-// page's own row) in a Box of the selected columns on the selection colour,
-// shifted left so its cells stay over the page's. The page is transparent,
-// so the colour shows under its text. Each row in the flow (overlayRows).
-// Nothing while the strips are cut: they draw once there.
-function drawSelected(
- $: EngineInterface, e: ResolveInput<RenderComponent, 'terminal'>, page: { file: string; width: number; height: number },
- plan: { rowPx: number; rows: number; columns: number }, rows: readonly SelRow[], keys: string,
-) {
- if (!rows.length || !S.env) return null
- if (!hasStrips(page.file, plan.rowPx)) {
- cutStrips($, S.env, page, plan, () => redrawSelection($))
- return null
- }
- const { Box, Image } = uiResolve($, e)
- const bg = selectColor(S.theme, picked.gen)
- return overlayRows(rows).map(r => (
- <Box key={`${keys}r${r.row}`} marginTop={r.top} marginLeft={r.x0} width={r.x1 - r.x0 + 1} height={1} flexShrink={0} overflow="hidden" backgroundColor={bg}>
- <Box marginLeft={-r.x0} width={plan.columns} flexShrink={0}>
- <Image key={`${keys}s${r.row}`} source={{ file: stripFile(page.file, plan.rows, r.row), format: 'png' }} columns={plan.columns} rows={1} alt=" " />
- </Box>
- </Box>
- ))
-}
-
// ---- Links ------------------------------------------------------------------
// A click on a picture cannot open a link (an Image is a leaf: no Link holds
// it, and a press on it gives no cell). So the links, file paths and commands
diff --git a/hooks/lib/png.ts b/hooks/lib/png.ts
new file mode 100644
index 0000000..3a57513
--- /dev/null
+++ b/hooks/lib/png.ts
@@ -0,0 +1,447 @@
+// PNG in plain JS for picture select's highlight (blit.ts): the hooks
+// runtime has no zlib, so a page is decoded here once (inflate, unfilter),
+// and each frame is a PNG of pieces: one raw deflate piece per page row
+// (fixed Huffman, matches one pixel left or one pixel up), each made alone
+// and ended on a byte, so any pieces join into one stream. Typst's pages
+// are 8-bit RGBA (or RGB), not interlaced; anything else is refused.
+
+// A picture's straight-alpha RGBA pixels, row after row.
+export type Rgba = { width: number; height: number; data: Uint8Array }
+
+// ---- Checksums -----------------------------------------------------------------
+
+const CRC_TABLE = (() => {
+ const t = new Uint32Array(256)
+ for (let n = 0; n < 256; n++) {
+ let c = n
+ for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1
+ t[n] = c >>> 0
+ }
+ return t
+})()
+
+// CRC-32 of `bytes` (PNG chunks), going on from `crc`.
+export function crc32(bytes: Uint8Array, crc = 0): number {
+ let c = ~crc >>> 0
+ for (let i = 0; i < bytes.length; i++) c = CRC_TABLE[(c ^ bytes[i]!) & 0xff]! ^ (c >>> 8)
+ return ~c >>> 0
+}
+
+const BASE = 65521
+// Adler-32 of `bytes` (a zlib stream's check).
+export function adler32(bytes: Uint8Array): number {
+ let a = 1
+ let b = 0
+ for (let i = 0; i < bytes.length; ) {
+ const end = Math.min(bytes.length, i + 5552)
+ for (; i < end; i++) b += a += bytes[i]!
+ a %= BASE
+ b %= BASE
+ }
+ return ((b << 16) | a) >>> 0
+}
+
+// The Adler-32 of two runs joined, from each one's and the second's length
+// (zlib's adler32_combine).
+export function adlerJoin(a1: number, a2: number, len2: number): number {
+ const rem = len2 % BASE
+ let s1 = a1 & 0xffff
+ let s2 = (rem * s1) % BASE
+ s1 += (a2 & 0xffff) + BASE - 1
+ s2 += ((a1 >>> 16) & 0xffff) + ((a2 >>> 16) & 0xffff) + BASE - rem
+ if (s1 >= BASE) s1 -= BASE
+ if (s1 >= BASE) s1 -= BASE
+ if (s2 >= BASE * 2) s2 -= BASE * 2
+ if (s2 >= BASE) s2 -= BASE
+ return ((s2 << 16) | s1) >>> 0
+}
+
+// ---- Inflate (RFC 1950, 1951) ------------------------------------------------------
+
+const LEN_BASE = [3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 17, 19, 23, 27, 31, 35, 43, 51, 59, 67, 83, 99, 115, 131, 163, 195, 227, 258]
+const LEN_EXTRA = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 0]
+const DIST_BASE = [1, 2, 3, 4, 5, 7, 9, 13, 17, 25, 33, 49, 65, 97, 129, 193, 257, 385, 513, 769, 1025, 1537, 2049, 3073, 4097, 6145, 8193, 12289, 16385, 24577]
+const DIST_EXTRA = [0, 0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13]
+const CL_ORDER = [16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15]
+
+// A Huffman decode table: indexed by the next `bits` input bits, each entry
+// the symbol << 4 | its code length (0: no code).
+type Table = { table: Int32Array; bits: number }
+
+function tableOf(lens: ArrayLike<number>, n: number): Table {
+ let bits = 0
+ const count = new Uint16Array(16)
+ for (let i = 0; i < n; i++) {
+ count[lens[i]!]!++
+ if (lens[i]! > bits) bits = lens[i]!
+ }
+ count[0] = 0
+ const next = new Uint16Array(16)
+ for (let len = 1, code = 0; len < 16; len++) next[len] = code = (code + count[len - 1]!) << 1
+ const size = 1 << Math.max(1, bits)
+ const table = new Int32Array(size)
+ for (let sym = 0; sym < n; sym++) {
+ const len = lens[sym]!
+ if (!len) continue
+ let code = next[len]!++
+ let rev = 0
+ for (let k = 0; k < len; k++, code >>= 1) rev = (rev << 1) | (code & 1)
+ for (let i = rev; i < size; i += 1 << len) table[i] = (sym << 4) | len
+ }
+ return { table, bits: Math.max(1, bits) }
+}
+
+const FIXED_LIT = tableOf(Array.from({ length: 288 }, (_, i) => (i < 144 ? 8 : i < 256 ? 9 : i < 280 ? 7 : 8)), 288)
+const FIXED_DIST = tableOf(new Array<number>(32).fill(5), 32)
+
+// Inflates the zlib stream `src` into `out` (its size known); returns the
+// bytes written. Throws on a stream that is not one.
+export function inflate(src: Uint8Array, out: Uint8Array): number {
+ if (src.length < 2 || (src[0]! & 0x0f) !== 8 || ((src[0]! << 8) | src[1]!) % 31 !== 0 || src[1]! & 0x20) throw new Error('not a zlib stream')
+ let pos = 2
+ let bb = 0
+ let bc = 0
+ let op = 0
+ const bitsOf = (n: number) => {
+ while (bc < n) {
+ bb |= (src[pos++] ?? 0) << bc
+ bc += 8
+ }
+ const v = bb & ((1 << n) - 1)
+ bb >>>= n
+ bc -= n
+ return v
+ }
+ const symOf = (t: Table) => {
+ while (bc < t.bits) {
+ bb |= (src[pos++] ?? 0) << bc
+ bc += 8
+ }
+ const e = t.table[bb & ((1 << t.bits) - 1)]!
+ const len = e & 15
+ if (!len) throw new Error('bad Huffman code')
+ bb >>>= len
+ bc -= len
+ return e >> 4
+ }
+ for (let last = 0; !last; ) {
+ last = bitsOf(1)
+ const type = bitsOf(2)
+ if (type === 0) {
+ pos -= bc >> 3
+ bb = bc = 0
+ const len = src[pos]! | (src[pos + 1]! << 8)
+ pos += 4
+ if (pos + len > src.length || op + len > out.length) throw new Error('stored block past the end')
+ out.set(src.subarray(pos, pos + len), op)
+ pos += len
+ op += len
+ continue
+ }
+ let lit = FIXED_LIT
+ let dist = FIXED_DIST
+ if (type === 2) {
+ const hlit = bitsOf(5) + 257
+ const hdist = bitsOf(5) + 1
+ const hclen = bitsOf(4) + 4
+ const cl = new Uint8Array(19)
+ for (let i = 0; i < hclen; i++) cl[CL_ORDER[i]!] = bitsOf(3)
+ const clt = tableOf(cl, 19)
+ const lens = new Uint8Array(hlit + hdist)
+ for (let i = 0; i < hlit + hdist; ) {
+ const sym = symOf(clt)
+ if (sym < 16) lens[i++] = sym
+ else {
+ const [rep, prev] = sym === 16 ? [3 + bitsOf(2), i ? lens[i - 1]! : -1] : sym === 17 ? [3 + bitsOf(3), 0] : [11 + bitsOf(7), 0]
+ if (prev < 0 || i + rep > hlit + hdist) throw new Error('bad code lengths')
+ lens.fill(prev, i, i + rep)
+ i += rep
+ }
+ }
+ lit = tableOf(lens.subarray(0, hlit), hlit)
+ dist = tableOf(lens.subarray(hlit), hdist)
+ } else if (type !== 1) throw new Error('bad block type')
+ // The hot loop, inlined: refill, look up, consume.
+ const lt = lit.table
+ const lb = lit.bits
+ const lm = (1 << lb) - 1
+ const dt = dist.table
+ const db = dist.bits
+ const dm = (1 << db) - 1
+ for (;;) {
+ while (bc < lb) {
+ bb |= (src[pos++] ?? 0) << bc
+ bc += 8
+ }
+ let e = lt[bb & lm]!
+ let n = e & 15
+ if (!n) throw new Error('bad Huffman code')
+ bb >>>= n
+ bc -= n
+ const sym = e >> 4
+ if (sym < 256) {
+ if (op >= out.length) throw new Error('output past the end')
+ out[op++] = sym
+ continue
+ }
+ if (sym === 256) break
+ const li = sym - 257
+ if (li >= 29) throw new Error('bad length code')
+ let len = LEN_BASE[li]!
+ n = LEN_EXTRA[li]!
+ while (bc < 15) {
+ bb |= (src[pos++] ?? 0) << bc
+ bc += 8
+ }
+ if (n) {
+ len += bb & ((1 << n) - 1)
+ bb >>>= n
+ bc -= n
+ }
+ while (bc < db) {
+ bb |= (src[pos++] ?? 0) << bc
+ bc += 8
+ }
+ e = dt[bb & dm]!
+ n = e & 15
+ if (!n) throw new Error('bad Huffman code')
+ bb >>>= n
+ bc -= n
+ const di = e >> 4
+ if (di >= 30) throw new Error('bad distance code')
+ n = DIST_EXTRA[di]!
+ while (bc < n) {
+ bb |= (src[pos++] ?? 0) << bc
+ bc += 8
+ }
+ const d = DIST_BASE[di]! + (bb & ((1 << n) - 1))
+ bb >>>= n
+ bc -= n
+ if (d > op || op + len > out.length) throw new Error('bad match')
+ if (d >= len) {
+ out.copyWithin(op, op - d, op - d + len)
+ op += len
+ } else for (const end = op + len; op < end; op++) out[op] = out[op - d]!
+ }
+ }
+ return op
+}
+
+// ---- Decode ------------------------------------------------------------------------
+
+const SIGNATURE = [137, 80, 78, 71, 13, 10, 26, 10]
+const u32 = (b: Uint8Array, i: number) => ((b[i]! << 24) | (b[i + 1]! << 16) | (b[i + 2]! << 8) | b[i + 3]!) >>> 0
+
+// A PNG's pixels as RGBA. Throws on one that is not 8-bit RGBA or RGB, not
+// interlaced, or is broken.
+export function decodePng(png: Uint8Array): Rgba {
+ if (png.length < 33 || SIGNATURE.some((v, i) => png[i] !== v)) throw new Error('not a PNG')
+ let width = 0
+ let height = 0
+ let channels = 0
+ const idat: Uint8Array[] = []
+ for (let i = 8; i + 8 <= png.length; ) {
+ const len = u32(png, i)
+ const type = String.fromCharCode(png[i + 4]!, png[i + 5]!, png[i + 6]!, png[i + 7]!)
+ const data = png.subarray(i + 8, i + 8 + len)
+ if (type === 'IHDR') {
+ width = u32(data, 0)
+ height = u32(data, 4)
+ const [depth, color, , , interlace] = data.subarray(8, 13)
+ channels = color === 6 ? 4 : color === 2 ? 3 : 0
+ if (depth !== 8 || !channels || interlace) throw new Error(`a PNG of depth ${depth}, color type ${color}, interlace ${interlace}`)
+ } else if (type === 'IDAT') idat.push(data)
+ else if (type === 'IEND') break
+ i += 12 + len
+ }
+ if (!width || !height || !idat.length) throw new Error('no IHDR or IDAT')
+ const z = new Uint8Array(idat.reduce((n, d) => n + d.length, 0))
+ idat.reduce((at, d) => (z.set(d, at), at + d.length), 0)
+ const stride = width * channels
+ const filtered = new Uint8Array(height * (stride + 1))
+ if (inflate(z, filtered) !== filtered.length) throw new Error('short image data')
+ const rows = unfilter(filtered, width, height, channels)
+ if (channels === 4) return { width, height, data: rows }
+ const data = new Uint8Array(width * height * 4)
+ for (let p = 0, q = 0; p < rows.length; p += 3, q += 4) {
+ data[q] = rows[p]!
+ data[q + 1] = rows[p + 1]!
+ data[q + 2] = rows[p + 2]!
+ data[q + 3] = 255
+ }
+ return { width, height, data }
+}
+
+// The scanlines' filters undone (PNG filter types 0 to 4).
+function unfilter(f: Uint8Array, width: number, height: number, bpp: number): Uint8Array {
+ const stride = width * bpp
+ const out = new Uint8Array(height * stride)
+ for (let y = 0; y < height; y++) {
+ const type = f[y * (stride + 1)]!
+ const s = y * (stride + 1) + 1
+ const o = y * stride
+ const u = o - stride
+ // The first pixel has no left neighbour (a and c are 0).
+ if (type === 0) out.set(f.subarray(s, s + stride), o)
+ else if (type === 1) {
+ for (let x = 0; x < bpp; x++) out[o + x] = f[s + x]!
+ for (let x = bpp; x < stride; x++) out[o + x] = f[s + x]! + out[o + x - bpp]!
+ } else if (!y) {
+ // The row above is 0: Up is None, Average halves the left, Paeth is Sub.
+ for (let x = 0; x < bpp; x++) out[o + x] = f[s + x]!
+ for (let x = bpp; x < stride; x++) out[o + x] = f[s + x]! + (type === 2 ? 0 : type === 3 ? out[o + x - bpp]! >> 1 : out[o + x - bpp]!)
+ } else if (type === 2) for (let x = 0; x < stride; x++) out[o + x] = f[s + x]! + out[u + x]!
+ else if (type === 3) {
+ for (let x = 0; x < bpp; x++) out[o + x] = f[s + x]! + (out[u + x]! >> 1)
+ for (let x = bpp; x < stride; x++) out[o + x] = f[s + x]! + ((out[o + x - bpp]! + out[u + x]!) >> 1)
+ } else if (type === 4) {
+ for (let x = 0; x < bpp; x++) out[o + x] = f[s + x]! + out[u + x]!
+ for (let x = bpp; x < stride; x++) {
+ const a = out[o + x - bpp]!
+ const b = out[u + x]!
+ const c = out[u + x - bpp]!
+ let pa = b - c
+ let pb = a - c
+ let pc = pa + pb
+ if (pa < 0) pa = -pa
+ if (pb < 0) pb = -pb
+ if (pc < 0) pc = -pc
+ out[o + x] = f[s + x]! + (pa <= pb && pa <= pc ? a : pb <= pc ? b : c)
+ }
+ } else throw new Error(`bad filter type ${type}`)
+ }
+ return out
+}
+
+// ---- Deflate pieces -------------------------------------------------------------------
+
+// The fixed Huffman codes (bit-reversed, as deflate writes them) and lengths.
+const FIXED_CODE = new Uint16Array(288)
+const FIXED_BITS = new Uint8Array(288)
+for (let sym = 0; sym < 288; sym++) {
+ const [len, code] = sym < 144 ? [8, 0x30 + sym] : sym < 256 ? [9, 0x190 + sym - 144] : sym < 280 ? [7, sym - 256] : [8, 0xc0 + sym - 280]
+ let rev = 0
+ for (let k = 0, c = code; k < len; k++, c >>= 1) rev = (rev << 1) | (c & 1)
+ FIXED_CODE[sym] = rev
+ FIXED_BITS[sym] = len
+}
+// For each match length 3 to 258: its code's index (into LEN_BASE).
+const LEN_INDEX = new Uint8Array(259)
+for (let i = 0; i < 29; i++) for (let l = LEN_BASE[i]!; l < (i < 28 ? LEN_BASE[i + 1]! : 259); l++) LEN_INDEX[l] = i
+const reverse5 = (v: number) => ((v & 1) << 4) | ((v & 2) << 2) | (v & 4) | ((v & 8) >> 2) | ((v & 16) >> 4)
+
+// A distance's code bits, written as one: the 5-bit code then its extra bits.
+function distBits(d: number): { value: number; bits: number } {
+ let i = 29
+ while (DIST_BASE[i]! > d) i--
+ return { value: reverse5(i) | ((d - DIST_BASE[i]!) << 5), bits: 5 + DIST_EXTRA[i]! }
+}
+
+// One piece of a stream: its compressed bytes (no final block, ended on a
+// byte), and the Adler-32 and length of what it holds.
+export type Piece = { bytes: Uint8Array; adler: number; length: number }
+
+// `raw` (filtered scanlines) as a piece: fixed Huffman blocks whose matches
+// are one pixel back (`bpp`) or one scanline up (`line`), never before the
+// piece's start, then an empty stored block (a sync flush).
+export function deflatePiece(raw: Uint8Array, bpp: number, line: number): Piece {
+ const out = new Uint8Array(Math.ceil(raw.length * 1.15) + 64)
+ let op = 0
+ let bb = 0
+ let bc = 0
+ const put = (value: number, bits: number) => {
+ bb |= value << bc
+ bc += bits
+ while (bc >= 8) {
+ out[op++] = bb & 0xff
+ bb >>>= 8
+ bc -= 8
+ }
+ }
+ const near = distBits(bpp)
+ const up = line <= 32768 ? distBits(line) : null
+ // A block holds at most this many symbols (any size is valid; keeps it small).
+ put(0b010, 3)
+ for (let i = 0; i < raw.length; ) {
+ const max = Math.min(258, raw.length - i)
+ let best = 0
+ let dist = near
+ if (i >= bpp && max >= 3) {
+ let n = 0
+ while (n < max && raw[i + n] === raw[i + n - bpp]) n++
+ best = n
+ }
+ if (up && i >= line && max >= 3 && best < max) {
+ let n = 0
+ while (n < max && raw[i + n] === raw[i + n - line]) n++
+ if (n > best) {
+ best = n
+ dist = up
+ }
+ }
+ if (best >= 3) {
+ const li = LEN_INDEX[best]!
+ put(FIXED_CODE[257 + li]!, FIXED_BITS[257 + li]!)
+ if (LEN_EXTRA[li]) put(best - LEN_BASE[li]!, LEN_EXTRA[li]!)
+ put(dist.value & 0xffff, Math.min(16, dist.bits))
+ if (dist.bits > 16) put(dist.value >>> 16, dist.bits - 16)
+ i += best
+ } else {
+ put(FIXED_CODE[raw[i]!]!, FIXED_BITS[raw[i]!]!)
+ i++
+ }
+ }
+ put(FIXED_CODE[256]!, FIXED_BITS[256]!)
+ // An empty stored block: its header, then up to the byte, LEN 0, NLEN ~0.
+ put(0, 3)
+ if (bc) put(0, 8 - bc)
+ out.set([0, 0, 0xff, 0xff], op)
+ op += 4
+ return { bytes: out.slice(0, op), adler: adler32(raw), length: raw.length }
+}
+
+// ---- Assemble -------------------------------------------------------------------------
+
+const ascii = (s: string) => Uint8Array.from(s, c => c.charCodeAt(0))
+const be32 = (v: number) => Uint8Array.of(v >>> 24, (v >>> 16) & 255, (v >>> 8) & 255, v & 255)
+
+// One PNG chunk: length, type, data, CRC.
+export function chunk(type: string, data: Uint8Array): Uint8Array {
+ const t = ascii(type)
+ const out = new Uint8Array(12 + data.length)
+ out.set(be32(data.length), 0)
+ out.set(t, 4)
+ out.set(data, 8)
+ out.set(be32(crc32(data, crc32(t))), 8 + data.length)
+ return out
+}
+
+// The head of an RGBA PNG of `width` x `height` (signature, IHDR, the zlib
+// header's IDAT): what every frame starts with.
+export function headOf(width: number, height: number): Uint8Array {
+ const ihdr = new Uint8Array(13)
+ ihdr.set(be32(width), 0)
+ ihdr.set(be32(height), 4)
+ ihdr.set([8, 6, 0, 0, 0], 8)
+ return concat([Uint8Array.from(SIGNATURE), chunk('IHDR', ihdr), chunk('IDAT', Uint8Array.of(0x78, 0x01))])
+}
+
+// The tail after the pieces: the final empty block and the Adler-32 of all
+// the pieces, then IEND.
+export function tailOf(pieces: readonly Piece[]): Uint8Array {
+ const adler = pieces.reduce((a, p) => adlerJoin(a, p.adler, p.length), 1)
+ const end = new Uint8Array(6)
+ end.set([0x03, 0x00], 0)
+ end.set(be32(adler), 2)
+ return concat([chunk('IDAT', end), chunk('IEND', new Uint8Array(0))])
+}
+
+// Uint8Array's base64 methods: the runtime has them, the TypeScript lib not yet.
+export const fromBase64 = (text: string) => (Uint8Array as unknown as { fromBase64(t: string): Uint8Array }).fromBase64(text)
+export const toBase64 = (bytes: Uint8Array) => (bytes as unknown as { toBase64(): string }).toBase64()
+
+export function concat(parts: readonly Uint8Array[]): Uint8Array {
+ const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0))
+ parts.reduce((at, p) => (out.set(p, at), at + p.length), 0)
+ return out
+}
diff --git a/hooks/lib/select.ts b/hooks/lib/select.ts
index 16b3bed..e9eff3a 100644
--- a/hooks/lib/select.ts
+++ b/hooks/lib/select.ts
@@ -2,18 +2,17 @@
// or a Markdown file Claude sent. Since Claude Code 2.1.305 its own
// selection skips an Image's cells (no highlight, no text), so a Client
// (hooks/sel.tsx) over each page takes the pointer and posts the selection;
-// the hooks module draws it over the page and copies its text on the
-// button's up. The highlight is "clip, do not crop": each selected row is
-// the page's own row (a strip PNG, strips.ts) on the selection colour, cut
-// to the selected columns. The pure parts are here, with the selection now.
+// the hooks module shows it on the page and copies its text on the
+// button's up. The highlight swaps the page's picture (blit.ts); the page's
+// layout never changes. The pure parts are here, with the selection now.
import type { EngineInterface } from 'claude-code'
-import type { IuiTheme } from '../../types'
import { SELECT_SINCE } from './settings'
-import { S, type Cell } from './state'
+import { drawnAnswers, S } from './state'
import { compareVersions } from './updates'
import { joinSegments, pageTopsOf, textOfRun, type Drawn, type Item, type RunLine, type TextBlock } from './copy'
-import { invalidate } from './api'
+import { endFold, foldNew, FRAME_MS, pokeFold, showSelection, startFold } from './blit'
+import { clockAfter } from './api'
export type Point = { x: number; y: number }
export type SelectPhase = 'down' | 'move' | 'up' | 'clear'
@@ -32,19 +31,35 @@ export type SelRow = { row: number; x0: number; x1: number }
type Box = { columns: number; rows: number }
// The one selection over the pictures (null: none), the order of the last
-// post taken, its redraws (`gen`: the overlay colour toggles with it), and
-// a Client that failed (`isBroken`: no select until a reload).
-export const picked = { now: null as Selection | null, seq: 0, gen: 0, isBroken: false }
+// post taken, and a Client that failed (`isBroken`: no select until a reload).
+export const picked = { now: null as Selection | null, seq: 0, isBroken: false }
-// Draws the selection again (its colour one step off: selectColor).
+// Shows the selection now on the pages (blit.ts), at most once a frame: the
+// posts between fold into one, the newest.
+let fold = foldNew()
export function redrawSelection($: EngineInterface) {
- picked.gen++
- invalidate($, 'ui.render')
+ const next = pokeFold(fold, performance.now(), FRAME_MS)
+ if (next.kind === 'wait') clockAfter($, next.ms, () => void runSelection($))
+ else if (next.kind === 'run') void runSelection($)
}
-// The selection's part of a drawing's memo: its redraw count while the
-// selection is on it, '' otherwise.
-export const selStamp = (id: string) => (picked.now?.id === id ? `sel${picked.gen}` : '')
+async function runSelection($: EngineInterface) {
+ startFold(fold, performance.now())
+ await showSelection($, selectedNow()).catch(() => undefined)
+ if (endFold(fold)) redrawSelection($)
+}
+
+// The selection's drawing, its pages and their selected rows (none: null).
+function selectedNow() {
+ const sel = picked.now
+ const d = sel ? drawnAnswers.get(sel.id) : undefined
+ return sel && d ? { id: sel.id, pages: d.pages, rows: selectedRows(sel, d.pages) } : null
+}
+
+// Forgets the fold (for tests).
+export function forgetRedraws() {
+ fold = foldNew()
+}
const isInt = (v: unknown): v is number => typeof v === 'number' && Number.isInteger(v)
const pointOf = (v: unknown): Point | null => {
@@ -106,20 +121,6 @@ export function selectedRows(sel: { anchor: Point; head: Point }, pages: readonl
return out
}
-// The selected rows as the overlay draws them, in the flow of a Box over
-// the page: each row `top` rows under the end of the one before (the first
-// under the page's top). Not absolute Boxes: the engine moves an absolute
-// Box whose top is above the screen down to row 0, children and pointer
-// too, so a page scrolled part away had its rows and its Client too high.
-export function overlayRows(rows: readonly SelRow[]): (SelRow & { top: number })[] {
- let next = 0
- return rows.map(r => {
- const top = r.row - next
- next = r.row + 1
- return { ...r, top }
- })
-}
-
// The rows of a page scrolled away above the viewport: the message's first
// row on screen less the page's top in the message, 0 to the page's rows.
// Undefined when either is not known.
@@ -168,32 +169,6 @@ export function textOfRuns(
return joinSegments(runs.map(run => ({ kind: 'image', lines: run.lines })), texts)
}
-// The strips of a page drawn at `box` in cells of `cell`: one PNG per cell
-// row, `rowPx` tall, each drawn as a 1-row Image of `box.columns`. The
-// terminal fits an image in its box with the aspect kept, so a strip shows
-// its row of the page only when both are at one scale with no space above
-// or below: the page is exactly `box.rows` x rowPx (a whole number) and
-// fills its box's height (its width at most the box's at that scale).
-// Null otherwise (a page drawn smaller, a cell of a fraction of a pixel):
-// that page has no select.
-export function stripPlan(page: { width: number; height: number }, box: Box, cell: Cell): { rowPx: number; rows: number } | null {
- if (box.rows < 1 || box.columns < 1 || page.width < 1) return null
- const rowPx = page.height / box.rows
- if (!Number.isInteger(rowPx)) return null
- return page.width * cell.height <= rowPx * box.columns * cell.width + 1e-6 ? { rowPx, rows: box.rows } : null
-}
-
-// The strips' folder of a page (beside it in its key folder, so the cache
-// cleanup takes them with it), and the file of row `row` (0-based) as
-// Typst names its pages: 1-based, padded to the count's digits.
-export const stripDir = (file: string) => file.replace(/\.png$/, '') + '.strips'
-export const stripFile = (file: string, rows: number, row: number) => `${stripDir(file)}/s-${String(row + 1).padStart(String(rows).length, '0')}.png`
-
-// The selection's background: VS Code's selection colour of the mode, one
-// step off on every other redraw (the engine repaints an overlay whole only
-// when it changes).
-export const selectColor = (mode: IuiTheme, gen: number) => (mode === 'light' ? (gen % 2 ? '#add6ff' : '#add6fe') : gen % 2 ? '#264f78' : '#264f79')
-
// Whether Claude Code is a version whose selection skips pictures
// (SELECT_SINCE or newer; a dev build by its release). Before it, its own
// selection copies them, and a Client over a picture would take that away.
@@ -205,8 +180,7 @@ export function needsSelect(version: string): boolean {
// Whether a drawing's pages take picture select now: `/iui select` on, in
// fullscreen (on the main screen the terminal selects), pages drawn from
// files (not as bytes over ssh), the Client element there and a Claude Code
-// that needs it, with the tty's cell known (the strips' scale).
-export function selectOn(o: { hasClient: boolean; isFullscreen?: boolean }): Cell | null {
- if (!S.select || picked.isBroken || !o.hasClient || o.isFullscreen !== true || !S.env?.canDraw || S.env.inline || !needsSelect(S.engine)) return null
- return S.tty?.cell ?? null
+// that needs it.
+export function selectOn(o: { hasClient: boolean; isFullscreen?: boolean }): boolean {
+ return S.select && !picked.isBroken && o.hasClient && o.isFullscreen === true && !!S.env?.canDraw && !S.env.inline && needsSelect(S.engine)
}
diff --git a/hooks/lib/strips.ts b/hooks/lib/strips.ts
deleted file mode 100644
index b726605..0000000
--- a/hooks/lib/strips.ts
+++ /dev/null
@@ -1,71 +0,0 @@
-// Picture select's strips (select.ts): a page cut into one PNG per row by
-// Typst (render/strips.typ, about 0.2 s a page), once, the first time a
-// selection reaches it. They are kept beside the page in its key folder
-// (stripDir), so the cache cleanup takes them with it, and a folder cut
-// before is used at once. A PNG's rows can not be cut out of its bytes (its
-// filters run across rows), so Typst draws each row again, pixel for pixel.
-import type { EngineInterface } from 'claude-code'
-
-import { COMPILE_TIMEOUT_MS } from './settings'
-import { keep, type Env } from './state'
-import { stripDir, stripFile } from './select'
-import { logDebug, noNetworkEnv } from './typst'
-import { fsExists, processRun } from './api'
-
-// The Typst run that cuts `file` (`width` x `height` px) into rows of
-// `rowPx`, into its strips' folder (made first).
-export function stripsCall(o: { typst: string; root: string; cache: string; file: string; width: number; height: number; rowPx: number }): { argv: string[]; env: Record<string, string> } {
- const argv = [
- 'sh', '-c', 'mkdir -p "$1" && shift && exec "$@"', 'sh', stripDir(o.file),
- o.typst, 'compile',
- '--root', '/',
- '--format', 'png',
- '--diagnostic-format', 'short',
- '--ppi', '72',
- '--input', `src=${o.file}`,
- '--input', `w=${o.width}`,
- '--input', `h=${o.height}`,
- '--input', `row=${o.rowPx}`,
- `${o.root}/render/strips.typ`,
- `${stripDir(o.file)}/s-{0p}.png`,
- ]
- return { argv, env: noNetworkEnv(o.cache) }
-}
-
-// The pages whose strips are there (true) or failed (false), and the cuts
-// that run now, by `<file>@<rowPx>`.
-const STRIPS_KEPT = 256
-const cut = new Map<string, boolean>()
-const cutting = new Map<string, Promise<boolean>>()
-
-const keyOf = (file: string, rowPx: number) => `${file}@${rowPx}`
-
-// Whether a page's strips are there (cut in this load, or found).
-export const hasStrips = (file: string, rowPx: number) => cut.get(keyOf(file, rowPx)) === true
-
-// Cuts a page's strips once (a run that failed is not tried again in this
-// load); `done` runs when they are there. Never throws.
-export function cutStrips($: EngineInterface, env: Env, page: { file: string; width: number; height: number }, plan: { rowPx: number; rows: number }, done: () => void) {
- const key = keyOf(page.file, plan.rowPx)
- if (cut.has(key) || cutting.has(key) || !env.typst) return
- const typst = env.typst
- const run = (async () => {
- if (await fsExists($, stripFile(page.file, plan.rows, plan.rows - 1)).catch(() => false)) return true
- const { argv, env: vars } = stripsCall({ typst, root: env.root, cache: env.cache, file: page.file, width: page.width, height: page.height, rowPx: plan.rowPx })
- const ran = await processRun($, argv, { env: vars, timeoutMs: COMPILE_TIMEOUT_MS })
- if (ran.exitCode !== 0) await logDebug($, env.cache, `strips of ${page.file} failed:\n${ran.stderr}`)
- return ran.exitCode === 0
- })().catch(() => false)
- cutting.set(key, run)
- void run.then(isCut => {
- cutting.delete(key)
- keep(cut, key, isCut, STRIPS_KEPT)
- if (isCut) done()
- })
-}
-
-// Forgets the strips cut (for tests).
-export function forgetStrips() {
- cut.clear()
- cutting.clear()
-}
diff --git a/hooks/register.tsx b/hooks/register.tsx
index 03330ac..dbdad9e 100644
--- a/hooks/register.tsx
+++ b/hooks/register.tsx
@@ -33,8 +33,8 @@ import { runSetup } from './lib/setup'
import { pollHost, PROBE_RUNS, probeOsc11, showMode, USE_OSC11, watchHost } from './lib/host'
import { measureFonts } from './lib/typst'
import { drawPages, linksFor, type ShownLink } from './lib/draw'
-import { picked, postOf, redrawSelection, runsOf, selectedRows, selectionAfter, selectTrace, selStamp, stripPlan, textOfRuns, topsOf, type SelectPost, type Selection } from './lib/select'
-import { cutStrips } from './lib/strips'
+import { picked, postOf, redrawSelection, runsOf, selectedRows, selectionAfter, selectTrace, textOfRuns, topsOf, type SelectPost, type Selection } from './lib/select'
+import { blits, warmPage } from './lib/blit'
import { traceDebug } from './lib/log'
import { finalizeAll } from './lib/stream'
import { noteDrawn, noteDrawnAgain, noteOnScreen, wordsOf, type PageWords } from './lib/probe'
@@ -161,9 +161,8 @@ const stampOf = (ready: { env: Env; done: number }, view?: RenderViewport) =>
// one pointer), `extra` its other inputs.
type Memo<T> = { text: string; extra: string; stamp: string; onScreen: boolean | undefined; value: T }
// An answer's tree is kept with its link rows (the copy counts them under
-// its pages), its pages and its selection's stamp (selStamp: a new one
-// draws the tree again from the pages), a plan's drawing with its link rows.
-type AnswerMemo = { tree: RenderElement; links: readonly ShownLink[]; pages: { result: Result; bytes?: readonly string[] }; sel: string }
+// its pages) and its pages, a plan's drawing with its link rows.
+type AnswerMemo = { tree: RenderElement; links: readonly ShownLink[]; pages: { result: Result; bytes?: readonly string[] } }
const answers = new Map<string, Memo<AnswerMemo>>()
const plans = new Map<string, Memo<{ drawing: Drawing; links: ShownLink[] }>>()
const unders = new Map<string, Memo<Under[]>>()
@@ -493,13 +492,14 @@ async function dump($: EngineInterface, on: boolean): Promise<string> {
}
// ---- Picture select -------------------------------------------------------
-// The Client over each page (hooks/sel.tsx) posts the selection; it is drawn
-// over the pages (lib/draw.tsx) and its text copied on the button's up, read
-// as a shim's copy is (wordsOf, textOfRun). See lib/select.ts.
+// The Client over each page (hooks/sel.tsx) posts the selection; the pages
+// show it (a blit of each page's picture, lib/blit.ts) and its text is
+// copied on the button's up, read as a shim's copy is (wordsOf, textOfRun).
+// See lib/select.ts.
-// A Client's post: the selection after it, drawn again (the redraw takes
-// the newest selection, so posts that come faster than redraws merge). A
-// down cuts its page's strips now, so they are there for the drag.
+// A Client's post: the selection after it, shown (once a frame at most, the
+// newest selection). A down decodes its page now, for the drag's frames,
+// and lets a page whose blit was refused try again.
function onSelect($: EngineInterface, post: SelectPost, surface: RenderSurface) {
if (post.seq <= picked.seq) return
picked.seq = post.seq
@@ -507,13 +507,11 @@ function onSelect($: EngineInterface, post: SelectPost, surface: RenderSurface)
const was = picked.now
picked.now = d ? selectionAfter(post, topsOf(d.pages)) : null
if (S.debug) traceDebug($, selectTrace(post, d, picked.now))
- if (was || picked.now) redrawSelection($)
- const page = d?.pages[post.page]
- const cell = S.tty?.cell
- if (post.phase === 'down' && page && cell && S.env) {
- const plan = stripPlan(page, page, cell)
- if (plan) cutStrips($, S.env, page, plan, () => redrawSelection($))
+ if (post.phase === 'down') {
+ blits.refused.clear()
+ warmPage($, d?.pages[post.page])
}
+ if (was || picked.now) redrawSelection($)
// The copy waits for the page's words (a probe): the next posts do not.
if (post.phase === 'up' && d && picked.now) void copySelection($, d, picked.now, surface).catch(() => undefined)
}
@@ -848,11 +846,6 @@ export const register: Register = on => {
noteDrawnAgain(e.requestId, roomColumns(e.viewport), { top: MARGIN_TOP, after: kept.links.length })
noteOnScreen(e.requestId, e.props.onScreen)
})
- const sel = selStamp(e.requestId)
- if (kept.sel !== sel) {
- kept.tree = drawPages($, e, kept.pages.result, kept.pages.bytes, { bullet, links: kept.links, id: e.requestId })
- kept.sel = sel
- }
return kept.tree
}
const drawing = await pagesOf($, { id: e.requestId, text: e.props.text, frame: frameFor(e.viewport), env: ready.env, canStream: true, onScreen })
@@ -864,7 +857,7 @@ export const register: Register = on => {
noteDrawn(e.requestId, result, roomColumns(e.viewport), { top: MARGIN_TOP, after: links.length })
noteOnScreen(e.requestId, e.props.onScreen)
})
- memoSet(answers, e.requestId, k, { tree, links, pages: { result, ...(drawing.bytes ? { bytes: drawing.bytes } : {}) }, sel: selStamp(e.requestId) })
+ memoSet(answers, e.requestId, k, { tree, links, pages: { result, ...(drawing.bytes ? { bytes: drawing.bytes } : {}) } })
return tree
}
answers.delete(e.requestId)
diff --git a/render/strips.typ b/render/strips.typ
deleted file mode 100644
index f1a49ab..0000000
--- a/render/strips.typ
+++ /dev/null
@@ -1,16 +0,0 @@
-// Intelligent UI: a page PNG cut into one PNG per terminal row, for picture
-// select (hooks/lib/strips.ts). At --ppi 72, 1 pt is 1 pixel, so each strip
-// is the page's own pixels, unscaled.
-//
-// typst compile --root / --ppi 72 --input src=<page.png> --input w=<px>
-// --input h=<px> --input row=<px> <render>/strips.typ <dir>/s-{0p}.png
-#let src = sys.inputs.at("src")
-#let w = int(sys.inputs.at("w"))
-#let h = int(sys.inputs.at("h"))
-#let row = int(sys.inputs.at("row"))
-#set page(width: w * 1pt, height: row * 1pt, margin: 0pt, fill: none)
-#let img = image(src, width: w * 1pt, height: h * 1pt, scaling: "pixelated")
-#for k in range(calc.ceil(h / row)) {
- if k > 0 { pagebreak() }
- box(width: w * 1pt, height: row * 1pt, clip: true, place(dy: -k * row * 1pt, img))
-}
diff --git a/tests/blit.test.ts b/tests/blit.test.ts
new file mode 100644
index 0000000..816d86c
--- /dev/null
+++ b/tests/blit.test.ts
@@ -0,0 +1,172 @@
+import { expect, test } from 'claude-code/testing'
+
+import { bandOf, edgesOf, endFold, foldNew, frameOf, framesOf, over, pokeFold, rgbOf, selectColor, sigOf, startFold } from '../hooks/lib/blit'
+import { adler32, adlerJoin, concat, crc32, decodePng, deflatePiece, fromBase64, headOf, inflate, tailOf, type Rgba } from '../hooks/lib/png'
+
+// ---- PNG ---------------------------------------------------------------------------
+
+// A 64 x 10 RGBA PNG made by zlib (dynamic Huffman blocks, two IDAT chunks),
+// row y with filter y % 5 (None, Sub, Up, Average, Paeth); its pixels are pixelOf.
+const FIXTURE = [
+ 'iVBORw0KGgoAAAANSUhEUgAAAEAAAAAKCAYAAAAEqljUAAACYUlEQVR42uXVf0yUBRzH8TuPh+sePO7Bu1MOumNwwnHKxQ71iE6CSw67mKgRIh',
+ 'oSirIbihKRomH5o1SmdlhgJhlOcIgTNNFJKS11KlObutL5I01HGtOc5NTynPZ2ydYfzj8q/5Lt2YvPvnx4GBsfZDI+lHLBGBwgvqRXaXKel3Tv',
+ 'mA2G6kEDTS12u/nwi6mxv6aMtQnphQlRoyoSU97wDX9z4iZXxeT96bXeixnbZ8nGHJsdmf3b/LQJ4ofefMtyX2HaJ+3egs+7SuavDylb25Q6Z1',
+ 'draeWPOxsW/r7n9BLNfml53GFPtefE4tppp79bu+hnWX39FVfjnuuLm8/c6my9c09q0yly2+1iQ0dmSM++4tDUQ0sjqo82xnSd2GtLOnVhqO/c',
+ 'PWf3RcMraZcdnvVXs8bIbszKKbi1YtL+u81TrQ8OTvcFdJX9qZLPLQw2LTiudS5NDR3/8TZjee3AqFVfrLG0NkhxRzYvs3d/FeAIbF/gNMvVck',
+ 'GhlPuNSoXGqBRQiSoMQjVqMAS1qMcBaMBwNGIERqIZo9GCVhyMNoxHOw7BYZiISejEZExBF45AN45ED2bgKByNYzELszEHc3Ei5mE+FuAUnIpF',
+ '6MVinIEzsRTLsBxnYwXOw0q/sc/DX4BaruHp4fErnrWsCOsjBopyvyT2kUmiAgN0kihgIFmJz5FVfp8okoPIfclqDCZr0EcOwX4yn6hFHVmP/b',
+ 'mvxlCyAcPI4XyfOrKRbCJH+KWqenIkOYpsxgZyNMaQLdhEtuIg8maM0/lEG7aQ43ErdzsmkLfj0E8lcRjv2UlOJO/inoRfk53Q35KvAAACYklE',
+ 'QVS4m5yMHfy8KX4p4O8/AZFHePT89I/Pe5+nc1f/b99feMLd9sS+zCUYNK8G6uJGq8yeccGmaXl6YVGhSVZfbNXsedshnqlwu+58MG64bok3w7',
+ '5yfnpmTY2tuK4ldumGzsTG5ssJe7cJ3gu7YgrvfespMxwoKXEcrcnO+qFjzKyz3fkrLvWf0Nztrj54493lXXc2rZXfP19rEvSVzr6Zc8Zrly0p',
+ 'DzuwcFWk0N4am952JH7Zvm7H9x2ByfpGc9qk+tSMTa15r99unpvrvrj6rdpzbUXdV4+XvHz5ennN0aD3rh+KXeQ55a7aeGJytaB6/7NpAXVfdm',
+ 'rbN74QfHJLza2bbfdvSN94H9j2nrz7WqfbXnRsZ9ziU1bn+vPrHB2/6I1nr60M/eOmyqK/+1FUgjwz0NTv4X8BpaKHEUQlqjAI1ahxMYKoJetx',
+ 'ABowHI0YwT0SzeRotKAVB6MN47nbcQh5GCZiEjoxGVO4u3AE2Y0j0YMZOApHcx+LWeRszMFcnIh5mM+9AKeQp2IRerEYZ+BM7qVYRi7H2ViB87',
+ 'BS6B3B3nF49lQ4hQHBolxkBDWMIAMRYGAETYygmRGMZQR3MIKXJFG8wgheYwR7GMHbjCBDomFYfDZGUPSJ/ehr6evo6+n3N0tVq+mH0jfQD7vi',
+ 'E8Pp19E30jfRj6BfTz+S90fRNzNwDfSj6cfwfgv9JvpW+oN4/2b6cT2MIP0W+vH0t25hBOkn0N/O+4eukcQd63yiYwMj2MQI0k+azAgWMYLTpa',
+ 'rdpYwg/Q76KbLeEfQ/GorH+e/u6v/Yfxp39WO+/i/KnT1vncz3VAAAAABJRU5ErkJggg==',
+].join('')
+const pixelOf = (x: number, y: number) => [(x * 7 + y * 13) & 255, (x * x + y) & 255, ((x ^ y) * 5) & 255, (x * 29 + y * 3) & 255]
+const imageOf = (width: number, height: number, at: (x: number, y: number) => number[]): Rgba => {
+ const data = new Uint8Array(width * height * 4)
+ for (let y = 0; y < height; y++) for (let x = 0; x < width; x++) data.set(at(x, y), (y * width + x) * 4)
+ return { width, height, data }
+}
+
+test('a PNG of every filter and dynamic Huffman blocks decodes to its pixels', () => {
+ const img = decodePng(fromBase64(FIXTURE))
+ expect([img.width, img.height]).toEqual([64, 10])
+ expect([...img.data]).toEqual([...imageOf(64, 10, pixelOf).data])
+})
+
+test('a PNG that is not 8-bit RGBA or RGB, or not a PNG, is refused', () => {
+ const png = fromBase64(FIXTURE)
+ const gray = png.slice()
+ gray[25] = 0
+ expect(() => decodePng(gray)).toThrow('color type 0')
+ expect(() => decodePng(new Uint8Array(40))).toThrow('not a PNG')
+})
+
+test('the checksums: CRC-32 and Adler-32 as zlib makes them; two Adler-32 runs join as one', () => {
+ const bytes = Uint8Array.from('The quick brown fox', c => c.charCodeAt(0))
+ expect(crc32(bytes)).toBe(0xb74574de)
+ expect(adler32(bytes)).toBe(0x452e0714)
+ const [a, b] = [bytes.subarray(0, 7), bytes.subarray(7)]
+ expect(adlerJoin(adler32(a), adler32(b), b.length)).toBe(adler32(bytes))
+ const big = new Uint8Array(100_000).map((_, i) => (i * 31) & 255)
+ expect(adlerJoin(adler32(big.subarray(0, 40_000)), adler32(big.subarray(40_000)), 60_000)).toBe(adler32(big))
+})
+
+test('deflate pieces join into one stream: each alone, ended on a byte, inflated as they were', () => {
+ const line = 4 * 50 + 1
+ const raws = [0, 1, 2].map(k => new Uint8Array(line * 3).map((_, i) => (i % line === 0 ? 0 : i % 7 < 3 ? 9 : (i * k) & 255)))
+ const pieces = raws.map(raw => deflatePiece(raw, 4, line))
+ // Long runs come out small.
+ expect(deflatePiece(new Uint8Array(line * 30), 4, line).bytes.length).toBeLessThan(line)
+ const stream = concat([Uint8Array.of(0x78, 0x01), ...pieces.map(p => p.bytes), Uint8Array.of(0x03, 0x00)])
+ const out = new Uint8Array(line * 9)
+ expect(inflate(stream, out)).toBe(out.length)
+ expect([...out]).toEqual([...concat(raws)])
+})
+
+// ---- Cells to pixels -------------------------------------------------------------
+
+test('cell edges: whole pixels a cell are exact; any other ratio tiles the picture with no gap or overlap', () => {
+ // Ghostty, 16 x 36 px cells: 212 columns, 53 rows.
+ expect(edgesOf(3392, 212).slice(0, 4)).toEqual([0, 16, 32, 48])
+ expect(edgesOf(1908, 53).at(-1)).toBe(1908)
+ expect(edgesOf(100, 3)).toEqual([0, 33, 67, 100])
+ // A page drawn smaller than its own cells (150 columns for 3392 px).
+ const xs = edgesOf(3392, 150)
+ expect(xs[0]).toBe(0)
+ expect(xs.at(-1)).toBe(3392)
+ for (let i = 0; i < 150; i++) expect(xs[i + 1]! - xs[i]!).toBeGreaterThanOrEqual(22)
+})
+
+// ---- The frame ---------------------------------------------------------------------
+
+// 8 x 4 px, 4 columns and 2 rows of 2 x 2 px cells: alpha by column 0, 128, 255, 64.
+const ALPHA = [0, 128, 255, 64]
+const SMALL = imageOf(8, 4, (x, y) => [10 * x, 20 * y, 200, ALPHA[x >> 1]!])
+const COLOR = rgbOf('#264f78')
+
+test('the selection colour is VS Code\'s of the mode', () => {
+ expect(selectColor('dark')).toBe('#264f78')
+ expect(selectColor('light')).toBe('#add6ff')
+ expect(rgbOf('#264f78')).toEqual([0x26, 0x4f, 0x78])
+})
+
+test('a pixel over the colour: straight alpha over an opaque colour, rounded', () => {
+ expect(over(200, 0, 0x26)).toBe(0x26)
+ expect(over(200, 255, 0x26)).toBe(200)
+ expect(over(200, 128, 0)).toBe(100)
+ expect(over(0, 128, 255)).toBe(127)
+})
+
+test('a frame: the selected cells over the colour (opaque), every other pixel as it was', () => {
+ const f = framesOf(SMALL, { columns: 4, rows: 2 }, COLOR)
+ // Row 0 columns 1 to 2, row 1 whole.
+ const img = decodePng(frameOf(f, [{ row: 0, x0: 1, x1: 2 }, { row: 1, x0: 0, x1: 3 }]))
+ expect([img.width, img.height]).toEqual([8, 4])
+ for (let y = 0; y < 4; y++) {
+ for (let x = 0; x < 8; x++) {
+ const p = [...img.data.subarray((y * 8 + x) * 4, (y * 8 + x) * 4 + 4)]
+ const was = [...SMALL.data.subarray((y * 8 + x) * 4, (y * 8 + x) * 4 + 4)]
+ const isSelected = y >= 2 || (x >= 2 && x < 6)
+ const a = was[3]!
+ expect(p).toEqual(isSelected ? [over(was[0]!, a, COLOR[0]), over(was[1]!, a, COLOR[1]), over(was[2]!, a, COLOR[2]), 255] : was)
+ }
+ }
+})
+
+test('a frame with no rows selected is the page as it is (a clear)', () => {
+ const f = framesOf(SMALL, { columns: 4, rows: 2 }, COLOR)
+ expect([...decodePng(frameOf(f, [])).data]).toEqual([...SMALL.data])
+ // A whole row's tinted piece is made once and kept.
+ frameOf(f, [{ row: 1, x0: 0, x1: 3 }])
+ const kept = f.tint[1]
+ frameOf(f, [{ row: 0, x0: 2, x1: 3 }, { row: 1, x0: 0, x1: 3 }])
+ expect(f.tint[1]).toBe(kept)
+ expect(f.tint[0]).toBe(undefined)
+})
+
+test('a band is the rows as filter-0 scanlines, its pixels from x0 to x1 over the colour', () => {
+ const band = bandOf(SMALL, 1, 3, 2, 4, COLOR)
+ expect(band.length).toBe(2 * 33)
+ expect(band[0]).toBe(0)
+ expect([...band.subarray(1, 9)]).toEqual([...SMALL.data.subarray(32, 40)])
+ expect([...band.subarray(9, 13)]).toEqual([over(20, 128, 0x26), over(20, 128, 0x4f), over(200, 128, 0x78), 255])
+})
+
+test('a frame is a PNG whose head and tail are its own: the right size, the Adler-32 of all its rows', () => {
+ const f = framesOf(SMALL, { columns: 4, rows: 2 }, COLOR)
+ expect([...f.head.subarray(0, 8)]).toEqual([137, 80, 78, 71, 13, 10, 26, 10])
+ expect(headOf(8, 4)).toEqual(f.head)
+ const tail = tailOf(f.plain.map(p => p.piece))
+ const all = concat([bandOf(SMALL, 0, 2, 0, 0, COLOR), bandOf(SMALL, 2, 4, 0, 0, COLOR)])
+ const adler = tail.subarray(10, 14)
+ expect(((adler[0]! << 24) | (adler[1]! << 16) | (adler[2]! << 8) | adler[3]!) >>> 0).toBe(adler32(all))
+})
+
+test('what a page shows: nothing selected is its own PNG, else the colour and the rows', () => {
+ expect(sigOf([], '#264f78')).toBe('')
+ expect(sigOf([{ row: 2, x0: 3, x1: 9 }, { row: 3, x0: 0, x1: 4 }], '#264f78')).toBe('#264f78|2:3-9,3:0-4')
+})
+
+// ---- Throttle ----------------------------------------------------------------------
+
+test('the fold: pokes while a frame is made fold into one more run; a run starts at most once a gap', () => {
+ const f = foldNew()
+ expect(pokeFold(f, 0, 16)).toEqual({ kind: 'run' })
+ startFold(f, 0)
+ // Five pokes while it runs: folded, one more run after it.
+ for (let i = 0; i < 5; i++) expect(pokeFold(f, 3, 16)).toEqual({ kind: 'fold' })
+ expect(endFold(f)).toBe(true)
+ // The run took 5 ms: the next waits out the gap since its start.
+ expect(pokeFold(f, 5, 16)).toEqual({ kind: 'wait', ms: 11 })
+ // While it waits, a poke folds into it.
+ expect(pokeFold(f, 8, 16)).toEqual({ kind: 'fold' })
+ startFold(f, 16)
+ expect(endFold(f)).toBe(false)
+ // A poke after the gap runs at once.
+ expect(pokeFold(f, 40, 16)).toEqual({ kind: 'run' })
+})
diff --git a/tests/copy.test.ts b/tests/copy.test.ts
index 2694211..6f35f98 100644
--- a/tests/copy.test.ts
+++ b/tests/copy.test.ts
@@ -12,6 +12,9 @@ import { HINT_WAIT_MS } from '../hooks/lib/settings'
import { rowsAfter, type Under } from '../hooks/lib/pictures'
import { forgetProbed, noteDrawn, noteDrawnAgain, noteOnScreen, wordsOf } from '../hooks/lib/probe'
import { clip, drawnAnswers, lastKey, results, type Job, type Result } from '../hooks/lib/state'
+import { forgetBlits, over, rgbOf, WRITE_PL } from '../hooks/lib/blit'
+import { forgetRedraws } from '../hooks/lib/select'
+import { chunk, concat, decodePng, deflatePiece, fromBase64, headOf, tailOf, toBase64 } from '../hooks/lib/png'
import { LONG_MARKS, LONG_MD, LONG_PAGE, SMALL_MARKS, SMALL_MD, SMALL_PAGE, SMALL_PPI } from './copy-fixture'
// One screen row of a picture as Claude Code draws it: `lead` (the gutter),
@@ -501,8 +504,12 @@ const inputsOf = (argv: readonly string[]) => Object.fromEntries(argv.flatMap((a
// with no ask (another session's, the terminal's own on the main screen,
// another app's); `change` is one more change with the same text.
// `tty`: the tty size line its read gives (none: no read), `version`
-// Claude Code's (picture select); `strips` the strip cuts run.
-function world(on: any, o: { failFirst?: boolean; noBridge?: boolean; env?: Record<string, string>; tty?: string; version?: string } = {}) {
+// Claude Code's (picture select); `frames` the highlight's frame files
+// written, `blits` the blits asked (`deny`: each refused with it), `logs`
+// the debug log lines.
+function world(on: any, o: { failFirst?: boolean; noBridge?: boolean; env?: Record<string, string>; tty?: string; version?: string; deny?: string } = {}) {
+ forgetBlits()
+ forgetRedraws()
mock.store(on)
const clock = mock.clock(on)
const vars: Record<string, string | undefined> = { PATH: '/usr/bin:/bin', ...(o.env ?? { HOME: '/h', CLAUDE_CODE_FORCE_TERMINAL_IMAGES: '1' }) }
@@ -525,7 +532,8 @@ function world(on: any, o: { failFirst?: boolean; noBridge?: boolean; env?: Reco
const w = {
clock, files, compiles, queries, kills, watchers, readers, vars, envSets,
ended: [] as string[][], answers: [] as string[], late: [] as string[], cleaned: [] as string[], asks: 0, reads: [] as string[],
- clipboard: 'nothing yet', count: 40, pastes: 0, copies: [] as string[], toasts: [] as string[], strips: [] as string[][], traces: [] as string[],
+ clipboard: 'nothing yet', count: 40, pastes: 0, copies: [] as string[], toasts: [] as string[], traces: [] as string[], logs: [] as string[],
+ frames: [] as { path: string; old: string[]; png: Uint8Array }[], blits: [] as any[],
selection: undefined as { text: string; requestId?: string } | undefined,
selects: true,
gate: null as Promise<void> | null, release: () => {},
@@ -646,8 +654,8 @@ function world(on: any, o: { failFirst?: boolean; noBridge?: boolean; env?: Reco
}
if (/cc-intelligent-ui-/.test(cmd)) return ok('/c\nDarwin\n')
if (o.tty && argv[0] === 'sh' && cmd.includes('ioctl(T')) return ok(o.tty)
- if (argv[0] === 'sh' && argv.some(a => a.endsWith('/render/strips.typ'))) {
- w.strips.push(argv)
+ if (argv[0] === '/usr/bin/perl' && argv[2] === WRITE_PL) {
+ w.frames.push({ path: argv[3]!, old: argv.slice(4), png: fromBase64(e.init?.stdin ?? '') })
return ok('')
}
if (argv[0] === 'sh' && cmd.includes('debug.log')) {
@@ -679,6 +687,7 @@ function world(on: any, o: { failFirst?: boolean; noBridge?: boolean; env?: Reco
e.path.startsWith(`${ASK_DIR}/`) && files.has(e.path) ? { value: { kind: 'file', size: files.get(e.path)!.length, mtimeMs: 1, isLink: false } } : next(e))
on('fs.read', async ($: any, e: any, next: any) => {
w.reads.push(e.path)
+ if (e.as === 'bytes' && /^\/c\/.*\.png$/.test(e.path)) return { value: { base64: PAGE_PNG } }
return e.path.endsWith('/hooks/clip-shim.sh') ? { value: SHIM_STUB } : files.has(e.path) ? { value: files.get(e.path) } : e.path.startsWith('/c/') ? { deny: 'ENOENT' } : next(e)
})
// One page per segment of the folder's doc.json.
@@ -694,6 +703,14 @@ function world(on: any, o: { failFirst?: boolean; noBridge?: boolean; env?: Reco
return { value: { isCopied: true } }
})
on('ui.selection', async () => ({ value: w.selection }))
+ on('ui.blit', async ($: any, e: any) => {
+ w.blits.push(e)
+ return { value: o.deny ? { deny: o.deny } : {} }
+ })
+ on('ui.log', async ($: any, e: any) => {
+ w.logs.push(String(e.text))
+ return { value: undefined }
+ })
on('session.version', async () => ({ value: { version: o.version ?? '2.1.306', base: o.version ?? '2.1.306' } }))
on('ui.toast', async ($: any, e: any) => {
w.toasts.push(String(e.text))
@@ -1401,64 +1418,135 @@ test('the fullscreen advice is not shown with Intelligent UI off', async ($, on)
// The tty of the fixture's page: 100 columns (98 for the picture) of 16 x 32 px.
const TTY = '40 100 1600 1280'
const tree = (node: any): any[] => (node && typeof node === 'object' ? [node, ...(node.children ?? []).flatMap(tree)] : [])
-const stripImages = (drawn: any) => tree(drawn).filter(n => n.type === 'Image' && /\.strips\//.test(n.props.source?.file ?? ''))
-// The selected rows drawn: each strip's row (its Box in the flow, its
-// marginTop the rows after the row before), and its Box's columns.
-const selectedOf = (drawn: any) => {
- let next = 0
- return tree(drawn)
- .filter(n => n.type === 'Box' && n.props.overflow === 'hidden' && n.props.height === 1 && n.props.backgroundColor)
- .map(n => {
- const row = next + n.props.marginTop
- next = row + 1
- return { row, x0: n.props.marginLeft, x1: n.props.marginLeft + n.props.width - 1 }
- })
+// The fixture's page, 1568 x 512 px (98 x 16 cells of 16 x 32): ink (opaque)
+// on every third pixel, the rest transparent.
+const pagePixel = (x: number, y: number) => ((x + y) % 3 ? [0, 0, 0, 0] : [200, 210, 220, 255])
+const PAGE_PNG = (() => {
+ const line = 1568 * 4 + 1
+ const raw = new Uint8Array(512 * line)
+ for (let y = 0; y < 512; y++) for (let x = 0; x < 1568; x++) raw.set(pagePixel(x, y), y * line + 1 + x * 4)
+ const piece = deflatePiece(raw, 4, line)
+ return toBase64(concat([headOf(1568, 512), chunk('IDAT', piece.bytes), tailOf([piece])]))
+})()
+// Whether a frame's cell (row, column) is the selection's: a transparent
+// pixel of it is the colour (an opaque one is the same either way).
+const cellOf = (png: Uint8Array, row: number, col: number) => {
+ const img = decodePng(png)
+ const y = row * 32 + 2
+ const x = col * 16 + ((col * 16 + y) % 3 ? 0 : 1)
+ const p = [...img.data.subarray((y * 1568 + x) * 4, (y * 1568 + x) * 4 + 4)]
+ const was = pagePixel(x, y)
+ const c = rgbOf('#264f78')
+ if (p.join() === was.join()) return 'page'
+ return p.join() === [over(was[0]!, was[3]!, c[0]), over(was[1]!, was[3]!, c[1]), over(was[2]!, was[3]!, c[2]), 255].join() ? 'selected' : 'other'
+}
+// The page Box of the drawing: the Image, then the Client pulled up over it.
+const pageBox = (drawn: any) => tree(drawn).find(n => n.type === 'Box' && n.props.flexDirection === 'column' && n.props.width === 98 && n.props.height === 16)
+const settle = async (w: ReturnType<typeof world>) => {
+ for (let i = 0; i < 4; i++) {
+ await w.clock.settle()
+ await w.clock.advance(20)
+ }
+ await w.clock.settle()
}
-test('picture select: a drag over a page draws its rows over it and copies their text on the up', async ($, on) => {
+test('picture select: a drag swaps the page\'s picture for a frame of its selected cells, copies their text on the up; a click puts the page back', async ($, on) => {
const w = world(on, { tty: TTY })
await $.command.run({ command: 'iui', args: '' } as any)
const ui = await drawAnswer($, w)
- const client = tree(await ui.drawn()).find(n => n.type === 'Client')
+ const before = await ui.drawn()
+ const client = tree(before).find(n => n.type === 'Client')
expect(client?.props).toMatchObject({ key: 'sel0', width: 98, height: 16, props: { id: 'ans-1', page: 0 } })
+ // The Image and the Client over it, in the flow: no absolute Box (the
+ // engine moves one scrolled above the screen to row 0), no overlay.
+ const page = pageBox(before)
+ expect(page.children.map((c: any) => c.type)).toEqual(['Image', 'Box'])
+ expect(page.children.map((c: any) => c.props.marginTop)).toEqual([undefined, -16])
+ expect(page.children[0].props.key).toBe('p0')
+ expect(tree(page).some(n => n.props?.position === 'absolute' || n.props?.backgroundColor)).toBe(false)
await ui.pointer({ type: 'down', x: 0, y: 1, button: 'left', in: 'sel0' })
- await w.clock.settle()
- // The page's strips are cut at the down: rows of 32 px.
- expect(w.strips.length).toBe(1)
- expect(w.strips[0]).toContain('row=32')
+ await settle(w)
+ // A down selects nothing yet: no frame.
+ expect(w.frames).toEqual([])
await ui.pointer({ type: 'move', x: 10, y: 2, button: 'left', in: 'sel0' })
await ui.pointer({ type: 'up', x: 21, y: 3, button: 'left', in: 'sel0' })
- await w.clock.settle()
+ await settle(w)
expect(w.copies).toEqual([COPIED])
expect(w.toasts).toContain(`Intelligent UI: copied ${COPIED.length} chars of text`)
- const drawn = await ui.drawn()
- expect(selectedOf(drawn)).toEqual([{ row: 1, x0: 0, x1: 97 }, { row: 2, x0: 0, x1: 97 }, { row: 3, x0: 0, x1: 21 }])
- expect(stripImages(drawn).map(n => n.props.source.file.replace(/^.*\//, ''))).toEqual(['s-02.png', 's-03.png', 's-04.png'])
- expect(stripImages(drawn).every(n => n.props.columns === 98 && n.props.rows === 1)).toBe(true)
- // The rows, then the Client, pulled up over the page in the flow: no
- // absolute Box (the engine moves one scrolled above the screen to row 0).
- const page = tree(drawn).find(n => n.type === 'Box' && n.props.flexDirection === 'column' && n.props.width === 98 && n.props.height === 16)
- expect(page.children.map((c: any) => c.props.marginTop)).toEqual([undefined, -16, -16])
- expect(page.children[2].children[0].type).toBe('Client')
- expect(tree(page).some(n => n.props?.position === 'absolute')).toBe(false)
- // A click with no drag clears it, and copies nothing.
+ // The last blit: the Image p0 under the message's requestId, its newest frame file.
+ const last = w.blits.at(-1)
+ expect(last).toMatchObject({ requestId: 'ans-1', key: 'p0', source: { format: 'png' } })
+ const frame = w.frames.at(-1)!
+ expect(last.source.file).toBe(frame.path)
+ expect(frame.path).toMatch(/^\/c\/select-[0-9a-z]+\/f-\d+\.png$/)
+ // Rows 1 and 2 whole, row 3 to column 21; the rest of the page as it is.
+ expect([cellOf(frame.png, 1, 0), cellOf(frame.png, 1, 97), cellOf(frame.png, 2, 50), cellOf(frame.png, 3, 21)]).toEqual(['selected', 'selected', 'selected', 'selected'])
+ expect([cellOf(frame.png, 0, 5), cellOf(frame.png, 3, 22), cellOf(frame.png, 4, 0), cellOf(frame.png, 15, 97)]).toEqual(['page', 'page', 'page', 'page'])
+ // The layout never changes with the selection.
+ expect(JSON.stringify(pageBox(await ui.drawn()))).toBe(JSON.stringify(page))
+ // A click with no drag clears it: the page's own PNG back, nothing copied.
+ const frames = w.frames.length
await ui.pointer({ type: 'down', x: 5, y: 5, button: 'left', in: 'sel0' })
await ui.pointer({ type: 'up', x: 5, y: 5, button: 'left', in: 'sel0' })
- await w.clock.settle()
- expect(selectedOf(await ui.drawn())).toEqual([])
+ await settle(w)
+ expect(w.blits.at(-1)).toMatchObject({ requestId: 'ans-1', key: 'p0', source: { file: drawnAnswers.get('ans-1')?.pages[0]?.file ?? w.blits.at(-1).source.file, format: 'png' } })
+ expect(w.blits.at(-1).source.file).toMatch(/^\/c\/[0-9a-f]+\/a-1\.png$/)
+ expect(w.frames.length).toBe(frames)
+ expect(w.copies).toEqual([COPIED])
+ // The page is read (decoded) once.
+ expect(w.reads.filter(r => r.endsWith('/a-1.png')).length).toBe(1)
+})
+
+test('picture select: a fast drag folds into few frames, the last one the newest selection', async ($, on) => {
+ const w = world(on, { tty: TTY })
+ await $.command.run({ command: 'iui', args: '' } as any)
+ const ui = await drawAnswer($, w)
+ await ui.pointer({ type: 'down', x: 0, y: 0, button: 'left', in: 'sel0' })
+ await settle(w)
+ for (let i = 1; i <= 15; i++) await ui.pointer({ type: 'move', x: 5 * i, y: i, button: 'left', in: 'sel0' })
+ await settle(w)
+ // Fifteen moves, far fewer frames; the last shows rows 0 to 15 (to column 75).
+ expect(w.frames.length).toBeGreaterThan(0)
+ expect(w.frames.length).toBeLessThan(15)
+ const png = w.frames.at(-1)!.png
+ expect([cellOf(png, 14, 97), cellOf(png, 15, 75), cellOf(png, 15, 76)]).toEqual(['selected', 'selected', 'page'])
+ expect(w.blits.at(-1).source.file).toBe(w.frames.at(-1)!.path)
+ // Old frame files go once four newer ones are there and 2 s have passed.
+ for (let i = 0; i < 6; i++) {
+ await ui.pointer({ type: 'move', x: 3 + i, y: 2, button: 'left', in: 'sel0' })
+ await settle(w)
+ }
+ expect(w.frames.flatMap(f => f.old)).toEqual([])
+ await w.clock.advance(3000)
+ await ui.pointer({ type: 'move', x: 30, y: 2, button: 'left', in: 'sel0' })
+ await settle(w)
+ expect(w.frames.at(-1)!.old.length).toBeGreaterThan(0)
+})
+
+test('picture select: a blit the engine refuses is logged once and shows no highlight; the copy still works', async ($, on) => {
+ const w = world(on, { tty: TTY, deny: 'no Image of its own is mounted under key "p0"' })
+ await $.command.run({ command: 'iui', args: '' } as any)
+ const ui = await drawAnswer($, w)
+ await ui.pointer({ type: 'down', x: 0, y: 1, button: 'left', in: 'sel0' })
+ await ui.pointer({ type: 'move', x: 10, y: 2, button: 'left', in: 'sel0' })
+ await settle(w)
+ await ui.pointer({ type: 'move', x: 15, y: 2, button: 'left', in: 'sel0' })
+ await ui.pointer({ type: 'up', x: 21, y: 3, button: 'left', in: 'sel0' })
+ await settle(w)
+ // One refused blit: the page is not tried again until the next down.
+ expect(w.blits.length).toBe(1)
+ expect(w.logs.filter(l => l.includes('refused')).length).toBe(1)
expect(w.copies).toEqual([COPIED])
- // The strips are cut once.
- expect(w.strips.length).toBe(1)
})
-test('picture select: with IUI_DEBUG, each post writes a trace line (the post, its region, the rows selected)', async ($, on) => {
+test('picture select: with IUI_DEBUG, each post writes a trace line (the post, its region, the rows selected), each blit its times', async ($, on) => {
const w = world(on, { tty: TTY, env: { HOME: '/h', CLAUDE_CODE_FORCE_TERMINAL_IMAGES: '1', IUI_DEBUG: '1' } })
await $.command.run({ command: 'iui', args: '' } as any)
const ui = await drawAnswer($, w)
await ui.pointer({ type: 'down', x: 0, y: 1, button: 'left', in: 'sel0' })
await ui.pointer({ type: 'move', x: 10, y: 2, button: 'left', in: 'sel0' })
- await w.clock.settle()
- const lines = w.traces.filter(t => t.includes('select: '))
+ await settle(w)
+ const lines = w.traces.filter(t => /select: (down|move|up|clear) /.test(t))
expect(lines.length).toBe(2)
expect(lines[0]).toContain('select: down #')
// The kit gives a Client the terminal's size, the engine its region's.
@@ -1466,6 +1554,8 @@ test('picture select: with IUI_DEBUG, each post writes a trace line (the post, i
expect(lines[0]).toContain('rows none')
expect(lines[1]).toContain('anchor 0,1 head 10,2 size ')
expect(lines[1]).toContain('rows p0 1..2 (2)')
+ const blit = w.traces.find(t => t.includes('select: blit '))
+ expect(blit).toMatch(/select: blit ans-1 page 0 rows 1\.\.2 \(2\) decode \d+ms frame \d+ms write \d+ms \d+ bytes blit \d+ms/)
})
test('picture select: with no IUI_DEBUG, a post writes no trace line', async ($, on) => {
diff --git a/tests/select.test.ts b/tests/select.test.ts
index 688d08e..0b13a18 100644
--- a/tests/select.test.ts
+++ b/tests/select.test.ts
@@ -1,11 +1,7 @@
import { expect, test } from 'claude-code/testing'
import { decodeSelection, placeholderCell, positionsOf, textOfRun, type Mark } from '../hooks/register'
-import {
- needsSelect, overlayRows, postOf, rowsAbove, runsOf, selectColor, selectedRows, selectionAfter, selectTrace, stripDir, stripFile, stripPlan, textOfRuns, topsOf,
- type SelectPost,
-} from '../hooks/lib/select'
-import { stripsCall } from '../hooks/lib/strips'
+import { needsSelect, postOf, rowsAbove, runsOf, selectedRows, selectionAfter, selectTrace, textOfRuns, topsOf, type SelectPost } from '../hooks/lib/select'
import { SMALL_MARKS, SMALL_PAGE, SMALL_PPI } from './copy-fixture'
// ---- The posts --------------------------------------------------------------
@@ -46,20 +42,6 @@ test('a post keeps its region\'s size when it is whole cells; a bad one is dropp
// ---- Scrolled pages -------------------------------------------------------------
-test('the overlay rows are in the flow: each row\'s marginTop is the rows after the one before', () => {
- expect(overlayRows([])).toEqual([])
- expect(overlayRows([{ row: 0, x0: 3, x1: 9 }])).toEqual([{ row: 0, x0: 3, x1: 9, top: 0 }])
- // Rows 5, 6 and 7: 5 rows down, then each under the one before.
- expect(overlayRows([{ row: 5, x0: 4, x1: 97 }, { row: 6, x0: 0, x1: 97 }, { row: 7, x0: 0, x1: 20 }]).map(r => r.top)).toEqual([5, 0, 0])
- // A gap (rows that are not selected) is the margin; out of order still lands on its row.
- expect(overlayRows([{ row: 2, x0: 0, x1: 1 }, { row: 9, x0: 0, x1: 1 }]).map(r => r.top)).toEqual([2, 6])
- expect(overlayRows([{ row: 9, x0: 0, x1: 1 }, { row: 2, x0: 0, x1: 1 }]).map(r => r.top)).toEqual([9, -8])
- // Each row's top: the sum of the margins and rows before it.
- let at = 0
- const rows = [3, 4, 10, 15].map(row => ({ row, x0: 0, x1: 1 }))
- expect(overlayRows(rows).map(r => (at += r.top + 1) - 1)).toEqual([3, 4, 10, 15])
-})
-
test('the rows of a page above the viewport: the message\'s first row on screen less the page\'s top, 0 to its rows', () => {
// Page at row 1 of the message (the top margin), 16 rows.
expect(rowsAbove(0, 1, 16)).toBe(0)
@@ -155,41 +137,7 @@ test('a selection over two pages is the text of each, one line apart; a page wit
expect(textOfRuns([], words, pages)).toBe('')
})
-// ---- The strips ------------------------------------------------------------------
-
-test('strips: the page must be exactly rows x rowPx (a whole number) at the cell\'s aspect', () => {
- // Ghostty, 16 x 36 px cells: 212 columns of 16 px, 53 rows of 36 px.
- expect(stripPlan({ width: 3392, height: 1908 }, { columns: 212, rows: 53 }, { width: 16, height: 36 })).toEqual({ rowPx: 36, rows: 53 })
- expect(stripPlan({ width: 1568, height: 512 }, { columns: 98, rows: 16 }, { width: 16, height: 32 })).toEqual({ rowPx: 32, rows: 16 })
- // A page not a whole number of rows (a page cut at the most rows an Image takes).
- expect(stripPlan({ width: 3392, height: 1909 }, { columns: 212, rows: 53 }, { width: 16, height: 36 })).toBe(null)
- // A page wider than its box at that scale: the terminal would fit it by
- // its width, with space above and below (strips would not line up).
- expect(stripPlan({ width: 3408, height: 1908 }, { columns: 212, rows: 53 }, { width: 16, height: 36 })).toBe(null)
- // A cell of a fraction of a pixel (16.062 px a column): the page rounded up is wider than its box.
- expect(stripPlan({ width: 3406, height: 1855 }, { columns: 212, rows: 53 }, { width: 16.062, height: 35 })).toBe(null)
- // A page drawn smaller (an old one in an open epoch): its rows are not whole.
- expect(stripPlan({ width: 3392, height: 1908 }, { columns: 150, rows: 38 }, { width: 16, height: 36 })).toBe(null)
- // A page narrower than its box at that scale: space at the side only, the same for the strips.
- expect(stripPlan({ width: 3380, height: 1908 }, { columns: 212, rows: 53 }, { width: 16, height: 36 })).toEqual({ rowPx: 36, rows: 53 })
-})
-
-test('strips: each row is a page of its own, named as Typst numbers pages (1-based, padded to the count)', () => {
- expect(stripDir('/c/k/a-1.png')).toBe('/c/k/a-1.strips')
- expect(stripFile('/c/k/a-1.png', 53, 0)).toBe('/c/k/a-1.strips/s-01.png')
- expect(stripFile('/c/k/a-1.png', 53, 52)).toBe('/c/k/a-1.strips/s-53.png')
- expect(stripFile('/c/k/b-2.png', 120, 7)).toBe('/c/k/b-2.strips/s-008.png')
- expect(stripFile('/c/k/a-1.png', 9, 3)).toBe('/c/k/a-1.strips/s-4.png')
- const { argv, env } = stripsCall({ typst: '/usr/bin/typst', root: '/p', cache: '/c', file: '/c/k/a-1.png', width: 3392, height: 1908, rowPx: 36 })
- expect(argv.slice(0, 5)).toEqual(['sh', '-c', 'mkdir -p "$1" && shift && exec "$@"', 'sh', '/c/k/a-1.strips'])
- expect(argv.slice(5)).toEqual([
- '/usr/bin/typst', 'compile', '--root', '/', '--format', 'png', '--diagnostic-format', 'short', '--ppi', '72',
- '--input', 'src=/c/k/a-1.png', '--input', 'w=3392', '--input', 'h=1908', '--input', 'row=36', '/p/render/strips.typ', '/c/k/a-1.strips/s-{0p}.png',
- ])
- expect(env.TYPST_PACKAGE_CACHE_PATH).toBe('/c/pkg-cache')
-})
-
-// ---- The gate and the colour ---------------------------------------------------
+// ---- The gate --------------------------------------------------------------------
test('picture select is for Claude Code 2.1.305 and newer (a dev build by its release)', () => {
expect(needsSelect('2.1.305')).toBe(true)
@@ -201,10 +149,3 @@ test('picture select is for Claude Code 2.1.305 and newer (a dev build by its re
expect(needsSelect('')).toBe(false)
expect(needsSelect('dev')).toBe(false)
})
-
-test('the selection colour is one step off on every other redraw, in both modes', () => {
- expect(selectColor('dark', 1)).toBe('#264f78')
- expect(selectColor('dark', 2)).toBe('#264f79')
- expect(selectColor('light', 1)).toBe('#add6ff')
- expect(selectColor('light', 2)).toBe('#add6fe')
-})