DesignCraft: An open-source page and layout design editor written by scratch in pure Rust
CJK: plan for the East Asian feature set
aionggd6a67bdcd1e8
diff --git a/docs/cjk.md b/docs/cjk.mdnew file mode 100644index 000000000..534b36c90--- /dev/null+++ b/docs/cjk.md@@ -0,0 +1,273 @@+# CJK typesetting plan++Goal: the Chinese, Japanese and Korean typesetting that InDesign offers with its East Asian+feature set (the "Japanese/CJK" feature set, `Feature Set Locale Setting` 257), so that a+DesignCraft document can be set to the standard of a Japanese, Chinese or Korean publisher,+round-trip through IDML, and export correct PDF and EPUB.++It builds on the CJK import and composition described in [cjk-typography.md](cjk-typography.md)+(composite fonts, kinsoku sets, aki, tsume, jidori, hanging punctuation, em alignment and leading+models, kenten characters, tate-chu-yoko offsets, preserved mojikumi tables) and plans what+remains.++## Sources (clean-room)++Behaviour comes from public specifications, not from InDesign's implementation:++- W3C *Requirements for Japanese Text Layout* (JLREQ): character classes, line composition,+ spacing (aki), line breaking (kinsoku), ruby, kenten, warichu, tate-chu-yoko, grids.+- W3C *Requirements for Chinese Text Layout* (CLREQ) and *Korean* (KLREQ): punctuation placement+ for Simplified and Traditional Chinese, Korean word-based line breaking.+- JIS X 4051 (published standard; JLREQ follows it).+- OpenType: `vmtx`/`vhea`/`VORG`, `BASE`, the `vert`/`vrt2`/`vkrn`/`vpal`/`palt`/`halt`/`vhal`,+ `locl`, `ruby`, width (`fwid`/`hwid`/`pwid`/`qwid`/`twid`) and form (`jp78`/`jp83`/`jp90`/`jp04`,+ `trad`, `expt`, `nlck`, `hojo`) features.+- The published IDML specification for attribute names and values.++InDesign itself is only observed black-box (feature names, which options exist); no wording,+presets or tables are copied. Our default mojikumi and kinsoku tables are built from JLREQ.++## Where we are (2026-10)++| Area | State |+|---|---|+| Vertical frames, upright CJK, rotated Latin | done (per frame; glyphs turned by transforms) |+| `vert`/`vrt2` in vertical frames | done (applied to every run, Latin included); Horizontal Kana (`hkna`) becomes `vkna` |+| Kinsoku | named (hard, soft, Chinese, Korean) and custom kinsoku sets, bunri-kinshi, rensuuji, hanging punctuation (regular, force); without a set, fixed no-start / no-end lists and a break anywhere between CJK characters; kinsoku type (push in / push out priorities) kept but not applied |+| Mojikumi | tables and paragraph references kept through IDML; not applied (Preflight says so) |+| Aki, tsume, jidori | done ([cjk-typography.md](cjk-typography.md)) |+| Character alignment, leading model | em box top / centre / bottom, ICF from ascender and descender, roman baseline; aki above / below, centre (centre down as centre) |+| Tate-chu-yoko | manual, with offsets; no auto, 3+ digits overflow the em |+| Ruby | group ruby only, fixed 50 % size, no options, doesn't affect leading |+| Kenten | the IDML kinds and a custom character, drawn as characters; no position, size, alignment, colour or font |+| Composite fonts | done: model, `style.compositeFont.*`, shaping, IDML, font list, replace and Preflight; no dialog |+| CJK fonts | Japanese faces (Shippori Mincho first) from the optional craft-fonts build input; language-blind fallback; no per-document fonts |+| Everything else below | missing |++Known bugs to fix first:++- Korean breaks between every syllable (it should break at spaces by default).+- Only Arabic, Persian, Urdu, Hebrew and English language names reach the shaper as languages+ it knows, so `locl` never applies for the others (Japanese, Chinese, Korean, Turkish…); Chinese+ and Korean text get Japanese glyph forms and the Japanese fallback font.+- Vertical advances are the horizontal ones and upright glyphs turn about a guessed centre+ (`adv × 0.38`), so proportional and non-square glyphs sit wrong in vertical text.+- A justified CJK line has no stretch except Single Word Justification.++## Architecture++### Character classes++A `cjk::class_of(char, &Context) -> Class` in `designcraft-compose` with the JLREQ classes+(opening brackets, closing brackets, hyphens, dividing punctuation, middle dots, full stops,+commas, inseparable characters, iteration marks, prolonged sound mark, small kana, prefixed and+postfixed abbreviations, full-width ideographic space, hiragana, katakana, math symbols and+operators, ideographs, Western characters, numerals, ruby bases, warichu brackets, …). Language+and the paragraph's punctuation style decide the ambiguous ones (e.g. Chinese full-width+punctuation centred vs corner-placed).++### Spacing: mojikumi++A *mojikumi set* is a table indexed by (class before, class after) giving the space between them+as a min/optimum/max triple in em fractions, plus line-start and line-end rules. Sets are+document resources (`Styles::mojikumi_tables`, read from and written to IDML with their override+rows), referenced by name from the paragraph's `mojikumi`; what remains is applying them. We+ship a few built-in sets derived from JLREQ (no line-start indent, one-em indent, half-width+punctuation, full-width punctuation…), all original data, for the IDML built-in names.++Shaping keeps the glyphs' own advances; mojikumi is applied after shaping as *aki* glue between+glyphs: a full-width bracket's empty half becomes negative aki (compression) and the composer+can widen or narrow each gap within its range. This keeps `palt`-style proportional fonts+working (tsume removes the side bearings before mojikumi runs).++### Line breaking: kinsoku and the Japanese composer++- Kinsoku sets (`doc::cjk::Kinsoku`: characters not allowed at line start, at line end, that may+ hang, and inseparable pairs), the paragraph's set, hanging (none / regular / force),+ bunri-kinshi and rensuuji are in place: they constrain breaks in both breakers. Remaining: the+ built-in hard and soft sets from JLREQ's classes (today's named sets are common punctuation+ lists), and the kinsoku type (push in first / push out first / push out only / prioritise the+ adjustment amount), which needs the Japanese composer's push-in and push-out candidates.+- A `Composer::Japanese` (paragraph) and `Composer::JapaneseSingleLine` beside the Western+ composers. In `breaker::items` the Japanese composer emits:+ - boxes per glyph,+ - mojikumi glue between glyphs with priority tiers (compress punctuation first, then+ expand inter-ideograph space, then Western/CJK spacing, then letter-spacing),+ - kinsoku penalties (infinite inside prohibited pairs; push-in candidates scored by+ compression needed),+ - hanging punctuation as zero-width boxes allowed past the measure.+ Knuth–Plass already takes tiered stretch (`TIER_COST`), so this plugs into `kp_pass`.+- Justification in `layout_line` distributes by the same tiers; the last line follows the+ paragraph's last-line alignment; CJK lines justify by default (JLREQ).+- Korean: the Japanese composer with word-based breaking (spaces) by default; per-paragraph+ option for character-based breaking.++### Vertical text, properly++- Read `vmtx`, `vhea`, `VORG` and the ideographic em box (`BASE` `ideo`/`idtp`, else OS/2 typo+ metrics). `FontFace` gets `v_advance(gid)`, `v_origin(gid)` and `em_box()`.+- Shape upright runs with direction `TopToBottom` so `vert`, `vkrn` and `vpal` apply and the+ shaper reports vertical advances; rotated (Latin) runs stay horizontal and turn as a whole.+ `vrt2` only for runs that are rotated, never with an extra turn.+- Glyph placement in vertical lines uses the vertical origin, not a guessed centre.+- Story direction per story (IDML `StoryOrientation`) rather than per frame; frames of one story+ agree.+- Arrow keys and carets follow the vertical line direction.++### Annotations and inline structures++All set after line layout like today's ruby, but able to affect line height:++- **Ruby**: per-character (mono), group and jukugo ruby; alignment (1-2-1, centre, left/right,+ equal space, JIS rule), position (above/right, below/left), size, font, colour, x/y scale,+ offsets, overhang onto neighbours (none / ½ / 1 ruby character / ½ base / unlimited), automatic+ scaling, spacing to the base, "ruby adjusts leading".+- **Kenten**: position, size, alignment, colour, font (the kinds and a custom character are+ there).+- **Warichu**: an inline run set in 2+ smaller lines inside one line height, with size, line+ spacing, alignment and breaking rules.+- **Tate-chu-yoko**: scale-to-fit-em, automatic TCY for runs of N digits (with or without+ Latin), rensuji (offsets are there).+- **Shatai** (oblique in the CJK sense: angle and magnification, keeping the em box),+ **character rotation**.+- **Character alignment and leading model**: ICF top and bottom from the font's ideographic+ character face (`BASE` `icft`/`icfb`) rather than its ascender and descender, and centre down+ as its own model.++Jidori, tsume, aki before and after, em-box alignment and the other leading models are in place+([cjk-typography.md](cjk-typography.md)).++### Grids++- **Layout grid** (per page / parent): character size, characters per line, lines per column,+ columns and gutter, origin; drawn like the baseline grid; snap target.+- **Frame grids**: text frames that show a character grid (horizontal or vertical), with a+ character count, named grids as document resources (font, size, scale, aki between characters+ and lines, alignment, view options), grid alignment of paragraphs (em-box centre, ICF top/bottom,+ roman baseline, gyoudori: lines spanning N grid lines), grid tracking.+- Frame Grid tools (horizontal and vertical) and Object › Frame Type (text frame ↔ frame grid).++### Fonts++- **Composite fonts** (a named font made of a base font plus fonts for character sets, with size,+ baseline, x/y scale and centre adjustments) are resolved before shaping, read from and written+ to IDML and listed, replaced and preflighted by their member fonts; PDF embeds the member fonts.+ Remaining: the Composite Fonts dialog, and the predefined character classes (kanji, kana,+ full-width symbols, punctuation, Latin, numerals) beside custom sets.+- **Language-aware fallback**: Japanese, Simplified Chinese, Traditional Chinese and Korean each+ have their own fallback chain of system fonts (Japanese: the craft-fonts Japanese faces first);+ `locl` follows the character language (IDML language names).+- **No CJK fonts in the repo.** CJK fonts are large (15–25 MB each even subset) and documents+ should name the fonts they use. The Japanese UI and document faces come from the optional+ craft-fonts build input when DesignCraft is built with it (release builds are).+- **Document fonts**: fonts in a `Document Fonts` folder beside an opened `.designcraft` or+ `.idml` file are available to that document only, ahead of system fonts, and unloaded when it+ closes. File › Package copies the document's fonts into the package's `Document Fonts` folder+ (font licences permitting: OS/2 `fsType` restricted-licence fonts are listed, not copied), so a+ packaged CJK document opens with its fonts on another machine. On the web, the fonts of a+ dropped or opened package folder load the same way.++### Editing and output++- Find/Change: kana-sensitive and width-sensitive matching; full/half-width and+ hiragana/katakana conversion (Type › Change Case).+- Numbering styles: kanji, full-width, circled, parenthesised.+- Typographer's quotes per language (Simplified/Traditional Chinese, Korean).+- Character count (Info panel) by CJK conventions.+- EPUB/HTML: `writing-mode: vertical-rl`, `<ruby>`, `text-combine-upright`, `text-emphasis`.+- PDF: correct `ActualText` for ruby/warichu; vertical glyph placement as above.+- IDML: every attribute below, import and export, round-trip tested.++## Feature set (UI)++Preferences › Type gets a *Feature Set* choice (Western, CJK). CJK shows the East Asian+controls in the Character and Paragraph panels, styles dialogs and menus (Kinsoku, Mojikumi,+Composite Fonts, Named Grids, Layout Grid), makes the Japanese composer the default for new+documents and offers the frame grid tools. Western hides them; documents keep their settings+either way (opening a CJK document in Western mode loses nothing). Every feature stays reachable+as a command regardless of the feature set (agents, MCP).++## IDML attributes to support++Already read and written: `KentenKind`, `KentenCustomCharacter`, `RubyFlag`, `RubyString`,+`Tatechuyoko`, `TatechuyokoXOffset`, `TatechuyokoYOffset`, `Jidori`, `Tsume`, `LeadingAki`,+`TrailingAki`, `CharacterAlignment`, `LeadingModel`, `GlyphForm`, `OTFProportionalMetrics`,+`OTFHVKana`, `OTFRomanItalics`; paragraph `KinsokuSet`, `KinsokuType`, `KinsokuHangType`,+`BunriKinshi`, `Rensuuji`, `MojikumiTable`, `TreatIdeographicSpaceAsSpace`; resources+`KinsokuTable`, `MojikumiTable`, `CompositeFont`, `StoryPreference` direction.++Character: `KentenFont`, `KentenFontStyle`, `KentenSize`, `KentenPosition`, `KentenAlignment`,+`KentenCharacterSet`, `KentenXScale`, `KentenYScale`, `KentenFillColor`/`Tint`, `RubyType`,+`RubyAlignment`, `RubyPosition`, `RubyFont`, `RubyFontStyle`, `RubyFontSize`, `RubyXScale`,+`RubyYScale`, `RubyXOffset`, `RubyYOffset`, `RubyOverhang`, `RubyParentSpacing`,+`RubyParentOverhangAmount`, `RubyAutoAlign`, `RubyAutoScaling`, `RubyAutoTcyDigits`, `Rensuji`,+`CharacterRotation`, `ShataiMagnification`, `ShataiDegreeAngle`, `ShataiAdjustRotation`,+`ShataiAdjustTsume`, `Warichu`, `WarichuLines`, `WarichuSize`, `WarichuLineSpacing`,+`WarichuAlignment`, `WarichuCharsBeforeBreak`, `WarichuCharsAfterBreak`.++Paragraph: `Composer` (the Japanese composers), `GridAlignment` (all values),+`GridAlignFirstLineOnly`, `GridGyoudori`, `ParagraphGyoudori`, `CjkGridTracking`, `AutoTcy`,+`AutoTcyIncludeRoman`, `RotateSingleByteCharacters`, `ScaleAffectsLineHeight`.++Resources: `MojikumiTableSet`, `NamedGrid`, layout grid and frame grid preferences,+`StoryPreference` orientation.++Exact names and enumerations are checked against the IDML specification as each phase lands.++## Phases++Each phase lands as reviewable commits with tests (compose tests for layout, engine tests for+commands, IDML round trips, a rendered sample page) and `cargo xtask ci`.++| # | Phase | Contents | Estimate (agent hours) |+|---|---|---|---|+| 1 | Foundations | language passed to the shaper (`locl`); Simplified/Traditional Chinese and Korean languages; language-aware CJK fallback; document fonts folder; Korean word breaking; `vmtx`/`VORG`/em box and vertical shaping; story-level direction; vertical caret keys | 12–17 |+| 2 | Japanese composer | character classes; mojikumi applied (built-in sets and the document's tables) as aki glue; JLREQ kinsoku sets and kinsoku types (push in / push out); Japanese paragraph and single-line composers; CJK justification and last line; Chinese punctuation styles; IDML | 20–30 |+| 3 | Inline CJK | full ruby, kenten options, warichu, auto TCY and scale-to-fit, rensuji, shatai, character rotation, ICF alignment from the font; IDML | 20–30 |+| 4 | Grids | layout grid, frame grids and tools, named grids, grid alignment and gyoudori, grid tracking, character count; IDML | 20–30 |+| 5 | Fonts | Composite Fonts dialog and predefined character classes; CJK OpenType features in the OpenType menu | 5–8 |+| 6 | Editing and output | kana/width-sensitive Find/Change, width and kana conversion, CJK numbering, quotes, EPUB/HTML vertical text and ruby | 10–15 |+| 7 | Feature set UI | Preferences › Feature Set, panels and dialogs for every attribute, Japanese defaults for new documents, translations | 15–20 |++Total: roughly 100–150 agent hours.++### Phase 1 in detail++1. `CharProps.language` reaches `designcraft_fonts::shape` (harfrust buffer language) and the+ word cache key; tests: a font with `locl` for JAN vs ZHS draws different glyphs for U+76F4.+2. Language list: "Chinese: Simplified", "Chinese: Traditional", "Korean", "Japanese" (IDML+ names), with quotes and spelling/hyphenation off for them.+3. Fallback chains by language in `FontDb::fallback_for` (system fonts per platform; the+ craft-fonts Japanese faces first for Japanese); tests with stand-in fonts and a fake catalog.+4. Document fonts: opening a document loads the fonts in its `Document Fonts` folder for that+ document; Package copies used fonts there (licence permitting) and reports the ones it+ couldn't copy; tests with a synthetic package folder.+5. Korean: `cjk_break_between` doesn't break between Hangul syllables unless the paragraph asks+ for character breaking; test with a Korean sentence.+6. `FontFace::v_advance/v_origin/em_box` from `vmtx`/`vhea`/`VORG`/`BASE`; upright glyphs in+ vertical lines use them; tests against synthetic fonts with and without `vmtx` (without, the+ em box), and Shippori Mincho's tables when built with craft-fonts.+7. Upright runs shaped top-to-bottom (`vert`, `vkrn`, `vpal` apply); `vrt2` never on turned+ runs; test that a full-width comma takes its vertical form and position.+8. Story direction at story level (frames of a story agree; IDML maps both ways); arrow keys move+ along the vertical line.++## Decisions++Each follows what an InDesign user expects.++1. **Feature set gating.** East Asian controls are hidden in the Western feature set and shown in+ the CJK feature set, as in InDesign. Every feature stays reachable as a command either way.+2. **Korean line breaking** is word-based by default (KLREQ), with a per-paragraph option to break+ between syllables.+3. **PDF vertical text** keeps transformed horizontal glyphs (correct-looking, copyable via+ ActualText) rather than vertical CID fonts (`Identity-V`).+4. **No CJK fonts in the repo**; documents carry theirs in `Document Fonts`, scoped to the+ document as in InDesign. DesignCraft embeds Japanese UI and document faces only when built with+ the optional craft-fonts input (release builds are).+5. **Package copies every font that draws text**, fallback fonts included, so a package+ reproduces what is on screen.+6. **Korean fallback is serif first** (Myungjo), as Chinese and Japanese fall back to Song and+ Mincho: the serif faces CJK editions of layout software default to.+7. **Language names on IDML import** accept the common spellings of Simplified and Traditional+ Chinese, since the exact names InDesign writes are unconfirmed without a sample file.