Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
fix(script): preserve complete prose across annotation boundaries
kashorin3bcd105cd2c2
diff --git a/docs/quickstart/script.md b/docs/quickstart/script.mdindex 3498d1fa..7e723ffa 100644--- a/docs/quickstart/script.md+++ b/docs/quickstart/script.md@@ -189,7 +189,7 @@ marker; the value is a pair of semantic anchors, not a frame span:Every marker begins with `@{` and ends with `}`; `/`, `!` and `~` are inside. Names match`[a-z][a-z0-9_-]{0,63}`; whitespace and nesting inside a marker are invalid. `@{beat!}` is a Moment;-`@{part}!` opens a Selection followed by a prose exclamation mark. A marker cannot split a speech Token.+`@{part}!` opens a Selection followed by a prose exclamation mark. A marker cannot split a speech Token or its attached punctuation: write `@{beat!}“测试”`, not `“@{beat!}测试”`.| Marker | Meaning |diff --git a/docs/zh/quickstart/script.md b/docs/zh/quickstart/script.mdindex 3d659aa3..dcb73889 100644--- a/docs/zh/quickstart/script.md+++ b/docs/zh/quickstart/script.md@@ -176,7 +176,7 @@ Selection 是内联声明的具名语义**范围**。每个名字只能有一对所有标记都以 `@{` 开始、以 `}` 结束;`/`、`!`、`~` 均在内部。名称以小写字母开头,后续可用小写字母、数字、`_`、`-`,总长不超过 64;内部不允许空白或嵌套。-`@{beat!}` 是 Moment,`@{part}!` 则是区间首后跟正文感叹号。标记不能切开一个语音 Token。+`@{beat!}` 是 Moment,`@{part}!` 则是区间首后跟正文感叹号。标记不能切开语音 Token,也不能插到它与附着标点之间:应写 `@{beat!}“测试”`,不能写 `“@{beat!}测试”`。| 标记 | 含义 |diff --git a/examples/complex-explainer/productions/explainer/authors/script.svml b/examples/complex-explainer/productions/explainer/authors/script.svmlindex cb5ed157..53d92313 100644--- a/examples/complex-explainer/productions/explainer/authors/script.svml+++ b/examples/complex-explainer/productions/explainer/authors/script.svml@@ -43,7 +43,7 @@<HOST> 时间上呢, || 它是把稿子当做时间线的: || @{sem-bind!} 动效不是绑定在秒上, || 而是绑定在说的话上。</semantic-time><semantic-demo>- <HOST> @{sem-examples!} 比如榜单在“@{rank-reveal!} 第一名”这个词上翻开, || 产品图在说“@{product-reveal!} hypit”这时候飞进来。 || @{sem-rewrite-scene!} 就算是重写 @{sem-rewrite!} 稿子、@{sem-voice!} 换掉了配音, || @{sem-adapt!} 动效依然都会精准出现, || @{sem-syntax!} 一个时间点都不用改了!+ <HOST> @{sem-examples!} 比如榜单在@{rank-reveal!}“ 第一名”这个词上翻开, || 产品图在说@{product-reveal!}“ hypit”这时候飞进来。 || @{sem-rewrite-scene!} 就算是重写 @{sem-rewrite!} 稿子、@{sem-voice!} 换掉了配音, || @{sem-adapt!} 动效依然都会精准出现, || @{sem-syntax!} 一个时间点都不用改了!</semantic-demo><result><HOST> 所以用 Hypit 来做视频, || @{result-assets!} 素材生成完就能直接拿到成片, || @{result-upload!} 连一秒钟的剪辑都不需要, || @{result-compare!} 而且跟原来那条爆款的效果 || 完全一模一样!diff --git a/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml b/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svmlindex c3194a61..65816545 100644--- a/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml+++ b/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml@@ -29,10 +29,10 @@TierBoard consumes windows without finding frames. Cue breaks are author intent. --><script id="story"><ronaldo>- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}</ronaldo><messi>- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.</messi></script>diff --git a/examples/ranking-football/reference-banana/reference-banana.svml b/examples/ranking-football/reference-banana/reference-banana.svmlindex c3194a61..65816545 100644--- a/examples/ranking-football/reference-banana/reference-banana.svml+++ b/examples/ranking-football/reference-banana/reference-banana.svml@@ -29,10 +29,10 @@TierBoard consumes windows without finding frames. Cue breaks are author intent. --><script id="story"><ronaldo>- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}</ronaldo><messi>- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.</messi></script>diff --git a/examples/ranking-football/reference-gpt.svml b/examples/ranking-football/reference-gpt.svmlindex b2b02842..53a26252 100644--- a/examples/ranking-football/reference-gpt.svml+++ b/examples/ranking-football/reference-gpt.svml@@ -29,10 +29,10 @@TierBoard consumes windows without finding frames. Cue breaks are author intent. --><script id="story"><ronaldo>- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}</ronaldo><messi>- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.</messi></script>diff --git a/examples/ranking-football/reference.svml b/examples/ranking-football/reference.svmlindex 529df8fd..873bfeec 100644--- a/examples/ranking-football/reference.svml+++ b/examples/ranking-football/reference.svml@@ -31,10 +31,10 @@TierBoard consumes windows without finding frames. Cue breaks are author intent. --><script id="story"><ronaldo>- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}</ronaldo><messi>- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.</messi></script>diff --git a/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml b/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svmlindex 8de5a7e2..10781c60 100644--- a/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml+++ b/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml@@ -29,10 +29,10 @@TierBoard consumes windows without finding frames. Cue breaks are author intent. --><script id="story"><ronaldo>- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}</ronaldo><messi>- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.</messi></script>diff --git a/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml b/examples/ranking-football/swap-effect-banana/swap-effect-banana.svmlindex cc3a6cdb..10391668 100644--- a/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml+++ b/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml@@ -28,10 +28,10 @@Ronaldo Segments are also the two disjoint Column reveal windows. --><script id="story"><messi>- <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>@{/ikea-sign}@{/pitch-pov}. || @{retirement-room} Went to Miami || and turned the league || into a retirement home@{/retirement-room}.+ <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>.@{/ikea-sign}@{/pitch-pov} || @{retirement-room} Went to Miami || and turned the league || into a retirement home.@{/retirement-room}</messi><ronaldo>- <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent@{/rent-jump}. || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals}. || Absolutely the top one.+ <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent.@{/rent-jump} || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum.@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals} || Absolutely the top one.</ronaldo></script>diff --git a/examples/ranking-football/swap-effect.svml b/examples/ranking-football/swap-effect.svmlindex 5568c289..bc9c6db4 100644--- a/examples/ranking-football/swap-effect.svml+++ b/examples/ranking-football/swap-effect.svml@@ -28,10 +28,10 @@Ronaldo Segments are also the two disjoint Column reveal windows. --><script id="story"><messi>- <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>@{/ikea-sign}@{/pitch-pov}. || @{retirement-room} Went to Miami || and turned the league || into a retirement home@{/retirement-room}.+ <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>.@{/ikea-sign}@{/pitch-pov} || @{retirement-room} Went to Miami || and turned the league || into a retirement home.@{/retirement-room}</messi><ronaldo>- <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent@{/rent-jump}. || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals}. || Absolutely the top one.+ <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent.@{/rent-jump} || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum.@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals} || Absolutely the top one.</ronaldo></script>diff --git a/examples/ranking-football/swap-host.svml b/examples/ranking-football/swap-host.svmlindex 5a907cbe..2367856d 100644--- a/examples/ranking-football/swap-host.svml+++ b/examples/ranking-football/swap-host.svml@@ -29,10 +29,10 @@TierBoard consumes windows without finding frames. Cue breaks are author intent. --><script id="story"><ronaldo>- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}</ronaldo><messi>- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.</messi></script>diff --git a/examples/ranking-football/swap-topic.svml b/examples/ranking-football/swap-topic.svmlindex 7f2eb3b9..dc5ce52e 100644--- a/examples/ranking-football/swap-topic.svml+++ b/examples/ranking-football/swap-topic.svml@@ -28,10 +28,10 @@Every B-roll insert consumes an authored semantic Selection. --><script id="story"><elon>- <HOST>Elon is <D | Dee> tier. || @{money-rain} Man bought Twitter || for 44 billion dollars || @{twitter-x} just to change || the name@{/twitter-x}@{/money-rain}. || @{mars-rocket} Sends rockets || to Mars, || but @{market-crash} can't send a tweet || without crashing || the stock market@{/market-crash}@{/mars-rocket}. || @{richest-chair} Richest guy || on Earth, || but somehow || @{haircut-x} still can't buy || a good haircut@{/haircut-x}@{/richest-chair}.+ <HOST>Elon is <D | Dee> tier. || @{money-rain} Man bought Twitter || for 44 billion dollars || @{twitter-x} just to change || the name.@{/twitter-x}@{/money-rain} || @{mars-rocket} Sends rockets || to Mars, || but @{market-crash} can't send a tweet || without crashing || the stock market.@{/market-crash}@{/mars-rocket} || @{richest-chair} Richest guy || on Earth, || but somehow || @{haircut-x} still can't buy || a good haircut.@{/haircut-x}@{/richest-chair}</elon><sam>- <HOST>Sam Altman? || @{fired-friday} Guy got fired || on a Friday, || @{company-quit} the entire company || threatened to quit, || and @{back-monday} he was back || in the chair || by Monday@{/back-monday}@{/company-quit}@{/fired-friday}. || @{elon-lawsuit} Elon sued him, || @{google-fears} Google fears him, || @{congress-grilled} Congress grilled him, || @{still-here} and he's still here@{/still-here}@{/congress-grilled}@{/google-fears}@{/elon-lawsuit}. || I have to || put him in || <S | Ess> tier.+ <HOST>Sam Altman? || @{fired-friday} Guy got fired || on a Friday, || @{company-quit} the entire company || threatened to quit, || and @{back-monday} he was back || in the chair || by Monday.@{/back-monday}@{/company-quit}@{/fired-friday} || @{elon-lawsuit} Elon sued him, || @{google-fears} Google fears him, || @{congress-grilled} Congress grilled him, || @{still-here} and he's still here.@{/still-here}@{/congress-grilled}@{/google-fears}@{/elon-lawsuit} || I have to || put him in || <S | Ess> tier.</sam></script>diff --git a/packages/script/README.md b/packages/script/README.mdindex 44db0f94..d2def36c 100644--- a/packages/script/README.md+++ b/packages/script/README.md@@ -117,6 +117,15 @@ following characters. Display punctuation attaches to neighboring words withoutThese units support precise timing and highlighting. A Caption Cue can hold a whole phrase of them;`||` chooses its handoff independently of character counts or visual line wrapping.+Annotations do not create speech boundaries. Comments are transparent (`hel<!--note-->lo`+remains `hello`); a postfix attribute or `||` inside a word is invalid. Script analyzes a complete+prose run before binding these constructs. An explicit Dual correspondence and a speaker/Segment+boundary remain authored structure. Shared Dual groups still expose their internal speech anchors.++A Dual display side is literal authored text, including symbols and emoji: `<😀|smile>` and+`<.|dot>` have explicit speech correspondence and require no invented speech token for the symbol.+A literal-only display is one display surface within that correspondence.+## Display spelling and separatorsScript preserves the normalized display spelling independently of speech tokenization. Ordinary@@ -139,7 +148,8 @@ Every marker starts with `@{` and ends with `}`. The complete marker is zero-widdisplay; surrounding prose whitespace remains prose. All control sigils belong inside the braces:`@{beat!}` is a Moment, whereas `@{part}!` opens a Selection followed by a literal exclamation mark.Names match `[a-z][a-z0-9_-]{0,63}`; whitespace and nesting inside a marker are invalid.-A marker cannot split a speech Token. `hello{emphasis}` is a postfix display attribute; `@{part}`+A marker cannot split a speech Token or separate it from attached punctuation: place+`@{beat!}“测试”`, not `“@{beat!}测试”`. `hello{emphasis}` is a postfix display attribute; `@{part}`is consumed as one marker and cannot be mistaken for that attribute. Write a literal `@{part}` as`\@\{part\}`.@@ -199,6 +209,9 @@ Use `--body` for a file containing a raw Script body rather than outer SVML. Themarkers only inside Script bodies, leaves comments and escapes intact, and does not touch providerprompt references such as `@image1`. It neither installs anything nor runs during a build.+The tool changes marker spelling, not marker placement. Move a marker that separates a word from+its attached quote or punctuation to the complete word boundary before using that source.+Review authored whitespace after migration: spaces previously discarded by Chinese/punctuationnormalization now appear. The tool preserves source spaces rather than guessing the author's intent.Regenerate affected Narrative, caption and Build results with the new reader/writer together;diff --git a/packages/script/src/lexical.ts b/packages/script/src/lexical.tsindex 21514133..b54a8d26 100644--- a/packages/script/src/lexical.ts+++ b/packages/script/src/lexical.ts@@ -9,146 +9,88 @@ const WORD_CHARACTER = String.raw`(?:(?!${CHARACTER_UNIT})[\p{L}\p{M}\p{N}])`;const LEXICAL_UNIT = new RegExp([String.raw`(?:\p{N}{1,3}(?:[,,]\p{N}{3})+|\p{N}+)(?:[..]\p{N}+)?(?:-\p{N}+(?:[..]\p{N}+)?)*(?!\p{N}|-[\p{L}\p{M}])`,String.raw`${CHARACTER_UNIT}\p{M}*`,- String.raw`${WORD_CHARACTER}+(?:['’.-]${WORD_CHARACTER}+)*`,+ String.raw`(?:(?!${CHARACTER_UNIT})[\p{L}\p{N}])${WORD_CHARACTER}*(?:['’.-]${WORD_CHARACTER}+)*`,].join("|"), "gu");-const OPENING_PUNCTUATION = new Set([- "(", "[", "{", "(", "【", "《", "「", "『", "〔", "〈", "“", "‘",- "$", "¥", "¥", "€", "£",-]);--const QUOTE_PUNCTUATION = new Set(["\"", "'", "`", "’", "ʼ"]);-const UNICODE_OPENING_PUNCTUATION = /[\p{Ps}\p{Pi}]/u;--/**- * ASCII quotation marks have no Unicode opening/closing category. In a gap between- * lexical units, whitespace before the mark is the useful authoring signal:- * `said "hello` opens a quote, while `hello" world` closes one. A quote before the- * first unit is opening by definition; contractions stay inside one lexical unit.- */-function isOpeningPunctuation(- character: string,- gap: string,- position: number,- hasPreviousSurface: boolean,-): boolean {- if (OPENING_PUNCTUATION.has(character) || UNICODE_OPENING_PUNCTUATION.test(character)) return true;- if (!QUOTE_PUNCTUATION.has(character)) return false;- if (!hasPreviousSurface) return true;- const before = [...gap.slice(0, position)].at(-1);- return before !== undefined && /\s/u.test(before);+// Unicode owns bracket/quote categories. ASCII symmetric quotes need local context;+// they have no opening/closing category. This is prose analysis, not marker syntax.+function openingAt(gap: string, hasPrevious: boolean, beforeWord = true): number {+ for (const match of gap.matchAll(/./gu)) {+ const character = match[0];+ if (/[\p{Ps}\p{Pi}\p{Sc}]/u.test(character)+ || ((/["'`]/u.test(character) || (beforeWord && /\p{Pf}/u.test(character))) && (!hasPrevious || /\s$/u.test(gap.slice(0, match.index))))) return match.index;+ }+ return gap.length;}export function lexicalUnits(value: string): readonly LexicalUnit[] {- return [...value.matchAll(LEXICAL_UNIT)].map((match) => ({- text: match[0],- index: match.index,- }));+ return [...value.matchAll(LEXICAL_UNIT)].map(match => ({ text: match[0], index: match.index }));}export function lexicalCount(value: string): number {return lexicalUnits(value).length;}-/** Source positions for markers; unlike timing units, these include attached punctuation. */-export function lexicalEditRanges(value: string): readonly { start: number; end: number }[] {- const units = lexicalUnits(value);- return units.map((unit, index) => {- const previousEnd = index === 0 ? 0 : units[index - 1]!.index + units[index - 1]!.text.length;- const nextStart = units[index + 1]?.index ?? value.length;- const leading = value.slice(previousEnd, unit.index);- let start = unit.index;- for (const [position, character] of [...leading.matchAll(/./gu)].map((match) => [match.index, match[0]] as const)) {- if (isOpeningPunctuation(character, leading, position, index > 0)) { start = previousEnd + position; break; }- }- let end = unit.index + unit.text.length;- const trailing = value.slice(end, nextStart);- for (const match of trailing.matchAll(/./gu)) {- if (/\s/u.test(match[0]) || isOpeningPunctuation(match[0], trailing, match.index, true)) break;- end += match[0].length;- }- return { start, end };- });-}--/** Normalize source formatting without inventing or removing language-specific separators. */+/** Normalize source formatting without inventing language-specific separators. */export function cleanProjection(value: string): string {return value.replace(/\s+/gu, " ").trim();}-/** Structural markers split atoms but must not invent prose whitespace when those atoms rejoin. */export function joinProjection(parts: readonly string[]): string {return cleanProjection(parts.join(""));}-export type DisplaySurface = {- readonly text: string;- readonly separatorBefore: "" | " ";-};+export type DisplaySurface = { readonly text: string; readonly separatorBefore: "" | " " };+type ProseWord = { start: number; end: number; separatorBefore: "" | " " };-/** Keep the authored display spelling while assigning punctuation to lexical surfaces. */-export function displaySurfaces(value: string): readonly DisplaySurface[] {- const prose = cleanProjection(value);- const units = lexicalUnits(prose);- const surfaces: Array<{ text: string; separatorBefore: "" | " " }> = [];+/** Analyze complete prose once. Speech cores and their written surfaces are distinct:+ * punctuation has spelling and source extent, but does not acquire speech timing.+ * A display-only literal (for example an emoji in Dual Text) needs no speech core.+ */+export function analyzeProse(value: string, followsDisplay = false): {+ readonly units: readonly LexicalUnit[];+ readonly surfaces: readonly DisplaySurface[];+ readonly editRanges: readonly { start: number; end: number }[];+} {+ const units = lexicalUnits(value);+ const words: ProseWord[] = [];let cursor = 0;for (const unit of units) {- const gap = prose.slice(cursor, unit.index);- let openingAt = gap.length;- for (const match of gap.matchAll(/./gu)) {- if (!/\s/u.test(match[0]) && isOpeningPunctuation(match[0], gap, match.index, surfaces.length > 0)) {- openingAt = match.index;- break;- }- }- const closing = gap.slice(0, openingAt);- const previous = surfaces.at(-1);- if (previous) previous.text += closing.trimEnd();- const prefix = (previous ? "" : closing.trimStart()) + gap.slice(openingAt);- surfaces.push({ text: prefix + unit.text, separatorBefore: previous && /\s$/u.test(closing) ? " " : "" });+ const gap = value.slice(cursor, unit.index);+ const opening = openingAt(gap, words.length > 0 || followsDisplay);+ const closing = gap.slice(0, opening);+ const previous = words.at(-1);+ if (previous) previous.end = cursor + closing.trimEnd().length;+ const start = previous || followsDisplay ? cursor + opening : value.slice(0, unit.index).search(/\S/u);+ words.push({ start: start < 0 ? unit.index : start, end: unit.index + unit.text.length,+ separatorBefore: previous && /\s$/u.test(closing) ? " " : "" });cursor = unit.index + unit.text.length;}- const last = surfaces.at(-1);- if (last) last.text += prose.slice(cursor);- return surfaces;+ const last = words.at(-1);+ if (last) last.end = value.trimEnd().length;+ const surfaces = words.map(word => ({ text: value.slice(word.start, word.end).replace(/\s+/gu, " "), separatorBefore: word.separatorBefore }));+ const editRanges = words.map((word, index) => {+ const unit = units[index]!;+ let end = unit.index + unit.text.length;+ // Whitespace is a legal insertion boundary. Only adjacent trailing punctuation+ // belongs to the indivisible edit surface ("word," versus "word ,").+ const trailing = value.slice(end, word.end);+ const whitespace = trailing.search(/\s/u);+ end += whitespace < 0 ? trailing.length : whitespace;+ return { start: word.start, end };+ });+ if (surfaces.length === 0 && value.trim()) surfaces.push({ text: cleanProjection(value), separatorBefore: "" });+ return { units, surfaces, editRanges };}-export function displayWordSurfaces(value: string): readonly string[] {- return displaySurfaces(value).map(surface => surface.text);+export function displaySurfaces(value: string): readonly DisplaySurface[] {+ return analyzeProse(value).surfaces;}-/** Split punctuation before the first lexical unit, retaining every authored separator. */-export function splitDisplayPrefix(value: string): { previous: string; current: string } {+/** Correspondence boundaries retain punctuation on either side of an explicit Dual. */+export function splitDisplayPrefix(value: string, hasPrevious: boolean): { previous: string; current: string } {const first = lexicalUnits(value)[0];- if (!first) return { previous: value, current: "" };- const leading = value.slice(0, first.index);- for (const match of leading.matchAll(/./gu)) {- if (!/\s/u.test(match[0]) && isOpeningPunctuation(match[0], leading, match.index, false)) {- return { previous: leading.slice(0, match.index), current: value.slice(match.index) };- }- }- return { previous: leading, current: value.slice(first.index) };-}--/** Closing punctuation before a lexical unit belongs to the previous display unit when one exists. */-export function splitLeadingClosingPunctuation(value: string): {- readonly previous: string;- readonly current: string;-} {- const prose = cleanProjection(value);- const first = lexicalUnits(prose)[0];- if (first === undefined) return { previous: prose.replace(/\s+/gu, ""), current: "" };- const leading = prose.slice(0, first.index);- const punctuation = [...leading]- .map((character, position) => ({ character, position }))- .filter(({ character }) => !/\s/u.test(character));- const previous = punctuation- .filter(({ character, position }) => !isOpeningPunctuation(character, leading, position, false))- .map(({ character }) => character)- .join("");- const opening = punctuation- .filter(({ character, position }) => isOpeningPunctuation(character, leading, position, false))- .map(({ character }) => character)- .join("");- return { previous, current: `${opening}${prose.slice(first.index)}` };+ const end = first?.index ?? value.length;+ const split = openingAt(value.slice(0, end), hasPrevious, first !== undefined);+ return { previous: value.slice(0, split), current: value.slice(split) };}diff --git a/packages/script/src/parser.ts b/packages/script/src/parser.tsindex e3a6dbad..7ca9cd67 100644--- a/packages/script/src/parser.ts+++ b/packages/script/src/parser.ts@@ -2,11 +2,9 @@import { ScriptSyntaxError } from "./error.js";import {cleanProjection,- displayWordSurfaces,joinProjection,lexicalUnits,- lexicalEditRanges,- splitLeadingClosingPunctuation,+ analyzeProse,splitDisplayPrefix,} from "./lexical.js";import type {@@ -38,8 +36,6 @@ type MutableSegment = {readonly tokenStart: number;readonly sourceStart: number;readonly contentStart: number;- lexicalRun: string;- readonly lexicalMarkers: Array<{ readonly position: number; readonly offset: number }>;};const SEGMENT_ID = /^[a-z][a-z0-9_-]{0,63}$/u;@@ -50,7 +46,7 @@ const ATTRIBUTE_NAME = /^[a-z][a-z0-9_-]{0,63}$/u;const ATTRIBUTE_VALUE = /^[A-Za-z0-9_.:-]+$/u;/** A marker's structural position before its affinity picks one anchor. */-type RawMarkerBoundary = Omit<MarkerBoundary, "anchorId">;+type RawMarkerBoundary = { -readonly [K in keyof Omit<MarkerBoundary, "anchorId">]: Omit<MarkerBoundary, "anchorId">[K] };type RawEdge<T> = Omit<T, "boundary"> & { readonly boundary: RawMarkerBoundary };type RawSelection = Omit<ParsedSelection, "startAnchorId" | "endAnchorId" | "open" | "close"> & {readonly open: RawEdge<ParsedSelection["open"]>;@@ -127,7 +123,7 @@ export function parseScript(const tokens: ParsedToken[] = [];const captionRegions: ParsedCaptionRegion[] = [];let precedingCaptionRegion: number | undefined;- let pendingCaptionSpace = false;+ let pendingCaptionText = "";const selections = new Map<string, RawSelection>();const moments = new Map<string, RawMoment>();const captionBreaks: Array<{ readonly tokenIndex: number; readonly range: { readonly start: number; readonly end: number } }> = [];@@ -141,33 +137,64 @@ export function parseScript(let offset = 0;let structuralPosition = 0;- const boundary = (): RawMarkerBoundary => ({- tokenIndex: tokens.length,- structuralPosition,- ...(current ? { segmentId: current.id } : {}),- });+ // One prose run crosses annotations and comments. Only an explicit correspondence,+ // speaker change or Segment edge ends it. Source positions stay local to parsing.+ let runText = "";+ let runCaption = "";+ let runCapture = true;+ let runStarts: number[] = [];+ let runEnds: number[] = [];+ let runBoundaries: Array<{ position: number; offset: number; kind: "marker" | "cue"; value: { tokenIndex: number } }> = [];+ let runAttributes: Array<{ position: number; offset: number; end: number; attributes: readonly CaptionWordAttribute[] }> = [];++ const boundary = (start: number): RawMarkerBoundary => {+ const value = { tokenIndex: tokens.length, structuralPosition,+ ...(current ? { segmentId: current.id } : {}) };+ if (current) runBoundaries.push({ position: runText.length, offset: start, kind: "marker", value });+ return value;+ };- const finishLexicalRun = (): void => {- if (!current) return;- for (const match of lexicalUnits(current.lexicalRun)) {- const start = match.index;- const end = start + match.text.length;- const split = current.lexicalMarkers.find((marker) => start < marker.position && marker.position < end);- if (split) {- fail(- "SCRIPT_MARKER_TOKEN_BOUNDARY",- "A Selection or Moment marker cannot split a speech token.",- split.offset,- );+ const finishLexicalRun = (): ParsedCaptionRegion["marks"] => {+ if (!current) return [];+ const { units, editRanges: ranges } = analyzeProse(runText, runCapture && precedingCaptionRegion !== undefined);+ const tokenStart = tokens.length;+ for (const event of runBoundaries) {+ if (ranges.some(range => range.start < event.position && event.position < range.end)) {+ fail(event.kind === "marker" ? "SCRIPT_MARKER_TOKEN_BOUNDARY" : "SCRIPT_CAPTION_BREAK_TOKEN_BOUNDARY",+ "A semantic marker or Cue break cannot split a word and its attached punctuation.", event.offset);}+ event.value.tokenIndex = tokenStart + units.filter(unit => unit.index + unit.text.length <= event.position).length;+ if (event.kind === "cue" && event.value.tokenIndex === current.tokenStart) {+ fail("SCRIPT_CAPTION_BREAK_EMPTY", "Caption Cue break must follow visible Script text.", event.offset);+ }+ }+ if (units.length === 0 && runAttributes.length > 0) fail("SCRIPT_ATTRIBUTE_TARGET", "A spoken display attribute needs a complete word.", runAttributes[0]!.offset);+ const marks = bindDisplayAttributes(runText, runAttributes, units);+ for (const [ordinal, unit] of units.entries()) {+ const index = tokens.length;+ const segmentTokenIndex = index - current.tokenStart;+ const range = ranges[ordinal]!;+ const attachedAttributes = runAttributes.filter(attribute =>+ attribute.position >= unit.index + unit.text.length && attribute.position <= range.end);+ tokens.push({ id: tokenId(current.id, segmentTokenIndex), index, segmentId: current.id, segmentTokenIndex,+ startAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "start"),+ endAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "end"),+ text: unit.text, normalized: normalizeWord(unit.text),+ range: { start: runStarts[unit.index]!, end: runEnds[unit.index + unit.text.length - 1]! },+ editRange: { start: runStarts[range.start]!, end: Math.max(runEnds[range.end - 1]!, ...attachedAttributes.map(attribute => sourceOffset + attribute.end)) },+ });+ }+ if (runText || runCaption) {+ const range = { start: runStarts[0]!, end: runEnds.at(-1)! };+ current.atoms.push({ kind: "text", speech: runText, caption: runCaption, tokenStart, tokenEndExclusive: tokens.length, range });+ if (runCapture) addCaptionRegion(runCaption, current.id, tokenStart, tokens.length, "identity", range, marks);}- current.lexicalRun = "";- current.lexicalMarkers.length = 0;+ runText = ""; runCaption = ""; runStarts = []; runEnds = []; runBoundaries = []; runAttributes = [];+ return marks;};const addMarker = (marker: Marker, start: number): void => {if (!TEMPORAL_ID.test(marker.id)) fail("SCRIPT_TEMPORAL_ID", `Invalid temporal id "${marker.id}".`, start);- if (current) current.lexicalMarkers.push({ position: current.lexicalRun.length, offset: start });if (marker.kind === "moment") {if (openSelections.has(marker.id) || selections.has(marker.id)) {fail("SCRIPT_TEMPORAL_TYPE", `Temporal name "${marker.id}" cannot be both Selection and Moment.`, start);@@ -178,7 +205,7 @@ export function parseScript(moments.set(marker.id, {id: marker.id,affinity: marker.affinity,- boundary: boundary(),+ boundary: boundary(start),range: { start: sourceOffset + start, end: sourceOffset + start + marker.length },});return;@@ -195,7 +222,7 @@ export function parseScript(}openSelections.set(marker.id, {affinity: marker.affinity,- boundary: boundary(),+ boundary: boundary(start),start: sourceOffset + start,end: sourceOffset + start + marker.length,});@@ -213,7 +240,7 @@ export function parseScript(},close: {affinity: marker.affinity,- boundary: boundary(),+ boundary: boundary(start),range: { start: sourceOffset + start, end: sourceOffset + start + marker.length },},});@@ -286,13 +313,32 @@ export function parseScript(return { attributes, length: close + 1 };};+ const bindDisplayAttributes = (+ display: string,+ annotations: readonly { position: number; offset: number; attributes: readonly CaptionWordAttribute[] }[],+ units = lexicalUnits(display),+ ): ParsedCaptionRegion["marks"] => {+ const marks: Array<ParsedCaptionRegion["marks"][number]> = [];+ for (const annotation of annotations) {+ const displayIndex = units.length === 0+ ? (display.trim() && annotation.position === display.trimEnd().length ? 0 : -1)+ : units.findLastIndex(unit => unit.index + unit.text.length <= annotation.position);+ if (displayIndex < 0 || units.some(unit => unit.index < annotation.position && annotation.position < unit.index + unit.text.length)) {+ fail("SCRIPT_ATTRIBUTE_TARGET", "A display attribute must follow a complete word.", annotation.offset);+ }+ if (marks.some(mark => mark.displayIndex === displayIndex)) fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display word cannot receive two separate attribute blocks.", annotation.offset);+ marks.push({ displayIndex, attributes: annotation.attributes });+ }+ return marks;+ };+const parseMarkedDisplay = (raw: string,absoluteStart: number,): { readonly display: string; readonly marks: readonly ParsedCaptionRegion["marks"][number][] } => {const parts: string[] = [];- const marks: ParsedCaptionRegion["marks"][number][] = [];let cursor = 0;+ const attributePositions: Array<{ position: number; offset: number; attributes: readonly CaptionWordAttribute[] }> = [];for (let index = 0; index < raw.length; index += 1) {if (raw[index] === "\\") {index += 1;@@ -304,21 +350,15 @@ export function parseScript(if (!before || /\s$/u.test(before)) {fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must immediately follow a display token.", absoluteStart + index);}- const cleanBefore = literalString(before, absoluteStart, true);- const displayIndex = displayWordSurfaces(cleanBefore).length - 1;- if (displayIndex < 0) {- fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", absoluteStart + index);- }- if (marks.some((mark) => mark.displayIndex === displayIndex)) {- fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display token cannot receive two separate attribute blocks.", absoluteStart + index);- }+ const cleanBefore = parts.join("") + literalString(raw.slice(cursor, index), absoluteStart + cursor, true);parts.push(literalString(raw.slice(cursor, index), absoluteStart + cursor, true));- marks.push({ displayIndex, attributes: block.attributes });+ attributePositions.push({ position: cleanBefore.length, offset: absoluteStart + index, attributes: block.attributes });cursor = index + block.length;index += block.length - 1;}parts.push(literalString(raw.slice(cursor), absoluteStart + cursor, true));- return { display: parts.join(""), marks };+ const display = parts.join("");+ return { display, marks: bindDisplayAttributes(display, attributePositions) };};const assertDualDisplayLiteral = (raw: string, absoluteStart: number): void => {@@ -363,8 +403,8 @@ export function parseScript(}// Dual Text edge whitespace is syntax padding; ordinary prose whitespace is content.const raw = kind === "alias" ? cleanProjection(displayValue) : displayValue;- const value = ((pendingCaptionSpace ? " " : "") + raw).replace(/\s+/gu, " ");- const split = splitDisplayPrefix(value);+ const value = (pendingCaptionText + raw).replace(/\s+/gu, " ");+ const split = splitDisplayPrefix(value, precedingCaptionRegion !== undefined);const previous = precedingCaptionRegion === undefined ? undefined : captionRegions[precedingCaptionRegion];if (previous && split.previous.trim()) {captionRegions[precedingCaptionRegion!] = {@@ -372,9 +412,15 @@ export function parseScript(display: previous.display + split.previous.trimEnd(),};}- pendingCaptionSpace = /\s$/u.test(value);+ pendingCaptionText = /\s$/u.test(value) ? " " : "";const display = (previous ? split.current : split.previous + split.current).trim();- if (!display || tokenEndExclusive <= tokenStart) return;+ if (tokenEndExclusive <= tokenStart) {+ // A literal opener outside an explicit correspondence awaits that display,+ // while closing punctuation has already been attached to the preceding one.+ pendingCaptionText = previous ? split.current || pendingCaptionText : value;+ return;+ }+ if (!display) return;captionRegions.push({id: `caption-region:${captionRegions.length + 1}`,display,@@ -389,100 +435,24 @@ export function parseScript(precedingCaptionRegion = captionRegions.length - 1;};- const markLastCaptionSurface = (- attributes: readonly CaptionWordAttribute[],- offset: number,- ): void => {- const previous = captionRegions.at(-1);- if (previous === undefined || previous.kind === "hidden") {- fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", offset);- }- const target = previous!;- if (target.kind === "alias") {- fail("SCRIPT_ATTRIBUTE_DUAL", "Annotate a Dual display word before the pipe; a mark cannot wrap a Dual Text.", offset);- }- const displayIndex = displayWordSurfaces(target.display).length - 1;- if (displayIndex < 0) fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", offset);- const existing = target.marks.find((mark) => mark.displayIndex === displayIndex);- if (existing !== undefined) {- fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display token cannot receive two separate attribute blocks.", offset);- }- captionRegions[captionRegions.length - 1] = {- ...target,- marks: [...target.marks, { displayIndex, attributes }],- };- };-const addText = (speech: string,caption: string,start: number,- end: number,+ _end: number,captureCaptionRegion = true,sourcePositions?: readonly number[],): void => {if (!current) {- if (speech.trim() || caption.trim()) {- fail("SCRIPT_TEXT_OUTSIDE_SEGMENT", "Natural-language text is only allowed inside a named Segment.", start);- }+ if (speech.trim() || caption.trim()) fail("SCRIPT_TEXT_OUTSIDE_SEGMENT", "Natural-language text is only allowed inside a named Segment.", start);return;}- current.lexicalRun += speech;- const tokenStart = tokens.length;- const editRanges = lexicalEditRanges(speech);- const units = lexicalUnits(speech);- // Decoded characters may occupy more than one source character (for example \@).- const position = (index: number): number => sourceOffset + (sourcePositions?.[index] ?? start + index);- // A punctuation-only piece can follow a display attribute or escaped source piece.- const preceding = tokens.at(-1);- const closing = splitLeadingClosingPunctuation(speech).previous;- if (preceding?.segmentId === current.id && preceding.editRange.end === sourceOffset + start && closing && speech.startsWith(closing)) {- tokens[tokens.length - 1] = { ...preceding, editRange: {- start: preceding.editRange.start, end: position(closing.length),- } };- }- for (const [unitIndex, match] of units.entries()) {- const normalized = normalizeWord(match.text);- if (!normalized) continue;- const index = tokens.length;- const segmentTokenIndex = index - current.tokenStart;- const id = tokenId(current.id, segmentTokenIndex);- tokens.push({- id,- index,- segmentId: current.id,- segmentTokenIndex,- startAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "start"),- endAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "end"),- text: match.text,- normalized,- range: {- start: position(match.index),- end: position(match.index + match.text.length),- },- editRange: {- start: position(editRanges[unitIndex]!.start),- end: position(editRanges[unitIndex]!.end),- },- });- }- current.atoms.push({- kind: "text",- speech,- caption,- tokenStart,- tokenEndExclusive: tokens.length,- range: { start: sourceOffset + start, end: sourceOffset + end },- });- if (captureCaptionRegion) {- addCaptionRegion(- caption,- current.id,- tokenStart,- tokens.length,- "identity",- { start: sourceOffset + start, end: sourceOffset + end },- );+ runCapture = captureCaptionRegion;+ runText += speech;+ runCaption += caption;+ for (let index = 0; index < speech.length; index++) {+ runStarts.push(sourceOffset + (sourcePositions?.[index] ?? start + index));+ runEnds.push(sourceOffset + (sourcePositions?.[index + 1] ?? start + index + 1));}};@@ -501,7 +471,6 @@ export function parseScript(// marker edits target these words, not a synthetic copy beyond the pipe.const shared = caption === undefined;const spokenParts: string[] = [];- const sharedMarks: Array<ParsedCaptionRegion["marks"][number]> = [];let partStart = 0;let index = 0;let emittedCaption = false;@@ -528,18 +497,8 @@ export function parseScript(}addLiteral(before, absoluteStart + partStart);const block = parseAttributeBlock(raw.slice(index), absoluteStart + index);- const displayIndex = displayWordSurfaces(spokenParts.join("")).length - 1;- if (displayIndex < 0 || tokens.length === startToken) {- fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", absoluteStart + index);- }- if (sharedMarks.some(mark => mark.displayIndex === displayIndex)) {- fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display token cannot receive two separate attribute blocks.", absoluteStart + index);- }- sharedMarks.push({ displayIndex, attributes: block.attributes });- const token = tokens.at(-1)!;- tokens[tokens.length - 1] = { ...token, editRange: {- start: token.editRange.start, end: sourceOffset + absoluteStart + index + block.length,- } };+ runAttributes.push({ position: runText.length, offset: absoluteStart + index,+ end: absoluteStart + index + block.length, attributes: block.attributes });index += block.length;partStart = index;continue;@@ -557,6 +516,7 @@ export function parseScript(}addLiteral(raw.slice(partStart), absoluteStart + partStart);const display = caption ?? spokenParts.join("");+ const sharedMarks = finishLexicalRun();if (shared && tokens.length === startToken) {fail("SCRIPT_DUAL_EMPTY", "Dual Text with omitted speech must contain spoken text on its display side.", absoluteStart);}@@ -595,7 +555,6 @@ export function parseScript(while (offset < source.length) {if (source.startsWith("<!--", offset)) {- finishLexicalRun();const end = source.indexOf("-->", offset + 4);if (end < 0) fail("SCRIPT_COMMENT", "Unclosed Script comment.", offset);offset = end + 3;@@ -615,7 +574,7 @@ export function parseScript(}const contentStart = offset + open[0].length;precedingCaptionRegion = undefined;- pendingCaptionSpace = false;+ pendingCaptionText = "";current = {id,index: segments.length,@@ -623,8 +582,6 @@ export function parseScript(tokenStart: tokens.length,sourceStart: sourceOffset + offset,contentStart: sourceOffset + contentStart,- lexicalRun: "",- lexicalMarkers: [],};offset = contentStart;if (self) closeCurrent(offset, true);@@ -687,7 +644,7 @@ export function parseScript(if (!ROLE_LABEL.test(inside)) fail("SCRIPT_ANGLE", `Unknown Script construct <${inside}>.`, offset);finishLexicalRun();precedingCaptionRegion = undefined;- pendingCaptionSpace = false;+ pendingCaptionText = "";current.atoms.push({kind: "role",label: inside,@@ -720,22 +677,15 @@ export function parseScript(if (!raw || /\s$/u.test(raw)) {fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must immediately follow a display token.", offset);}- markLastCaptionSurface(block.attributes, offset);- const token = tokens.at(-1)!;- tokens[tokens.length - 1] = { ...token, editRange: {- start: token.editRange.start, end: sourceOffset + offset + block.length,- } };+ runAttributes.push({ position: runText.length, offset, end: offset + block.length, attributes: block.attributes });offset += block.length;continue;}if (source.startsWith("||", offset)) {- if (!current || tokens.length === current.tokenStart) {- fail("SCRIPT_CAPTION_BREAK_EMPTY", "Caption Cue break must follow visible Script text.", offset);- }- captionBreaks.push({- tokenIndex: tokens.length,- range: { start: sourceOffset + offset, end: sourceOffset + offset + 2 },- });+ if (!current) fail("SCRIPT_CAPTION_BREAK_EMPTY", "Caption Cue break belongs inside a Segment.", offset);+ const value = { tokenIndex: tokens.length, range: { start: sourceOffset + offset, end: sourceOffset + offset + 2 } };+ captionBreaks.push(value);+ runBoundaries.push({ position: runText.length, offset, kind: "cue", value });offset += 2;}}diff --git a/packages/script/src/types.ts b/packages/script/src/types.tsindex 79268788..123dd187 100644--- a/packages/script/src/types.ts+++ b/packages/script/src/types.ts@@ -68,7 +68,7 @@ export type ParsedCaptionRegion = {readonly startToken: number;readonly endTokenExclusive: number;readonly kind: "identity" | "alias" | "hidden";- /** Flat word attributes authored on the display side; indices address displayWordSurfaces(display). */+ /** Flat word attributes authored on the display side; indices address displaySurfaces(display). */readonly marks: readonly {readonly displayIndex: number;readonly attributes: readonly CaptionWordAttribute[];diff --git a/packages/script/test/prose-boundaries.test.ts b/packages/script/test/prose-boundaries.test.tsnew file mode 100644index 00000000..841fb801--- /dev/null+++ b/packages/script/test/prose-boundaries.test.ts@@ -0,0 +1,54 @@+import assert from "node:assert/strict";+import test from "node:test";+import { adjustScriptMoment, captionDocument, parseScript, serializeCaption, serializeSpeech } from "@hypit/script";++const parse = (text: string) => parseScript("boundary", `<line>${text}</line>`);+const caption = (text: string) => {+ const document = captionDocument(parse(text), "caption", "story");+ return document.words.map(word => word.separatorBefore + word.text).join("");+};++test("annotations cannot create speech boundaries inside words or attached punctuation", () => {+ for (const body of ['hel@{beat!}lo', '“@{beat!}测试”', 'hello@{beat!}, world',+ 'hel{emphasis}lo', '한{emphasis}글', 'hel||lo', '<hel{emphasis}lo|hello>', '<hel{emphasis}lo|>']) {+ assert.throws(() => parse(body), /SCRIPT_(?:MARKER_TOKEN_BOUNDARY|CAPTION_BREAK_TOKEN_BOUNDARY|ATTRIBUTE_TARGET)/u, body);+ }+ assert.equal(caption('@{beat!}“测试”'), '“测试”');+ assert.equal(caption('hello,@{beat!} world'), 'hello, world');+});++test("comments do not divide words, and source edits retain the comment", () => {+ const source = '<line>hel<!-- note -->lo 한<!-- 설명 -->글. @{beat!}</line>';+ const parsed = parseScript("boundary", source);+ assert.deepEqual(parsed.tokens.map(token => token.text), ['hello', '한글']);+ assert.equal(serializeSpeech(parsed), 'hello 한글.');+ const changed = adjustScriptMoment({ sourceName: "boundary", source, parsed,+ adjustment: { id: 'beat', anchorId: parsed.tokens[0]!.startAnchorId } });+ assert.equal(changed, '<line>@{beat!}hel<!-- note -->lo 한<!-- 설명 -->글. </line>');+});++test("Dual display spelling does not require speech characters", () => {+ for (const [body, expected] of [+ ['<😀|smile>', '😀'], ['<❤️|love>', '❤️'], ['<.|dot>', '.'],+ ['<!?|surprise>', '!?'], ['“<测试|>”', '“测试”'], ['<😀{emphasis}|smile>', '😀'],+ ]) assert.equal(caption(body!), expected, body);+});++test("explicit and shared Dual attributes address visible words, not prior attribute syntax", () => {+ for (const body of ['<one{emphasis} two{strong}|one two>', '<one{emphasis} two{strong}|>',+ '<하나{emphasis} 둘{strong}|하나 둘>']) {+ const document = captionDocument(parse(body), 'caption', 'story');+ assert.deepEqual(document.words.map(word => word.attributes), [+ [{ name: 'emphasis', value: true }], [{ name: 'strong', value: true }],+ ]);+ }+});++test("explicit groups keep internal speech anchors while display separators stay authored", () => {+ const parsed = parse('<组@{beat!}件{emphasis}化|> 3D 한글');+ assert.equal(serializeCaption(parsed), '组件化 3D 한글');+ const document = captionDocument(parsed, 'caption', 'story');+ assert.deepEqual(document.units[0]!.sourceTokenIds, parsed.tokens.slice(0, 3).map(token => token.id));+ assert.equal(parsed.moments[0]!.anchorId, parsed.tokens[1]!.startAnchorId);+ assert.equal(document.words.map(word => word.separatorBefore + word.text).join(''), '组件化 3D 한글');+});diff --git a/skills/hypit/references/creation/script-and-time.md b/skills/hypit/references/creation/script-and-time.mdindex b9caaf83..f87ed5ae 100644--- a/skills/hypit/references/creation/script-and-time.md+++ b/skills/hypit/references/creation/script-and-time.md@@ -295,7 +295,7 @@ Use explicit seconds for genuinely clock-based or speechless design.Every semantic marker is enclosed in `@{...}`, with `/`, `!` and `~` inside. The marker contributesno text or whitespace. `是的@{part}就是这样@{/part}` stays joined; `是的 @{part}就是这样@{/part}`keeps its authored space. Do not add spaces to make a marker parse. Markers cannot split a speech-Token. Attributes stay attached to their display word: `word{emphasis}@{beat!}`. Write literal+Token or its attached punctuation: write `@{beat!}“测试”`, not `“@{beat!}测试”`. Attributes stay attached to their display word: `word{emphasis}@{beat!}`. Write literal`@{part}` as `\@\{part\}`. Bare 0.1 markers require explicit migration before use with 0.2.Selections may overlap, cross, or span Segments; they are named semantic ranges rather than nested