Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

fix(script): preserve complete prose across annotation boundaries

kashorin3bcd105cd2c2
diff --git a/docs/quickstart/script.md b/docs/quickstart/script.md
index 3498d1fa..7e723ffa 100644
--- a/docs/quickstart/script.md
+++ b/docs/quickstart/script.md
@@ -189,7 +189,7 @@ marker; the value is a pair of semantic anchors, not a frame span:
Every marker begins with `@{` and ends with `}`; `/`, `!` and `~` are inside. Names match
`[a-z][a-z0-9_-]{0,63}`; whitespace and nesting inside a marker are invalid. `@{beat!}` is a Moment;
-`@{part}!` opens a Selection followed by a prose exclamation mark. A marker cannot split a speech Token.
+`@{part}!` opens a Selection followed by a prose exclamation mark. A marker cannot split a speech Token or its attached punctuation: write `@{beat!}“测试”`, not `“@{beat!}测试”`.
| Marker | Meaning |
diff --git a/docs/zh/quickstart/script.md b/docs/zh/quickstart/script.md
index 3d659aa3..dcb73889 100644
--- a/docs/zh/quickstart/script.md
+++ b/docs/zh/quickstart/script.md
@@ -176,7 +176,7 @@ Selection 是内联声明的具名语义**范围**。每个名字只能有一对
所有标记都以 `@{` 开始、以 `}` 结束;`/`、`!`、`~` 均在内部。名称以小写字母开头,
后续可用小写字母、数字、`_`、`-`,总长不超过 64;内部不允许空白或嵌套。
-`@{beat!}` 是 Moment,`@{part}!` 则是区间首后跟正文感叹号。标记不能切开一个语音 Token。
+`@{beat!}` 是 Moment,`@{part}!` 则是区间首后跟正文感叹号。标记不能切开语音 Token,也不能插到它与附着标点之间:应写 `@{beat!}“测试”`,不能写 `“@{beat!}测试”`。
| 标记 | 含义 |
diff --git a/examples/complex-explainer/productions/explainer/authors/script.svml b/examples/complex-explainer/productions/explainer/authors/script.svml
index cb5ed157..53d92313 100644
--- a/examples/complex-explainer/productions/explainer/authors/script.svml
+++ b/examples/complex-explainer/productions/explainer/authors/script.svml
@@ -43,7 +43,7 @@
<HOST> 时间上呢, || 它是把稿子当做时间线的: || @{sem-bind!} 动效不是绑定在秒上, || 而是绑定在说的话上。
</semantic-time>
<semantic-demo>
- <HOST> @{sem-examples!} 比如榜单在“@{rank-reveal!} 第一名”这个词上翻开, || 产品图在说“@{product-reveal!} hypit”这时候飞进来。 || @{sem-rewrite-scene!} 就算是重写 @{sem-rewrite!} 稿子、@{sem-voice!} 换掉了配音, || @{sem-adapt!} 动效依然都会精准出现, || @{sem-syntax!} 一个时间点都不用改了!
+ <HOST> @{sem-examples!} 比如榜单在@{rank-reveal!}“ 第一名”这个词上翻开, || 产品图在说@{product-reveal!}“ hypit”这时候飞进来。 || @{sem-rewrite-scene!} 就算是重写 @{sem-rewrite!} 稿子、@{sem-voice!} 换掉了配音, || @{sem-adapt!} 动效依然都会精准出现, || @{sem-syntax!} 一个时间点都不用改了!
</semantic-demo>
<result>
<HOST> 所以用 Hypit 来做视频, || @{result-assets!} 素材生成完就能直接拿到成片, || @{result-upload!} 连一秒钟的剪辑都不需要, || @{result-compare!} 而且跟原来那条爆款的效果 || 完全一模一样!
diff --git a/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml b/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml
index c3194a61..65816545 100644
--- a/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml
+++ b/examples/ranking-football/reference-banana-from-swap/reference-banana-from-swap.svml
@@ -29,10 +29,10 @@
TierBoard consumes windows without finding frames. Cue breaks are author intent. -->
<script id="story">
<ronaldo>
- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.
+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}
</ronaldo>
<messi>
- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.
+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.
</messi>
</script>
diff --git a/examples/ranking-football/reference-banana/reference-banana.svml b/examples/ranking-football/reference-banana/reference-banana.svml
index c3194a61..65816545 100644
--- a/examples/ranking-football/reference-banana/reference-banana.svml
+++ b/examples/ranking-football/reference-banana/reference-banana.svml
@@ -29,10 +29,10 @@
TierBoard consumes windows without finding frames. Cue breaks are author intent. -->
<script id="story">
<ronaldo>
- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.
+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}
</ronaldo>
<messi>
- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.
+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.
</messi>
</script>
diff --git a/examples/ranking-football/reference-gpt.svml b/examples/ranking-football/reference-gpt.svml
index b2b02842..53a26252 100644
--- a/examples/ranking-football/reference-gpt.svml
+++ b/examples/ranking-football/reference-gpt.svml
@@ -29,10 +29,10 @@
TierBoard consumes windows without finding frames. Cue breaks are author intent. -->
<script id="story">
<ronaldo>
- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.
+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}
</ronaldo>
<messi>
- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.
+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.
</messi>
</script>
diff --git a/examples/ranking-football/reference.svml b/examples/ranking-football/reference.svml
index 529df8fd..873bfeec 100644
--- a/examples/ranking-football/reference.svml
+++ b/examples/ranking-football/reference.svml
@@ -31,10 +31,10 @@
TierBoard consumes windows without finding frames. Cue breaks are author intent. -->
<script id="story">
<ronaldo>
- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.
+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}
</ronaldo>
<messi>
- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.
+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.
</messi>
</script>
diff --git a/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml b/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml
index 8de5a7e2..10781c60 100644
--- a/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml
+++ b/examples/ranking-football/swap-effect-banana-reference-sync/swap-effect-banana-reference-sync.svml
@@ -29,10 +29,10 @@
TierBoard consumes windows without finding frames. Cue breaks are author intent. -->
<script id="story">
<ronaldo>
- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.
+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}
</ronaldo>
<messi>
- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.
+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.
</messi>
</script>
diff --git a/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml b/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml
index cc3a6cdb..10391668 100644
--- a/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml
+++ b/examples/ranking-football/swap-effect-banana/swap-effect-banana.svml
@@ -28,10 +28,10 @@
Ronaldo Segments are also the two disjoint Column reveal windows. -->
<script id="story">
<messi>
- <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>@{/ikea-sign}@{/pitch-pov}. || @{retirement-room} Went to Miami || and turned the league || into a retirement home@{/retirement-room}.
+ <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>.@{/ikea-sign}@{/pitch-pov} || @{retirement-room} Went to Miami || and turned the league || into a retirement home.@{/retirement-room}
</messi>
<ronaldo>
- <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent@{/rent-jump}. || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals}. || Absolutely the top one.
+ <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent.@{/rent-jump} || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum.@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals} || Absolutely the top one.
</ronaldo>
</script>
diff --git a/examples/ranking-football/swap-effect.svml b/examples/ranking-football/swap-effect.svml
index 5568c289..bc9c6db4 100644
--- a/examples/ranking-football/swap-effect.svml
+++ b/examples/ranking-football/swap-effect.svml
@@ -28,10 +28,10 @@
Ronaldo Segments are also the two disjoint Column reveal windows. -->
<script id="story">
<messi>
- <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>@{/ikea-sign}@{/pitch-pov}. || @{retirement-room} Went to Miami || and turned the league || into a retirement home@{/retirement-room}.
+ <HOST>Messi is the bottom. || Five foot seven, || no muscles, || no speed, || just vibes. || @{pitch-pov} Man walks around || the pitch like || @{ikea-sign} he's lost in <IKEA | eye KEE uh>.@{/ikea-sign}@{/pitch-pov} || @{retirement-room} Went to Miami || and turned the league || into a retirement home.@{/retirement-room}
</messi>
<ronaldo>
- <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent@{/rent-jump}. || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals}. || Absolutely the top one.
+ <HOST>Ronaldo? Bro is || 39 years old || @{rent-jump} and still jumps || higher than your rent.@{/rent-jump} || @{nine-hundred-goals} Nine hundred goals, || @{laboratory-body} a body built || in a laboratory, || @{museum-jawline} and a jawline || that belongs || in a museum.@{/museum-jawline}@{/laboratory-body}@{/nine-hundred-goals} || Absolutely the top one.
</ronaldo>
</script>
diff --git a/examples/ranking-football/swap-host.svml b/examples/ranking-football/swap-host.svml
index 5a907cbe..2367856d 100644
--- a/examples/ranking-football/swap-host.svml
+++ b/examples/ranking-football/swap-host.svml
@@ -29,10 +29,10 @@
TierBoard consumes windows without finding frames. Cue breaks are author intent. -->
<script id="story">
<ronaldo>
- <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point@{/trophy}@{/hair-gel}. || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy@{/greek-tragedy}@{/greek-god}. || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram@{/penaldo-social}.
+ <HOST>Ronaldo is <D | Dee> tier. || Bro has @{hair-gel} more || hair gel than @{trophy} trophies || at this point.@{/trophy}@{/hair-gel} || Built like || @{greek-god} a Greek god, || plays like || @{greek-tragedy} a Greek tragedy.@{/greek-tragedy}@{/greek-god} || But hey, || at least @{penaldo-social} he's got || the most followers || on Instagram.@{/penaldo-social}
</ronaldo>
<messi>
- <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad@{/tired-dad}. || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick@{/hat-trick}@{/walks-around}. || Definitely <S | Ess> tier.
+ <HOST>Messi? Man won || a World Cup || @{tired-dad} looking like || somebody's tired dad.@{/tired-dad} || Walks around || @{walks-around} for 89 minutes, || touches the ball || three times, || and somehow || @{hat-trick} you just watched || a hat trick.@{/hat-trick}@{/walks-around} || Definitely <S | Ess> tier.
</messi>
</script>
diff --git a/examples/ranking-football/swap-topic.svml b/examples/ranking-football/swap-topic.svml
index 7f2eb3b9..dc5ce52e 100644
--- a/examples/ranking-football/swap-topic.svml
+++ b/examples/ranking-football/swap-topic.svml
@@ -28,10 +28,10 @@
Every B-roll insert consumes an authored semantic Selection. -->
<script id="story">
<elon>
- <HOST>Elon is <D | Dee> tier. || @{money-rain} Man bought Twitter || for 44 billion dollars || @{twitter-x} just to change || the name@{/twitter-x}@{/money-rain}. || @{mars-rocket} Sends rockets || to Mars, || but @{market-crash} can't send a tweet || without crashing || the stock market@{/market-crash}@{/mars-rocket}. || @{richest-chair} Richest guy || on Earth, || but somehow || @{haircut-x} still can't buy || a good haircut@{/haircut-x}@{/richest-chair}.
+ <HOST>Elon is <D | Dee> tier. || @{money-rain} Man bought Twitter || for 44 billion dollars || @{twitter-x} just to change || the name.@{/twitter-x}@{/money-rain} || @{mars-rocket} Sends rockets || to Mars, || but @{market-crash} can't send a tweet || without crashing || the stock market.@{/market-crash}@{/mars-rocket} || @{richest-chair} Richest guy || on Earth, || but somehow || @{haircut-x} still can't buy || a good haircut.@{/haircut-x}@{/richest-chair}
</elon>
<sam>
- <HOST>Sam Altman? || @{fired-friday} Guy got fired || on a Friday, || @{company-quit} the entire company || threatened to quit, || and @{back-monday} he was back || in the chair || by Monday@{/back-monday}@{/company-quit}@{/fired-friday}. || @{elon-lawsuit} Elon sued him, || @{google-fears} Google fears him, || @{congress-grilled} Congress grilled him, || @{still-here} and he's still here@{/still-here}@{/congress-grilled}@{/google-fears}@{/elon-lawsuit}. || I have to || put him in || <S | Ess> tier.
+ <HOST>Sam Altman? || @{fired-friday} Guy got fired || on a Friday, || @{company-quit} the entire company || threatened to quit, || and @{back-monday} he was back || in the chair || by Monday.@{/back-monday}@{/company-quit}@{/fired-friday} || @{elon-lawsuit} Elon sued him, || @{google-fears} Google fears him, || @{congress-grilled} Congress grilled him, || @{still-here} and he's still here.@{/still-here}@{/congress-grilled}@{/google-fears}@{/elon-lawsuit} || I have to || put him in || <S | Ess> tier.
</sam>
</script>
diff --git a/packages/script/README.md b/packages/script/README.md
index 44db0f94..d2def36c 100644
--- a/packages/script/README.md
+++ b/packages/script/README.md
@@ -117,6 +117,15 @@ following characters. Display punctuation attaches to neighboring words without
These units support precise timing and highlighting. A Caption Cue can hold a whole phrase of them;
`||` chooses its handoff independently of character counts or visual line wrapping.
+Annotations do not create speech boundaries. Comments are transparent (`hel<!--note-->lo`
+remains `hello`); a postfix attribute or `||` inside a word is invalid. Script analyzes a complete
+prose run before binding these constructs. An explicit Dual correspondence and a speaker/Segment
+boundary remain authored structure. Shared Dual groups still expose their internal speech anchors.
+
+A Dual display side is literal authored text, including symbols and emoji: `<😀|smile>` and
+`<.|dot>` have explicit speech correspondence and require no invented speech token for the symbol.
+A literal-only display is one display surface within that correspondence.
+
## Display spelling and separators
Script preserves the normalized display spelling independently of speech tokenization. Ordinary
@@ -139,7 +148,8 @@ Every marker starts with `@{` and ends with `}`. The complete marker is zero-wid
display; surrounding prose whitespace remains prose. All control sigils belong inside the braces:
`@{beat!}` is a Moment, whereas `@{part}!` opens a Selection followed by a literal exclamation mark.
Names match `[a-z][a-z0-9_-]{0,63}`; whitespace and nesting inside a marker are invalid.
-A marker cannot split a speech Token. `hello{emphasis}` is a postfix display attribute; `@{part}`
+A marker cannot split a speech Token or separate it from attached punctuation: place
+`@{beat!}“测试”`, not `“@{beat!}测试”`. `hello{emphasis}` is a postfix display attribute; `@{part}`
is consumed as one marker and cannot be mistaken for that attribute. Write a literal `@{part}` as
`\@\{part\}`.
@@ -199,6 +209,9 @@ Use `--body` for a file containing a raw Script body rather than outer SVML. The
markers only inside Script bodies, leaves comments and escapes intact, and does not touch provider
prompt references such as `@image1`. It neither installs anything nor runs during a build.
+The tool changes marker spelling, not marker placement. Move a marker that separates a word from
+its attached quote or punctuation to the complete word boundary before using that source.
+
Review authored whitespace after migration: spaces previously discarded by Chinese/punctuation
normalization now appear. The tool preserves source spaces rather than guessing the author's intent.
Regenerate affected Narrative, caption and Build results with the new reader/writer together;
diff --git a/packages/script/src/lexical.ts b/packages/script/src/lexical.ts
index 21514133..b54a8d26 100644
--- a/packages/script/src/lexical.ts
+++ b/packages/script/src/lexical.ts
@@ -9,146 +9,88 @@ const WORD_CHARACTER = String.raw`(?:(?!${CHARACTER_UNIT})[\p{L}\p{M}\p{N}])`;
const LEXICAL_UNIT = new RegExp([
String.raw`(?:\p{N}{1,3}(?:[,,]\p{N}{3})+|\p{N}+)(?:[..]\p{N}+)?(?:-\p{N}+(?:[..]\p{N}+)?)*(?!\p{N}|-[\p{L}\p{M}])`,
String.raw`${CHARACTER_UNIT}\p{M}*`,
- String.raw`${WORD_CHARACTER}+(?:['’.-]${WORD_CHARACTER}+)*`,
+ String.raw`(?:(?!${CHARACTER_UNIT})[\p{L}\p{N}])${WORD_CHARACTER}*(?:['’.-]${WORD_CHARACTER}+)*`,
].join("|"), "gu");
-const OPENING_PUNCTUATION = new Set([
- "(", "[", "{", "(", "【", "《", "「", "『", "〔", "〈", "“", "‘",
- "$", "¥", "¥", "€", "£",
-]);
-
-const QUOTE_PUNCTUATION = new Set(["\"", "'", "`", "’", "ʼ"]);
-const UNICODE_OPENING_PUNCTUATION = /[\p{Ps}\p{Pi}]/u;
-
-/**
- * ASCII quotation marks have no Unicode opening/closing category. In a gap between
- * lexical units, whitespace before the mark is the useful authoring signal:
- * `said "hello` opens a quote, while `hello" world` closes one. A quote before the
- * first unit is opening by definition; contractions stay inside one lexical unit.
- */
-function isOpeningPunctuation(
- character: string,
- gap: string,
- position: number,
- hasPreviousSurface: boolean,
-): boolean {
- if (OPENING_PUNCTUATION.has(character) || UNICODE_OPENING_PUNCTUATION.test(character)) return true;
- if (!QUOTE_PUNCTUATION.has(character)) return false;
- if (!hasPreviousSurface) return true;
- const before = [...gap.slice(0, position)].at(-1);
- return before !== undefined && /\s/u.test(before);
+// Unicode owns bracket/quote categories. ASCII symmetric quotes need local context;
+// they have no opening/closing category. This is prose analysis, not marker syntax.
+function openingAt(gap: string, hasPrevious: boolean, beforeWord = true): number {
+ for (const match of gap.matchAll(/./gu)) {
+ const character = match[0];
+ if (/[\p{Ps}\p{Pi}\p{Sc}]/u.test(character)
+ || ((/["'`]/u.test(character) || (beforeWord && /\p{Pf}/u.test(character))) && (!hasPrevious || /\s$/u.test(gap.slice(0, match.index))))) return match.index;
+ }
+ return gap.length;
}
export function lexicalUnits(value: string): readonly LexicalUnit[] {
- return [...value.matchAll(LEXICAL_UNIT)].map((match) => ({
- text: match[0],
- index: match.index,
- }));
+ return [...value.matchAll(LEXICAL_UNIT)].map(match => ({ text: match[0], index: match.index }));
}
export function lexicalCount(value: string): number {
return lexicalUnits(value).length;
}
-/** Source positions for markers; unlike timing units, these include attached punctuation. */
-export function lexicalEditRanges(value: string): readonly { start: number; end: number }[] {
- const units = lexicalUnits(value);
- return units.map((unit, index) => {
- const previousEnd = index === 0 ? 0 : units[index - 1]!.index + units[index - 1]!.text.length;
- const nextStart = units[index + 1]?.index ?? value.length;
- const leading = value.slice(previousEnd, unit.index);
- let start = unit.index;
- for (const [position, character] of [...leading.matchAll(/./gu)].map((match) => [match.index, match[0]] as const)) {
- if (isOpeningPunctuation(character, leading, position, index > 0)) { start = previousEnd + position; break; }
- }
- let end = unit.index + unit.text.length;
- const trailing = value.slice(end, nextStart);
- for (const match of trailing.matchAll(/./gu)) {
- if (/\s/u.test(match[0]) || isOpeningPunctuation(match[0], trailing, match.index, true)) break;
- end += match[0].length;
- }
- return { start, end };
- });
-}
-
-/** Normalize source formatting without inventing or removing language-specific separators. */
+/** Normalize source formatting without inventing language-specific separators. */
export function cleanProjection(value: string): string {
return value.replace(/\s+/gu, " ").trim();
}
-/** Structural markers split atoms but must not invent prose whitespace when those atoms rejoin. */
export function joinProjection(parts: readonly string[]): string {
return cleanProjection(parts.join(""));
}
-export type DisplaySurface = {
- readonly text: string;
- readonly separatorBefore: "" | " ";
-};
+export type DisplaySurface = { readonly text: string; readonly separatorBefore: "" | " " };
+type ProseWord = { start: number; end: number; separatorBefore: "" | " " };
-/** Keep the authored display spelling while assigning punctuation to lexical surfaces. */
-export function displaySurfaces(value: string): readonly DisplaySurface[] {
- const prose = cleanProjection(value);
- const units = lexicalUnits(prose);
- const surfaces: Array<{ text: string; separatorBefore: "" | " " }> = [];
+/** Analyze complete prose once. Speech cores and their written surfaces are distinct:
+ * punctuation has spelling and source extent, but does not acquire speech timing.
+ * A display-only literal (for example an emoji in Dual Text) needs no speech core.
+ */
+export function analyzeProse(value: string, followsDisplay = false): {
+ readonly units: readonly LexicalUnit[];
+ readonly surfaces: readonly DisplaySurface[];
+ readonly editRanges: readonly { start: number; end: number }[];
+} {
+ const units = lexicalUnits(value);
+ const words: ProseWord[] = [];
let cursor = 0;
for (const unit of units) {
- const gap = prose.slice(cursor, unit.index);
- let openingAt = gap.length;
- for (const match of gap.matchAll(/./gu)) {
- if (!/\s/u.test(match[0]) && isOpeningPunctuation(match[0], gap, match.index, surfaces.length > 0)) {
- openingAt = match.index;
- break;
- }
- }
- const closing = gap.slice(0, openingAt);
- const previous = surfaces.at(-1);
- if (previous) previous.text += closing.trimEnd();
- const prefix = (previous ? "" : closing.trimStart()) + gap.slice(openingAt);
- surfaces.push({ text: prefix + unit.text, separatorBefore: previous && /\s$/u.test(closing) ? " " : "" });
+ const gap = value.slice(cursor, unit.index);
+ const opening = openingAt(gap, words.length > 0 || followsDisplay);
+ const closing = gap.slice(0, opening);
+ const previous = words.at(-1);
+ if (previous) previous.end = cursor + closing.trimEnd().length;
+ const start = previous || followsDisplay ? cursor + opening : value.slice(0, unit.index).search(/\S/u);
+ words.push({ start: start < 0 ? unit.index : start, end: unit.index + unit.text.length,
+ separatorBefore: previous && /\s$/u.test(closing) ? " " : "" });
cursor = unit.index + unit.text.length;
}
- const last = surfaces.at(-1);
- if (last) last.text += prose.slice(cursor);
- return surfaces;
+ const last = words.at(-1);
+ if (last) last.end = value.trimEnd().length;
+ const surfaces = words.map(word => ({ text: value.slice(word.start, word.end).replace(/\s+/gu, " "), separatorBefore: word.separatorBefore }));
+ const editRanges = words.map((word, index) => {
+ const unit = units[index]!;
+ let end = unit.index + unit.text.length;
+ // Whitespace is a legal insertion boundary. Only adjacent trailing punctuation
+ // belongs to the indivisible edit surface ("word," versus "word ,").
+ const trailing = value.slice(end, word.end);
+ const whitespace = trailing.search(/\s/u);
+ end += whitespace < 0 ? trailing.length : whitespace;
+ return { start: word.start, end };
+ });
+ if (surfaces.length === 0 && value.trim()) surfaces.push({ text: cleanProjection(value), separatorBefore: "" });
+ return { units, surfaces, editRanges };
}
-export function displayWordSurfaces(value: string): readonly string[] {
- return displaySurfaces(value).map(surface => surface.text);
+export function displaySurfaces(value: string): readonly DisplaySurface[] {
+ return analyzeProse(value).surfaces;
}
-/** Split punctuation before the first lexical unit, retaining every authored separator. */
-export function splitDisplayPrefix(value: string): { previous: string; current: string } {
+/** Correspondence boundaries retain punctuation on either side of an explicit Dual. */
+export function splitDisplayPrefix(value: string, hasPrevious: boolean): { previous: string; current: string } {
const first = lexicalUnits(value)[0];
- if (!first) return { previous: value, current: "" };
- const leading = value.slice(0, first.index);
- for (const match of leading.matchAll(/./gu)) {
- if (!/\s/u.test(match[0]) && isOpeningPunctuation(match[0], leading, match.index, false)) {
- return { previous: leading.slice(0, match.index), current: value.slice(match.index) };
- }
- }
- return { previous: leading, current: value.slice(first.index) };
-}
-
-/** Closing punctuation before a lexical unit belongs to the previous display unit when one exists. */
-export function splitLeadingClosingPunctuation(value: string): {
- readonly previous: string;
- readonly current: string;
-} {
- const prose = cleanProjection(value);
- const first = lexicalUnits(prose)[0];
- if (first === undefined) return { previous: prose.replace(/\s+/gu, ""), current: "" };
- const leading = prose.slice(0, first.index);
- const punctuation = [...leading]
- .map((character, position) => ({ character, position }))
- .filter(({ character }) => !/\s/u.test(character));
- const previous = punctuation
- .filter(({ character, position }) => !isOpeningPunctuation(character, leading, position, false))
- .map(({ character }) => character)
- .join("");
- const opening = punctuation
- .filter(({ character, position }) => isOpeningPunctuation(character, leading, position, false))
- .map(({ character }) => character)
- .join("");
- return { previous, current: `${opening}${prose.slice(first.index)}` };
+ const end = first?.index ?? value.length;
+ const split = openingAt(value.slice(0, end), hasPrevious, first !== undefined);
+ return { previous: value.slice(0, split), current: value.slice(split) };
}
diff --git a/packages/script/src/parser.ts b/packages/script/src/parser.ts
index e3a6dbad..7ca9cd67 100644
--- a/packages/script/src/parser.ts
+++ b/packages/script/src/parser.ts
@@ -2,11 +2,9 @@
import { ScriptSyntaxError } from "./error.js";
import {
cleanProjection,
- displayWordSurfaces,
joinProjection,
lexicalUnits,
- lexicalEditRanges,
- splitLeadingClosingPunctuation,
+ analyzeProse,
splitDisplayPrefix,
} from "./lexical.js";
import type {
@@ -38,8 +36,6 @@ type MutableSegment = {
readonly tokenStart: number;
readonly sourceStart: number;
readonly contentStart: number;
- lexicalRun: string;
- readonly lexicalMarkers: Array<{ readonly position: number; readonly offset: number }>;
};
const SEGMENT_ID = /^[a-z][a-z0-9_-]{0,63}$/u;
@@ -50,7 +46,7 @@ const ATTRIBUTE_NAME = /^[a-z][a-z0-9_-]{0,63}$/u;
const ATTRIBUTE_VALUE = /^[A-Za-z0-9_.:-]+$/u;
/** A marker's structural position before its affinity picks one anchor. */
-type RawMarkerBoundary = Omit<MarkerBoundary, "anchorId">;
+type RawMarkerBoundary = { -readonly [K in keyof Omit<MarkerBoundary, "anchorId">]: Omit<MarkerBoundary, "anchorId">[K] };
type RawEdge<T> = Omit<T, "boundary"> & { readonly boundary: RawMarkerBoundary };
type RawSelection = Omit<ParsedSelection, "startAnchorId" | "endAnchorId" | "open" | "close"> & {
readonly open: RawEdge<ParsedSelection["open"]>;
@@ -127,7 +123,7 @@ export function parseScript(
const tokens: ParsedToken[] = [];
const captionRegions: ParsedCaptionRegion[] = [];
let precedingCaptionRegion: number | undefined;
- let pendingCaptionSpace = false;
+ let pendingCaptionText = "";
const selections = new Map<string, RawSelection>();
const moments = new Map<string, RawMoment>();
const captionBreaks: Array<{ readonly tokenIndex: number; readonly range: { readonly start: number; readonly end: number } }> = [];
@@ -141,33 +137,64 @@ export function parseScript(
let offset = 0;
let structuralPosition = 0;
- const boundary = (): RawMarkerBoundary => ({
- tokenIndex: tokens.length,
- structuralPosition,
- ...(current ? { segmentId: current.id } : {}),
- });
+ // One prose run crosses annotations and comments. Only an explicit correspondence,
+ // speaker change or Segment edge ends it. Source positions stay local to parsing.
+ let runText = "";
+ let runCaption = "";
+ let runCapture = true;
+ let runStarts: number[] = [];
+ let runEnds: number[] = [];
+ let runBoundaries: Array<{ position: number; offset: number; kind: "marker" | "cue"; value: { tokenIndex: number } }> = [];
+ let runAttributes: Array<{ position: number; offset: number; end: number; attributes: readonly CaptionWordAttribute[] }> = [];
+
+ const boundary = (start: number): RawMarkerBoundary => {
+ const value = { tokenIndex: tokens.length, structuralPosition,
+ ...(current ? { segmentId: current.id } : {}) };
+ if (current) runBoundaries.push({ position: runText.length, offset: start, kind: "marker", value });
+ return value;
+ };
- const finishLexicalRun = (): void => {
- if (!current) return;
- for (const match of lexicalUnits(current.lexicalRun)) {
- const start = match.index;
- const end = start + match.text.length;
- const split = current.lexicalMarkers.find((marker) => start < marker.position && marker.position < end);
- if (split) {
- fail(
- "SCRIPT_MARKER_TOKEN_BOUNDARY",
- "A Selection or Moment marker cannot split a speech token.",
- split.offset,
- );
+ const finishLexicalRun = (): ParsedCaptionRegion["marks"] => {
+ if (!current) return [];
+ const { units, editRanges: ranges } = analyzeProse(runText, runCapture && precedingCaptionRegion !== undefined);
+ const tokenStart = tokens.length;
+ for (const event of runBoundaries) {
+ if (ranges.some(range => range.start < event.position && event.position < range.end)) {
+ fail(event.kind === "marker" ? "SCRIPT_MARKER_TOKEN_BOUNDARY" : "SCRIPT_CAPTION_BREAK_TOKEN_BOUNDARY",
+ "A semantic marker or Cue break cannot split a word and its attached punctuation.", event.offset);
}
+ event.value.tokenIndex = tokenStart + units.filter(unit => unit.index + unit.text.length <= event.position).length;
+ if (event.kind === "cue" && event.value.tokenIndex === current.tokenStart) {
+ fail("SCRIPT_CAPTION_BREAK_EMPTY", "Caption Cue break must follow visible Script text.", event.offset);
+ }
+ }
+ if (units.length === 0 && runAttributes.length > 0) fail("SCRIPT_ATTRIBUTE_TARGET", "A spoken display attribute needs a complete word.", runAttributes[0]!.offset);
+ const marks = bindDisplayAttributes(runText, runAttributes, units);
+ for (const [ordinal, unit] of units.entries()) {
+ const index = tokens.length;
+ const segmentTokenIndex = index - current.tokenStart;
+ const range = ranges[ordinal]!;
+ const attachedAttributes = runAttributes.filter(attribute =>
+ attribute.position >= unit.index + unit.text.length && attribute.position <= range.end);
+ tokens.push({ id: tokenId(current.id, segmentTokenIndex), index, segmentId: current.id, segmentTokenIndex,
+ startAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "start"),
+ endAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "end"),
+ text: unit.text, normalized: normalizeWord(unit.text),
+ range: { start: runStarts[unit.index]!, end: runEnds[unit.index + unit.text.length - 1]! },
+ editRange: { start: runStarts[range.start]!, end: Math.max(runEnds[range.end - 1]!, ...attachedAttributes.map(attribute => sourceOffset + attribute.end)) },
+ });
+ }
+ if (runText || runCaption) {
+ const range = { start: runStarts[0]!, end: runEnds.at(-1)! };
+ current.atoms.push({ kind: "text", speech: runText, caption: runCaption, tokenStart, tokenEndExclusive: tokens.length, range });
+ if (runCapture) addCaptionRegion(runCaption, current.id, tokenStart, tokens.length, "identity", range, marks);
}
- current.lexicalRun = "";
- current.lexicalMarkers.length = 0;
+ runText = ""; runCaption = ""; runStarts = []; runEnds = []; runBoundaries = []; runAttributes = [];
+ return marks;
};
const addMarker = (marker: Marker, start: number): void => {
if (!TEMPORAL_ID.test(marker.id)) fail("SCRIPT_TEMPORAL_ID", `Invalid temporal id "${marker.id}".`, start);
- if (current) current.lexicalMarkers.push({ position: current.lexicalRun.length, offset: start });
if (marker.kind === "moment") {
if (openSelections.has(marker.id) || selections.has(marker.id)) {
fail("SCRIPT_TEMPORAL_TYPE", `Temporal name "${marker.id}" cannot be both Selection and Moment.`, start);
@@ -178,7 +205,7 @@ export function parseScript(
moments.set(marker.id, {
id: marker.id,
affinity: marker.affinity,
- boundary: boundary(),
+ boundary: boundary(start),
range: { start: sourceOffset + start, end: sourceOffset + start + marker.length },
});
return;
@@ -195,7 +222,7 @@ export function parseScript(
}
openSelections.set(marker.id, {
affinity: marker.affinity,
- boundary: boundary(),
+ boundary: boundary(start),
start: sourceOffset + start,
end: sourceOffset + start + marker.length,
});
@@ -213,7 +240,7 @@ export function parseScript(
},
close: {
affinity: marker.affinity,
- boundary: boundary(),
+ boundary: boundary(start),
range: { start: sourceOffset + start, end: sourceOffset + start + marker.length },
},
});
@@ -286,13 +313,32 @@ export function parseScript(
return { attributes, length: close + 1 };
};
+ const bindDisplayAttributes = (
+ display: string,
+ annotations: readonly { position: number; offset: number; attributes: readonly CaptionWordAttribute[] }[],
+ units = lexicalUnits(display),
+ ): ParsedCaptionRegion["marks"] => {
+ const marks: Array<ParsedCaptionRegion["marks"][number]> = [];
+ for (const annotation of annotations) {
+ const displayIndex = units.length === 0
+ ? (display.trim() && annotation.position === display.trimEnd().length ? 0 : -1)
+ : units.findLastIndex(unit => unit.index + unit.text.length <= annotation.position);
+ if (displayIndex < 0 || units.some(unit => unit.index < annotation.position && annotation.position < unit.index + unit.text.length)) {
+ fail("SCRIPT_ATTRIBUTE_TARGET", "A display attribute must follow a complete word.", annotation.offset);
+ }
+ if (marks.some(mark => mark.displayIndex === displayIndex)) fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display word cannot receive two separate attribute blocks.", annotation.offset);
+ marks.push({ displayIndex, attributes: annotation.attributes });
+ }
+ return marks;
+ };
+
const parseMarkedDisplay = (
raw: string,
absoluteStart: number,
): { readonly display: string; readonly marks: readonly ParsedCaptionRegion["marks"][number][] } => {
const parts: string[] = [];
- const marks: ParsedCaptionRegion["marks"][number][] = [];
let cursor = 0;
+ const attributePositions: Array<{ position: number; offset: number; attributes: readonly CaptionWordAttribute[] }> = [];
for (let index = 0; index < raw.length; index += 1) {
if (raw[index] === "\\") {
index += 1;
@@ -304,21 +350,15 @@ export function parseScript(
if (!before || /\s$/u.test(before)) {
fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must immediately follow a display token.", absoluteStart + index);
}
- const cleanBefore = literalString(before, absoluteStart, true);
- const displayIndex = displayWordSurfaces(cleanBefore).length - 1;
- if (displayIndex < 0) {
- fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", absoluteStart + index);
- }
- if (marks.some((mark) => mark.displayIndex === displayIndex)) {
- fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display token cannot receive two separate attribute blocks.", absoluteStart + index);
- }
+ const cleanBefore = parts.join("") + literalString(raw.slice(cursor, index), absoluteStart + cursor, true);
parts.push(literalString(raw.slice(cursor, index), absoluteStart + cursor, true));
- marks.push({ displayIndex, attributes: block.attributes });
+ attributePositions.push({ position: cleanBefore.length, offset: absoluteStart + index, attributes: block.attributes });
cursor = index + block.length;
index += block.length - 1;
}
parts.push(literalString(raw.slice(cursor), absoluteStart + cursor, true));
- return { display: parts.join(""), marks };
+ const display = parts.join("");
+ return { display, marks: bindDisplayAttributes(display, attributePositions) };
};
const assertDualDisplayLiteral = (raw: string, absoluteStart: number): void => {
@@ -363,8 +403,8 @@ export function parseScript(
}
// Dual Text edge whitespace is syntax padding; ordinary prose whitespace is content.
const raw = kind === "alias" ? cleanProjection(displayValue) : displayValue;
- const value = ((pendingCaptionSpace ? " " : "") + raw).replace(/\s+/gu, " ");
- const split = splitDisplayPrefix(value);
+ const value = (pendingCaptionText + raw).replace(/\s+/gu, " ");
+ const split = splitDisplayPrefix(value, precedingCaptionRegion !== undefined);
const previous = precedingCaptionRegion === undefined ? undefined : captionRegions[precedingCaptionRegion];
if (previous && split.previous.trim()) {
captionRegions[precedingCaptionRegion!] = {
@@ -372,9 +412,15 @@ export function parseScript(
display: previous.display + split.previous.trimEnd(),
};
}
- pendingCaptionSpace = /\s$/u.test(value);
+ pendingCaptionText = /\s$/u.test(value) ? " " : "";
const display = (previous ? split.current : split.previous + split.current).trim();
- if (!display || tokenEndExclusive <= tokenStart) return;
+ if (tokenEndExclusive <= tokenStart) {
+ // A literal opener outside an explicit correspondence awaits that display,
+ // while closing punctuation has already been attached to the preceding one.
+ pendingCaptionText = previous ? split.current || pendingCaptionText : value;
+ return;
+ }
+ if (!display) return;
captionRegions.push({
id: `caption-region:${captionRegions.length + 1}`,
display,
@@ -389,100 +435,24 @@ export function parseScript(
precedingCaptionRegion = captionRegions.length - 1;
};
- const markLastCaptionSurface = (
- attributes: readonly CaptionWordAttribute[],
- offset: number,
- ): void => {
- const previous = captionRegions.at(-1);
- if (previous === undefined || previous.kind === "hidden") {
- fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", offset);
- }
- const target = previous!;
- if (target.kind === "alias") {
- fail("SCRIPT_ATTRIBUTE_DUAL", "Annotate a Dual display word before the pipe; a mark cannot wrap a Dual Text.", offset);
- }
- const displayIndex = displayWordSurfaces(target.display).length - 1;
- if (displayIndex < 0) fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", offset);
- const existing = target.marks.find((mark) => mark.displayIndex === displayIndex);
- if (existing !== undefined) {
- fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display token cannot receive two separate attribute blocks.", offset);
- }
- captionRegions[captionRegions.length - 1] = {
- ...target,
- marks: [...target.marks, { displayIndex, attributes }],
- };
- };
-
const addText = (
speech: string,
caption: string,
start: number,
- end: number,
+ _end: number,
captureCaptionRegion = true,
sourcePositions?: readonly number[],
): void => {
if (!current) {
- if (speech.trim() || caption.trim()) {
- fail("SCRIPT_TEXT_OUTSIDE_SEGMENT", "Natural-language text is only allowed inside a named Segment.", start);
- }
+ if (speech.trim() || caption.trim()) fail("SCRIPT_TEXT_OUTSIDE_SEGMENT", "Natural-language text is only allowed inside a named Segment.", start);
return;
}
- current.lexicalRun += speech;
- const tokenStart = tokens.length;
- const editRanges = lexicalEditRanges(speech);
- const units = lexicalUnits(speech);
- // Decoded characters may occupy more than one source character (for example \@).
- const position = (index: number): number => sourceOffset + (sourcePositions?.[index] ?? start + index);
- // A punctuation-only piece can follow a display attribute or escaped source piece.
- const preceding = tokens.at(-1);
- const closing = splitLeadingClosingPunctuation(speech).previous;
- if (preceding?.segmentId === current.id && preceding.editRange.end === sourceOffset + start && closing && speech.startsWith(closing)) {
- tokens[tokens.length - 1] = { ...preceding, editRange: {
- start: preceding.editRange.start, end: position(closing.length),
- } };
- }
- for (const [unitIndex, match] of units.entries()) {
- const normalized = normalizeWord(match.text);
- if (!normalized) continue;
- const index = tokens.length;
- const segmentTokenIndex = index - current.tokenStart;
- const id = tokenId(current.id, segmentTokenIndex);
- tokens.push({
- id,
- index,
- segmentId: current.id,
- segmentTokenIndex,
- startAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "start"),
- endAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "end"),
- text: match.text,
- normalized,
- range: {
- start: position(match.index),
- end: position(match.index + match.text.length),
- },
- editRange: {
- start: position(editRanges[unitIndex]!.start),
- end: position(editRanges[unitIndex]!.end),
- },
- });
- }
- current.atoms.push({
- kind: "text",
- speech,
- caption,
- tokenStart,
- tokenEndExclusive: tokens.length,
- range: { start: sourceOffset + start, end: sourceOffset + end },
- });
- if (captureCaptionRegion) {
- addCaptionRegion(
- caption,
- current.id,
- tokenStart,
- tokens.length,
- "identity",
- { start: sourceOffset + start, end: sourceOffset + end },
- );
+ runCapture = captureCaptionRegion;
+ runText += speech;
+ runCaption += caption;
+ for (let index = 0; index < speech.length; index++) {
+ runStarts.push(sourceOffset + (sourcePositions?.[index] ?? start + index));
+ runEnds.push(sourceOffset + (sourcePositions?.[index + 1] ?? start + index + 1));
}
};
@@ -501,7 +471,6 @@ export function parseScript(
// marker edits target these words, not a synthetic copy beyond the pipe.
const shared = caption === undefined;
const spokenParts: string[] = [];
- const sharedMarks: Array<ParsedCaptionRegion["marks"][number]> = [];
let partStart = 0;
let index = 0;
let emittedCaption = false;
@@ -528,18 +497,8 @@ export function parseScript(
}
addLiteral(before, absoluteStart + partStart);
const block = parseAttributeBlock(raw.slice(index), absoluteStart + index);
- const displayIndex = displayWordSurfaces(spokenParts.join("")).length - 1;
- if (displayIndex < 0 || tokens.length === startToken) {
- fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must follow a visible display token.", absoluteStart + index);
- }
- if (sharedMarks.some(mark => mark.displayIndex === displayIndex)) {
- fail("SCRIPT_ATTRIBUTE_OVERLAP", "A display token cannot receive two separate attribute blocks.", absoluteStart + index);
- }
- sharedMarks.push({ displayIndex, attributes: block.attributes });
- const token = tokens.at(-1)!;
- tokens[tokens.length - 1] = { ...token, editRange: {
- start: token.editRange.start, end: sourceOffset + absoluteStart + index + block.length,
- } };
+ runAttributes.push({ position: runText.length, offset: absoluteStart + index,
+ end: absoluteStart + index + block.length, attributes: block.attributes });
index += block.length;
partStart = index;
continue;
@@ -557,6 +516,7 @@ export function parseScript(
}
addLiteral(raw.slice(partStart), absoluteStart + partStart);
const display = caption ?? spokenParts.join("");
+ const sharedMarks = finishLexicalRun();
if (shared && tokens.length === startToken) {
fail("SCRIPT_DUAL_EMPTY", "Dual Text with omitted speech must contain spoken text on its display side.", absoluteStart);
}
@@ -595,7 +555,6 @@ export function parseScript(
while (offset < source.length) {
if (source.startsWith("<!--", offset)) {
- finishLexicalRun();
const end = source.indexOf("-->", offset + 4);
if (end < 0) fail("SCRIPT_COMMENT", "Unclosed Script comment.", offset);
offset = end + 3;
@@ -615,7 +574,7 @@ export function parseScript(
}
const contentStart = offset + open[0].length;
precedingCaptionRegion = undefined;
- pendingCaptionSpace = false;
+ pendingCaptionText = "";
current = {
id,
index: segments.length,
@@ -623,8 +582,6 @@ export function parseScript(
tokenStart: tokens.length,
sourceStart: sourceOffset + offset,
contentStart: sourceOffset + contentStart,
- lexicalRun: "",
- lexicalMarkers: [],
};
offset = contentStart;
if (self) closeCurrent(offset, true);
@@ -687,7 +644,7 @@ export function parseScript(
if (!ROLE_LABEL.test(inside)) fail("SCRIPT_ANGLE", `Unknown Script construct <${inside}>.`, offset);
finishLexicalRun();
precedingCaptionRegion = undefined;
- pendingCaptionSpace = false;
+ pendingCaptionText = "";
current.atoms.push({
kind: "role",
label: inside,
@@ -720,22 +677,15 @@ export function parseScript(
if (!raw || /\s$/u.test(raw)) {
fail("SCRIPT_ATTRIBUTE_TARGET", "A token attribute must immediately follow a display token.", offset);
}
- markLastCaptionSurface(block.attributes, offset);
- const token = tokens.at(-1)!;
- tokens[tokens.length - 1] = { ...token, editRange: {
- start: token.editRange.start, end: sourceOffset + offset + block.length,
- } };
+ runAttributes.push({ position: runText.length, offset, end: offset + block.length, attributes: block.attributes });
offset += block.length;
continue;
}
if (source.startsWith("||", offset)) {
- if (!current || tokens.length === current.tokenStart) {
- fail("SCRIPT_CAPTION_BREAK_EMPTY", "Caption Cue break must follow visible Script text.", offset);
- }
- captionBreaks.push({
- tokenIndex: tokens.length,
- range: { start: sourceOffset + offset, end: sourceOffset + offset + 2 },
- });
+ if (!current) fail("SCRIPT_CAPTION_BREAK_EMPTY", "Caption Cue break belongs inside a Segment.", offset);
+ const value = { tokenIndex: tokens.length, range: { start: sourceOffset + offset, end: sourceOffset + offset + 2 } };
+ captionBreaks.push(value);
+ runBoundaries.push({ position: runText.length, offset, kind: "cue", value });
offset += 2;
}
}
diff --git a/packages/script/src/types.ts b/packages/script/src/types.ts
index 79268788..123dd187 100644
--- a/packages/script/src/types.ts
+++ b/packages/script/src/types.ts
@@ -68,7 +68,7 @@ export type ParsedCaptionRegion = {
readonly startToken: number;
readonly endTokenExclusive: number;
readonly kind: "identity" | "alias" | "hidden";
- /** Flat word attributes authored on the display side; indices address displayWordSurfaces(display). */
+ /** Flat word attributes authored on the display side; indices address displaySurfaces(display). */
readonly marks: readonly {
readonly displayIndex: number;
readonly attributes: readonly CaptionWordAttribute[];
diff --git a/packages/script/test/prose-boundaries.test.ts b/packages/script/test/prose-boundaries.test.ts
new file mode 100644
index 00000000..841fb801
--- /dev/null
+++ b/packages/script/test/prose-boundaries.test.ts
@@ -0,0 +1,54 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { adjustScriptMoment, captionDocument, parseScript, serializeCaption, serializeSpeech } from "@hypit/script";
+
+const parse = (text: string) => parseScript("boundary", `<line>${text}</line>`);
+const caption = (text: string) => {
+ const document = captionDocument(parse(text), "caption", "story");
+ return document.words.map(word => word.separatorBefore + word.text).join("");
+};
+
+test("annotations cannot create speech boundaries inside words or attached punctuation", () => {
+ for (const body of ['hel@{beat!}lo', '“@{beat!}测试”', 'hello@{beat!}, world',
+ 'hel{emphasis}lo', '한{emphasis}글', 'hel||lo', '<hel{emphasis}lo|hello>', '<hel{emphasis}lo|>']) {
+ assert.throws(() => parse(body), /SCRIPT_(?:MARKER_TOKEN_BOUNDARY|CAPTION_BREAK_TOKEN_BOUNDARY|ATTRIBUTE_TARGET)/u, body);
+ }
+ assert.equal(caption('@{beat!}“测试”'), '“测试”');
+ assert.equal(caption('hello,@{beat!} world'), 'hello, world');
+});
+
+test("comments do not divide words, and source edits retain the comment", () => {
+ const source = '<line>hel<!-- note -->lo 한<!-- 설명 -->글. @{beat!}</line>';
+ const parsed = parseScript("boundary", source);
+ assert.deepEqual(parsed.tokens.map(token => token.text), ['hello', '한글']);
+ assert.equal(serializeSpeech(parsed), 'hello 한글.');
+ const changed = adjustScriptMoment({ sourceName: "boundary", source, parsed,
+ adjustment: { id: 'beat', anchorId: parsed.tokens[0]!.startAnchorId } });
+ assert.equal(changed, '<line>@{beat!}hel<!-- note -->lo 한<!-- 설명 -->글. </line>');
+});
+
+test("Dual display spelling does not require speech characters", () => {
+ for (const [body, expected] of [
+ ['<😀|smile>', '😀'], ['<❤️|love>', '❤️'], ['<.|dot>', '.'],
+ ['<!?|surprise>', '!?'], ['“<测试|>”', '“测试”'], ['<😀{emphasis}|smile>', '😀'],
+ ]) assert.equal(caption(body!), expected, body);
+});
+
+test("explicit and shared Dual attributes address visible words, not prior attribute syntax", () => {
+ for (const body of ['<one{emphasis} two{strong}|one two>', '<one{emphasis} two{strong}|>',
+ '<하나{emphasis} 둘{strong}|하나 둘>']) {
+ const document = captionDocument(parse(body), 'caption', 'story');
+ assert.deepEqual(document.words.map(word => word.attributes), [
+ [{ name: 'emphasis', value: true }], [{ name: 'strong', value: true }],
+ ]);
+ }
+});
+
+test("explicit groups keep internal speech anchors while display separators stay authored", () => {
+ const parsed = parse('<组@{beat!}件{emphasis}化|> 3D 한글');
+ assert.equal(serializeCaption(parsed), '组件化 3D 한글');
+ const document = captionDocument(parsed, 'caption', 'story');
+ assert.deepEqual(document.units[0]!.sourceTokenIds, parsed.tokens.slice(0, 3).map(token => token.id));
+ assert.equal(parsed.moments[0]!.anchorId, parsed.tokens[1]!.startAnchorId);
+ assert.equal(document.words.map(word => word.separatorBefore + word.text).join(''), '组件化 3D 한글');
+});
diff --git a/skills/hypit/references/creation/script-and-time.md b/skills/hypit/references/creation/script-and-time.md
index b9caaf83..f87ed5ae 100644
--- a/skills/hypit/references/creation/script-and-time.md
+++ b/skills/hypit/references/creation/script-and-time.md
@@ -295,7 +295,7 @@ Use explicit seconds for genuinely clock-based or speechless design.
Every semantic marker is enclosed in `@{...}`, with `/`, `!` and `~` inside. The marker contributes
no text or whitespace. `是的@{part}就是这样@{/part}` stays joined; `是的 @{part}就是这样@{/part}`
keeps its authored space. Do not add spaces to make a marker parse. Markers cannot split a speech
-Token. Attributes stay attached to their display word: `word{emphasis}@{beat!}`. Write literal
+Token or its attached punctuation: write `@{beat!}“测试”`, not `“@{beat!}测试”`. Attributes stay attached to their display word: `word{emphasis}@{beat!}`. Write literal
`@{part}` as `\@\{part\}`. Bare 0.1 markers require explicit migration before use with 0.2.
Selections may overlap, cross, or span Segments; they are named semantic ranges rather than nested