Skip to content

🎥 Make videos programmatically with React

Skills: Fix caption timing guidance for edited clips

Jonny Burgerf46b453284d7
diff --git a/packages/skills/skills/remotion-captions/display-captions.md b/packages/skills/skills/remotion-captions/display-captions.md
index 1bee82282b..0e4d1cb365 100644
--- a/packages/skills/skills/remotion-captions/display-captions.md
+++ b/packages/skills/skills/remotion-captions/display-captions.md
@@ -33,8 +33,27 @@ Fetch its source code from https://www.remotion.dev/elements/captions/basic-capt
## Displaying captions alongside video content
-Put the captions next to the video so they stay in sync. The Caption editor
-writes back to the array literal on the selected `<BasicCaptions>` source node,
+Keep each clip's video and captions inside the same `<Sequence>`,
+`<Series.Sequence>`, or `<TransitionSeries.Sequence>`. Put the clip's
+`trimBefore`, `durationInFrames`, and `playbackRate` on that shared parent so
+trimming or changing the speed affects both. For independently positioned
+clips, put `from` on the shared `<Sequence>` too. Do not repeat the shared trim
+or playback rate on the video or captions, which would apply it twice.
+
+Keep caption timestamps in milliseconds relative to the original media file.
+Do not add the clip's start time in the main composition or subtract its trim
+from the caption data. The shared parent maps that source time to the edited
+timeline. Give every captioned clip its own group and inline caption array;
+do not combine captions from multiple clips into one overlay outside the
+scene timeline. Moving, reordering, trimming, or deleting the group then
+affects its video and captions together.
+
+Putting captions inside a scene is insufficient if `trimBefore` or
+`playbackRate` is still applied only to `<Video>`: its sibling captions do not
+inherit those props. Apply them to the shared parent instead.
+
+The Caption editor writes back to the array literal on the selected
+`<BasicCaptions>` source node,
so inline the captions directly in its `captions` prop. Passing a variable or
component prop such as `captions={captions}` leaves no source array for the
editor to update. If the captions were transcribed to a JSON file, copy its
@@ -46,7 +65,7 @@ Give the caption area a width and position it over the video. In this
```tsx
import { Video } from "@remotion/media";
-import { Composition, staticFile } from "remotion";
+import { Composition, Series, staticFile, useVideoConfig } from "remotion";
import { BasicCaptions } from "./basic-captions";
export const MyComposition: React.FC = () => {
@@ -54,7 +73,7 @@ export const MyComposition: React.FC = () => {
<Composition
id="MyComposition"
component={MyComponent}
- durationInFrames={150}
+ durationInFrames={750}
fps={30}
width={1920}
height={1080}
@@ -63,38 +82,105 @@ export const MyComposition: React.FC = () => {
};
export const MyComponent: React.FC = () => {
+ const { fps } = useVideoConfig();
+
return (
- <>
- <Video name="Camera footage" src={staticFile("video.mp4")} />
- <BasicCaptions
- width={900}
- style={{
- position: "absolute",
- bottom: 120,
- translate: "510px 0px",
- }}
- captions={[
- {
- text: "Hello",
- startMs: 0,
- endMs: 400,
- timestampMs: 200,
- confidence: null,
- },
- {
- text: " world",
- startMs: 400,
- endMs: 900,
- timestampMs: 650,
- confidence: null,
- },
- ]}
- />
- </>
+ <Series>
+ <Series.Sequence
+ name="Opening"
+ trimBefore={4 * fps}
+ durationInFrames={20 * fps}
+ premountFor={fps}
+ >
+ <Video
+ name="Opening footage"
+ src={staticFile("opening.mp4")}
+ premountFor={fps}
+ objectFit="cover"
+ style={{ width: "100%", height: "100%" }}
+ />
+ <BasicCaptions
+ name="Opening captions"
+ premountFor={fps}
+ width={900}
+ style={{
+ position: "absolute",
+ bottom: 120,
+ translate: "510px 0px",
+ }}
+ captions={[
+ {
+ text: "Hello",
+ startMs: 5000,
+ endMs: 5400,
+ timestampMs: 5200,
+ confidence: null,
+ },
+ {
+ text: " world",
+ startMs: 5400,
+ endMs: 5900,
+ timestampMs: 5650,
+ confidence: null,
+ },
+ ]}
+ />
+ </Series.Sequence>
+ <Series.Sequence
+ name="Next clip"
+ trimBefore={1 * fps}
+ durationInFrames={5 * fps}
+ premountFor={fps}
+ >
+ <Video
+ name="Next footage"
+ src={staticFile("next.mp4")}
+ premountFor={fps}
+ objectFit="cover"
+ style={{ width: "100%", height: "100%" }}
+ />
+ <BasicCaptions
+ name="Next captions"
+ premountFor={fps}
+ width={900}
+ style={{
+ position: "absolute",
+ bottom: 120,
+ translate: "510px 0px",
+ }}
+ captions={[
+ {
+ text: "Next",
+ startMs: 2000,
+ endMs: 2400,
+ timestampMs: 2200,
+ confidence: null,
+ },
+ {
+ text: " clip",
+ startMs: 2400,
+ endMs: 2900,
+ timestampMs: 2650,
+ confidence: null,
+ },
+ ]}
+ />
+ </Series.Sequence>
+ </Series>
);
};
```
+The opening skips four seconds of source footage, so its caption at `5000`
+milliseconds appears one second into the scene. The next clip's timestamps
+refer to `next.mp4`, not the main composition. Changing the opening's duration
+moves the next clip and its captions together.
+
+To change a captioned clip's speed, set `playbackRate` on its shared sequence.
+`durationInFrames` selects a range of source frames; the group occupies
+`durationInFrames / playbackRate` frames in its parent timeline. Update the
+composition duration when changing the total timeline length.
+
If a caption has `pageBreakAfter: true`, the current page ends after that caption and the next caption starts a new page.
## White-space preservation
diff --git a/packages/skills/skills/remotion-captions/transcribe-captions.md b/packages/skills/skills/remotion-captions/transcribe-captions.md
index 49ef11795c..1751da46d1 100644
--- a/packages/skills/skills/remotion-captions/transcribe-captions.md
+++ b/packages/skills/skills/remotion-captions/transcribe-captions.md
@@ -165,6 +165,11 @@ export const transcribeFile = async (file: File) => {
};
```
-Transcribe each clip individually.
+Transcribe each original source video or audio file individually. Keep the
+resulting caption timestamps relative to the beginning of that source file,
+including portions currently trimmed out of the edit. Do not add composition
+offsets or flatten multiple transcripts into composition-wide timestamps.
+Apply editing offsets, trims, and playback rates through each clip's shared
+timing group when displaying its captions.
See [Displaying captions](display-captions.md) for how to display the captions in Remotion.
diff --git a/packages/skills/skills/remotion-markup/video-editing.md b/packages/skills/skills/remotion-markup/video-editing.md
index af8eab9f92..36aa1f2e28 100644
--- a/packages/skills/skills/remotion-markup/video-editing.md
+++ b/packages/skills/skills/remotion-markup/video-editing.md
@@ -28,10 +28,31 @@ directly in each clip's JSX, for example `trimBefore={4 * fps}`. Do not use a
separate fixed `FPS` constant: Studio must be able to update each clip's source
expression when the user trims or splits it.
+## Clips with synchronized captions or overlays
+
+For a clip with synchronized captions or overlays, make a shared timing group
+the editable timeline item. Keep the video and its synchronized layers inside
+one `<Sequence>` for independently positioned clips, or inside the same
+`<Series.Sequence>` or `<TransitionSeries.Sequence>` for consecutive clips.
+
+Put the clip's `trimBefore`, `durationInFrames`, and `playbackRate` on that
+shared parent, plus `from` when using an independently positioned `<Sequence>`.
+Apply the shared timing once, rather than repeating it on the children.
+Moving, trimming, changing the speed, reordering, or deleting the group then
+affects all its layers together. A shared wrapper has a purpose here even
+though the children accept timing props themselves.
+
+Keep each clip's captions in an inline array on its own `<BasicCaptions>` node,
+with timestamps relative to the original media file. Do not put a combined
+caption overlay outside the clip groups. See
+[Displaying captions alongside video content](../remotion-captions/display-captions.md#displaying-captions-alongside-video-content)
+for the complete example.
+
## Independently positioned clips
-Place every `<Video>` directly in the composition and write its timing props
-on the JSX node. Use literal frame counts when they are already known;
+For clips without synchronized captions or overlays, place each `<Video>`
+directly in the composition and write its timing props on the JSX node.
+Use literal frame counts when they are already known;
`from={0}` may be omitted:
```tsx
@@ -72,7 +93,9 @@ overlaps are therefore allowed.
## Consecutive clips with `Series`
Use `<Series>` when the clips should remain adjacent and do not need
-transitions:
+transitions. The example below has no synchronized captions; for captioned
+clips, put the captions inside each sequence and move the video's trim to
+that shared sequence as described above:
```tsx
<Series>
@@ -116,7 +139,9 @@ Keep every sequence as a separate JSX node with a hardcoded `name` and
## Consecutive clips with transitions
Use `<TransitionSeries>` when the timeline needs transitions or overlays.
-Preserve the same one-source-node-per-clip structure:
+Preserve the same one-source-node-per-clip structure. As with `<Series>`, put
+a clip's synchronized captions inside its sequence and apply shared timing
+to that parent:
```tsx
<TransitionSeries name="Video timeline">