Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

[fix] loop support for score and refinement

zhengzangwb70b749e5570
diff --git a/README.md b/README.md
index 43c6ddd..1ca8378 100644
--- a/README.md
+++ b/README.md
@@ -278,7 +278,7 @@ In the Gradio application, the basic options are as follows:
![Gradio Demo](assets/readme/gradio_basic.png)
-The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button. The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.
+The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button (scroll down if you cannot find). The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.
Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 80G H100 GPU, the generation speed and peak memory usage is:
diff --git a/gradio/README.md b/gradio/README.md
index 3718d9a..a785cde 100644
--- a/gradio/README.md
+++ b/gradio/README.md
@@ -2,7 +2,7 @@
We have provided a Gradio demo app for you to generate videos via a web interface. You can choose to run it locally or deploy it to Hugging Face by following the instructions given below.
-## πŸš€ Run Gradio Locally
+## πŸš€ Run Gradio Locally (Outdated)
We assume that you have already installed `opensora` based on the instructions given in the [main README](../README.md). Follow the steps below to run this app on your local machine.
@@ -33,7 +33,7 @@ python gradio/app.py --share
3. You should then be able to access this demo via the link which appears in your terminal.
-## πŸ“¦ Deploy Gradio to Hugging Face Space
+## πŸ“¦ Deploy Gradio to Hugging Face Space (Outdated)
We have also tested this Gradio app on Hugging Face Spaces. You can follow the steps below.
@@ -68,3 +68,5 @@ We have also tested this Gradio app on Hugging Face Spaces. You can follow the s
For the "**FPS**" option, as now we fix the output video's FPS to 24, this option will not affect the output video's length. Thus, for a smaller FPS, the video is supposed to be longer but accelerated due to 24 FPS. Thus, the video will be less smooth but faster. For a larger FPS, the video will be smoother but slower.
For the "**Number of Loops**", it will affect the output video's length and generation speed. For example, if you set the number of loops to 2, the output video will be twice as long as the original video. This is achieved by conditioning the next generation on 1/4 of the previous generated frames and then concatenating all the frames together.
+
+A trick to give different text prompts for different parts of the video is to use the `|x|` symbol to separate the text prompts, where x is the start frame of the next text prompt. This format requires a `|0|` at the start of the prompt. For example, if you want to generate a video with the text prompt "A cat" for the first 2 generations and "A dog" for the rest generations, you can use the text prompt "|0|A cat|2|A dog". You can still check the "**Enhance prompt with GPT4o**" to refine your prompts in each part separately.
diff --git a/opensora/schedulers/rf/__init__.py b/opensora/schedulers/rf/__init__.py
index db485a2..4bc4ca6 100644
--- a/opensora/schedulers/rf/__init__.py
+++ b/opensora/schedulers/rf/__init__.py
@@ -65,9 +65,11 @@ class RFLOW:
if mask is not None:
noise_added = torch.zeros_like(mask, dtype=torch.bool)
+ noise_added = noise_added | (mask == 1)
progress_wrap = tqdm if progress else (lambda x: x)
for i, t in progress_wrap(enumerate(timesteps)):
+ # mask for adding noise
if mask is not None:
mask_t = mask * self.num_timesteps
x0 = z.clone()
@@ -75,10 +77,10 @@ class RFLOW:
mask_t_upper = mask_t >= t.unsqueeze(1)
model_args["x_mask"] = mask_t_upper.repeat(2, 1)
+ mask_add_noise = mask_t_upper & ~noise_added
- mask_add_noise = mask_t_upper & ~noise_added & (mask != 1)
- noise_added = noise_added | mask_add_noise
z = torch.where(mask_add_noise[:, None, :, None, None], x_noise, x0)
+ noise_added = mask_t_upper
# classifier-free guidance
z_in = torch.cat([z, z], 0)
diff --git a/opensora/utils/config_utils.py b/opensora/utils/config_utils.py
index ecac612..1f67a76 100644
--- a/opensora/utils/config_utils.py
+++ b/opensora/utils/config_utils.py
@@ -54,6 +54,7 @@ def parse_args(training=False):
parser.add_argument("--image-size", default=None, type=int, nargs=2, help="image size")
parser.add_argument("--frame-interval", default=None, type=int, help="frame interval")
parser.add_argument("--aspect-ratio", default=None, type=str, help="aspect ratio (h:w)")
+ parser.add_argument("--watermark", default=None, type=str2bool, help="watermark video")
# hyperparameters
parser.add_argument("--num-sampling-steps", default=None, type=int, help="sampling steps")
@@ -66,6 +67,7 @@ def parse_args(training=False):
parser.add_argument("--mask-strategy", default=None, type=str, nargs="+", help="mask strategy")
parser.add_argument("--aes", default=None, type=float, help="aesthetic score")
parser.add_argument("--flow", default=None, type=float, help="flow score")
+ parser.add_argument("--camera-motion", default=None, type=str, help="camera motion")
# ======================================================
# Training
# ======================================================
diff --git a/opensora/utils/inference_utils.py b/opensora/utils/inference_utils.py
index 0c177da..a00dc22 100644
--- a/opensora/utils/inference_utils.py
+++ b/opensora/utils/inference_utils.py
@@ -52,22 +52,18 @@ def get_save_path_name(
return save_path
-def append_score_to_prompts(prompts, aes=None, flow=None):
- score_prompts = []
- if aes is not None:
- score_prompts.append(f"aesthetic score: {aes:.1f}")
- if flow is not None:
- score_prompts.append(f"motion score: {flow:.1f}")
- if len(score_prompts) > 0:
- score_text = ", ".join(score_prompts)
- new_prompts = []
- for prompt in prompts:
- if "score:" in prompt:
- new_prompts.append(prompt)
- else:
- new_prompts.append(f"{prompt} {score_text}")
- return new_prompts
- return prompts
+def append_score_to_prompts(prompts, aes=None, flow=None, camera_motion=None):
+ new_prompts = []
+ for prompt in prompts:
+ new_prompt = prompt
+ if aes is not None and "aesthetic score:" not in prompt:
+ new_prompt = f"{new_prompt} aesthetic score: {aes:.1f}."
+ if flow is not None and "motion score:" not in prompt:
+ new_prompt = f"{new_prompt} motion score: {flow:.1f}."
+ if camera_motion is not None and "camera motion:" not in prompt:
+ new_prompt = f"{new_prompt} camera motion: {camera_motion}."
+ new_prompts.append(new_prompt)
+ return new_prompts
def extract_json_from_prompts(prompts, reference, mask_strategy):
@@ -113,7 +109,7 @@ def extract_prompts_loop(prompts, num_loop):
for i in range(0, len(prompt_list), 2):
start_loop = int(prompt_list[i])
text = prompt_list[i + 1]
- end_loop = int(prompt_list[i + 2]) if i + 2 < len(prompt_list) else num_loop
+ end_loop = int(prompt_list[i + 2]) if i + 2 < len(prompt_list) else num_loop + 1
text_list.extend([text] * (end_loop - start_loop))
prompt = text_list[num_loop]
ret_prompts.append(prompt)
@@ -180,18 +176,20 @@ def apply_mask_strategy(z, refs_x, mask_strategys, loop_i, align=None):
return masks
-def append_generated(vae, generated_video, refs_x, mask_strategy, loop_i, condition_frame_length):
+def append_generated(vae, generated_video, refs_x, mask_strategy, loop_i, condition_frame_length, condition_frame_edit):
ref_x = vae.encode(generated_video)
for j, refs in enumerate(refs_x):
if refs is None:
refs_x[j] = [ref_x[j]]
else:
refs.append(ref_x[j])
- if mask_strategy[j] is None:
+ if mask_strategy[j] is None or mask_strategy[j] == "":
mask_strategy[j] = ""
else:
mask_strategy[j] += ";"
- mask_strategy[j] += f"{loop_i},{len(refs)-1},-{condition_frame_length},0,{condition_frame_length}"
+ mask_strategy[
+ j
+ ] += f"{loop_i},{len(refs)-1},-{condition_frame_length},0,{condition_frame_length},{condition_frame_edit}"
return refs_x, mask_strategy
@@ -279,9 +277,13 @@ def refine_prompts_by_openai(prompts):
return new_prompts
-def add_watermark(input_video_path, watermark_image_path, output_video_path):
+def add_watermark(
+ input_video_path, watermark_image_path="./assets/images/watermark/watermark.png", output_video_path=None
+):
# execute this command in terminal with subprocess
# return if the process is successful
- cmd = f"ffmpeg -y -i {input_video_path} -i {watermark_image_path} -filter_complex \"[1][0]scale2ref=oh*mdar:ih*0.1[logo][video];[video][logo]overlay\" {output_video_path}"
+ if output_video_path is None:
+ output_video_path = input_video_path.replace(".mp4", "_watermark.mp4")
+ cmd = f'ffmpeg -y -i {input_video_path} -i {watermark_image_path} -filter_complex "[1][0]scale2ref=oh*mdar:ih*0.1[logo][video];[video][logo]overlay" {output_video_path}'
exit_code = os.system(cmd)
return exit_code == 0
diff --git a/scripts/inference.py b/scripts/inference.py
index 1461198..d7ab9f1 100644
--- a/scripts/inference.py
+++ b/scripts/inference.py
@@ -1,4 +1,5 @@
import os
+import time
from pprint import pformat
import colossalai
@@ -15,6 +16,7 @@ from opensora.models.text_encoder.t5 import text_preprocessing
from opensora.registry import MODELS, SCHEDULERS, build_module
from opensora.utils.config_utils import parse_configs
from opensora.utils.inference_utils import (
+ add_watermark,
append_generated,
append_score_to_prompts,
apply_mask_strategy,
@@ -128,6 +130,7 @@ def main():
num_sample = cfg.get("num_sample", 1)
loop = cfg.get("loop", 1)
condition_frame_length = cfg.get("condition_frame_length", 5)
+ condition_frame_edit = cfg.get("condition_frame_edit", 0.0)
align = cfg.get("align", None)
save_dir = cfg.save_dir
@@ -149,13 +152,6 @@ def main():
# == get reference for condition ==
refs = collect_references_batch(refs, vae, image_size)
- # == refine prompt by openai ==
- if cfg.get("llm_refine", False):
- batch_prompts = refine_prompts_by_openai(batch_prompts)
-
- # == score ==
- batch_prompts = append_score_to_prompts(batch_prompts, aes=cfg.get("aes", None), flow=cfg.get("flow", None))
-
# == multi-resolution info ==
model_args = prepare_multi_resolution_info(
multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype
@@ -185,12 +181,29 @@ def main():
# == Iter over loop generation ==
video_clips = []
for loop_i in range(loop):
+ # == get prompt for loop i ==
batch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)
+
+ # == refine prompt by openai ==
+ if cfg.get("llm_refine", False):
+ batch_prompts_loop = refine_prompts_by_openai(batch_prompts_loop)
+
+ # == add score to prompt ==
+ batch_prompts_loop = append_score_to_prompts(
+ batch_prompts_loop,
+ aes=cfg.get("aes", None),
+ flow=cfg.get("flow", None),
+ camera_motion=cfg.get("camera_motion", None),
+ )
+
+ # == clean prompt for t5 ==
batch_prompts_cleaned = [text_preprocessing(prompt) for prompt in batch_prompts_loop]
- # == loop ==
+ # == add condition frames for loop ==
if loop_i > 0:
- refs, ms = append_generated(vae, video_clips[-1], refs, ms, loop_i, condition_frame_length)
+ refs, ms = append_generated(
+ vae, video_clips[-1], refs, ms, loop_i, condition_frame_length, condition_frame_edit
+ )
# == sampling ==
z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)
@@ -218,12 +231,15 @@ def main():
for i in range(1, loop):
video[i] = video[i][:, dframe_to_frame(condition_frame_length) :]
video = torch.cat(video, dim=1)
- save_sample(
+ save_path = save_sample(
video,
fps=save_fps,
save_path=save_path,
verbose=verbose >= 2,
)
+ if save_path.endswith(".mp4") and cfg.get("watermark", False):
+ time.sleep(1) # prevent loading previous generated video
+ add_watermark(save_path)
start_idx += len(batch_prompts)
logger.info("Inference finished.")
logger.info("Saved %s samples to %s", start_idx, save_dir)