Open-Sora: Democratizing Efficient Video Production for All
Python
[fix] loop support for score and refinement
zhengzangwb70b749e5570
diff --git a/README.md b/README.mdindex 43c6ddd..1ca8378 100644--- a/README.md+++ b/README.md@@ -278,7 +278,7 @@ In the Gradio application, the basic options are as follows:-The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button. The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.+The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button (scroll down if you cannot find). The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 80G H100 GPU, the generation speed and peak memory usage is:diff --git a/gradio/README.md b/gradio/README.mdindex 3718d9a..a785cde 100644--- a/gradio/README.md+++ b/gradio/README.md@@ -2,7 +2,7 @@We have provided a Gradio demo app for you to generate videos via a web interface. You can choose to run it locally or deploy it to Hugging Face by following the instructions given below.-## π Run Gradio Locally+## π Run Gradio Locally (Outdated)We assume that you have already installed `opensora` based on the instructions given in the [main README](../README.md). Follow the steps below to run this app on your local machine.@@ -33,7 +33,7 @@ python gradio/app.py --share3. You should then be able to access this demo via the link which appears in your terminal.-## π¦ Deploy Gradio to Hugging Face Space+## π¦ Deploy Gradio to Hugging Face Space (Outdated)We have also tested this Gradio app on Hugging Face Spaces. You can follow the steps below.@@ -68,3 +68,5 @@ We have also tested this Gradio app on Hugging Face Spaces. You can follow the sFor the "**FPS**" option, as now we fix the output video's FPS to 24, this option will not affect the output video's length. Thus, for a smaller FPS, the video is supposed to be longer but accelerated due to 24 FPS. Thus, the video will be less smooth but faster. For a larger FPS, the video will be smoother but slower.For the "**Number of Loops**", it will affect the output video's length and generation speed. For example, if you set the number of loops to 2, the output video will be twice as long as the original video. This is achieved by conditioning the next generation on 1/4 of the previous generated frames and then concatenating all the frames together.++A trick to give different text prompts for different parts of the video is to use the `|x|` symbol to separate the text prompts, where x is the start frame of the next text prompt. This format requires a `|0|` at the start of the prompt. For example, if you want to generate a video with the text prompt "A cat" for the first 2 generations and "A dog" for the rest generations, you can use the text prompt "|0|A cat|2|A dog". You can still check the "**Enhance prompt with GPT4o**" to refine your prompts in each part separately.diff --git a/opensora/schedulers/rf/__init__.py b/opensora/schedulers/rf/__init__.pyindex db485a2..4bc4ca6 100644--- a/opensora/schedulers/rf/__init__.py+++ b/opensora/schedulers/rf/__init__.py@@ -65,9 +65,11 @@ class RFLOW:if mask is not None:noise_added = torch.zeros_like(mask, dtype=torch.bool)+ noise_added = noise_added | (mask == 1)progress_wrap = tqdm if progress else (lambda x: x)for i, t in progress_wrap(enumerate(timesteps)):+ # mask for adding noiseif mask is not None:mask_t = mask * self.num_timestepsx0 = z.clone()@@ -75,10 +77,10 @@ class RFLOW:mask_t_upper = mask_t >= t.unsqueeze(1)model_args["x_mask"] = mask_t_upper.repeat(2, 1)+ mask_add_noise = mask_t_upper & ~noise_added- mask_add_noise = mask_t_upper & ~noise_added & (mask != 1)- noise_added = noise_added | mask_add_noisez = torch.where(mask_add_noise[:, None, :, None, None], x_noise, x0)+ noise_added = mask_t_upper# classifier-free guidancez_in = torch.cat([z, z], 0)diff --git a/opensora/utils/config_utils.py b/opensora/utils/config_utils.pyindex ecac612..1f67a76 100644--- a/opensora/utils/config_utils.py+++ b/opensora/utils/config_utils.py@@ -54,6 +54,7 @@ def parse_args(training=False):parser.add_argument("--image-size", default=None, type=int, nargs=2, help="image size")parser.add_argument("--frame-interval", default=None, type=int, help="frame interval")parser.add_argument("--aspect-ratio", default=None, type=str, help="aspect ratio (h:w)")+ parser.add_argument("--watermark", default=None, type=str2bool, help="watermark video")# hyperparametersparser.add_argument("--num-sampling-steps", default=None, type=int, help="sampling steps")@@ -66,6 +67,7 @@ def parse_args(training=False):parser.add_argument("--mask-strategy", default=None, type=str, nargs="+", help="mask strategy")parser.add_argument("--aes", default=None, type=float, help="aesthetic score")parser.add_argument("--flow", default=None, type=float, help="flow score")+ parser.add_argument("--camera-motion", default=None, type=str, help="camera motion")# ======================================================# Training# ======================================================diff --git a/opensora/utils/inference_utils.py b/opensora/utils/inference_utils.pyindex 0c177da..a00dc22 100644--- a/opensora/utils/inference_utils.py+++ b/opensora/utils/inference_utils.py@@ -52,22 +52,18 @@ def get_save_path_name(return save_path-def append_score_to_prompts(prompts, aes=None, flow=None):- score_prompts = []- if aes is not None:- score_prompts.append(f"aesthetic score: {aes:.1f}")- if flow is not None:- score_prompts.append(f"motion score: {flow:.1f}")- if len(score_prompts) > 0:- score_text = ", ".join(score_prompts)- new_prompts = []- for prompt in prompts:- if "score:" in prompt:- new_prompts.append(prompt)- else:- new_prompts.append(f"{prompt} {score_text}")- return new_prompts- return prompts+def append_score_to_prompts(prompts, aes=None, flow=None, camera_motion=None):+ new_prompts = []+ for prompt in prompts:+ new_prompt = prompt+ if aes is not None and "aesthetic score:" not in prompt:+ new_prompt = f"{new_prompt} aesthetic score: {aes:.1f}."+ if flow is not None and "motion score:" not in prompt:+ new_prompt = f"{new_prompt} motion score: {flow:.1f}."+ if camera_motion is not None and "camera motion:" not in prompt:+ new_prompt = f"{new_prompt} camera motion: {camera_motion}."+ new_prompts.append(new_prompt)+ return new_promptsdef extract_json_from_prompts(prompts, reference, mask_strategy):@@ -113,7 +109,7 @@ def extract_prompts_loop(prompts, num_loop):for i in range(0, len(prompt_list), 2):start_loop = int(prompt_list[i])text = prompt_list[i + 1]- end_loop = int(prompt_list[i + 2]) if i + 2 < len(prompt_list) else num_loop+ end_loop = int(prompt_list[i + 2]) if i + 2 < len(prompt_list) else num_loop + 1text_list.extend([text] * (end_loop - start_loop))prompt = text_list[num_loop]ret_prompts.append(prompt)@@ -180,18 +176,20 @@ def apply_mask_strategy(z, refs_x, mask_strategys, loop_i, align=None):return masks-def append_generated(vae, generated_video, refs_x, mask_strategy, loop_i, condition_frame_length):+def append_generated(vae, generated_video, refs_x, mask_strategy, loop_i, condition_frame_length, condition_frame_edit):ref_x = vae.encode(generated_video)for j, refs in enumerate(refs_x):if refs is None:refs_x[j] = [ref_x[j]]else:refs.append(ref_x[j])- if mask_strategy[j] is None:+ if mask_strategy[j] is None or mask_strategy[j] == "":mask_strategy[j] = ""else:mask_strategy[j] += ";"- mask_strategy[j] += f"{loop_i},{len(refs)-1},-{condition_frame_length},0,{condition_frame_length}"+ mask_strategy[+ j+ ] += f"{loop_i},{len(refs)-1},-{condition_frame_length},0,{condition_frame_length},{condition_frame_edit}"return refs_x, mask_strategy@@ -279,9 +277,13 @@ def refine_prompts_by_openai(prompts):return new_prompts-def add_watermark(input_video_path, watermark_image_path, output_video_path):+def add_watermark(+ input_video_path, watermark_image_path="./assets/images/watermark/watermark.png", output_video_path=None+):# execute this command in terminal with subprocess# return if the process is successful- cmd = f"ffmpeg -y -i {input_video_path} -i {watermark_image_path} -filter_complex \"[1][0]scale2ref=oh*mdar:ih*0.1[logo][video];[video][logo]overlay\" {output_video_path}"+ if output_video_path is None:+ output_video_path = input_video_path.replace(".mp4", "_watermark.mp4")+ cmd = f'ffmpeg -y -i {input_video_path} -i {watermark_image_path} -filter_complex "[1][0]scale2ref=oh*mdar:ih*0.1[logo][video];[video][logo]overlay" {output_video_path}'exit_code = os.system(cmd)return exit_code == 0diff --git a/scripts/inference.py b/scripts/inference.pyindex 1461198..d7ab9f1 100644--- a/scripts/inference.py+++ b/scripts/inference.py@@ -1,4 +1,5 @@import os+import timefrom pprint import pformatimport colossalai@@ -15,6 +16,7 @@ from opensora.models.text_encoder.t5 import text_preprocessingfrom opensora.registry import MODELS, SCHEDULERS, build_modulefrom opensora.utils.config_utils import parse_configsfrom opensora.utils.inference_utils import (+ add_watermark,append_generated,append_score_to_prompts,apply_mask_strategy,@@ -128,6 +130,7 @@ def main():num_sample = cfg.get("num_sample", 1)loop = cfg.get("loop", 1)condition_frame_length = cfg.get("condition_frame_length", 5)+ condition_frame_edit = cfg.get("condition_frame_edit", 0.0)align = cfg.get("align", None)save_dir = cfg.save_dir@@ -149,13 +152,6 @@ def main():# == get reference for condition ==refs = collect_references_batch(refs, vae, image_size)- # == refine prompt by openai ==- if cfg.get("llm_refine", False):- batch_prompts = refine_prompts_by_openai(batch_prompts)-- # == score ==- batch_prompts = append_score_to_prompts(batch_prompts, aes=cfg.get("aes", None), flow=cfg.get("flow", None))-# == multi-resolution info ==model_args = prepare_multi_resolution_info(multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype@@ -185,12 +181,29 @@ def main():# == Iter over loop generation ==video_clips = []for loop_i in range(loop):+ # == get prompt for loop i ==batch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)++ # == refine prompt by openai ==+ if cfg.get("llm_refine", False):+ batch_prompts_loop = refine_prompts_by_openai(batch_prompts_loop)++ # == add score to prompt ==+ batch_prompts_loop = append_score_to_prompts(+ batch_prompts_loop,+ aes=cfg.get("aes", None),+ flow=cfg.get("flow", None),+ camera_motion=cfg.get("camera_motion", None),+ )++ # == clean prompt for t5 ==batch_prompts_cleaned = [text_preprocessing(prompt) for prompt in batch_prompts_loop]- # == loop ==+ # == add condition frames for loop ==if loop_i > 0:- refs, ms = append_generated(vae, video_clips[-1], refs, ms, loop_i, condition_frame_length)+ refs, ms = append_generated(+ vae, video_clips[-1], refs, ms, loop_i, condition_frame_length, condition_frame_edit+ )# == sampling ==z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)@@ -218,12 +231,15 @@ def main():for i in range(1, loop):video[i] = video[i][:, dframe_to_frame(condition_frame_length) :]video = torch.cat(video, dim=1)- save_sample(+ save_path = save_sample(video,fps=save_fps,save_path=save_path,verbose=verbose >= 2,)+ if save_path.endswith(".mp4") and cfg.get("watermark", False):+ time.sleep(1) # prevent loading previous generated video+ add_watermark(save_path)start_idx += len(batch_prompts)logger.info("Inference finished.")logger.info("Saved %s samples to %s", start_idx, save_dir)