Open-Sora: Democratizing Efficient Video Production for All
Python
format and some fix
Zangwei Zheng52aeb3769f33
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex b2ef579..d606846 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -4,7 +4,7 @@ The Open-Sora project welcomes any constructive contribution from the community## Development Environment Setup-To contribute to Open-Sora, we would like to first guide you to set up a proper development environment so that you can better implement your code. You can install this library from source with the `editable` flag (`-e`, for development mode) so that your change to the source code will be reflected in runtime without re-installation.+To contribute to Open-Sora, we would like to first guide you to set up a proper development environment so that you can better implement your code. You can install this library from source with the `editable` flag (`-e`, for development mode) so that your change to the source code will be reflected in runtime without re-installation.You can refer to the [Installation Section](./README.md#installation) and replace `pip install -v .` with `pip install -v -e .`.diff --git a/LICENSE b/LICENSEindex 7327c12..553df3c 100644--- a/LICENSE+++ b/LICENSE@@ -313,7 +313,7 @@such as asking that all changes be marked or described.Although not required by our licenses, you are encouraged torespect those requests where reasonable. More_considerations- for the public:+ for the public:wiki.creativecommons.org/Considerations_for_licensees=======================================================================@@ -677,5 +677,3 @@WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.--diff --git a/README.md b/README.mdindex 7aac775..8000852 100644--- a/README.md+++ b/README.md@@ -118,7 +118,7 @@ conda create -n opensora python=3.10conda activate opensora# install torch-# the command below is for CUDA 12.1, choose install commands from+# the command below is for CUDA 12.1, choose install commands from# https://pytorch.org/get-started/locally/ based on your own CUDA versionpip install torch torchvisiondiff --git a/configs/opensora-v1-1/train/Vx360p.py b/configs/opensora-v1-1/train/Vx360p.pyindex 81ff50a..72afafa 100644--- a/configs/opensora-v1-1/train/Vx360p.py+++ b/configs/opensora-v1-1/train/Vx360p.py@@ -15,7 +15,7 @@ bucket_config = {}# Define acceleration-num_workers = 0+num_workers = 4dtype = "bf16"grad_checkpoint = Trueplugin = "zero2"diff --git a/configs/opensora/inference-long/16x512x512-extend.py b/configs/opensora/inference-long/16x512x512-extend.pyindex cdd1cde..8feb1a4 100644--- a/configs/opensora/inference-long/16x512x512-extend.py+++ b/configs/opensora/inference-long/16x512x512-extend.py@@ -35,7 +35,7 @@ dtype = "fp16"prompt_path = Noneprompt = ["Drone view of waves crashing against the rugged cliffs along Big Sur’s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff’s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff's edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.",- "In an ornate, historical hall, a massive tidal wave peaks and begins to crash. Two surfers, seizing the moment, skillfully navigate the face of the wave."+ "In an ornate, historical hall, a massive tidal wave peaks and begins to crash. Two surfers, seizing the moment, skillfully navigate the face of the wave.",]loop = 10diff --git a/configs/opensora/inference/16x256x256.py b/configs/opensora/inference/16x256x256.pyindex 492543b..7508846 100644--- a/configs/opensora/inference/16x256x256.py+++ b/configs/opensora/inference/16x256x256.py@@ -25,13 +25,13 @@ scheduler = dict(type="iddpm",num_sampling_steps=100,cfg_scale=7.0,- cfg_channel=3, # or None+ cfg_channel=3, # or None)dtype = "fp16"# Conditionprompt_path = "./assets/texts/t2v_samples.txt"-prompt = None # prompt has higher priority than prompt_path+prompt = None # prompt has higher priority than prompt_path# Othersbatch_size = 1diff --git a/configs/opensora/inference/16x512x512.py b/configs/opensora/inference/16x512x512.pyindex 2064074..af67602 100644--- a/configs/opensora/inference/16x512x512.py+++ b/configs/opensora/inference/16x512x512.py@@ -9,7 +9,7 @@ model = dict(time_scale=1.0,enable_flashattn=True,enable_layernorm_kernel=True,- from_pretrained="PRETRAINED_MODEL"+ from_pretrained="PRETRAINED_MODEL",)vae = dict(type="VideoAutoencoderKL",diff --git a/docs/zh_CN/README.md b/docs/zh_CN/README.mdindex 8e52abf..265c3dc 100644--- a/docs/zh_CN/README.md+++ b/docs/zh_CN/README.md@@ -91,7 +91,7 @@conda create -n opensora python=3.10# install torch-# the command below is for CUDA 12.1, choose install commands from+# the command below is for CUDA 12.1, choose install commands from# https://pytorch.org/get-started/locally/ based on your own CUDA versionpip3 install torch torchvisiondiff --git a/opensora/datasets/aspect.py b/opensora/datasets/aspect.pyindex 9191ef2..5a8ac05 100644--- a/opensora/datasets/aspect.py+++ b/opensora/datasets/aspect.py@@ -1,6 +1,5 @@import math-# Oursdiff --git a/opensora/datasets/bucket.py b/opensora/datasets/bucket.pyindex e78baa9..574e5cb 100644--- a/opensora/datasets/bucket.py+++ b/opensora/datasets/bucket.py@@ -33,19 +33,11 @@ class Bucket:# wrap config with OrderedDictbucket_probs = OrderedDict()bucket_bs = OrderedDict()- bucket_names = sorted(- bucket_config.keys(), key=lambda x: ASPECT_RATIOS[x][0], reverse=True- )+ bucket_names = sorted(bucket_config.keys(), key=lambda x: ASPECT_RATIOS[x][0], reverse=True)for key in bucket_names:- bucket_time_names = sorted(- bucket_config[key].keys(), key=lambda x: x, reverse=True- )- bucket_probs[key] = OrderedDict(- {k: bucket_config[key][k][0] for k in bucket_time_names}- )- bucket_bs[key] = OrderedDict(- {k: bucket_config[key][k][1] for k in bucket_time_names}- )+ bucket_time_names = sorted(bucket_config[key].keys(), key=lambda x: x, reverse=True)+ bucket_probs[key] = OrderedDict({k: bucket_config[key][k][0] for k in bucket_time_names})+ bucket_bs[key] = OrderedDict({k: bucket_config[key][k][1] for k in bucket_time_names})# first level: HWnum_bucket = 0diff --git a/opensora/datasets/dataloader.py b/opensora/datasets/dataloader.pyindex ec65805..5077a07 100644--- a/opensora/datasets/dataloader.py+++ b/opensora/datasets/dataloader.py@@ -8,7 +8,6 @@ from torch.distributed.distributed_c10d import _get_default_groupfrom torch.utils.data import DataLoader, Datasetfrom torch.utils.data.distributed import DistributedSampler-from .bucket import Bucketfrom .sampler import DistributedVariableVideoSampler, VariableVideoBatchSampler@@ -98,38 +97,6 @@ def prepare_dataloader()-class _VariableVideoBatchSampler(torch.utils.data.BatchSampler):- def __init__(self, sampler, batch_size, drop_last, dataset, buckect_config):- self.sampler = sampler- self.dataset = dataset- self.batch_size = batch_size- self.drop_last = drop_last- self.bucket = Bucket(buckect_config)- self.frame_interval = self.dataset.frame_interval- self.bucket.info_bucket(self.dataset, self.frame_interval)-- def __iter__(self):- for idx in self.sampler:- T, H, W = self.dataset.get_data_info(idx)- bucket_id = self.bucket.get_bucket_id(T, H, W, self.frame_interval)- if bucket_id is None:- continue- rT, rH, rW = self.bucket.get_thw(bucket_id)- self.dataset.set_data_info(idx, rT, rH, rW)- buffer = self.bucket[bucket_id]- buffer.append(idx)- if len(buffer) >= self.bucket.get_batch_size(bucket_id):- yield buffer- self.bucket.set_empty(bucket_id)-- for k1, v1 in self.bucket.bucket.items():- for k2, v2 in v1.items():- for k3, buffer in v2.items():- if len(buffer) > 0 and not self.drop_last:- yield buffer- self.bucket.set_empty((k1, k2, k3))--def prepare_variable_dataloader(dataset,batch_size,diff --git a/opensora/datasets/datasets.py b/opensora/datasets/datasets.pyindex e0cae5f..f9cbf83 100644--- a/opensora/datasets/datasets.py+++ b/opensora/datasets/datasets.py@@ -39,6 +39,16 @@ class VideoTextDataset(torch.utils.data.Dataset):"video": get_transforms_video(transform_name, image_size),}+ def _print_data_number(self):+ num_videos = 0+ num_images = 0+ for path in self.data["path"]:+ if self.get_type(path) == "video":+ num_videos += 1+ else:+ num_images += 1+ print(f"Dataset contains {num_videos} videos and {num_images} images.")+def get_type(self, path):ext = os.path.splitext(path)[-1].lower()if ext.lower() in VID_EXTENSIONS:@@ -148,7 +158,6 @@ class VariableVideoTextDataset(VideoTextDataset):return {"video": video, "text": text, "num_frames": num_frames, "height": height, "width": width, "ar": ar}def __getitem__(self, index):- return self.getitem(index)for _ in range(10):try:return self.getitem(index)diff --git a/opensora/datasets/sampler.py b/opensora/datasets/sampler.pyindex 633bbc2..df7ccd7 100644--- a/opensora/datasets/sampler.py+++ b/opensora/datasets/sampler.py@@ -1,6 +1,7 @@import mathimport warnings-from collections import OrderedDict+from collections import OrderedDict, defaultdict+from pprint import pprintfrom typing import Iterator, List, Optional, Tupleimport torch@@ -43,9 +44,7 @@ class DistributedVariableVideoSampler(DistributedSampler):# group by bucketfor i in range(len(self.dataset)):t, h, w = self.dataset.get_data_info(i)- bucket_id = self.bucket.get_bucket_id(- t, h, w, self.dataset.frame_interval, g- )+ bucket_id = self.bucket.get_bucket_id(t, h, w, self.dataset.frame_interval, g)if bucket_id is None:continuereal_t, real_h, real_w = self.bucket.get_thw(bucket_id)@@ -56,12 +55,8 @@ class DistributedVariableVideoSampler(DistributedSampler):# shuffleif self.shuffle:# sort buckets- bucket_indices = torch.randperm(- len(bucket_sample_dict), generator=g- ).tolist()- bucket_order = {- k: bucket_indices[i] for i, k in enumerate(bucket_sample_dict)- }+ bucket_indices = torch.randperm(len(bucket_sample_dict), generator=g).tolist()+ bucket_order = {k: bucket_indices[i] for i, k in enumerate(bucket_sample_dict)}# sort samples in each bucketfor k, v in bucket_sample_dict.items():sample_indices = torch.randperm(len(v), generator=g).tolist()@@ -90,11 +85,7 @@ class DistributedVariableVideoSampler(DistributedSampler):if self.verbose:self._print_bucket_info(bucket_sample_dict)if self.shuffle:- bucket_sample_dict = OrderedDict(- sorted(- bucket_sample_dict.items(), key=lambda item: bucket_order[item[0]]- )- )+ bucket_sample_dict = OrderedDict(sorted(bucket_sample_dict.items(), key=lambda item: bucket_order[item[0]]))# iteratefound_last_bucket = self.last_bucket_id is Nonefor k, v in bucket_sample_dict.items():@@ -126,13 +117,21 @@ class DistributedVariableVideoSampler(DistributedSampler):def _print_bucket_info(self, bucket_sample_dict: dict) -> None:total_samples = 0num_dict = {}+ num_aspect_dict = defaultdict(int)+ num_hwt_dict = defaultdict(int)for k, v in bucket_sample_dict.items():size = len(v) * self.num_replicastotal_samples += sizenum_dict[k] = size- print(- f"Total training samples: {total_samples}, num buckets: {len(num_dict)}, bucket samples: {num_dict}"- )+ num_aspect_dict[k[-1]] += size+ num_hwt_dict[k[:-1]] += size+ print(f"Total training samples: {total_samples}, num buckets: {len(num_dict)}")+ print("Bucket samples:")+ pprint(num_dict)+ print("Bucket samples by HxWxT:")+ pprint(num_hwt_dict)+ print("Bucket samples by aspect ratio:")+ pprint(num_aspect_dict)def state_dict(self) -> dict:# users must ensure bucket config is the same@@ -175,9 +174,7 @@ class VariableVideoBatchSampler(Sampler[List[int]]):cur_sample_indices = [sample_idx]else:cur_sample_indices.append(sample_idx)- if len(cur_sample_indices) > 0 and (- not self.drop_last or len(cur_sample_indices) == cur_batch_size- ):+ if len(cur_sample_indices) > 0 and (not self.drop_last or len(cur_sample_indices) == cur_batch_size):yield cur_sample_indicesdef state_dict(self) -> dict:diff --git a/opensora/datasets/utils.py b/opensora/datasets/utils.pyindex f5c71c9..85d1d18 100644--- a/opensora/datasets/utils.py+++ b/opensora/datasets/utils.py@@ -1,5 +1,3 @@-import numbers-import numpy as npimport torchimport torchvision@@ -146,8 +144,8 @@ def center_crop_arr(pil_image, image_size):def resize_crop_to_fill(pil_image, image_size):- w, h = pil_image.size # PIL is (W, H)- th, tw = image_size+ w, h = pil_image.size # PIL is (W, H)+ th, tw = image_sizerh, rw = th / h, tw / wif rh > rw:sh, sw = th, int(w * rh)diff --git a/opensora/models/layers/blocks.py b/opensora/models/layers/blocks.pyindex 1f44f1f..f7c67b1 100644--- a/opensora/models/layers/blocks.py+++ b/opensora/models/layers/blocks.py@@ -23,10 +23,7 @@ import xformers.opsfrom einops import rearrangefrom timm.models.vision_transformer import Mlp-from opensora.acceleration.communications import (- all_to_all,- split_forward_gather_backward,-)+from opensora.acceleration.communications import all_to_all, split_forward_gather_backwardfrom opensora.acceleration.parallel_states import get_sequence_parallel_groupapprox_gelu = lambda: nn.GELU(approximate="tanh")@@ -568,7 +565,6 @@ class CaptionEmbedder(nn.Module):self.register_buffer("y_embedding",torch.randn(token_num, in_channels) / in_channels**0.5,- persistent=False,)self.uncond_prob = uncond_probdiff --git a/opensora/models/text_encoder/t5.py b/opensora/models/text_encoder/t5.pyindex 7f55ef9..1bd535d 100644--- a/opensora/models/text_encoder/t5.py+++ b/opensora/models/text_encoder/t5.py@@ -23,14 +23,12 @@import html-import osimport reimport urllib.parse as ulimport ftfyimport torchfrom bs4 import BeautifulSoup-from huggingface_hub import hf_hub_downloadfrom transformers import AutoTokenizer, T5EncoderModelfrom opensora.registry import MODELSdiff --git a/opensora/models/vae/vae.py b/opensora/models/vae/vae.pyindex 6f96dff..e6e1441 100644--- a/opensora/models/vae/vae.py+++ b/opensora/models/vae/vae.py@@ -53,7 +53,9 @@ class VideoAutoencoderKL(nn.Module):def get_latent_size(self, input_size):latent_size = []for i in range(3):- assert input_size[i] is None or input_size[i] % self.patch_size[i] == 0, "Input size must be divisible by patch size"+ assert (+ input_size[i] is None or input_size[i] % self.patch_size[i] == 0+ ), "Input size must be divisible by patch size"latent_size.append(input_size[i] // self.patch_size[i] if input_size[i] is not None else None)return latent_size@@ -87,7 +89,9 @@ class VideoAutoencoderKLTemporalDecoder(nn.Module):def get_latent_size(self, input_size):latent_size = []for i in range(3):- assert input_size[i] is None or input_size[i] % self.patch_size[i] == 0, "Input size must be divisible by patch size"+ assert (+ input_size[i] is None or input_size[i] % self.patch_size[i] == 0+ ), "Input size must be divisible by patch size"latent_size.append(input_size[i] // self.patch_size[i] if input_size[i] is not None else None)return latent_sizediff --git a/opensora/schedulers/iddpm/speed.py b/opensora/schedulers/iddpm/speed.pyindex 3231d02..e70063a 100644--- a/opensora/schedulers/iddpm/speed.py+++ b/opensora/schedulers/iddpm/speed.py@@ -1,75 +1,75 @@-import numpy as np-import torch-import torch.nn.functional as F--from opensora.registry import SCHEDULERS--from . import gaussian_diffusion as gd-from .respace import SpacedDiffusion, space_timesteps--[email protected]_module("iddpm-speed")-class SpeeDiffusion(SpacedDiffusion):- def __init__(- self,- num_sampling_steps=None,- timestep_respacing=None,- noise_schedule="linear",- use_kl=False,- sigma_small=False,- predict_xstart=False,- learn_sigma=True,- rescale_learned_sigmas=False,- diffusion_steps=1000,- cfg_scale=4.0,- ):- betas = gd.get_named_beta_schedule(noise_schedule, diffusion_steps)- if use_kl:- loss_type = gd.LossType.RESCALED_KL- elif rescale_learned_sigmas:- loss_type = gd.LossType.RESCALED_MSE- else:- loss_type = gd.LossType.MSE- if num_sampling_steps is not None:- assert timestep_respacing is None- timestep_respacing = str(num_sampling_steps)- if timestep_respacing is None or timestep_respacing == "":- timestep_respacing = [diffusion_steps]- super().__init__(- use_timesteps=space_timesteps(diffusion_steps, timestep_respacing),- betas=betas,- model_mean_type=(gd.ModelMeanType.EPSILON if not predict_xstart else gd.ModelMeanType.START_X),- model_var_type=(- (gd.ModelVarType.FIXED_LARGE if not sigma_small else gd.ModelVarType.FIXED_SMALL)- if not learn_sigma- else gd.ModelVarType.LEARNED_RANGE- ),- loss_type=loss_type,- )-- self.cfg_scale = cfg_scale-- grad = np.gradient(self.sqrt_one_minus_alphas_cumprod)- self.meaningful_steps = np.argmax(grad < 5e-5) + 1-- # p2 weighting from: Perception Prioritized Training of Diffusion Models- self.p2_gamma = 1- self.p2_k = 1- self.snr = 1.0 / (1 - self.alphas_cumprod) - 1- sqrt_one_minus_alphas_bar = torch.from_numpy(self.sqrt_one_minus_alphas_cumprod)- p = torch.tanh(1e6 * (torch.gradient(sqrt_one_minus_alphas_bar)[0] - 1e-4)) + 1.5- self.p = F.normalize(p, p=1, dim=0)- self.weights = 1 / (self.p2_k + self.snr) ** self.p2_gamma-- def t_sample(self, n, device):- t = torch.multinomial(self.p, n // 2 + 1, replacement=True).to(device)- dual_t = torch.where(t < self.meaningful_steps, self.meaningful_steps - t, t - self.meaningful_steps)- t = torch.cat([t, dual_t], dim=0)[:n]- return t-- def training_losses(self, model, x, t, *args, **kwargs): # pylint: disable=signature-differs- t = self.t_sample(x.shape[0], x.device)- return super().training_losses(model, x, t, weights=self.weights, *args, **kwargs)-- def sample(self, *args, **kwargs):- raise NotImplementedError("SpeeDiffusion is only for training")+import numpy as np+import torch+import torch.nn.functional as F++from opensora.registry import SCHEDULERS++from . import gaussian_diffusion as gd+from .respace import SpacedDiffusion, space_timesteps++[email protected]_module("iddpm-speed")+class SpeeDiffusion(SpacedDiffusion):+ def __init__(+ self,+ num_sampling_steps=None,+ timestep_respacing=None,+ noise_schedule="linear",+ use_kl=False,+ sigma_small=False,+ predict_xstart=False,+ learn_sigma=True,+ rescale_learned_sigmas=False,+ diffusion_steps=1000,+ cfg_scale=4.0,+ ):+ betas = gd.get_named_beta_schedule(noise_schedule, diffusion_steps)+ if use_kl:+ loss_type = gd.LossType.RESCALED_KL+ elif rescale_learned_sigmas:+ loss_type = gd.LossType.RESCALED_MSE+ else:+ loss_type = gd.LossType.MSE+ if num_sampling_steps is not None:+ assert timestep_respacing is None+ timestep_respacing = str(num_sampling_steps)+ if timestep_respacing is None or timestep_respacing == "":+ timestep_respacing = [diffusion_steps]+ super().__init__(+ use_timesteps=space_timesteps(diffusion_steps, timestep_respacing),+ betas=betas,+ model_mean_type=(gd.ModelMeanType.EPSILON if not predict_xstart else gd.ModelMeanType.START_X),+ model_var_type=(+ (gd.ModelVarType.FIXED_LARGE if not sigma_small else gd.ModelVarType.FIXED_SMALL)+ if not learn_sigma+ else gd.ModelVarType.LEARNED_RANGE+ ),+ loss_type=loss_type,+ )++ self.cfg_scale = cfg_scale++ grad = np.gradient(self.sqrt_one_minus_alphas_cumprod)+ self.meaningful_steps = np.argmax(grad < 5e-5) + 1++ # p2 weighting from: Perception Prioritized Training of Diffusion Models+ self.p2_gamma = 1+ self.p2_k = 1+ self.snr = 1.0 / (1 - self.alphas_cumprod) - 1+ sqrt_one_minus_alphas_bar = torch.from_numpy(self.sqrt_one_minus_alphas_cumprod)+ p = torch.tanh(1e6 * (torch.gradient(sqrt_one_minus_alphas_bar)[0] - 1e-4)) + 1.5+ self.p = F.normalize(p, p=1, dim=0)+ self.weights = 1 / (self.p2_k + self.snr) ** self.p2_gamma++ def t_sample(self, n, device):+ t = torch.multinomial(self.p, n // 2 + 1, replacement=True).to(device)+ dual_t = torch.where(t < self.meaningful_steps, self.meaningful_steps - t, t - self.meaningful_steps)+ t = torch.cat([t, dual_t], dim=0)[:n]+ return t++ def training_losses(self, model, x, t, *args, **kwargs): # pylint: disable=signature-differs+ t = self.t_sample(x.shape[0], x.device)+ return super().training_losses(model, x, t, weights=self.weights, *args, **kwargs)++ def sample(self, *args, **kwargs):+ raise NotImplementedError("SpeeDiffusion is only for training")diff --git a/opensora/utils/ckpt_utils.py b/opensora/utils/ckpt_utils.pyindex 871fc59..87c5ac1 100644--- a/opensora/utils/ckpt_utils.py+++ b/opensora/utils/ckpt_utils.py@@ -5,7 +5,6 @@ import operatorimport osfrom typing import Tuple-import colossalaiimport torchimport torch.distributed as distimport torch.nn as nn@@ -55,9 +54,7 @@ def find_model(model_name):model = reparameter(model, model_name)return modelelse: # Load a custom DiT checkpoint:- assert os.path.isfile(- model_name- ), f"Could not find DiT checkpoint at {model_name}"+ assert os.path.isfile(model_name), f"Could not find DiT checkpoint at {model_name}"checkpoint = torch.load(model_name, map_location=lambda storage, loc: storage)if "pos_embed_temporal" in checkpoint:del checkpoint["pos_embed_temporal"]@@ -93,9 +90,7 @@ def model_sharding(model: torch.nn.Module):for _, param in model.named_parameters():padding_size = (world_size - param.numel() % world_size) % world_sizeif padding_size > 0:- padding_param = torch.nn.functional.pad(- param.data.view(-1), [0, padding_size]- )+ padding_param = torch.nn.functional.pad(param.data.view(-1), [0, padding_size])else:padding_param = param.data.view(-1)splited_params = padding_param.split(padding_param.numel() // world_size)@@ -125,9 +120,7 @@ def model_gathering(model: torch.nn.Module, model_shape_dict: dict):dist.all_gather(all_params, param.data, group=dist.group.WORLD)if int(global_rank) == 0:all_params = torch.cat(all_params)- param.data = remove_padding(all_params, model_shape_dict[name]).view(- model_shape_dict[name]- )+ param.data = remove_padding(all_params, model_shape_dict[name]).view(model_shape_dict[name])dist.barrier()@@ -164,9 +157,7 @@ def save(torch.save(ema.state_dict(), os.path.join(save_dir, "ema.pt"))model_sharding(ema)- booster.save_optimizer(- optimizer, os.path.join(save_dir, "optimizer"), shard=True, size_per_shard=4096- )+ booster.save_optimizer(optimizer, os.path.join(save_dir, "optimizer"), shard=True, size_per_shard=4096)if lr_scheduler is not None:booster.save_lr_scheduler(lr_scheduler, os.path.join(save_dir, "lr_scheduler"))running_states = {@@ -194,9 +185,7 @@ def load(booster.load_model(model, os.path.join(load_dir, "model"))# ema is not boosted, so we don't use booster.load_model# ema.load_state_dict(torch.load(os.path.join(load_dir, "ema.pt")))- ema.load_state_dict(- torch.load(os.path.join(load_dir, "ema.pt"), map_location=torch.device("cpu"))- )+ ema.load_state_dict(torch.load(os.path.join(load_dir, "ema.pt"), map_location=torch.device("cpu")))booster.load_optimizer(optimizer, os.path.join(load_dir, "optimizer"))if lr_scheduler is not None:booster.load_lr_scheduler(lr_scheduler, os.path.join(load_dir, "lr_scheduler"))diff --git a/opensora/utils/config_utils.py b/opensora/utils/config_utils.pyindex 2840605..d47c0a7 100644--- a/opensora/utils/config_utils.py+++ b/opensora/utils/config_utils.py@@ -47,14 +47,13 @@ def merge_args(cfg, args, training=False):if args.ckpt_path is not None:cfg.model["from_pretrained"] = args.ckpt_pathargs.ckpt_path = None-for k, v in vars(args).items():if k in cfg and v is not None:cfg[k] = vif not training:- # Inference only+ # Inference onlyif "reference_path" not in cfg:cfg["reference_path"] = Noneif "loop" not in cfg:@@ -63,7 +62,7 @@ def merge_args(cfg, args, training=False):assert cfg["prompt_path"] is not None, "prompt or prompt_path must be provided"cfg["prompt"] = load_prompts(cfg["prompt_path"])else:- # Training only+ # Training onlyif args.data_path is not None:cfg.dataset["data_path"] = args.data_pathargs.data_path = Nonediff --git a/scripts/train.py b/scripts/train.pyindex a8cac61..439b461 100644--- a/scripts/train.py+++ b/scripts/train.py@@ -1,4 +1,5 @@from copy import deepcopy+from pprint import pprintimport colossalaiimport torch@@ -20,26 +21,14 @@ from opensora.acceleration.parallel_states import (from opensora.acceleration.plugin import ZeroSeqParallelPluginfrom opensora.datasets import prepare_dataloader, prepare_variable_dataloaderfrom opensora.registry import DATASETS, MODELS, SCHEDULERS, build_module-from opensora.utils.ckpt_utils import (- create_logger,- load,- model_sharding,- record_model_param_shape,- save,-)+from opensora.utils.ckpt_utils import create_logger, load, model_sharding, record_model_param_shape, savefrom opensora.utils.config_utils import (create_experiment_workspace,create_tensorboard_writer,parse_configs,save_training_config,)-from opensora.utils.misc import (- all_reduce_mean,- format_numel_str,- get_model_numel,- requires_grad,- to_torch_dtype,-)+from opensora.utils.misc import all_reduce_mean, format_numel_str, get_model_numel, requires_grad, to_torch_dtypefrom opensora.utils.train_utils import MaskGenerator, update_ema@@ -48,7 +37,8 @@ def main():# 1. args & cfg# ======================================================cfg = parse_configs(training=True)- print(cfg)+ print("Training configuration:")+ pprint(cfg._cfg_dict)exp_name, exp_dir = create_experiment_workspace(cfg)save_training_config(cfg._cfg_dict, exp_dir)@@ -115,12 +105,10 @@ def main():if cfg.bucket_config is None:dataloader = prepare_dataloader(**dataloader_args)else:- dataloader = prepare_variable_dataloader(- bucket_config=cfg.bucket_config, **dataloader_args- )- total_batch_size = cfg.batch_size * dist.get_world_size() // cfg.sp_size- logger.info(f"Dataset contains {len(dataset):,} videos ({dataset.data_path})")- logger.info(f"Total batch size: {total_batch_size}")+ dataloader = prepare_variable_dataloader(bucket_config=cfg.bucket_config, **dataloader_args)+ if cfg.dataset.type == "VideoTextDataset":+ total_batch_size = cfg.batch_size * dist.get_world_size() // cfg.sp_size+ logger.info(f"Total batch size: {total_batch_size}")# ======================================================# 4. build model@@ -193,11 +181,7 @@ def main():# =======================================================start_epoch = start_step = log_step = sampler_start_idx = 0running_loss = 0.0- sampler_to_io = (- dataloader.batch_sampler- if cfg.dataset.type == "VariableVideoTextDataset"- else None- )+ sampler_to_io = dataloader.batch_sampler if cfg.dataset.type == "VariableVideoTextDataset" else None# 6.1. resume trainingif cfg.load is not None:logger.info("Loading checkpoint")@@ -210,12 +194,8 @@ def main():cfg.load,sampler=sampler_to_io,)- logger.info(- f"Loaded checkpoint {cfg.load} at epoch {start_epoch} step {start_step}"- )- logger.info(- f"Training for {cfg.epochs} epochs with {num_steps_per_epoch} steps per epoch"- )+ logger.info(f"Loaded checkpoint {cfg.load} at epoch {start_epoch} step {start_step}")+ logger.info(f"Training for {cfg.epochs} epochs with {num_steps_per_epoch} steps per epoch")if cfg.dataset.type == "VideoTextDataset":dataloader.sampler.set_start_index(sampler_start_idx)@@ -257,12 +237,8 @@ def main():model_args[k] = v.to(device, dtype)# Diffusion- t = torch.randint(- 0, scheduler.num_timesteps, (x.shape[0],), device=device- )- loss_dict = scheduler.training_losses(- model, x, t, model_args, mask=mask- )+ t = torch.randint(0, scheduler.num_timesteps, (x.shape[0],), device=device)+ loss_dict = scheduler.training_losses(model, x, t, model_args, mask=mask)# Backward & updateloss = loss_dict["loss"].mean()@@ -282,9 +258,7 @@ def main():# Log to tensorboardif coordinator.is_master() and (global_step + 1) % cfg.log_every == 0:avg_loss = running_loss / log_step- pbar.set_postfix(- {"loss": avg_loss, "step": step, "global_step": global_step}- )+ pbar.set_postfix({"loss": avg_loss, "step": step, "global_step": global_step})running_loss = 0log_step = 0writer.add_scalar("loss", loss.item(), global_step)@@ -292,7 +266,6 @@ def main():wandb.log({"iter": global_step,- "num_samples": global_step * total_batch_size,"epoch": epoch,"loss": loss.item(),"avg_loss": avg_loss,diff --git a/tests/test_seq_parallel_attention.py b/tests/test_seq_parallel_attention.pyindex 00966ad..9cce2b2 100644--- a/tests/test_seq_parallel_attention.py+++ b/tests/test_seq_parallel_attention.py@@ -69,16 +69,24 @@ def run_cross_attention(rank, world_size):# create modeltorch.manual_seed(1024)set_sequence_parallel_group(dist.group.WORLD)- seq_parallel_attention = SeqParallelMultiHeadCrossAttention(- d_model=256,- num_heads=4,- ).cuda().to(torch.bfloat16)+ seq_parallel_attention = (+ SeqParallelMultiHeadCrossAttention(+ d_model=256,+ num_heads=4,+ )+ .cuda()+ .to(torch.bfloat16)+ )torch.manual_seed(1024)- attention = MultiHeadCrossAttention(- d_model=256,- num_heads=4,- ).cuda().to(torch.bfloat16)+ attention = (+ MultiHeadCrossAttention(+ d_model=256,+ num_heads=4,+ )+ .cuda()+ .to(torch.bfloat16)+ )# make sure the weights are the samefor p1, p2 in zip(seq_parallel_attention.parameters(), attention.parameters()):@@ -128,7 +136,9 @@ def run_cross_attention(rank, world_size):# # check gradfor p1, p2 in zip(seq_parallel_attention.named_parameters(), attention.named_parameters()):- assert torch.allclose(p1[1].grad, p2[1].grad, rtol=1e-3, atol=1e-4), f"\n{p1[0]}\nvs\n{p2[0]}:\n{p1[1].grad}\nvs\n{p2[1].grad}"+ assert torch.allclose(+ p1[1].grad, p2[1].grad, rtol=1e-3, atol=1e-4+ ), f"\n{p1[0]}\nvs\n{p2[0]}:\n{p1[1].grad}\nvs\n{p2[1].grad}"# # check input gradassert torch.allclose(x.grad, seq_x.grad, atol=1e-7), f"{x.grad}\nvs\n{seq_x.grad}"diff --git a/tools/caption/README.md b/tools/caption/README.mdindex fc1ae31..3c4c056 100644--- a/tools/caption/README.md+++ b/tools/caption/README.md@@ -39,7 +39,7 @@ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvipip install flash-attn --no-build-isolation```-First, install LLaVA according to their [official instructions](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#install). We use the `liuhaotian/llava-v1.6-34b` model for captioning, which can be download [here](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b).+First, install LLaVA according to their [official instructions](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#install). We use the `liuhaotian/llava-v1.6-34b` model for captioning, which can be download [here](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b).### Usagediff --git a/tools/datasets/README.md b/tools/datasets/README.mdindex 354a858..2ea5cea 100644--- a/tools/datasets/README.md+++ b/tools/datasets/README.md@@ -60,7 +60,7 @@ head -n 10 DATA1.csvwc -l DATA1.csv```-Additionally, Ww provide `csvutils.py` to manage the CSV files.+Additionally, Ww provide `csvutils.py` to manage the CSV files.### Requirementdiff --git a/tools/datasets/convert_dataset.py b/tools/datasets/convert_dataset.pyindex aefc661..f832eb7 100644--- a/tools/datasets/convert_dataset.py+++ b/tools/datasets/convert_dataset.py@@ -4,7 +4,6 @@ import osimport pandas as pdfrom torchvision.datasets import ImageNet-IMG_EXTENSIONS = (".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp")VID_EXTENSIONS = (".mp4", ".avi", ".mov", ".mkv")diff --git a/tools/scenedetect/README.md b/tools/scenedetect/README.mdindex 8052733..5e7ac29 100644--- a/tools/scenedetect/README.md+++ b/tools/scenedetect/README.md@@ -1,6 +1,6 @@# Scene Detection and Video Split-Raw videos from the Internet may be too long for training.+Raw videos from the Internet may be too long for training.Thus, we detect scenes in raw videos and split them into short clips based on the scenes.First prepare the video processing packages.```bash