Open-Sora: Democratizing Efficient Video Production for All
Python
removed old files (#63)
Frank Lee2f87a9af75c2
diff --git a/README.md b/README.mddeleted file mode 100644index 46d75c2..0000000--- a/README.md+++ /dev/null@@ -1,165 +0,0 @@-# 🎥 Open-Sora-<div id="top" align="center">-- [](https://github.com/hpcaitech/Open-Sora/stargazers)- [](https://github.com/hpcaitech/public_assets/tree/main/colossalai/contact/slack)- [](https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/WeChat.png)--</div>--## 📎 Table of Contents--- [🎥 Open-Sora](#-open-sora)- - [📎 Table of Contents](#-table-of-contents)- - [📍 Overview](#-overview)- - [📂 Dataset Preparation](#-dataset-preparation)- - [Use MSR-VTT](#use-msr-vtt)- - [Use Customized Datasets](#use-customized-datasets)- - [🚀 Get Started](#-get-started)- - [Training](#training)- - [Inference](#inference)- - [🪄 Acknowledgement](#-acknowledgement)--## Latest News-* [2024/03] [Open-Sora:Sora Replication Solution with 46% Cost Reduction, Sequence Expansion to Nearly a Million](https://hpc-ai.com/blog/open-sora)--## 📍 Overview-Open-Sora is an open-source project that provides a high-performance implementation of the development pipeline that Sora might use powered by [Colossal-AI](https://github.com/hpcaitech/ColossalAI), including:--- Provides **a complete Sora reproduction architecture solution**, including the whole process from data processing to training and deployment.-- Supports **dynamic resolution**, training can directly train any resolution of the video, without scaling.-- Supports **multiple model structures**. Since the actual model structure of Sora is unknown, we implement three common multimodal model structures such as adaLN-zero, cross attention, and in-context conditioning (token concat).-- Supports **multiple video compression methods**. Users can choose to use original video, VQVAE (video native model), or SD-VAE (image native model) for training.-- Supports **multiple parallel training optimizations**. Including the AI large model system optimization capability compatible with Colossal-AI, and hybrid sequence parallelism with Ulysses and FastSeq.--<p id="diffusion_demo" align="center">-<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/open-sora-1.png" width=800/>-</p>--<p id="diffusion_demo" align="center">-<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/open-sora-2.png" width=800/>-</p>--<p id="diffusion_demo" align="center">-<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/dit-model-arch.png" width=800/>-</p>--## 📂 Dataset Preparation--### Use MSR-VTT--We use [MSR-VTT](https://cove.thecvf.com/datasets/839) dataset, which is a large-scale video description dataset. Users should preprocess the raw videos before training the model. You can use the following scripts to perform data processing.---```bash-# Step 1: download the dataset to ./dataset/MSRVTT-bash scripts/data/download_msr_vtt_dataset.sh--# Step 2: collate the video and annotations-python scripts/data/collate_msr_vtt_dataset.py -d ./dataset/MSRVTT/ -o ./dataset/MSRVTT-collated--# Step 3: perform data processing-# NOTE: each script could several minutes so we apply the script to the dataset split individually-python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/train/annotations.json -v ./dataset/MSRVTT-collated/train/videos -o ./dataset/MSRVTT-processed/train-python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/val/annotations.json -v ./dataset/MSRVTT-collated/val/videos -o ./dataset/MSRVTT-processed/val-python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/test/annotations.json -v ./dataset/MSRVTT-collated/test/videos -o ./dataset/MSRVTT-processed/test-```--**If you want to use adaLN-zero, you should use `--use_pooled_text` when running `preprocess_data.py`**--```bash--After completing these steps, you should have a processed MSR-VTT dataset in `./dataset/MSRVTT-processed`.---### Use Customized Datasets--You can also use other datasets and transform the dataset to the required format. You should prepare a captions file and a video directory. The captions file should be a JSON file or a JSONL file. The video directory should contain all the videos.--Here is an example of the captions file:--```json-[- {- "file": "video0.mp4",- "captions": ["a girl is throwing away folded clothes", "a girl throwing cloths around"]- },- {- "file": "video1.mp4",- "captions": ["a comparison of two opposing team football athletes"]- }-]-```--Here is an example of the video directory:--```-.-├── video0.mp4-├── video1.mp4-└── ...-```--Each video may have multiple captions. So the outputs are video-caption pairs. E.g., If the first video has two captions, then the output will be two video-caption pairs.--We use [VQ-VAE](https://github.com/wilson1yan/VideoGPT/) to quantize the video frames. And we use [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#clip) to extract the text features.--The output is an arrow dataset, which contains the following columns: "video_file", "video_latent_states", "text_latent_states". The dimension of "video_latent_states" is (T, H, W), and the dimension of "text_latent_states" is (S, D).--Then you can run the data processing script with the command below:--```bash-python preprocess_data.py -c /path/to/captions.json -v /path/to/video_dir -o /path/to/output_dir-```--**If you want to use adaLN-zero, you should use `--use_pooled_text` when running `preprocess_data.py`**--Note that this script needs to be run on a machine with a GPU. To avoid CUDA OOM, we filter out the videos that are too long.---## 🚀 Get Started--In this section, we will guide how to run training and inference. Before that, make sure you installed the dependencies with the command below.--```bash-pip install -r requirements.txt-```--### Training--You can invoke the training via the command below.--```bash-bash ./scripts/train.sh-```--You can also modify the arguments in `train.sh` for your own need.--We provide three model architectures: "adaln", "cross-attn" and "token-concat". You can specify the model architecture with the `-x` option.--### Inference--We've provided a script to perform inference, allowing you to generate videos from the trained model. You can invoke the inference via the command below.--```bash-python sample.py -m "DiT/XL-2" --text "a person is walking on the street" --ckpt /path/to/checkpoint --height 256 --width 256 --fps 10 --sec 5 --disable-cfg-```-This will generate a "sample.mp4" file in the current directory.--For more command line options, you can use the following command to check the help message.--```bash-python sample.py -h-```--Please make sure the video compressor and model architecture are consistent with the training settings.--## 🪄 Acknowledgement--During the development of the project, we learned a lot from the following information:--- [OpenAI Sora Technical Report](https://openai.com/research/video-generation-models-as-world-simulators)-- [VideoGPT Project](https://github.com/wilson1yan/VideoGPT)-- [Diffusion Transformers](https://github.com/facebookresearch/DiT)-- [Deepspeed Ulysses](https://arxiv.org/abs/2309.14509)-- [OpenDiT](https://github.com/NUS-HPC-AI-Lab/OpenDiT)diff --git a/benchmark.py b/benchmark.pydeleted file mode 100644index 69eef8d..0000000--- a/benchmark.py+++ /dev/null@@ -1,180 +0,0 @@-# Copyright (c) Meta Platforms, Inc. and affiliates.-# All rights reserved.--# This source code is licensed under the license found in the-# LICENSE file in the root directory of this source tree.--"""-A minimal training script for DiT using PyTorch DDP.-"""-import argparse-import time--import torch-import torch.distributed as dist-from colossalai import launch_from_torch-from colossalai.accelerator import get_accelerator-from colossalai.booster import Booster-from colossalai.booster.plugin import TorchDDPPlugin-from colossalai.cluster import DistCoordinator-from colossalai.logging import get_dist_logger-from colossalai.nn.optimizer import HybridAdam-from colossalai.utils import get_current_device-from tqdm import tqdm--from open_sora.diffusion import create_diffusion-from open_sora.modeling import DiT_models-from open_sora.modeling.dit import SUPPORTED_SEQ_PARALLEL_MODES-from open_sora.utils.data import create_video_compressor, preprocess_batch-from open_sora.utils.plugin import ZeroSeqParallelPlugin--#################################################################################-# Training Loop #-#################################################################################---def main(args):- """- Trains a new DiT model.- """- # init distributed environment- launch_from_torch({})- coordinator = DistCoordinator()- logger = get_dist_logger()-- # set up acceleration plugins- if args.plugin == "ddp":- plugin = TorchDDPPlugin()- elif args.plugin == "zero2":- # use bf16 to avoid skipping the first few iterations due to NaNs- plugin = ZeroSeqParallelPlugin(sp_size=args.sp_size, stage=2, precision="bf16")- else:- raise ValueError(f"Unknown plugin {args.plugin}")- booster = Booster(plugin=plugin)-- # Create video compressor- video_compressor = create_video_compressor(args.compressor)- model_kwargs = {- "in_channels": video_compressor.out_channels,- "seq_parallel_group": getattr(plugin, "sp_group", None),- "seq_parallel_mode": args.sp_mode,- "seq_parallel_overlap": args.sp_overlap,- }-- # Create DiT and EMA- model = DiT_models[args.model](**model_kwargs).to(get_current_device())- patch_size = model.patch_size- model.train() # important! This enables embedding dropout for classifier-free guidance-- # configure gradient checkpointing- if args.grad_checkpoint:- model.enable_gradient_checkpointing()-- # create diffusion pipeline- diffusion = create_diffusion(- timestep_respacing=""- ) # default: 1000 steps, linear noise schedule-- # setup optimizer (we used default Adam betas=(0.9, 0.999) and a constant learning rate of 1e-4 in our paper):- opt = HybridAdam(model.parameters(), lr=1e-4, weight_decay=0)-- # Setup dataloader- videos = [- torch.randn(args.num_frames, args.height, args.width, 3)- for _ in range(args.batch_size)- ]- assert args.num_tokens % args.sp_size == 0- input_ids = torch.randn(args.batch_size, args.num_tokens, args.text_embed_dim)- text_mask = torch.ones(input_ids.shape[:2], dtype=torch.int)- batch = {- "videos": videos,- "text_latent_states": input_ids,- "text_padding_mask": text_mask,- }- batch = preprocess_batch(- batch, patch_size, video_compressor, pad_to_multiple=args.sp_size- )- video_inputs = batch.pop("video_latent_states")- mask = batch.pop("video_padding_mask")- logger.info(- f"Num patches: {video_inputs.shape[1]}, num tokens: {batch['text_latent_states'].shape[1]}",- ranks=[0],- )-- # setup booster- model, opt, *_ = booster.boost(model, opt)- logger.info(- f"Booster init max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",- ranks=[0],- )-- # Train- total_samples = 0- total_duration = 0.0- for i in tqdm(- range(args.warmup_steps + args.steps),- desc="Steps",- disable=not coordinator.is_master(),- ):- start = time.time()- t = torch.randint(- 0,- diffusion.num_timesteps,- (video_inputs.shape[0],),- device=video_inputs.device,- )- loss_dict = diffusion.training_losses(model, video_inputs, t, batch, mask=mask)- loss = loss_dict["loss"].mean()- booster.backward(loss, opt)- opt.step()- opt.zero_grad()- get_accelerator().empty_cache()- time_per_iter = time.time() - start- if i >= args.warmup_steps:- total_samples += args.batch_size * coordinator.world_size- total_duration += time_per_iter- total_duration = torch.tensor([total_duration], device=get_current_device())- dist.all_reduce(total_duration)- total_duration = total_duration / coordinator.world_size- total_duration = total_duration.item()- total_samples *= coordinator.world_size // args.sp_size-- throughput = total_samples / total_duration- logger.info(- f"Training complete, max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",- ranks=[0],- )- logger.info(- f"Throughput: {throughput:.2f} samples/s",- ranks=[0],- )---if __name__ == "__main__":- # Default args here will train DiT-XL/2 with the hyperparameters we used in our paper (except training iters).- parser = argparse.ArgumentParser()- parser.add_argument(- "-m", "--model", type=str, choices=list(DiT_models.keys()), default="DiT-S/8"- )- parser.add_argument(- "-p", "--plugin", type=str, default="zero2", choices=["ddp", "zero2"]- )- parser.add_argument("--sp_size", type=int, default=1)- parser.add_argument(- "--sp_mode", type=str, default="ulysses", choices=SUPPORTED_SEQ_PARALLEL_MODES- )- parser.add_argument("--sp_overlap", action="store_true", default=False)- parser.add_argument("-w", "--warmup_steps", type=int, default=2)- parser.add_argument("-s", "--steps", type=int, default=3)- parser.add_argument("-b", "--batch_size", type=int, default=4)- parser.add_argument("-f", "--num_frames", type=int, default=300)- parser.add_argument("--height", type=int, default=256)- parser.add_argument("--width", type=int, default=256)- parser.add_argument("--num_tokens", type=int, default=20)- parser.add_argument("--text_embed_dim", type=int, default=512)- parser.add_argument("-g", "--grad_checkpoint", action="store_true", default=False)- parser.add_argument(- "-c", "--compressor", choices=["raw", "vqvae", "vae"], default="raw"- )- args = parser.parse_args()- main(args)diff --git a/download.py b/download.pydeleted file mode 100644index f4c3700..0000000--- a/download.py+++ /dev/null@@ -1,50 +0,0 @@-# Copyright (c) Meta Platforms, Inc. and affiliates.-# All rights reserved.--# This source code is licensed under the license found in the-# LICENSE file in the root directory of this source tree.--"""-Functions for downloading pre-trained DiT models-"""-import os--import torch-from torchvision.datasets.utils import download_url--pretrained_models = {"DiT-XL-2-512x512.pt", "DiT-XL-2-256x256.pt"}---def find_model(model_name):- """- Finds a pre-trained DiT model, downloading it if necessary. Alternatively, loads a model from a local path.- """- if model_name in pretrained_models: # Find/download our pre-trained DiT checkpoints- return download_model(model_name)- else: # Load a custom DiT checkpoint:- assert os.path.isfile(model_name), f"Could not find DiT checkpoint at {model_name}"- checkpoint = torch.load(model_name, map_location=lambda storage, loc: storage)- if "ema" in checkpoint: # supports checkpoints from train.py- checkpoint = checkpoint["ema"]- return checkpoint---def download_model(model_name):- """- Downloads a pre-trained DiT model from the web.- """- assert model_name in pretrained_models- local_path = f"pretrained_models/{model_name}"- if not os.path.isfile(local_path):- os.makedirs("pretrained_models", exist_ok=True)- web_path = f"https://dl.fbaipublicfiles.com/DiT/models/{model_name}"- download_url(web_path, "pretrained_models")- model = torch.load(local_path, map_location=lambda storage, loc: storage)- return model---if __name__ == "__main__":- # Download all DiT checkpoints- for model in pretrained_models:- download_model(model)- print("Done.")diff --git a/open_sora/__init__.py b/open_sora/__init__.pydeleted file mode 100644index e69de29..0000000diff --git a/open_sora/diffusion/__init__.py b/open_sora/diffusion/__init__.pydeleted file mode 100644index 427068d..0000000--- a/open_sora/diffusion/__init__.py+++ /dev/null@@ -1,40 +0,0 @@-# Modified from OpenAI's diffusion repos-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py--from . import gaussian_diffusion as gd-from .respace import SpacedDiffusion, space_timesteps---def create_diffusion(- timestep_respacing,- noise_schedule="linear",- use_kl=False,- sigma_small=False,- predict_xstart=False,- learn_sigma=True,- rescale_learned_sigmas=False,- diffusion_steps=1000,-):- betas = gd.get_named_beta_schedule(noise_schedule, diffusion_steps)- if use_kl:- loss_type = gd.LossType.RESCALED_KL- elif rescale_learned_sigmas:- loss_type = gd.LossType.RESCALED_MSE- else:- loss_type = gd.LossType.MSE- if timestep_respacing is None or timestep_respacing == "":- timestep_respacing = [diffusion_steps]- return SpacedDiffusion(- use_timesteps=space_timesteps(diffusion_steps, timestep_respacing),- betas=betas,- model_mean_type=(gd.ModelMeanType.EPSILON if not predict_xstart else gd.ModelMeanType.START_X),- model_var_type=(- (gd.ModelVarType.FIXED_LARGE if not sigma_small else gd.ModelVarType.FIXED_SMALL)- if not learn_sigma- else gd.ModelVarType.LEARNED_RANGE- ),- loss_type=loss_type- # rescale_timesteps=rescale_timesteps,- )diff --git a/open_sora/diffusion/diffusion_utils.py b/open_sora/diffusion/diffusion_utils.pydeleted file mode 100644index 056471c..0000000--- a/open_sora/diffusion/diffusion_utils.py+++ /dev/null@@ -1,79 +0,0 @@-# Modified from OpenAI's diffusion repos-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py--import numpy as np-import torch as th---def normal_kl(mean1, logvar1, mean2, logvar2):- """- Compute the KL divergence between two gaussians.- Shapes are automatically broadcasted, so batches can be compared to- scalars, among other use cases.- """- tensor = None- for obj in (mean1, logvar1, mean2, logvar2):- if isinstance(obj, th.Tensor):- tensor = obj- break- assert tensor is not None, "at least one argument must be a Tensor"-- # Force variances to be Tensors. Broadcasting helps convert scalars to- # Tensors, but it does not work for th.exp().- logvar1, logvar2 = [x if isinstance(x, th.Tensor) else th.tensor(x).to(tensor) for x in (logvar1, logvar2)]-- return 0.5 * (-1.0 + logvar2 - logvar1 + th.exp(logvar1 - logvar2) + ((mean1 - mean2) ** 2) * th.exp(-logvar2))---def approx_standard_normal_cdf(x):- """- A fast approximation of the cumulative distribution function of the- standard normal.- """- return 0.5 * (1.0 + th.tanh(np.sqrt(2.0 / np.pi) * (x + 0.044715 * th.pow(x, 3))))---def continuous_gaussian_log_likelihood(x, *, means, log_scales):- """- Compute the log-likelihood of a continuous Gaussian distribution.- :param x: the targets- :param means: the Gaussian mean Tensor.- :param log_scales: the Gaussian log stddev Tensor.- :return: a tensor like x of log probabilities (in nats).- """- centered_x = x - means- inv_stdv = th.exp(-log_scales)- normalized_x = centered_x * inv_stdv- log_probs = th.distributions.Normal(th.zeros_like(x), th.ones_like(x)).log_prob(normalized_x)- return log_probs---def discretized_gaussian_log_likelihood(x, *, means, log_scales):- """- Compute the log-likelihood of a Gaussian distribution discretizing to a- given image.- :param x: the target images. It is assumed that this was uint8 values,- rescaled to the range [-1, 1].- :param means: the Gaussian mean Tensor.- :param log_scales: the Gaussian log stddev Tensor.- :return: a tensor like x of log probabilities (in nats).- """- assert x.shape == means.shape == log_scales.shape- centered_x = x - means- inv_stdv = th.exp(-log_scales)- plus_in = inv_stdv * (centered_x + 1.0 / 255.0)- cdf_plus = approx_standard_normal_cdf(plus_in)- min_in = inv_stdv * (centered_x - 1.0 / 255.0)- cdf_min = approx_standard_normal_cdf(min_in)- log_cdf_plus = th.log(cdf_plus.clamp(min=1e-12))- log_one_minus_cdf_min = th.log((1.0 - cdf_min).clamp(min=1e-12))- cdf_delta = cdf_plus - cdf_min- log_probs = th.where(- x < -0.999,- log_cdf_plus,- th.where(x > 0.999, log_one_minus_cdf_min, th.log(cdf_delta.clamp(min=1e-12))),- )- assert log_probs.shape == x.shape- return log_probsdiff --git a/open_sora/diffusion/gaussian_diffusion.py b/open_sora/diffusion/gaussian_diffusion.pydeleted file mode 100644index fe94622..0000000--- a/open_sora/diffusion/gaussian_diffusion.py+++ /dev/null@@ -1,842 +0,0 @@-# Modified from OpenAI's diffusion repos-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py---import enum-import math--import numpy as np-import torch as th--from .diffusion_utils import discretized_gaussian_log_likelihood, normal_kl---def mean_flat(tensor, mask=None):- """- Take the mean over all non-batch dimensions.- """- if mask is not None:- tensor = tensor * mask- return tensor.mean(dim=list(range(1, len(tensor.shape))))---class ModelMeanType(enum.Enum):- """- Which type of output the model predicts.- """-- PREVIOUS_X = enum.auto() # the model predicts x_{t-1}- START_X = enum.auto() # the model predicts x_0- EPSILON = enum.auto() # the model predicts epsilon---class ModelVarType(enum.Enum):- """- What is used as the model's output variance.- The LEARNED_RANGE option has been added to allow the model to predict- values between FIXED_SMALL and FIXED_LARGE, making its job easier.- """-- LEARNED = enum.auto()- FIXED_SMALL = enum.auto()- FIXED_LARGE = enum.auto()- LEARNED_RANGE = enum.auto()---class LossType(enum.Enum):- MSE = enum.auto() # use raw MSE loss (and KL when learning variances)- RESCALED_MSE = enum.auto() # use raw MSE loss (with RESCALED_KL when learning variances)- KL = enum.auto() # use the variational lower-bound- RESCALED_KL = enum.auto() # like KL, but rescale to estimate the full VLB-- def is_vb(self):- return self == LossType.KL or self == LossType.RESCALED_KL---def _warmup_beta(beta_start, beta_end, num_diffusion_timesteps, warmup_frac):- betas = beta_end * np.ones(num_diffusion_timesteps, dtype=np.float64)- warmup_time = int(num_diffusion_timesteps * warmup_frac)- betas[:warmup_time] = np.linspace(beta_start, beta_end, warmup_time, dtype=np.float64)- return betas---def get_beta_schedule(beta_schedule, *, beta_start, beta_end, num_diffusion_timesteps):- """- This is the deprecated API for creating beta schedules.- See get_named_beta_schedule() for the new library of schedules.- """- if beta_schedule == "quad":- betas = (- np.linspace(- beta_start**0.5,- beta_end**0.5,- num_diffusion_timesteps,- dtype=np.float64,- )- ** 2- )- elif beta_schedule == "linear":- betas = np.linspace(beta_start, beta_end, num_diffusion_timesteps, dtype=np.float64)- elif beta_schedule == "warmup10":- betas = _warmup_beta(beta_start, beta_end, num_diffusion_timesteps, 0.1)- elif beta_schedule == "warmup50":- betas = _warmup_beta(beta_start, beta_end, num_diffusion_timesteps, 0.5)- elif beta_schedule == "const":- betas = beta_end * np.ones(num_diffusion_timesteps, dtype=np.float64)- elif beta_schedule == "jsd": # 1/T, 1/(T-1), 1/(T-2), ..., 1- betas = 1.0 / np.linspace(num_diffusion_timesteps, 1, num_diffusion_timesteps, dtype=np.float64)- else:- raise NotImplementedError(beta_schedule)- assert betas.shape == (num_diffusion_timesteps,)- return betas---def get_named_beta_schedule(schedule_name, num_diffusion_timesteps):- """- Get a pre-defined beta schedule for the given name.- The beta schedule library consists of beta schedules which remain similar- in the limit of num_diffusion_timesteps.- Beta schedules may be added, but should not be removed or changed once- they are committed to maintain backwards compatibility.- """- if schedule_name == "linear":- # Linear schedule from Ho et al, extended to work for any number of- # diffusion steps.- scale = 1000 / num_diffusion_timesteps- return get_beta_schedule(- "linear",- beta_start=scale * 0.0001,- beta_end=scale * 0.02,- num_diffusion_timesteps=num_diffusion_timesteps,- )- elif schedule_name == "squaredcos_cap_v2":- return betas_for_alpha_bar(- num_diffusion_timesteps,- lambda t: math.cos((t + 0.008) / 1.008 * math.pi / 2) ** 2,- )- else:- raise NotImplementedError(f"unknown beta schedule: {schedule_name}")---def betas_for_alpha_bar(num_diffusion_timesteps, alpha_bar, max_beta=0.999):- """- Create a beta schedule that discretizes the given alpha_t_bar function,- which defines the cumulative product of (1-beta) over time from t = [0,1].- :param num_diffusion_timesteps: the number of betas to produce.- :param alpha_bar: a lambda that takes an argument t from 0 to 1 and- produces the cumulative product of (1-beta) up to that- part of the diffusion process.- :param max_beta: the maximum beta to use; use values lower than 1 to- prevent singularities.- """- betas = []- for i in range(num_diffusion_timesteps):- t1 = i / num_diffusion_timesteps- t2 = (i + 1) / num_diffusion_timesteps- betas.append(min(1 - alpha_bar(t2) / alpha_bar(t1), max_beta))- return np.array(betas)---class GaussianDiffusion:- """- Utilities for training and sampling diffusion models.- Original ported from this codebase:- https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/diffusion_utils_2.py#L42- :param betas: a 1-D numpy array of betas for each diffusion timestep,- starting at T and going to 1.- """-- def __init__(self, *, betas, model_mean_type, model_var_type, loss_type):- self.model_mean_type = model_mean_type- self.model_var_type = model_var_type- self.loss_type = loss_type-- # Use float64 for accuracy.- betas = np.array(betas, dtype=np.float64)- self.betas = betas- assert len(betas.shape) == 1, "betas must be 1-D"- assert (betas > 0).all() and (betas <= 1).all()-- self.num_timesteps = int(betas.shape[0])-- alphas = 1.0 - betas- self.alphas_cumprod = np.cumprod(alphas, axis=0)- self.alphas_cumprod_prev = np.append(1.0, self.alphas_cumprod[:-1])- self.alphas_cumprod_next = np.append(self.alphas_cumprod[1:], 0.0)- assert self.alphas_cumprod_prev.shape == (self.num_timesteps,)-- # calculations for diffusion q(x_t | x_{t-1}) and others- self.sqrt_alphas_cumprod = np.sqrt(self.alphas_cumprod)- self.sqrt_one_minus_alphas_cumprod = np.sqrt(1.0 - self.alphas_cumprod)- self.log_one_minus_alphas_cumprod = np.log(1.0 - self.alphas_cumprod)- self.sqrt_recip_alphas_cumprod = np.sqrt(1.0 / self.alphas_cumprod)- self.sqrt_recipm1_alphas_cumprod = np.sqrt(1.0 / self.alphas_cumprod - 1)-- # calculations for posterior q(x_{t-1} | x_t, x_0)- self.posterior_variance = betas * (1.0 - self.alphas_cumprod_prev) / (1.0 - self.alphas_cumprod)- # below: log calculation clipped because the posterior variance is 0 at the beginning of the diffusion chain- self.posterior_log_variance_clipped = (- np.log(np.append(self.posterior_variance[1], self.posterior_variance[1:]))- if len(self.posterior_variance) > 1- else np.array([])- )-- self.posterior_mean_coef1 = betas * np.sqrt(self.alphas_cumprod_prev) / (1.0 - self.alphas_cumprod)- self.posterior_mean_coef2 = (1.0 - self.alphas_cumprod_prev) * np.sqrt(alphas) / (1.0 - self.alphas_cumprod)-- def q_mean_variance(self, x_start, t):- """- Get the distribution q(x_t | x_0).- :param x_start: the [N x T x C x ...] tensor of noiseless inputs.- :param t: the number of diffusion steps (minus 1). Here, 0 means one step.- :return: A tuple (mean, variance, log_variance), all of x_start's shape.- """- mean = _extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start- variance = _extract_into_tensor(1.0 - self.alphas_cumprod, t, x_start.shape)- log_variance = _extract_into_tensor(self.log_one_minus_alphas_cumprod, t, x_start.shape)- return mean, variance, log_variance-- def q_sample(self, x_start, t, noise=None):- """- Diffuse the data for a given number of diffusion steps.- In other words, sample from q(x_t | x_0).- :param x_start: the initial data batch.- :param t: the number of diffusion steps (minus 1). Here, 0 means one step.- :param noise: if specified, the split-out normal noise.- :return: A noisy version of x_start.- """- if noise is None:- noise = th.randn_like(x_start)- assert noise.shape == x_start.shape- return (- _extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start- + _extract_into_tensor(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise- )-- def q_posterior_mean_variance(self, x_start, x_t, t):- """- Compute the mean and variance of the diffusion posterior:- q(x_{t-1} | x_t, x_0)- """- assert x_start.shape == x_t.shape- posterior_mean = (- _extract_into_tensor(self.posterior_mean_coef1, t, x_t.shape) * x_start- + _extract_into_tensor(self.posterior_mean_coef2, t, x_t.shape) * x_t- )- posterior_variance = _extract_into_tensor(self.posterior_variance, t, x_t.shape)- posterior_log_variance_clipped = _extract_into_tensor(self.posterior_log_variance_clipped, t, x_t.shape)- assert (- posterior_mean.shape[0]- == posterior_variance.shape[0]- == posterior_log_variance_clipped.shape[0]- == x_start.shape[0]- )- return posterior_mean, posterior_variance, posterior_log_variance_clipped-- def p_mean_variance(self, model, x, t, clip_denoised=True, denoised_fn=None, model_kwargs=None):- """- Apply the model to get p(x_{t-1} | x_t), as well as a prediction of- the initial x, x_0.- :param model: the model, which takes a signal and a batch of timesteps- as input.- :param x: the [N x T x C x ...] tensor at time t.- :param t: a 1-D Tensor of timesteps.- :param clip_denoised: if True, clip the denoised signal into [-1, 1].- :param denoised_fn: if not None, a function which applies to the- x_start prediction before it is used to sample. Applies before- clip_denoised.- :param model_kwargs: if not None, a dict of extra keyword arguments to- pass to the model. This can be used for conditioning.- :return: a dict with the following keys:- - 'mean': the model mean output.- - 'variance': the model variance output.- - 'log_variance': the log of 'variance'.- - 'pred_xstart': the prediction for x_0.- """- if model_kwargs is None:- model_kwargs = {}-- B, S, C = x.shape[:3]- assert t.shape == (B,)- model_output = model(x, t, **model_kwargs)- if isinstance(model_output, tuple):- model_output, extra = model_output- else:- extra = None-- if self.model_var_type in [ModelVarType.LEARNED, ModelVarType.LEARNED_RANGE]:- assert model_output.shape == (B, S, C * 2, *x.shape[3:])- model_output, model_var_values = th.split(model_output, C, dim=2)- min_log = _extract_into_tensor(self.posterior_log_variance_clipped, t, x.shape)- max_log = _extract_into_tensor(np.log(self.betas), t, x.shape)- # The model_var_values is [-1, 1] for [min_var, max_var].- frac = (model_var_values + 1) / 2- model_log_variance = frac * max_log + (1 - frac) * min_log- model_variance = th.exp(model_log_variance)- else:- model_variance, model_log_variance = {- # for fixedlarge, we set the initial (log-)variance like so- # to get a better decoder log likelihood.- ModelVarType.FIXED_LARGE: (- np.append(self.posterior_variance[1], self.betas[1:]),- np.log(np.append(self.posterior_variance[1], self.betas[1:])),- ),- ModelVarType.FIXED_SMALL: (- self.posterior_variance,- self.posterior_log_variance_clipped,- ),- }[self.model_var_type]- model_variance = _extract_into_tensor(model_variance, t, x.shape)- model_log_variance = _extract_into_tensor(model_log_variance, t, x.shape)-- def process_xstart(x):- if denoised_fn is not None:- x = denoised_fn(x)- if clip_denoised:- return x.clamp(-1, 1)- return x-- if self.model_mean_type == ModelMeanType.START_X:- pred_xstart = process_xstart(model_output)- else:- pred_xstart = process_xstart(self._predict_xstart_from_eps(x_t=x, t=t, eps=model_output))- model_mean, _, _ = self.q_posterior_mean_variance(x_start=pred_xstart, x_t=x, t=t)-- assert model_mean.shape == model_log_variance.shape == pred_xstart.shape == x.shape- return {- "mean": model_mean,- "variance": model_variance,- "log_variance": model_log_variance,- "pred_xstart": pred_xstart,- "extra": extra,- }-- def _predict_xstart_from_eps(self, x_t, t, eps):- assert x_t.shape == eps.shape- return (- _extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x_t.shape) * x_t- - _extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x_t.shape) * eps- )-- def _predict_eps_from_xstart(self, x_t, t, pred_xstart):- return (- _extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x_t.shape) * x_t - pred_xstart- ) / _extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x_t.shape)-- def condition_mean(self, cond_fn, p_mean_var, x, t, model_kwargs=None):- """- Compute the mean for the previous step, given a function cond_fn that- computes the gradient of a conditional log probability with respect to- x. In particular, cond_fn computes grad(log(p(y|x))), and we want to- condition on y.- This uses the conditioning strategy from Sohl-Dickstein et al. (2015).- """- gradient = cond_fn(x, t, **model_kwargs)- new_mean = p_mean_var["mean"].float() + p_mean_var["variance"] * gradient.float()- return new_mean-- def condition_score(self, cond_fn, p_mean_var, x, t, model_kwargs=None):- """- Compute what the p_mean_variance output would have been, should the- model's score function be conditioned by cond_fn.- See condition_mean() for details on cond_fn.- Unlike condition_mean(), this instead uses the conditioning strategy- from Song et al (2020).- """- alpha_bar = _extract_into_tensor(self.alphas_cumprod, t, x.shape)-- eps = self._predict_eps_from_xstart(x, t, p_mean_var["pred_xstart"])- eps = eps - (1 - alpha_bar).sqrt() * cond_fn(x, t, **model_kwargs)-- out = p_mean_var.copy()- out["pred_xstart"] = self._predict_xstart_from_eps(x, t, eps)- out["mean"], _, _ = self.q_posterior_mean_variance(x_start=out["pred_xstart"], x_t=x, t=t)- return out-- def p_sample(- self,- model,- x,- t,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- ):- """- Sample x_{t-1} from the model at the given timestep.- :param model: the model to sample from.- :param x: the current tensor at x_{t-1}.- :param t: the value of t, starting at 0 for the first diffusion step.- :param clip_denoised: if True, clip the x_start prediction to [-1, 1].- :param denoised_fn: if not None, a function which applies to the- x_start prediction before it is used to sample.- :param cond_fn: if not None, this is a gradient function that acts- similarly to the model.- :param model_kwargs: if not None, a dict of extra keyword arguments to- pass to the model. This can be used for conditioning.- :return: a dict containing the following keys:- - 'sample': a random sample from the model.- - 'pred_xstart': a prediction of x_0.- """- out = self.p_mean_variance(- model,- x,- t,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- model_kwargs=model_kwargs,- )- noise = th.randn_like(x)- nonzero_mask = (t != 0).float().view(-1, *([1] * (len(x.shape) - 1))) # no noise when t == 0- if cond_fn is not None:- out["mean"] = self.condition_mean(cond_fn, out, x, t, model_kwargs=model_kwargs)- sample = out["mean"] + nonzero_mask * th.exp(0.5 * out["log_variance"]) * noise- return {"sample": sample, "pred_xstart": out["pred_xstart"]}-- def p_sample_loop(- self,- model,- shape,- noise=None,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- device=None,- progress=False,- ):- """- Generate samples from the model.- :param model: the model module.- :param shape: the shape of the samples, (N, T, C, H, W).- :param noise: if specified, the noise from the encoder to sample.- Should be of the same shape as `shape`.- :param clip_denoised: if True, clip x_start predictions to [-1, 1].- :param denoised_fn: if not None, a function which applies to the- x_start prediction before it is used to sample.- :param cond_fn: if not None, this is a gradient function that acts- similarly to the model.- :param model_kwargs: if not None, a dict of extra keyword arguments to- pass to the model. This can be used for conditioning.- :param device: if specified, the device to create the samples on.- If not specified, use a model parameter's device.- :param progress: if True, show a tqdm progress bar.- :return: a non-differentiable batch of samples.- """- final = None- for sample in self.p_sample_loop_progressive(- model,- shape,- noise=noise,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- cond_fn=cond_fn,- model_kwargs=model_kwargs,- device=device,- progress=progress,- ):- final = sample- return final["sample"]-- def p_sample_loop_progressive(- self,- model,- shape,- noise=None,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- device=None,- progress=False,- ):- """- Generate samples from the model and yield intermediate samples from- each timestep of diffusion.- Arguments are the same as p_sample_loop().- Returns a generator over dicts, where each dict is the return value of- p_sample().- """- if device is None:- device = next(model.parameters()).device- assert isinstance(shape, (tuple, list))- if noise is not None:- img = noise- else:- img = th.randn(*shape, device=device)- indices = list(range(self.num_timesteps))[::-1]-- if progress:- # Lazy import so that we don't depend on tqdm.- from tqdm.auto import tqdm-- indices = tqdm(indices)-- for i in indices:- t = th.tensor([i] * shape[0], device=device)- with th.no_grad():- out = self.p_sample(- model,- img,- t,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- cond_fn=cond_fn,- model_kwargs=model_kwargs,- )- yield out- img = out["sample"]-- def ddim_sample(- self,- model,- x,- t,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- eta=0.0,- ):- """- Sample x_{t-1} from the model using DDIM.- Same usage as p_sample().- """- out = self.p_mean_variance(- model,- x,- t,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- model_kwargs=model_kwargs,- )- if cond_fn is not None:- out = self.condition_score(cond_fn, out, x, t, model_kwargs=model_kwargs)-- # Usually our model outputs epsilon, but we re-derive it- # in case we used x_start or x_prev prediction.- eps = self._predict_eps_from_xstart(x, t, out["pred_xstart"])-- alpha_bar = _extract_into_tensor(self.alphas_cumprod, t, x.shape)- alpha_bar_prev = _extract_into_tensor(self.alphas_cumprod_prev, t, x.shape)- sigma = eta * th.sqrt((1 - alpha_bar_prev) / (1 - alpha_bar)) * th.sqrt(1 - alpha_bar / alpha_bar_prev)- # Equation 12.- noise = th.randn_like(x)- mean_pred = out["pred_xstart"] * th.sqrt(alpha_bar_prev) + th.sqrt(1 - alpha_bar_prev - sigma**2) * eps- nonzero_mask = (t != 0).float().view(-1, *([1] * (len(x.shape) - 1))) # no noise when t == 0- sample = mean_pred + nonzero_mask * sigma * noise- return {"sample": sample, "pred_xstart": out["pred_xstart"]}-- def ddim_reverse_sample(- self,- model,- x,- t,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- eta=0.0,- ):- """- Sample x_{t+1} from the model using DDIM reverse ODE.- """- assert eta == 0.0, "Reverse ODE only for deterministic path"- out = self.p_mean_variance(- model,- x,- t,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- model_kwargs=model_kwargs,- )- if cond_fn is not None:- out = self.condition_score(cond_fn, out, x, t, model_kwargs=model_kwargs)- # Usually our model outputs epsilon, but we re-derive it- # in case we used x_start or x_prev prediction.- eps = (- _extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x.shape) * x - out["pred_xstart"]- ) / _extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x.shape)- alpha_bar_next = _extract_into_tensor(self.alphas_cumprod_next, t, x.shape)-- # Equation 12. reversed- mean_pred = out["pred_xstart"] * th.sqrt(alpha_bar_next) + th.sqrt(1 - alpha_bar_next) * eps-- return {"sample": mean_pred, "pred_xstart": out["pred_xstart"]}-- def ddim_sample_loop(- self,- model,- shape,- noise=None,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- device=None,- progress=False,- eta=0.0,- ):- """- Generate samples from the model using DDIM.- Same usage as p_sample_loop().- """- final = None- for sample in self.ddim_sample_loop_progressive(- model,- shape,- noise=noise,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- cond_fn=cond_fn,- model_kwargs=model_kwargs,- device=device,- progress=progress,- eta=eta,- ):- final = sample- return final["sample"]-- def ddim_sample_loop_progressive(- self,- model,- shape,- noise=None,- clip_denoised=True,- denoised_fn=None,- cond_fn=None,- model_kwargs=None,- device=None,- progress=False,- eta=0.0,- ):- """- Use DDIM to sample from the model and yield intermediate samples from- each timestep of DDIM.- Same usage as p_sample_loop_progressive().- """- if device is None:- device = next(model.parameters()).device- assert isinstance(shape, (tuple, list))- if noise is not None:- img = noise- else:- img = th.randn(*shape, device=device)- indices = list(range(self.num_timesteps))[::-1]-- if progress:- # Lazy import so that we don't depend on tqdm.- from tqdm.auto import tqdm-- indices = tqdm(indices)-- for i in indices:- t = th.tensor([i] * shape[0], device=device)- with th.no_grad():- out = self.ddim_sample(- model,- img,- t,- clip_denoised=clip_denoised,- denoised_fn=denoised_fn,- cond_fn=cond_fn,- model_kwargs=model_kwargs,- eta=eta,- )- yield out- img = out["sample"]-- def _vb_terms_bpd(self, model, x_start, x_t, t, clip_denoised=True, model_kwargs=None, mask=None):- """- Get a term for the variational lower-bound.- The resulting units are bits (rather than nats, as one might expect).- This allows for comparison to other papers.- :return: a dict with the following keys:- - 'output': a shape [N] tensor of NLLs or KLs.- - 'pred_xstart': the x_0 predictions.- """- true_mean, _, true_log_variance_clipped = self.q_posterior_mean_variance(x_start=x_start, x_t=x_t, t=t)- out = self.p_mean_variance(model, x_t, t, clip_denoised=clip_denoised, model_kwargs=model_kwargs)- kl = normal_kl(true_mean, true_log_variance_clipped, out["mean"], out["log_variance"])- kl = mean_flat(kl, mask=mask) / np.log(2.0)-- decoder_nll = -discretized_gaussian_log_likelihood(- x_start, means=out["mean"], log_scales=0.5 * out["log_variance"]- )- assert decoder_nll.shape == x_start.shape- decoder_nll = mean_flat(decoder_nll, mask=mask) / np.log(2.0)-- # At the first timestep return the decoder NLL,- # otherwise return KL(q(x_{t-1}|x_t,x_0) || p(x_{t-1}|x_t))- output = th.where((t == 0), decoder_nll, kl)- return {"output": output, "pred_xstart": out["pred_xstart"]}-- def _expand_mask(self, mask, ndim: int):- assert mask.ndim == 2- # [B, S] -> [B, S, ...]- mask = mask.view(*mask.shape, *([1] * (ndim - mask.ndim)))- return mask-- def training_losses(self, model, x_start, t, model_kwargs=None, noise=None, mask=None):- """- Compute training losses for a single timestep.- :param model: the model to evaluate loss on.- :param x_start: the [N x T x C x ...] tensor of inputs.- :param t: a batch of timestep indices.- :param model_kwargs: if not None, a dict of extra keyword arguments to- pass to the model. This can be used for conditioning.- :param noise: if specified, the specific Gaussian noise to try to remove.- :return: a dict with the key "loss" containing a tensor of shape [N].- Some mean or variance settings may also have other keys.- """- if model_kwargs is None:- model_kwargs = {}- if noise is None:- noise = th.randn_like(x_start)- x_t = self.q_sample(x_start, t, noise=noise)-- terms = {}-- if mask is not None:- mask = self._expand_mask(mask, x_start.ndim)-- if self.loss_type == LossType.KL or self.loss_type == LossType.RESCALED_KL:- terms["loss"] = self._vb_terms_bpd(- model=model,- x_start=x_start,- x_t=x_t,- t=t,- clip_denoised=False,- model_kwargs=model_kwargs,- mask=mask,- )["output"]- if self.loss_type == LossType.RESCALED_KL:- terms["loss"] *= self.num_timesteps- elif self.loss_type == LossType.MSE or self.loss_type == LossType.RESCALED_MSE:- model_output = model(x_t, t, **model_kwargs)-- if self.model_var_type in [- ModelVarType.LEARNED,- ModelVarType.LEARNED_RANGE,- ]:- B, S, C = x_t.shape[:3]- assert model_output.shape == (B, S, C * 2, *x_t.shape[3:])- model_output, model_var_values = th.split(model_output, C, dim=2)- # Learn the variance using the variational bound, but don't let- # it affect our mean prediction.- frozen_out = th.cat([model_output.detach(), model_var_values], dim=2)- terms["vb"] = self._vb_terms_bpd(- model=lambda *args, r=frozen_out: r,- x_start=x_start,- x_t=x_t,- t=t,- clip_denoised=False,- mask=mask,- )["output"]- if self.loss_type == LossType.RESCALED_MSE:- # Divide by 1000 for equivalence with initial implementation.- # Without a factor of 1/1000, the VB term hurts the MSE term.- terms["vb"] *= self.num_timesteps / 1000.0-- target = {- ModelMeanType.PREVIOUS_X: self.q_posterior_mean_variance(x_start=x_start, x_t=x_t, t=t)[0],- ModelMeanType.START_X: x_start,- ModelMeanType.EPSILON: noise,- }[self.model_mean_type]- assert model_output.shape == target.shape == x_start.shape- terms["mse"] = mean_flat((target - model_output) ** 2, mask=mask)- if "vb" in terms:- terms["loss"] = terms["mse"] + terms["vb"]- else:- terms["loss"] = terms["mse"]- else:- raise NotImplementedError(self.loss_type)-- return terms-- def _prior_bpd(self, x_start):- """- Get the prior KL term for the variational lower-bound, measured in- bits-per-dim.- This term can't be optimized, as it only depends on the encoder.- :param x_start: the [N x T x C x ...] tensor of inputs.- :return: a batch of [N] KL values (in bits), one per batch element.- """- batch_size = x_start.shape[0]- t = th.tensor([self.num_timesteps - 1] * batch_size, device=x_start.device)- qt_mean, _, qt_log_variance = self.q_mean_variance(x_start, t)- kl_prior = normal_kl(mean1=qt_mean, logvar1=qt_log_variance, mean2=0.0, logvar2=0.0)- return mean_flat(kl_prior) / np.log(2.0)-- def calc_bpd_loop(self, model, x_start, clip_denoised=True, model_kwargs=None):- """- Compute the entire variational lower-bound, measured in bits-per-dim,- as well as other related quantities.- :param model: the model to evaluate loss on.- :param x_start: the [N x T x C x ...] tensor of inputs.- :param clip_denoised: if True, clip denoised samples.- :param model_kwargs: if not None, a dict of extra keyword arguments to- pass to the model. This can be used for conditioning.- :return: a dict containing the following keys:- - total_bpd: the total variational lower-bound, per batch element.- - prior_bpd: the prior term in the lower-bound.- - vb: an [N x T] tensor of terms in the lower-bound.- - xstart_mse: an [N x T] tensor of x_0 MSEs for each timestep.- - mse: an [N x T] tensor of epsilon MSEs for each timestep.- """- device = x_start.device- batch_size = x_start.shape[0]-- vb = []- xstart_mse = []- mse = []- for t in list(range(self.num_timesteps))[::-1]:- t_batch = th.tensor([t] * batch_size, device=device)- noise = th.randn_like(x_start)- x_t = self.q_sample(x_start=x_start, t=t_batch, noise=noise)- # Calculate VLB term at the current timestep- with th.no_grad():- out = self._vb_terms_bpd(- model,- x_start=x_start,- x_t=x_t,- t=t_batch,- clip_denoised=clip_denoised,- model_kwargs=model_kwargs,- )- vb.append(out["output"])- xstart_mse.append(mean_flat((out["pred_xstart"] - x_start) ** 2))- eps = self._predict_eps_from_xstart(x_t, t_batch, out["pred_xstart"])- mse.append(mean_flat((eps - noise) ** 2))-- vb = th.stack(vb, dim=1)- xstart_mse = th.stack(xstart_mse, dim=1)- mse = th.stack(mse, dim=1)-- prior_bpd = self._prior_bpd(x_start)- total_bpd = vb.sum(dim=1) + prior_bpd- return {- "total_bpd": total_bpd,- "prior_bpd": prior_bpd,- "vb": vb,- "xstart_mse": xstart_mse,- "mse": mse,- }---def _extract_into_tensor(arr, timesteps, broadcast_shape):- """- Extract values from a 1-D numpy array for a batch of indices.- :param arr: the 1-D numpy array.- :param timesteps: a tensor of indices into the array to extract.- :param broadcast_shape: a larger shape of K dimensions with the batch- dimension equal to the length of timesteps.- :return: a tensor of shape [batch_size, 1, ...] where the shape has K dims.- """- res = th.from_numpy(arr).to(device=timesteps.device)[timesteps].float()- while len(res.shape) < len(broadcast_shape):- res = res[..., None]- return res + th.zeros(broadcast_shape, device=timesteps.device)diff --git a/open_sora/diffusion/respace.py b/open_sora/diffusion/respace.pydeleted file mode 100644index e5754aa..0000000--- a/open_sora/diffusion/respace.py+++ /dev/null@@ -1,119 +0,0 @@-# Modified from OpenAI's diffusion repos-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py--import numpy as np-import torch as th--from .gaussian_diffusion import GaussianDiffusion---def space_timesteps(num_timesteps, section_counts):- """- Create a list of timesteps to use from an original diffusion process,- given the number of timesteps we want to take from equally-sized portions- of the original process.- For example, if there's 300 timesteps and the section counts are [10,15,20]- then the first 100 timesteps are strided to be 10 timesteps, the second 100- are strided to be 15 timesteps, and the final 100 are strided to be 20.- If the stride is a string starting with "ddim", then the fixed striding- from the DDIM paper is used, and only one section is allowed.- :param num_timesteps: the number of diffusion steps in the original- process to divide up.- :param section_counts: either a list of numbers, or a string containing- comma-separated numbers, indicating the step count- per section. As a special case, use "ddimN" where N- is a number of steps to use the striding from the- DDIM paper.- :return: a set of diffusion steps from the original process to use.- """- if isinstance(section_counts, str):- if section_counts.startswith("ddim"):- desired_count = int(section_counts[len("ddim") :])- for i in range(1, num_timesteps):- if len(range(0, num_timesteps, i)) == desired_count:- return set(range(0, num_timesteps, i))- raise ValueError(f"cannot create exactly {num_timesteps} steps with an integer stride")- section_counts = [int(x) for x in section_counts.split(",")]- size_per = num_timesteps // len(section_counts)- extra = num_timesteps % len(section_counts)- start_idx = 0- all_steps = []- for i, section_count in enumerate(section_counts):- size = size_per + (1 if i < extra else 0)- if size < section_count:- raise ValueError(f"cannot divide section of {size} steps into {section_count}")- if section_count <= 1:- frac_stride = 1- else:- frac_stride = (size - 1) / (section_count - 1)- cur_idx = 0.0- taken_steps = []- for _ in range(section_count):- taken_steps.append(start_idx + round(cur_idx))- cur_idx += frac_stride- all_steps += taken_steps- start_idx += size- return set(all_steps)---class SpacedDiffusion(GaussianDiffusion):- """- A diffusion process which can skip steps in a base diffusion process.- :param use_timesteps: a collection (sequence or set) of timesteps from the- original diffusion process to retain.- :param kwargs: the kwargs to create the base diffusion process.- """-- def __init__(self, use_timesteps, **kwargs):- self.use_timesteps = set(use_timesteps)- self.timestep_map = []- self.original_num_steps = len(kwargs["betas"])-- base_diffusion = GaussianDiffusion(**kwargs) # pylint: disable=missing-kwoa- last_alpha_cumprod = 1.0- new_betas = []- for i, alpha_cumprod in enumerate(base_diffusion.alphas_cumprod):- if i in self.use_timesteps:- new_betas.append(1 - alpha_cumprod / last_alpha_cumprod)- last_alpha_cumprod = alpha_cumprod- self.timestep_map.append(i)- kwargs["betas"] = np.array(new_betas)- super().__init__(**kwargs)-- def p_mean_variance(self, model, *args, **kwargs): # pylint: disable=signature-differs- return super().p_mean_variance(self._wrap_model(model), *args, **kwargs)-- def training_losses(self, model, *args, **kwargs): # pylint: disable=signature-differs- return super().training_losses(self._wrap_model(model), *args, **kwargs)-- def condition_mean(self, cond_fn, *args, **kwargs):- return super().condition_mean(self._wrap_model(cond_fn), *args, **kwargs)-- def condition_score(self, cond_fn, *args, **kwargs):- return super().condition_score(self._wrap_model(cond_fn), *args, **kwargs)-- def _wrap_model(self, model):- if isinstance(model, _WrappedModel):- return model- return _WrappedModel(model, self.timestep_map, self.original_num_steps)-- def _scale_timesteps(self, t):- # Scaling is done by the wrapped model.- return t---class _WrappedModel:- def __init__(self, model, timestep_map, original_num_steps):- self.model = model- self.timestep_map = timestep_map- # self.rescale_timesteps = rescale_timesteps- self.original_num_steps = original_num_steps-- def __call__(self, x, ts, **kwargs):- map_tensor = th.tensor(self.timestep_map, device=ts.device, dtype=ts.dtype)- new_ts = map_tensor[ts]- # if self.rescale_timesteps:- # new_ts = new_ts.float() * (1000.0 / self.original_num_steps)- return self.model(x, new_ts, **kwargs)diff --git a/open_sora/diffusion/timestep_sampler.py b/open_sora/diffusion/timestep_sampler.pydeleted file mode 100644index fdaa45a..0000000--- a/open_sora/diffusion/timestep_sampler.py+++ /dev/null@@ -1,143 +0,0 @@-# Modified from OpenAI's diffusion repos-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py--from abc import ABC, abstractmethod--import numpy as np-import torch as th-import torch.distributed as dist---def create_named_schedule_sampler(name, diffusion):- """- Create a ScheduleSampler from a library of pre-defined samplers.- :param name: the name of the sampler.- :param diffusion: the diffusion object to sample for.- """- if name == "uniform":- return UniformSampler(diffusion)- elif name == "loss-second-moment":- return LossSecondMomentResampler(diffusion)- else:- raise NotImplementedError(f"unknown schedule sampler: {name}")---class ScheduleSampler(ABC):- """- A distribution over timesteps in the diffusion process, intended to reduce- variance of the objective.- By default, samplers perform unbiased importance sampling, in which the- objective's mean is unchanged.- However, subclasses may override sample() to change how the resampled- terms are reweighted, allowing for actual changes in the objective.- """-- @abstractmethod- def weights(self):- """- Get a numpy array of weights, one per diffusion step.- The weights needn't be normalized, but must be positive.- """-- def sample(self, batch_size, device):- """- Importance-sample timesteps for a batch.- :param batch_size: the number of timesteps.- :param device: the torch device to save to.- :return: a tuple (timesteps, weights):- - timesteps: a tensor of timestep indices.- - weights: a tensor of weights to scale the resulting losses.- """- w = self.weights()- p = w / np.sum(w)- indices_np = np.random.choice(len(p), size=(batch_size,), p=p)- indices = th.from_numpy(indices_np).long().to(device)- weights_np = 1 / (len(p) * p[indices_np])- weights = th.from_numpy(weights_np).float().to(device)- return indices, weights---class UniformSampler(ScheduleSampler):- def __init__(self, diffusion):- self.diffusion = diffusion- self._weights = np.ones([diffusion.num_timesteps])-- def weights(self):- return self._weights---class LossAwareSampler(ScheduleSampler):- def update_with_local_losses(self, local_ts, local_losses):- """- Update the reweighting using losses from a model.- Call this method from each rank with a batch of timesteps and the- corresponding losses for each of those timesteps.- This method will perform synchronization to make sure all of the ranks- maintain the exact same reweighting.- :param local_ts: an integer Tensor of timesteps.- :param local_losses: a 1D Tensor of losses.- """- batch_sizes = [th.tensor([0], dtype=th.int32, device=local_ts.device) for _ in range(dist.get_world_size())]- dist.all_gather(- batch_sizes,- th.tensor([len(local_ts)], dtype=th.int32, device=local_ts.device),- )-- # Pad all_gather batches to be the maximum batch size.- batch_sizes = [x.item() for x in batch_sizes]- max_bs = max(batch_sizes)-- timestep_batches = [th.zeros(max_bs).to(local_ts) for bs in batch_sizes]- loss_batches = [th.zeros(max_bs).to(local_losses) for bs in batch_sizes]- dist.all_gather(timestep_batches, local_ts)- dist.all_gather(loss_batches, local_losses)- timesteps = [x.item() for y, bs in zip(timestep_batches, batch_sizes) for x in y[:bs]]- losses = [x.item() for y, bs in zip(loss_batches, batch_sizes) for x in y[:bs]]- self.update_with_all_losses(timesteps, losses)-- @abstractmethod- def update_with_all_losses(self, ts, losses):- """- Update the reweighting using losses from a model.- Sub-classes should override this method to update the reweighting- using losses from the model.- This method directly updates the reweighting without synchronizing- between workers. It is called by update_with_local_losses from all- ranks with identical arguments. Thus, it should have deterministic- behavior to maintain state across workers.- :param ts: a list of int timesteps.- :param losses: a list of float losses, one per timestep.- """---class LossSecondMomentResampler(LossAwareSampler):- def __init__(self, diffusion, history_per_term=10, uniform_prob=0.001):- self.diffusion = diffusion- self.history_per_term = history_per_term- self.uniform_prob = uniform_prob- self._loss_history = np.zeros([diffusion.num_timesteps, history_per_term], dtype=np.float64)- self._loss_counts = np.zeros([diffusion.num_timesteps], dtype=np.int)-- def weights(self):- if not self._warmed_up():- return np.ones([self.diffusion.num_timesteps], dtype=np.float64)- weights = np.sqrt(np.mean(self._loss_history**2, axis=-1))- weights /= np.sum(weights)- weights *= 1 - self.uniform_prob- weights += self.uniform_prob / len(weights)- return weights-- def update_with_all_losses(self, ts, losses):- for t, loss in zip(ts, losses):- if self._loss_counts[t] == self.history_per_term:- # Shift out the oldest loss term.- self._loss_history[t, :-1] = self._loss_history[t, 1:]- self._loss_history[t, -1] = loss- else:- self._loss_history[t, self._loss_counts[t]] = loss- self._loss_counts[t] += 1-- def _warmed_up(self):- return (self._loss_counts == self.history_per_term).all()diff --git a/open_sora/modeling/__init__.py b/open_sora/modeling/__init__.pydeleted file mode 100644index f1355ce..0000000--- a/open_sora/modeling/__init__.py+++ /dev/null@@ -1,4 +0,0 @@-from .dit import DiT, DiT_models-from .latte import LatteT2V--__all__ = ["DiT_models", "DiT", "LatteT2V"]diff --git a/open_sora/modeling/dit/__init__.py b/open_sora/modeling/dit/__init__.pydeleted file mode 100644index 73813b5..0000000--- a/open_sora/modeling/dit/__init__.py+++ /dev/null@@ -1,3 +0,0 @@-from .dit import SUPPORTED_MODEL_ARCH, SUPPORTED_SEQ_PARALLEL_MODES, DiT, DiT_models--__all__ = ["DiT_models", "DiT", "SUPPORTED_SEQ_PARALLEL_MODES", "SUPPORTED_MODEL_ARCH"]diff --git a/open_sora/modeling/dit/attn.py b/open_sora/modeling/dit/attn.pydeleted file mode 100644index 6ff95f2..0000000--- a/open_sora/modeling/dit/attn.py+++ /dev/null@@ -1,276 +0,0 @@-from typing import Optional--import torch-import torch.distributed as dist-import torch.nn as nn-import torch.nn.functional as F-from colossalai.logging import get_dist_logger-from colossalai.shardformer.layer._operation import gather_forward_split_backward--from open_sora.utils.comm import all_to_all, async_all_gather_proj_for_two---class CrossAttention(nn.Module):- r"""- A cross attention layer.-- Parameters:- query_dim (`int`): The number of channels in the query.- cross_attention_dim (`int`, *optional*):- The number of channels in the context. If not given, defaults to `query_dim`.- num_heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention.- head_dim (`int`, *optional*, defaults to 64): The number of channels in each head.- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.- bias (`bool`, *optional*, defaults to False):- Set to `True` for the query, key, and value linear layers to contain a bias parameter.- """-- def __init__(- self,- query_dim: int,- cross_attention_dim: Optional[int] = None,- num_heads: int = 8,- head_dim: int = 64,- dropout: float = 0.0,- bias=False,- sdpa=True,- ):- super().__init__()- self.hidden_size = head_dim * num_heads- cross_attention_dim = cross_attention_dim if cross_attention_dim is not None else query_dim-- self.scale = head_dim**-0.5- self.num_heads = num_heads- self.head_dim = head_dim- self.sdpa = sdpa-- self.to_q = nn.Linear(query_dim, self.hidden_size, bias=bias)- self.to_k = nn.Linear(cross_attention_dim, self.hidden_size, bias=bias)- self.to_v = nn.Linear(cross_attention_dim, self.hidden_size, bias=bias)-- self.to_out = nn.Sequential(nn.Linear(self.hidden_size, query_dim), nn.Dropout(dropout))-- def forward(self, hidden_states, context=None, mask=None):- bsz, q_len, _ = hidden_states.shape-- query = self.to_q(hidden_states)- context = context if context is not None else hidden_states- kv_seq_len = context.shape[1]- key = self.to_k(context)- value = self.to_v(context)-- # [B, S, H, D]- query = query.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)- key = key.view(bsz, kv_seq_len, self.num_heads, self.head_dim).transpose(1, 2)- value = value.view(bsz, kv_seq_len, self.num_heads, self.head_dim).transpose(1, 2)-- if mask is not None:- assert mask.shape == (bsz, 1, q_len, kv_seq_len)- if self.sdpa:- attn_output = F.scaled_dot_product_attention(query, key, value, attn_mask=mask, scale=self.scale)- else:- attn_weights = torch.matmul(query, key.transpose(2, 3)) / self.scale- assert attn_weights.shape == (bsz, self.num_heads, q_len, kv_seq_len)- if mask is not None:- attn_weights = attn_weights + mask- attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)- attn_output = torch.matmul(attn_weights, value)- assert attn_output.shape == (bsz, self.num_heads, q_len, self.head_dim)- attn_output = attn_output.transpose(1, 2).contiguous()- attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)- attn_output = self.to_out(attn_output)- return attn_output---class SeqParallelCrossAttention(CrossAttention):- r"""- A cross attention layer.-- Parameters:- query_dim (`int`): The number of channels in the query.- cross_attention_dim (`int`, *optional*):- The number of channels in the context. If not given, defaults to `query_dim`.- num_heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention.- head_dim (`int`, *optional*, defaults to 64): The number of channels in each head.- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.- bias (`bool`, *optional*, defaults to False):- Set to `True` for the query, key, and value linear layers to contain a bias parameter.- """-- def __init__(- self,- query_dim: int,- cross_attention_dim: Optional[int] = None,- num_heads: int = 8,- head_dim: int = 64,- dropout: float = 0.0,- bias=False,- sdpa=True,- seq_parallel_group=None,- ):- super().__init__(- query_dim,- cross_attention_dim,- num_heads,- head_dim,- dropout,- bias,- sdpa,- )- self.seq_parallel_group = seq_parallel_group- self.seq_parallel_size = dist.get_world_size(self.seq_parallel_group) if seq_parallel_group is not None else 1- assert self.num_heads % self.seq_parallel_size == 0-- def forward(self, hidden_states, context=None, mask=None):- bsz, q_len, _ = hidden_states.shape-- query = self.to_q(hidden_states)- context = context if context is not None else hidden_states- kv_seq_len = context.shape[1]- key = self.to_k(context)- value = self.to_v(context)-- # [B, S/P, H] -> [B, S, H/P]- num_heads_parallel = self.num_heads // self.seq_parallel_size- hidden_size_parallel = self.hidden_size // self.seq_parallel_size- if self.seq_parallel_size > 1:- query = all_to_all(query, self.seq_parallel_group, scatter_dim=2, gather_dim=1)- key = all_to_all(key, self.seq_parallel_group, scatter_dim=2, gather_dim=1)- value = all_to_all(value, self.seq_parallel_group, scatter_dim=2, gather_dim=1)-- q_len *= self.seq_parallel_size- kv_seq_len *= self.seq_parallel_size-- # [B, S, H/P] -> [B, S, N/P, D] -> [B, N/P, S, D]- query = query.view(bsz, q_len, num_heads_parallel, self.head_dim).transpose(1, 2)- key = key.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)- value = value.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)-- if mask is not None:- assert mask.shape == (bsz, 1, q_len, kv_seq_len)- if self.sdpa:- attn_output = F.scaled_dot_product_attention(query, key, value, attn_mask=mask, scale=self.scale)- else:- attn_weights = torch.matmul(query, key.transpose(2, 3)) / self.scale- assert attn_weights.shape == (bsz, num_heads_parallel, q_len, kv_seq_len)- if mask is not None:- attn_weights = attn_weights + mask- attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)- attn_output = torch.matmul(attn_weights, value)- assert attn_output.shape == (bsz, num_heads_parallel, q_len, self.head_dim)- attn_output = attn_output.transpose(1, 2).contiguous()- attn_output = attn_output.reshape(bsz, q_len, hidden_size_parallel)- # [B, S, H/P] -> [B, S/P, H]- if self.seq_parallel_size > 1:- attn_output = all_to_all(attn_output, self.seq_parallel_group, scatter_dim=1, gather_dim=2)- attn_output = self.to_out(attn_output)- return attn_output---class FastSeqParallelCrossAttention(SeqParallelCrossAttention):- def __init__(- self,- query_dim: int,- cross_attention_dim: Optional[int] = None,- num_heads: int = 8,- head_dim: int = 64,- dropout: float = 0,- bias=False,- sdpa=True,- seq_parallel_group=None,- overlap=False,- ):- super().__init__(- query_dim,- cross_attention_dim,- num_heads,- head_dim,- dropout,- bias,- sdpa,- seq_parallel_group,- )- self.seq_parallel_rank = dist.get_rank(self.seq_parallel_group) if seq_parallel_group is not None else 0- self.sequence_parallel_param_slice = slice(- self.hidden_size // self.seq_parallel_size * self.seq_parallel_rank,- self.hidden_size // self.seq_parallel_size * (self.seq_parallel_rank + 1),- )- if overlap and self.seq_parallel_size != 2:- logger = get_dist_logger()- logger.warning(- "FastSeqParallelCrossAttention only supports overlap with seq_parallel_size=2. Fallback to non-overlap",- ranks=[0],- )- overlap = False- self.overlap = overlap-- def _get_sliced_params(self, proj_layer: nn.Linear):- bias = bias = proj_layer.bias[self.sequence_parallel_param_slice] if proj_layer.bias is not None else None- return proj_layer.weight[self.sequence_parallel_param_slice], bias-- def _proj(self, x: torch.Tensor, proj_layer: nn.Linear):- return F.linear(- x,- *self._get_sliced_params(proj_layer),- )-- def forward(self, hidden_states, context=None, mask=None):- bsz, q_len, _ = hidden_states.shape-- context = context if context is not None else hidden_states- kv_seq_len = context.shape[1]- if self.seq_parallel_size > 1:- if self.overlap and self.seq_parallel_size == 2:- query, key, value = async_all_gather_proj_for_two(- hidden_states,- context,- *self._get_sliced_params(self.to_q),- *self._get_sliced_params(self.to_k),- *self._get_sliced_params(self.to_v),- dim=1,- process_group=self.seq_parallel_group,- sp_size=self.seq_parallel_size,- sp_rank=self.seq_parallel_rank,- )- else:- # [B, S/P, H] -> [B, S, H]- hidden_states = gather_forward_split_backward(hidden_states, 1, self.seq_parallel_group)- context = gather_forward_split_backward(context, 1, self.seq_parallel_group)- query = self._proj(hidden_states, self.to_q)- key = self._proj(context, self.to_k)- value = self._proj(context, self.to_v)- else:- query = self.to_q(hidden_states)- key = self.to_k(context)- value = self.to_v(context)- # output is [B, S, H/P]-- num_heads_parallel = self.num_heads // self.seq_parallel_size- hidden_size_parallel = self.hidden_size // self.seq_parallel_size- q_len *= self.seq_parallel_size- kv_seq_len *= self.seq_parallel_size-- # [B, S, H/P] -> [B, S, N/P, D] -> [B, N/P, S, D]- query = query.view(bsz, q_len, num_heads_parallel, self.head_dim).transpose(1, 2)- key = key.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)- value = value.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)-- if mask is not None:- assert mask.shape == (bsz, 1, q_len, kv_seq_len)- if self.sdpa:- attn_output = F.scaled_dot_product_attention(query, key, value, attn_mask=mask, scale=self.scale)- else:- attn_weights = torch.matmul(query, key.transpose(2, 3)) / self.scale- assert attn_weights.shape == (bsz, num_heads_parallel, q_len, kv_seq_len)- if mask is not None:- attn_weights = attn_weights + mask- attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)- attn_output = torch.matmul(attn_weights, value)- assert attn_output.shape == (bsz, num_heads_parallel, q_len, self.head_dim)- attn_output = attn_output.transpose(1, 2).contiguous()- attn_output = attn_output.reshape(bsz, q_len, hidden_size_parallel)- # [B, S, H/P] -> [B, S/P, H]- if self.seq_parallel_size > 1:- attn_output = all_to_all(attn_output, self.seq_parallel_group, scatter_dim=1, gather_dim=2)- attn_output = self.to_out(attn_output)- return attn_outputdiff --git a/open_sora/modeling/dit/dit.py b/open_sora/modeling/dit/dit.pydeleted file mode 100644index 6b7bd00..0000000--- a/open_sora/modeling/dit/dit.py+++ /dev/null@@ -1,659 +0,0 @@-# Copyright (c) Meta Platforms, Inc. and affiliates.-# All rights reserved.--# This source code is licensed under the license found in the-# LICENSE file in the root directory of this source tree.-# ---------------------------------------------------------# References:-# GLIDE: https://github.com/openai/glide-text2im-# MAE: https://github.com/facebookresearch/mae/blob/main/models_mae.py-# ----------------------------------------------------------import math-from typing import Callable, Optional--import numpy as np-import torch-import torch.distributed as dist-import torch.nn as nn-import torch.nn.functional as F-from timm.models.vision_transformer import Mlp--from open_sora.utils.comm import gather_seq, split_seq--from .attn import (- CrossAttention,- FastSeqParallelCrossAttention,- SeqParallelCrossAttention,-)--SUPPORTED_SEQ_PARALLEL_MODES = ["ulysses", "fastseq"]-SUPPORTED_MODEL_ARCH = ["adaln", "cross-attn", "token-concat"]---def modulate(x, shift, scale):- return x * (1 + scale.unsqueeze(1)) + shift.unsqueeze(1)---#################################################################################-# Embedding Layers for Timesteps and Class Labels #-#################################################################################---class TimestepEmbedder(nn.Module):- """- Embeds scalar timesteps into vector representations.- """-- def __init__(self, hidden_size, frequency_embedding_size=256):- super().__init__()- self.mlp = nn.Sequential(- nn.Linear(frequency_embedding_size, hidden_size, bias=True),- nn.SiLU(),- nn.Linear(hidden_size, hidden_size, bias=True),- )- self.frequency_embedding_size = frequency_embedding_size-- @staticmethod- def timestep_embedding(t, dim, max_period=10000):- """- Create sinusoidal timestep embeddings.- :param t: a 1-D Tensor of N indices, one per batch element.- These may be fractional.- :param dim: the dimension of the output.- :param max_period: controls the minimum frequency of the embeddings.- :return: an (N, D) Tensor of positional embeddings.- """- # https://github.com/openai/glide-text2im/blob/main/glide_text2im/nn.py- half = dim // 2- freqs = torch.exp(- -math.log(max_period)- * torch.arange(start=0, end=half, dtype=torch.float32)- / half- ).to(device=t.device)- args = t[:, None].float() * freqs[None]- embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)- if dim % 2:- embedding = torch.cat(- [embedding, torch.zeros_like(embedding[:, :1])], dim=-1- )- return embedding-- def forward(self, t):- t_freq = self.timestep_embedding(t, self.frequency_embedding_size).to(- self.mlp[0].weight.dtype- )- t_emb = self.mlp(t_freq)- return t_emb---class LabelEmbedder(nn.Module):- """- Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance.- """-- def __init__(self, num_classes, hidden_size, dropout_prob):- super().__init__()- use_cfg_embedding = dropout_prob > 0- self.embedding_table = nn.Embedding(- num_classes + use_cfg_embedding, hidden_size- )- self.num_classes = num_classes- self.dropout_prob = dropout_prob-- def token_drop(self, labels, force_drop_ids=None):- """- Drops labels to enable classifier-free guidance.- """- if force_drop_ids is None:- drop_ids = (- torch.rand(labels.shape[0], device=labels.device) < self.dropout_prob- )- else:- drop_ids = force_drop_ids == 1- labels = torch.where(drop_ids, self.num_classes, labels)- return labels-- def forward(self, labels, train, force_drop_ids=None):- use_dropout = self.dropout_prob > 0- if (train and use_dropout) or (force_drop_ids is not None):- labels = self.token_drop(labels, force_drop_ids)- embeddings = self.embedding_table(labels)- return embeddings---class PatchEmbedder(nn.Module):- """Patch Embedding Layer for flat 4D video tensors."""-- def __init__(- self,- patch_size: int = 16,- in_chans: int = 3,- embed_dim: int = 768,- norm_layer: Optional[Callable] = None,- bias: bool = True,- ) -> None:- super().__init__()- self.patch_size = patch_size- self.proj = nn.Conv2d(- in_chans, embed_dim, kernel_size=patch_size, stride=patch_size, bias=bias- )- self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()-- def forward(self, x: torch.Tensor) -> torch.Tensor:- # [B, S, C, P, P] -> [B, S, C*P*P]- x = x.view(*x.shape[:2], -1)- out = F.linear(- x, self.proj.weight.view(self.proj.weight.shape[0], -1), self.proj.bias- )- out = self.norm(out)- # [B, S, H]- return out---class TextEmbedder(nn.Module):- def __init__(- self,- in_features: int,- embed_dim: int = 768,- bias: bool = True,- dropout_prob: float = 0.0,- use_proj: bool = True,- ) -> None:- super().__init__()- self.dropout_prob = dropout_prob- self.use_proj = use_proj- if self.use_proj:- self.proj = nn.Linear(in_features, embed_dim, bias=bias)-- def drop_sample(self, x: torch.Tensor) -> torch.Tensor:- drop_ids = (- torch.rand(x.shape[0], *([1] * (x.ndim - 1)), device=x.device)- < self.dropout_prob- )- x = torch.where(drop_ids, torch.zeros_like(x), x)- return x-- def forward(self, x: torch.Tensor) -> torch.Tensor:- use_dropout = self.dropout_prob > 0- if self.training and use_dropout:- x = self.drop_sample(x)- if self.use_proj:- # [B, S, C] -> [B, S, H]- x = self.proj(x)- return x---class PositionEmbedding(nn.Module):- def __init__(self, dim: int, max_position_embeddings=262114) -> None:- super().__init__()- self.dim = dim- self.max_position_embeddings = max_position_embeddings- self._set_pos_embed_cache(max_position_embeddings)-- def _set_pos_embed_cache(self, seq_len: int, device="cpu", dtype=torch.float):- self.max_seq_len_cached = seq_len- pos_embed = get_2d_sincos_pos_embed(self.dim, math.ceil(seq_len**0.5))- pos_embed = torch.from_numpy(pos_embed).to(device=device, dtype=dtype)- # [S, H]- self.register_buffer("pos_embed_cache", pos_embed, persistent=False)-- def forward(self, x: torch.Tensor) -> torch.Tensor:- # [B, S, H]- seq_len = x.shape[1]- if seq_len > self.max_seq_len_cached:- self._set_pos_embed_cache(seq_len, x.device, x.dtype)- pos_embed = self.pos_embed_cache[None, :seq_len]- return pos_embed---#################################################################################-# Core DiT Model #-#################################################################################---class DiTBlock(nn.Module):- """- A DiT block with adaptive layer norm zero (adaLN-Zero) conditioning.- """-- def __init__(- self,- hidden_size,- num_heads,- cross_attention_dim=None,- mlp_ratio=4.0,- seq_parallel_group=None,- seq_parallel_mode="ulysses",- seq_parallel_overlap=False,- ):- super().__init__()- self.norm1 = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)- attn_kwargs = {}- if seq_parallel_group is not None:- attn_kwargs["seq_parallel_group"] = seq_parallel_group- if seq_parallel_mode == "ulysses":- attn_cls = SeqParallelCrossAttention- elif seq_parallel_mode == "fastseq":- attn_cls = FastSeqParallelCrossAttention- attn_kwargs["overlap"] = seq_parallel_overlap- else:- raise ValueError(- f"seq_parallel_mode must be one of {SUPPORTED_SEQ_PARALLEL_MODES}"- )- else:- attn_cls = CrossAttention- self.attn = attn_cls(- query_dim=hidden_size,- cross_attention_dim=cross_attention_dim,- num_heads=num_heads,- head_dim=hidden_size // num_heads,- bias=True,- sdpa=True,- **attn_kwargs,- )-- self.norm2 = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)- mlp_hidden_dim = int(hidden_size * mlp_ratio)- approx_gelu = lambda: nn.GELU(approximate="tanh")- self.mlp = Mlp(- in_features=hidden_size,- hidden_features=mlp_hidden_dim,- act_layer=approx_gelu,- drop=0,- )- self.mlp = Mlp(- in_features=hidden_size,- hidden_features=mlp_hidden_dim,- act_layer=approx_gelu,- drop=0,- )- self.adaLN_modulation = nn.Sequential(- nn.SiLU(), nn.Linear(hidden_size, 6 * hidden_size, bias=True)- )-- def forward(self, x, attention_mask, t, context=None):- shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (- self.adaLN_modulation(t).chunk(6, dim=1)- )- x = x + gate_msa.unsqueeze(1) * self.attn(- modulate(self.norm1(x), shift_msa, scale_msa), context, attention_mask- )- x = x + gate_mlp.unsqueeze(1) * self.mlp(- modulate(self.norm2(x), shift_mlp, scale_mlp)- )- return x---class FinalLayer(nn.Module):- """- The final layer of DiT.- """-- def __init__(self, hidden_size, patch_size, out_channels):- super().__init__()- self.norm_final = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)- self.linear = nn.Linear(- hidden_size, patch_size * patch_size * out_channels, bias=True- )- self.patch_size = patch_size- self.adaLN_modulation = nn.Sequential(- nn.SiLU(), nn.Linear(hidden_size, 2 * hidden_size, bias=True)- )-- def unpatchify(self, x):- b, s, h = x.shape- return x.view(b, s, -1, self.patch_size, self.patch_size)-- def forward(self, x, t):- shift, scale = self.adaLN_modulation(t).chunk(2, dim=1)- x = modulate(self.norm_final(x), shift, scale)- x = self.linear(x)- x = self.unpatchify(x)- return x---class DiT(nn.Module):- """- Diffusion model with a Transformer backbone.- """-- def __init__(- self,- patch_size=2,- in_channels=3,- text_embed_dim=512,- hidden_size=1152,- depth=28,- num_heads=16,- mlp_ratio=4.0,- max_num_embeddings=256 * 1024,- text_dropout_prob=0.1,- learn_sigma=True,- seq_parallel_group=None,- seq_parallel_mode="ulysses",- seq_parallel_overlap=False,- model_arch="cross-attn",- ):- super().__init__()- assert model_arch in SUPPORTED_MODEL_ARCH- self.model_arch = model_arch- self.grad_checkpointing = False- self.learn_sigma = learn_sigma- self.in_channels = in_channels- self.out_channels = in_channels * 2 if learn_sigma else in_channels- self.patch_size = patch_size- self.num_heads = num_heads- self.seq_parallel_group = seq_parallel_group- self.seq_parallel_size = (- dist.get_world_size(self.seq_parallel_group)- if seq_parallel_group is not None- else 1- )- self.seq_parallel_rank = (- dist.get_rank(self.seq_parallel_group)- if seq_parallel_group is not None- else 0- )-- self.video_embedder = PatchEmbedder(- patch_size, in_channels, hidden_size, bias=True- )- self.t_embedder = TimestepEmbedder(hidden_size)- self.pos_embed = PositionEmbedding(hidden_size, max_num_embeddings)- self.text_embedder = TextEmbedder(- text_embed_dim,- hidden_size,- bias=True,- dropout_prob=text_dropout_prob,- use_proj=model_arch != "cross-attn",- )- if model_arch != "cross-attn":- cross_attn_dim = None- else:- cross_attn_dim = text_embed_dim-- self.blocks = nn.ModuleList(- [- DiTBlock(- hidden_size,- num_heads,- cross_attn_dim,- mlp_ratio=mlp_ratio,- seq_parallel_group=seq_parallel_group,- seq_parallel_mode=seq_parallel_mode,- seq_parallel_overlap=seq_parallel_overlap,- )- for _ in range(depth)- ]- )- self.final_layer = FinalLayer(hidden_size, patch_size, self.out_channels)- self.initialize_weights()-- def initialize_weights(self):- # Initialize transformer layers:- def _basic_init(module):- if isinstance(module, nn.Linear):- torch.nn.init.xavier_uniform_(module.weight)- if module.bias is not None:- nn.init.constant_(module.bias, 0)-- self.apply(_basic_init)-- # Initialize text embedding layer- if self.text_embedder.use_proj:- nn.init.normal_(self.text_embedder.proj.weight, std=0.02)-- # Initialize timestep embedding MLP:- nn.init.normal_(self.t_embedder.mlp[0].weight, std=0.02)- nn.init.normal_(self.t_embedder.mlp[2].weight, std=0.02)-- # Zero-out adaLN modulation layers- for block in self.blocks:- nn.init.constant_(block.adaLN_modulation[-1].weight, 0)- nn.init.constant_(block.adaLN_modulation[-1].bias, 0)-- # Zero-out output- nn.init.constant_(self.final_layer.adaLN_modulation[-1].weight, 0)- nn.init.constant_(self.final_layer.adaLN_modulation[-1].bias, 0)- nn.init.constant_(self.final_layer.linear.weight, 0)- nn.init.constant_(self.final_layer.linear.bias, 0)-- def _prepare_mask(self, attention_mask: Optional[torch.Tensor], dtype: torch.dtype):- if attention_mask is not None:- assert attention_mask.ndim == 4- attention_mask = attention_mask.to(dtype)- inverted_mask = 1.0 - attention_mask- return inverted_mask.masked_fill(- inverted_mask.to(torch.bool), torch.finfo(dtype).min- )- return attention_mask-- def enable_gradient_checkpointing(self):- self.grad_checkpointing = True-- def disable_gradient_checkpointing(self):- self.grad_checkpointing = False-- def forward(- self,- video_latent_states,- t,- text_latent_states=None,- attention_mask=None,- ):- """- video_latent_states: [B, S, C, P, P]- t: [B]- text_latent_states: [B, H] if model_arch == "adaln" else [B, S, H]- attention_mask: [B, 1, Sv, St] or [B, 1, Sv, Sv]- """- video_latent_states = self.video_embedder(video_latent_states)- text_len = text_latent_states.shape[1]- text_latent_states = self.text_embedder(text_latent_states)- if self.model_arch == "token-concat":- video_latent_states = torch.cat(- [text_latent_states, video_latent_states], dim=1- )- text_latent_states = None- pos_embed = self.pos_embed(video_latent_states)- video_latent_states = video_latent_states + pos_embed- t = self.t_embedder(t) # (N, D)- if self.model_arch == "adaln":- t = t + text_latent_states- text_latent_states = None- attention_mask = self._prepare_mask(attention_mask, video_latent_states.dtype)-- if self.seq_parallel_group is not None and self.seq_parallel_size > 1:- assert video_latent_states.shape[1] % self.seq_parallel_size == 0- video_latent_states = split_seq(- video_latent_states, self.seq_parallel_size, self.seq_parallel_rank- )- if text_latent_states is not None:- assert text_latent_states.shape[1] % self.seq_parallel_size == 0- text_latent_states = split_seq(- text_latent_states, self.seq_parallel_size, self.seq_parallel_rank- )-- for block in self.blocks:- if self.grad_checkpointing and self.training:- video_latent_states = torch.utils.checkpoint.checkpoint(- block,- video_latent_states,- attention_mask,- t,- text_latent_states,- )- else:- video_latent_states = block(- video_latent_states, attention_mask, t, text_latent_states- )-- if self.seq_parallel_group is not None and self.seq_parallel_size > 1:- video_latent_states = gather_seq(- video_latent_states,- self.seq_parallel_size,- self.seq_parallel_rank,- self.seq_parallel_group,- )-- if self.model_arch == "token-concat":- video_latent_states = video_latent_states[:, text_len:]- video_latent_states = self.final_layer(video_latent_states, t)- return video_latent_states-- def forward_with_cfg(- self, x, t, text_latent_states, cfg_scale, attention_mask=None- ):- """- Forward pass of DiT, but also batches the unconditional forward pass for classifier-free guidance.- """- # https://github.com/openai/glide-text2im/blob/main/notebooks/text2im.ipynb- half = x[: len(x) // 2]- combined = torch.cat([half, half], dim=0)- model_out = self.forward(- combined, t, text_latent_states, attention_mask=attention_mask- )- # For exact reproducibility reasons, we apply classifier-free guidance on only- # three channels by default. The standard approach to cfg applies it to all channels.- # This can be done by uncommenting the following line and commenting-out the line following that.- # eps, rest = model_out[:, :self.in_channels], model_out[:, self.in_channels:]- if self.learn_sigma:- c = model_out.shape[2]- assert c == 2 * self.in_channels- eps, rest = model_out.chunk(2, dim=2)- else:- eps = model_out- cond_eps, uncond_eps = torch.split(eps, len(eps) // 2, dim=0)- half_eps = uncond_eps + cfg_scale * (cond_eps - uncond_eps)- eps = torch.cat([half_eps, half_eps], dim=0)- if self.learn_sigma:- return torch.cat([eps, rest], dim=2)- return eps---#################################################################################-# Sine/Cosine Positional Embedding Functions #-#################################################################################-# https://github.com/facebookresearch/mae/blob/main/util/pos_embed.py---def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False, extra_tokens=0):- """- grid_size: int of the grid height and width- return:- pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token)- """- grid_h = np.arange(grid_size, dtype=np.float32)- grid_w = np.arange(grid_size, dtype=np.float32)- grid = np.meshgrid(grid_w, grid_h) # here w goes first- grid = np.stack(grid, axis=0)-- grid = grid.reshape([2, 1, grid_size, grid_size])- pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)- if cls_token and extra_tokens > 0:- pos_embed = np.concatenate(- [np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0- )- return pos_embed---def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):- assert embed_dim % 2 == 0-- # use half of dimensions to encode grid_h- emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2)- emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2)-- emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)- return emb---def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):- """- embed_dim: output dimension for each position- pos: a list of positions to be encoded: size (M,)- out: (M, D)- """- assert embed_dim % 2 == 0- omega = np.arange(embed_dim // 2, dtype=np.float64)- omega /= embed_dim / 2.0- omega = 1.0 / 10000**omega # (D/2,)-- pos = pos.reshape(-1) # (M,)- out = np.einsum("m,d->md", pos, omega) # (M, D/2), outer product-- emb_sin = np.sin(out) # (M, D/2)- emb_cos = np.cos(out) # (M, D/2)-- emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D)- return emb---#################################################################################-# DiT Configs #-#################################################################################---def DiT_XL_2(**kwargs):- return DiT(depth=28, hidden_size=1152, patch_size=2, num_heads=16, **kwargs)---def DiT_XL_4(**kwargs):- return DiT(depth=28, hidden_size=1152, patch_size=4, num_heads=16, **kwargs)---def DiT_XL_8(**kwargs):- return DiT(depth=28, hidden_size=1152, patch_size=8, num_heads=16, **kwargs)---def DiT_L_2(**kwargs):- return DiT(depth=24, hidden_size=1024, patch_size=2, num_heads=16, **kwargs)---def DiT_L_4(**kwargs):- return DiT(depth=24, hidden_size=1024, patch_size=4, num_heads=16, **kwargs)---def DiT_L_8(**kwargs):- return DiT(depth=24, hidden_size=1024, patch_size=8, num_heads=16, **kwargs)---def DiT_B_2(**kwargs):- return DiT(depth=12, hidden_size=768, patch_size=2, num_heads=12, **kwargs)---def DiT_B_4(**kwargs):- return DiT(depth=12, hidden_size=768, patch_size=4, num_heads=12, **kwargs)---def DiT_B_8(**kwargs):- return DiT(depth=12, hidden_size=768, patch_size=8, num_heads=12, **kwargs)---def DiT_S_2(**kwargs):- return DiT(depth=12, hidden_size=384, patch_size=2, num_heads=6, **kwargs)---def DiT_S_4(**kwargs):- return DiT(depth=12, hidden_size=384, patch_size=4, num_heads=6, **kwargs)---def DiT_S_8(**kwargs):- return DiT(depth=12, hidden_size=384, patch_size=8, num_heads=6, **kwargs)---DiT_models = {- "DiT-XL/2": DiT_XL_2,- "DiT-XL/4": DiT_XL_4,- "DiT-XL/8": DiT_XL_8,- "DiT-L/2": DiT_L_2,- "DiT-L/4": DiT_L_4,- "DiT-L/8": DiT_L_8,- "DiT-B/2": DiT_B_2,- "DiT-B/4": DiT_B_4,- "DiT-B/8": DiT_B_8,- "DiT-S/2": DiT_S_2,- "DiT-S/4": DiT_S_4,- "DiT-S/8": DiT_S_8,-}diff --git a/open_sora/modeling/latte/__init__.py b/open_sora/modeling/latte/__init__.pydeleted file mode 100644index 159db0f..0000000--- a/open_sora/modeling/latte/__init__.py+++ /dev/null@@ -1,3 +0,0 @@-from .latte_t2v import LatteT2V--__all__ = ["LatteT2V"]diff --git a/open_sora/modeling/latte/latte_t2v.py b/open_sora/modeling/latte/latte_t2v.pydeleted file mode 100644index 7c4de50..0000000--- a/open_sora/modeling/latte/latte_t2v.py+++ /dev/null@@ -1,1003 +0,0 @@-import json-import os-from dataclasses import dataclass-from typing import Any, Dict, Optional, Tuple--import torch-import torch.nn.functional as F-from diffusers.configuration_utils import ConfigMixin, register_to_config-from diffusers.models.activations import GEGLU, GELU, ApproximateGELU-from diffusers.models.attention import BasicTransformerBlock-from diffusers.models.attention_processor import Attention-from diffusers.models.embeddings import (- CaptionProjection,- CombinedTimestepSizeEmbeddings,- ImagePositionalEmbeddings,- PatchEmbed,- SinusoidalPositionalEmbedding,- get_1d_sincos_pos_embed_from_grid,-)-from diffusers.models.lora import LoRACompatibleConv, LoRACompatibleLinear-from diffusers.models.modeling_utils import ModelMixin-from diffusers.models.normalization import AdaLayerNorm, AdaLayerNormZero-from diffusers.utils import USE_PEFT_BACKEND, BaseOutput, deprecate-from diffusers.utils.torch_utils import maybe_allow_in_graph-from einops import rearrange, repeat-from torch import nn---@maybe_allow_in_graph-class GatedSelfAttentionDense(nn.Module):- r"""- A gated self-attention dense layer that combines visual features and object features.-- Parameters:- query_dim (`int`): The number of channels in the query.- context_dim (`int`): The number of channels in the context.- n_heads (`int`): The number of heads to use for attention.- d_head (`int`): The number of channels in each head.- """-- def __init__(self, query_dim: int, context_dim: int, n_heads: int, d_head: int):- super().__init__()-- # we need a linear projection since we need cat visual feature and obj feature- self.linear = nn.Linear(context_dim, query_dim)-- self.attn = Attention(query_dim=query_dim, heads=n_heads, dim_head=d_head)- self.ff = FeedForward(query_dim, activation_fn="geglu")-- self.norm1 = nn.LayerNorm(query_dim)- self.norm2 = nn.LayerNorm(query_dim)-- self.register_parameter("alpha_attn", nn.Parameter(torch.tensor(0.0)))- self.register_parameter("alpha_dense", nn.Parameter(torch.tensor(0.0)))-- self.enabled = True-- def forward(self, x: torch.Tensor, objs: torch.Tensor) -> torch.Tensor:- if not self.enabled:- return x-- n_visual = x.shape[1]- objs = self.linear(objs)-- x = x + self.alpha_attn.tanh() * self.attn(self.norm1(torch.cat([x, objs], dim=1)))[:, :n_visual, :]- x = x + self.alpha_dense.tanh() * self.ff(self.norm2(x))-- return x---class FeedForward(nn.Module):- r"""- A feed-forward layer.-- Parameters:- dim (`int`): The number of channels in the input.- dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.- mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.- activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.- final_dropout (`bool` *optional*, defaults to False): Apply a final dropout.- """-- def __init__(- self,- dim: int,- dim_out: Optional[int] = None,- mult: int = 4,- dropout: float = 0.0,- activation_fn: str = "geglu",- final_dropout: bool = False,- ):- super().__init__()- inner_dim = int(dim * mult)- dim_out = dim_out if dim_out is not None else dim- linear_cls = LoRACompatibleLinear if not USE_PEFT_BACKEND else nn.Linear-- if activation_fn == "gelu":- act_fn = GELU(dim, inner_dim)- if activation_fn == "gelu-approximate":- act_fn = GELU(dim, inner_dim, approximate="tanh")- elif activation_fn == "geglu":- act_fn = GEGLU(dim, inner_dim)- elif activation_fn == "geglu-approximate":- act_fn = ApproximateGELU(dim, inner_dim)-- self.net = nn.ModuleList([])- # project in- self.net.append(act_fn)- # project dropout- self.net.append(nn.Dropout(dropout))- # project out- self.net.append(linear_cls(inner_dim, dim_out))- # FF as used in Vision Transformer, MLP-Mixer, etc. have a final dropout- if final_dropout:- self.net.append(nn.Dropout(dropout))-- def forward(self, hidden_states: torch.Tensor, scale: float = 1.0) -> torch.Tensor:- compatible_cls = (GEGLU,) if USE_PEFT_BACKEND else (GEGLU, LoRACompatibleLinear)- for module in self.net:- if isinstance(module, compatible_cls):- hidden_states = module(hidden_states, scale)- else:- hidden_states = module(hidden_states)- return hidden_states---@maybe_allow_in_graph-class BasicTransformerBlock_(nn.Module):- r"""- A basic Transformer block.-- Parameters:- dim (`int`): The number of channels in the input and output.- num_attention_heads (`int`): The number of heads to use for multi-head attention.- attention_head_dim (`int`): The number of channels in each head.- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.- cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention.- activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.- num_embeds_ada_norm (:- obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`.- attention_bias (:- obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.- only_cross_attention (`bool`, *optional*):- Whether to use only cross-attention layers. In this case two cross attention layers are used.- double_self_attention (`bool`, *optional*):- Whether to use two self-attention layers. In this case no cross attention layers are used.- upcast_attention (`bool`, *optional*):- Whether to upcast the attention computation to float32. This is useful for mixed precision training.- norm_elementwise_affine (`bool`, *optional*, defaults to `True`):- Whether to use learnable elementwise affine parameters for normalization.- norm_type (`str`, *optional*, defaults to `"layer_norm"`):- The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`.- final_dropout (`bool` *optional*, defaults to False):- Whether to apply a final dropout after the last feed-forward layer.- attention_type (`str`, *optional*, defaults to `"default"`):- The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`.- positional_embeddings (`str`, *optional*, defaults to `None`):- The type of positional embeddings to apply to.- num_positional_embeddings (`int`, *optional*, defaults to `None`):- The maximum number of positional embeddings to apply.- """-- def __init__(- self,- dim: int,- num_attention_heads: int,- attention_head_dim: int,- dropout=0.0,- cross_attention_dim: Optional[int] = None,- activation_fn: str = "geglu",- num_embeds_ada_norm: Optional[int] = None,- attention_bias: bool = False,- only_cross_attention: bool = False,- double_self_attention: bool = False,- upcast_attention: bool = False,- norm_elementwise_affine: bool = True,- norm_type: str = "layer_norm", # 'layer_norm', 'ada_norm', 'ada_norm_zero', 'ada_norm_single'- norm_eps: float = 1e-5,- final_dropout: bool = False,- attention_type: str = "default",- positional_embeddings: Optional[str] = None,- num_positional_embeddings: Optional[int] = None,- ):- super().__init__()- self.only_cross_attention = only_cross_attention-- self.use_ada_layer_norm_zero = (num_embeds_ada_norm is not None) and norm_type == "ada_norm_zero"- self.use_ada_layer_norm = (num_embeds_ada_norm is not None) and norm_type == "ada_norm"- self.use_ada_layer_norm_single = norm_type == "ada_norm_single"- self.use_layer_norm = norm_type == "layer_norm"-- if norm_type in ("ada_norm", "ada_norm_zero") and num_embeds_ada_norm is None:- raise ValueError(- f"`norm_type` is set to {norm_type}, but `num_embeds_ada_norm` is not defined. Please make sure to"- f" define `num_embeds_ada_norm` if setting `norm_type` to {norm_type}."- )-- if positional_embeddings and (num_positional_embeddings is None):- raise ValueError(- "If `positional_embedding` type is defined, `num_positition_embeddings` must also be defined."- )-- if positional_embeddings == "sinusoidal":- self.pos_embed = SinusoidalPositionalEmbedding(dim, max_seq_length=num_positional_embeddings)- else:- self.pos_embed = None-- # Define 3 blocks. Each block has its own normalization layer.- # 1. Self-Attn- if self.use_ada_layer_norm:- self.norm1 = AdaLayerNorm(dim, num_embeds_ada_norm)- elif self.use_ada_layer_norm_zero:- self.norm1 = AdaLayerNormZero(dim, num_embeds_ada_norm)- else:- self.norm1 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)-- self.attn1 = Attention(- query_dim=dim,- heads=num_attention_heads,- dim_head=attention_head_dim,- dropout=dropout,- bias=attention_bias,- cross_attention_dim=cross_attention_dim if only_cross_attention else None,- upcast_attention=upcast_attention,- )-- # # 2. Cross-Attn- # if cross_attention_dim is not None or double_self_attention:- # # We currently only use AdaLayerNormZero for self attention where there will only be one attention block.- # # I.e. the number of returned modulation chunks from AdaLayerZero would not make sense if returned during- # # the second cross attention block.- # self.norm2 = (- # AdaLayerNorm(dim, num_embeds_ada_norm)- # if self.use_ada_layer_norm- # else nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)- # )- # self.attn2 = Attention(- # query_dim=dim,- # cross_attention_dim=cross_attention_dim if not double_self_attention else None,- # heads=num_attention_heads,- # dim_head=attention_head_dim,- # dropout=dropout,- # bias=attention_bias,- # upcast_attention=upcast_attention,- # ) # is self-attn if encoder_hidden_states is none- # else:- # self.norm2 = None- # self.attn2 = None-- # 3. Feed-forward- # if not self.use_ada_layer_norm_single:- # self.norm3 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)- self.norm3 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)-- self.ff = FeedForward(dim, dropout=dropout, activation_fn=activation_fn, final_dropout=final_dropout)-- # 4. Fuser- if attention_type == "gated" or attention_type == "gated-text-image":- self.fuser = GatedSelfAttentionDense(dim, cross_attention_dim, num_attention_heads, attention_head_dim)-- # 5. Scale-shift for PixArt-Alpha.- if self.use_ada_layer_norm_single:- self.scale_shift_table = nn.Parameter(torch.randn(6, dim) / dim**0.5)-- # let chunk size default to None- self._chunk_size = None- self._chunk_dim = 0-- def set_chunk_feed_forward(self, chunk_size: Optional[int], dim: int):- # Sets chunk feed-forward- self._chunk_size = chunk_size- self._chunk_dim = dim-- def forward(- self,- hidden_states: torch.FloatTensor,- attention_mask: Optional[torch.FloatTensor] = None,- encoder_hidden_states: Optional[torch.FloatTensor] = None,- encoder_attention_mask: Optional[torch.FloatTensor] = None,- timestep: Optional[torch.LongTensor] = None,- cross_attention_kwargs: Dict[str, Any] = None,- class_labels: Optional[torch.LongTensor] = None,- ) -> torch.FloatTensor:- # Notice that normalization is always applied before the real computation in the following blocks.- # 0. Self-Attention- batch_size = hidden_states.shape[0]-- if self.use_ada_layer_norm:- norm_hidden_states = self.norm1(hidden_states, timestep)- elif self.use_ada_layer_norm_zero:- norm_hidden_states, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.norm1(- hidden_states, timestep, class_labels, hidden_dtype=hidden_states.dtype- )- elif self.use_layer_norm:- norm_hidden_states = self.norm1(hidden_states)- elif self.use_ada_layer_norm_single:- shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (- self.scale_shift_table[None] + timestep.reshape(batch_size, 6, -1)- ).chunk(6, dim=1)- norm_hidden_states = self.norm1(hidden_states)- norm_hidden_states = norm_hidden_states * (1 + scale_msa) + shift_msa- norm_hidden_states = norm_hidden_states.squeeze(1)- else:- raise ValueError("Incorrect norm used")-- if self.pos_embed is not None:- norm_hidden_states = self.pos_embed(norm_hidden_states)-- # 1. Retrieve lora scale.- lora_scale = cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0-- # 2. Prepare GLIGEN inputs- cross_attention_kwargs = cross_attention_kwargs.copy() if cross_attention_kwargs is not None else {}- gligen_kwargs = cross_attention_kwargs.pop("gligen", None)-- attn_output = self.attn1(- norm_hidden_states,- encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,- attention_mask=attention_mask,- **cross_attention_kwargs,- )- if self.use_ada_layer_norm_zero:- attn_output = gate_msa.unsqueeze(1) * attn_output- elif self.use_ada_layer_norm_single:- attn_output = gate_msa * attn_output-- hidden_states = attn_output + hidden_states- if hidden_states.ndim == 4:- hidden_states = hidden_states.squeeze(1)-- # 2.5 GLIGEN Control- if gligen_kwargs is not None:- hidden_states = self.fuser(hidden_states, gligen_kwargs["objs"])-- # # 3. Cross-Attention- # if self.attn2 is not None:- # if self.use_ada_layer_norm:- # norm_hidden_states = self.norm2(hidden_states, timestep)- # elif self.use_ada_layer_norm_zero or self.use_layer_norm:- # norm_hidden_states = self.norm2(hidden_states)- # elif self.use_ada_layer_norm_single:- # # For PixArt norm2 isn't applied here:- # # https://github.com/PixArt-alpha/PixArt-alpha/blob/0f55e922376d8b797edd44d25d0e7464b260dcab/diffusion/model/nets/PixArtMS.py#L70C1-L76C103- # norm_hidden_states = hidden_states- # else:- # raise ValueError("Incorrect norm")-- # if self.pos_embed is not None and self.use_ada_layer_norm_single is False:- # norm_hidden_states = self.pos_embed(norm_hidden_states)-- # attn_output = self.attn2(- # norm_hidden_states,- # encoder_hidden_states=encoder_hidden_states,- # attention_mask=encoder_attention_mask,- # **cross_attention_kwargs,- # )- # hidden_states = attn_output + hidden_states-- # 4. Feed-forward- # if not self.use_ada_layer_norm_single:- # norm_hidden_states = self.norm3(hidden_states)-- if self.use_ada_layer_norm_zero:- norm_hidden_states = norm_hidden_states * (1 + scale_mlp[:, None]) + shift_mlp[:, None]-- if self.use_ada_layer_norm_single:- # norm_hidden_states = self.norm2(hidden_states)- norm_hidden_states = self.norm3(hidden_states)- norm_hidden_states = norm_hidden_states * (1 + scale_mlp) + shift_mlp-- if self._chunk_size is not None:- # "feed_forward_chunk_size" can be used to save memory- if norm_hidden_states.shape[self._chunk_dim] % self._chunk_size != 0:- raise ValueError(- f"`hidden_states` dimension to be chunked: {norm_hidden_states.shape[self._chunk_dim]} has to be divisible by chunk size: {self._chunk_size}. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`."- )-- num_chunks = norm_hidden_states.shape[self._chunk_dim] // self._chunk_size- ff_output = torch.cat(- [- self.ff(hid_slice, scale=lora_scale)- for hid_slice in norm_hidden_states.chunk(num_chunks, dim=self._chunk_dim)- ],- dim=self._chunk_dim,- )- else:- ff_output = self.ff(norm_hidden_states, scale=lora_scale)-- if self.use_ada_layer_norm_zero:- ff_output = gate_mlp.unsqueeze(1) * ff_output- elif self.use_ada_layer_norm_single:- ff_output = gate_mlp * ff_output-- hidden_states = ff_output + hidden_states- if hidden_states.ndim == 4:- hidden_states = hidden_states.squeeze(1)-- return hidden_states---class AdaLayerNormSingle(nn.Module):- r"""- Norm layer adaptive layer norm single (adaLN-single).-- As proposed in PixArt-Alpha (see: https://arxiv.org/abs/2310.00426; Section 2.3).-- Parameters:- embedding_dim (`int`): The size of each embedding vector.- use_additional_conditions (`bool`): To use additional conditions for normalization or not.- """-- def __init__(self, embedding_dim: int, use_additional_conditions: bool = False):- super().__init__()-- self.emb = CombinedTimestepSizeEmbeddings(- embedding_dim, size_emb_dim=embedding_dim // 3, use_additional_conditions=use_additional_conditions- )-- self.silu = nn.SiLU()- self.linear = nn.Linear(embedding_dim, 6 * embedding_dim, bias=True)-- def forward(- self,- timestep: torch.Tensor,- added_cond_kwargs: Dict[str, torch.Tensor] = None,- batch_size: int = None,- hidden_dtype: Optional[torch.dtype] = None,- ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:- # No modulation happening here.- embedded_timestep = self.emb(- timestep, batch_size=batch_size, hidden_dtype=hidden_dtype, resolution=None, aspect_ratio=None- )- return self.linear(self.silu(embedded_timestep)), embedded_timestep---@dataclass-class Transformer3DModelOutput(BaseOutput):- """- The output of [`Transformer2DModel`].-- Args:- sample (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `(batch size, num_vector_embeds - 1, num_latent_pixels)` if [`Transformer2DModel`] is discrete):- The hidden states output conditioned on the `encoder_hidden_states` input. If discrete, returns probability- distributions for the unnoised latent pixels.- """-- sample: torch.FloatTensor---class LatteT2V(ModelMixin, ConfigMixin):- _supports_gradient_checkpointing = True-- """- A 2D Transformer model for image-like data.-- Parameters:- num_attention_heads (`int`, *optional*, defaults to 16): The number of heads to use for multi-head attention.- attention_head_dim (`int`, *optional*, defaults to 88): The number of channels in each head.- in_channels (`int`, *optional*):- The number of channels in the input and output (specify if the input is **continuous**).- num_layers (`int`, *optional*, defaults to 1): The number of layers of Transformer blocks to use.- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.- cross_attention_dim (`int`, *optional*): The number of `encoder_hidden_states` dimensions to use.- sample_size (`int`, *optional*): The width of the latent images (specify if the input is **discrete**).- This is fixed during training since it is used to learn a number of position embeddings.- num_vector_embeds (`int`, *optional*):- The number of classes of the vector embeddings of the latent pixels (specify if the input is **discrete**).- Includes the class for the masked latent pixel.- activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to use in feed-forward.- num_embeds_ada_norm ( `int`, *optional*):- The number of diffusion steps used during training. Pass if at least one of the norm_layers is- `AdaLayerNorm`. This is fixed during training since it is used to learn a number of embeddings that are- added to the hidden states.-- During inference, you can denoise for up to but not more steps than `num_embeds_ada_norm`.- attention_bias (`bool`, *optional*):- Configure if the `TransformerBlocks` attention should contain a bias parameter.- """-- @register_to_config- def __init__(- self,- num_attention_heads: int = 16,- attention_head_dim: int = 88,- in_channels: Optional[int] = None,- out_channels: Optional[int] = None,- num_layers: int = 1,- dropout: float = 0.0,- norm_num_groups: int = 32,- cross_attention_dim: Optional[int] = None,- attention_bias: bool = False,- sample_size: Optional[int] = None,- num_vector_embeds: Optional[int] = None,- patch_size: Optional[int] = None,- activation_fn: str = "geglu",- num_embeds_ada_norm: Optional[int] = None,- use_linear_projection: bool = False,- only_cross_attention: bool = False,- double_self_attention: bool = False,- upcast_attention: bool = False,- norm_type: str = "layer_norm",- norm_elementwise_affine: bool = True,- norm_eps: float = 1e-5,- attention_type: str = "default",- caption_channels: int = None,- video_length: int = 16,- ):- super().__init__()- self.use_linear_projection = use_linear_projection- self.num_attention_heads = num_attention_heads- self.attention_head_dim = attention_head_dim- inner_dim = num_attention_heads * attention_head_dim- self.video_length = video_length-- conv_cls = nn.Conv2d if USE_PEFT_BACKEND else LoRACompatibleConv- linear_cls = nn.Linear if USE_PEFT_BACKEND else LoRACompatibleLinear-- # 1. Transformer2DModel can process both standard continuous images of shape `(batch_size, num_channels, width, height)` as well as quantized image embeddings of shape `(batch_size, num_image_vectors)`- # Define whether input is continuous or discrete depending on configuration- self.is_input_continuous = (in_channels is not None) and (patch_size is None)- self.is_input_vectorized = num_vector_embeds is not None- self.is_input_patches = in_channels is not None and patch_size is not None-- if norm_type == "layer_norm" and num_embeds_ada_norm is not None:- deprecation_message = (- f"The configuration file of this model: {self.__class__} is outdated. `norm_type` is either not set or"- " incorrectly set to `'layer_norm'`.Make sure to set `norm_type` to `'ada_norm'` in the config."- " Please make sure to update the config accordingly as leaving `norm_type` might led to incorrect"- " results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it"- " would be very nice if you could open a Pull request for the `transformer/config.json` file"- )- deprecate("norm_type!=num_embeds_ada_norm", "1.0.0", deprecation_message, standard_warn=False)- norm_type = "ada_norm"-- if self.is_input_continuous and self.is_input_vectorized:- raise ValueError(- f"Cannot define both `in_channels`: {in_channels} and `num_vector_embeds`: {num_vector_embeds}. Make"- " sure that either `in_channels` or `num_vector_embeds` is None."- )- elif self.is_input_vectorized and self.is_input_patches:- raise ValueError(- f"Cannot define both `num_vector_embeds`: {num_vector_embeds} and `patch_size`: {patch_size}. Make"- " sure that either `num_vector_embeds` or `num_patches` is None."- )- elif not self.is_input_continuous and not self.is_input_vectorized and not self.is_input_patches:- raise ValueError(- f"Has to define `in_channels`: {in_channels}, `num_vector_embeds`: {num_vector_embeds}, or patch_size:"- f" {patch_size}. Make sure that `in_channels`, `num_vector_embeds` or `num_patches` is not None."- )-- # 2. Define input layers- if self.is_input_continuous:- self.in_channels = in_channels-- self.norm = torch.nn.GroupNorm(num_groups=norm_num_groups, num_channels=in_channels, eps=1e-6, affine=True)- if use_linear_projection:- self.proj_in = linear_cls(in_channels, inner_dim)- else:- self.proj_in = conv_cls(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)- elif self.is_input_vectorized:- assert sample_size is not None, "Transformer2DModel over discrete input must provide sample_size"- assert num_vector_embeds is not None, "Transformer2DModel over discrete input must provide num_embed"-- self.height = sample_size- self.width = sample_size- self.num_vector_embeds = num_vector_embeds- self.num_latent_pixels = self.height * self.width-- self.latent_image_embedding = ImagePositionalEmbeddings(- num_embed=num_vector_embeds, embed_dim=inner_dim, height=self.height, width=self.width- )- elif self.is_input_patches:- assert sample_size is not None, "Transformer2DModel over patched input must provide sample_size"-- self.height = sample_size- self.width = sample_size-- self.patch_size = patch_size- interpolation_scale = self.config.sample_size // 64 # => 64 (= 512 pixart) has interpolation scale 1- interpolation_scale = max(interpolation_scale, 1)- self.pos_embed = PatchEmbed(- height=sample_size,- width=sample_size,- patch_size=patch_size,- in_channels=in_channels,- embed_dim=inner_dim,- interpolation_scale=interpolation_scale,- )-- # 3. Define transformers blocks- self.transformer_blocks = nn.ModuleList(- [- BasicTransformerBlock(- inner_dim,- num_attention_heads,- attention_head_dim,- dropout=dropout,- cross_attention_dim=cross_attention_dim,- activation_fn=activation_fn,- num_embeds_ada_norm=num_embeds_ada_norm,- attention_bias=attention_bias,- only_cross_attention=only_cross_attention,- double_self_attention=double_self_attention,- upcast_attention=upcast_attention,- norm_type=norm_type,- norm_elementwise_affine=norm_elementwise_affine,- norm_eps=norm_eps,- attention_type=attention_type,- )- for d in range(num_layers)- ]- )-- # Define temporal transformers blocks- self.temporal_transformer_blocks = nn.ModuleList(- [- BasicTransformerBlock_( # one attention- inner_dim,- num_attention_heads, # num_attention_heads- attention_head_dim, # attention_head_dim 72- dropout=dropout,- cross_attention_dim=None,- activation_fn=activation_fn,- num_embeds_ada_norm=num_embeds_ada_norm,- attention_bias=attention_bias,- only_cross_attention=only_cross_attention,- double_self_attention=False,- upcast_attention=upcast_attention,- norm_type=norm_type,- norm_elementwise_affine=norm_elementwise_affine,- norm_eps=norm_eps,- attention_type=attention_type,- )- for d in range(num_layers)- ]- )-- # 4. Define output layers- self.out_channels = in_channels if out_channels is None else out_channels- if self.is_input_continuous:- # TODO: should use out_channels for continuous projections- if use_linear_projection:- self.proj_out = linear_cls(inner_dim, in_channels)- else:- self.proj_out = conv_cls(inner_dim, in_channels, kernel_size=1, stride=1, padding=0)- elif self.is_input_vectorized:- self.norm_out = nn.LayerNorm(inner_dim)- self.out = nn.Linear(inner_dim, self.num_vector_embeds - 1)- elif self.is_input_patches and norm_type != "ada_norm_single":- self.norm_out = nn.LayerNorm(inner_dim, elementwise_affine=False, eps=1e-6)- self.proj_out_1 = nn.Linear(inner_dim, 2 * inner_dim)- self.proj_out_2 = nn.Linear(inner_dim, patch_size * patch_size * self.out_channels)- elif self.is_input_patches and norm_type == "ada_norm_single":- self.norm_out = nn.LayerNorm(inner_dim, elementwise_affine=False, eps=1e-6)- self.scale_shift_table = nn.Parameter(torch.randn(2, inner_dim) / inner_dim**0.5)- self.proj_out = nn.Linear(inner_dim, patch_size * patch_size * self.out_channels)-- # 5. PixArt-Alpha blocks.- self.adaln_single = None- self.use_additional_conditions = False- if norm_type == "ada_norm_single":- self.use_additional_conditions = self.config.sample_size == 128 # False, 128 -> 1024- # TODO(Sayak, PVP) clean this, for now we use sample size to determine whether to use- # additional conditions until we find better name- self.adaln_single = AdaLayerNormSingle(inner_dim, use_additional_conditions=self.use_additional_conditions)-- self.caption_projection = None- if caption_channels is not None:- self.caption_projection = CaptionProjection(in_features=caption_channels, hidden_size=inner_dim)-- self.gradient_checkpointing = False-- # define temporal positional embedding- temp_pos_embed = self.get_1d_sincos_temp_embed(inner_dim, video_length) # 1152 hidden size- self.register_buffer("temp_pos_embed", torch.from_numpy(temp_pos_embed).float().unsqueeze(0), persistent=False)-- def _set_gradient_checkpointing(self, module, value=False):- self.gradient_checkpointing = value-- def forward(- self,- hidden_states: torch.Tensor,- timestep: Optional[torch.LongTensor] = None,- encoder_hidden_states: Optional[torch.Tensor] = None,- added_cond_kwargs: Dict[str, torch.Tensor] = None,- class_labels: Optional[torch.LongTensor] = None,- cross_attention_kwargs: Dict[str, Any] = None,- attention_mask: Optional[torch.Tensor] = None,- encoder_attention_mask: Optional[torch.Tensor] = None,- use_image_num: int = 0,- enable_temporal_attentions: bool = True,- return_dict: bool = True,- ):- """- The [`Transformer2DModel`] forward method.-- Args:- hidden_states (`torch.LongTensor` of shape `(batch size, num latent pixels)` if discrete, `torch.FloatTensor` of shape `(batch size, frame, channel, height, width)` if continuous):- Input `hidden_states`.- encoder_hidden_states ( `torch.FloatTensor` of shape `(batch size, sequence len, embed dims)`, *optional*):- Conditional embeddings for cross attention layer. If not given, cross-attention defaults to- self-attention.- timestep ( `torch.LongTensor`, *optional*):- Used to indicate denoising step. Optional timestep to be applied as an embedding in `AdaLayerNorm`.- class_labels ( `torch.LongTensor` of shape `(batch size, num classes)`, *optional*):- Used to indicate class labels conditioning. Optional class labels to be applied as an embedding in- `AdaLayerZeroNorm`.- cross_attention_kwargs ( `Dict[str, Any]`, *optional*):- A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under- `self.processor` in- [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).- attention_mask ( `torch.Tensor`, *optional*):- An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask- is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large- negative values to the attention scores corresponding to "discard" tokens.- encoder_attention_mask ( `torch.Tensor`, *optional*):- Cross-attention mask applied to `encoder_hidden_states`. Two formats supported:-- * Mask `(batch, sequence_length)` True = keep, False = discard.- * Bias `(batch, 1, sequence_length)` 0 = keep, -10000 = discard.-- If `ndim == 2`: will be interpreted as a mask, then converted into a bias consistent with the format- above. This bias will be added to the cross-attention scores.- return_dict (`bool`, *optional*, defaults to `True`):- Whether or not to return a [`~models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain- tuple.-- Returns:- If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a- `tuple` where the first element is the sample tensor.- """- input_batch_size, c, frame, h, w = hidden_states.shape- frame = frame - use_image_num- hidden_states = rearrange(hidden_states, "b c f h w -> (b f) c h w").contiguous()-- # ensure attention_mask is a bias, and give it a singleton query_tokens dimension.- # we may have done this conversion already, e.g. if we came here via UNet2DConditionModel#forward.- # we can tell by counting dims; if ndim == 2: it's a mask rather than a bias.- # expects mask of shape:- # [batch, key_tokens]- # adds singleton query_tokens dimension:- # [batch, 1, key_tokens]- # this helps to broadcast it as a bias over attention scores, which will be in one of the following shapes:- # [batch, heads, query_tokens, key_tokens] (e.g. torch sdp attn)- # [batch * heads, query_tokens, key_tokens] (e.g. xformers or classic attn)- if attention_mask is not None and attention_mask.ndim == 2:- # assume that mask is expressed as:- # (1 = keep, 0 = discard)- # convert mask into a bias that can be added to attention scores:- # (keep = +0, discard = -10000.0)- attention_mask = (1 - attention_mask.to(hidden_states.dtype)) * -10000.0- attention_mask = attention_mask.unsqueeze(1)-- # convert encoder_attention_mask to a bias the same way we do for attention_mask- if encoder_attention_mask is not None and encoder_attention_mask.ndim == 2: # ndim == 2 means no image joint- encoder_attention_mask = (1 - encoder_attention_mask.to(hidden_states.dtype)) * -10000.0- encoder_attention_mask = encoder_attention_mask.unsqueeze(1)- encoder_attention_mask = repeat(encoder_attention_mask, "b 1 l -> (b f) 1 l", f=frame).contiguous()- elif encoder_attention_mask is not None and encoder_attention_mask.ndim == 3: # ndim == 3 means image joint- encoder_attention_mask = (1 - encoder_attention_mask.to(hidden_states.dtype)) * -10000.0- encoder_attention_mask_video = encoder_attention_mask[:, :1, ...]- encoder_attention_mask_video = repeat(- encoder_attention_mask_video, "b 1 l -> b (1 f) l", f=frame- ).contiguous()- encoder_attention_mask_image = encoder_attention_mask[:, 1:, ...]- encoder_attention_mask = torch.cat([encoder_attention_mask_video, encoder_attention_mask_image], dim=1)- encoder_attention_mask = rearrange(encoder_attention_mask, "b n l -> (b n) l").contiguous().unsqueeze(1)-- # Retrieve lora scale.- cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0-- # 1. Input- if self.is_input_patches: # here- height, width = hidden_states.shape[-2] // self.patch_size, hidden_states.shape[-1] // self.patch_size- num_patches = height * width-- hidden_states = self.pos_embed(hidden_states) # already add positional embeddings-- if self.adaln_single is not None:- if self.use_additional_conditions and added_cond_kwargs is None:- raise ValueError(- "`added_cond_kwargs` cannot be None when using additional conditions for `adaln_single`."- )- # batch_size = hidden_states.shape[0]- batch_size = input_batch_size- timestep, embedded_timestep = self.adaln_single(- timestep, added_cond_kwargs, batch_size=batch_size, hidden_dtype=hidden_states.dtype- )-- # 2. Blocks- if self.caption_projection is not None:- batch_size = hidden_states.shape[0]- encoder_hidden_states = self.caption_projection(encoder_hidden_states) # 3 120 1152-- if use_image_num != 0 and self.training:- encoder_hidden_states_video = encoder_hidden_states[:, :1, ...]- encoder_hidden_states_video = repeat(- encoder_hidden_states_video, "b 1 t d -> b (1 f) t d", f=frame- ).contiguous()- encoder_hidden_states_image = encoder_hidden_states[:, 1:, ...]- encoder_hidden_states = torch.cat([encoder_hidden_states_video, encoder_hidden_states_image], dim=1)- encoder_hidden_states_spatial = rearrange(encoder_hidden_states, "b f t d -> (b f) t d").contiguous()- else:- encoder_hidden_states_spatial = repeat(- encoder_hidden_states, "b t d -> (b f) t d", f=frame- ).contiguous()-- # prepare timesteps for spatial and temporal block- timestep_spatial = repeat(timestep, "b d -> (b f) d", f=frame + use_image_num).contiguous()- timestep_temp = repeat(timestep, "b d -> (b p) d", p=num_patches).contiguous()-- for i, (spatial_block, temp_block) in enumerate(zip(self.transformer_blocks, self.temporal_transformer_blocks)):- if self.training and self.gradient_checkpointing:- hidden_states = torch.utils.checkpoint.checkpoint(- spatial_block,- hidden_states,- attention_mask,- encoder_hidden_states_spatial,- encoder_attention_mask,- timestep_spatial,- cross_attention_kwargs,- class_labels,- use_reentrant=False,- )-- if enable_temporal_attentions:- hidden_states = rearrange(hidden_states, "(b f) t d -> (b t) f d", b=input_batch_size).contiguous()-- if use_image_num != 0: # image-video joitn training- hidden_states_video = hidden_states[:, :frame, ...]- hidden_states_image = hidden_states[:, frame:, ...]-- if i == 0:- hidden_states_video = hidden_states_video + self.temp_pos_embed-- hidden_states_video = torch.utils.checkpoint.checkpoint(- temp_block,- hidden_states_video,- None, # attention_mask- None, # encoder_hidden_states- None, # encoder_attention_mask- timestep_temp,- cross_attention_kwargs,- class_labels,- use_reentrant=False,- )-- hidden_states = torch.cat([hidden_states_video, hidden_states_image], dim=1)- hidden_states = rearrange(- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size- ).contiguous()-- else:- if i == 0:- hidden_states = hidden_states + self.temp_pos_embed-- hidden_states = torch.utils.checkpoint.checkpoint(- temp_block,- hidden_states,- None, # attention_mask- None, # encoder_hidden_states- None, # encoder_attention_mask- timestep_temp,- cross_attention_kwargs,- class_labels,- use_reentrant=False,- )-- hidden_states = rearrange(- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size- ).contiguous()- else:- hidden_states = spatial_block(- hidden_states,- attention_mask,- encoder_hidden_states_spatial,- encoder_attention_mask,- timestep_spatial,- cross_attention_kwargs,- class_labels,- )-- if enable_temporal_attentions:- hidden_states = rearrange(hidden_states, "(b f) t d -> (b t) f d", b=input_batch_size).contiguous()-- if use_image_num != 0 and self.training:- hidden_states_video = hidden_states[:, :frame, ...]- hidden_states_image = hidden_states[:, frame:, ...]-- hidden_states_video = temp_block(- hidden_states_video,- None, # attention_mask- None, # encoder_hidden_states- None, # encoder_attention_mask- timestep_temp,- cross_attention_kwargs,- class_labels,- )-- hidden_states = torch.cat([hidden_states_video, hidden_states_image], dim=1)- hidden_states = rearrange(- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size- ).contiguous()-- else:- if i == 0:- hidden_states = hidden_states + self.temp_pos_embed-- hidden_states = temp_block(- hidden_states,- None, # attention_mask- None, # encoder_hidden_states- None, # encoder_attention_mask- timestep_temp,- cross_attention_kwargs,- class_labels,- )-- hidden_states = rearrange(- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size- ).contiguous()-- if self.is_input_patches:- if self.config.norm_type != "ada_norm_single":- conditioning = self.transformer_blocks[0].norm1.emb(- timestep, class_labels, hidden_dtype=hidden_states.dtype- )- shift, scale = self.proj_out_1(F.silu(conditioning)).chunk(2, dim=1)- hidden_states = self.norm_out(hidden_states) * (1 + scale[:, None]) + shift[:, None]- hidden_states = self.proj_out_2(hidden_states)- elif self.config.norm_type == "ada_norm_single":- embedded_timestep = repeat(embedded_timestep, "b d -> (b f) d", f=frame + use_image_num).contiguous()- shift, scale = (self.scale_shift_table[None] + embedded_timestep[:, None]).chunk(2, dim=1)- hidden_states = self.norm_out(hidden_states)- # Modulation- hidden_states = hidden_states * (1 + scale) + shift- hidden_states = self.proj_out(hidden_states)-- # unpatchify- if self.adaln_single is None:- height = width = int(hidden_states.shape[1] ** 0.5)- hidden_states = hidden_states.reshape(- shape=(-1, height, width, self.patch_size, self.patch_size, self.out_channels)- )- hidden_states = torch.einsum("nhwpqc->nchpwq", hidden_states)- output = hidden_states.reshape(- shape=(-1, self.out_channels, height * self.patch_size, width * self.patch_size)- )- output = rearrange(output, "(b f) c h w -> b c f h w", b=input_batch_size).contiguous()-- if not return_dict:- return (output,)-- return Transformer3DModelOutput(sample=output)-- def get_1d_sincos_temp_embed(self, embed_dim, length):- pos = torch.arange(0, length).unsqueeze(1)- return get_1d_sincos_pos_embed_from_grid(embed_dim, pos)-- @classmethod- def from_pretrained_2d(cls, pretrained_model_path, subfolder=None, **kwargs):- if subfolder is not None:- pretrained_model_path = os.path.join(pretrained_model_path, subfolder)-- config_file = os.path.join(pretrained_model_path, "config.json")- if not os.path.isfile(config_file):- raise RuntimeError(f"{config_file} does not exist")- with open(config_file, "r") as f:- config = json.load(f)-- model = cls.from_config(config, **kwargs)-- # model_files = [- # os.path.join(pretrained_model_path, 'diffusion_pytorch_model.bin'),- # os.path.join(pretrained_model_path, 'diffusion_pytorch_model.safetensors')- # ]-- # model_file = None-- # for fp in model_files:- # if os.path.exists(fp):- # model_file = fp-- # if not model_file:- # raise RuntimeError(f"{model_file} does not exist")-- # if model_file.split(".")[-1] == "safetensors":- # from safetensors import safe_open- # state_dict = {}- # with safe_open(model_file, framework="pt", device="cpu") as f:- # for key in f.keys():- # state_dict[key] = f.get_tensor(key)- # else:- # state_dict = torch.load(model_file, map_location="cpu")-- # for k, v in model.state_dict().items():- # if 'temporal_transformer_blocks' in k:- # state_dict.update({k: v})-- # model.load_state_dict(state_dict)-- return modeldiff --git a/open_sora/utils/__init__.py b/open_sora/utils/__init__.pydeleted file mode 100644index e69de29..0000000diff --git a/open_sora/utils/comm.py b/open_sora/utils/comm.pydeleted file mode 100644index 9b947ff..0000000--- a/open_sora/utils/comm.py+++ /dev/null@@ -1,292 +0,0 @@-from typing import Optional, Tuple--import torch-import torch.distributed as dist-import torch.nn.functional as F-from colossalai.moe._operation import MoeInGradScaler, MoeOutGradScaler-from colossalai.shardformer.layer._operation import gather_forward_split_backward-from torch.distributed.distributed_c10d import get_global_rank---def _all_to_all(- input_: torch.Tensor,- world_size: int,- group: dist.ProcessGroup,- scatter_dim: int,- gather_dim: int,-):- input_list = [- t.contiguous() for t in torch.tensor_split(input_, world_size, scatter_dim)- ]- output_list = [torch.empty_like(input_list[0]) for _ in range(world_size)]- dist.all_to_all(output_list, input_list, group=group)- return torch.cat(output_list, dim=gather_dim).contiguous()---class _AllToAll(torch.autograd.Function):- """All-to-all communication.-- Args:- input_: input matrix- process_group: communication group- scatter_dim: scatter dimension- gather_dim: gather dimension- """-- @staticmethod- def forward(ctx, input_, process_group, scatter_dim, gather_dim):- ctx.process_group = process_group- ctx.scatter_dim = scatter_dim- ctx.gather_dim = gather_dim- ctx.world_size = dist.get_world_size(process_group)- return _all_to_all(- input_, ctx.world_size, process_group, scatter_dim, gather_dim- )-- @staticmethod- def backward(ctx, grad_output):- return (- _all_to_all(- grad_output,- ctx.world_size,- ctx.process_group,- ctx.gather_dim,- ctx.scatter_dim,- ),- None,- None,- None,- )---def all_to_all(- input_: torch.Tensor,- process_group: dist.ProcessGroup,- scatter_dim: int = 2,- gather_dim: int = 1,-):- return _AllToAll.apply(input_, process_group, scatter_dim, gather_dim)---def split_seq(input_: torch.Tensor, sp_size: int, sp_rank: int, dim: int = 1):- """Split a tensor along sequence dimension. It will split input and divide grad by sp_size.-- Args:- input_ (torch.Tensor): The common shape is (bs, seq, *).- sp_size (int): Sequence parallel size.- sp_rank (int): Sequence parallel rank.- dim (int, optional): Sequence dimension. Defaults to 1.- """- input_ = input_.chunk(sp_size, dim=dim)[sp_rank].clone()- return MoeOutGradScaler.apply(input_, sp_size)---def gather_seq(- input_: torch.Tensor,- sp_size: int,- sp_rank: int,- sp_group: dist.ProcessGroup,- dim: int = 1,-):- """Gather a tensor along sequence dimension. It will gather input and multiply grad by sp_size.-- Args:- input_ (torch.Tensor): The common shape is (bs, seq, *).- sp_size (int): Sequence parallel size.- sp_rank (int): Sequence parallel rank.- dim (int, optional): Sequence dimension. Defaults to 1.- """- input_ = gather_forward_split_backward(input_, dim, sp_group)- return MoeInGradScaler.apply(input_, sp_size)---class AsyncAllGatherProjForTwo(torch.autograd.Function):- @staticmethod- def forward(- ctx,- hidden_states: torch.Tensor,- context: torch.Tensor,- q_proj_weight: torch.Tensor,- q_proj_bias: Optional[torch.Tensor],- k_proj_weight: torch.Tensor,- k_proj_bias: Optional[torch.Tensor],- v_proj_weight: torch.Tensor,- v_proj_bias: Optional[torch.Tensor],- dim: int,- process_group: dist.ProcessGroup,- sp_size: int,- sp_rank: int,- ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:- assert sp_size == 2- ctx.process_group = process_group- ctx.sp_size = sp_size- ctx.sp_rank = sp_rank- ctx.dim = dim-- is_cross_attn = not (hidden_states is context)- ctx.is_cross_attn = is_cross_attn-- recv_hidden_states = torch.empty_like(hidden_states)- if is_cross_attn:- recv_context = torch.empty_like(context)- else:- recv_context = recv_hidden_states-- peer_global_rank = get_global_rank(process_group, 1 - sp_rank)- ops = [- dist.P2POp(dist.isend, hidden_states, peer_global_rank, process_group),- dist.P2POp(dist.irecv, recv_hidden_states, peer_global_rank, process_group),- ]- if is_cross_attn:- ops.extend(- [- dist.P2POp(dist.isend, context, peer_global_rank, process_group),- dist.P2POp(- dist.irecv, recv_context, peer_global_rank, process_group- ),- ]- )-- reqs = dist.batch_isend_irecv(ops)-- # [B, S/P, H]- q = F.linear(hidden_states, q_proj_weight, q_proj_bias)- k = F.linear(context, k_proj_weight, k_proj_bias)- v = F.linear(context, v_proj_weight, v_proj_bias)-- for req in reqs:- req.wait()-- q_other = F.linear(recv_hidden_states, q_proj_weight, q_proj_bias)- k_other = F.linear(recv_context, k_proj_weight, k_proj_bias)- v_other = F.linear(recv_context, v_proj_weight, v_proj_bias)-- if sp_rank == 0:- q = torch.cat([q, q_other], dim=dim)- k = torch.cat([k, k_other], dim=dim)- v = torch.cat([v, v_other], dim=dim)- else:- q = torch.cat([q_other, q], dim=dim)- k = torch.cat([k_other, k], dim=dim)- v = torch.cat([v_other, v], dim=dim)-- ctx.save_for_backward(- hidden_states,- context,- q_proj_weight,- q_proj_bias,- k_proj_weight,- k_proj_bias,- v_proj_weight,- v_proj_bias,- recv_hidden_states,- recv_context,- )-- return q, k, v-- @staticmethod- def backward(- ctx: torch.Any, q_grad: torch.Tensor, k_grad: torch.Tensor, v_grad: torch.Tensor- ) -> torch.Any:- (- hidden_states,- context,- q_proj_weight,- q_proj_bias,- k_proj_weight,- k_proj_bias,- v_proj_weight,- v_proj_bias,- recv_hidden_states,- recv_context,- ) = ctx.saved_tensors-- # compute param grads- if ctx.sp_rank == 0:- hidden_states = torch.cat([hidden_states, recv_hidden_states], dim=ctx.dim)- if ctx.is_cross_attn:- context = torch.cat([context, recv_context], dim=ctx.dim)- else:- context = hidden_states- else:- hidden_states = torch.cat([recv_hidden_states, hidden_states], dim=ctx.dim)- if ctx.is_cross_attn:- context = torch.cat([recv_context, context], dim=ctx.dim)- else:- context = hidden_states- q_proj_weight_grad = q_grad.transpose(-1, -2).matmul(hidden_states).sum(dim=0)- q_proj_bias_grad = (- q_grad.sum(dim=0).sum(dim=0) if q_proj_bias is not None else None- )- k_proj_weight_grad = k_grad.transpose(-1, -2).matmul(context).sum(dim=0)- k_proj_bias_grad = (- k_grad.sum(dim=0).sum(dim=0) if k_proj_bias is not None else None- )- v_proj_weight_grad = v_grad.transpose(-1, -2).matmul(context).sum(dim=0)- v_proj_bias_grad = (- v_grad.sum(dim=0).sum(dim=0) if v_proj_bias is not None else None- )-- # split grads- q_grad = q_grad.chunk(ctx.sp_size, dim=ctx.dim)[ctx.sp_rank].clone()- k_grad = k_grad.chunk(ctx.sp_size, dim=ctx.dim)[ctx.sp_rank].clone()- v_grad = v_grad.chunk(ctx.sp_size, dim=ctx.dim)[ctx.sp_rank].clone()-- if ctx.is_cross_attn:- hidden_states_grad = torch.matmul(q_grad, q_proj_weight)- context_grad = torch.matmul(k_grad, k_proj_weight) + torch.matmul(- v_grad, v_proj_weight- )- else:- hidden_states_grad = (- torch.matmul(q_grad, q_proj_weight)- + torch.matmul(k_grad, k_proj_weight)- + torch.matmul(v_grad, v_proj_weight)- )- context_grad = hidden_states_grad-- return (- hidden_states_grad,- context_grad,- q_proj_weight_grad,- q_proj_bias_grad,- k_proj_weight_grad,- k_proj_bias_grad,- v_proj_weight_grad,- v_proj_bias_grad,- None,- None,- None,- None,- )---def async_all_gather_proj_for_two(- hidden_states: torch.Tensor,- context: torch.Tensor,- q_proj_weight: torch.Tensor,- q_proj_bias: Optional[torch.Tensor],- k_proj_weight: torch.Tensor,- k_proj_bias: Optional[torch.Tensor],- v_proj_weight: torch.Tensor,- v_proj_bias: Optional[torch.Tensor],- dim: int,- process_group: dist.ProcessGroup,- sp_size: int,- sp_rank: int,-) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:- return AsyncAllGatherProjForTwo.apply(- hidden_states,- context,- q_proj_weight,- q_proj_bias,- k_proj_weight,- k_proj_bias,- v_proj_weight,- v_proj_bias,- dim,- process_group,- sp_size,- sp_rank,- )diff --git a/open_sora/utils/data.py b/open_sora/utils/data.pydeleted file mode 100644index a68b760..0000000--- a/open_sora/utils/data.py+++ /dev/null@@ -1,356 +0,0 @@-import os-from typing import List, Optional, Tuple, Union--import torch-import torch.nn as nn-import torch.nn.functional as F-from colossalai.utils import get_current_device-from datasets import Dataset as HFDataset-from datasets import dataset_dict, load_from_disk-from diffusers.models import AutoencoderKL-from torch.utils.data import ConcatDataset, Dataset-from torchvision.io import read_video-from transformers import AutoModel--DatasetType = Union[Dataset, ConcatDataset, dataset_dict.Dataset]-PathType = Union[str, os.PathLike]---def ceil_to_multiple(x: int, multiple: int) -> int:- m = x % multiple- if m == 0:- return x- return x + multiple - m---def video2col(video_4d: torch.Tensor, patch_size: int) -> torch.Tensor:- """- Convert a 4D video tensor to a 2D tensor where each row is a patch of the video.-- Args:- video_4d (torch.Tensor): A tensor of shape [T, C, H, W]- patch_size (int): The size of the patches.-- Returns:- torch.Tensor: A tensor of shape [S, C, P, P] where S is the number of patches and P is the patch size.- """- t, c, h, w = video_4d.shape- out = []- for y in range(0, h, patch_size):- for x in range(0, w, patch_size):- if y + patch_size > h or x + patch_size > w:- continue- patch = video_4d[:, :, y : y + patch_size, x : x + patch_size]- out.append(patch)- # [S, C, P, P]- return torch.stack(out, dim=1).view(-1, c, patch_size, patch_size)---def col2video(- patches: torch.Tensor, video_shape: Tuple[int, int, int, int]-) -> torch.Tensor:- """- Convert a 2D tensor of patches to a 4D video tensor.-- Args:- patches (torch.Tensor): A tensor of shape [S, C, P, P] where S is the number of patches and P is the patch size.- video_shape (Tuple[int, int, int, int]): The shape of the video tensor [T, C, H, W].-- Returns:- torch.Tensor: A tensor of shape [T, C, H, W].- """- t, c, h, w = video_shape- video = torch.empty(t, c, h, w, dtype=patches.dtype, device=patches.device)- patch_size = patches.shape[2]- num_x_patches = w // patch_size- patches = patches.view(t, -1, c, patch_size, patch_size)- for y in range(0, h, patch_size):- for x in range(0, w, patch_size):- if y + patch_size > h or x + patch_size > w:- continue- # [T, C, P, P]- patch = patches[:, (y // patch_size) * num_x_patches + x // patch_size]- video[:, :, y : y + patch_size, x : x + patch_size].copy_(patch)- return video---def pad_sequences(- sequences: List[torch.Tensor], pad_to_multiple: Optional[int] = None-) -> Tuple[torch.Tensor, torch.Tensor]:- """Pad a list of sequences.-- Args:- sequences (List[torch.Tensor]): Each sequence is a tensor of shape [T, ...].-- Returns:- Tuple[torch.Tensor, torch.Tensor]: Padded batch of sequences ([B, T, ...]) and padding mask ([B, T]).- """- max_len = max([sequence.shape[0] for sequence in sequences])- if pad_to_multiple is not None:- max_len = ceil_to_multiple(max_len, pad_to_multiple)- padded_sequences = [- F.pad(- sequence, [0] * (sequence.ndim - 1) * 2 + [0, max_len - sequence.shape[0]]- )- for sequence in sequences- ]- padded_sequences = torch.stack(padded_sequences, dim=0)- padding_mask = torch.zeros(- padded_sequences.shape[0],- padded_sequences.shape[1],- dtype=torch.int,- device=padded_sequences.device,- )- for i, sequence in enumerate(sequences):- padding_mask[i, : sequence.shape[0]] = 1- return padded_sequences, padding_mask---def patchify_batch(- videos: List[torch.Tensor], patch_size: int, pad_to_multiple: Optional[int] = None-) -> Tuple[torch.Tensor, torch.Tensor]:- """Patchify a batch of videos.-- Args:- videos (List[torch.Tensor]): A list of tensors of shape [T, C, H, W]- patch_size (int): The size of the patches.-- Returns:- Tuple[torch.Tensor, torch.Tensor]: Padded batch of patches ([B, S, C, P, P]) and padding mask ([B, S]).- """- video_patches = [video2col(video, patch_size) for video in videos]- return pad_sequences(video_patches, pad_to_multiple=pad_to_multiple)---def expand_mask_4d(q_mask: torch.Tensor, kv_mask: torch.Tensor) -> torch.Tensor:- """expand 2D mask to 4D mask-- Args:- q_mask (torch.Tensor): [B, Sq]- kv_mask (torch.Tensor): [B, Skv]-- Returns:- torch.Tensor: [B, 1, Sq, Skv]- """- q_len = q_mask.shape[1]- mask = kv_mask.unsqueeze(1).repeat(1, q_len, 1)- mask = mask * q_mask.unsqueeze(-1)- return mask.unsqueeze(1)---def make_batch(- samples: List[dict],- video_dir: str,- pad_to_multiple: Optional[int] = None,- use_pooled_text: bool = False,-) -> dict:- """Make a batch of samples.-- Args:- samples (List[dict]): A list of samples.-- Returns:- dict: A batch of samples.- """- videos = [- read_video(os.path.join(video_dir, sample["video_file"]), pts_unit="sec")[0]- for sample in samples- ]- texts = [sample["text_latent_states"] for sample in samples]- if use_pooled_text:- texts = torch.stack(texts, dim=0)- return {- "videos": videos,- "text_latent_states": texts,- }- texts, text_padding_mask = pad_sequences(texts, pad_to_multiple=pad_to_multiple)- return {- "videos": videos,- "text_latent_states": texts,- "text_padding_mask": text_padding_mask,- }---def normalize_video(video: torch.Tensor) -> torch.Tensor:- return video.float() / 255 - 0.5---def unnormalize_video(video: torch.Tensor) -> torch.Tensor:- return (video + 0.5) * 255---class VideoCompressor:- t_factor: int- h_w_factor: int- out_channels: int-- def encode(self, video: torch.Tensor) -> torch.Tensor:- """Encode a video.-- Args:- video (torch.Tensor): [T, H, W, C]-- Returns:- torch.Tensor: [T, C, H, W]- """- raise NotImplementedError-- def decode(self, latent: torch.Tensor) -> torch.Tensor:- """Decode a latent tensor.-- Args:- latent (torch.Tensor): [T, C, H, W]-- Returns:- torch.Tensor: [T, H, W, C]- """- raise NotImplementedError---class RawVideoCompressor(VideoCompressor):- t_factor = 1- h_w_factor = 1- out_channels = 3-- def encode(self, video: torch.Tensor) -> torch.Tensor:- # [T, H, W, C] -> [T, C, H, W]- return video.permute(0, 3, 1, 2).contiguous()-- def decode(self, latent: torch.Tensor) -> torch.Tensor:- # [T, C, H, W] -> [T, H, W, C]- return latent.permute(0, 2, 3, 1).contiguous()---class VqvaeVideoCompressor(VideoCompressor):- t_factor = 2- h_w_factor = 4-- def __init__(self, vqvae: nn.Module):- self.vqvae = vqvae- self.out_channels = vqvae.embedding_dim-- def encode(self, video: torch.Tensor) -> torch.Tensor:- # [T, H, W, C] -> [B, C, T, H, W]- video = video.permute(3, 0, 1, 2).unsqueeze(0)- latent_indices, embeddings = self.vqvae.encode(video, include_embeddings=True)- # [B, C, T, H, W] -> [T, C, H, W]- return embeddings.squeeze(0).permute(1, 0, 2, 3)-- def decode(self, latent: torch.Tensor) -> torch.Tensor:- # [T, C, H, W] -> [B, C, T, H, W]- latent = latent.permute(1, 0, 2, 3).unsqueeze(0)- video = self.vqvae.decode_from_embeddings(latent)- # [B, C, T, H, W] -> [T, H, W, C]- video = video.squeeze(0).permute(1, 2, 3, 0)- return video---class VaeVideoCompressor(VideoCompressor):- t_factor = 1- h_w_factor = 8- out_channels = 4-- def __init__(self, vae: nn.Module):- self.vae = vae-- def encode(self, video: torch.Tensor) -> torch.Tensor:- # [T, H, W, C] -> [T, C, H, W]- video = video.permute(0, 3, 1, 2)- return self.vae.encode(video).latent_dist.sample().mul_(0.18215)-- def decode(self, latent: torch.Tensor) -> torch.Tensor:- video = self.vae.decode(latent / 0.18215).sample- # [T, C, H, W] -> [T, H, W, C]- return video.permute(0, 2, 3, 1).contiguous()---def create_video_compressor(- compressor_type: str,- vqvae_path="hpcai-tech/vqvae",- vae_path="stabilityai/sd-vae-ft-mse",-) -> VideoCompressor:- if compressor_type == "raw":- return RawVideoCompressor()- if compressor_type == "vqvae":- vqvae = (- AutoModel.from_pretrained(vqvae_path, trust_remote_code=True)- .to(get_current_device())- .eval()- )- return VqvaeVideoCompressor(vqvae)- if compressor_type == "vae":- vae = AutoencoderKL.from_pretrained(vae_path).to(get_current_device()).eval()- return VaeVideoCompressor(vae)- raise ValueError(f"Unsupported video compressor type {compressor_type}")--[email protected]_grad()-def preprocess_batch(- batch: dict,- patch_size: int,- video_compressor: VideoCompressor,- device=None,- pad_to_multiple: Optional[int] = None,- model_arch: str = "cross-attn",-) -> dict:- if device is None:- device = get_current_device()- videos = []- for video in batch.pop("videos"):- video = video.to(device)- video = normalize_video(video)- video = video_compressor.encode(video)- videos.append(video)- video_latent_states, video_padding_mask = patchify_batch(- videos, patch_size, pad_to_multiple- )- batch["video_latent_states"] = video_latent_states- batch["video_padding_mask"] = video_padding_mask- if model_arch == "adaln":- batch["attention_mask"] = expand_mask_4d(video_padding_mask, video_padding_mask)- else:- text_padding_mask = batch.pop("text_padding_mask").to(device)- if model_arch == "cross-attn":- batch["attention_mask"] = expand_mask_4d(- video_padding_mask, text_padding_mask- )- else:- attention_mask = torch.cat([text_padding_mask, video_padding_mask], dim=1)- batch["attention_mask"] = expand_mask_4d(attention_mask, attention_mask)- batch["text_latent_states"] = batch["text_latent_states"].to(device)- return batch---def load_datasets(- dataset_paths: Union[PathType, List[PathType]], mode: str = "train"-) -> Optional[DatasetType]:- """- Load pre-tokenized dataset.- Each instance of dataset is a dictionary with- `{'input_ids': List[int], 'labels': List[int], sequence: str}` format.- """- mode_map = {"train": "train", "dev": "validation", "test": "test"}- assert mode in tuple(- mode_map- ), f"Unsupported mode {mode}, it must be in {tuple(mode_map)}"-- if isinstance(dataset_paths, (str, os.PathLike)):- dataset_paths = [dataset_paths]-- datasets = [] # `List[datasets.dataset_dict.Dataset]`- for ds_path in dataset_paths:- ds_path = os.path.abspath(ds_path)- assert os.path.exists(ds_path), f"Not existed file path {ds_path}"- ds_dict = load_from_disk(- dataset_path=ds_path, keep_in_memory=False- ).with_format("torch")- if isinstance(ds_dict, HFDataset):- datasets.append(ds_dict)- else:- if mode_map[mode] in ds_dict:- datasets.append(ds_dict[mode_map[mode]])- if len(datasets) == 0:- return None- if len(datasets) == 1:- return datasets.pop()- return ConcatDataset(datasets=datasets)diff --git a/open_sora/utils/plugin.py b/open_sora/utils/plugin.pydeleted file mode 100644index f687b41..0000000--- a/open_sora/utils/plugin.py+++ /dev/null@@ -1,102 +0,0 @@-import random-from typing import Optional--import numpy as np-import torch-from colossalai.booster.plugin import LowLevelZeroPlugin-from colossalai.cluster import ProcessGroupMesh-from torch.utils.data import DataLoader-from torch.utils.data.distributed import DistributedSampler--DP_AXIS, SP_AXIS = 0, 1---class ZeroSeqParallelPlugin(LowLevelZeroPlugin):- def __init__(- self,- sp_size: int = 1,- stage: int = 2,- precision: str = "fp16",- initial_scale: float = 2**32,- min_scale: float = 1,- growth_factor: float = 2,- backoff_factor: float = 0.5,- growth_interval: int = 1000,- hysteresis: int = 2,- max_scale: float = 2**32,- max_norm: float = 0.0,- norm_type: float = 2.0,- reduce_bucket_size_in_m: int = 12,- communication_dtype: Optional[torch.dtype] = None,- overlap_communication: bool = True,- cpu_offload: bool = False,- master_weights: bool = True,- verbose: bool = False,- ) -> None:- super().__init__(- stage=stage,- precision=precision,- initial_scale=initial_scale,- min_scale=min_scale,- growth_factor=growth_factor,- backoff_factor=backoff_factor,- growth_interval=growth_interval,- hysteresis=hysteresis,- max_scale=max_scale,- max_norm=max_norm,- norm_type=norm_type,- reduce_bucket_size_in_m=reduce_bucket_size_in_m,- communication_dtype=communication_dtype,- overlap_communication=overlap_communication,- cpu_offload=cpu_offload,- master_weights=master_weights,- verbose=verbose,- )- self.sp_size = sp_size- assert self.world_size % sp_size == 0, "world_size must be divisible by sp_size"- self.dp_size = self.world_size // sp_size- self.pg_mesh = ProcessGroupMesh(self.dp_size, self.sp_size)- self.dp_group = self.pg_mesh.get_group_along_axis(DP_AXIS)- self.sp_group = self.pg_mesh.get_group_along_axis(SP_AXIS)- self.dp_rank = self.pg_mesh.coordinate(DP_AXIS)- self.sp_rank = self.pg_mesh.coordinate(SP_AXIS)-- def __del__(self):- """Destroy the prcess groups in ProcessGroupMesh"""- self.pg_mesh.destroy_mesh_process_groups()-- def prepare_dataloader(- self,- dataset,- batch_size,- shuffle=False,- seed=1024,- drop_last=False,- pin_memory=False,- num_workers=0,- distributed_sampler_cls=None,- **kwargs- ):- _kwargs = kwargs.copy()- distributed_sampler_cls = distributed_sampler_cls or DistributedSampler- sampler = distributed_sampler_cls(- dataset, num_replicas=self.dp_size, rank=self.dp_rank, shuffle=shuffle- )-- # Deterministic dataloader- def seed_worker(worker_id):- worker_seed = seed- np.random.seed(worker_seed)- torch.manual_seed(worker_seed)- random.seed(worker_seed)-- return DataLoader(- dataset,- batch_size=batch_size,- sampler=sampler,- worker_init_fn=seed_worker,- drop_last=drop_last,- pin_memory=pin_memory,- num_workers=num_workers,- **_kwargs,- )diff --git a/requirements.txt b/requirements.txtdeleted file mode 100644index 3a8077d..0000000--- a/requirements.txt+++ /dev/null@@ -1,9 +0,0 @@-torch-torchvision-datasets-transformers-av-tensorboard-git+https://github.com/hpcaitech/ColossalAI.git@main#egg=colossalai-timm-diffusers==0.24.0\ No newline at end of filediff --git a/sample.py b/sample.pydeleted file mode 100644index 67aec4b..0000000--- a/sample.py+++ /dev/null@@ -1,150 +0,0 @@-# Copyright (c) Meta Platforms, Inc. and affiliates.-# All rights reserved.--# This source code is licensed under the license found in the-# LICENSE file in the root directory of this source tree.--"""-Sample new images from a pre-trained DiT.-"""-import torch--torch.backends.cuda.matmul.allow_tf32 = True-torch.backends.cudnn.allow_tf32 = True-import argparse--from colossalai.utils import get_current_device-from torchvision.io import write_video-from transformers import AutoTokenizer, CLIPTextModel--from open_sora.diffusion import create_diffusion-from open_sora.modeling import DiT_models-from open_sora.modeling.dit import SUPPORTED_MODEL_ARCH-from open_sora.utils.data import col2video, create_video_compressor, unnormalize_video---def main(args):- # Setup PyTorch:- torch.manual_seed(args.seed)- torch.set_grad_enabled(False)- device = get_current_device()-- video_compressor = create_video_compressor(args.compressor)- model_kwargs = {- "in_channels": video_compressor.out_channels,- "model_arch": args.model_arch,- }-- text_model = CLIPTextModel.from_pretrained(args.text_model).to(device).eval()- tokenizer = AutoTokenizer.from_pretrained(args.text_model)-- model = DiT_models[args.model](**model_kwargs).to(device).eval()- patch_size = model.patch_size- model.load_state_dict(torch.load(args.ckpt))- diffusion = create_diffusion(str(args.num_sampling_steps))-- # Create sampling noise:- text_inputs = tokenizer(args.text, return_tensors="pt")- text_inputs = {k: v.to(device) for k, v in text_inputs.items()}- text_latent_states = text_model(**text_inputs)- if args.model_arch == "adaln":- text_latent_states = text_latent_states.pooler_output- else:- text_latent_states = text_latent_states.last_hidden_state-- num_frames = args.fps * args.sec- z = torch.randn(- 1,- (args.height // patch_size // video_compressor.h_w_factor)- * (args.width // patch_size // video_compressor.h_w_factor)- * (num_frames // video_compressor.t_factor),- video_compressor.out_channels,- patch_size,- patch_size,- device=device,- )-- # Setup classifier-free guidance:- model_kwargs = {}- if not args.disable_cfg:- z = torch.cat([z, z], 0)- model_kwargs["text_latent_states"] = torch.cat(- [text_latent_states, torch.zeros_like(text_latent_states)], 0- )- model_kwargs["cfg_scale"] = args.cfg_scale- else:- model_kwargs["text_latent_states"] = text_latent_states- context_len = (- text_latent_states.shape[1] if args.model_arch == "cross-attn" else z.shape[1]- )- model_kwargs["attention_mask"] = torch.ones(- z.shape[0],- 1,- z.shape[1],- context_len,- device=device,- dtype=torch.int,- )-- # Sample images:- samples = diffusion.p_sample_loop(- model if args.disable_cfg else model.forward_with_cfg,- z.shape,- z,- clip_denoised=False,- model_kwargs=model_kwargs,- progress=True,- device=device,- )- if not args.disable_cfg:- samples, _ = samples.chunk(2, dim=0) # Remove null class samples- samples = col2video(- samples.squeeze(),- (- num_frames // video_compressor.t_factor,- video_compressor.out_channels,- args.height // video_compressor.h_w_factor,- args.width // video_compressor.h_w_factor,- ),- )- samples = video_compressor.decode(samples)- samples = unnormalize_video(samples).to(torch.uint8)-- write_video("sample.mp4", samples.cpu(), args.fps)---if __name__ == "__main__":- parser = argparse.ArgumentParser()- parser.add_argument(- "-m", "--model", type=str, choices=list(DiT_models.keys()), default="DiT-S/8"- )- parser.add_argument(- "-x", "--model_arch", choices=SUPPORTED_MODEL_ARCH, default="cross-attn"- )- parser.add_argument(- "--text",- type=str,- default="a cartoon animals runs through an ice cave in a video game",- )- parser.add_argument("--cfg-scale", type=float, default=4.0)- parser.add_argument("--num-sampling-steps", type=int, default=250)- parser.add_argument("--seed", type=int, default=0)- parser.add_argument(- "--ckpt",- type=str,- required=True,- help="Optional path to a DiT checkpoint (default: auto-download a pre-trained DiT-XL/2 model).",- )- parser.add_argument(- "-c", "--compressor", choices=["raw", "vqvae", "vae"], default="raw"- )- parser.add_argument(- "--text_model", type=str, default="openai/clip-vit-base-patch32"- )- parser.add_argument("--width", type=int, default=480)- parser.add_argument("--height", type=int, default=320)- parser.add_argument("--fps", type=int, default=15)- parser.add_argument("--sec", type=int, default=8)- parser.add_argument("--disable-cfg", action="store_true", default=False)- args = parser.parse_args()- main(args)diff --git a/scripts/data/collate_msr_vtt_dataset.py b/scripts/data/collate_msr_vtt_dataset.pydeleted file mode 100644index 04f63cb..0000000--- a/scripts/data/collate_msr_vtt_dataset.py+++ /dev/null@@ -1,182 +0,0 @@-import argparse-import json-import multiprocessing-import os-import shutil-import warnings-from typing import Dict, Tuple--from tqdm import tqdm--DEFAULT_TYPES = ["train", "val", "test"]---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument("-d", "--data-path", type=str, help="The path to the MSR-VTT dataset")- parser.add_argument("-o", "--output-path", type=str, help="The output to the collated MSR-VTT dataset")- return parser.parse_args()---def get_annotations(root_path: str):- """- Get the annotation data from the MSR-VTT dataset. The annotations are in the format of:-- {- "annotations": [- {- "image_id": "video1",- "caption": "some- }- ]- }-- Args:- root_path (str): The root path to the MSR-VTT dataset- """- annotation_json_file = os.path.join(root_path, "annotation/MSR_VTT.json")- with open(annotation_json_file, "r") as f:- data = json.load(f)- return data---def get_video_list(root_path: str, dataset_type: str):- """- Get the list of videos in the dataset split.-- Args:- root_path (str): The root path to the MSR-VTT dataset- dataset_type (str): The dataset split type. It should be one of "train", "val", or "test"- """- assert dataset_type in DEFAULT_TYPES, f"Expected the dataset type to be in {DEFAULT_TYPES}, but got {dataset_type}"- dataset_file_path = os.path.join(root_path, f"structured-symlinks/{dataset_type}_list_full.txt")- with open(dataset_file_path, "r") as f:- video_list = f.readlines()- video_list = [x.strip() for x in video_list]- return video_list---def copy_video(video_id: str, root_path: str, output_path: str, dataset_type: str):- """- Copy the video from the source path to the destination path.-- Args:- video_id (str): The video id- root_path (str): The root path to the MSR-VTT dataset- output_path (str): The output path to the collated MSR-VTT dataset- dataset_type (str): The dataset split type. It should be one of "train", "val", or "test"- """- assert dataset_type in DEFAULT_TYPES, f"Expected the dataset type to be in {DEFAULT_TYPES}, but got {dataset_type}"- src_file = os.path.join(root_path, f"videos/all/{video_id}.mp4")- dst_folder = os.path.join(output_path, f"{dataset_type}/videos")- dst_file = os.path.join(dst_folder, f"{video_id}.mp4")- os.makedirs(dst_folder, exist_ok=True)-- # create symlink- assert os.path.isfile(src_file), f"Expected the source file {src_file} to exist"- if not os.path.islink(dst_file):- shutil.copy(src_file, dst_file)---def get_annotation_file_path(output_path: str, dataset_type: str):- file_path = os.path.join(output_path, f"{dataset_type}/annotations.json")- return file_path---def collate_annotation_files(- annotations: Dict,- root_path: str,- output_path: str,-):- """- Collate the video and caption data into a single folder.-- Args:- annotations (Dict): The annotations data- root_path (str): The root path to the MSR-VTT dataset- output_path (str): The output path to the collated MSR-VTT dataset- """- # get all video list- train_video_list = get_video_list(root_path, "train")- val_video_list = get_video_list(root_path, "val")- test_video_list = get_video_list(root_path, "test")-- # iterate over annotations- collated_train_data = []- collated_val_data = []- collated_test_data = []-- print("Collating annotations files")-- for anno in tqdm(annotations["annotations"]):- video_id = anno["image_id"]- caption = anno["caption"]-- obj = {"file": f"{video_id}.mp4", "captions": [caption]}-- if video_id in train_video_list:- collated_train_data.append(obj)- elif video_id in val_video_list:- collated_val_data.append(obj)- elif video_id in test_video_list:- collated_test_data.append(obj)- else:- warnings.warn(f"Video {video_id} not found in any of the dataset splits")-- def _save_caption_files(obj, dataset_type):- dst_file = get_annotation_file_path(output_path, dataset_type)- os.makedirs(os.path.dirname(dst_file), exist_ok=True)- with open(dst_file, "w") as f:- json.dump(obj, f, indent=4)-- _save_caption_files(collated_train_data, "train")- _save_caption_files(collated_val_data, "val")- _save_caption_files(collated_test_data, "test")---def copy_file(path_pair: Tuple[str, str]):- src_path, dst_path = path_pair- shutil.copyfile(src_path, dst_path)---def copy_videos(root_path: str, output_path: str, num_workers: int = 8):- """- Batch copy the video files to the output path.-- Args:- root_path (str): The root path to the MSR-VTT dataset- output_path (str): The output path to the collated MSR-VTT dataset- num_workers (int): The number of workers to use for the copy operation- """- pool = multiprocessing.Pool(num_workers)-- for dataset_type in DEFAULT_TYPES:- print(f"Copying videos for the {dataset_type} dataset")- annotation_file_path = get_annotation_file_path(output_path, dataset_type)- output_video_folder_path = os.path.join(output_path, f"{dataset_type}/videos")- os.makedirs(output_video_folder_path, exist_ok=True)-- with open(annotation_file_path, "r") as f:- annotation_data = json.load(f)-- video_ids = [obj["file"] for obj in annotation_data]- unique_video_ids = list(set(video_ids))-- path_pairs = [- (os.path.join(root_path, f"videos/all/{video_id}"), os.path.join(output_video_folder_path, video_id))- for video_id in unique_video_ids- ]-- for _ in tqdm(pool.imap_unordered(copy_file, path_pairs), total=len(path_pairs)):- pass---def main():- args = parse_args()- annotations = get_annotations(args.data_path)- collate_annotation_files(annotations, args.data_path, args.output_path)- copy_videos(args.data_path, args.output_path)---if __name__ == "__main__":- main()diff --git a/scripts/data/download_msr_vtt_dataset.sh b/scripts/data/download_msr_vtt_dataset.shdeleted file mode 100644index 141a7d6..0000000--- a/scripts/data/download_msr_vtt_dataset.sh+++ /dev/null@@ -1,11 +0,0 @@-#!/usr/bin/env bash--# get root dir-FOLDER_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"-ROOT_DIR=$FOLDER_DIR/../..--# download at root dir-cd $ROOT_DIR-mkdir -p dataset && cd ./dataset-wget https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip-unzip MSRVTT.zipdiff --git a/scripts/data/preprocess_data.py b/scripts/data/preprocess_data.pydeleted file mode 100644index bc0dfbb..0000000--- a/scripts/data/preprocess_data.py+++ /dev/null@@ -1,135 +0,0 @@-import argparse-import math-import os--import torch-from datasets import load_dataset-from torchvision.io import read_video-from transformers import AutoTokenizer, CLIPTextModel--EMPTY_SAMPLE = {"video_file": [], "text_latent_states": []}---def process_text(text, tokenizer, text_model, use_pooled_text):- inputs = tokenizer(text, padding=True, return_tensors="pt")- inputs = {k: v.cuda() for k, v in inputs.items()}- outputs = text_model(**inputs)- if use_pooled_text:- return list(outputs.pooler_output.cpu().unbind(0))- output_states = []- for i, x in enumerate(outputs.last_hidden_state):- valid_x = x[inputs["attention_mask"][i].bool()]- output_states.append(valid_x.cpu())- return output_states--[email protected]_grad()-def process_item(item, video_dir, tokenizer, text_model, use_pooled_text):- video_path = os.path.join(video_dir, item["file"])- video = read_video(video_path, pts_unit="sec")[0]- if video.size(0) > 600:- return EMPTY_SAMPLE- text_latent_states = process_text(- item["captions"], tokenizer, text_model, use_pooled_text- )- torch.cuda.empty_cache()- return {- "video_file": [item["file"]] * len(text_latent_states),- "text_latent_states": text_latent_states,- }---def process_batch(batch, video_dir, tokenizer, text_model, use_pooled_text):- item = {"file": batch["file"][0], "captions": batch["captions"][0]}- return process_item(item, video_dir, tokenizer, text_model, use_pooled_text)---def process_dataset(- captions_file,- video_dir,- output_dir,- num_spliced_dataset_bins=10,- text_model="openai/clip-vit-base-patch32",- use_pooled_text=False,-):- tokenizer = AutoTokenizer.from_pretrained(text_model)- text_model = CLIPTextModel.from_pretrained(text_model).cuda().eval()-- if not os.path.exists(output_dir):- os.makedirs(output_dir)-- # Prepare to data splitting.- train_splits = []- split_interval = math.ceil(100 / num_spliced_dataset_bins)- for i in range(0, 100, split_interval):- start = i- end = i + split_interval- if end > 100:- end = 100- train_splits.append(f"train[{start}%:{end}%]")-- ds = load_dataset(- "json",- data_files=captions_file,- keep_in_memory=False,- split=train_splits,- num_proc=1,- )-- for i, part_ds in enumerate(ds):- print(f"Processing part {i+1}/{len(ds)}")- part_ds = part_ds.with_format("torch")- part_ds = part_ds.map(- process_batch,- fn_kwargs={- "video_dir": video_dir,- "tokenizer": tokenizer,- "text_model": text_model,- "use_pooled_text": use_pooled_text,- },- batched=True,- batch_size=1,- keep_in_memory=False,- remove_columns=part_ds.column_names,- )- output_path = os.path.join(output_dir, f"part-{i:05d}")- part_ds.save_to_disk(output_path)---if __name__ == "__main__":- parser = argparse.ArgumentParser(description="Preprocess data")- parser.add_argument(- "-c",- "--captions-file",- type=str,- help="Path to the captions file. It should be a JSON file or a JSONL file",- )- parser.add_argument(- "-v", "--video-dir", type=str, help="Path to the video directory"- )- parser.add_argument(- "-o", "--output_dir", type=str, help="Path to the output directory"- )- parser.add_argument(- "-n",- "--num_spliced_dataset_bins",- type=int,- default=10,- help="Number of bins for spliced dataset",- )- parser.add_argument(- "--text_model",- type=str,- default="openai/clip-vit-base-patch32",- help="CLIP text model",- )- parser.add_argument("--use_pooled_text", action="store_true", default=False)- args = parser.parse_args()- process_dataset(- args.captions_file,- args.video_dir,- args.output_dir,- args.num_spliced_dataset_bins,- args.text_model,- args.use_pooled_text,- )diff --git a/scripts/sample/latte/pipeline_videogen.py b/scripts/sample/latte/pipeline_videogen.pydeleted file mode 100644index 8acd627..0000000--- a/scripts/sample/latte/pipeline_videogen.py+++ /dev/null@@ -1,792 +0,0 @@-# All rights reserved.-# Copyright 2024 Vchitect/Latte-# Licensed under the Apache License, Version 2.0 (the "License");-# you may not use this file except in compliance with the License.-# You may obtain a copy of the License at-#-# http://www.apache.org/licenses/LICENSE-2.0-#-# Unless required by applicable law or agreed to in writing, software-# distributed under the License is distributed on an "AS IS" BASIS,-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-# See the License for the specific language governing permissions and-# limitations under the License.-#-# copied from https://github.com/Vchitect/Latte/blob/main/sample/pipeline_videogen.py--import html-import inspect-import re-import urllib.parse as ul-from dataclasses import dataclass-from typing import Callable, List, Optional, Tuple, Union--import einops-import torch-from diffusers.image_processor import VaeImageProcessor-from diffusers.models import AutoencoderKL, Transformer2DModel-from diffusers.pipelines.pipeline_utils import DiffusionPipeline-from diffusers.schedulers import DPMSolverMultistepScheduler-from diffusers.utils import (- BACKENDS_MAPPING,- BaseOutput,- is_bs4_available,- is_ftfy_available,- logging,- replace_example_docstring,-)-from diffusers.utils.torch_utils import randn_tensor-from transformers import T5EncoderModel, T5Tokenizer--logger = logging.get_logger(__name__) # pylint: disable=invalid-name--if is_bs4_available():- from bs4 import BeautifulSoup--if is_ftfy_available():- import ftfy---EXAMPLE_DOC_STRING = """- Examples:- ```py- >>> import torch- >>> from diffusers import PixArtAlphaPipeline-- >>> # You can replace the checkpoint id with "PixArt-alpha/PixArt-XL-2-512x512" too.- >>> pipe = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-1024-MS", torch_dtype=torch.float16)- >>> # Enable memory optimizations.- >>> pipe.enable_model_cpu_offload()-- >>> prompt = "A small cactus with a happy face in the Sahara desert."- >>> image = pipe(prompt).images[0]- ```-"""---@dataclass-class VideoPipelineOutput(BaseOutput):- video: torch.Tensor---class VideoGenPipeline(DiffusionPipeline):- r"""- Pipeline for text-to-image generation using PixArt-Alpha.-- This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the- library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)-- Args:- vae ([`AutoencoderKL`]):- Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations.- text_encoder ([`T5EncoderModel`]):- Frozen text-encoder. PixArt-Alpha uses- [T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5EncoderModel), specifically the- [t5-v1_1-xxl](https://huggingface.co/PixArt-alpha/PixArt-alpha/tree/main/t5-v1_1-xxl) variant.- tokenizer (`T5Tokenizer`):- Tokenizer of class- [T5Tokenizer](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Tokenizer).- transformer ([`Transformer2DModel`]):- A text conditioned `Transformer2DModel` to denoise the encoded image latents.- scheduler ([`SchedulerMixin`]):- A scheduler to be used in combination with `transformer` to denoise the encoded image latents.- """- bad_punct_regex = re.compile(- r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"- ) # noqa-- _optional_components = ["tokenizer", "text_encoder"]- model_cpu_offload_seq = "text_encoder->transformer->vae"-- def __init__(- self,- tokenizer: T5Tokenizer,- text_encoder: T5EncoderModel,- vae: AutoencoderKL,- transformer: Transformer2DModel,- scheduler: DPMSolverMultistepScheduler,- ):- super().__init__()-- self.register_modules(- tokenizer=tokenizer, text_encoder=text_encoder, vae=vae, transformer=transformer, scheduler=scheduler- )-- self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1)- self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor)-- # Adapted from https://github.com/PixArt-alpha/PixArt-alpha/blob/master/diffusion/model/utils.py- def mask_text_embeddings(self, emb, mask):- if emb.shape[0] == 1:- keep_index = mask.sum().item()- return emb[:, :, :keep_index, :], keep_index # 1, 120, 4096 -> 1 7 4096- else:- masked_feature = emb * mask[:, None, :, None] # 1 120 4096- return masked_feature, emb.shape[2]-- # Adapted from diffusers.pipelines.deepfloyd_if.pipeline_if.encode_prompt- def encode_prompt(- self,- prompt: Union[str, List[str]],- do_classifier_free_guidance: bool = True,- negative_prompt: str = "",- num_images_per_prompt: int = 1,- device: Optional[torch.device] = None,- prompt_embeds: Optional[torch.FloatTensor] = None,- negative_prompt_embeds: Optional[torch.FloatTensor] = None,- clean_caption: bool = False,- mask_feature: bool = True,- ):- r"""- Encodes the prompt into text encoder hidden states.-- Args:- prompt (`str` or `List[str]`, *optional*):- prompt to be encoded- negative_prompt (`str` or `List[str]`, *optional*):- The prompt not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`- instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). For- PixArt-Alpha, this should be "".- do_classifier_free_guidance (`bool`, *optional*, defaults to `True`):- whether to use classifier free guidance or not- num_images_per_prompt (`int`, *optional*, defaults to 1):- number of images that should be generated per prompt- device: (`torch.device`, *optional*):- torch device to place the resulting embeddings on- prompt_embeds (`torch.FloatTensor`, *optional*):- Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not- provided, text embeddings will be generated from `prompt` input argument.- negative_prompt_embeds (`torch.FloatTensor`, *optional*):- Pre-generated negative text embeddings. For PixArt-Alpha, it's should be the embeddings of the ""- string.- clean_caption (bool, defaults to `False`):- If `True`, the function will preprocess and clean the provided caption before encoding.- mask_feature: (bool, defaults to `True`):- If `True`, the function will mask the text embeddings.- """- embeds_initially_provided = prompt_embeds is not None and negative_prompt_embeds is not None-- if device is None:- device = self._execution_device-- if prompt is not None and isinstance(prompt, str):- batch_size = 1- elif prompt is not None and isinstance(prompt, list):- batch_size = len(prompt)- else:- batch_size = prompt_embeds.shape[0]-- # See Section 3.1. of the paper.- max_length = 120-- if prompt_embeds is None:- prompt = self._text_preprocessing(prompt, clean_caption=clean_caption)- text_inputs = self.tokenizer(- prompt,- padding="max_length",- max_length=max_length,- truncation=True,- return_attention_mask=True,- add_special_tokens=True,- return_tensors="pt",- )- text_input_ids = text_inputs.input_ids- untruncated_ids = self.tokenizer(prompt, padding="longest", return_tensors="pt").input_ids-- if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(- text_input_ids, untruncated_ids- ):- removed_text = self.tokenizer.batch_decode(untruncated_ids[:, max_length - 1 : -1])- logger.warning(- "The following part of your input was truncated because CLIP can only handle sequences up to"- f" {max_length} tokens: {removed_text}"- )-- attention_mask = text_inputs.attention_mask.to(device)- prompt_embeds_attention_mask = attention_mask-- prompt_embeds = self.text_encoder(text_input_ids.to(device), attention_mask=attention_mask)- prompt_embeds = prompt_embeds[0]- else:- prompt_embeds_attention_mask = torch.ones_like(prompt_embeds)-- if self.text_encoder is not None:- dtype = self.text_encoder.dtype- elif self.transformer is not None:- dtype = self.transformer.dtype- else:- dtype = None-- prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)-- bs_embed, seq_len, _ = prompt_embeds.shape- # duplicate text embeddings and attention mask for each generation per prompt, using mps friendly method- prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)- prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)- prompt_embeds_attention_mask = prompt_embeds_attention_mask.view(bs_embed, -1)- prompt_embeds_attention_mask = prompt_embeds_attention_mask.repeat(num_images_per_prompt, 1)-- # get unconditional embeddings for classifier free guidance- if do_classifier_free_guidance and negative_prompt_embeds is None:- uncond_tokens = [negative_prompt] * batch_size- uncond_tokens = self._text_preprocessing(uncond_tokens, clean_caption=clean_caption)- max_length = prompt_embeds.shape[1]- uncond_input = self.tokenizer(- uncond_tokens,- padding="max_length",- max_length=max_length,- truncation=True,- return_attention_mask=True,- add_special_tokens=True,- return_tensors="pt",- )- attention_mask = uncond_input.attention_mask.to(device)-- negative_prompt_embeds = self.text_encoder(- uncond_input.input_ids.to(device),- attention_mask=attention_mask,- )- negative_prompt_embeds = negative_prompt_embeds[0]-- if do_classifier_free_guidance:- # duplicate unconditional embeddings for each generation per prompt, using mps friendly method- seq_len = negative_prompt_embeds.shape[1]-- negative_prompt_embeds = negative_prompt_embeds.to(dtype=dtype, device=device)-- negative_prompt_embeds = negative_prompt_embeds.repeat(1, num_images_per_prompt, 1)- negative_prompt_embeds = negative_prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)-- # For classifier free guidance, we need to do two forward passes.- # Here we concatenate the unconditional and text embeddings into a single batch- # to avoid doing two forward passes- else:- negative_prompt_embeds = None-- # print(prompt_embeds.shape) # 1 120 4096- # print(negative_prompt_embeds.shape) # 1 120 4096-- # Perform additional masking.- if mask_feature and not embeds_initially_provided:- prompt_embeds = prompt_embeds.unsqueeze(1)- masked_prompt_embeds, keep_indices = self.mask_text_embeddings(prompt_embeds, prompt_embeds_attention_mask)- masked_prompt_embeds = masked_prompt_embeds.squeeze(1)- masked_negative_prompt_embeds = (- negative_prompt_embeds[:, :keep_indices, :] if negative_prompt_embeds is not None else None- )-- # import torch.nn.functional as F-- # padding = (0, 0, 0, 113) # (左, 右, 下, 上)- # masked_prompt_embeds_ = F.pad(masked_prompt_embeds, padding, "constant", 0)- # masked_negative_prompt_embeds_ = F.pad(masked_negative_prompt_embeds, padding, "constant", 0)-- # print(masked_prompt_embeds == masked_prompt_embeds_[:, :masked_negative_prompt_embeds.shape[1], ...])-- return masked_prompt_embeds, masked_negative_prompt_embeds- # return masked_prompt_embeds_, masked_negative_prompt_embeds_-- return prompt_embeds, negative_prompt_embeds-- # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.prepare_extra_step_kwargs- def prepare_extra_step_kwargs(self, generator, eta):- # prepare extra kwargs for the scheduler step, since not all schedulers have the same signature- # eta (η) is only used with the DDIMScheduler, it will be ignored for other schedulers.- # eta corresponds to η in DDIM paper: https://arxiv.org/abs/2010.02502- # and should be between [0, 1]-- accepts_eta = "eta" in set(inspect.signature(self.scheduler.step).parameters.keys())- extra_step_kwargs = {}- if accepts_eta:- extra_step_kwargs["eta"] = eta-- # check if the scheduler accepts generator- accepts_generator = "generator" in set(inspect.signature(self.scheduler.step).parameters.keys())- if accepts_generator:- extra_step_kwargs["generator"] = generator- return extra_step_kwargs-- def check_inputs(- self,- prompt,- height,- width,- negative_prompt,- callback_steps,- prompt_embeds=None,- negative_prompt_embeds=None,- ):- if height % 8 != 0 or width % 8 != 0:- raise ValueError(f"`height` and `width` have to be divisible by 8 but are {height} and {width}.")-- if (callback_steps is None) or (- callback_steps is not None and (not isinstance(callback_steps, int) or callback_steps <= 0)- ):- raise ValueError(- f"`callback_steps` has to be a positive integer but is {callback_steps} of type"- f" {type(callback_steps)}."- )-- if prompt is not None and prompt_embeds is not None:- raise ValueError(- f"Cannot forward both `prompt`: {prompt} and `prompt_embeds`: {prompt_embeds}. Please make sure to"- " only forward one of the two."- )- elif prompt is None and prompt_embeds is None:- raise ValueError(- "Provide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined."- )- elif prompt is not None and (not isinstance(prompt, str) and not isinstance(prompt, list)):- raise ValueError(f"`prompt` has to be of type `str` or `list` but is {type(prompt)}")-- if prompt is not None and negative_prompt_embeds is not None:- raise ValueError(- f"Cannot forward both `prompt`: {prompt} and `negative_prompt_embeds`:"- f" {negative_prompt_embeds}. Please make sure to only forward one of the two."- )-- if negative_prompt is not None and negative_prompt_embeds is not None:- raise ValueError(- f"Cannot forward both `negative_prompt`: {negative_prompt} and `negative_prompt_embeds`:"- f" {negative_prompt_embeds}. Please make sure to only forward one of the two."- )-- if prompt_embeds is not None and negative_prompt_embeds is not None:- if prompt_embeds.shape != negative_prompt_embeds.shape:- raise ValueError(- "`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but"- f" got: `prompt_embeds` {prompt_embeds.shape} != `negative_prompt_embeds`"- f" {negative_prompt_embeds.shape}."- )-- # Copied from diffusers.pipelines.deepfloyd_if.pipeline_if.IFPipeline._text_preprocessing- def _text_preprocessing(self, text, clean_caption=False):- if clean_caption and not is_bs4_available():- logger.warn(BACKENDS_MAPPING["bs4"][-1].format("Setting `clean_caption=True`"))- logger.warn("Setting `clean_caption` to False...")- clean_caption = False-- if clean_caption and not is_ftfy_available():- logger.warn(BACKENDS_MAPPING["ftfy"][-1].format("Setting `clean_caption=True`"))- logger.warn("Setting `clean_caption` to False...")- clean_caption = False-- if not isinstance(text, (tuple, list)):- text = [text]-- def process(text: str):- if clean_caption:- text = self._clean_caption(text)- text = self._clean_caption(text)- else:- text = text.lower().strip()- return text-- return [process(t) for t in text]-- # Copied from diffusers.pipelines.deepfloyd_if.pipeline_if.IFPipeline._clean_caption- def _clean_caption(self, caption):- caption = str(caption)- caption = ul.unquote_plus(caption)- caption = caption.strip().lower()- caption = re.sub("<person>", "person", caption)- # urls:- caption = re.sub(- r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa- "",- caption,- ) # regex for urls- caption = re.sub(- r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa- "",- caption,- ) # regex for urls- # html:- caption = BeautifulSoup(caption, features="html.parser").text-- # @<nickname>- caption = re.sub(r"@[\w\d]+\b", "", caption)-- # 31C0—31EF CJK Strokes- # 31F0—31FF Katakana Phonetic Extensions- # 3200—32FF Enclosed CJK Letters and Months- # 3300—33FF CJK Compatibility- # 3400—4DBF CJK Unified Ideographs Extension A- # 4DC0—4DFF Yijing Hexagram Symbols- # 4E00—9FFF CJK Unified Ideographs- caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)- caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)- caption = re.sub(r"[\u3200-\u32ff]+", "", caption)- caption = re.sub(r"[\u3300-\u33ff]+", "", caption)- caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)- caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)- caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)- #######################################################-- # все виды тире / all types of dash --> "-"- caption = re.sub(- r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa- "-",- caption,- )-- # кавычки к одному стандарту- caption = re.sub(r"[`´«»“”¨]", '"', caption)- caption = re.sub(r"[‘’]", "'", caption)-- # "- caption = re.sub(r""?", "", caption)- # &- caption = re.sub(r"&", "", caption)-- # ip adresses:- caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)-- # article ids:- caption = re.sub(r"\d:\d\d\s+$", "", caption)-- # \n- caption = re.sub(r"\\n", " ", caption)-- # "#123"- caption = re.sub(r"#\d{1,3}\b", "", caption)- # "#12345.."- caption = re.sub(r"#\d{5,}\b", "", caption)- # "123456.."- caption = re.sub(r"\b\d{6,}\b", "", caption)- # filenames:- caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)-- #- caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""- caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""-- caption = re.sub(self.bad_punct_regex, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT- caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "-- # this-is-my-cute-cat / this_is_my_cute_cat- regex2 = re.compile(r"(?:\-|\_)")- if len(re.findall(regex2, caption)) > 3:- caption = re.sub(regex2, " ", caption)-- caption = ftfy.fix_text(caption)- caption = html.unescape(html.unescape(caption))-- caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640- caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc- caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231-- caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)- caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)- caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)- caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)- caption = re.sub(r"\bpage\s+\d+\b", "", caption)-- caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...-- caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)-- caption = re.sub(r"\b\s+\:\s+", r": ", caption)- caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)- caption = re.sub(r"\s+", " ", caption)-- caption.strip()-- caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)- caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)- caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)- caption = re.sub(r"^\.\S+$", "", caption)-- return caption.strip()-- # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.prepare_latents- def prepare_latents(- self, batch_size, num_channels_latents, video_length, height, width, dtype, device, generator, latents=None- ):- shape = (- batch_size,- num_channels_latents,- video_length,- height // self.vae_scale_factor,- width // self.vae_scale_factor,- )- if isinstance(generator, list) and len(generator) != batch_size:- raise ValueError(- f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"- f" size of {batch_size}. Make sure the batch size matches the length of the generators."- )-- if latents is None:- latents = randn_tensor(shape, generator=generator, device=device, dtype=dtype)- else:- latents = latents.to(device)-- # scale the initial noise by the standard deviation required by the scheduler- latents = latents * self.scheduler.init_noise_sigma- return latents-- @torch.no_grad()- @replace_example_docstring(EXAMPLE_DOC_STRING)- def __call__(- self,- prompt: Union[str, List[str]] = None,- negative_prompt: str = "",- num_inference_steps: int = 20,- timesteps: List[int] = None,- guidance_scale: float = 4.5,- num_images_per_prompt: Optional[int] = 1,- video_length: Optional[int] = None,- height: Optional[int] = None,- width: Optional[int] = None,- eta: float = 0.0,- generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,- latents: Optional[torch.FloatTensor] = None,- prompt_embeds: Optional[torch.FloatTensor] = None,- negative_prompt_embeds: Optional[torch.FloatTensor] = None,- output_type: Optional[str] = "pil",- return_dict: bool = True,- callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None,- callback_steps: int = 1,- clean_caption: bool = True,- mask_feature: bool = True,- enable_temporal_attentions: bool = True,- enable_vae_temporal_decoder: bool = False,- ) -> Union[VideoPipelineOutput, Tuple]:- """- Function invoked when calling the pipeline for generation.-- Args:- prompt (`str` or `List[str]`, *optional*):- The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`.- instead.- negative_prompt (`str` or `List[str]`, *optional*):- The prompt or prompts not to guide the image generation. If not defined, one has to pass- `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is- less than `1`).- num_inference_steps (`int`, *optional*, defaults to 100):- The number of denoising steps. More denoising steps usually lead to a higher quality image at the- expense of slower inference.- timesteps (`List[int]`, *optional*):- Custom timesteps to use for the denoising process. If not defined, equal spaced `num_inference_steps`- timesteps are used. Must be in descending order.- guidance_scale (`float`, *optional*, defaults to 7.0):- Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).- `guidance_scale` is defined as `w` of equation 2. of [Imagen- Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >- 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,- usually at the expense of lower image quality.- num_images_per_prompt (`int`, *optional*, defaults to 1):- The number of images to generate per prompt.- height (`int`, *optional*, defaults to self.unet.config.sample_size):- The height in pixels of the generated image.- width (`int`, *optional*, defaults to self.unet.config.sample_size):- The width in pixels of the generated image.- eta (`float`, *optional*, defaults to 0.0):- Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to- [`schedulers.DDIMScheduler`], will be ignored for others.- generator (`torch.Generator` or `List[torch.Generator]`, *optional*):- One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)- to make generation deterministic.- latents (`torch.FloatTensor`, *optional*):- Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image- generation. Can be used to tweak the same generation with different prompts. If not provided, a latents- tensor will ge generated by sampling using the supplied random `generator`.- prompt_embeds (`torch.FloatTensor`, *optional*):- Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not- provided, text embeddings will be generated from `prompt` input argument.- negative_prompt_embeds (`torch.FloatTensor`, *optional*):- Pre-generated negative text embeddings. For PixArt-Alpha this negative prompt should be "". If not- provided, negative_prompt_embeds will be generated from `negative_prompt` input argument.- output_type (`str`, *optional*, defaults to `"pil"`):- The output format of the generate image. Choose between- [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.- return_dict (`bool`, *optional*, defaults to `True`):- Whether or not to return a [`~pipelines.stable_diffusion.IFPipelineOutput`] instead of a plain tuple.- callback (`Callable`, *optional*):- A function that will be called every `callback_steps` steps during inference. The function will be- called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.- callback_steps (`int`, *optional*, defaults to 1):- The frequency at which the `callback` function will be called. If not specified, the callback will be- called at every step.- clean_caption (`bool`, *optional*, defaults to `True`):- Whether or not to clean the caption before creating embeddings. Requires `beautifulsoup4` and `ftfy` to- be installed. If the dependencies are not installed, the embeddings will be created from the raw- prompt.- mask_feature (`bool` defaults to `True`): If set to `True`, the text embeddings will be masked.-- Examples:-- Returns:- [`~pipelines.ImagePipelineOutput`] or `tuple`:- If `return_dict` is `True`, [`~pipelines.ImagePipelineOutput`] is returned, otherwise a `tuple` is- returned where the first element is a list with the generated images- """- # 1. Check inputs. Raise error if not correct- height = height or self.transformer.config.sample_size * self.vae_scale_factor- width = width or self.transformer.config.sample_size * self.vae_scale_factor- self.check_inputs(prompt, height, width, negative_prompt, callback_steps, prompt_embeds, negative_prompt_embeds)-- # 2. Default height and width to transformer- if prompt is not None and isinstance(prompt, str):- batch_size = 1- elif prompt is not None and isinstance(prompt, list):- batch_size = len(prompt)- else:- batch_size = prompt_embeds.shape[0]-- device = self._execution_device-- # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)- # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`- # corresponds to doing no classifier free guidance.- do_classifier_free_guidance = guidance_scale > 1.0-- # 3. Encode input prompt- prompt_embeds, negative_prompt_embeds = self.encode_prompt(- prompt,- do_classifier_free_guidance,- negative_prompt=negative_prompt,- num_images_per_prompt=num_images_per_prompt,- device=device,- prompt_embeds=prompt_embeds,- negative_prompt_embeds=negative_prompt_embeds,- clean_caption=clean_caption,- mask_feature=mask_feature,- )- if do_classifier_free_guidance:- prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)-- # 4. Prepare timesteps- self.scheduler.set_timesteps(num_inference_steps, device=device)- timesteps = self.scheduler.timesteps-- # 5. Prepare latents.- latent_channels = self.transformer.config.in_channels- latents = self.prepare_latents(- batch_size * num_images_per_prompt,- latent_channels,- video_length,- height,- width,- prompt_embeds.dtype,- device,- generator,- latents,- )-- # 6. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline- extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)-- # 6.1 Prepare micro-conditions.- added_cond_kwargs = {"resolution": None, "aspect_ratio": None}- if self.transformer.config.sample_size == 128:- resolution = torch.tensor([height, width]).repeat(batch_size * num_images_per_prompt, 1)- aspect_ratio = torch.tensor([float(height / width)]).repeat(batch_size * num_images_per_prompt, 1)- resolution = resolution.to(dtype=prompt_embeds.dtype, device=device)- aspect_ratio = aspect_ratio.to(dtype=prompt_embeds.dtype, device=device)- added_cond_kwargs = {"resolution": resolution, "aspect_ratio": aspect_ratio}-- # 7. Denoising loop- num_warmup_steps = max(len(timesteps) - num_inference_steps * self.scheduler.order, 0)-- with self.progress_bar(total=num_inference_steps) as progress_bar:- for i, t in enumerate(timesteps):- latent_model_input = torch.cat([latents] * 2) if do_classifier_free_guidance else latents- latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)-- current_timestep = t- if not torch.is_tensor(current_timestep):- # TODO: this requires sync between CPU and GPU. So try to pass timesteps as tensors if you can- # This would be a good case for the `match` statement (Python 3.10+)- is_mps = latent_model_input.device.type == "mps"- if isinstance(current_timestep, float):- dtype = torch.float32 if is_mps else torch.float64- else:- dtype = torch.int32 if is_mps else torch.int64- current_timestep = torch.tensor([current_timestep], dtype=dtype, device=latent_model_input.device)- elif len(current_timestep.shape) == 0:- current_timestep = current_timestep[None].to(latent_model_input.device)- # broadcast to batch dimension in a way that's compatible with ONNX/Core ML- current_timestep = current_timestep.expand(latent_model_input.shape[0])-- # predict noise model_output- noise_pred = self.transformer(- latent_model_input,- encoder_hidden_states=prompt_embeds,- timestep=current_timestep,- added_cond_kwargs=added_cond_kwargs,- enable_temporal_attentions=enable_temporal_attentions,- return_dict=False,- )[0]-- # perform guidance- if do_classifier_free_guidance:- noise_pred_uncond, noise_pred_text = noise_pred.chunk(2)- noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)-- # learned sigma- if self.transformer.config.out_channels // 2 == latent_channels:- noise_pred = noise_pred.chunk(2, dim=1)[0]- else:- noise_pred = noise_pred-- # compute previous image: x_t -> x_t-1- latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs, return_dict=False)[0]-- # call the callback, if provided- if i == len(timesteps) - 1 or ((i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0):- progress_bar.update()- if callback is not None and i % callback_steps == 0:- step_idx = i // getattr(self.scheduler, "order", 1)- callback(step_idx, t, latents)-- if not output_type == "latents":- if enable_vae_temporal_decoder:- video = self.decode_latents_with_temporal_decoder(latents)- else:- video = self.decode_latents(latents)- else:- video = latents- return VideoPipelineOutput(video=video)-- # Offload all models- self.maybe_free_model_hooks()-- if not return_dict:- return (video,)-- return VideoPipelineOutput(video=video)-- def decode_latents(self, latents):- video_length = latents.shape[2]- latents = 1 / self.vae.config.scaling_factor * latents- latents = einops.rearrange(latents, "b c f h w -> (b f) c h w")- video = []- for frame_idx in range(latents.shape[0]):- video.append(self.vae.decode(latents[frame_idx : frame_idx + 1]).sample)- video = torch.cat(video)- video = einops.rearrange(video, "(b f) c h w -> b f h w c", f=video_length)- video = ((video / 2.0 + 0.5).clamp(0, 1) * 255).to(dtype=torch.uint8).cpu().contiguous()- # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16- return video-- def decode_latents_with_temporal_decoder(self, latents):- video_length = latents.shape[2]- latents = 1 / self.vae.config.scaling_factor * latents- latents = einops.rearrange(latents, "b c f h w -> (b f) c h w")- video = []-- decode_chunk_size = 14- for frame_idx in range(0, latents.shape[0], decode_chunk_size):- num_frames_in = latents[frame_idx : frame_idx + decode_chunk_size].shape[0]-- decode_kwargs = {}- decode_kwargs["num_frames"] = num_frames_in-- video.append(self.vae.decode(latents[frame_idx : frame_idx + decode_chunk_size], **decode_kwargs).sample)-- video = torch.cat(video)- video = einops.rearrange(video, "(b f) c h w -> b f h w c", f=video_length)- video = ((video / 2.0 + 0.5).clamp(0, 1) * 255).to(dtype=torch.uint8).cpu().contiguous()- # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16- return videodiff --git a/scripts/sample/latte/sample.sh b/scripts/sample/latte/sample.shdeleted file mode 100644index c85272f..0000000--- a/scripts/sample/latte/sample.sh+++ /dev/null@@ -1,19 +0,0 @@-#!/usr/bin/env bash--# get args-GPUS=${1:-8}--# get root dir-FOLDER_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"-ROOT_DIR=$FOLDER_DIR/../../..--# go to root dir-cd $ROOT_DIR---export PYTHONPATH=$FOLDER_DIR:$PYTHONPATH-python $FOLDER_DIR/sample_t2v.py \- --checkpoint /home/lishenggui/projects/sora/hf-weights/models--maxin-cn--Latte/snapshots/8f0591220fa329f9d917086810b3c0f6544a87c7/t2v.pt \- --model_path /home/lishenggui/projects/sora/hf-weights/models--maxin-cn--Latte/snapshots/8f0591220fa329f9d917086810b3c0f6544a87c7/t2v_required_models/ \- --text_prompt "A dog in astronaut suit and sunglasses floating in space" \- --output_path $ROOT_DIR/outputs/lattediff --git a/scripts/sample/latte/sample_t2v.py b/scripts/sample/latte/sample_t2v.pydeleted file mode 100644index c65d007..0000000--- a/scripts/sample/latte/sample_t2v.py+++ /dev/null@@ -1,242 +0,0 @@-# All rights reserved.-# Copyright 2024 Vchitect/Latte-# Licensed under the Apache License, Version 2.0 (the "License");-# you may not use this file except in compliance with the License.-# You may obtain a copy of the License at-#-# http://www.apache.org/licenses/LICENSE-2.0-#-# Unless required by applicable law or agreed to in writing, software-# distributed under the License is distributed on an "AS IS" BASIS,-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-# See the License for the specific language governing permissions and-# limitations under the License.-#-# modified from https://github.com/Vchitect/Latte/blob/main/sample/sample_t2v.py--import argparse-import os-import sys--import torch-from diffusers.models import AutoencoderKL, AutoencoderKLTemporalDecoder-from diffusers.schedulers import (- DDIMScheduler,- DDPMScheduler,- DEISMultistepScheduler,- DPMSolverMultistepScheduler,- EulerAncestralDiscreteScheduler,- EulerDiscreteScheduler,- HeunDiscreteScheduler,- KDPM2AncestralDiscreteScheduler,- PNDMScheduler,-)-from diffusers.schedulers.scheduling_dpmsolver_singlestep import DPMSolverSinglestepScheduler-from transformers import T5EncoderModel, T5Tokenizer--sys.path.append(os.path.split(sys.path[0])[0])-import imageio-from pipeline_videogen import VideoGenPipeline-from utils import save_video_grid--from download import find_model-from open_sora.modeling import LatteT2V---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument("--model_path", type=str, required=True, help="The path to the pretrained model files")- parser.add_argument("--checkpoint", type=str, required=True, help="The path to the t2v.pt file.")- parser.add_argument("--output_path", type=str, required=True, help="The path to save the output")-- # generation configs- parser.add_argument(- "--text_prompt", type=str, nargs="+", required=True, help="The text prompt to generate the video."- )- parser.add_argument("--video_length", type=int, default=16, help="The number of frames in the video.")- parser.add_argument("--image_height", type=int, default=256, help="The size of the generated images.")- parser.add_argument("--image_width", type=int, default=256, help="The size of the generated images.")- parser.add_argument("--guidance_scale", type=float, default=7.5, help="The scale of the guidance loss.")- parser.add_argument("--sample_method", type=str, default="PNDM", help="The sampling method to use.")- parser.add_argument("--num_sampling_steps", type=int, default=50, help="The number of sampling steps.")- parser.add_argument(- "--enable_temporal_attentions", action="store_true", default=True, help="Whether to enable temporal attentions."- )- parser.add_argument(- "--enable_vae_temporal_decoder",- action="store_true",- default=True,- help="Whether to enable the VAE temporal decoder.",- )-- # Scheduler configs- parser.add_argument("--beta_start", type=float, default=0.0001)- parser.add_argument("--beta_end", type=float, default=0.02)- parser.add_argument("--beta_schedule", type=str, default="linear")- parser.add_argument("--variance_type", type=str, default="learned_range")-- args = parser.parse_args()- return args---def main(args):- torch.set_grad_enabled(False)- device = "cuda" if torch.cuda.is_available() else "cpu"-- transformer_model = LatteT2V.from_pretrained_2d(- args.model_path, subfolder="transformer", video_length=args.video_length- ).to(device, dtype=torch.float16)- state_dict = find_model(args.checkpoint)- transformer_model.load_state_dict(state_dict["model"])-- if args.enable_vae_temporal_decoder:- vae = AutoencoderKLTemporalDecoder.from_pretrained(- args.model_path, subfolder="vae_temporal_decoder", torch_dtype=torch.float16- ).to(device)- else:- vae = AutoencoderKL.from_pretrained(args.model_path, subfolder="vae", torch_dtype=torch.float16).to(device)- tokenizer = T5Tokenizer.from_pretrained(args.model_path, subfolder="tokenizer")- text_encoder = T5EncoderModel.from_pretrained(- args.model_path, subfolder="text_encoder", torch_dtype=torch.float16- ).to(device)-- # set eval mode- transformer_model.eval()- vae.eval()- text_encoder.eval()-- if args.sample_method == "DDIM":- scheduler = DDIMScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "EulerDiscrete":- scheduler = EulerDiscreteScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "DDPM":- scheduler = DDPMScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "DPMSolverMultistep":- scheduler = DPMSolverMultistepScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "DPMSolverSinglestep":- scheduler = DPMSolverSinglestepScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "PNDM":- scheduler = PNDMScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "HeunDiscrete":- scheduler = HeunDiscreteScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "EulerAncestralDiscrete":- scheduler = EulerAncestralDiscreteScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "DEISMultistep":- scheduler = DEISMultistepScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )- elif args.sample_method == "KDPM2AncestralDiscrete":- scheduler = KDPM2AncestralDiscreteScheduler.from_pretrained(- args.model_path,- subfolder="scheduler",- beta_start=args.beta_start,- beta_end=args.beta_end,- beta_schedule=args.beta_schedule,- variance_type=args.variance_type,- )-- videogen_pipeline = VideoGenPipeline(- vae=vae, text_encoder=text_encoder, tokenizer=tokenizer, scheduler=scheduler, transformer=transformer_model- ).to(device)- # videogen_pipeline.enable_xformers_memory_efficient_attention()-- if not os.path.exists(args.output_path):- os.makedirs(args.output_path, exist_ok=True)-- video_grids = []- for prompt in args.text_prompt:- print("Processing the ({}) prompt".format(prompt))- videos = videogen_pipeline(- prompt,- video_length=args.video_length,- height=args.image_height,- width=args.image_width,- num_inference_steps=args.num_sampling_steps,- guidance_scale=args.guidance_scale,- enable_temporal_attentions=args.enable_temporal_attentions,- num_images_per_prompt=1,- mask_feature=True,- enable_vae_temporal_decoder=args.enable_vae_temporal_decoder,- ).video- try:- save_path = os.path.join(args.output_path, prompt.replace(" ", "_") + "_webv-imageio.mp4")- imageio.mimwrite(save_path, videos[0], fps=8, quality=9) # highest quality is 10, lowest is 0- except:- print("Error when saving {}".format(prompt))- video_grids.append(videos)- video_grids = torch.cat(video_grids, dim=0)-- video_grids = save_video_grid(video_grids)-- # torchvision.io.write_video(args.output_path + '_%04d' % args.run_time + '-.mp4', video_grids, fps=6)- save_path = os.path.join(args.output_path, "grid.mp4")- imageio.mimwrite(save_path, video_grids, fps=8, quality=5)- print("save path {}".format(abspath(args.output_path)))-- # save_videos_grid(video, f"./{prompt}.gif")---if __name__ == "__main__":- args = parse_args()- main(args)diff --git a/scripts/sample/latte/utils.py b/scripts/sample/latte/utils.pydeleted file mode 100644index 3a9e4f1..0000000--- a/scripts/sample/latte/utils.py+++ /dev/null@@ -1,471 +0,0 @@-# All rights reserved.-# Copyright 2024 Vchitect/Latte-# Licensed under the Apache License, Version 2.0 (the "License");-# you may not use this file except in compliance with the License.-# You may obtain a copy of the License at-#-# http://www.apache.org/licenses/LICENSE-2.0-#-# Unless required by applicable law or agreed to in writing, software-# distributed under the License is distributed on an "AS IS" BASIS,-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-# See the License for the specific language governing permissions and-# limitations under the License.-#-# copied from https://github.com/Vchitect/Latte/blob/main/utils.py--import html-import logging-import math-import os-import re-import subprocess-import urllib.parse as ul-from collections import OrderedDict-from typing import Iterable, Union--import torch-import torch.distributed as dist-from diffusers.utils import is_bs4_available, is_ftfy_available--# from torch._six import inf-from torch import inf-from torch.utils.tensorboard import SummaryWriter--if is_bs4_available():- from bs4 import BeautifulSoup--if is_ftfy_available():- import ftfy--_tensor_or_tensors = Union[torch.Tensor, Iterable[torch.Tensor]]---#################################################################################-# Training Clip Gradients #-#################################################################################---def get_grad_norm(parameters: _tensor_or_tensors, norm_type: float = 2.0) -> torch.Tensor:- r"""- Copy from torch.nn.utils.clip_grad_norm_-- Clips gradient norm of an iterable of parameters.-- The norm is computed over all gradients together, as if they were- concatenated into a single vector. Gradients are modified in-place.-- Args:- parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a- single Tensor that will have gradients normalized- max_norm (float or int): max norm of the gradients- norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for- infinity norm.- error_if_nonfinite (bool): if True, an error is thrown if the total- norm of the gradients from :attr:`parameters` is ``nan``,- ``inf``, or ``-inf``. Default: False (will switch to True in the future)-- Returns:- Total norm of the parameter gradients (viewed as a single vector).- """- if isinstance(parameters, torch.Tensor):- parameters = [parameters]- grads = [p.grad for p in parameters if p.grad is not None]- norm_type = float(norm_type)- if len(grads) == 0:- return torch.tensor(0.0)- device = grads[0].device- if norm_type == inf:- norms = [g.detach().abs().max().to(device) for g in grads]- total_norm = norms[0] if len(norms) == 1 else torch.max(torch.stack(norms))- else:- total_norm = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)- return total_norm---def clip_grad_norm_(- parameters: _tensor_or_tensors,- max_norm: float,- norm_type: float = 2.0,- error_if_nonfinite: bool = False,- clip_grad=True,-) -> torch.Tensor:- r"""- Copy from torch.nn.utils.clip_grad_norm_-- Clips gradient norm of an iterable of parameters.-- The norm is computed over all gradients together, as if they were- concatenated into a single vector. Gradients are modified in-place.-- Args:- parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a- single Tensor that will have gradients normalized- max_norm (float or int): max norm of the gradients- norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for- infinity norm.- error_if_nonfinite (bool): if True, an error is thrown if the total- norm of the gradients from :attr:`parameters` is ``nan``,- ``inf``, or ``-inf``. Default: False (will switch to True in the future)-- Returns:- Total norm of the parameter gradients (viewed as a single vector).- """- if isinstance(parameters, torch.Tensor):- parameters = [parameters]- grads = [p.grad for p in parameters if p.grad is not None]- max_norm = float(max_norm)- norm_type = float(norm_type)- if len(grads) == 0:- return torch.tensor(0.0)- device = grads[0].device- if norm_type == inf:- norms = [g.detach().abs().max().to(device) for g in grads]- total_norm = norms[0] if len(norms) == 1 else torch.max(torch.stack(norms))- else:- total_norm = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)- # print(total_norm)-- if clip_grad:- if error_if_nonfinite and torch.logical_or(total_norm.isnan(), total_norm.isinf()):- raise RuntimeError(- f"The total norm of order {norm_type} for gradients from "- "`parameters` is non-finite, so it cannot be clipped. To disable "- "this error and scale the gradients by the non-finite norm anyway, "- "set `error_if_nonfinite=False`"- )- clip_coef = max_norm / (total_norm + 1e-6)- # Note: multiplying by the clamped coef is redundant when the coef is clamped to 1, but doing so- # avoids a `if clip_coef < 1:` conditional which can require a CPU <=> device synchronization- # when the gradients do not reside in CPU memory.- clip_coef_clamped = torch.clamp(clip_coef, max=1.0)- for g in grads:- g.detach().mul_(clip_coef_clamped.to(g.device))- # gradient_cliped = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)- # print(gradient_cliped)- return total_norm---def get_experiment_dir(root_dir, args):- # if args.pretrained is not None and 'Latte-XL-2-256x256.pt' not in args.pretrained:- # root_dir += '-WOPRE'- if args.use_compile:- root_dir += "-Compile" # speedup by torch compile- if args.fixed_spatial:- root_dir += "-FixedSpa"- if args.enable_xformers_memory_efficient_attention:- root_dir += "-Xfor"- if args.gradient_checkpointing:- root_dir += "-Gc"- if args.mixed_precision:- root_dir += "-Amp"- if args.image_size == 512:- root_dir += "-512"- return root_dir---#################################################################################-# Training Logger #-#################################################################################---def create_logger(logging_dir):- """- Create a logger that writes to a log file and stdout.- """- if dist.get_rank() == 0: # real logger- logging.basicConfig(- level=logging.INFO,- # format='[\033[34m%(asctime)s\033[0m] %(message)s',- format="[%(asctime)s] %(message)s",- datefmt="%Y-%m-%d %H:%M:%S",- handlers=[logging.StreamHandler(), logging.FileHandler(f"{logging_dir}/log.txt")],- )- logger = logging.getLogger(__name__)-- else: # dummy logger (does nothing)- logger = logging.getLogger(__name__)- logger.addHandler(logging.NullHandler())- return logger---def create_tensorboard(tensorboard_dir):- """- Create a tensorboard that saves losses.- """- if dist.get_rank() == 0: # real tensorboard- # tensorboard- writer = SummaryWriter(tensorboard_dir)-- return writer---def write_tensorboard(writer, *args):- """- write the loss information to a tensorboard file.- Only for pytorch DDP mode.- """- if dist.get_rank() == 0: # real tensorboard- writer.add_scalar(args[0], args[1], args[2])---#################################################################################-# EMA Update/ DDP Training Utils #-#################################################################################--[email protected]_grad()-def update_ema(ema_model, model, decay=0.9999):- """- Step the EMA model towards the current model.- """- ema_params = OrderedDict(ema_model.named_parameters())- model_params = OrderedDict(model.named_parameters())-- for name, param in model_params.items():- # TODO: Consider applying only to params that require_grad to avoid small numerical changes of pos_embed- ema_params[name].mul_(decay).add_(param.data, alpha=1 - decay)---def requires_grad(model, flag=True):- """- Set requires_grad flag for all parameters in a model.- """- for p in model.parameters():- p.requires_grad = flag---def cleanup():- """- End DDP training.- """- dist.destroy_process_group()---def setup_distributed(backend="nccl", port=None):- """Initialize distributed training environment.- support both slurm and torch.distributed.launch- see torch.distributed.init_process_group() for more details- """- num_gpus = torch.cuda.device_count()-- if "SLURM_JOB_ID" in os.environ:- rank = int(os.environ["SLURM_PROCID"])- world_size = int(os.environ["SLURM_NTASKS"])- node_list = os.environ["SLURM_NODELIST"]- addr = subprocess.getoutput(f"scontrol show hostname {node_list} | head -n1")- # specify master port- if port is not None:- os.environ["MASTER_PORT"] = str(port)- elif "MASTER_PORT" not in os.environ:- # os.environ["MASTER_PORT"] = "29566"- os.environ["MASTER_PORT"] = str(29567 + num_gpus)- if "MASTER_ADDR" not in os.environ:- os.environ["MASTER_ADDR"] = addr- os.environ["WORLD_SIZE"] = str(world_size)- os.environ["LOCAL_RANK"] = str(rank % num_gpus)- os.environ["RANK"] = str(rank)- else:- rank = int(os.environ["RANK"])- world_size = int(os.environ["WORLD_SIZE"])-- # torch.cuda.set_device(rank % num_gpus)-- dist.init_process_group(- backend=backend,- world_size=world_size,- rank=rank,- )---#################################################################################-# Testing Utils #-#################################################################################---def save_video_grid(video, nrow=None):- b, t, h, w, c = video.shape-- if nrow is None:- nrow = math.ceil(math.sqrt(b))- ncol = math.ceil(b / nrow)- padding = 1- video_grid = torch.zeros((t, (padding + h) * nrow + padding, (padding + w) * ncol + padding, c), dtype=torch.uint8)-- print(video_grid.shape)- for i in range(b):- r = i // ncol- c = i % ncol- start_r = (padding + h) * r- start_c = (padding + w) * c- video_grid[:, start_r : start_r + h, start_c : start_c + w] = video[i]-- return video_grid---#################################################################################-# MMCV Utils #-#################################################################################---def collect_env():- # Copyright (c) OpenMMLab. All rights reserved.- from mmcv.utils import collect_env as collect_base_env- from mmcv.utils import get_git_hash-- """Collect the information of the running environments."""-- env_info = collect_base_env()- env_info["MMClassification"] = get_git_hash()[:7]-- for name, val in env_info.items():- print(f"{name}: {val}")-- print(torch.cuda.get_arch_list())- print(torch.version.cuda)---#################################################################################-# Pixart-alpha Utils #-#################################################################################--bad_punct_regex = re.compile(- r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"-)---def text_preprocessing(text, clean_caption=False):- if clean_caption and not is_bs4_available():- clean_caption = False-- if clean_caption and not is_ftfy_available():- clean_caption = False-- if not isinstance(text, (tuple, list)):- text = [text]-- def process(text: str):- if clean_caption:- text = clean_caption(text)- text = clean_caption(text)- else:- text = text.lower().strip()- return text-- return [process(t) for t in text]---# Copied from diffusers.pipelines.deepfloyd_if.pipeline_if.IFPipeline._clean_caption-def clean_caption(caption):- caption = str(caption)- caption = ul.unquote_plus(caption)- caption = caption.strip().lower()- caption = re.sub("<person>", "person", caption)- # urls:- caption = re.sub(- r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa- "",- caption,- ) # regex for urls- caption = re.sub(- r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa- "",- caption,- ) # regex for urls- # html:- caption = BeautifulSoup(caption, features="html.parser").text-- # @<nickname>- caption = re.sub(r"@[\w\d]+\b", "", caption)-- # 31C0—31EF CJK Strokes- # 31F0—31FF Katakana Phonetic Extensions- # 3200—32FF Enclosed CJK Letters and Months- # 3300—33FF CJK Compatibility- # 3400—4DBF CJK Unified Ideographs Extension A- # 4DC0—4DFF Yijing Hexagram Symbols- # 4E00—9FFF CJK Unified Ideographs- caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)- caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)- caption = re.sub(r"[\u3200-\u32ff]+", "", caption)- caption = re.sub(r"[\u3300-\u33ff]+", "", caption)- caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)- caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)- caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)- #######################################################-- # все виды тире / all types of dash --> "-"- caption = re.sub(- r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa- "-",- caption,- )-- # кавычки к одному стандарту- caption = re.sub(r"[`´«»“”¨]", '"', caption)- caption = re.sub(r"[‘’]", "'", caption)-- # "- caption = re.sub(r""?", "", caption)- # &- caption = re.sub(r"&", "", caption)-- # ip adresses:- caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)-- # article ids:- caption = re.sub(r"\d:\d\d\s+$", "", caption)-- # \n- caption = re.sub(r"\\n", " ", caption)-- # "#123"- caption = re.sub(r"#\d{1,3}\b", "", caption)- # "#12345.."- caption = re.sub(r"#\d{5,}\b", "", caption)- # "123456.."- caption = re.sub(r"\b\d{6,}\b", "", caption)- # filenames:- caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)-- #- caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""- caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""-- caption = re.sub(bad_punct_regex, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT- caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "-- # this-is-my-cute-cat / this_is_my_cute_cat- regex2 = re.compile(r"(?:\-|\_)")- if len(re.findall(regex2, caption)) > 3:- caption = re.sub(regex2, " ", caption)-- caption = ftfy.fix_text(caption)- caption = html.unescape(html.unescape(caption))-- caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640- caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc- caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231-- caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)- caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)- caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)- caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)- caption = re.sub(r"\bpage\s+\d+\b", "", caption)-- caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...-- caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)-- caption = re.sub(r"\b\s+\:\s+", r": ", caption)- caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)- caption = re.sub(r"\s+", " ", caption)-- caption.strip()-- caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)- caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)- caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)- caption = re.sub(r"^\.\S+$", "", caption)-- return caption.strip()diff --git a/scripts/train.sh b/scripts/train.shdeleted file mode 100644index b38de61..0000000--- a/scripts/train.sh+++ /dev/null@@ -1,44 +0,0 @@-#!/usr/bin/env bash--# get args-GPUS=${1:-8}--# get root dir-FOLDER_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"-ROOT_DIR=$FOLDER_DIR/..--# go to root dir-cd $ROOT_DIR--# define dataset shards-COLLATED_VIDEO_DIR=./dataset/MSRVTT-collated/train/videos-PROCESSED_DATASET=(- ./dataset/MSRVTT-processed/train/part-00000- ./dataset/MSRVTT-processed/train/part-00001- ./dataset/MSRVTT-processed/train/part-00002- ./dataset/MSRVTT-processed/train/part-00003- ./dataset/MSRVTT-processed/train/part-00004- ./dataset/MSRVTT-processed/train/part-00005- ./dataset/MSRVTT-processed/train/part-00006- ./dataset/MSRVTT-processed/train/part-00007- ./dataset/MSRVTT-processed/train/part-00008- ./dataset/MSRVTT-processed/train/part-00009-)--# create timestamp to differentiate between runs-timestamp=$(date +%Y-%m-%d-%H-%M)--# run single node training-torchrun --standalone \- --nproc_per_node $GPUS \- train.py \- --epochs 1 \- --batch_size 1 \- --lr 1e-4 \- --accumulation_steps 32 \- --grad_checkpoint \- --dataset "${PROCESSED_DATASET[@]}" \- --video_dir $COLLATED_VIDEO_DIR \- --save_interval 224 \- --checkpoint_dir ./checkpoints/$timestamp \- --tensorboard_dir ./runs/$timestampdiff --git a/setup.py b/setup.pydeleted file mode 100644index 582beef..0000000--- a/setup.py+++ /dev/null@@ -1,82 +0,0 @@-import os-from typing import List--from setuptools import find_packages, setup---def fetch_requirements(path) -> List[str]:- """- This function reads the requirements file.-- Args:- path (str): the path to the requirements file.-- Returns:- The lines in the requirements file.- """- with open(path, "r") as fd:- return [r.strip() for r in fd.readlines()]---def fetch_readme() -> str:- """- This function reads the README.md file in the current directory.-- Returns:- The lines in the README file.- """- with open("README.md", encoding="utf-8") as f:- return f.read()---def get_version() -> str:- """- This function reads the version.txt and generates the colossalai/version.py file.-- Returns:- The library version stored in version.txt.- """-- setup_file_path = os.path.abspath(__file__)- project_path = os.path.dirname(setup_file_path)- version_txt_path = os.path.join(project_path, "version.txt")-- with open(version_txt_path) as f:- version = f.read().strip()- return version---setup(- name="open-sora",- version=get_version(),- packages=find_packages(- exclude=(- "docker",- "tests",- "docs",- "examples",- "tests",- "scripts",- "requirements",- "extensions",- "*.egg-info",- ),- ),- description="Unofficial implementation of OpenAI's Sora by the Colossal-AI Team",- long_description=fetch_readme(),- long_description_content_type="text/markdown",- license="Apache Software License 2.0",- url="https://www.colossalai.org",- project_urls={- "Github": "https://github.com/hpcaitech/Open-Sora",- },- install_requires=fetch_requirements("requirements.txt"),- python_requires=">=3.6",- classifiers=[- "Programming Language :: Python :: 3",- "License :: OSI Approved :: Apache Software License",- "Environment :: GPU :: NVIDIA CUDA",- "Topic :: Scientific/Engineering :: Artificial Intelligence",- "Topic :: System :: Distributed Computing",- ],-)diff --git a/tests/test_model.py b/tests/test_model.pydeleted file mode 100644index 1c2462f..0000000--- a/tests/test_model.py+++ /dev/null@@ -1,117 +0,0 @@-import colossalai-import pytest-import torch-import torch.distributed as dist-import torch.nn.functional as F-from colossalai.booster import Booster-from colossalai.logging import disable_existing_loggers-from colossalai.nn.optimizer import HybridAdam-from colossalai.testing import parameterize, rerun_if_address_is_in_use, spawn-from colossalai.utils import get_current_device-from torch.testing import assert_close--from open_sora.modeling import DiT_models-from open_sora.utils.plugin import ZeroSeqParallelPlugin---@parameterize("sp_size", [2, 4])-@parameterize("sp_mode", ["ulysses", "fastseq"])-@parameterize("sp_overlap", [True, False])-@parameterize("cross_attn", [True, False])-def check_dit_model_fwd_bwd(- sp_size: int,- sp_mode: str,- sp_overlap: bool,- cross_attn: bool,- video_latent_states,- text_latent_states,- t,- mask,-):- if sp_overlap and sp_mode == "ulysses":- return- if not cross_attn:- mask = mask.new_ones(- *mask.shape[:2],- mask.shape[2] + mask.shape[3],- mask.shape[2] + mask.shape[3],- )- plugin = ZeroSeqParallelPlugin(- sp_size=sp_size, stage=2, precision="fp32", master_weights=False- )- booster = Booster(plugin=plugin)- model = DiT_models["DiT-B/8"](text_dropout_prob=0.0, use_cross_attn=cross_attn).to(- get_current_device()- )- parallel_model = DiT_models["DiT-B/8"](- text_dropout_prob=0.0,- use_cross_attn=cross_attn,- seq_parallel_group=plugin.sp_group,- seq_parallel_mode=sp_mode,- seq_parallel_overlap=sp_overlap,- ).to(get_current_device())- parallel_model.load_state_dict(model.state_dict())- opt = HybridAdam(parallel_model.parameters(), lr=1e-3)- parallel_model, opt, *_ = booster.boost(parallel_model, opt)-- target = model(video_latent_states, t, text_latent_states, mask)- noise = torch.randn_like(target)- target_loss = F.mse_loss(target, noise)- target_loss.backward()-- dp_video_latent_states = video_latent_states.chunk(plugin.dp_size)[plugin.dp_rank]- dp_text_latent_states = text_latent_states.chunk(plugin.dp_size)[plugin.dp_rank]- dp_t = t.chunk(plugin.dp_size)[plugin.dp_rank]- dp_mask = mask.chunk(plugin.dp_size)[plugin.dp_rank]- dp_noise = noise.chunk(plugin.dp_size)[plugin.dp_rank]-- output = parallel_model(- dp_video_latent_states, dp_t, dp_text_latent_states, dp_mask- )- loss = F.mse_loss(output, dp_noise)- booster.backward(loss, opt)-- if plugin.dp_size == 1:- assert_close(target, output)-- for p1, p2 in zip(model.parameters(), opt._master_param_groups_of_current_rank[0]):- working_p = opt._param_store.master_to_working_param[id(p2)]- grads = opt._grad_store.get_partitioned_gradients_by_param_id(0, id(working_p))- grad_index = 0 if opt._partition_grads else opt._local_rank- grad = grads[grad_index]- sharded_grad = p1.grad.view(-1).chunk(dist.get_world_size())[dist.get_rank()]- assert_close(sharded_grad, grad[: sharded_grad.shape[0]])---def run_dist(rank, world_size, port):- disable_existing_loggers()- colossalai.launch(- config={},- rank=rank,- world_size=world_size,- port=port,- host="localhost",- backend="nccl",- )- b, s, c, p = 4, 20, 3, 8- dim_text, s_text = 512, 12- video_latent_states = torch.rand(b, s, c, p, p, device=get_current_device())- text_latent_states = torch.rand(b, s_text, dim_text, device=get_current_device())- t = torch.randint(0, 1000, (b,), device=get_current_device())- mask = torch.ones(b, 1, s, s_text, device=get_current_device(), dtype=torch.int)- check_dit_model_fwd_bwd(- video_latent_states=video_latent_states,- text_latent_states=text_latent_states,- t=t,- mask=mask,- )---@rerun_if_address_is_in_use()-def test_dit_model():- spawn(run_dist, 4)---if __name__ == "__main__":- test_dit_model()diff --git a/tests/test_sp_attn.py b/tests/test_sp_attn.pydeleted file mode 100644index d62a4ab..0000000--- a/tests/test_sp_attn.py+++ /dev/null@@ -1,102 +0,0 @@-import colossalai-import pytest-import torch-import torch.distributed as dist-from colossalai.logging import disable_existing_loggers-from colossalai.testing import parameterize, rerun_if_address_is_in_use, spawn-from colossalai.utils import get_current_device-from torch.testing import assert_close--from open_sora.modeling.dit import (- CrossAttention,- FastSeqParallelCrossAttention,- SeqParallelCrossAttention,-)-from open_sora.utils.comm import gather_seq, split_seq---@parameterize("layer_cls", [SeqParallelCrossAttention, FastSeqParallelCrossAttention])-@parameterize("overlap", [True, False])-@parameterize("cross_attn", [True, False])-def check_sp_attn(layer_cls, overlap, cross_attn):- if overlap and layer_cls == SeqParallelCrossAttention:- return- model_kwargs = {}- if layer_cls == FastSeqParallelCrossAttention:- model_kwargs["overlap"] = overlap- sp_size = dist.get_world_size()- sp_rank = dist.get_rank()- q_dim, context_dim = 8, 4- num_heads = 4- head_dim = 16- bs = 2- sq = 8- skv = 4- if not cross_attn:- context_dim = None- skv = sq- attn = CrossAttention(q_dim, context_dim, num_heads, head_dim).to(- get_current_device()- )- parallel_attn = layer_cls(- q_dim,- context_dim,- num_heads,- head_dim,- seq_parallel_group=dist.group.WORLD,- **model_kwargs,- ).to(get_current_device())- parallel_attn.load_state_dict(attn.state_dict())- hidden_states = torch.rand(bs, sq, q_dim, device=get_current_device())- if cross_attn:- context = torch.rand(bs, skv, context_dim, device=get_current_device())- else:- context = None- mask = torch.zeros(bs, 1, sq, skv, device=get_current_device())- target = attn(hidden_states, context, mask)- hidden_states_parallel = split_seq(hidden_states, sp_size, sp_rank)- if cross_attn:- context_parallel = split_seq(context, sp_size, sp_rank)- else:- context_parallel = None- output_parallel = parallel_attn(- hidden_states_parallel,- context_parallel,- mask,- )- assert torch.equal(target.chunk(sp_size, dim=1)[sp_rank], output_parallel)- output = gather_seq(output_parallel, sp_size, sp_rank, dist.group.WORLD)- assert torch.equal(target, output)- target.mean().backward()- output.mean().backward()-- # all-reduce mean of grads- for p in parallel_attn.parameters():- p.grad.data.div_(sp_size)- dist.all_reduce(p.grad.data)-- for (n, p1), p2 in zip(attn.named_parameters(), parallel_attn.parameters()):- assert_close(p1.grad, p2.grad, msg=lambda m: f"Check {n}\n{m}")---def run_dist(rank, world_size, port):- disable_existing_loggers()- colossalai.launch(- config={},- rank=rank,- world_size=world_size,- port=port,- host="localhost",- backend="nccl",- )- check_sp_attn()---@rerun_if_address_is_in_use()-def test_seq_parallel_attn():- spawn(run_dist, 2)---if __name__ == "__main__":- test_seq_parallel_attn()diff --git a/train.py b/train.pydeleted file mode 100644index 2d48f63..0000000--- a/train.py+++ /dev/null@@ -1,278 +0,0 @@-# Copyright (c) Meta Platforms, Inc. and affiliates.-# All rights reserved.--# This source code is licensed under the license found in the-# LICENSE file in the root directory of this source tree.--"""-A minimal training script for DiT using PyTorch DDP.-"""-import argparse-import os-from functools import partial--import torch-import torch.distributed as dist-from colossalai import launch_from_torch-from colossalai.accelerator import get_accelerator-from colossalai.booster import Booster-from colossalai.cluster import DistCoordinator-from colossalai.logging import get_dist_logger-from colossalai.nn.lr_scheduler import CosineAnnealingLR-from colossalai.nn.optimizer import HybridAdam-from colossalai.utils import get_current_device-from torch.utils.tensorboard import SummaryWriter-from tqdm import tqdm--from open_sora.diffusion import create_diffusion-from open_sora.modeling import DiT_models-from open_sora.modeling.dit import SUPPORTED_MODEL_ARCH, SUPPORTED_SEQ_PARALLEL_MODES-from open_sora.utils.data import (- create_video_compressor,- load_datasets,- make_batch,- preprocess_batch,-)-from open_sora.utils.plugin import ZeroSeqParallelPlugin--#################################################################################-# Training Helper Functions #-#################################################################################---def configure_backends():- # the first flag below was False when we tested this script but True makes A100 training a lot faster:- torch.backends.cuda.matmul.allow_tf32 = True- torch.backends.cudnn.allow_tf32 = True--[email protected]_grad()-def update_ema(ema_model, model, decay=0.9999):- """- Step the EMA model towards the current model.- """-- for ema_p, p in zip(ema_model.parameters(), model.parameters()):- ema_p.mul_(decay).add_(p.data, alpha=1 - decay)---def requires_grad(model, flag=True):- """- Set requires_grad flag for all parameters in a model.- """- for p in model.parameters():- p.requires_grad = flag---def all_reduce_mean(tensor: torch.Tensor) -> torch.Tensor:- dist.all_reduce(tensor=tensor, op=dist.ReduceOp.SUM)- tensor.div_(dist.get_world_size())- return tensor---def save_checkpoints(booster, model, optimizer, ema, save_path, coordinator):- os.makedirs(save_path, exist_ok=True)- booster.save_model(model, os.path.join(save_path, "model"), shard=False)- booster.save_optimizer(optimizer, os.path.join(save_path, "optimizer"), shard=True)- if coordinator.is_master():- ema_state_dict = ema.state_dict()- for k, v in ema_state_dict.items():- ema_state_dict[k] = v.cpu()- torch.save(ema_state_dict, os.path.join(save_path, "ema.pt"))- dist.barrier()---#################################################################################-# Training Loop #-#################################################################################---def main(args):- """- Trains a new DiT model.- """- # Step 1: init distributed environment- launch_from_torch({})- coordinator = DistCoordinator()- logger = get_dist_logger()- configure_backends()-- # Step 2: set up acceleration plugins- plugin = ZeroSeqParallelPlugin(sp_size=args.sp_size, stage=2, precision="fp16")- booster = Booster(plugin=plugin)-- if coordinator.is_master():- os.makedirs(args.checkpoint_dir, exist_ok=True)- os.makedirs(args.tensorboard_dir, exist_ok=True)- writer = SummaryWriter(args.tensorboard_dir)-- # Step 3: Create video compressor- video_compressor = create_video_compressor(args.compressor)- model_kwargs = {- "in_channels": video_compressor.out_channels,- "seq_parallel_group": plugin.sp_group,- "seq_parallel_mode": args.sp_mode,- "seq_parallel_overlap": args.sp_overlap,- "model_arch": args.model_arch,- }-- # Step 4: Create DiT and EMA- model = DiT_models[args.model](**model_kwargs).to(get_current_device())- patch_size = model.patch_size- ema = DiT_models[args.model](**model_kwargs).to(get_current_device())- update_ema(ema, model, decay=0)- requires_grad(ema, False)-- model.train() # important! This enables embedding dropout for classifier-free guidance- ema.eval() # EMA model should always be in eval mode-- # configure gradient checkpointing- if args.grad_checkpoint:- model.enable_gradient_checkpointing()-- # Step 5: create diffusion pipeline- diffusion = create_diffusion(- timestep_respacing=""- ) # default: 1000 steps, linear noise schedule-- # Step 6: setup optimizer (we used default Adam betas=(0.9, 0.999) and a constant learning rate of 1e-4 in our paper):- opt = HybridAdam(model.parameters(), lr=args.lr, weight_decay=0)-- # Step 7: Setup dataloader- dataset = load_datasets(args.dataset)- dataloader = plugin.prepare_dataloader(- dataset,- batch_size=args.batch_size,- collate_fn=partial(- make_batch,- video_dir=args.video_dir,- pad_to_multiple=args.sp_size,- use_pooled_text=args.model_arch == "adaln",- ),- shuffle=True,- drop_last=True,- )- lr_scheduler = CosineAnnealingLR(- opt, args.epochs * len(dataloader) // args.accumulation_steps- )- logger.info(f"Dataset contains {len(dataset)} samples", ranks=[0])-- # Step 8: setup booster- model, opt, _, dataloader, lr_scheduler = booster.boost(- model, opt, dataloader=dataloader, lr_scheduler=lr_scheduler- )- if args.load_model is not None:- booster.load_model(model, args.load_model)- if args.load_optimizer is not None:- booster.load_optimizer(opt, args.load_optimizer)- logger.info(- f"Booster init max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",- ranks=[0],- )-- # Step 9: Train- num_steps_per_epoch = len(dataloader) // args.accumulation_steps-- for epoch in range(args.epochs):- dataloader.sampler.set_epoch(epoch)- with tqdm(- desc=f"Epoch {epoch}",- disable=not coordinator.is_master(),- total=num_steps_per_epoch,- ) as pbar:- total_loss = torch.tensor(0.0, device=get_current_device())- for step, batch in enumerate(dataloader):- batch = preprocess_batch(- batch,- patch_size,- video_compressor,- pad_to_multiple=args.sp_size,- model_arch=args.model_arch,- )- video_inputs = batch.pop("video_latent_states")- mask = batch.pop("video_padding_mask")- t = torch.randint(- 0,- diffusion.num_timesteps,- (video_inputs.shape[0],),- device=video_inputs.device,- )- loss_dict = diffusion.training_losses(- model, video_inputs, t, batch, mask=mask- )- loss = loss_dict["loss"].mean() / args.accumulation_steps- total_loss.add_(loss.data)- booster.backward(loss, opt)-- if (step + 1) % args.accumulation_steps == 0:- opt.step()- opt.zero_grad()- lr_scheduler.step()- update_ema(ema, model)-- all_reduce_mean(total_loss)- pbar.set_postfix({"Loss": f"{total_loss.item():.4f}"})- if coordinator.is_master():- global_step = (epoch * num_steps_per_epoch) + (- step + 1- ) // args.accumulation_steps- writer.add_scalar(- tag="Loss",- scalar_value=total_loss.item(),- global_step=global_step,- )- pbar.update()- total_loss.zero_()-- # Save DiT checkpoint:- if args.save_interval > 0 and (- (step + 1) % (args.save_interval * args.accumulation_steps) == 0- or (step + 1) == len(dataloader)- ):- save_path = os.path.join(- args.checkpoint_dir, f"epoch-{epoch}-step-{step}"- )- save_checkpoints(booster, model, opt, ema, save_path, coordinator)- logger.info(f"Saved checkpoint to {save_path}", ranks=[0])-- get_accelerator().empty_cache()- final_save_path = os.path.join(args.checkpoint_dir, "final")- save_checkpoints(booster, model, opt, ema, final_save_path, coordinator)- logger.info(f"Saved checkpoint to {final_save_path}", ranks=[0])- logger.info(- f"Training complete, max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",- ranks=[0],- )---if __name__ == "__main__":- # Default args here will train DiT-XL/2 with the hyperparameters we used in our paper (except training iters).- parser = argparse.ArgumentParser()- parser.add_argument(- "-m", "--model", type=str, choices=list(DiT_models.keys()), default="DiT-S/8"- )- parser.add_argument(- "-x", "--model_arch", choices=SUPPORTED_MODEL_ARCH, default="cross-attn"- )- parser.add_argument("-d", "--dataset", nargs="+", default=[])- parser.add_argument("-v", "--video_dir", type=str, required=True)- parser.add_argument("-e", "--epochs", type=int, default=10)- parser.add_argument("-b", "--batch_size", type=int, default=4)- parser.add_argument("-g", "--grad_checkpoint", action="store_true", default=False)- parser.add_argument("-a", "--accumulation_steps", default=1, type=int)- parser.add_argument("--sp_size", type=int, default=1)- parser.add_argument(- "--sp_mode", type=str, default="ulysses", choices=SUPPORTED_SEQ_PARALLEL_MODES- )- parser.add_argument("--sp_overlap", action="store_true", default=False)- parser.add_argument("--lr", type=float, default=1e-4)- parser.add_argument("--save_interval", type=int, default=20)- parser.add_argument("--checkpoint_dir", type=str, default="checkpoints")- parser.add_argument("--tensorboard_dir", type=str, default="runs")- parser.add_argument(- "-c", "--compressor", choices=["raw", "vqvae", "vae"], default="raw"- )- parser.add_argument("--load_model", default=None)- parser.add_argument("--load_optimizer", default=None)- args = parser.parse_args()- main(args)diff --git a/version.txt b/version.txtdeleted file mode 100644index 8acdd82..0000000--- a/version.txt+++ /dev/null@@ -1 +0,0 @@-0.0.1