Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

removed old files (#63)

Frank Lee2f87a9af75c2
diff --git a/README.md b/README.md
deleted file mode 100644
index 46d75c2..0000000
--- a/README.md
+++ /dev/null
@@ -1,165 +0,0 @@
-# 🎥 Open-Sora
-<div id="top" align="center">
-
- [![GitHub Repo stars](https://img.shields.io/github/stars/hpcaitech/Open-Sora?style=social)](https://github.com/hpcaitech/Open-Sora/stargazers)
- [![slack badge](https://img.shields.io/badge/Slack-join-blueviolet?logo=slack&amp)](https://github.com/hpcaitech/public_assets/tree/main/colossalai/contact/slack)
- [![WeChat badge](https://img.shields.io/badge/微信-加入-green?logo=wechat&amp)](https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/WeChat.png)
-
-</div>
-
-## 📎 Table of Contents
-
-- [🎥 Open-Sora](#-open-sora)
- - [📎 Table of Contents](#-table-of-contents)
- - [📍 Overview](#-overview)
- - [📂 Dataset Preparation](#-dataset-preparation)
- - [Use MSR-VTT](#use-msr-vtt)
- - [Use Customized Datasets](#use-customized-datasets)
- - [🚀 Get Started](#-get-started)
- - [Training](#training)
- - [Inference](#inference)
- - [🪄 Acknowledgement](#-acknowledgement)
-
-## Latest News
-* [2024/03] [Open-Sora:Sora Replication Solution with 46% Cost Reduction, Sequence Expansion to Nearly a Million](https://hpc-ai.com/blog/open-sora)
-
-## 📍 Overview
-Open-Sora is an open-source project that provides a high-performance implementation of the development pipeline that Sora might use powered by [Colossal-AI](https://github.com/hpcaitech/ColossalAI), including:
-
-- Provides **a complete Sora reproduction architecture solution**, including the whole process from data processing to training and deployment.
-- Supports **dynamic resolution**, training can directly train any resolution of the video, without scaling.
-- Supports **multiple model structures**. Since the actual model structure of Sora is unknown, we implement three common multimodal model structures such as adaLN-zero, cross attention, and in-context conditioning (token concat).
-- Supports **multiple video compression methods**. Users can choose to use original video, VQVAE (video native model), or SD-VAE (image native model) for training.
-- Supports **multiple parallel training optimizations**. Including the AI large model system optimization capability compatible with Colossal-AI, and hybrid sequence parallelism with Ulysses and FastSeq.
-
-<p id="diffusion_demo" align="center">
-<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/open-sora-1.png" width=800/>
-</p>
-
-<p id="diffusion_demo" align="center">
-<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/open-sora-2.png" width=800/>
-</p>
-
-<p id="diffusion_demo" align="center">
-<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/dit-model-arch.png" width=800/>
-</p>
-
-## 📂 Dataset Preparation
-
-### Use MSR-VTT
-
-We use [MSR-VTT](https://cove.thecvf.com/datasets/839) dataset, which is a large-scale video description dataset. Users should preprocess the raw videos before training the model. You can use the following scripts to perform data processing.
-
-
-```bash
-# Step 1: download the dataset to ./dataset/MSRVTT
-bash scripts/data/download_msr_vtt_dataset.sh
-
-# Step 2: collate the video and annotations
-python scripts/data/collate_msr_vtt_dataset.py -d ./dataset/MSRVTT/ -o ./dataset/MSRVTT-collated
-
-# Step 3: perform data processing
-# NOTE: each script could several minutes so we apply the script to the dataset split individually
-python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/train/annotations.json -v ./dataset/MSRVTT-collated/train/videos -o ./dataset/MSRVTT-processed/train
-python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/val/annotations.json -v ./dataset/MSRVTT-collated/val/videos -o ./dataset/MSRVTT-processed/val
-python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/test/annotations.json -v ./dataset/MSRVTT-collated/test/videos -o ./dataset/MSRVTT-processed/test
-```
-
-**If you want to use adaLN-zero, you should use `--use_pooled_text` when running `preprocess_data.py`**
-
-```bash
-
-After completing these steps, you should have a processed MSR-VTT dataset in `./dataset/MSRVTT-processed`.
-
-
-### Use Customized Datasets
-
-You can also use other datasets and transform the dataset to the required format. You should prepare a captions file and a video directory. The captions file should be a JSON file or a JSONL file. The video directory should contain all the videos.
-
-Here is an example of the captions file:
-
-```json
-[
- {
- "file": "video0.mp4",
- "captions": ["a girl is throwing away folded clothes", "a girl throwing cloths around"]
- },
- {
- "file": "video1.mp4",
- "captions": ["a comparison of two opposing team football athletes"]
- }
-]
-```
-
-Here is an example of the video directory:
-
-```
-.
-├── video0.mp4
-├── video1.mp4
-└── ...
-```
-
-Each video may have multiple captions. So the outputs are video-caption pairs. E.g., If the first video has two captions, then the output will be two video-caption pairs.
-
-We use [VQ-VAE](https://github.com/wilson1yan/VideoGPT/) to quantize the video frames. And we use [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#clip) to extract the text features.
-
-The output is an arrow dataset, which contains the following columns: "video_file", "video_latent_states", "text_latent_states". The dimension of "video_latent_states" is (T, H, W), and the dimension of "text_latent_states" is (S, D).
-
-Then you can run the data processing script with the command below:
-
-```bash
-python preprocess_data.py -c /path/to/captions.json -v /path/to/video_dir -o /path/to/output_dir
-```
-
-**If you want to use adaLN-zero, you should use `--use_pooled_text` when running `preprocess_data.py`**
-
-Note that this script needs to be run on a machine with a GPU. To avoid CUDA OOM, we filter out the videos that are too long.
-
-
-## 🚀 Get Started
-
-In this section, we will guide how to run training and inference. Before that, make sure you installed the dependencies with the command below.
-
-```bash
-pip install -r requirements.txt
-```
-
-### Training
-
-You can invoke the training via the command below.
-
-```bash
-bash ./scripts/train.sh
-```
-
-You can also modify the arguments in `train.sh` for your own need.
-
-We provide three model architectures: "adaln", "cross-attn" and "token-concat". You can specify the model architecture with the `-x` option.
-
-### Inference
-
-We've provided a script to perform inference, allowing you to generate videos from the trained model. You can invoke the inference via the command below.
-
-```bash
-python sample.py -m "DiT/XL-2" --text "a person is walking on the street" --ckpt /path/to/checkpoint --height 256 --width 256 --fps 10 --sec 5 --disable-cfg
-```
-This will generate a "sample.mp4" file in the current directory.
-
-For more command line options, you can use the following command to check the help message.
-
-```bash
-python sample.py -h
-```
-
-Please make sure the video compressor and model architecture are consistent with the training settings.
-
-## 🪄 Acknowledgement
-
-During the development of the project, we learned a lot from the following information:
-
-- [OpenAI Sora Technical Report](https://openai.com/research/video-generation-models-as-world-simulators)
-- [VideoGPT Project](https://github.com/wilson1yan/VideoGPT)
-- [Diffusion Transformers](https://github.com/facebookresearch/DiT)
-- [Deepspeed Ulysses](https://arxiv.org/abs/2309.14509)
-- [OpenDiT](https://github.com/NUS-HPC-AI-Lab/OpenDiT)
diff --git a/benchmark.py b/benchmark.py
deleted file mode 100644
index 69eef8d..0000000
--- a/benchmark.py
+++ /dev/null
@@ -1,180 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""
-A minimal training script for DiT using PyTorch DDP.
-"""
-import argparse
-import time
-
-import torch
-import torch.distributed as dist
-from colossalai import launch_from_torch
-from colossalai.accelerator import get_accelerator
-from colossalai.booster import Booster
-from colossalai.booster.plugin import TorchDDPPlugin
-from colossalai.cluster import DistCoordinator
-from colossalai.logging import get_dist_logger
-from colossalai.nn.optimizer import HybridAdam
-from colossalai.utils import get_current_device
-from tqdm import tqdm
-
-from open_sora.diffusion import create_diffusion
-from open_sora.modeling import DiT_models
-from open_sora.modeling.dit import SUPPORTED_SEQ_PARALLEL_MODES
-from open_sora.utils.data import create_video_compressor, preprocess_batch
-from open_sora.utils.plugin import ZeroSeqParallelPlugin
-
-#################################################################################
-# Training Loop #
-#################################################################################
-
-
-def main(args):
- """
- Trains a new DiT model.
- """
- # init distributed environment
- launch_from_torch({})
- coordinator = DistCoordinator()
- logger = get_dist_logger()
-
- # set up acceleration plugins
- if args.plugin == "ddp":
- plugin = TorchDDPPlugin()
- elif args.plugin == "zero2":
- # use bf16 to avoid skipping the first few iterations due to NaNs
- plugin = ZeroSeqParallelPlugin(sp_size=args.sp_size, stage=2, precision="bf16")
- else:
- raise ValueError(f"Unknown plugin {args.plugin}")
- booster = Booster(plugin=plugin)
-
- # Create video compressor
- video_compressor = create_video_compressor(args.compressor)
- model_kwargs = {
- "in_channels": video_compressor.out_channels,
- "seq_parallel_group": getattr(plugin, "sp_group", None),
- "seq_parallel_mode": args.sp_mode,
- "seq_parallel_overlap": args.sp_overlap,
- }
-
- # Create DiT and EMA
- model = DiT_models[args.model](**model_kwargs).to(get_current_device())
- patch_size = model.patch_size
- model.train() # important! This enables embedding dropout for classifier-free guidance
-
- # configure gradient checkpointing
- if args.grad_checkpoint:
- model.enable_gradient_checkpointing()
-
- # create diffusion pipeline
- diffusion = create_diffusion(
- timestep_respacing=""
- ) # default: 1000 steps, linear noise schedule
-
- # setup optimizer (we used default Adam betas=(0.9, 0.999) and a constant learning rate of 1e-4 in our paper):
- opt = HybridAdam(model.parameters(), lr=1e-4, weight_decay=0)
-
- # Setup dataloader
- videos = [
- torch.randn(args.num_frames, args.height, args.width, 3)
- for _ in range(args.batch_size)
- ]
- assert args.num_tokens % args.sp_size == 0
- input_ids = torch.randn(args.batch_size, args.num_tokens, args.text_embed_dim)
- text_mask = torch.ones(input_ids.shape[:2], dtype=torch.int)
- batch = {
- "videos": videos,
- "text_latent_states": input_ids,
- "text_padding_mask": text_mask,
- }
- batch = preprocess_batch(
- batch, patch_size, video_compressor, pad_to_multiple=args.sp_size
- )
- video_inputs = batch.pop("video_latent_states")
- mask = batch.pop("video_padding_mask")
- logger.info(
- f"Num patches: {video_inputs.shape[1]}, num tokens: {batch['text_latent_states'].shape[1]}",
- ranks=[0],
- )
-
- # setup booster
- model, opt, *_ = booster.boost(model, opt)
- logger.info(
- f"Booster init max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",
- ranks=[0],
- )
-
- # Train
- total_samples = 0
- total_duration = 0.0
- for i in tqdm(
- range(args.warmup_steps + args.steps),
- desc="Steps",
- disable=not coordinator.is_master(),
- ):
- start = time.time()
- t = torch.randint(
- 0,
- diffusion.num_timesteps,
- (video_inputs.shape[0],),
- device=video_inputs.device,
- )
- loss_dict = diffusion.training_losses(model, video_inputs, t, batch, mask=mask)
- loss = loss_dict["loss"].mean()
- booster.backward(loss, opt)
- opt.step()
- opt.zero_grad()
- get_accelerator().empty_cache()
- time_per_iter = time.time() - start
- if i >= args.warmup_steps:
- total_samples += args.batch_size * coordinator.world_size
- total_duration += time_per_iter
- total_duration = torch.tensor([total_duration], device=get_current_device())
- dist.all_reduce(total_duration)
- total_duration = total_duration / coordinator.world_size
- total_duration = total_duration.item()
- total_samples *= coordinator.world_size // args.sp_size
-
- throughput = total_samples / total_duration
- logger.info(
- f"Training complete, max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",
- ranks=[0],
- )
- logger.info(
- f"Throughput: {throughput:.2f} samples/s",
- ranks=[0],
- )
-
-
-if __name__ == "__main__":
- # Default args here will train DiT-XL/2 with the hyperparameters we used in our paper (except training iters).
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "-m", "--model", type=str, choices=list(DiT_models.keys()), default="DiT-S/8"
- )
- parser.add_argument(
- "-p", "--plugin", type=str, default="zero2", choices=["ddp", "zero2"]
- )
- parser.add_argument("--sp_size", type=int, default=1)
- parser.add_argument(
- "--sp_mode", type=str, default="ulysses", choices=SUPPORTED_SEQ_PARALLEL_MODES
- )
- parser.add_argument("--sp_overlap", action="store_true", default=False)
- parser.add_argument("-w", "--warmup_steps", type=int, default=2)
- parser.add_argument("-s", "--steps", type=int, default=3)
- parser.add_argument("-b", "--batch_size", type=int, default=4)
- parser.add_argument("-f", "--num_frames", type=int, default=300)
- parser.add_argument("--height", type=int, default=256)
- parser.add_argument("--width", type=int, default=256)
- parser.add_argument("--num_tokens", type=int, default=20)
- parser.add_argument("--text_embed_dim", type=int, default=512)
- parser.add_argument("-g", "--grad_checkpoint", action="store_true", default=False)
- parser.add_argument(
- "-c", "--compressor", choices=["raw", "vqvae", "vae"], default="raw"
- )
- args = parser.parse_args()
- main(args)
diff --git a/download.py b/download.py
deleted file mode 100644
index f4c3700..0000000
--- a/download.py
+++ /dev/null
@@ -1,50 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""
-Functions for downloading pre-trained DiT models
-"""
-import os
-
-import torch
-from torchvision.datasets.utils import download_url
-
-pretrained_models = {"DiT-XL-2-512x512.pt", "DiT-XL-2-256x256.pt"}
-
-
-def find_model(model_name):
- """
- Finds a pre-trained DiT model, downloading it if necessary. Alternatively, loads a model from a local path.
- """
- if model_name in pretrained_models: # Find/download our pre-trained DiT checkpoints
- return download_model(model_name)
- else: # Load a custom DiT checkpoint:
- assert os.path.isfile(model_name), f"Could not find DiT checkpoint at {model_name}"
- checkpoint = torch.load(model_name, map_location=lambda storage, loc: storage)
- if "ema" in checkpoint: # supports checkpoints from train.py
- checkpoint = checkpoint["ema"]
- return checkpoint
-
-
-def download_model(model_name):
- """
- Downloads a pre-trained DiT model from the web.
- """
- assert model_name in pretrained_models
- local_path = f"pretrained_models/{model_name}"
- if not os.path.isfile(local_path):
- os.makedirs("pretrained_models", exist_ok=True)
- web_path = f"https://dl.fbaipublicfiles.com/DiT/models/{model_name}"
- download_url(web_path, "pretrained_models")
- model = torch.load(local_path, map_location=lambda storage, loc: storage)
- return model
-
-
-if __name__ == "__main__":
- # Download all DiT checkpoints
- for model in pretrained_models:
- download_model(model)
- print("Done.")
diff --git a/open_sora/__init__.py b/open_sora/__init__.py
deleted file mode 100644
index e69de29..0000000
diff --git a/open_sora/diffusion/__init__.py b/open_sora/diffusion/__init__.py
deleted file mode 100644
index 427068d..0000000
--- a/open_sora/diffusion/__init__.py
+++ /dev/null
@@ -1,40 +0,0 @@
-# Modified from OpenAI's diffusion repos
-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py
-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion
-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py
-
-from . import gaussian_diffusion as gd
-from .respace import SpacedDiffusion, space_timesteps
-
-
-def create_diffusion(
- timestep_respacing,
- noise_schedule="linear",
- use_kl=False,
- sigma_small=False,
- predict_xstart=False,
- learn_sigma=True,
- rescale_learned_sigmas=False,
- diffusion_steps=1000,
-):
- betas = gd.get_named_beta_schedule(noise_schedule, diffusion_steps)
- if use_kl:
- loss_type = gd.LossType.RESCALED_KL
- elif rescale_learned_sigmas:
- loss_type = gd.LossType.RESCALED_MSE
- else:
- loss_type = gd.LossType.MSE
- if timestep_respacing is None or timestep_respacing == "":
- timestep_respacing = [diffusion_steps]
- return SpacedDiffusion(
- use_timesteps=space_timesteps(diffusion_steps, timestep_respacing),
- betas=betas,
- model_mean_type=(gd.ModelMeanType.EPSILON if not predict_xstart else gd.ModelMeanType.START_X),
- model_var_type=(
- (gd.ModelVarType.FIXED_LARGE if not sigma_small else gd.ModelVarType.FIXED_SMALL)
- if not learn_sigma
- else gd.ModelVarType.LEARNED_RANGE
- ),
- loss_type=loss_type
- # rescale_timesteps=rescale_timesteps,
- )
diff --git a/open_sora/diffusion/diffusion_utils.py b/open_sora/diffusion/diffusion_utils.py
deleted file mode 100644
index 056471c..0000000
--- a/open_sora/diffusion/diffusion_utils.py
+++ /dev/null
@@ -1,79 +0,0 @@
-# Modified from OpenAI's diffusion repos
-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py
-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion
-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py
-
-import numpy as np
-import torch as th
-
-
-def normal_kl(mean1, logvar1, mean2, logvar2):
- """
- Compute the KL divergence between two gaussians.
- Shapes are automatically broadcasted, so batches can be compared to
- scalars, among other use cases.
- """
- tensor = None
- for obj in (mean1, logvar1, mean2, logvar2):
- if isinstance(obj, th.Tensor):
- tensor = obj
- break
- assert tensor is not None, "at least one argument must be a Tensor"
-
- # Force variances to be Tensors. Broadcasting helps convert scalars to
- # Tensors, but it does not work for th.exp().
- logvar1, logvar2 = [x if isinstance(x, th.Tensor) else th.tensor(x).to(tensor) for x in (logvar1, logvar2)]
-
- return 0.5 * (-1.0 + logvar2 - logvar1 + th.exp(logvar1 - logvar2) + ((mean1 - mean2) ** 2) * th.exp(-logvar2))
-
-
-def approx_standard_normal_cdf(x):
- """
- A fast approximation of the cumulative distribution function of the
- standard normal.
- """
- return 0.5 * (1.0 + th.tanh(np.sqrt(2.0 / np.pi) * (x + 0.044715 * th.pow(x, 3))))
-
-
-def continuous_gaussian_log_likelihood(x, *, means, log_scales):
- """
- Compute the log-likelihood of a continuous Gaussian distribution.
- :param x: the targets
- :param means: the Gaussian mean Tensor.
- :param log_scales: the Gaussian log stddev Tensor.
- :return: a tensor like x of log probabilities (in nats).
- """
- centered_x = x - means
- inv_stdv = th.exp(-log_scales)
- normalized_x = centered_x * inv_stdv
- log_probs = th.distributions.Normal(th.zeros_like(x), th.ones_like(x)).log_prob(normalized_x)
- return log_probs
-
-
-def discretized_gaussian_log_likelihood(x, *, means, log_scales):
- """
- Compute the log-likelihood of a Gaussian distribution discretizing to a
- given image.
- :param x: the target images. It is assumed that this was uint8 values,
- rescaled to the range [-1, 1].
- :param means: the Gaussian mean Tensor.
- :param log_scales: the Gaussian log stddev Tensor.
- :return: a tensor like x of log probabilities (in nats).
- """
- assert x.shape == means.shape == log_scales.shape
- centered_x = x - means
- inv_stdv = th.exp(-log_scales)
- plus_in = inv_stdv * (centered_x + 1.0 / 255.0)
- cdf_plus = approx_standard_normal_cdf(plus_in)
- min_in = inv_stdv * (centered_x - 1.0 / 255.0)
- cdf_min = approx_standard_normal_cdf(min_in)
- log_cdf_plus = th.log(cdf_plus.clamp(min=1e-12))
- log_one_minus_cdf_min = th.log((1.0 - cdf_min).clamp(min=1e-12))
- cdf_delta = cdf_plus - cdf_min
- log_probs = th.where(
- x < -0.999,
- log_cdf_plus,
- th.where(x > 0.999, log_one_minus_cdf_min, th.log(cdf_delta.clamp(min=1e-12))),
- )
- assert log_probs.shape == x.shape
- return log_probs
diff --git a/open_sora/diffusion/gaussian_diffusion.py b/open_sora/diffusion/gaussian_diffusion.py
deleted file mode 100644
index fe94622..0000000
--- a/open_sora/diffusion/gaussian_diffusion.py
+++ /dev/null
@@ -1,842 +0,0 @@
-# Modified from OpenAI's diffusion repos
-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py
-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion
-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py
-
-
-import enum
-import math
-
-import numpy as np
-import torch as th
-
-from .diffusion_utils import discretized_gaussian_log_likelihood, normal_kl
-
-
-def mean_flat(tensor, mask=None):
- """
- Take the mean over all non-batch dimensions.
- """
- if mask is not None:
- tensor = tensor * mask
- return tensor.mean(dim=list(range(1, len(tensor.shape))))
-
-
-class ModelMeanType(enum.Enum):
- """
- Which type of output the model predicts.
- """
-
- PREVIOUS_X = enum.auto() # the model predicts x_{t-1}
- START_X = enum.auto() # the model predicts x_0
- EPSILON = enum.auto() # the model predicts epsilon
-
-
-class ModelVarType(enum.Enum):
- """
- What is used as the model's output variance.
- The LEARNED_RANGE option has been added to allow the model to predict
- values between FIXED_SMALL and FIXED_LARGE, making its job easier.
- """
-
- LEARNED = enum.auto()
- FIXED_SMALL = enum.auto()
- FIXED_LARGE = enum.auto()
- LEARNED_RANGE = enum.auto()
-
-
-class LossType(enum.Enum):
- MSE = enum.auto() # use raw MSE loss (and KL when learning variances)
- RESCALED_MSE = enum.auto() # use raw MSE loss (with RESCALED_KL when learning variances)
- KL = enum.auto() # use the variational lower-bound
- RESCALED_KL = enum.auto() # like KL, but rescale to estimate the full VLB
-
- def is_vb(self):
- return self == LossType.KL or self == LossType.RESCALED_KL
-
-
-def _warmup_beta(beta_start, beta_end, num_diffusion_timesteps, warmup_frac):
- betas = beta_end * np.ones(num_diffusion_timesteps, dtype=np.float64)
- warmup_time = int(num_diffusion_timesteps * warmup_frac)
- betas[:warmup_time] = np.linspace(beta_start, beta_end, warmup_time, dtype=np.float64)
- return betas
-
-
-def get_beta_schedule(beta_schedule, *, beta_start, beta_end, num_diffusion_timesteps):
- """
- This is the deprecated API for creating beta schedules.
- See get_named_beta_schedule() for the new library of schedules.
- """
- if beta_schedule == "quad":
- betas = (
- np.linspace(
- beta_start**0.5,
- beta_end**0.5,
- num_diffusion_timesteps,
- dtype=np.float64,
- )
- ** 2
- )
- elif beta_schedule == "linear":
- betas = np.linspace(beta_start, beta_end, num_diffusion_timesteps, dtype=np.float64)
- elif beta_schedule == "warmup10":
- betas = _warmup_beta(beta_start, beta_end, num_diffusion_timesteps, 0.1)
- elif beta_schedule == "warmup50":
- betas = _warmup_beta(beta_start, beta_end, num_diffusion_timesteps, 0.5)
- elif beta_schedule == "const":
- betas = beta_end * np.ones(num_diffusion_timesteps, dtype=np.float64)
- elif beta_schedule == "jsd": # 1/T, 1/(T-1), 1/(T-2), ..., 1
- betas = 1.0 / np.linspace(num_diffusion_timesteps, 1, num_diffusion_timesteps, dtype=np.float64)
- else:
- raise NotImplementedError(beta_schedule)
- assert betas.shape == (num_diffusion_timesteps,)
- return betas
-
-
-def get_named_beta_schedule(schedule_name, num_diffusion_timesteps):
- """
- Get a pre-defined beta schedule for the given name.
- The beta schedule library consists of beta schedules which remain similar
- in the limit of num_diffusion_timesteps.
- Beta schedules may be added, but should not be removed or changed once
- they are committed to maintain backwards compatibility.
- """
- if schedule_name == "linear":
- # Linear schedule from Ho et al, extended to work for any number of
- # diffusion steps.
- scale = 1000 / num_diffusion_timesteps
- return get_beta_schedule(
- "linear",
- beta_start=scale * 0.0001,
- beta_end=scale * 0.02,
- num_diffusion_timesteps=num_diffusion_timesteps,
- )
- elif schedule_name == "squaredcos_cap_v2":
- return betas_for_alpha_bar(
- num_diffusion_timesteps,
- lambda t: math.cos((t + 0.008) / 1.008 * math.pi / 2) ** 2,
- )
- else:
- raise NotImplementedError(f"unknown beta schedule: {schedule_name}")
-
-
-def betas_for_alpha_bar(num_diffusion_timesteps, alpha_bar, max_beta=0.999):
- """
- Create a beta schedule that discretizes the given alpha_t_bar function,
- which defines the cumulative product of (1-beta) over time from t = [0,1].
- :param num_diffusion_timesteps: the number of betas to produce.
- :param alpha_bar: a lambda that takes an argument t from 0 to 1 and
- produces the cumulative product of (1-beta) up to that
- part of the diffusion process.
- :param max_beta: the maximum beta to use; use values lower than 1 to
- prevent singularities.
- """
- betas = []
- for i in range(num_diffusion_timesteps):
- t1 = i / num_diffusion_timesteps
- t2 = (i + 1) / num_diffusion_timesteps
- betas.append(min(1 - alpha_bar(t2) / alpha_bar(t1), max_beta))
- return np.array(betas)
-
-
-class GaussianDiffusion:
- """
- Utilities for training and sampling diffusion models.
- Original ported from this codebase:
- https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/diffusion_utils_2.py#L42
- :param betas: a 1-D numpy array of betas for each diffusion timestep,
- starting at T and going to 1.
- """
-
- def __init__(self, *, betas, model_mean_type, model_var_type, loss_type):
- self.model_mean_type = model_mean_type
- self.model_var_type = model_var_type
- self.loss_type = loss_type
-
- # Use float64 for accuracy.
- betas = np.array(betas, dtype=np.float64)
- self.betas = betas
- assert len(betas.shape) == 1, "betas must be 1-D"
- assert (betas > 0).all() and (betas <= 1).all()
-
- self.num_timesteps = int(betas.shape[0])
-
- alphas = 1.0 - betas
- self.alphas_cumprod = np.cumprod(alphas, axis=0)
- self.alphas_cumprod_prev = np.append(1.0, self.alphas_cumprod[:-1])
- self.alphas_cumprod_next = np.append(self.alphas_cumprod[1:], 0.0)
- assert self.alphas_cumprod_prev.shape == (self.num_timesteps,)
-
- # calculations for diffusion q(x_t | x_{t-1}) and others
- self.sqrt_alphas_cumprod = np.sqrt(self.alphas_cumprod)
- self.sqrt_one_minus_alphas_cumprod = np.sqrt(1.0 - self.alphas_cumprod)
- self.log_one_minus_alphas_cumprod = np.log(1.0 - self.alphas_cumprod)
- self.sqrt_recip_alphas_cumprod = np.sqrt(1.0 / self.alphas_cumprod)
- self.sqrt_recipm1_alphas_cumprod = np.sqrt(1.0 / self.alphas_cumprod - 1)
-
- # calculations for posterior q(x_{t-1} | x_t, x_0)
- self.posterior_variance = betas * (1.0 - self.alphas_cumprod_prev) / (1.0 - self.alphas_cumprod)
- # below: log calculation clipped because the posterior variance is 0 at the beginning of the diffusion chain
- self.posterior_log_variance_clipped = (
- np.log(np.append(self.posterior_variance[1], self.posterior_variance[1:]))
- if len(self.posterior_variance) > 1
- else np.array([])
- )
-
- self.posterior_mean_coef1 = betas * np.sqrt(self.alphas_cumprod_prev) / (1.0 - self.alphas_cumprod)
- self.posterior_mean_coef2 = (1.0 - self.alphas_cumprod_prev) * np.sqrt(alphas) / (1.0 - self.alphas_cumprod)
-
- def q_mean_variance(self, x_start, t):
- """
- Get the distribution q(x_t | x_0).
- :param x_start: the [N x T x C x ...] tensor of noiseless inputs.
- :param t: the number of diffusion steps (minus 1). Here, 0 means one step.
- :return: A tuple (mean, variance, log_variance), all of x_start's shape.
- """
- mean = _extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start
- variance = _extract_into_tensor(1.0 - self.alphas_cumprod, t, x_start.shape)
- log_variance = _extract_into_tensor(self.log_one_minus_alphas_cumprod, t, x_start.shape)
- return mean, variance, log_variance
-
- def q_sample(self, x_start, t, noise=None):
- """
- Diffuse the data for a given number of diffusion steps.
- In other words, sample from q(x_t | x_0).
- :param x_start: the initial data batch.
- :param t: the number of diffusion steps (minus 1). Here, 0 means one step.
- :param noise: if specified, the split-out normal noise.
- :return: A noisy version of x_start.
- """
- if noise is None:
- noise = th.randn_like(x_start)
- assert noise.shape == x_start.shape
- return (
- _extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start
- + _extract_into_tensor(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise
- )
-
- def q_posterior_mean_variance(self, x_start, x_t, t):
- """
- Compute the mean and variance of the diffusion posterior:
- q(x_{t-1} | x_t, x_0)
- """
- assert x_start.shape == x_t.shape
- posterior_mean = (
- _extract_into_tensor(self.posterior_mean_coef1, t, x_t.shape) * x_start
- + _extract_into_tensor(self.posterior_mean_coef2, t, x_t.shape) * x_t
- )
- posterior_variance = _extract_into_tensor(self.posterior_variance, t, x_t.shape)
- posterior_log_variance_clipped = _extract_into_tensor(self.posterior_log_variance_clipped, t, x_t.shape)
- assert (
- posterior_mean.shape[0]
- == posterior_variance.shape[0]
- == posterior_log_variance_clipped.shape[0]
- == x_start.shape[0]
- )
- return posterior_mean, posterior_variance, posterior_log_variance_clipped
-
- def p_mean_variance(self, model, x, t, clip_denoised=True, denoised_fn=None, model_kwargs=None):
- """
- Apply the model to get p(x_{t-1} | x_t), as well as a prediction of
- the initial x, x_0.
- :param model: the model, which takes a signal and a batch of timesteps
- as input.
- :param x: the [N x T x C x ...] tensor at time t.
- :param t: a 1-D Tensor of timesteps.
- :param clip_denoised: if True, clip the denoised signal into [-1, 1].
- :param denoised_fn: if not None, a function which applies to the
- x_start prediction before it is used to sample. Applies before
- clip_denoised.
- :param model_kwargs: if not None, a dict of extra keyword arguments to
- pass to the model. This can be used for conditioning.
- :return: a dict with the following keys:
- - 'mean': the model mean output.
- - 'variance': the model variance output.
- - 'log_variance': the log of 'variance'.
- - 'pred_xstart': the prediction for x_0.
- """
- if model_kwargs is None:
- model_kwargs = {}
-
- B, S, C = x.shape[:3]
- assert t.shape == (B,)
- model_output = model(x, t, **model_kwargs)
- if isinstance(model_output, tuple):
- model_output, extra = model_output
- else:
- extra = None
-
- if self.model_var_type in [ModelVarType.LEARNED, ModelVarType.LEARNED_RANGE]:
- assert model_output.shape == (B, S, C * 2, *x.shape[3:])
- model_output, model_var_values = th.split(model_output, C, dim=2)
- min_log = _extract_into_tensor(self.posterior_log_variance_clipped, t, x.shape)
- max_log = _extract_into_tensor(np.log(self.betas), t, x.shape)
- # The model_var_values is [-1, 1] for [min_var, max_var].
- frac = (model_var_values + 1) / 2
- model_log_variance = frac * max_log + (1 - frac) * min_log
- model_variance = th.exp(model_log_variance)
- else:
- model_variance, model_log_variance = {
- # for fixedlarge, we set the initial (log-)variance like so
- # to get a better decoder log likelihood.
- ModelVarType.FIXED_LARGE: (
- np.append(self.posterior_variance[1], self.betas[1:]),
- np.log(np.append(self.posterior_variance[1], self.betas[1:])),
- ),
- ModelVarType.FIXED_SMALL: (
- self.posterior_variance,
- self.posterior_log_variance_clipped,
- ),
- }[self.model_var_type]
- model_variance = _extract_into_tensor(model_variance, t, x.shape)
- model_log_variance = _extract_into_tensor(model_log_variance, t, x.shape)
-
- def process_xstart(x):
- if denoised_fn is not None:
- x = denoised_fn(x)
- if clip_denoised:
- return x.clamp(-1, 1)
- return x
-
- if self.model_mean_type == ModelMeanType.START_X:
- pred_xstart = process_xstart(model_output)
- else:
- pred_xstart = process_xstart(self._predict_xstart_from_eps(x_t=x, t=t, eps=model_output))
- model_mean, _, _ = self.q_posterior_mean_variance(x_start=pred_xstart, x_t=x, t=t)
-
- assert model_mean.shape == model_log_variance.shape == pred_xstart.shape == x.shape
- return {
- "mean": model_mean,
- "variance": model_variance,
- "log_variance": model_log_variance,
- "pred_xstart": pred_xstart,
- "extra": extra,
- }
-
- def _predict_xstart_from_eps(self, x_t, t, eps):
- assert x_t.shape == eps.shape
- return (
- _extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x_t.shape) * x_t
- - _extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x_t.shape) * eps
- )
-
- def _predict_eps_from_xstart(self, x_t, t, pred_xstart):
- return (
- _extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x_t.shape) * x_t - pred_xstart
- ) / _extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x_t.shape)
-
- def condition_mean(self, cond_fn, p_mean_var, x, t, model_kwargs=None):
- """
- Compute the mean for the previous step, given a function cond_fn that
- computes the gradient of a conditional log probability with respect to
- x. In particular, cond_fn computes grad(log(p(y|x))), and we want to
- condition on y.
- This uses the conditioning strategy from Sohl-Dickstein et al. (2015).
- """
- gradient = cond_fn(x, t, **model_kwargs)
- new_mean = p_mean_var["mean"].float() + p_mean_var["variance"] * gradient.float()
- return new_mean
-
- def condition_score(self, cond_fn, p_mean_var, x, t, model_kwargs=None):
- """
- Compute what the p_mean_variance output would have been, should the
- model's score function be conditioned by cond_fn.
- See condition_mean() for details on cond_fn.
- Unlike condition_mean(), this instead uses the conditioning strategy
- from Song et al (2020).
- """
- alpha_bar = _extract_into_tensor(self.alphas_cumprod, t, x.shape)
-
- eps = self._predict_eps_from_xstart(x, t, p_mean_var["pred_xstart"])
- eps = eps - (1 - alpha_bar).sqrt() * cond_fn(x, t, **model_kwargs)
-
- out = p_mean_var.copy()
- out["pred_xstart"] = self._predict_xstart_from_eps(x, t, eps)
- out["mean"], _, _ = self.q_posterior_mean_variance(x_start=out["pred_xstart"], x_t=x, t=t)
- return out
-
- def p_sample(
- self,
- model,
- x,
- t,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- ):
- """
- Sample x_{t-1} from the model at the given timestep.
- :param model: the model to sample from.
- :param x: the current tensor at x_{t-1}.
- :param t: the value of t, starting at 0 for the first diffusion step.
- :param clip_denoised: if True, clip the x_start prediction to [-1, 1].
- :param denoised_fn: if not None, a function which applies to the
- x_start prediction before it is used to sample.
- :param cond_fn: if not None, this is a gradient function that acts
- similarly to the model.
- :param model_kwargs: if not None, a dict of extra keyword arguments to
- pass to the model. This can be used for conditioning.
- :return: a dict containing the following keys:
- - 'sample': a random sample from the model.
- - 'pred_xstart': a prediction of x_0.
- """
- out = self.p_mean_variance(
- model,
- x,
- t,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- model_kwargs=model_kwargs,
- )
- noise = th.randn_like(x)
- nonzero_mask = (t != 0).float().view(-1, *([1] * (len(x.shape) - 1))) # no noise when t == 0
- if cond_fn is not None:
- out["mean"] = self.condition_mean(cond_fn, out, x, t, model_kwargs=model_kwargs)
- sample = out["mean"] + nonzero_mask * th.exp(0.5 * out["log_variance"]) * noise
- return {"sample": sample, "pred_xstart": out["pred_xstart"]}
-
- def p_sample_loop(
- self,
- model,
- shape,
- noise=None,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- device=None,
- progress=False,
- ):
- """
- Generate samples from the model.
- :param model: the model module.
- :param shape: the shape of the samples, (N, T, C, H, W).
- :param noise: if specified, the noise from the encoder to sample.
- Should be of the same shape as `shape`.
- :param clip_denoised: if True, clip x_start predictions to [-1, 1].
- :param denoised_fn: if not None, a function which applies to the
- x_start prediction before it is used to sample.
- :param cond_fn: if not None, this is a gradient function that acts
- similarly to the model.
- :param model_kwargs: if not None, a dict of extra keyword arguments to
- pass to the model. This can be used for conditioning.
- :param device: if specified, the device to create the samples on.
- If not specified, use a model parameter's device.
- :param progress: if True, show a tqdm progress bar.
- :return: a non-differentiable batch of samples.
- """
- final = None
- for sample in self.p_sample_loop_progressive(
- model,
- shape,
- noise=noise,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- cond_fn=cond_fn,
- model_kwargs=model_kwargs,
- device=device,
- progress=progress,
- ):
- final = sample
- return final["sample"]
-
- def p_sample_loop_progressive(
- self,
- model,
- shape,
- noise=None,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- device=None,
- progress=False,
- ):
- """
- Generate samples from the model and yield intermediate samples from
- each timestep of diffusion.
- Arguments are the same as p_sample_loop().
- Returns a generator over dicts, where each dict is the return value of
- p_sample().
- """
- if device is None:
- device = next(model.parameters()).device
- assert isinstance(shape, (tuple, list))
- if noise is not None:
- img = noise
- else:
- img = th.randn(*shape, device=device)
- indices = list(range(self.num_timesteps))[::-1]
-
- if progress:
- # Lazy import so that we don't depend on tqdm.
- from tqdm.auto import tqdm
-
- indices = tqdm(indices)
-
- for i in indices:
- t = th.tensor([i] * shape[0], device=device)
- with th.no_grad():
- out = self.p_sample(
- model,
- img,
- t,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- cond_fn=cond_fn,
- model_kwargs=model_kwargs,
- )
- yield out
- img = out["sample"]
-
- def ddim_sample(
- self,
- model,
- x,
- t,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- eta=0.0,
- ):
- """
- Sample x_{t-1} from the model using DDIM.
- Same usage as p_sample().
- """
- out = self.p_mean_variance(
- model,
- x,
- t,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- model_kwargs=model_kwargs,
- )
- if cond_fn is not None:
- out = self.condition_score(cond_fn, out, x, t, model_kwargs=model_kwargs)
-
- # Usually our model outputs epsilon, but we re-derive it
- # in case we used x_start or x_prev prediction.
- eps = self._predict_eps_from_xstart(x, t, out["pred_xstart"])
-
- alpha_bar = _extract_into_tensor(self.alphas_cumprod, t, x.shape)
- alpha_bar_prev = _extract_into_tensor(self.alphas_cumprod_prev, t, x.shape)
- sigma = eta * th.sqrt((1 - alpha_bar_prev) / (1 - alpha_bar)) * th.sqrt(1 - alpha_bar / alpha_bar_prev)
- # Equation 12.
- noise = th.randn_like(x)
- mean_pred = out["pred_xstart"] * th.sqrt(alpha_bar_prev) + th.sqrt(1 - alpha_bar_prev - sigma**2) * eps
- nonzero_mask = (t != 0).float().view(-1, *([1] * (len(x.shape) - 1))) # no noise when t == 0
- sample = mean_pred + nonzero_mask * sigma * noise
- return {"sample": sample, "pred_xstart": out["pred_xstart"]}
-
- def ddim_reverse_sample(
- self,
- model,
- x,
- t,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- eta=0.0,
- ):
- """
- Sample x_{t+1} from the model using DDIM reverse ODE.
- """
- assert eta == 0.0, "Reverse ODE only for deterministic path"
- out = self.p_mean_variance(
- model,
- x,
- t,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- model_kwargs=model_kwargs,
- )
- if cond_fn is not None:
- out = self.condition_score(cond_fn, out, x, t, model_kwargs=model_kwargs)
- # Usually our model outputs epsilon, but we re-derive it
- # in case we used x_start or x_prev prediction.
- eps = (
- _extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x.shape) * x - out["pred_xstart"]
- ) / _extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x.shape)
- alpha_bar_next = _extract_into_tensor(self.alphas_cumprod_next, t, x.shape)
-
- # Equation 12. reversed
- mean_pred = out["pred_xstart"] * th.sqrt(alpha_bar_next) + th.sqrt(1 - alpha_bar_next) * eps
-
- return {"sample": mean_pred, "pred_xstart": out["pred_xstart"]}
-
- def ddim_sample_loop(
- self,
- model,
- shape,
- noise=None,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- device=None,
- progress=False,
- eta=0.0,
- ):
- """
- Generate samples from the model using DDIM.
- Same usage as p_sample_loop().
- """
- final = None
- for sample in self.ddim_sample_loop_progressive(
- model,
- shape,
- noise=noise,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- cond_fn=cond_fn,
- model_kwargs=model_kwargs,
- device=device,
- progress=progress,
- eta=eta,
- ):
- final = sample
- return final["sample"]
-
- def ddim_sample_loop_progressive(
- self,
- model,
- shape,
- noise=None,
- clip_denoised=True,
- denoised_fn=None,
- cond_fn=None,
- model_kwargs=None,
- device=None,
- progress=False,
- eta=0.0,
- ):
- """
- Use DDIM to sample from the model and yield intermediate samples from
- each timestep of DDIM.
- Same usage as p_sample_loop_progressive().
- """
- if device is None:
- device = next(model.parameters()).device
- assert isinstance(shape, (tuple, list))
- if noise is not None:
- img = noise
- else:
- img = th.randn(*shape, device=device)
- indices = list(range(self.num_timesteps))[::-1]
-
- if progress:
- # Lazy import so that we don't depend on tqdm.
- from tqdm.auto import tqdm
-
- indices = tqdm(indices)
-
- for i in indices:
- t = th.tensor([i] * shape[0], device=device)
- with th.no_grad():
- out = self.ddim_sample(
- model,
- img,
- t,
- clip_denoised=clip_denoised,
- denoised_fn=denoised_fn,
- cond_fn=cond_fn,
- model_kwargs=model_kwargs,
- eta=eta,
- )
- yield out
- img = out["sample"]
-
- def _vb_terms_bpd(self, model, x_start, x_t, t, clip_denoised=True, model_kwargs=None, mask=None):
- """
- Get a term for the variational lower-bound.
- The resulting units are bits (rather than nats, as one might expect).
- This allows for comparison to other papers.
- :return: a dict with the following keys:
- - 'output': a shape [N] tensor of NLLs or KLs.
- - 'pred_xstart': the x_0 predictions.
- """
- true_mean, _, true_log_variance_clipped = self.q_posterior_mean_variance(x_start=x_start, x_t=x_t, t=t)
- out = self.p_mean_variance(model, x_t, t, clip_denoised=clip_denoised, model_kwargs=model_kwargs)
- kl = normal_kl(true_mean, true_log_variance_clipped, out["mean"], out["log_variance"])
- kl = mean_flat(kl, mask=mask) / np.log(2.0)
-
- decoder_nll = -discretized_gaussian_log_likelihood(
- x_start, means=out["mean"], log_scales=0.5 * out["log_variance"]
- )
- assert decoder_nll.shape == x_start.shape
- decoder_nll = mean_flat(decoder_nll, mask=mask) / np.log(2.0)
-
- # At the first timestep return the decoder NLL,
- # otherwise return KL(q(x_{t-1}|x_t,x_0) || p(x_{t-1}|x_t))
- output = th.where((t == 0), decoder_nll, kl)
- return {"output": output, "pred_xstart": out["pred_xstart"]}
-
- def _expand_mask(self, mask, ndim: int):
- assert mask.ndim == 2
- # [B, S] -> [B, S, ...]
- mask = mask.view(*mask.shape, *([1] * (ndim - mask.ndim)))
- return mask
-
- def training_losses(self, model, x_start, t, model_kwargs=None, noise=None, mask=None):
- """
- Compute training losses for a single timestep.
- :param model: the model to evaluate loss on.
- :param x_start: the [N x T x C x ...] tensor of inputs.
- :param t: a batch of timestep indices.
- :param model_kwargs: if not None, a dict of extra keyword arguments to
- pass to the model. This can be used for conditioning.
- :param noise: if specified, the specific Gaussian noise to try to remove.
- :return: a dict with the key "loss" containing a tensor of shape [N].
- Some mean or variance settings may also have other keys.
- """
- if model_kwargs is None:
- model_kwargs = {}
- if noise is None:
- noise = th.randn_like(x_start)
- x_t = self.q_sample(x_start, t, noise=noise)
-
- terms = {}
-
- if mask is not None:
- mask = self._expand_mask(mask, x_start.ndim)
-
- if self.loss_type == LossType.KL or self.loss_type == LossType.RESCALED_KL:
- terms["loss"] = self._vb_terms_bpd(
- model=model,
- x_start=x_start,
- x_t=x_t,
- t=t,
- clip_denoised=False,
- model_kwargs=model_kwargs,
- mask=mask,
- )["output"]
- if self.loss_type == LossType.RESCALED_KL:
- terms["loss"] *= self.num_timesteps
- elif self.loss_type == LossType.MSE or self.loss_type == LossType.RESCALED_MSE:
- model_output = model(x_t, t, **model_kwargs)
-
- if self.model_var_type in [
- ModelVarType.LEARNED,
- ModelVarType.LEARNED_RANGE,
- ]:
- B, S, C = x_t.shape[:3]
- assert model_output.shape == (B, S, C * 2, *x_t.shape[3:])
- model_output, model_var_values = th.split(model_output, C, dim=2)
- # Learn the variance using the variational bound, but don't let
- # it affect our mean prediction.
- frozen_out = th.cat([model_output.detach(), model_var_values], dim=2)
- terms["vb"] = self._vb_terms_bpd(
- model=lambda *args, r=frozen_out: r,
- x_start=x_start,
- x_t=x_t,
- t=t,
- clip_denoised=False,
- mask=mask,
- )["output"]
- if self.loss_type == LossType.RESCALED_MSE:
- # Divide by 1000 for equivalence with initial implementation.
- # Without a factor of 1/1000, the VB term hurts the MSE term.
- terms["vb"] *= self.num_timesteps / 1000.0
-
- target = {
- ModelMeanType.PREVIOUS_X: self.q_posterior_mean_variance(x_start=x_start, x_t=x_t, t=t)[0],
- ModelMeanType.START_X: x_start,
- ModelMeanType.EPSILON: noise,
- }[self.model_mean_type]
- assert model_output.shape == target.shape == x_start.shape
- terms["mse"] = mean_flat((target - model_output) ** 2, mask=mask)
- if "vb" in terms:
- terms["loss"] = terms["mse"] + terms["vb"]
- else:
- terms["loss"] = terms["mse"]
- else:
- raise NotImplementedError(self.loss_type)
-
- return terms
-
- def _prior_bpd(self, x_start):
- """
- Get the prior KL term for the variational lower-bound, measured in
- bits-per-dim.
- This term can't be optimized, as it only depends on the encoder.
- :param x_start: the [N x T x C x ...] tensor of inputs.
- :return: a batch of [N] KL values (in bits), one per batch element.
- """
- batch_size = x_start.shape[0]
- t = th.tensor([self.num_timesteps - 1] * batch_size, device=x_start.device)
- qt_mean, _, qt_log_variance = self.q_mean_variance(x_start, t)
- kl_prior = normal_kl(mean1=qt_mean, logvar1=qt_log_variance, mean2=0.0, logvar2=0.0)
- return mean_flat(kl_prior) / np.log(2.0)
-
- def calc_bpd_loop(self, model, x_start, clip_denoised=True, model_kwargs=None):
- """
- Compute the entire variational lower-bound, measured in bits-per-dim,
- as well as other related quantities.
- :param model: the model to evaluate loss on.
- :param x_start: the [N x T x C x ...] tensor of inputs.
- :param clip_denoised: if True, clip denoised samples.
- :param model_kwargs: if not None, a dict of extra keyword arguments to
- pass to the model. This can be used for conditioning.
- :return: a dict containing the following keys:
- - total_bpd: the total variational lower-bound, per batch element.
- - prior_bpd: the prior term in the lower-bound.
- - vb: an [N x T] tensor of terms in the lower-bound.
- - xstart_mse: an [N x T] tensor of x_0 MSEs for each timestep.
- - mse: an [N x T] tensor of epsilon MSEs for each timestep.
- """
- device = x_start.device
- batch_size = x_start.shape[0]
-
- vb = []
- xstart_mse = []
- mse = []
- for t in list(range(self.num_timesteps))[::-1]:
- t_batch = th.tensor([t] * batch_size, device=device)
- noise = th.randn_like(x_start)
- x_t = self.q_sample(x_start=x_start, t=t_batch, noise=noise)
- # Calculate VLB term at the current timestep
- with th.no_grad():
- out = self._vb_terms_bpd(
- model,
- x_start=x_start,
- x_t=x_t,
- t=t_batch,
- clip_denoised=clip_denoised,
- model_kwargs=model_kwargs,
- )
- vb.append(out["output"])
- xstart_mse.append(mean_flat((out["pred_xstart"] - x_start) ** 2))
- eps = self._predict_eps_from_xstart(x_t, t_batch, out["pred_xstart"])
- mse.append(mean_flat((eps - noise) ** 2))
-
- vb = th.stack(vb, dim=1)
- xstart_mse = th.stack(xstart_mse, dim=1)
- mse = th.stack(mse, dim=1)
-
- prior_bpd = self._prior_bpd(x_start)
- total_bpd = vb.sum(dim=1) + prior_bpd
- return {
- "total_bpd": total_bpd,
- "prior_bpd": prior_bpd,
- "vb": vb,
- "xstart_mse": xstart_mse,
- "mse": mse,
- }
-
-
-def _extract_into_tensor(arr, timesteps, broadcast_shape):
- """
- Extract values from a 1-D numpy array for a batch of indices.
- :param arr: the 1-D numpy array.
- :param timesteps: a tensor of indices into the array to extract.
- :param broadcast_shape: a larger shape of K dimensions with the batch
- dimension equal to the length of timesteps.
- :return: a tensor of shape [batch_size, 1, ...] where the shape has K dims.
- """
- res = th.from_numpy(arr).to(device=timesteps.device)[timesteps].float()
- while len(res.shape) < len(broadcast_shape):
- res = res[..., None]
- return res + th.zeros(broadcast_shape, device=timesteps.device)
diff --git a/open_sora/diffusion/respace.py b/open_sora/diffusion/respace.py
deleted file mode 100644
index e5754aa..0000000
--- a/open_sora/diffusion/respace.py
+++ /dev/null
@@ -1,119 +0,0 @@
-# Modified from OpenAI's diffusion repos
-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py
-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion
-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py
-
-import numpy as np
-import torch as th
-
-from .gaussian_diffusion import GaussianDiffusion
-
-
-def space_timesteps(num_timesteps, section_counts):
- """
- Create a list of timesteps to use from an original diffusion process,
- given the number of timesteps we want to take from equally-sized portions
- of the original process.
- For example, if there's 300 timesteps and the section counts are [10,15,20]
- then the first 100 timesteps are strided to be 10 timesteps, the second 100
- are strided to be 15 timesteps, and the final 100 are strided to be 20.
- If the stride is a string starting with "ddim", then the fixed striding
- from the DDIM paper is used, and only one section is allowed.
- :param num_timesteps: the number of diffusion steps in the original
- process to divide up.
- :param section_counts: either a list of numbers, or a string containing
- comma-separated numbers, indicating the step count
- per section. As a special case, use "ddimN" where N
- is a number of steps to use the striding from the
- DDIM paper.
- :return: a set of diffusion steps from the original process to use.
- """
- if isinstance(section_counts, str):
- if section_counts.startswith("ddim"):
- desired_count = int(section_counts[len("ddim") :])
- for i in range(1, num_timesteps):
- if len(range(0, num_timesteps, i)) == desired_count:
- return set(range(0, num_timesteps, i))
- raise ValueError(f"cannot create exactly {num_timesteps} steps with an integer stride")
- section_counts = [int(x) for x in section_counts.split(",")]
- size_per = num_timesteps // len(section_counts)
- extra = num_timesteps % len(section_counts)
- start_idx = 0
- all_steps = []
- for i, section_count in enumerate(section_counts):
- size = size_per + (1 if i < extra else 0)
- if size < section_count:
- raise ValueError(f"cannot divide section of {size} steps into {section_count}")
- if section_count <= 1:
- frac_stride = 1
- else:
- frac_stride = (size - 1) / (section_count - 1)
- cur_idx = 0.0
- taken_steps = []
- for _ in range(section_count):
- taken_steps.append(start_idx + round(cur_idx))
- cur_idx += frac_stride
- all_steps += taken_steps
- start_idx += size
- return set(all_steps)
-
-
-class SpacedDiffusion(GaussianDiffusion):
- """
- A diffusion process which can skip steps in a base diffusion process.
- :param use_timesteps: a collection (sequence or set) of timesteps from the
- original diffusion process to retain.
- :param kwargs: the kwargs to create the base diffusion process.
- """
-
- def __init__(self, use_timesteps, **kwargs):
- self.use_timesteps = set(use_timesteps)
- self.timestep_map = []
- self.original_num_steps = len(kwargs["betas"])
-
- base_diffusion = GaussianDiffusion(**kwargs) # pylint: disable=missing-kwoa
- last_alpha_cumprod = 1.0
- new_betas = []
- for i, alpha_cumprod in enumerate(base_diffusion.alphas_cumprod):
- if i in self.use_timesteps:
- new_betas.append(1 - alpha_cumprod / last_alpha_cumprod)
- last_alpha_cumprod = alpha_cumprod
- self.timestep_map.append(i)
- kwargs["betas"] = np.array(new_betas)
- super().__init__(**kwargs)
-
- def p_mean_variance(self, model, *args, **kwargs): # pylint: disable=signature-differs
- return super().p_mean_variance(self._wrap_model(model), *args, **kwargs)
-
- def training_losses(self, model, *args, **kwargs): # pylint: disable=signature-differs
- return super().training_losses(self._wrap_model(model), *args, **kwargs)
-
- def condition_mean(self, cond_fn, *args, **kwargs):
- return super().condition_mean(self._wrap_model(cond_fn), *args, **kwargs)
-
- def condition_score(self, cond_fn, *args, **kwargs):
- return super().condition_score(self._wrap_model(cond_fn), *args, **kwargs)
-
- def _wrap_model(self, model):
- if isinstance(model, _WrappedModel):
- return model
- return _WrappedModel(model, self.timestep_map, self.original_num_steps)
-
- def _scale_timesteps(self, t):
- # Scaling is done by the wrapped model.
- return t
-
-
-class _WrappedModel:
- def __init__(self, model, timestep_map, original_num_steps):
- self.model = model
- self.timestep_map = timestep_map
- # self.rescale_timesteps = rescale_timesteps
- self.original_num_steps = original_num_steps
-
- def __call__(self, x, ts, **kwargs):
- map_tensor = th.tensor(self.timestep_map, device=ts.device, dtype=ts.dtype)
- new_ts = map_tensor[ts]
- # if self.rescale_timesteps:
- # new_ts = new_ts.float() * (1000.0 / self.original_num_steps)
- return self.model(x, new_ts, **kwargs)
diff --git a/open_sora/diffusion/timestep_sampler.py b/open_sora/diffusion/timestep_sampler.py
deleted file mode 100644
index fdaa45a..0000000
--- a/open_sora/diffusion/timestep_sampler.py
+++ /dev/null
@@ -1,143 +0,0 @@
-# Modified from OpenAI's diffusion repos
-# GLIDE: https://github.com/openai/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py
-# ADM: https://github.com/openai/guided-diffusion/blob/main/guided_diffusion
-# IDDPM: https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py
-
-from abc import ABC, abstractmethod
-
-import numpy as np
-import torch as th
-import torch.distributed as dist
-
-
-def create_named_schedule_sampler(name, diffusion):
- """
- Create a ScheduleSampler from a library of pre-defined samplers.
- :param name: the name of the sampler.
- :param diffusion: the diffusion object to sample for.
- """
- if name == "uniform":
- return UniformSampler(diffusion)
- elif name == "loss-second-moment":
- return LossSecondMomentResampler(diffusion)
- else:
- raise NotImplementedError(f"unknown schedule sampler: {name}")
-
-
-class ScheduleSampler(ABC):
- """
- A distribution over timesteps in the diffusion process, intended to reduce
- variance of the objective.
- By default, samplers perform unbiased importance sampling, in which the
- objective's mean is unchanged.
- However, subclasses may override sample() to change how the resampled
- terms are reweighted, allowing for actual changes in the objective.
- """
-
- @abstractmethod
- def weights(self):
- """
- Get a numpy array of weights, one per diffusion step.
- The weights needn't be normalized, but must be positive.
- """
-
- def sample(self, batch_size, device):
- """
- Importance-sample timesteps for a batch.
- :param batch_size: the number of timesteps.
- :param device: the torch device to save to.
- :return: a tuple (timesteps, weights):
- - timesteps: a tensor of timestep indices.
- - weights: a tensor of weights to scale the resulting losses.
- """
- w = self.weights()
- p = w / np.sum(w)
- indices_np = np.random.choice(len(p), size=(batch_size,), p=p)
- indices = th.from_numpy(indices_np).long().to(device)
- weights_np = 1 / (len(p) * p[indices_np])
- weights = th.from_numpy(weights_np).float().to(device)
- return indices, weights
-
-
-class UniformSampler(ScheduleSampler):
- def __init__(self, diffusion):
- self.diffusion = diffusion
- self._weights = np.ones([diffusion.num_timesteps])
-
- def weights(self):
- return self._weights
-
-
-class LossAwareSampler(ScheduleSampler):
- def update_with_local_losses(self, local_ts, local_losses):
- """
- Update the reweighting using losses from a model.
- Call this method from each rank with a batch of timesteps and the
- corresponding losses for each of those timesteps.
- This method will perform synchronization to make sure all of the ranks
- maintain the exact same reweighting.
- :param local_ts: an integer Tensor of timesteps.
- :param local_losses: a 1D Tensor of losses.
- """
- batch_sizes = [th.tensor([0], dtype=th.int32, device=local_ts.device) for _ in range(dist.get_world_size())]
- dist.all_gather(
- batch_sizes,
- th.tensor([len(local_ts)], dtype=th.int32, device=local_ts.device),
- )
-
- # Pad all_gather batches to be the maximum batch size.
- batch_sizes = [x.item() for x in batch_sizes]
- max_bs = max(batch_sizes)
-
- timestep_batches = [th.zeros(max_bs).to(local_ts) for bs in batch_sizes]
- loss_batches = [th.zeros(max_bs).to(local_losses) for bs in batch_sizes]
- dist.all_gather(timestep_batches, local_ts)
- dist.all_gather(loss_batches, local_losses)
- timesteps = [x.item() for y, bs in zip(timestep_batches, batch_sizes) for x in y[:bs]]
- losses = [x.item() for y, bs in zip(loss_batches, batch_sizes) for x in y[:bs]]
- self.update_with_all_losses(timesteps, losses)
-
- @abstractmethod
- def update_with_all_losses(self, ts, losses):
- """
- Update the reweighting using losses from a model.
- Sub-classes should override this method to update the reweighting
- using losses from the model.
- This method directly updates the reweighting without synchronizing
- between workers. It is called by update_with_local_losses from all
- ranks with identical arguments. Thus, it should have deterministic
- behavior to maintain state across workers.
- :param ts: a list of int timesteps.
- :param losses: a list of float losses, one per timestep.
- """
-
-
-class LossSecondMomentResampler(LossAwareSampler):
- def __init__(self, diffusion, history_per_term=10, uniform_prob=0.001):
- self.diffusion = diffusion
- self.history_per_term = history_per_term
- self.uniform_prob = uniform_prob
- self._loss_history = np.zeros([diffusion.num_timesteps, history_per_term], dtype=np.float64)
- self._loss_counts = np.zeros([diffusion.num_timesteps], dtype=np.int)
-
- def weights(self):
- if not self._warmed_up():
- return np.ones([self.diffusion.num_timesteps], dtype=np.float64)
- weights = np.sqrt(np.mean(self._loss_history**2, axis=-1))
- weights /= np.sum(weights)
- weights *= 1 - self.uniform_prob
- weights += self.uniform_prob / len(weights)
- return weights
-
- def update_with_all_losses(self, ts, losses):
- for t, loss in zip(ts, losses):
- if self._loss_counts[t] == self.history_per_term:
- # Shift out the oldest loss term.
- self._loss_history[t, :-1] = self._loss_history[t, 1:]
- self._loss_history[t, -1] = loss
- else:
- self._loss_history[t, self._loss_counts[t]] = loss
- self._loss_counts[t] += 1
-
- def _warmed_up(self):
- return (self._loss_counts == self.history_per_term).all()
diff --git a/open_sora/modeling/__init__.py b/open_sora/modeling/__init__.py
deleted file mode 100644
index f1355ce..0000000
--- a/open_sora/modeling/__init__.py
+++ /dev/null
@@ -1,4 +0,0 @@
-from .dit import DiT, DiT_models
-from .latte import LatteT2V
-
-__all__ = ["DiT_models", "DiT", "LatteT2V"]
diff --git a/open_sora/modeling/dit/__init__.py b/open_sora/modeling/dit/__init__.py
deleted file mode 100644
index 73813b5..0000000
--- a/open_sora/modeling/dit/__init__.py
+++ /dev/null
@@ -1,3 +0,0 @@
-from .dit import SUPPORTED_MODEL_ARCH, SUPPORTED_SEQ_PARALLEL_MODES, DiT, DiT_models
-
-__all__ = ["DiT_models", "DiT", "SUPPORTED_SEQ_PARALLEL_MODES", "SUPPORTED_MODEL_ARCH"]
diff --git a/open_sora/modeling/dit/attn.py b/open_sora/modeling/dit/attn.py
deleted file mode 100644
index 6ff95f2..0000000
--- a/open_sora/modeling/dit/attn.py
+++ /dev/null
@@ -1,276 +0,0 @@
-from typing import Optional
-
-import torch
-import torch.distributed as dist
-import torch.nn as nn
-import torch.nn.functional as F
-from colossalai.logging import get_dist_logger
-from colossalai.shardformer.layer._operation import gather_forward_split_backward
-
-from open_sora.utils.comm import all_to_all, async_all_gather_proj_for_two
-
-
-class CrossAttention(nn.Module):
- r"""
- A cross attention layer.
-
- Parameters:
- query_dim (`int`): The number of channels in the query.
- cross_attention_dim (`int`, *optional*):
- The number of channels in the context. If not given, defaults to `query_dim`.
- num_heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention.
- head_dim (`int`, *optional*, defaults to 64): The number of channels in each head.
- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
- bias (`bool`, *optional*, defaults to False):
- Set to `True` for the query, key, and value linear layers to contain a bias parameter.
- """
-
- def __init__(
- self,
- query_dim: int,
- cross_attention_dim: Optional[int] = None,
- num_heads: int = 8,
- head_dim: int = 64,
- dropout: float = 0.0,
- bias=False,
- sdpa=True,
- ):
- super().__init__()
- self.hidden_size = head_dim * num_heads
- cross_attention_dim = cross_attention_dim if cross_attention_dim is not None else query_dim
-
- self.scale = head_dim**-0.5
- self.num_heads = num_heads
- self.head_dim = head_dim
- self.sdpa = sdpa
-
- self.to_q = nn.Linear(query_dim, self.hidden_size, bias=bias)
- self.to_k = nn.Linear(cross_attention_dim, self.hidden_size, bias=bias)
- self.to_v = nn.Linear(cross_attention_dim, self.hidden_size, bias=bias)
-
- self.to_out = nn.Sequential(nn.Linear(self.hidden_size, query_dim), nn.Dropout(dropout))
-
- def forward(self, hidden_states, context=None, mask=None):
- bsz, q_len, _ = hidden_states.shape
-
- query = self.to_q(hidden_states)
- context = context if context is not None else hidden_states
- kv_seq_len = context.shape[1]
- key = self.to_k(context)
- value = self.to_v(context)
-
- # [B, S, H, D]
- query = query.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
- key = key.view(bsz, kv_seq_len, self.num_heads, self.head_dim).transpose(1, 2)
- value = value.view(bsz, kv_seq_len, self.num_heads, self.head_dim).transpose(1, 2)
-
- if mask is not None:
- assert mask.shape == (bsz, 1, q_len, kv_seq_len)
- if self.sdpa:
- attn_output = F.scaled_dot_product_attention(query, key, value, attn_mask=mask, scale=self.scale)
- else:
- attn_weights = torch.matmul(query, key.transpose(2, 3)) / self.scale
- assert attn_weights.shape == (bsz, self.num_heads, q_len, kv_seq_len)
- if mask is not None:
- attn_weights = attn_weights + mask
- attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
- attn_output = torch.matmul(attn_weights, value)
- assert attn_output.shape == (bsz, self.num_heads, q_len, self.head_dim)
- attn_output = attn_output.transpose(1, 2).contiguous()
- attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
- attn_output = self.to_out(attn_output)
- return attn_output
-
-
-class SeqParallelCrossAttention(CrossAttention):
- r"""
- A cross attention layer.
-
- Parameters:
- query_dim (`int`): The number of channels in the query.
- cross_attention_dim (`int`, *optional*):
- The number of channels in the context. If not given, defaults to `query_dim`.
- num_heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention.
- head_dim (`int`, *optional*, defaults to 64): The number of channels in each head.
- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
- bias (`bool`, *optional*, defaults to False):
- Set to `True` for the query, key, and value linear layers to contain a bias parameter.
- """
-
- def __init__(
- self,
- query_dim: int,
- cross_attention_dim: Optional[int] = None,
- num_heads: int = 8,
- head_dim: int = 64,
- dropout: float = 0.0,
- bias=False,
- sdpa=True,
- seq_parallel_group=None,
- ):
- super().__init__(
- query_dim,
- cross_attention_dim,
- num_heads,
- head_dim,
- dropout,
- bias,
- sdpa,
- )
- self.seq_parallel_group = seq_parallel_group
- self.seq_parallel_size = dist.get_world_size(self.seq_parallel_group) if seq_parallel_group is not None else 1
- assert self.num_heads % self.seq_parallel_size == 0
-
- def forward(self, hidden_states, context=None, mask=None):
- bsz, q_len, _ = hidden_states.shape
-
- query = self.to_q(hidden_states)
- context = context if context is not None else hidden_states
- kv_seq_len = context.shape[1]
- key = self.to_k(context)
- value = self.to_v(context)
-
- # [B, S/P, H] -> [B, S, H/P]
- num_heads_parallel = self.num_heads // self.seq_parallel_size
- hidden_size_parallel = self.hidden_size // self.seq_parallel_size
- if self.seq_parallel_size > 1:
- query = all_to_all(query, self.seq_parallel_group, scatter_dim=2, gather_dim=1)
- key = all_to_all(key, self.seq_parallel_group, scatter_dim=2, gather_dim=1)
- value = all_to_all(value, self.seq_parallel_group, scatter_dim=2, gather_dim=1)
-
- q_len *= self.seq_parallel_size
- kv_seq_len *= self.seq_parallel_size
-
- # [B, S, H/P] -> [B, S, N/P, D] -> [B, N/P, S, D]
- query = query.view(bsz, q_len, num_heads_parallel, self.head_dim).transpose(1, 2)
- key = key.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)
- value = value.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)
-
- if mask is not None:
- assert mask.shape == (bsz, 1, q_len, kv_seq_len)
- if self.sdpa:
- attn_output = F.scaled_dot_product_attention(query, key, value, attn_mask=mask, scale=self.scale)
- else:
- attn_weights = torch.matmul(query, key.transpose(2, 3)) / self.scale
- assert attn_weights.shape == (bsz, num_heads_parallel, q_len, kv_seq_len)
- if mask is not None:
- attn_weights = attn_weights + mask
- attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
- attn_output = torch.matmul(attn_weights, value)
- assert attn_output.shape == (bsz, num_heads_parallel, q_len, self.head_dim)
- attn_output = attn_output.transpose(1, 2).contiguous()
- attn_output = attn_output.reshape(bsz, q_len, hidden_size_parallel)
- # [B, S, H/P] -> [B, S/P, H]
- if self.seq_parallel_size > 1:
- attn_output = all_to_all(attn_output, self.seq_parallel_group, scatter_dim=1, gather_dim=2)
- attn_output = self.to_out(attn_output)
- return attn_output
-
-
-class FastSeqParallelCrossAttention(SeqParallelCrossAttention):
- def __init__(
- self,
- query_dim: int,
- cross_attention_dim: Optional[int] = None,
- num_heads: int = 8,
- head_dim: int = 64,
- dropout: float = 0,
- bias=False,
- sdpa=True,
- seq_parallel_group=None,
- overlap=False,
- ):
- super().__init__(
- query_dim,
- cross_attention_dim,
- num_heads,
- head_dim,
- dropout,
- bias,
- sdpa,
- seq_parallel_group,
- )
- self.seq_parallel_rank = dist.get_rank(self.seq_parallel_group) if seq_parallel_group is not None else 0
- self.sequence_parallel_param_slice = slice(
- self.hidden_size // self.seq_parallel_size * self.seq_parallel_rank,
- self.hidden_size // self.seq_parallel_size * (self.seq_parallel_rank + 1),
- )
- if overlap and self.seq_parallel_size != 2:
- logger = get_dist_logger()
- logger.warning(
- "FastSeqParallelCrossAttention only supports overlap with seq_parallel_size=2. Fallback to non-overlap",
- ranks=[0],
- )
- overlap = False
- self.overlap = overlap
-
- def _get_sliced_params(self, proj_layer: nn.Linear):
- bias = bias = proj_layer.bias[self.sequence_parallel_param_slice] if proj_layer.bias is not None else None
- return proj_layer.weight[self.sequence_parallel_param_slice], bias
-
- def _proj(self, x: torch.Tensor, proj_layer: nn.Linear):
- return F.linear(
- x,
- *self._get_sliced_params(proj_layer),
- )
-
- def forward(self, hidden_states, context=None, mask=None):
- bsz, q_len, _ = hidden_states.shape
-
- context = context if context is not None else hidden_states
- kv_seq_len = context.shape[1]
- if self.seq_parallel_size > 1:
- if self.overlap and self.seq_parallel_size == 2:
- query, key, value = async_all_gather_proj_for_two(
- hidden_states,
- context,
- *self._get_sliced_params(self.to_q),
- *self._get_sliced_params(self.to_k),
- *self._get_sliced_params(self.to_v),
- dim=1,
- process_group=self.seq_parallel_group,
- sp_size=self.seq_parallel_size,
- sp_rank=self.seq_parallel_rank,
- )
- else:
- # [B, S/P, H] -> [B, S, H]
- hidden_states = gather_forward_split_backward(hidden_states, 1, self.seq_parallel_group)
- context = gather_forward_split_backward(context, 1, self.seq_parallel_group)
- query = self._proj(hidden_states, self.to_q)
- key = self._proj(context, self.to_k)
- value = self._proj(context, self.to_v)
- else:
- query = self.to_q(hidden_states)
- key = self.to_k(context)
- value = self.to_v(context)
- # output is [B, S, H/P]
-
- num_heads_parallel = self.num_heads // self.seq_parallel_size
- hidden_size_parallel = self.hidden_size // self.seq_parallel_size
- q_len *= self.seq_parallel_size
- kv_seq_len *= self.seq_parallel_size
-
- # [B, S, H/P] -> [B, S, N/P, D] -> [B, N/P, S, D]
- query = query.view(bsz, q_len, num_heads_parallel, self.head_dim).transpose(1, 2)
- key = key.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)
- value = value.view(bsz, kv_seq_len, num_heads_parallel, self.head_dim).transpose(1, 2)
-
- if mask is not None:
- assert mask.shape == (bsz, 1, q_len, kv_seq_len)
- if self.sdpa:
- attn_output = F.scaled_dot_product_attention(query, key, value, attn_mask=mask, scale=self.scale)
- else:
- attn_weights = torch.matmul(query, key.transpose(2, 3)) / self.scale
- assert attn_weights.shape == (bsz, num_heads_parallel, q_len, kv_seq_len)
- if mask is not None:
- attn_weights = attn_weights + mask
- attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
- attn_output = torch.matmul(attn_weights, value)
- assert attn_output.shape == (bsz, num_heads_parallel, q_len, self.head_dim)
- attn_output = attn_output.transpose(1, 2).contiguous()
- attn_output = attn_output.reshape(bsz, q_len, hidden_size_parallel)
- # [B, S, H/P] -> [B, S/P, H]
- if self.seq_parallel_size > 1:
- attn_output = all_to_all(attn_output, self.seq_parallel_group, scatter_dim=1, gather_dim=2)
- attn_output = self.to_out(attn_output)
- return attn_output
diff --git a/open_sora/modeling/dit/dit.py b/open_sora/modeling/dit/dit.py
deleted file mode 100644
index 6b7bd00..0000000
--- a/open_sora/modeling/dit/dit.py
+++ /dev/null
@@ -1,659 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-# --------------------------------------------------------
-# References:
-# GLIDE: https://github.com/openai/glide-text2im
-# MAE: https://github.com/facebookresearch/mae/blob/main/models_mae.py
-# --------------------------------------------------------
-
-import math
-from typing import Callable, Optional
-
-import numpy as np
-import torch
-import torch.distributed as dist
-import torch.nn as nn
-import torch.nn.functional as F
-from timm.models.vision_transformer import Mlp
-
-from open_sora.utils.comm import gather_seq, split_seq
-
-from .attn import (
- CrossAttention,
- FastSeqParallelCrossAttention,
- SeqParallelCrossAttention,
-)
-
-SUPPORTED_SEQ_PARALLEL_MODES = ["ulysses", "fastseq"]
-SUPPORTED_MODEL_ARCH = ["adaln", "cross-attn", "token-concat"]
-
-
-def modulate(x, shift, scale):
- return x * (1 + scale.unsqueeze(1)) + shift.unsqueeze(1)
-
-
-#################################################################################
-# Embedding Layers for Timesteps and Class Labels #
-#################################################################################
-
-
-class TimestepEmbedder(nn.Module):
- """
- Embeds scalar timesteps into vector representations.
- """
-
- def __init__(self, hidden_size, frequency_embedding_size=256):
- super().__init__()
- self.mlp = nn.Sequential(
- nn.Linear(frequency_embedding_size, hidden_size, bias=True),
- nn.SiLU(),
- nn.Linear(hidden_size, hidden_size, bias=True),
- )
- self.frequency_embedding_size = frequency_embedding_size
-
- @staticmethod
- def timestep_embedding(t, dim, max_period=10000):
- """
- Create sinusoidal timestep embeddings.
- :param t: a 1-D Tensor of N indices, one per batch element.
- These may be fractional.
- :param dim: the dimension of the output.
- :param max_period: controls the minimum frequency of the embeddings.
- :return: an (N, D) Tensor of positional embeddings.
- """
- # https://github.com/openai/glide-text2im/blob/main/glide_text2im/nn.py
- half = dim // 2
- freqs = torch.exp(
- -math.log(max_period)
- * torch.arange(start=0, end=half, dtype=torch.float32)
- / half
- ).to(device=t.device)
- args = t[:, None].float() * freqs[None]
- embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)
- if dim % 2:
- embedding = torch.cat(
- [embedding, torch.zeros_like(embedding[:, :1])], dim=-1
- )
- return embedding
-
- def forward(self, t):
- t_freq = self.timestep_embedding(t, self.frequency_embedding_size).to(
- self.mlp[0].weight.dtype
- )
- t_emb = self.mlp(t_freq)
- return t_emb
-
-
-class LabelEmbedder(nn.Module):
- """
- Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance.
- """
-
- def __init__(self, num_classes, hidden_size, dropout_prob):
- super().__init__()
- use_cfg_embedding = dropout_prob > 0
- self.embedding_table = nn.Embedding(
- num_classes + use_cfg_embedding, hidden_size
- )
- self.num_classes = num_classes
- self.dropout_prob = dropout_prob
-
- def token_drop(self, labels, force_drop_ids=None):
- """
- Drops labels to enable classifier-free guidance.
- """
- if force_drop_ids is None:
- drop_ids = (
- torch.rand(labels.shape[0], device=labels.device) < self.dropout_prob
- )
- else:
- drop_ids = force_drop_ids == 1
- labels = torch.where(drop_ids, self.num_classes, labels)
- return labels
-
- def forward(self, labels, train, force_drop_ids=None):
- use_dropout = self.dropout_prob > 0
- if (train and use_dropout) or (force_drop_ids is not None):
- labels = self.token_drop(labels, force_drop_ids)
- embeddings = self.embedding_table(labels)
- return embeddings
-
-
-class PatchEmbedder(nn.Module):
- """Patch Embedding Layer for flat 4D video tensors."""
-
- def __init__(
- self,
- patch_size: int = 16,
- in_chans: int = 3,
- embed_dim: int = 768,
- norm_layer: Optional[Callable] = None,
- bias: bool = True,
- ) -> None:
- super().__init__()
- self.patch_size = patch_size
- self.proj = nn.Conv2d(
- in_chans, embed_dim, kernel_size=patch_size, stride=patch_size, bias=bias
- )
- self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()
-
- def forward(self, x: torch.Tensor) -> torch.Tensor:
- # [B, S, C, P, P] -> [B, S, C*P*P]
- x = x.view(*x.shape[:2], -1)
- out = F.linear(
- x, self.proj.weight.view(self.proj.weight.shape[0], -1), self.proj.bias
- )
- out = self.norm(out)
- # [B, S, H]
- return out
-
-
-class TextEmbedder(nn.Module):
- def __init__(
- self,
- in_features: int,
- embed_dim: int = 768,
- bias: bool = True,
- dropout_prob: float = 0.0,
- use_proj: bool = True,
- ) -> None:
- super().__init__()
- self.dropout_prob = dropout_prob
- self.use_proj = use_proj
- if self.use_proj:
- self.proj = nn.Linear(in_features, embed_dim, bias=bias)
-
- def drop_sample(self, x: torch.Tensor) -> torch.Tensor:
- drop_ids = (
- torch.rand(x.shape[0], *([1] * (x.ndim - 1)), device=x.device)
- < self.dropout_prob
- )
- x = torch.where(drop_ids, torch.zeros_like(x), x)
- return x
-
- def forward(self, x: torch.Tensor) -> torch.Tensor:
- use_dropout = self.dropout_prob > 0
- if self.training and use_dropout:
- x = self.drop_sample(x)
- if self.use_proj:
- # [B, S, C] -> [B, S, H]
- x = self.proj(x)
- return x
-
-
-class PositionEmbedding(nn.Module):
- def __init__(self, dim: int, max_position_embeddings=262114) -> None:
- super().__init__()
- self.dim = dim
- self.max_position_embeddings = max_position_embeddings
- self._set_pos_embed_cache(max_position_embeddings)
-
- def _set_pos_embed_cache(self, seq_len: int, device="cpu", dtype=torch.float):
- self.max_seq_len_cached = seq_len
- pos_embed = get_2d_sincos_pos_embed(self.dim, math.ceil(seq_len**0.5))
- pos_embed = torch.from_numpy(pos_embed).to(device=device, dtype=dtype)
- # [S, H]
- self.register_buffer("pos_embed_cache", pos_embed, persistent=False)
-
- def forward(self, x: torch.Tensor) -> torch.Tensor:
- # [B, S, H]
- seq_len = x.shape[1]
- if seq_len > self.max_seq_len_cached:
- self._set_pos_embed_cache(seq_len, x.device, x.dtype)
- pos_embed = self.pos_embed_cache[None, :seq_len]
- return pos_embed
-
-
-#################################################################################
-# Core DiT Model #
-#################################################################################
-
-
-class DiTBlock(nn.Module):
- """
- A DiT block with adaptive layer norm zero (adaLN-Zero) conditioning.
- """
-
- def __init__(
- self,
- hidden_size,
- num_heads,
- cross_attention_dim=None,
- mlp_ratio=4.0,
- seq_parallel_group=None,
- seq_parallel_mode="ulysses",
- seq_parallel_overlap=False,
- ):
- super().__init__()
- self.norm1 = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
- attn_kwargs = {}
- if seq_parallel_group is not None:
- attn_kwargs["seq_parallel_group"] = seq_parallel_group
- if seq_parallel_mode == "ulysses":
- attn_cls = SeqParallelCrossAttention
- elif seq_parallel_mode == "fastseq":
- attn_cls = FastSeqParallelCrossAttention
- attn_kwargs["overlap"] = seq_parallel_overlap
- else:
- raise ValueError(
- f"seq_parallel_mode must be one of {SUPPORTED_SEQ_PARALLEL_MODES}"
- )
- else:
- attn_cls = CrossAttention
- self.attn = attn_cls(
- query_dim=hidden_size,
- cross_attention_dim=cross_attention_dim,
- num_heads=num_heads,
- head_dim=hidden_size // num_heads,
- bias=True,
- sdpa=True,
- **attn_kwargs,
- )
-
- self.norm2 = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
- mlp_hidden_dim = int(hidden_size * mlp_ratio)
- approx_gelu = lambda: nn.GELU(approximate="tanh")
- self.mlp = Mlp(
- in_features=hidden_size,
- hidden_features=mlp_hidden_dim,
- act_layer=approx_gelu,
- drop=0,
- )
- self.mlp = Mlp(
- in_features=hidden_size,
- hidden_features=mlp_hidden_dim,
- act_layer=approx_gelu,
- drop=0,
- )
- self.adaLN_modulation = nn.Sequential(
- nn.SiLU(), nn.Linear(hidden_size, 6 * hidden_size, bias=True)
- )
-
- def forward(self, x, attention_mask, t, context=None):
- shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (
- self.adaLN_modulation(t).chunk(6, dim=1)
- )
- x = x + gate_msa.unsqueeze(1) * self.attn(
- modulate(self.norm1(x), shift_msa, scale_msa), context, attention_mask
- )
- x = x + gate_mlp.unsqueeze(1) * self.mlp(
- modulate(self.norm2(x), shift_mlp, scale_mlp)
- )
- return x
-
-
-class FinalLayer(nn.Module):
- """
- The final layer of DiT.
- """
-
- def __init__(self, hidden_size, patch_size, out_channels):
- super().__init__()
- self.norm_final = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
- self.linear = nn.Linear(
- hidden_size, patch_size * patch_size * out_channels, bias=True
- )
- self.patch_size = patch_size
- self.adaLN_modulation = nn.Sequential(
- nn.SiLU(), nn.Linear(hidden_size, 2 * hidden_size, bias=True)
- )
-
- def unpatchify(self, x):
- b, s, h = x.shape
- return x.view(b, s, -1, self.patch_size, self.patch_size)
-
- def forward(self, x, t):
- shift, scale = self.adaLN_modulation(t).chunk(2, dim=1)
- x = modulate(self.norm_final(x), shift, scale)
- x = self.linear(x)
- x = self.unpatchify(x)
- return x
-
-
-class DiT(nn.Module):
- """
- Diffusion model with a Transformer backbone.
- """
-
- def __init__(
- self,
- patch_size=2,
- in_channels=3,
- text_embed_dim=512,
- hidden_size=1152,
- depth=28,
- num_heads=16,
- mlp_ratio=4.0,
- max_num_embeddings=256 * 1024,
- text_dropout_prob=0.1,
- learn_sigma=True,
- seq_parallel_group=None,
- seq_parallel_mode="ulysses",
- seq_parallel_overlap=False,
- model_arch="cross-attn",
- ):
- super().__init__()
- assert model_arch in SUPPORTED_MODEL_ARCH
- self.model_arch = model_arch
- self.grad_checkpointing = False
- self.learn_sigma = learn_sigma
- self.in_channels = in_channels
- self.out_channels = in_channels * 2 if learn_sigma else in_channels
- self.patch_size = patch_size
- self.num_heads = num_heads
- self.seq_parallel_group = seq_parallel_group
- self.seq_parallel_size = (
- dist.get_world_size(self.seq_parallel_group)
- if seq_parallel_group is not None
- else 1
- )
- self.seq_parallel_rank = (
- dist.get_rank(self.seq_parallel_group)
- if seq_parallel_group is not None
- else 0
- )
-
- self.video_embedder = PatchEmbedder(
- patch_size, in_channels, hidden_size, bias=True
- )
- self.t_embedder = TimestepEmbedder(hidden_size)
- self.pos_embed = PositionEmbedding(hidden_size, max_num_embeddings)
- self.text_embedder = TextEmbedder(
- text_embed_dim,
- hidden_size,
- bias=True,
- dropout_prob=text_dropout_prob,
- use_proj=model_arch != "cross-attn",
- )
- if model_arch != "cross-attn":
- cross_attn_dim = None
- else:
- cross_attn_dim = text_embed_dim
-
- self.blocks = nn.ModuleList(
- [
- DiTBlock(
- hidden_size,
- num_heads,
- cross_attn_dim,
- mlp_ratio=mlp_ratio,
- seq_parallel_group=seq_parallel_group,
- seq_parallel_mode=seq_parallel_mode,
- seq_parallel_overlap=seq_parallel_overlap,
- )
- for _ in range(depth)
- ]
- )
- self.final_layer = FinalLayer(hidden_size, patch_size, self.out_channels)
- self.initialize_weights()
-
- def initialize_weights(self):
- # Initialize transformer layers:
- def _basic_init(module):
- if isinstance(module, nn.Linear):
- torch.nn.init.xavier_uniform_(module.weight)
- if module.bias is not None:
- nn.init.constant_(module.bias, 0)
-
- self.apply(_basic_init)
-
- # Initialize text embedding layer
- if self.text_embedder.use_proj:
- nn.init.normal_(self.text_embedder.proj.weight, std=0.02)
-
- # Initialize timestep embedding MLP:
- nn.init.normal_(self.t_embedder.mlp[0].weight, std=0.02)
- nn.init.normal_(self.t_embedder.mlp[2].weight, std=0.02)
-
- # Zero-out adaLN modulation layers
- for block in self.blocks:
- nn.init.constant_(block.adaLN_modulation[-1].weight, 0)
- nn.init.constant_(block.adaLN_modulation[-1].bias, 0)
-
- # Zero-out output
- nn.init.constant_(self.final_layer.adaLN_modulation[-1].weight, 0)
- nn.init.constant_(self.final_layer.adaLN_modulation[-1].bias, 0)
- nn.init.constant_(self.final_layer.linear.weight, 0)
- nn.init.constant_(self.final_layer.linear.bias, 0)
-
- def _prepare_mask(self, attention_mask: Optional[torch.Tensor], dtype: torch.dtype):
- if attention_mask is not None:
- assert attention_mask.ndim == 4
- attention_mask = attention_mask.to(dtype)
- inverted_mask = 1.0 - attention_mask
- return inverted_mask.masked_fill(
- inverted_mask.to(torch.bool), torch.finfo(dtype).min
- )
- return attention_mask
-
- def enable_gradient_checkpointing(self):
- self.grad_checkpointing = True
-
- def disable_gradient_checkpointing(self):
- self.grad_checkpointing = False
-
- def forward(
- self,
- video_latent_states,
- t,
- text_latent_states=None,
- attention_mask=None,
- ):
- """
- video_latent_states: [B, S, C, P, P]
- t: [B]
- text_latent_states: [B, H] if model_arch == "adaln" else [B, S, H]
- attention_mask: [B, 1, Sv, St] or [B, 1, Sv, Sv]
- """
- video_latent_states = self.video_embedder(video_latent_states)
- text_len = text_latent_states.shape[1]
- text_latent_states = self.text_embedder(text_latent_states)
- if self.model_arch == "token-concat":
- video_latent_states = torch.cat(
- [text_latent_states, video_latent_states], dim=1
- )
- text_latent_states = None
- pos_embed = self.pos_embed(video_latent_states)
- video_latent_states = video_latent_states + pos_embed
- t = self.t_embedder(t) # (N, D)
- if self.model_arch == "adaln":
- t = t + text_latent_states
- text_latent_states = None
- attention_mask = self._prepare_mask(attention_mask, video_latent_states.dtype)
-
- if self.seq_parallel_group is not None and self.seq_parallel_size > 1:
- assert video_latent_states.shape[1] % self.seq_parallel_size == 0
- video_latent_states = split_seq(
- video_latent_states, self.seq_parallel_size, self.seq_parallel_rank
- )
- if text_latent_states is not None:
- assert text_latent_states.shape[1] % self.seq_parallel_size == 0
- text_latent_states = split_seq(
- text_latent_states, self.seq_parallel_size, self.seq_parallel_rank
- )
-
- for block in self.blocks:
- if self.grad_checkpointing and self.training:
- video_latent_states = torch.utils.checkpoint.checkpoint(
- block,
- video_latent_states,
- attention_mask,
- t,
- text_latent_states,
- )
- else:
- video_latent_states = block(
- video_latent_states, attention_mask, t, text_latent_states
- )
-
- if self.seq_parallel_group is not None and self.seq_parallel_size > 1:
- video_latent_states = gather_seq(
- video_latent_states,
- self.seq_parallel_size,
- self.seq_parallel_rank,
- self.seq_parallel_group,
- )
-
- if self.model_arch == "token-concat":
- video_latent_states = video_latent_states[:, text_len:]
- video_latent_states = self.final_layer(video_latent_states, t)
- return video_latent_states
-
- def forward_with_cfg(
- self, x, t, text_latent_states, cfg_scale, attention_mask=None
- ):
- """
- Forward pass of DiT, but also batches the unconditional forward pass for classifier-free guidance.
- """
- # https://github.com/openai/glide-text2im/blob/main/notebooks/text2im.ipynb
- half = x[: len(x) // 2]
- combined = torch.cat([half, half], dim=0)
- model_out = self.forward(
- combined, t, text_latent_states, attention_mask=attention_mask
- )
- # For exact reproducibility reasons, we apply classifier-free guidance on only
- # three channels by default. The standard approach to cfg applies it to all channels.
- # This can be done by uncommenting the following line and commenting-out the line following that.
- # eps, rest = model_out[:, :self.in_channels], model_out[:, self.in_channels:]
- if self.learn_sigma:
- c = model_out.shape[2]
- assert c == 2 * self.in_channels
- eps, rest = model_out.chunk(2, dim=2)
- else:
- eps = model_out
- cond_eps, uncond_eps = torch.split(eps, len(eps) // 2, dim=0)
- half_eps = uncond_eps + cfg_scale * (cond_eps - uncond_eps)
- eps = torch.cat([half_eps, half_eps], dim=0)
- if self.learn_sigma:
- return torch.cat([eps, rest], dim=2)
- return eps
-
-
-#################################################################################
-# Sine/Cosine Positional Embedding Functions #
-#################################################################################
-# https://github.com/facebookresearch/mae/blob/main/util/pos_embed.py
-
-
-def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False, extra_tokens=0):
- """
- grid_size: int of the grid height and width
- return:
- pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token)
- """
- grid_h = np.arange(grid_size, dtype=np.float32)
- grid_w = np.arange(grid_size, dtype=np.float32)
- grid = np.meshgrid(grid_w, grid_h) # here w goes first
- grid = np.stack(grid, axis=0)
-
- grid = grid.reshape([2, 1, grid_size, grid_size])
- pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)
- if cls_token and extra_tokens > 0:
- pos_embed = np.concatenate(
- [np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0
- )
- return pos_embed
-
-
-def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
- assert embed_dim % 2 == 0
-
- # use half of dimensions to encode grid_h
- emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2)
- emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2)
-
- emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)
- return emb
-
-
-def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
- """
- embed_dim: output dimension for each position
- pos: a list of positions to be encoded: size (M,)
- out: (M, D)
- """
- assert embed_dim % 2 == 0
- omega = np.arange(embed_dim // 2, dtype=np.float64)
- omega /= embed_dim / 2.0
- omega = 1.0 / 10000**omega # (D/2,)
-
- pos = pos.reshape(-1) # (M,)
- out = np.einsum("m,d->md", pos, omega) # (M, D/2), outer product
-
- emb_sin = np.sin(out) # (M, D/2)
- emb_cos = np.cos(out) # (M, D/2)
-
- emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D)
- return emb
-
-
-#################################################################################
-# DiT Configs #
-#################################################################################
-
-
-def DiT_XL_2(**kwargs):
- return DiT(depth=28, hidden_size=1152, patch_size=2, num_heads=16, **kwargs)
-
-
-def DiT_XL_4(**kwargs):
- return DiT(depth=28, hidden_size=1152, patch_size=4, num_heads=16, **kwargs)
-
-
-def DiT_XL_8(**kwargs):
- return DiT(depth=28, hidden_size=1152, patch_size=8, num_heads=16, **kwargs)
-
-
-def DiT_L_2(**kwargs):
- return DiT(depth=24, hidden_size=1024, patch_size=2, num_heads=16, **kwargs)
-
-
-def DiT_L_4(**kwargs):
- return DiT(depth=24, hidden_size=1024, patch_size=4, num_heads=16, **kwargs)
-
-
-def DiT_L_8(**kwargs):
- return DiT(depth=24, hidden_size=1024, patch_size=8, num_heads=16, **kwargs)
-
-
-def DiT_B_2(**kwargs):
- return DiT(depth=12, hidden_size=768, patch_size=2, num_heads=12, **kwargs)
-
-
-def DiT_B_4(**kwargs):
- return DiT(depth=12, hidden_size=768, patch_size=4, num_heads=12, **kwargs)
-
-
-def DiT_B_8(**kwargs):
- return DiT(depth=12, hidden_size=768, patch_size=8, num_heads=12, **kwargs)
-
-
-def DiT_S_2(**kwargs):
- return DiT(depth=12, hidden_size=384, patch_size=2, num_heads=6, **kwargs)
-
-
-def DiT_S_4(**kwargs):
- return DiT(depth=12, hidden_size=384, patch_size=4, num_heads=6, **kwargs)
-
-
-def DiT_S_8(**kwargs):
- return DiT(depth=12, hidden_size=384, patch_size=8, num_heads=6, **kwargs)
-
-
-DiT_models = {
- "DiT-XL/2": DiT_XL_2,
- "DiT-XL/4": DiT_XL_4,
- "DiT-XL/8": DiT_XL_8,
- "DiT-L/2": DiT_L_2,
- "DiT-L/4": DiT_L_4,
- "DiT-L/8": DiT_L_8,
- "DiT-B/2": DiT_B_2,
- "DiT-B/4": DiT_B_4,
- "DiT-B/8": DiT_B_8,
- "DiT-S/2": DiT_S_2,
- "DiT-S/4": DiT_S_4,
- "DiT-S/8": DiT_S_8,
-}
diff --git a/open_sora/modeling/latte/__init__.py b/open_sora/modeling/latte/__init__.py
deleted file mode 100644
index 159db0f..0000000
--- a/open_sora/modeling/latte/__init__.py
+++ /dev/null
@@ -1,3 +0,0 @@
-from .latte_t2v import LatteT2V
-
-__all__ = ["LatteT2V"]
diff --git a/open_sora/modeling/latte/latte_t2v.py b/open_sora/modeling/latte/latte_t2v.py
deleted file mode 100644
index 7c4de50..0000000
--- a/open_sora/modeling/latte/latte_t2v.py
+++ /dev/null
@@ -1,1003 +0,0 @@
-import json
-import os
-from dataclasses import dataclass
-from typing import Any, Dict, Optional, Tuple
-
-import torch
-import torch.nn.functional as F
-from diffusers.configuration_utils import ConfigMixin, register_to_config
-from diffusers.models.activations import GEGLU, GELU, ApproximateGELU
-from diffusers.models.attention import BasicTransformerBlock
-from diffusers.models.attention_processor import Attention
-from diffusers.models.embeddings import (
- CaptionProjection,
- CombinedTimestepSizeEmbeddings,
- ImagePositionalEmbeddings,
- PatchEmbed,
- SinusoidalPositionalEmbedding,
- get_1d_sincos_pos_embed_from_grid,
-)
-from diffusers.models.lora import LoRACompatibleConv, LoRACompatibleLinear
-from diffusers.models.modeling_utils import ModelMixin
-from diffusers.models.normalization import AdaLayerNorm, AdaLayerNormZero
-from diffusers.utils import USE_PEFT_BACKEND, BaseOutput, deprecate
-from diffusers.utils.torch_utils import maybe_allow_in_graph
-from einops import rearrange, repeat
-from torch import nn
-
-
-@maybe_allow_in_graph
-class GatedSelfAttentionDense(nn.Module):
- r"""
- A gated self-attention dense layer that combines visual features and object features.
-
- Parameters:
- query_dim (`int`): The number of channels in the query.
- context_dim (`int`): The number of channels in the context.
- n_heads (`int`): The number of heads to use for attention.
- d_head (`int`): The number of channels in each head.
- """
-
- def __init__(self, query_dim: int, context_dim: int, n_heads: int, d_head: int):
- super().__init__()
-
- # we need a linear projection since we need cat visual feature and obj feature
- self.linear = nn.Linear(context_dim, query_dim)
-
- self.attn = Attention(query_dim=query_dim, heads=n_heads, dim_head=d_head)
- self.ff = FeedForward(query_dim, activation_fn="geglu")
-
- self.norm1 = nn.LayerNorm(query_dim)
- self.norm2 = nn.LayerNorm(query_dim)
-
- self.register_parameter("alpha_attn", nn.Parameter(torch.tensor(0.0)))
- self.register_parameter("alpha_dense", nn.Parameter(torch.tensor(0.0)))
-
- self.enabled = True
-
- def forward(self, x: torch.Tensor, objs: torch.Tensor) -> torch.Tensor:
- if not self.enabled:
- return x
-
- n_visual = x.shape[1]
- objs = self.linear(objs)
-
- x = x + self.alpha_attn.tanh() * self.attn(self.norm1(torch.cat([x, objs], dim=1)))[:, :n_visual, :]
- x = x + self.alpha_dense.tanh() * self.ff(self.norm2(x))
-
- return x
-
-
-class FeedForward(nn.Module):
- r"""
- A feed-forward layer.
-
- Parameters:
- dim (`int`): The number of channels in the input.
- dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.
- mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.
- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
- activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.
- final_dropout (`bool` *optional*, defaults to False): Apply a final dropout.
- """
-
- def __init__(
- self,
- dim: int,
- dim_out: Optional[int] = None,
- mult: int = 4,
- dropout: float = 0.0,
- activation_fn: str = "geglu",
- final_dropout: bool = False,
- ):
- super().__init__()
- inner_dim = int(dim * mult)
- dim_out = dim_out if dim_out is not None else dim
- linear_cls = LoRACompatibleLinear if not USE_PEFT_BACKEND else nn.Linear
-
- if activation_fn == "gelu":
- act_fn = GELU(dim, inner_dim)
- if activation_fn == "gelu-approximate":
- act_fn = GELU(dim, inner_dim, approximate="tanh")
- elif activation_fn == "geglu":
- act_fn = GEGLU(dim, inner_dim)
- elif activation_fn == "geglu-approximate":
- act_fn = ApproximateGELU(dim, inner_dim)
-
- self.net = nn.ModuleList([])
- # project in
- self.net.append(act_fn)
- # project dropout
- self.net.append(nn.Dropout(dropout))
- # project out
- self.net.append(linear_cls(inner_dim, dim_out))
- # FF as used in Vision Transformer, MLP-Mixer, etc. have a final dropout
- if final_dropout:
- self.net.append(nn.Dropout(dropout))
-
- def forward(self, hidden_states: torch.Tensor, scale: float = 1.0) -> torch.Tensor:
- compatible_cls = (GEGLU,) if USE_PEFT_BACKEND else (GEGLU, LoRACompatibleLinear)
- for module in self.net:
- if isinstance(module, compatible_cls):
- hidden_states = module(hidden_states, scale)
- else:
- hidden_states = module(hidden_states)
- return hidden_states
-
-
-@maybe_allow_in_graph
-class BasicTransformerBlock_(nn.Module):
- r"""
- A basic Transformer block.
-
- Parameters:
- dim (`int`): The number of channels in the input and output.
- num_attention_heads (`int`): The number of heads to use for multi-head attention.
- attention_head_dim (`int`): The number of channels in each head.
- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
- cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention.
- activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.
- num_embeds_ada_norm (:
- obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`.
- attention_bias (:
- obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.
- only_cross_attention (`bool`, *optional*):
- Whether to use only cross-attention layers. In this case two cross attention layers are used.
- double_self_attention (`bool`, *optional*):
- Whether to use two self-attention layers. In this case no cross attention layers are used.
- upcast_attention (`bool`, *optional*):
- Whether to upcast the attention computation to float32. This is useful for mixed precision training.
- norm_elementwise_affine (`bool`, *optional*, defaults to `True`):
- Whether to use learnable elementwise affine parameters for normalization.
- norm_type (`str`, *optional*, defaults to `"layer_norm"`):
- The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`.
- final_dropout (`bool` *optional*, defaults to False):
- Whether to apply a final dropout after the last feed-forward layer.
- attention_type (`str`, *optional*, defaults to `"default"`):
- The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`.
- positional_embeddings (`str`, *optional*, defaults to `None`):
- The type of positional embeddings to apply to.
- num_positional_embeddings (`int`, *optional*, defaults to `None`):
- The maximum number of positional embeddings to apply.
- """
-
- def __init__(
- self,
- dim: int,
- num_attention_heads: int,
- attention_head_dim: int,
- dropout=0.0,
- cross_attention_dim: Optional[int] = None,
- activation_fn: str = "geglu",
- num_embeds_ada_norm: Optional[int] = None,
- attention_bias: bool = False,
- only_cross_attention: bool = False,
- double_self_attention: bool = False,
- upcast_attention: bool = False,
- norm_elementwise_affine: bool = True,
- norm_type: str = "layer_norm", # 'layer_norm', 'ada_norm', 'ada_norm_zero', 'ada_norm_single'
- norm_eps: float = 1e-5,
- final_dropout: bool = False,
- attention_type: str = "default",
- positional_embeddings: Optional[str] = None,
- num_positional_embeddings: Optional[int] = None,
- ):
- super().__init__()
- self.only_cross_attention = only_cross_attention
-
- self.use_ada_layer_norm_zero = (num_embeds_ada_norm is not None) and norm_type == "ada_norm_zero"
- self.use_ada_layer_norm = (num_embeds_ada_norm is not None) and norm_type == "ada_norm"
- self.use_ada_layer_norm_single = norm_type == "ada_norm_single"
- self.use_layer_norm = norm_type == "layer_norm"
-
- if norm_type in ("ada_norm", "ada_norm_zero") and num_embeds_ada_norm is None:
- raise ValueError(
- f"`norm_type` is set to {norm_type}, but `num_embeds_ada_norm` is not defined. Please make sure to"
- f" define `num_embeds_ada_norm` if setting `norm_type` to {norm_type}."
- )
-
- if positional_embeddings and (num_positional_embeddings is None):
- raise ValueError(
- "If `positional_embedding` type is defined, `num_positition_embeddings` must also be defined."
- )
-
- if positional_embeddings == "sinusoidal":
- self.pos_embed = SinusoidalPositionalEmbedding(dim, max_seq_length=num_positional_embeddings)
- else:
- self.pos_embed = None
-
- # Define 3 blocks. Each block has its own normalization layer.
- # 1. Self-Attn
- if self.use_ada_layer_norm:
- self.norm1 = AdaLayerNorm(dim, num_embeds_ada_norm)
- elif self.use_ada_layer_norm_zero:
- self.norm1 = AdaLayerNormZero(dim, num_embeds_ada_norm)
- else:
- self.norm1 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)
-
- self.attn1 = Attention(
- query_dim=dim,
- heads=num_attention_heads,
- dim_head=attention_head_dim,
- dropout=dropout,
- bias=attention_bias,
- cross_attention_dim=cross_attention_dim if only_cross_attention else None,
- upcast_attention=upcast_attention,
- )
-
- # # 2. Cross-Attn
- # if cross_attention_dim is not None or double_self_attention:
- # # We currently only use AdaLayerNormZero for self attention where there will only be one attention block.
- # # I.e. the number of returned modulation chunks from AdaLayerZero would not make sense if returned during
- # # the second cross attention block.
- # self.norm2 = (
- # AdaLayerNorm(dim, num_embeds_ada_norm)
- # if self.use_ada_layer_norm
- # else nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)
- # )
- # self.attn2 = Attention(
- # query_dim=dim,
- # cross_attention_dim=cross_attention_dim if not double_self_attention else None,
- # heads=num_attention_heads,
- # dim_head=attention_head_dim,
- # dropout=dropout,
- # bias=attention_bias,
- # upcast_attention=upcast_attention,
- # ) # is self-attn if encoder_hidden_states is none
- # else:
- # self.norm2 = None
- # self.attn2 = None
-
- # 3. Feed-forward
- # if not self.use_ada_layer_norm_single:
- # self.norm3 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)
- self.norm3 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)
-
- self.ff = FeedForward(dim, dropout=dropout, activation_fn=activation_fn, final_dropout=final_dropout)
-
- # 4. Fuser
- if attention_type == "gated" or attention_type == "gated-text-image":
- self.fuser = GatedSelfAttentionDense(dim, cross_attention_dim, num_attention_heads, attention_head_dim)
-
- # 5. Scale-shift for PixArt-Alpha.
- if self.use_ada_layer_norm_single:
- self.scale_shift_table = nn.Parameter(torch.randn(6, dim) / dim**0.5)
-
- # let chunk size default to None
- self._chunk_size = None
- self._chunk_dim = 0
-
- def set_chunk_feed_forward(self, chunk_size: Optional[int], dim: int):
- # Sets chunk feed-forward
- self._chunk_size = chunk_size
- self._chunk_dim = dim
-
- def forward(
- self,
- hidden_states: torch.FloatTensor,
- attention_mask: Optional[torch.FloatTensor] = None,
- encoder_hidden_states: Optional[torch.FloatTensor] = None,
- encoder_attention_mask: Optional[torch.FloatTensor] = None,
- timestep: Optional[torch.LongTensor] = None,
- cross_attention_kwargs: Dict[str, Any] = None,
- class_labels: Optional[torch.LongTensor] = None,
- ) -> torch.FloatTensor:
- # Notice that normalization is always applied before the real computation in the following blocks.
- # 0. Self-Attention
- batch_size = hidden_states.shape[0]
-
- if self.use_ada_layer_norm:
- norm_hidden_states = self.norm1(hidden_states, timestep)
- elif self.use_ada_layer_norm_zero:
- norm_hidden_states, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.norm1(
- hidden_states, timestep, class_labels, hidden_dtype=hidden_states.dtype
- )
- elif self.use_layer_norm:
- norm_hidden_states = self.norm1(hidden_states)
- elif self.use_ada_layer_norm_single:
- shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (
- self.scale_shift_table[None] + timestep.reshape(batch_size, 6, -1)
- ).chunk(6, dim=1)
- norm_hidden_states = self.norm1(hidden_states)
- norm_hidden_states = norm_hidden_states * (1 + scale_msa) + shift_msa
- norm_hidden_states = norm_hidden_states.squeeze(1)
- else:
- raise ValueError("Incorrect norm used")
-
- if self.pos_embed is not None:
- norm_hidden_states = self.pos_embed(norm_hidden_states)
-
- # 1. Retrieve lora scale.
- lora_scale = cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0
-
- # 2. Prepare GLIGEN inputs
- cross_attention_kwargs = cross_attention_kwargs.copy() if cross_attention_kwargs is not None else {}
- gligen_kwargs = cross_attention_kwargs.pop("gligen", None)
-
- attn_output = self.attn1(
- norm_hidden_states,
- encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
- attention_mask=attention_mask,
- **cross_attention_kwargs,
- )
- if self.use_ada_layer_norm_zero:
- attn_output = gate_msa.unsqueeze(1) * attn_output
- elif self.use_ada_layer_norm_single:
- attn_output = gate_msa * attn_output
-
- hidden_states = attn_output + hidden_states
- if hidden_states.ndim == 4:
- hidden_states = hidden_states.squeeze(1)
-
- # 2.5 GLIGEN Control
- if gligen_kwargs is not None:
- hidden_states = self.fuser(hidden_states, gligen_kwargs["objs"])
-
- # # 3. Cross-Attention
- # if self.attn2 is not None:
- # if self.use_ada_layer_norm:
- # norm_hidden_states = self.norm2(hidden_states, timestep)
- # elif self.use_ada_layer_norm_zero or self.use_layer_norm:
- # norm_hidden_states = self.norm2(hidden_states)
- # elif self.use_ada_layer_norm_single:
- # # For PixArt norm2 isn't applied here:
- # # https://github.com/PixArt-alpha/PixArt-alpha/blob/0f55e922376d8b797edd44d25d0e7464b260dcab/diffusion/model/nets/PixArtMS.py#L70C1-L76C103
- # norm_hidden_states = hidden_states
- # else:
- # raise ValueError("Incorrect norm")
-
- # if self.pos_embed is not None and self.use_ada_layer_norm_single is False:
- # norm_hidden_states = self.pos_embed(norm_hidden_states)
-
- # attn_output = self.attn2(
- # norm_hidden_states,
- # encoder_hidden_states=encoder_hidden_states,
- # attention_mask=encoder_attention_mask,
- # **cross_attention_kwargs,
- # )
- # hidden_states = attn_output + hidden_states
-
- # 4. Feed-forward
- # if not self.use_ada_layer_norm_single:
- # norm_hidden_states = self.norm3(hidden_states)
-
- if self.use_ada_layer_norm_zero:
- norm_hidden_states = norm_hidden_states * (1 + scale_mlp[:, None]) + shift_mlp[:, None]
-
- if self.use_ada_layer_norm_single:
- # norm_hidden_states = self.norm2(hidden_states)
- norm_hidden_states = self.norm3(hidden_states)
- norm_hidden_states = norm_hidden_states * (1 + scale_mlp) + shift_mlp
-
- if self._chunk_size is not None:
- # "feed_forward_chunk_size" can be used to save memory
- if norm_hidden_states.shape[self._chunk_dim] % self._chunk_size != 0:
- raise ValueError(
- f"`hidden_states` dimension to be chunked: {norm_hidden_states.shape[self._chunk_dim]} has to be divisible by chunk size: {self._chunk_size}. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`."
- )
-
- num_chunks = norm_hidden_states.shape[self._chunk_dim] // self._chunk_size
- ff_output = torch.cat(
- [
- self.ff(hid_slice, scale=lora_scale)
- for hid_slice in norm_hidden_states.chunk(num_chunks, dim=self._chunk_dim)
- ],
- dim=self._chunk_dim,
- )
- else:
- ff_output = self.ff(norm_hidden_states, scale=lora_scale)
-
- if self.use_ada_layer_norm_zero:
- ff_output = gate_mlp.unsqueeze(1) * ff_output
- elif self.use_ada_layer_norm_single:
- ff_output = gate_mlp * ff_output
-
- hidden_states = ff_output + hidden_states
- if hidden_states.ndim == 4:
- hidden_states = hidden_states.squeeze(1)
-
- return hidden_states
-
-
-class AdaLayerNormSingle(nn.Module):
- r"""
- Norm layer adaptive layer norm single (adaLN-single).
-
- As proposed in PixArt-Alpha (see: https://arxiv.org/abs/2310.00426; Section 2.3).
-
- Parameters:
- embedding_dim (`int`): The size of each embedding vector.
- use_additional_conditions (`bool`): To use additional conditions for normalization or not.
- """
-
- def __init__(self, embedding_dim: int, use_additional_conditions: bool = False):
- super().__init__()
-
- self.emb = CombinedTimestepSizeEmbeddings(
- embedding_dim, size_emb_dim=embedding_dim // 3, use_additional_conditions=use_additional_conditions
- )
-
- self.silu = nn.SiLU()
- self.linear = nn.Linear(embedding_dim, 6 * embedding_dim, bias=True)
-
- def forward(
- self,
- timestep: torch.Tensor,
- added_cond_kwargs: Dict[str, torch.Tensor] = None,
- batch_size: int = None,
- hidden_dtype: Optional[torch.dtype] = None,
- ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
- # No modulation happening here.
- embedded_timestep = self.emb(
- timestep, batch_size=batch_size, hidden_dtype=hidden_dtype, resolution=None, aspect_ratio=None
- )
- return self.linear(self.silu(embedded_timestep)), embedded_timestep
-
-
-@dataclass
-class Transformer3DModelOutput(BaseOutput):
- """
- The output of [`Transformer2DModel`].
-
- Args:
- sample (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `(batch size, num_vector_embeds - 1, num_latent_pixels)` if [`Transformer2DModel`] is discrete):
- The hidden states output conditioned on the `encoder_hidden_states` input. If discrete, returns probability
- distributions for the unnoised latent pixels.
- """
-
- sample: torch.FloatTensor
-
-
-class LatteT2V(ModelMixin, ConfigMixin):
- _supports_gradient_checkpointing = True
-
- """
- A 2D Transformer model for image-like data.
-
- Parameters:
- num_attention_heads (`int`, *optional*, defaults to 16): The number of heads to use for multi-head attention.
- attention_head_dim (`int`, *optional*, defaults to 88): The number of channels in each head.
- in_channels (`int`, *optional*):
- The number of channels in the input and output (specify if the input is **continuous**).
- num_layers (`int`, *optional*, defaults to 1): The number of layers of Transformer blocks to use.
- dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
- cross_attention_dim (`int`, *optional*): The number of `encoder_hidden_states` dimensions to use.
- sample_size (`int`, *optional*): The width of the latent images (specify if the input is **discrete**).
- This is fixed during training since it is used to learn a number of position embeddings.
- num_vector_embeds (`int`, *optional*):
- The number of classes of the vector embeddings of the latent pixels (specify if the input is **discrete**).
- Includes the class for the masked latent pixel.
- activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to use in feed-forward.
- num_embeds_ada_norm ( `int`, *optional*):
- The number of diffusion steps used during training. Pass if at least one of the norm_layers is
- `AdaLayerNorm`. This is fixed during training since it is used to learn a number of embeddings that are
- added to the hidden states.
-
- During inference, you can denoise for up to but not more steps than `num_embeds_ada_norm`.
- attention_bias (`bool`, *optional*):
- Configure if the `TransformerBlocks` attention should contain a bias parameter.
- """
-
- @register_to_config
- def __init__(
- self,
- num_attention_heads: int = 16,
- attention_head_dim: int = 88,
- in_channels: Optional[int] = None,
- out_channels: Optional[int] = None,
- num_layers: int = 1,
- dropout: float = 0.0,
- norm_num_groups: int = 32,
- cross_attention_dim: Optional[int] = None,
- attention_bias: bool = False,
- sample_size: Optional[int] = None,
- num_vector_embeds: Optional[int] = None,
- patch_size: Optional[int] = None,
- activation_fn: str = "geglu",
- num_embeds_ada_norm: Optional[int] = None,
- use_linear_projection: bool = False,
- only_cross_attention: bool = False,
- double_self_attention: bool = False,
- upcast_attention: bool = False,
- norm_type: str = "layer_norm",
- norm_elementwise_affine: bool = True,
- norm_eps: float = 1e-5,
- attention_type: str = "default",
- caption_channels: int = None,
- video_length: int = 16,
- ):
- super().__init__()
- self.use_linear_projection = use_linear_projection
- self.num_attention_heads = num_attention_heads
- self.attention_head_dim = attention_head_dim
- inner_dim = num_attention_heads * attention_head_dim
- self.video_length = video_length
-
- conv_cls = nn.Conv2d if USE_PEFT_BACKEND else LoRACompatibleConv
- linear_cls = nn.Linear if USE_PEFT_BACKEND else LoRACompatibleLinear
-
- # 1. Transformer2DModel can process both standard continuous images of shape `(batch_size, num_channels, width, height)` as well as quantized image embeddings of shape `(batch_size, num_image_vectors)`
- # Define whether input is continuous or discrete depending on configuration
- self.is_input_continuous = (in_channels is not None) and (patch_size is None)
- self.is_input_vectorized = num_vector_embeds is not None
- self.is_input_patches = in_channels is not None and patch_size is not None
-
- if norm_type == "layer_norm" and num_embeds_ada_norm is not None:
- deprecation_message = (
- f"The configuration file of this model: {self.__class__} is outdated. `norm_type` is either not set or"
- " incorrectly set to `'layer_norm'`.Make sure to set `norm_type` to `'ada_norm'` in the config."
- " Please make sure to update the config accordingly as leaving `norm_type` might led to incorrect"
- " results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it"
- " would be very nice if you could open a Pull request for the `transformer/config.json` file"
- )
- deprecate("norm_type!=num_embeds_ada_norm", "1.0.0", deprecation_message, standard_warn=False)
- norm_type = "ada_norm"
-
- if self.is_input_continuous and self.is_input_vectorized:
- raise ValueError(
- f"Cannot define both `in_channels`: {in_channels} and `num_vector_embeds`: {num_vector_embeds}. Make"
- " sure that either `in_channels` or `num_vector_embeds` is None."
- )
- elif self.is_input_vectorized and self.is_input_patches:
- raise ValueError(
- f"Cannot define both `num_vector_embeds`: {num_vector_embeds} and `patch_size`: {patch_size}. Make"
- " sure that either `num_vector_embeds` or `num_patches` is None."
- )
- elif not self.is_input_continuous and not self.is_input_vectorized and not self.is_input_patches:
- raise ValueError(
- f"Has to define `in_channels`: {in_channels}, `num_vector_embeds`: {num_vector_embeds}, or patch_size:"
- f" {patch_size}. Make sure that `in_channels`, `num_vector_embeds` or `num_patches` is not None."
- )
-
- # 2. Define input layers
- if self.is_input_continuous:
- self.in_channels = in_channels
-
- self.norm = torch.nn.GroupNorm(num_groups=norm_num_groups, num_channels=in_channels, eps=1e-6, affine=True)
- if use_linear_projection:
- self.proj_in = linear_cls(in_channels, inner_dim)
- else:
- self.proj_in = conv_cls(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)
- elif self.is_input_vectorized:
- assert sample_size is not None, "Transformer2DModel over discrete input must provide sample_size"
- assert num_vector_embeds is not None, "Transformer2DModel over discrete input must provide num_embed"
-
- self.height = sample_size
- self.width = sample_size
- self.num_vector_embeds = num_vector_embeds
- self.num_latent_pixels = self.height * self.width
-
- self.latent_image_embedding = ImagePositionalEmbeddings(
- num_embed=num_vector_embeds, embed_dim=inner_dim, height=self.height, width=self.width
- )
- elif self.is_input_patches:
- assert sample_size is not None, "Transformer2DModel over patched input must provide sample_size"
-
- self.height = sample_size
- self.width = sample_size
-
- self.patch_size = patch_size
- interpolation_scale = self.config.sample_size // 64 # => 64 (= 512 pixart) has interpolation scale 1
- interpolation_scale = max(interpolation_scale, 1)
- self.pos_embed = PatchEmbed(
- height=sample_size,
- width=sample_size,
- patch_size=patch_size,
- in_channels=in_channels,
- embed_dim=inner_dim,
- interpolation_scale=interpolation_scale,
- )
-
- # 3. Define transformers blocks
- self.transformer_blocks = nn.ModuleList(
- [
- BasicTransformerBlock(
- inner_dim,
- num_attention_heads,
- attention_head_dim,
- dropout=dropout,
- cross_attention_dim=cross_attention_dim,
- activation_fn=activation_fn,
- num_embeds_ada_norm=num_embeds_ada_norm,
- attention_bias=attention_bias,
- only_cross_attention=only_cross_attention,
- double_self_attention=double_self_attention,
- upcast_attention=upcast_attention,
- norm_type=norm_type,
- norm_elementwise_affine=norm_elementwise_affine,
- norm_eps=norm_eps,
- attention_type=attention_type,
- )
- for d in range(num_layers)
- ]
- )
-
- # Define temporal transformers blocks
- self.temporal_transformer_blocks = nn.ModuleList(
- [
- BasicTransformerBlock_( # one attention
- inner_dim,
- num_attention_heads, # num_attention_heads
- attention_head_dim, # attention_head_dim 72
- dropout=dropout,
- cross_attention_dim=None,
- activation_fn=activation_fn,
- num_embeds_ada_norm=num_embeds_ada_norm,
- attention_bias=attention_bias,
- only_cross_attention=only_cross_attention,
- double_self_attention=False,
- upcast_attention=upcast_attention,
- norm_type=norm_type,
- norm_elementwise_affine=norm_elementwise_affine,
- norm_eps=norm_eps,
- attention_type=attention_type,
- )
- for d in range(num_layers)
- ]
- )
-
- # 4. Define output layers
- self.out_channels = in_channels if out_channels is None else out_channels
- if self.is_input_continuous:
- # TODO: should use out_channels for continuous projections
- if use_linear_projection:
- self.proj_out = linear_cls(inner_dim, in_channels)
- else:
- self.proj_out = conv_cls(inner_dim, in_channels, kernel_size=1, stride=1, padding=0)
- elif self.is_input_vectorized:
- self.norm_out = nn.LayerNorm(inner_dim)
- self.out = nn.Linear(inner_dim, self.num_vector_embeds - 1)
- elif self.is_input_patches and norm_type != "ada_norm_single":
- self.norm_out = nn.LayerNorm(inner_dim, elementwise_affine=False, eps=1e-6)
- self.proj_out_1 = nn.Linear(inner_dim, 2 * inner_dim)
- self.proj_out_2 = nn.Linear(inner_dim, patch_size * patch_size * self.out_channels)
- elif self.is_input_patches and norm_type == "ada_norm_single":
- self.norm_out = nn.LayerNorm(inner_dim, elementwise_affine=False, eps=1e-6)
- self.scale_shift_table = nn.Parameter(torch.randn(2, inner_dim) / inner_dim**0.5)
- self.proj_out = nn.Linear(inner_dim, patch_size * patch_size * self.out_channels)
-
- # 5. PixArt-Alpha blocks.
- self.adaln_single = None
- self.use_additional_conditions = False
- if norm_type == "ada_norm_single":
- self.use_additional_conditions = self.config.sample_size == 128 # False, 128 -> 1024
- # TODO(Sayak, PVP) clean this, for now we use sample size to determine whether to use
- # additional conditions until we find better name
- self.adaln_single = AdaLayerNormSingle(inner_dim, use_additional_conditions=self.use_additional_conditions)
-
- self.caption_projection = None
- if caption_channels is not None:
- self.caption_projection = CaptionProjection(in_features=caption_channels, hidden_size=inner_dim)
-
- self.gradient_checkpointing = False
-
- # define temporal positional embedding
- temp_pos_embed = self.get_1d_sincos_temp_embed(inner_dim, video_length) # 1152 hidden size
- self.register_buffer("temp_pos_embed", torch.from_numpy(temp_pos_embed).float().unsqueeze(0), persistent=False)
-
- def _set_gradient_checkpointing(self, module, value=False):
- self.gradient_checkpointing = value
-
- def forward(
- self,
- hidden_states: torch.Tensor,
- timestep: Optional[torch.LongTensor] = None,
- encoder_hidden_states: Optional[torch.Tensor] = None,
- added_cond_kwargs: Dict[str, torch.Tensor] = None,
- class_labels: Optional[torch.LongTensor] = None,
- cross_attention_kwargs: Dict[str, Any] = None,
- attention_mask: Optional[torch.Tensor] = None,
- encoder_attention_mask: Optional[torch.Tensor] = None,
- use_image_num: int = 0,
- enable_temporal_attentions: bool = True,
- return_dict: bool = True,
- ):
- """
- The [`Transformer2DModel`] forward method.
-
- Args:
- hidden_states (`torch.LongTensor` of shape `(batch size, num latent pixels)` if discrete, `torch.FloatTensor` of shape `(batch size, frame, channel, height, width)` if continuous):
- Input `hidden_states`.
- encoder_hidden_states ( `torch.FloatTensor` of shape `(batch size, sequence len, embed dims)`, *optional*):
- Conditional embeddings for cross attention layer. If not given, cross-attention defaults to
- self-attention.
- timestep ( `torch.LongTensor`, *optional*):
- Used to indicate denoising step. Optional timestep to be applied as an embedding in `AdaLayerNorm`.
- class_labels ( `torch.LongTensor` of shape `(batch size, num classes)`, *optional*):
- Used to indicate class labels conditioning. Optional class labels to be applied as an embedding in
- `AdaLayerZeroNorm`.
- cross_attention_kwargs ( `Dict[str, Any]`, *optional*):
- A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under
- `self.processor` in
- [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
- attention_mask ( `torch.Tensor`, *optional*):
- An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask
- is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large
- negative values to the attention scores corresponding to "discard" tokens.
- encoder_attention_mask ( `torch.Tensor`, *optional*):
- Cross-attention mask applied to `encoder_hidden_states`. Two formats supported:
-
- * Mask `(batch, sequence_length)` True = keep, False = discard.
- * Bias `(batch, 1, sequence_length)` 0 = keep, -10000 = discard.
-
- If `ndim == 2`: will be interpreted as a mask, then converted into a bias consistent with the format
- above. This bias will be added to the cross-attention scores.
- return_dict (`bool`, *optional*, defaults to `True`):
- Whether or not to return a [`~models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain
- tuple.
-
- Returns:
- If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a
- `tuple` where the first element is the sample tensor.
- """
- input_batch_size, c, frame, h, w = hidden_states.shape
- frame = frame - use_image_num
- hidden_states = rearrange(hidden_states, "b c f h w -> (b f) c h w").contiguous()
-
- # ensure attention_mask is a bias, and give it a singleton query_tokens dimension.
- # we may have done this conversion already, e.g. if we came here via UNet2DConditionModel#forward.
- # we can tell by counting dims; if ndim == 2: it's a mask rather than a bias.
- # expects mask of shape:
- # [batch, key_tokens]
- # adds singleton query_tokens dimension:
- # [batch, 1, key_tokens]
- # this helps to broadcast it as a bias over attention scores, which will be in one of the following shapes:
- # [batch, heads, query_tokens, key_tokens] (e.g. torch sdp attn)
- # [batch * heads, query_tokens, key_tokens] (e.g. xformers or classic attn)
- if attention_mask is not None and attention_mask.ndim == 2:
- # assume that mask is expressed as:
- # (1 = keep, 0 = discard)
- # convert mask into a bias that can be added to attention scores:
- # (keep = +0, discard = -10000.0)
- attention_mask = (1 - attention_mask.to(hidden_states.dtype)) * -10000.0
- attention_mask = attention_mask.unsqueeze(1)
-
- # convert encoder_attention_mask to a bias the same way we do for attention_mask
- if encoder_attention_mask is not None and encoder_attention_mask.ndim == 2: # ndim == 2 means no image joint
- encoder_attention_mask = (1 - encoder_attention_mask.to(hidden_states.dtype)) * -10000.0
- encoder_attention_mask = encoder_attention_mask.unsqueeze(1)
- encoder_attention_mask = repeat(encoder_attention_mask, "b 1 l -> (b f) 1 l", f=frame).contiguous()
- elif encoder_attention_mask is not None and encoder_attention_mask.ndim == 3: # ndim == 3 means image joint
- encoder_attention_mask = (1 - encoder_attention_mask.to(hidden_states.dtype)) * -10000.0
- encoder_attention_mask_video = encoder_attention_mask[:, :1, ...]
- encoder_attention_mask_video = repeat(
- encoder_attention_mask_video, "b 1 l -> b (1 f) l", f=frame
- ).contiguous()
- encoder_attention_mask_image = encoder_attention_mask[:, 1:, ...]
- encoder_attention_mask = torch.cat([encoder_attention_mask_video, encoder_attention_mask_image], dim=1)
- encoder_attention_mask = rearrange(encoder_attention_mask, "b n l -> (b n) l").contiguous().unsqueeze(1)
-
- # Retrieve lora scale.
- cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0
-
- # 1. Input
- if self.is_input_patches: # here
- height, width = hidden_states.shape[-2] // self.patch_size, hidden_states.shape[-1] // self.patch_size
- num_patches = height * width
-
- hidden_states = self.pos_embed(hidden_states) # already add positional embeddings
-
- if self.adaln_single is not None:
- if self.use_additional_conditions and added_cond_kwargs is None:
- raise ValueError(
- "`added_cond_kwargs` cannot be None when using additional conditions for `adaln_single`."
- )
- # batch_size = hidden_states.shape[0]
- batch_size = input_batch_size
- timestep, embedded_timestep = self.adaln_single(
- timestep, added_cond_kwargs, batch_size=batch_size, hidden_dtype=hidden_states.dtype
- )
-
- # 2. Blocks
- if self.caption_projection is not None:
- batch_size = hidden_states.shape[0]
- encoder_hidden_states = self.caption_projection(encoder_hidden_states) # 3 120 1152
-
- if use_image_num != 0 and self.training:
- encoder_hidden_states_video = encoder_hidden_states[:, :1, ...]
- encoder_hidden_states_video = repeat(
- encoder_hidden_states_video, "b 1 t d -> b (1 f) t d", f=frame
- ).contiguous()
- encoder_hidden_states_image = encoder_hidden_states[:, 1:, ...]
- encoder_hidden_states = torch.cat([encoder_hidden_states_video, encoder_hidden_states_image], dim=1)
- encoder_hidden_states_spatial = rearrange(encoder_hidden_states, "b f t d -> (b f) t d").contiguous()
- else:
- encoder_hidden_states_spatial = repeat(
- encoder_hidden_states, "b t d -> (b f) t d", f=frame
- ).contiguous()
-
- # prepare timesteps for spatial and temporal block
- timestep_spatial = repeat(timestep, "b d -> (b f) d", f=frame + use_image_num).contiguous()
- timestep_temp = repeat(timestep, "b d -> (b p) d", p=num_patches).contiguous()
-
- for i, (spatial_block, temp_block) in enumerate(zip(self.transformer_blocks, self.temporal_transformer_blocks)):
- if self.training and self.gradient_checkpointing:
- hidden_states = torch.utils.checkpoint.checkpoint(
- spatial_block,
- hidden_states,
- attention_mask,
- encoder_hidden_states_spatial,
- encoder_attention_mask,
- timestep_spatial,
- cross_attention_kwargs,
- class_labels,
- use_reentrant=False,
- )
-
- if enable_temporal_attentions:
- hidden_states = rearrange(hidden_states, "(b f) t d -> (b t) f d", b=input_batch_size).contiguous()
-
- if use_image_num != 0: # image-video joitn training
- hidden_states_video = hidden_states[:, :frame, ...]
- hidden_states_image = hidden_states[:, frame:, ...]
-
- if i == 0:
- hidden_states_video = hidden_states_video + self.temp_pos_embed
-
- hidden_states_video = torch.utils.checkpoint.checkpoint(
- temp_block,
- hidden_states_video,
- None, # attention_mask
- None, # encoder_hidden_states
- None, # encoder_attention_mask
- timestep_temp,
- cross_attention_kwargs,
- class_labels,
- use_reentrant=False,
- )
-
- hidden_states = torch.cat([hidden_states_video, hidden_states_image], dim=1)
- hidden_states = rearrange(
- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size
- ).contiguous()
-
- else:
- if i == 0:
- hidden_states = hidden_states + self.temp_pos_embed
-
- hidden_states = torch.utils.checkpoint.checkpoint(
- temp_block,
- hidden_states,
- None, # attention_mask
- None, # encoder_hidden_states
- None, # encoder_attention_mask
- timestep_temp,
- cross_attention_kwargs,
- class_labels,
- use_reentrant=False,
- )
-
- hidden_states = rearrange(
- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size
- ).contiguous()
- else:
- hidden_states = spatial_block(
- hidden_states,
- attention_mask,
- encoder_hidden_states_spatial,
- encoder_attention_mask,
- timestep_spatial,
- cross_attention_kwargs,
- class_labels,
- )
-
- if enable_temporal_attentions:
- hidden_states = rearrange(hidden_states, "(b f) t d -> (b t) f d", b=input_batch_size).contiguous()
-
- if use_image_num != 0 and self.training:
- hidden_states_video = hidden_states[:, :frame, ...]
- hidden_states_image = hidden_states[:, frame:, ...]
-
- hidden_states_video = temp_block(
- hidden_states_video,
- None, # attention_mask
- None, # encoder_hidden_states
- None, # encoder_attention_mask
- timestep_temp,
- cross_attention_kwargs,
- class_labels,
- )
-
- hidden_states = torch.cat([hidden_states_video, hidden_states_image], dim=1)
- hidden_states = rearrange(
- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size
- ).contiguous()
-
- else:
- if i == 0:
- hidden_states = hidden_states + self.temp_pos_embed
-
- hidden_states = temp_block(
- hidden_states,
- None, # attention_mask
- None, # encoder_hidden_states
- None, # encoder_attention_mask
- timestep_temp,
- cross_attention_kwargs,
- class_labels,
- )
-
- hidden_states = rearrange(
- hidden_states, "(b t) f d -> (b f) t d", b=input_batch_size
- ).contiguous()
-
- if self.is_input_patches:
- if self.config.norm_type != "ada_norm_single":
- conditioning = self.transformer_blocks[0].norm1.emb(
- timestep, class_labels, hidden_dtype=hidden_states.dtype
- )
- shift, scale = self.proj_out_1(F.silu(conditioning)).chunk(2, dim=1)
- hidden_states = self.norm_out(hidden_states) * (1 + scale[:, None]) + shift[:, None]
- hidden_states = self.proj_out_2(hidden_states)
- elif self.config.norm_type == "ada_norm_single":
- embedded_timestep = repeat(embedded_timestep, "b d -> (b f) d", f=frame + use_image_num).contiguous()
- shift, scale = (self.scale_shift_table[None] + embedded_timestep[:, None]).chunk(2, dim=1)
- hidden_states = self.norm_out(hidden_states)
- # Modulation
- hidden_states = hidden_states * (1 + scale) + shift
- hidden_states = self.proj_out(hidden_states)
-
- # unpatchify
- if self.adaln_single is None:
- height = width = int(hidden_states.shape[1] ** 0.5)
- hidden_states = hidden_states.reshape(
- shape=(-1, height, width, self.patch_size, self.patch_size, self.out_channels)
- )
- hidden_states = torch.einsum("nhwpqc->nchpwq", hidden_states)
- output = hidden_states.reshape(
- shape=(-1, self.out_channels, height * self.patch_size, width * self.patch_size)
- )
- output = rearrange(output, "(b f) c h w -> b c f h w", b=input_batch_size).contiguous()
-
- if not return_dict:
- return (output,)
-
- return Transformer3DModelOutput(sample=output)
-
- def get_1d_sincos_temp_embed(self, embed_dim, length):
- pos = torch.arange(0, length).unsqueeze(1)
- return get_1d_sincos_pos_embed_from_grid(embed_dim, pos)
-
- @classmethod
- def from_pretrained_2d(cls, pretrained_model_path, subfolder=None, **kwargs):
- if subfolder is not None:
- pretrained_model_path = os.path.join(pretrained_model_path, subfolder)
-
- config_file = os.path.join(pretrained_model_path, "config.json")
- if not os.path.isfile(config_file):
- raise RuntimeError(f"{config_file} does not exist")
- with open(config_file, "r") as f:
- config = json.load(f)
-
- model = cls.from_config(config, **kwargs)
-
- # model_files = [
- # os.path.join(pretrained_model_path, 'diffusion_pytorch_model.bin'),
- # os.path.join(pretrained_model_path, 'diffusion_pytorch_model.safetensors')
- # ]
-
- # model_file = None
-
- # for fp in model_files:
- # if os.path.exists(fp):
- # model_file = fp
-
- # if not model_file:
- # raise RuntimeError(f"{model_file} does not exist")
-
- # if model_file.split(".")[-1] == "safetensors":
- # from safetensors import safe_open
- # state_dict = {}
- # with safe_open(model_file, framework="pt", device="cpu") as f:
- # for key in f.keys():
- # state_dict[key] = f.get_tensor(key)
- # else:
- # state_dict = torch.load(model_file, map_location="cpu")
-
- # for k, v in model.state_dict().items():
- # if 'temporal_transformer_blocks' in k:
- # state_dict.update({k: v})
-
- # model.load_state_dict(state_dict)
-
- return model
diff --git a/open_sora/utils/__init__.py b/open_sora/utils/__init__.py
deleted file mode 100644
index e69de29..0000000
diff --git a/open_sora/utils/comm.py b/open_sora/utils/comm.py
deleted file mode 100644
index 9b947ff..0000000
--- a/open_sora/utils/comm.py
+++ /dev/null
@@ -1,292 +0,0 @@
-from typing import Optional, Tuple
-
-import torch
-import torch.distributed as dist
-import torch.nn.functional as F
-from colossalai.moe._operation import MoeInGradScaler, MoeOutGradScaler
-from colossalai.shardformer.layer._operation import gather_forward_split_backward
-from torch.distributed.distributed_c10d import get_global_rank
-
-
-def _all_to_all(
- input_: torch.Tensor,
- world_size: int,
- group: dist.ProcessGroup,
- scatter_dim: int,
- gather_dim: int,
-):
- input_list = [
- t.contiguous() for t in torch.tensor_split(input_, world_size, scatter_dim)
- ]
- output_list = [torch.empty_like(input_list[0]) for _ in range(world_size)]
- dist.all_to_all(output_list, input_list, group=group)
- return torch.cat(output_list, dim=gather_dim).contiguous()
-
-
-class _AllToAll(torch.autograd.Function):
- """All-to-all communication.
-
- Args:
- input_: input matrix
- process_group: communication group
- scatter_dim: scatter dimension
- gather_dim: gather dimension
- """
-
- @staticmethod
- def forward(ctx, input_, process_group, scatter_dim, gather_dim):
- ctx.process_group = process_group
- ctx.scatter_dim = scatter_dim
- ctx.gather_dim = gather_dim
- ctx.world_size = dist.get_world_size(process_group)
- return _all_to_all(
- input_, ctx.world_size, process_group, scatter_dim, gather_dim
- )
-
- @staticmethod
- def backward(ctx, grad_output):
- return (
- _all_to_all(
- grad_output,
- ctx.world_size,
- ctx.process_group,
- ctx.gather_dim,
- ctx.scatter_dim,
- ),
- None,
- None,
- None,
- )
-
-
-def all_to_all(
- input_: torch.Tensor,
- process_group: dist.ProcessGroup,
- scatter_dim: int = 2,
- gather_dim: int = 1,
-):
- return _AllToAll.apply(input_, process_group, scatter_dim, gather_dim)
-
-
-def split_seq(input_: torch.Tensor, sp_size: int, sp_rank: int, dim: int = 1):
- """Split a tensor along sequence dimension. It will split input and divide grad by sp_size.
-
- Args:
- input_ (torch.Tensor): The common shape is (bs, seq, *).
- sp_size (int): Sequence parallel size.
- sp_rank (int): Sequence parallel rank.
- dim (int, optional): Sequence dimension. Defaults to 1.
- """
- input_ = input_.chunk(sp_size, dim=dim)[sp_rank].clone()
- return MoeOutGradScaler.apply(input_, sp_size)
-
-
-def gather_seq(
- input_: torch.Tensor,
- sp_size: int,
- sp_rank: int,
- sp_group: dist.ProcessGroup,
- dim: int = 1,
-):
- """Gather a tensor along sequence dimension. It will gather input and multiply grad by sp_size.
-
- Args:
- input_ (torch.Tensor): The common shape is (bs, seq, *).
- sp_size (int): Sequence parallel size.
- sp_rank (int): Sequence parallel rank.
- dim (int, optional): Sequence dimension. Defaults to 1.
- """
- input_ = gather_forward_split_backward(input_, dim, sp_group)
- return MoeInGradScaler.apply(input_, sp_size)
-
-
-class AsyncAllGatherProjForTwo(torch.autograd.Function):
- @staticmethod
- def forward(
- ctx,
- hidden_states: torch.Tensor,
- context: torch.Tensor,
- q_proj_weight: torch.Tensor,
- q_proj_bias: Optional[torch.Tensor],
- k_proj_weight: torch.Tensor,
- k_proj_bias: Optional[torch.Tensor],
- v_proj_weight: torch.Tensor,
- v_proj_bias: Optional[torch.Tensor],
- dim: int,
- process_group: dist.ProcessGroup,
- sp_size: int,
- sp_rank: int,
- ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
- assert sp_size == 2
- ctx.process_group = process_group
- ctx.sp_size = sp_size
- ctx.sp_rank = sp_rank
- ctx.dim = dim
-
- is_cross_attn = not (hidden_states is context)
- ctx.is_cross_attn = is_cross_attn
-
- recv_hidden_states = torch.empty_like(hidden_states)
- if is_cross_attn:
- recv_context = torch.empty_like(context)
- else:
- recv_context = recv_hidden_states
-
- peer_global_rank = get_global_rank(process_group, 1 - sp_rank)
- ops = [
- dist.P2POp(dist.isend, hidden_states, peer_global_rank, process_group),
- dist.P2POp(dist.irecv, recv_hidden_states, peer_global_rank, process_group),
- ]
- if is_cross_attn:
- ops.extend(
- [
- dist.P2POp(dist.isend, context, peer_global_rank, process_group),
- dist.P2POp(
- dist.irecv, recv_context, peer_global_rank, process_group
- ),
- ]
- )
-
- reqs = dist.batch_isend_irecv(ops)
-
- # [B, S/P, H]
- q = F.linear(hidden_states, q_proj_weight, q_proj_bias)
- k = F.linear(context, k_proj_weight, k_proj_bias)
- v = F.linear(context, v_proj_weight, v_proj_bias)
-
- for req in reqs:
- req.wait()
-
- q_other = F.linear(recv_hidden_states, q_proj_weight, q_proj_bias)
- k_other = F.linear(recv_context, k_proj_weight, k_proj_bias)
- v_other = F.linear(recv_context, v_proj_weight, v_proj_bias)
-
- if sp_rank == 0:
- q = torch.cat([q, q_other], dim=dim)
- k = torch.cat([k, k_other], dim=dim)
- v = torch.cat([v, v_other], dim=dim)
- else:
- q = torch.cat([q_other, q], dim=dim)
- k = torch.cat([k_other, k], dim=dim)
- v = torch.cat([v_other, v], dim=dim)
-
- ctx.save_for_backward(
- hidden_states,
- context,
- q_proj_weight,
- q_proj_bias,
- k_proj_weight,
- k_proj_bias,
- v_proj_weight,
- v_proj_bias,
- recv_hidden_states,
- recv_context,
- )
-
- return q, k, v
-
- @staticmethod
- def backward(
- ctx: torch.Any, q_grad: torch.Tensor, k_grad: torch.Tensor, v_grad: torch.Tensor
- ) -> torch.Any:
- (
- hidden_states,
- context,
- q_proj_weight,
- q_proj_bias,
- k_proj_weight,
- k_proj_bias,
- v_proj_weight,
- v_proj_bias,
- recv_hidden_states,
- recv_context,
- ) = ctx.saved_tensors
-
- # compute param grads
- if ctx.sp_rank == 0:
- hidden_states = torch.cat([hidden_states, recv_hidden_states], dim=ctx.dim)
- if ctx.is_cross_attn:
- context = torch.cat([context, recv_context], dim=ctx.dim)
- else:
- context = hidden_states
- else:
- hidden_states = torch.cat([recv_hidden_states, hidden_states], dim=ctx.dim)
- if ctx.is_cross_attn:
- context = torch.cat([recv_context, context], dim=ctx.dim)
- else:
- context = hidden_states
- q_proj_weight_grad = q_grad.transpose(-1, -2).matmul(hidden_states).sum(dim=0)
- q_proj_bias_grad = (
- q_grad.sum(dim=0).sum(dim=0) if q_proj_bias is not None else None
- )
- k_proj_weight_grad = k_grad.transpose(-1, -2).matmul(context).sum(dim=0)
- k_proj_bias_grad = (
- k_grad.sum(dim=0).sum(dim=0) if k_proj_bias is not None else None
- )
- v_proj_weight_grad = v_grad.transpose(-1, -2).matmul(context).sum(dim=0)
- v_proj_bias_grad = (
- v_grad.sum(dim=0).sum(dim=0) if v_proj_bias is not None else None
- )
-
- # split grads
- q_grad = q_grad.chunk(ctx.sp_size, dim=ctx.dim)[ctx.sp_rank].clone()
- k_grad = k_grad.chunk(ctx.sp_size, dim=ctx.dim)[ctx.sp_rank].clone()
- v_grad = v_grad.chunk(ctx.sp_size, dim=ctx.dim)[ctx.sp_rank].clone()
-
- if ctx.is_cross_attn:
- hidden_states_grad = torch.matmul(q_grad, q_proj_weight)
- context_grad = torch.matmul(k_grad, k_proj_weight) + torch.matmul(
- v_grad, v_proj_weight
- )
- else:
- hidden_states_grad = (
- torch.matmul(q_grad, q_proj_weight)
- + torch.matmul(k_grad, k_proj_weight)
- + torch.matmul(v_grad, v_proj_weight)
- )
- context_grad = hidden_states_grad
-
- return (
- hidden_states_grad,
- context_grad,
- q_proj_weight_grad,
- q_proj_bias_grad,
- k_proj_weight_grad,
- k_proj_bias_grad,
- v_proj_weight_grad,
- v_proj_bias_grad,
- None,
- None,
- None,
- None,
- )
-
-
-def async_all_gather_proj_for_two(
- hidden_states: torch.Tensor,
- context: torch.Tensor,
- q_proj_weight: torch.Tensor,
- q_proj_bias: Optional[torch.Tensor],
- k_proj_weight: torch.Tensor,
- k_proj_bias: Optional[torch.Tensor],
- v_proj_weight: torch.Tensor,
- v_proj_bias: Optional[torch.Tensor],
- dim: int,
- process_group: dist.ProcessGroup,
- sp_size: int,
- sp_rank: int,
-) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
- return AsyncAllGatherProjForTwo.apply(
- hidden_states,
- context,
- q_proj_weight,
- q_proj_bias,
- k_proj_weight,
- k_proj_bias,
- v_proj_weight,
- v_proj_bias,
- dim,
- process_group,
- sp_size,
- sp_rank,
- )
diff --git a/open_sora/utils/data.py b/open_sora/utils/data.py
deleted file mode 100644
index a68b760..0000000
--- a/open_sora/utils/data.py
+++ /dev/null
@@ -1,356 +0,0 @@
-import os
-from typing import List, Optional, Tuple, Union
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-from colossalai.utils import get_current_device
-from datasets import Dataset as HFDataset
-from datasets import dataset_dict, load_from_disk
-from diffusers.models import AutoencoderKL
-from torch.utils.data import ConcatDataset, Dataset
-from torchvision.io import read_video
-from transformers import AutoModel
-
-DatasetType = Union[Dataset, ConcatDataset, dataset_dict.Dataset]
-PathType = Union[str, os.PathLike]
-
-
-def ceil_to_multiple(x: int, multiple: int) -> int:
- m = x % multiple
- if m == 0:
- return x
- return x + multiple - m
-
-
-def video2col(video_4d: torch.Tensor, patch_size: int) -> torch.Tensor:
- """
- Convert a 4D video tensor to a 2D tensor where each row is a patch of the video.
-
- Args:
- video_4d (torch.Tensor): A tensor of shape [T, C, H, W]
- patch_size (int): The size of the patches.
-
- Returns:
- torch.Tensor: A tensor of shape [S, C, P, P] where S is the number of patches and P is the patch size.
- """
- t, c, h, w = video_4d.shape
- out = []
- for y in range(0, h, patch_size):
- for x in range(0, w, patch_size):
- if y + patch_size > h or x + patch_size > w:
- continue
- patch = video_4d[:, :, y : y + patch_size, x : x + patch_size]
- out.append(patch)
- # [S, C, P, P]
- return torch.stack(out, dim=1).view(-1, c, patch_size, patch_size)
-
-
-def col2video(
- patches: torch.Tensor, video_shape: Tuple[int, int, int, int]
-) -> torch.Tensor:
- """
- Convert a 2D tensor of patches to a 4D video tensor.
-
- Args:
- patches (torch.Tensor): A tensor of shape [S, C, P, P] where S is the number of patches and P is the patch size.
- video_shape (Tuple[int, int, int, int]): The shape of the video tensor [T, C, H, W].
-
- Returns:
- torch.Tensor: A tensor of shape [T, C, H, W].
- """
- t, c, h, w = video_shape
- video = torch.empty(t, c, h, w, dtype=patches.dtype, device=patches.device)
- patch_size = patches.shape[2]
- num_x_patches = w // patch_size
- patches = patches.view(t, -1, c, patch_size, patch_size)
- for y in range(0, h, patch_size):
- for x in range(0, w, patch_size):
- if y + patch_size > h or x + patch_size > w:
- continue
- # [T, C, P, P]
- patch = patches[:, (y // patch_size) * num_x_patches + x // patch_size]
- video[:, :, y : y + patch_size, x : x + patch_size].copy_(patch)
- return video
-
-
-def pad_sequences(
- sequences: List[torch.Tensor], pad_to_multiple: Optional[int] = None
-) -> Tuple[torch.Tensor, torch.Tensor]:
- """Pad a list of sequences.
-
- Args:
- sequences (List[torch.Tensor]): Each sequence is a tensor of shape [T, ...].
-
- Returns:
- Tuple[torch.Tensor, torch.Tensor]: Padded batch of sequences ([B, T, ...]) and padding mask ([B, T]).
- """
- max_len = max([sequence.shape[0] for sequence in sequences])
- if pad_to_multiple is not None:
- max_len = ceil_to_multiple(max_len, pad_to_multiple)
- padded_sequences = [
- F.pad(
- sequence, [0] * (sequence.ndim - 1) * 2 + [0, max_len - sequence.shape[0]]
- )
- for sequence in sequences
- ]
- padded_sequences = torch.stack(padded_sequences, dim=0)
- padding_mask = torch.zeros(
- padded_sequences.shape[0],
- padded_sequences.shape[1],
- dtype=torch.int,
- device=padded_sequences.device,
- )
- for i, sequence in enumerate(sequences):
- padding_mask[i, : sequence.shape[0]] = 1
- return padded_sequences, padding_mask
-
-
-def patchify_batch(
- videos: List[torch.Tensor], patch_size: int, pad_to_multiple: Optional[int] = None
-) -> Tuple[torch.Tensor, torch.Tensor]:
- """Patchify a batch of videos.
-
- Args:
- videos (List[torch.Tensor]): A list of tensors of shape [T, C, H, W]
- patch_size (int): The size of the patches.
-
- Returns:
- Tuple[torch.Tensor, torch.Tensor]: Padded batch of patches ([B, S, C, P, P]) and padding mask ([B, S]).
- """
- video_patches = [video2col(video, patch_size) for video in videos]
- return pad_sequences(video_patches, pad_to_multiple=pad_to_multiple)
-
-
-def expand_mask_4d(q_mask: torch.Tensor, kv_mask: torch.Tensor) -> torch.Tensor:
- """expand 2D mask to 4D mask
-
- Args:
- q_mask (torch.Tensor): [B, Sq]
- kv_mask (torch.Tensor): [B, Skv]
-
- Returns:
- torch.Tensor: [B, 1, Sq, Skv]
- """
- q_len = q_mask.shape[1]
- mask = kv_mask.unsqueeze(1).repeat(1, q_len, 1)
- mask = mask * q_mask.unsqueeze(-1)
- return mask.unsqueeze(1)
-
-
-def make_batch(
- samples: List[dict],
- video_dir: str,
- pad_to_multiple: Optional[int] = None,
- use_pooled_text: bool = False,
-) -> dict:
- """Make a batch of samples.
-
- Args:
- samples (List[dict]): A list of samples.
-
- Returns:
- dict: A batch of samples.
- """
- videos = [
- read_video(os.path.join(video_dir, sample["video_file"]), pts_unit="sec")[0]
- for sample in samples
- ]
- texts = [sample["text_latent_states"] for sample in samples]
- if use_pooled_text:
- texts = torch.stack(texts, dim=0)
- return {
- "videos": videos,
- "text_latent_states": texts,
- }
- texts, text_padding_mask = pad_sequences(texts, pad_to_multiple=pad_to_multiple)
- return {
- "videos": videos,
- "text_latent_states": texts,
- "text_padding_mask": text_padding_mask,
- }
-
-
-def normalize_video(video: torch.Tensor) -> torch.Tensor:
- return video.float() / 255 - 0.5
-
-
-def unnormalize_video(video: torch.Tensor) -> torch.Tensor:
- return (video + 0.5) * 255
-
-
-class VideoCompressor:
- t_factor: int
- h_w_factor: int
- out_channels: int
-
- def encode(self, video: torch.Tensor) -> torch.Tensor:
- """Encode a video.
-
- Args:
- video (torch.Tensor): [T, H, W, C]
-
- Returns:
- torch.Tensor: [T, C, H, W]
- """
- raise NotImplementedError
-
- def decode(self, latent: torch.Tensor) -> torch.Tensor:
- """Decode a latent tensor.
-
- Args:
- latent (torch.Tensor): [T, C, H, W]
-
- Returns:
- torch.Tensor: [T, H, W, C]
- """
- raise NotImplementedError
-
-
-class RawVideoCompressor(VideoCompressor):
- t_factor = 1
- h_w_factor = 1
- out_channels = 3
-
- def encode(self, video: torch.Tensor) -> torch.Tensor:
- # [T, H, W, C] -> [T, C, H, W]
- return video.permute(0, 3, 1, 2).contiguous()
-
- def decode(self, latent: torch.Tensor) -> torch.Tensor:
- # [T, C, H, W] -> [T, H, W, C]
- return latent.permute(0, 2, 3, 1).contiguous()
-
-
-class VqvaeVideoCompressor(VideoCompressor):
- t_factor = 2
- h_w_factor = 4
-
- def __init__(self, vqvae: nn.Module):
- self.vqvae = vqvae
- self.out_channels = vqvae.embedding_dim
-
- def encode(self, video: torch.Tensor) -> torch.Tensor:
- # [T, H, W, C] -> [B, C, T, H, W]
- video = video.permute(3, 0, 1, 2).unsqueeze(0)
- latent_indices, embeddings = self.vqvae.encode(video, include_embeddings=True)
- # [B, C, T, H, W] -> [T, C, H, W]
- return embeddings.squeeze(0).permute(1, 0, 2, 3)
-
- def decode(self, latent: torch.Tensor) -> torch.Tensor:
- # [T, C, H, W] -> [B, C, T, H, W]
- latent = latent.permute(1, 0, 2, 3).unsqueeze(0)
- video = self.vqvae.decode_from_embeddings(latent)
- # [B, C, T, H, W] -> [T, H, W, C]
- video = video.squeeze(0).permute(1, 2, 3, 0)
- return video
-
-
-class VaeVideoCompressor(VideoCompressor):
- t_factor = 1
- h_w_factor = 8
- out_channels = 4
-
- def __init__(self, vae: nn.Module):
- self.vae = vae
-
- def encode(self, video: torch.Tensor) -> torch.Tensor:
- # [T, H, W, C] -> [T, C, H, W]
- video = video.permute(0, 3, 1, 2)
- return self.vae.encode(video).latent_dist.sample().mul_(0.18215)
-
- def decode(self, latent: torch.Tensor) -> torch.Tensor:
- video = self.vae.decode(latent / 0.18215).sample
- # [T, C, H, W] -> [T, H, W, C]
- return video.permute(0, 2, 3, 1).contiguous()
-
-
-def create_video_compressor(
- compressor_type: str,
- vqvae_path="hpcai-tech/vqvae",
- vae_path="stabilityai/sd-vae-ft-mse",
-) -> VideoCompressor:
- if compressor_type == "raw":
- return RawVideoCompressor()
- if compressor_type == "vqvae":
- vqvae = (
- AutoModel.from_pretrained(vqvae_path, trust_remote_code=True)
- .to(get_current_device())
- .eval()
- )
- return VqvaeVideoCompressor(vqvae)
- if compressor_type == "vae":
- vae = AutoencoderKL.from_pretrained(vae_path).to(get_current_device()).eval()
- return VaeVideoCompressor(vae)
- raise ValueError(f"Unsupported video compressor type {compressor_type}")
-
-
-def preprocess_batch(
- batch: dict,
- patch_size: int,
- video_compressor: VideoCompressor,
- device=None,
- pad_to_multiple: Optional[int] = None,
- model_arch: str = "cross-attn",
-) -> dict:
- if device is None:
- device = get_current_device()
- videos = []
- for video in batch.pop("videos"):
- video = video.to(device)
- video = normalize_video(video)
- video = video_compressor.encode(video)
- videos.append(video)
- video_latent_states, video_padding_mask = patchify_batch(
- videos, patch_size, pad_to_multiple
- )
- batch["video_latent_states"] = video_latent_states
- batch["video_padding_mask"] = video_padding_mask
- if model_arch == "adaln":
- batch["attention_mask"] = expand_mask_4d(video_padding_mask, video_padding_mask)
- else:
- text_padding_mask = batch.pop("text_padding_mask").to(device)
- if model_arch == "cross-attn":
- batch["attention_mask"] = expand_mask_4d(
- video_padding_mask, text_padding_mask
- )
- else:
- attention_mask = torch.cat([text_padding_mask, video_padding_mask], dim=1)
- batch["attention_mask"] = expand_mask_4d(attention_mask, attention_mask)
- batch["text_latent_states"] = batch["text_latent_states"].to(device)
- return batch
-
-
-def load_datasets(
- dataset_paths: Union[PathType, List[PathType]], mode: str = "train"
-) -> Optional[DatasetType]:
- """
- Load pre-tokenized dataset.
- Each instance of dataset is a dictionary with
- `{'input_ids': List[int], 'labels': List[int], sequence: str}` format.
- """
- mode_map = {"train": "train", "dev": "validation", "test": "test"}
- assert mode in tuple(
- mode_map
- ), f"Unsupported mode {mode}, it must be in {tuple(mode_map)}"
-
- if isinstance(dataset_paths, (str, os.PathLike)):
- dataset_paths = [dataset_paths]
-
- datasets = [] # `List[datasets.dataset_dict.Dataset]`
- for ds_path in dataset_paths:
- ds_path = os.path.abspath(ds_path)
- assert os.path.exists(ds_path), f"Not existed file path {ds_path}"
- ds_dict = load_from_disk(
- dataset_path=ds_path, keep_in_memory=False
- ).with_format("torch")
- if isinstance(ds_dict, HFDataset):
- datasets.append(ds_dict)
- else:
- if mode_map[mode] in ds_dict:
- datasets.append(ds_dict[mode_map[mode]])
- if len(datasets) == 0:
- return None
- if len(datasets) == 1:
- return datasets.pop()
- return ConcatDataset(datasets=datasets)
diff --git a/open_sora/utils/plugin.py b/open_sora/utils/plugin.py
deleted file mode 100644
index f687b41..0000000
--- a/open_sora/utils/plugin.py
+++ /dev/null
@@ -1,102 +0,0 @@
-import random
-from typing import Optional
-
-import numpy as np
-import torch
-from colossalai.booster.plugin import LowLevelZeroPlugin
-from colossalai.cluster import ProcessGroupMesh
-from torch.utils.data import DataLoader
-from torch.utils.data.distributed import DistributedSampler
-
-DP_AXIS, SP_AXIS = 0, 1
-
-
-class ZeroSeqParallelPlugin(LowLevelZeroPlugin):
- def __init__(
- self,
- sp_size: int = 1,
- stage: int = 2,
- precision: str = "fp16",
- initial_scale: float = 2**32,
- min_scale: float = 1,
- growth_factor: float = 2,
- backoff_factor: float = 0.5,
- growth_interval: int = 1000,
- hysteresis: int = 2,
- max_scale: float = 2**32,
- max_norm: float = 0.0,
- norm_type: float = 2.0,
- reduce_bucket_size_in_m: int = 12,
- communication_dtype: Optional[torch.dtype] = None,
- overlap_communication: bool = True,
- cpu_offload: bool = False,
- master_weights: bool = True,
- verbose: bool = False,
- ) -> None:
- super().__init__(
- stage=stage,
- precision=precision,
- initial_scale=initial_scale,
- min_scale=min_scale,
- growth_factor=growth_factor,
- backoff_factor=backoff_factor,
- growth_interval=growth_interval,
- hysteresis=hysteresis,
- max_scale=max_scale,
- max_norm=max_norm,
- norm_type=norm_type,
- reduce_bucket_size_in_m=reduce_bucket_size_in_m,
- communication_dtype=communication_dtype,
- overlap_communication=overlap_communication,
- cpu_offload=cpu_offload,
- master_weights=master_weights,
- verbose=verbose,
- )
- self.sp_size = sp_size
- assert self.world_size % sp_size == 0, "world_size must be divisible by sp_size"
- self.dp_size = self.world_size // sp_size
- self.pg_mesh = ProcessGroupMesh(self.dp_size, self.sp_size)
- self.dp_group = self.pg_mesh.get_group_along_axis(DP_AXIS)
- self.sp_group = self.pg_mesh.get_group_along_axis(SP_AXIS)
- self.dp_rank = self.pg_mesh.coordinate(DP_AXIS)
- self.sp_rank = self.pg_mesh.coordinate(SP_AXIS)
-
- def __del__(self):
- """Destroy the prcess groups in ProcessGroupMesh"""
- self.pg_mesh.destroy_mesh_process_groups()
-
- def prepare_dataloader(
- self,
- dataset,
- batch_size,
- shuffle=False,
- seed=1024,
- drop_last=False,
- pin_memory=False,
- num_workers=0,
- distributed_sampler_cls=None,
- **kwargs
- ):
- _kwargs = kwargs.copy()
- distributed_sampler_cls = distributed_sampler_cls or DistributedSampler
- sampler = distributed_sampler_cls(
- dataset, num_replicas=self.dp_size, rank=self.dp_rank, shuffle=shuffle
- )
-
- # Deterministic dataloader
- def seed_worker(worker_id):
- worker_seed = seed
- np.random.seed(worker_seed)
- torch.manual_seed(worker_seed)
- random.seed(worker_seed)
-
- return DataLoader(
- dataset,
- batch_size=batch_size,
- sampler=sampler,
- worker_init_fn=seed_worker,
- drop_last=drop_last,
- pin_memory=pin_memory,
- num_workers=num_workers,
- **_kwargs,
- )
diff --git a/requirements.txt b/requirements.txt
deleted file mode 100644
index 3a8077d..0000000
--- a/requirements.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-torch
-torchvision
-datasets
-transformers
-av
-tensorboard
-git+https://github.com/hpcaitech/ColossalAI.git@main#egg=colossalai
-timm
-diffusers==0.24.0
\ No newline at end of file
diff --git a/sample.py b/sample.py
deleted file mode 100644
index 67aec4b..0000000
--- a/sample.py
+++ /dev/null
@@ -1,150 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""
-Sample new images from a pre-trained DiT.
-"""
-import torch
-
-torch.backends.cuda.matmul.allow_tf32 = True
-torch.backends.cudnn.allow_tf32 = True
-import argparse
-
-from colossalai.utils import get_current_device
-from torchvision.io import write_video
-from transformers import AutoTokenizer, CLIPTextModel
-
-from open_sora.diffusion import create_diffusion
-from open_sora.modeling import DiT_models
-from open_sora.modeling.dit import SUPPORTED_MODEL_ARCH
-from open_sora.utils.data import col2video, create_video_compressor, unnormalize_video
-
-
-def main(args):
- # Setup PyTorch:
- torch.manual_seed(args.seed)
- torch.set_grad_enabled(False)
- device = get_current_device()
-
- video_compressor = create_video_compressor(args.compressor)
- model_kwargs = {
- "in_channels": video_compressor.out_channels,
- "model_arch": args.model_arch,
- }
-
- text_model = CLIPTextModel.from_pretrained(args.text_model).to(device).eval()
- tokenizer = AutoTokenizer.from_pretrained(args.text_model)
-
- model = DiT_models[args.model](**model_kwargs).to(device).eval()
- patch_size = model.patch_size
- model.load_state_dict(torch.load(args.ckpt))
- diffusion = create_diffusion(str(args.num_sampling_steps))
-
- # Create sampling noise:
- text_inputs = tokenizer(args.text, return_tensors="pt")
- text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
- text_latent_states = text_model(**text_inputs)
- if args.model_arch == "adaln":
- text_latent_states = text_latent_states.pooler_output
- else:
- text_latent_states = text_latent_states.last_hidden_state
-
- num_frames = args.fps * args.sec
- z = torch.randn(
- 1,
- (args.height // patch_size // video_compressor.h_w_factor)
- * (args.width // patch_size // video_compressor.h_w_factor)
- * (num_frames // video_compressor.t_factor),
- video_compressor.out_channels,
- patch_size,
- patch_size,
- device=device,
- )
-
- # Setup classifier-free guidance:
- model_kwargs = {}
- if not args.disable_cfg:
- z = torch.cat([z, z], 0)
- model_kwargs["text_latent_states"] = torch.cat(
- [text_latent_states, torch.zeros_like(text_latent_states)], 0
- )
- model_kwargs["cfg_scale"] = args.cfg_scale
- else:
- model_kwargs["text_latent_states"] = text_latent_states
- context_len = (
- text_latent_states.shape[1] if args.model_arch == "cross-attn" else z.shape[1]
- )
- model_kwargs["attention_mask"] = torch.ones(
- z.shape[0],
- 1,
- z.shape[1],
- context_len,
- device=device,
- dtype=torch.int,
- )
-
- # Sample images:
- samples = diffusion.p_sample_loop(
- model if args.disable_cfg else model.forward_with_cfg,
- z.shape,
- z,
- clip_denoised=False,
- model_kwargs=model_kwargs,
- progress=True,
- device=device,
- )
- if not args.disable_cfg:
- samples, _ = samples.chunk(2, dim=0) # Remove null class samples
- samples = col2video(
- samples.squeeze(),
- (
- num_frames // video_compressor.t_factor,
- video_compressor.out_channels,
- args.height // video_compressor.h_w_factor,
- args.width // video_compressor.h_w_factor,
- ),
- )
- samples = video_compressor.decode(samples)
- samples = unnormalize_video(samples).to(torch.uint8)
-
- write_video("sample.mp4", samples.cpu(), args.fps)
-
-
-if __name__ == "__main__":
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "-m", "--model", type=str, choices=list(DiT_models.keys()), default="DiT-S/8"
- )
- parser.add_argument(
- "-x", "--model_arch", choices=SUPPORTED_MODEL_ARCH, default="cross-attn"
- )
- parser.add_argument(
- "--text",
- type=str,
- default="a cartoon animals runs through an ice cave in a video game",
- )
- parser.add_argument("--cfg-scale", type=float, default=4.0)
- parser.add_argument("--num-sampling-steps", type=int, default=250)
- parser.add_argument("--seed", type=int, default=0)
- parser.add_argument(
- "--ckpt",
- type=str,
- required=True,
- help="Optional path to a DiT checkpoint (default: auto-download a pre-trained DiT-XL/2 model).",
- )
- parser.add_argument(
- "-c", "--compressor", choices=["raw", "vqvae", "vae"], default="raw"
- )
- parser.add_argument(
- "--text_model", type=str, default="openai/clip-vit-base-patch32"
- )
- parser.add_argument("--width", type=int, default=480)
- parser.add_argument("--height", type=int, default=320)
- parser.add_argument("--fps", type=int, default=15)
- parser.add_argument("--sec", type=int, default=8)
- parser.add_argument("--disable-cfg", action="store_true", default=False)
- args = parser.parse_args()
- main(args)
diff --git a/scripts/data/collate_msr_vtt_dataset.py b/scripts/data/collate_msr_vtt_dataset.py
deleted file mode 100644
index 04f63cb..0000000
--- a/scripts/data/collate_msr_vtt_dataset.py
+++ /dev/null
@@ -1,182 +0,0 @@
-import argparse
-import json
-import multiprocessing
-import os
-import shutil
-import warnings
-from typing import Dict, Tuple
-
-from tqdm import tqdm
-
-DEFAULT_TYPES = ["train", "val", "test"]
-
-
-def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument("-d", "--data-path", type=str, help="The path to the MSR-VTT dataset")
- parser.add_argument("-o", "--output-path", type=str, help="The output to the collated MSR-VTT dataset")
- return parser.parse_args()
-
-
-def get_annotations(root_path: str):
- """
- Get the annotation data from the MSR-VTT dataset. The annotations are in the format of:
-
- {
- "annotations": [
- {
- "image_id": "video1",
- "caption": "some
- }
- ]
- }
-
- Args:
- root_path (str): The root path to the MSR-VTT dataset
- """
- annotation_json_file = os.path.join(root_path, "annotation/MSR_VTT.json")
- with open(annotation_json_file, "r") as f:
- data = json.load(f)
- return data
-
-
-def get_video_list(root_path: str, dataset_type: str):
- """
- Get the list of videos in the dataset split.
-
- Args:
- root_path (str): The root path to the MSR-VTT dataset
- dataset_type (str): The dataset split type. It should be one of "train", "val", or "test"
- """
- assert dataset_type in DEFAULT_TYPES, f"Expected the dataset type to be in {DEFAULT_TYPES}, but got {dataset_type}"
- dataset_file_path = os.path.join(root_path, f"structured-symlinks/{dataset_type}_list_full.txt")
- with open(dataset_file_path, "r") as f:
- video_list = f.readlines()
- video_list = [x.strip() for x in video_list]
- return video_list
-
-
-def copy_video(video_id: str, root_path: str, output_path: str, dataset_type: str):
- """
- Copy the video from the source path to the destination path.
-
- Args:
- video_id (str): The video id
- root_path (str): The root path to the MSR-VTT dataset
- output_path (str): The output path to the collated MSR-VTT dataset
- dataset_type (str): The dataset split type. It should be one of "train", "val", or "test"
- """
- assert dataset_type in DEFAULT_TYPES, f"Expected the dataset type to be in {DEFAULT_TYPES}, but got {dataset_type}"
- src_file = os.path.join(root_path, f"videos/all/{video_id}.mp4")
- dst_folder = os.path.join(output_path, f"{dataset_type}/videos")
- dst_file = os.path.join(dst_folder, f"{video_id}.mp4")
- os.makedirs(dst_folder, exist_ok=True)
-
- # create symlink
- assert os.path.isfile(src_file), f"Expected the source file {src_file} to exist"
- if not os.path.islink(dst_file):
- shutil.copy(src_file, dst_file)
-
-
-def get_annotation_file_path(output_path: str, dataset_type: str):
- file_path = os.path.join(output_path, f"{dataset_type}/annotations.json")
- return file_path
-
-
-def collate_annotation_files(
- annotations: Dict,
- root_path: str,
- output_path: str,
-):
- """
- Collate the video and caption data into a single folder.
-
- Args:
- annotations (Dict): The annotations data
- root_path (str): The root path to the MSR-VTT dataset
- output_path (str): The output path to the collated MSR-VTT dataset
- """
- # get all video list
- train_video_list = get_video_list(root_path, "train")
- val_video_list = get_video_list(root_path, "val")
- test_video_list = get_video_list(root_path, "test")
-
- # iterate over annotations
- collated_train_data = []
- collated_val_data = []
- collated_test_data = []
-
- print("Collating annotations files")
-
- for anno in tqdm(annotations["annotations"]):
- video_id = anno["image_id"]
- caption = anno["caption"]
-
- obj = {"file": f"{video_id}.mp4", "captions": [caption]}
-
- if video_id in train_video_list:
- collated_train_data.append(obj)
- elif video_id in val_video_list:
- collated_val_data.append(obj)
- elif video_id in test_video_list:
- collated_test_data.append(obj)
- else:
- warnings.warn(f"Video {video_id} not found in any of the dataset splits")
-
- def _save_caption_files(obj, dataset_type):
- dst_file = get_annotation_file_path(output_path, dataset_type)
- os.makedirs(os.path.dirname(dst_file), exist_ok=True)
- with open(dst_file, "w") as f:
- json.dump(obj, f, indent=4)
-
- _save_caption_files(collated_train_data, "train")
- _save_caption_files(collated_val_data, "val")
- _save_caption_files(collated_test_data, "test")
-
-
-def copy_file(path_pair: Tuple[str, str]):
- src_path, dst_path = path_pair
- shutil.copyfile(src_path, dst_path)
-
-
-def copy_videos(root_path: str, output_path: str, num_workers: int = 8):
- """
- Batch copy the video files to the output path.
-
- Args:
- root_path (str): The root path to the MSR-VTT dataset
- output_path (str): The output path to the collated MSR-VTT dataset
- num_workers (int): The number of workers to use for the copy operation
- """
- pool = multiprocessing.Pool(num_workers)
-
- for dataset_type in DEFAULT_TYPES:
- print(f"Copying videos for the {dataset_type} dataset")
- annotation_file_path = get_annotation_file_path(output_path, dataset_type)
- output_video_folder_path = os.path.join(output_path, f"{dataset_type}/videos")
- os.makedirs(output_video_folder_path, exist_ok=True)
-
- with open(annotation_file_path, "r") as f:
- annotation_data = json.load(f)
-
- video_ids = [obj["file"] for obj in annotation_data]
- unique_video_ids = list(set(video_ids))
-
- path_pairs = [
- (os.path.join(root_path, f"videos/all/{video_id}"), os.path.join(output_video_folder_path, video_id))
- for video_id in unique_video_ids
- ]
-
- for _ in tqdm(pool.imap_unordered(copy_file, path_pairs), total=len(path_pairs)):
- pass
-
-
-def main():
- args = parse_args()
- annotations = get_annotations(args.data_path)
- collate_annotation_files(annotations, args.data_path, args.output_path)
- copy_videos(args.data_path, args.output_path)
-
-
-if __name__ == "__main__":
- main()
diff --git a/scripts/data/download_msr_vtt_dataset.sh b/scripts/data/download_msr_vtt_dataset.sh
deleted file mode 100644
index 141a7d6..0000000
--- a/scripts/data/download_msr_vtt_dataset.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-
-# get root dir
-FOLDER_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-ROOT_DIR=$FOLDER_DIR/../..
-
-# download at root dir
-cd $ROOT_DIR
-mkdir -p dataset && cd ./dataset
-wget https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip
-unzip MSRVTT.zip
diff --git a/scripts/data/preprocess_data.py b/scripts/data/preprocess_data.py
deleted file mode 100644
index bc0dfbb..0000000
--- a/scripts/data/preprocess_data.py
+++ /dev/null
@@ -1,135 +0,0 @@
-import argparse
-import math
-import os
-
-import torch
-from datasets import load_dataset
-from torchvision.io import read_video
-from transformers import AutoTokenizer, CLIPTextModel
-
-EMPTY_SAMPLE = {"video_file": [], "text_latent_states": []}
-
-
-def process_text(text, tokenizer, text_model, use_pooled_text):
- inputs = tokenizer(text, padding=True, return_tensors="pt")
- inputs = {k: v.cuda() for k, v in inputs.items()}
- outputs = text_model(**inputs)
- if use_pooled_text:
- return list(outputs.pooler_output.cpu().unbind(0))
- output_states = []
- for i, x in enumerate(outputs.last_hidden_state):
- valid_x = x[inputs["attention_mask"][i].bool()]
- output_states.append(valid_x.cpu())
- return output_states
-
-
-def process_item(item, video_dir, tokenizer, text_model, use_pooled_text):
- video_path = os.path.join(video_dir, item["file"])
- video = read_video(video_path, pts_unit="sec")[0]
- if video.size(0) > 600:
- return EMPTY_SAMPLE
- text_latent_states = process_text(
- item["captions"], tokenizer, text_model, use_pooled_text
- )
- torch.cuda.empty_cache()
- return {
- "video_file": [item["file"]] * len(text_latent_states),
- "text_latent_states": text_latent_states,
- }
-
-
-def process_batch(batch, video_dir, tokenizer, text_model, use_pooled_text):
- item = {"file": batch["file"][0], "captions": batch["captions"][0]}
- return process_item(item, video_dir, tokenizer, text_model, use_pooled_text)
-
-
-def process_dataset(
- captions_file,
- video_dir,
- output_dir,
- num_spliced_dataset_bins=10,
- text_model="openai/clip-vit-base-patch32",
- use_pooled_text=False,
-):
- tokenizer = AutoTokenizer.from_pretrained(text_model)
- text_model = CLIPTextModel.from_pretrained(text_model).cuda().eval()
-
- if not os.path.exists(output_dir):
- os.makedirs(output_dir)
-
- # Prepare to data splitting.
- train_splits = []
- split_interval = math.ceil(100 / num_spliced_dataset_bins)
- for i in range(0, 100, split_interval):
- start = i
- end = i + split_interval
- if end > 100:
- end = 100
- train_splits.append(f"train[{start}%:{end}%]")
-
- ds = load_dataset(
- "json",
- data_files=captions_file,
- keep_in_memory=False,
- split=train_splits,
- num_proc=1,
- )
-
- for i, part_ds in enumerate(ds):
- print(f"Processing part {i+1}/{len(ds)}")
- part_ds = part_ds.with_format("torch")
- part_ds = part_ds.map(
- process_batch,
- fn_kwargs={
- "video_dir": video_dir,
- "tokenizer": tokenizer,
- "text_model": text_model,
- "use_pooled_text": use_pooled_text,
- },
- batched=True,
- batch_size=1,
- keep_in_memory=False,
- remove_columns=part_ds.column_names,
- )
- output_path = os.path.join(output_dir, f"part-{i:05d}")
- part_ds.save_to_disk(output_path)
-
-
-if __name__ == "__main__":
- parser = argparse.ArgumentParser(description="Preprocess data")
- parser.add_argument(
- "-c",
- "--captions-file",
- type=str,
- help="Path to the captions file. It should be a JSON file or a JSONL file",
- )
- parser.add_argument(
- "-v", "--video-dir", type=str, help="Path to the video directory"
- )
- parser.add_argument(
- "-o", "--output_dir", type=str, help="Path to the output directory"
- )
- parser.add_argument(
- "-n",
- "--num_spliced_dataset_bins",
- type=int,
- default=10,
- help="Number of bins for spliced dataset",
- )
- parser.add_argument(
- "--text_model",
- type=str,
- default="openai/clip-vit-base-patch32",
- help="CLIP text model",
- )
- parser.add_argument("--use_pooled_text", action="store_true", default=False)
- args = parser.parse_args()
- process_dataset(
- args.captions_file,
- args.video_dir,
- args.output_dir,
- args.num_spliced_dataset_bins,
- args.text_model,
- args.use_pooled_text,
- )
diff --git a/scripts/sample/latte/pipeline_videogen.py b/scripts/sample/latte/pipeline_videogen.py
deleted file mode 100644
index 8acd627..0000000
--- a/scripts/sample/latte/pipeline_videogen.py
+++ /dev/null
@@ -1,792 +0,0 @@
-# All rights reserved.
-# Copyright 2024 Vchitect/Latte
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# copied from https://github.com/Vchitect/Latte/blob/main/sample/pipeline_videogen.py
-
-import html
-import inspect
-import re
-import urllib.parse as ul
-from dataclasses import dataclass
-from typing import Callable, List, Optional, Tuple, Union
-
-import einops
-import torch
-from diffusers.image_processor import VaeImageProcessor
-from diffusers.models import AutoencoderKL, Transformer2DModel
-from diffusers.pipelines.pipeline_utils import DiffusionPipeline
-from diffusers.schedulers import DPMSolverMultistepScheduler
-from diffusers.utils import (
- BACKENDS_MAPPING,
- BaseOutput,
- is_bs4_available,
- is_ftfy_available,
- logging,
- replace_example_docstring,
-)
-from diffusers.utils.torch_utils import randn_tensor
-from transformers import T5EncoderModel, T5Tokenizer
-
-logger = logging.get_logger(__name__) # pylint: disable=invalid-name
-
-if is_bs4_available():
- from bs4 import BeautifulSoup
-
-if is_ftfy_available():
- import ftfy
-
-
-EXAMPLE_DOC_STRING = """
- Examples:
- ```py
- >>> import torch
- >>> from diffusers import PixArtAlphaPipeline
-
- >>> # You can replace the checkpoint id with "PixArt-alpha/PixArt-XL-2-512x512" too.
- >>> pipe = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-1024-MS", torch_dtype=torch.float16)
- >>> # Enable memory optimizations.
- >>> pipe.enable_model_cpu_offload()
-
- >>> prompt = "A small cactus with a happy face in the Sahara desert."
- >>> image = pipe(prompt).images[0]
- ```
-"""
-
-
-@dataclass
-class VideoPipelineOutput(BaseOutput):
- video: torch.Tensor
-
-
-class VideoGenPipeline(DiffusionPipeline):
- r"""
- Pipeline for text-to-image generation using PixArt-Alpha.
-
- This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the
- library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)
-
- Args:
- vae ([`AutoencoderKL`]):
- Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations.
- text_encoder ([`T5EncoderModel`]):
- Frozen text-encoder. PixArt-Alpha uses
- [T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5EncoderModel), specifically the
- [t5-v1_1-xxl](https://huggingface.co/PixArt-alpha/PixArt-alpha/tree/main/t5-v1_1-xxl) variant.
- tokenizer (`T5Tokenizer`):
- Tokenizer of class
- [T5Tokenizer](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Tokenizer).
- transformer ([`Transformer2DModel`]):
- A text conditioned `Transformer2DModel` to denoise the encoded image latents.
- scheduler ([`SchedulerMixin`]):
- A scheduler to be used in combination with `transformer` to denoise the encoded image latents.
- """
- bad_punct_regex = re.compile(
- r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"
- ) # noqa
-
- _optional_components = ["tokenizer", "text_encoder"]
- model_cpu_offload_seq = "text_encoder->transformer->vae"
-
- def __init__(
- self,
- tokenizer: T5Tokenizer,
- text_encoder: T5EncoderModel,
- vae: AutoencoderKL,
- transformer: Transformer2DModel,
- scheduler: DPMSolverMultistepScheduler,
- ):
- super().__init__()
-
- self.register_modules(
- tokenizer=tokenizer, text_encoder=text_encoder, vae=vae, transformer=transformer, scheduler=scheduler
- )
-
- self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1)
- self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor)
-
- # Adapted from https://github.com/PixArt-alpha/PixArt-alpha/blob/master/diffusion/model/utils.py
- def mask_text_embeddings(self, emb, mask):
- if emb.shape[0] == 1:
- keep_index = mask.sum().item()
- return emb[:, :, :keep_index, :], keep_index # 1, 120, 4096 -> 1 7 4096
- else:
- masked_feature = emb * mask[:, None, :, None] # 1 120 4096
- return masked_feature, emb.shape[2]
-
- # Adapted from diffusers.pipelines.deepfloyd_if.pipeline_if.encode_prompt
- def encode_prompt(
- self,
- prompt: Union[str, List[str]],
- do_classifier_free_guidance: bool = True,
- negative_prompt: str = "",
- num_images_per_prompt: int = 1,
- device: Optional[torch.device] = None,
- prompt_embeds: Optional[torch.FloatTensor] = None,
- negative_prompt_embeds: Optional[torch.FloatTensor] = None,
- clean_caption: bool = False,
- mask_feature: bool = True,
- ):
- r"""
- Encodes the prompt into text encoder hidden states.
-
- Args:
- prompt (`str` or `List[str]`, *optional*):
- prompt to be encoded
- negative_prompt (`str` or `List[str]`, *optional*):
- The prompt not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`
- instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). For
- PixArt-Alpha, this should be "".
- do_classifier_free_guidance (`bool`, *optional*, defaults to `True`):
- whether to use classifier free guidance or not
- num_images_per_prompt (`int`, *optional*, defaults to 1):
- number of images that should be generated per prompt
- device: (`torch.device`, *optional*):
- torch device to place the resulting embeddings on
- prompt_embeds (`torch.FloatTensor`, *optional*):
- Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
- provided, text embeddings will be generated from `prompt` input argument.
- negative_prompt_embeds (`torch.FloatTensor`, *optional*):
- Pre-generated negative text embeddings. For PixArt-Alpha, it's should be the embeddings of the ""
- string.
- clean_caption (bool, defaults to `False`):
- If `True`, the function will preprocess and clean the provided caption before encoding.
- mask_feature: (bool, defaults to `True`):
- If `True`, the function will mask the text embeddings.
- """
- embeds_initially_provided = prompt_embeds is not None and negative_prompt_embeds is not None
-
- if device is None:
- device = self._execution_device
-
- if prompt is not None and isinstance(prompt, str):
- batch_size = 1
- elif prompt is not None and isinstance(prompt, list):
- batch_size = len(prompt)
- else:
- batch_size = prompt_embeds.shape[0]
-
- # See Section 3.1. of the paper.
- max_length = 120
-
- if prompt_embeds is None:
- prompt = self._text_preprocessing(prompt, clean_caption=clean_caption)
- text_inputs = self.tokenizer(
- prompt,
- padding="max_length",
- max_length=max_length,
- truncation=True,
- return_attention_mask=True,
- add_special_tokens=True,
- return_tensors="pt",
- )
- text_input_ids = text_inputs.input_ids
- untruncated_ids = self.tokenizer(prompt, padding="longest", return_tensors="pt").input_ids
-
- if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(
- text_input_ids, untruncated_ids
- ):
- removed_text = self.tokenizer.batch_decode(untruncated_ids[:, max_length - 1 : -1])
- logger.warning(
- "The following part of your input was truncated because CLIP can only handle sequences up to"
- f" {max_length} tokens: {removed_text}"
- )
-
- attention_mask = text_inputs.attention_mask.to(device)
- prompt_embeds_attention_mask = attention_mask
-
- prompt_embeds = self.text_encoder(text_input_ids.to(device), attention_mask=attention_mask)
- prompt_embeds = prompt_embeds[0]
- else:
- prompt_embeds_attention_mask = torch.ones_like(prompt_embeds)
-
- if self.text_encoder is not None:
- dtype = self.text_encoder.dtype
- elif self.transformer is not None:
- dtype = self.transformer.dtype
- else:
- dtype = None
-
- prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
-
- bs_embed, seq_len, _ = prompt_embeds.shape
- # duplicate text embeddings and attention mask for each generation per prompt, using mps friendly method
- prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
- prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)
- prompt_embeds_attention_mask = prompt_embeds_attention_mask.view(bs_embed, -1)
- prompt_embeds_attention_mask = prompt_embeds_attention_mask.repeat(num_images_per_prompt, 1)
-
- # get unconditional embeddings for classifier free guidance
- if do_classifier_free_guidance and negative_prompt_embeds is None:
- uncond_tokens = [negative_prompt] * batch_size
- uncond_tokens = self._text_preprocessing(uncond_tokens, clean_caption=clean_caption)
- max_length = prompt_embeds.shape[1]
- uncond_input = self.tokenizer(
- uncond_tokens,
- padding="max_length",
- max_length=max_length,
- truncation=True,
- return_attention_mask=True,
- add_special_tokens=True,
- return_tensors="pt",
- )
- attention_mask = uncond_input.attention_mask.to(device)
-
- negative_prompt_embeds = self.text_encoder(
- uncond_input.input_ids.to(device),
- attention_mask=attention_mask,
- )
- negative_prompt_embeds = negative_prompt_embeds[0]
-
- if do_classifier_free_guidance:
- # duplicate unconditional embeddings for each generation per prompt, using mps friendly method
- seq_len = negative_prompt_embeds.shape[1]
-
- negative_prompt_embeds = negative_prompt_embeds.to(dtype=dtype, device=device)
-
- negative_prompt_embeds = negative_prompt_embeds.repeat(1, num_images_per_prompt, 1)
- negative_prompt_embeds = negative_prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)
-
- # For classifier free guidance, we need to do two forward passes.
- # Here we concatenate the unconditional and text embeddings into a single batch
- # to avoid doing two forward passes
- else:
- negative_prompt_embeds = None
-
- # print(prompt_embeds.shape) # 1 120 4096
- # print(negative_prompt_embeds.shape) # 1 120 4096
-
- # Perform additional masking.
- if mask_feature and not embeds_initially_provided:
- prompt_embeds = prompt_embeds.unsqueeze(1)
- masked_prompt_embeds, keep_indices = self.mask_text_embeddings(prompt_embeds, prompt_embeds_attention_mask)
- masked_prompt_embeds = masked_prompt_embeds.squeeze(1)
- masked_negative_prompt_embeds = (
- negative_prompt_embeds[:, :keep_indices, :] if negative_prompt_embeds is not None else None
- )
-
- # import torch.nn.functional as F
-
- # padding = (0, 0, 0, 113) # (左, 右, 下, 上)
- # masked_prompt_embeds_ = F.pad(masked_prompt_embeds, padding, "constant", 0)
- # masked_negative_prompt_embeds_ = F.pad(masked_negative_prompt_embeds, padding, "constant", 0)
-
- # print(masked_prompt_embeds == masked_prompt_embeds_[:, :masked_negative_prompt_embeds.shape[1], ...])
-
- return masked_prompt_embeds, masked_negative_prompt_embeds
- # return masked_prompt_embeds_, masked_negative_prompt_embeds_
-
- return prompt_embeds, negative_prompt_embeds
-
- # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.prepare_extra_step_kwargs
- def prepare_extra_step_kwargs(self, generator, eta):
- # prepare extra kwargs for the scheduler step, since not all schedulers have the same signature
- # eta (η) is only used with the DDIMScheduler, it will be ignored for other schedulers.
- # eta corresponds to η in DDIM paper: https://arxiv.org/abs/2010.02502
- # and should be between [0, 1]
-
- accepts_eta = "eta" in set(inspect.signature(self.scheduler.step).parameters.keys())
- extra_step_kwargs = {}
- if accepts_eta:
- extra_step_kwargs["eta"] = eta
-
- # check if the scheduler accepts generator
- accepts_generator = "generator" in set(inspect.signature(self.scheduler.step).parameters.keys())
- if accepts_generator:
- extra_step_kwargs["generator"] = generator
- return extra_step_kwargs
-
- def check_inputs(
- self,
- prompt,
- height,
- width,
- negative_prompt,
- callback_steps,
- prompt_embeds=None,
- negative_prompt_embeds=None,
- ):
- if height % 8 != 0 or width % 8 != 0:
- raise ValueError(f"`height` and `width` have to be divisible by 8 but are {height} and {width}.")
-
- if (callback_steps is None) or (
- callback_steps is not None and (not isinstance(callback_steps, int) or callback_steps <= 0)
- ):
- raise ValueError(
- f"`callback_steps` has to be a positive integer but is {callback_steps} of type"
- f" {type(callback_steps)}."
- )
-
- if prompt is not None and prompt_embeds is not None:
- raise ValueError(
- f"Cannot forward both `prompt`: {prompt} and `prompt_embeds`: {prompt_embeds}. Please make sure to"
- " only forward one of the two."
- )
- elif prompt is None and prompt_embeds is None:
- raise ValueError(
- "Provide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined."
- )
- elif prompt is not None and (not isinstance(prompt, str) and not isinstance(prompt, list)):
- raise ValueError(f"`prompt` has to be of type `str` or `list` but is {type(prompt)}")
-
- if prompt is not None and negative_prompt_embeds is not None:
- raise ValueError(
- f"Cannot forward both `prompt`: {prompt} and `negative_prompt_embeds`:"
- f" {negative_prompt_embeds}. Please make sure to only forward one of the two."
- )
-
- if negative_prompt is not None and negative_prompt_embeds is not None:
- raise ValueError(
- f"Cannot forward both `negative_prompt`: {negative_prompt} and `negative_prompt_embeds`:"
- f" {negative_prompt_embeds}. Please make sure to only forward one of the two."
- )
-
- if prompt_embeds is not None and negative_prompt_embeds is not None:
- if prompt_embeds.shape != negative_prompt_embeds.shape:
- raise ValueError(
- "`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but"
- f" got: `prompt_embeds` {prompt_embeds.shape} != `negative_prompt_embeds`"
- f" {negative_prompt_embeds.shape}."
- )
-
- # Copied from diffusers.pipelines.deepfloyd_if.pipeline_if.IFPipeline._text_preprocessing
- def _text_preprocessing(self, text, clean_caption=False):
- if clean_caption and not is_bs4_available():
- logger.warn(BACKENDS_MAPPING["bs4"][-1].format("Setting `clean_caption=True`"))
- logger.warn("Setting `clean_caption` to False...")
- clean_caption = False
-
- if clean_caption and not is_ftfy_available():
- logger.warn(BACKENDS_MAPPING["ftfy"][-1].format("Setting `clean_caption=True`"))
- logger.warn("Setting `clean_caption` to False...")
- clean_caption = False
-
- if not isinstance(text, (tuple, list)):
- text = [text]
-
- def process(text: str):
- if clean_caption:
- text = self._clean_caption(text)
- text = self._clean_caption(text)
- else:
- text = text.lower().strip()
- return text
-
- return [process(t) for t in text]
-
- # Copied from diffusers.pipelines.deepfloyd_if.pipeline_if.IFPipeline._clean_caption
- def _clean_caption(self, caption):
- caption = str(caption)
- caption = ul.unquote_plus(caption)
- caption = caption.strip().lower()
- caption = re.sub("<person>", "person", caption)
- # urls:
- caption = re.sub(
- r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
- "",
- caption,
- ) # regex for urls
- caption = re.sub(
- r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
- "",
- caption,
- ) # regex for urls
- # html:
- caption = BeautifulSoup(caption, features="html.parser").text
-
- # @<nickname>
- caption = re.sub(r"@[\w\d]+\b", "", caption)
-
- # 31C0—31EF CJK Strokes
- # 31F0—31FF Katakana Phonetic Extensions
- # 3200—32FF Enclosed CJK Letters and Months
- # 3300—33FF CJK Compatibility
- # 3400—4DBF CJK Unified Ideographs Extension A
- # 4DC0—4DFF Yijing Hexagram Symbols
- # 4E00—9FFF CJK Unified Ideographs
- caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)
- caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)
- caption = re.sub(r"[\u3200-\u32ff]+", "", caption)
- caption = re.sub(r"[\u3300-\u33ff]+", "", caption)
- caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)
- caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)
- caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)
- #######################################################
-
- # все виды тире / all types of dash --> "-"
- caption = re.sub(
- r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa
- "-",
- caption,
- )
-
- # кавычки к одному стандарту
- caption = re.sub(r"[`´«»“”¨]", '"', caption)
- caption = re.sub(r"[‘’]", "'", caption)
-
- # &quot;
- caption = re.sub(r"&quot;?", "", caption)
- # &amp
- caption = re.sub(r"&amp", "", caption)
-
- # ip adresses:
- caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)
-
- # article ids:
- caption = re.sub(r"\d:\d\d\s+$", "", caption)
-
- # \n
- caption = re.sub(r"\\n", " ", caption)
-
- # "#123"
- caption = re.sub(r"#\d{1,3}\b", "", caption)
- # "#12345.."
- caption = re.sub(r"#\d{5,}\b", "", caption)
- # "123456.."
- caption = re.sub(r"\b\d{6,}\b", "", caption)
- # filenames:
- caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)
-
- #
- caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""
- caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""
-
- caption = re.sub(self.bad_punct_regex, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT
- caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "
-
- # this-is-my-cute-cat / this_is_my_cute_cat
- regex2 = re.compile(r"(?:\-|\_)")
- if len(re.findall(regex2, caption)) > 3:
- caption = re.sub(regex2, " ", caption)
-
- caption = ftfy.fix_text(caption)
- caption = html.unescape(html.unescape(caption))
-
- caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640
- caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc
- caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231
-
- caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)
- caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)
- caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)
- caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)
- caption = re.sub(r"\bpage\s+\d+\b", "", caption)
-
- caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...
-
- caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)
-
- caption = re.sub(r"\b\s+\:\s+", r": ", caption)
- caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)
- caption = re.sub(r"\s+", " ", caption)
-
- caption.strip()
-
- caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)
- caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)
- caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)
- caption = re.sub(r"^\.\S+$", "", caption)
-
- return caption.strip()
-
- # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.prepare_latents
- def prepare_latents(
- self, batch_size, num_channels_latents, video_length, height, width, dtype, device, generator, latents=None
- ):
- shape = (
- batch_size,
- num_channels_latents,
- video_length,
- height // self.vae_scale_factor,
- width // self.vae_scale_factor,
- )
- if isinstance(generator, list) and len(generator) != batch_size:
- raise ValueError(
- f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"
- f" size of {batch_size}. Make sure the batch size matches the length of the generators."
- )
-
- if latents is None:
- latents = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
- else:
- latents = latents.to(device)
-
- # scale the initial noise by the standard deviation required by the scheduler
- latents = latents * self.scheduler.init_noise_sigma
- return latents
-
- @torch.no_grad()
- @replace_example_docstring(EXAMPLE_DOC_STRING)
- def __call__(
- self,
- prompt: Union[str, List[str]] = None,
- negative_prompt: str = "",
- num_inference_steps: int = 20,
- timesteps: List[int] = None,
- guidance_scale: float = 4.5,
- num_images_per_prompt: Optional[int] = 1,
- video_length: Optional[int] = None,
- height: Optional[int] = None,
- width: Optional[int] = None,
- eta: float = 0.0,
- generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
- latents: Optional[torch.FloatTensor] = None,
- prompt_embeds: Optional[torch.FloatTensor] = None,
- negative_prompt_embeds: Optional[torch.FloatTensor] = None,
- output_type: Optional[str] = "pil",
- return_dict: bool = True,
- callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None,
- callback_steps: int = 1,
- clean_caption: bool = True,
- mask_feature: bool = True,
- enable_temporal_attentions: bool = True,
- enable_vae_temporal_decoder: bool = False,
- ) -> Union[VideoPipelineOutput, Tuple]:
- """
- Function invoked when calling the pipeline for generation.
-
- Args:
- prompt (`str` or `List[str]`, *optional*):
- The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`.
- instead.
- negative_prompt (`str` or `List[str]`, *optional*):
- The prompt or prompts not to guide the image generation. If not defined, one has to pass
- `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is
- less than `1`).
- num_inference_steps (`int`, *optional*, defaults to 100):
- The number of denoising steps. More denoising steps usually lead to a higher quality image at the
- expense of slower inference.
- timesteps (`List[int]`, *optional*):
- Custom timesteps to use for the denoising process. If not defined, equal spaced `num_inference_steps`
- timesteps are used. Must be in descending order.
- guidance_scale (`float`, *optional*, defaults to 7.0):
- Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
- `guidance_scale` is defined as `w` of equation 2. of [Imagen
- Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >
- 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,
- usually at the expense of lower image quality.
- num_images_per_prompt (`int`, *optional*, defaults to 1):
- The number of images to generate per prompt.
- height (`int`, *optional*, defaults to self.unet.config.sample_size):
- The height in pixels of the generated image.
- width (`int`, *optional*, defaults to self.unet.config.sample_size):
- The width in pixels of the generated image.
- eta (`float`, *optional*, defaults to 0.0):
- Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
- [`schedulers.DDIMScheduler`], will be ignored for others.
- generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
- One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
- to make generation deterministic.
- latents (`torch.FloatTensor`, *optional*):
- Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
- generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
- tensor will ge generated by sampling using the supplied random `generator`.
- prompt_embeds (`torch.FloatTensor`, *optional*):
- Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
- provided, text embeddings will be generated from `prompt` input argument.
- negative_prompt_embeds (`torch.FloatTensor`, *optional*):
- Pre-generated negative text embeddings. For PixArt-Alpha this negative prompt should be "". If not
- provided, negative_prompt_embeds will be generated from `negative_prompt` input argument.
- output_type (`str`, *optional*, defaults to `"pil"`):
- The output format of the generate image. Choose between
- [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.
- return_dict (`bool`, *optional*, defaults to `True`):
- Whether or not to return a [`~pipelines.stable_diffusion.IFPipelineOutput`] instead of a plain tuple.
- callback (`Callable`, *optional*):
- A function that will be called every `callback_steps` steps during inference. The function will be
- called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
- callback_steps (`int`, *optional*, defaults to 1):
- The frequency at which the `callback` function will be called. If not specified, the callback will be
- called at every step.
- clean_caption (`bool`, *optional*, defaults to `True`):
- Whether or not to clean the caption before creating embeddings. Requires `beautifulsoup4` and `ftfy` to
- be installed. If the dependencies are not installed, the embeddings will be created from the raw
- prompt.
- mask_feature (`bool` defaults to `True`): If set to `True`, the text embeddings will be masked.
-
- Examples:
-
- Returns:
- [`~pipelines.ImagePipelineOutput`] or `tuple`:
- If `return_dict` is `True`, [`~pipelines.ImagePipelineOutput`] is returned, otherwise a `tuple` is
- returned where the first element is a list with the generated images
- """
- # 1. Check inputs. Raise error if not correct
- height = height or self.transformer.config.sample_size * self.vae_scale_factor
- width = width or self.transformer.config.sample_size * self.vae_scale_factor
- self.check_inputs(prompt, height, width, negative_prompt, callback_steps, prompt_embeds, negative_prompt_embeds)
-
- # 2. Default height and width to transformer
- if prompt is not None and isinstance(prompt, str):
- batch_size = 1
- elif prompt is not None and isinstance(prompt, list):
- batch_size = len(prompt)
- else:
- batch_size = prompt_embeds.shape[0]
-
- device = self._execution_device
-
- # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
- # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
- # corresponds to doing no classifier free guidance.
- do_classifier_free_guidance = guidance_scale > 1.0
-
- # 3. Encode input prompt
- prompt_embeds, negative_prompt_embeds = self.encode_prompt(
- prompt,
- do_classifier_free_guidance,
- negative_prompt=negative_prompt,
- num_images_per_prompt=num_images_per_prompt,
- device=device,
- prompt_embeds=prompt_embeds,
- negative_prompt_embeds=negative_prompt_embeds,
- clean_caption=clean_caption,
- mask_feature=mask_feature,
- )
- if do_classifier_free_guidance:
- prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)
-
- # 4. Prepare timesteps
- self.scheduler.set_timesteps(num_inference_steps, device=device)
- timesteps = self.scheduler.timesteps
-
- # 5. Prepare latents.
- latent_channels = self.transformer.config.in_channels
- latents = self.prepare_latents(
- batch_size * num_images_per_prompt,
- latent_channels,
- video_length,
- height,
- width,
- prompt_embeds.dtype,
- device,
- generator,
- latents,
- )
-
- # 6. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline
- extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
-
- # 6.1 Prepare micro-conditions.
- added_cond_kwargs = {"resolution": None, "aspect_ratio": None}
- if self.transformer.config.sample_size == 128:
- resolution = torch.tensor([height, width]).repeat(batch_size * num_images_per_prompt, 1)
- aspect_ratio = torch.tensor([float(height / width)]).repeat(batch_size * num_images_per_prompt, 1)
- resolution = resolution.to(dtype=prompt_embeds.dtype, device=device)
- aspect_ratio = aspect_ratio.to(dtype=prompt_embeds.dtype, device=device)
- added_cond_kwargs = {"resolution": resolution, "aspect_ratio": aspect_ratio}
-
- # 7. Denoising loop
- num_warmup_steps = max(len(timesteps) - num_inference_steps * self.scheduler.order, 0)
-
- with self.progress_bar(total=num_inference_steps) as progress_bar:
- for i, t in enumerate(timesteps):
- latent_model_input = torch.cat([latents] * 2) if do_classifier_free_guidance else latents
- latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)
-
- current_timestep = t
- if not torch.is_tensor(current_timestep):
- # TODO: this requires sync between CPU and GPU. So try to pass timesteps as tensors if you can
- # This would be a good case for the `match` statement (Python 3.10+)
- is_mps = latent_model_input.device.type == "mps"
- if isinstance(current_timestep, float):
- dtype = torch.float32 if is_mps else torch.float64
- else:
- dtype = torch.int32 if is_mps else torch.int64
- current_timestep = torch.tensor([current_timestep], dtype=dtype, device=latent_model_input.device)
- elif len(current_timestep.shape) == 0:
- current_timestep = current_timestep[None].to(latent_model_input.device)
- # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
- current_timestep = current_timestep.expand(latent_model_input.shape[0])
-
- # predict noise model_output
- noise_pred = self.transformer(
- latent_model_input,
- encoder_hidden_states=prompt_embeds,
- timestep=current_timestep,
- added_cond_kwargs=added_cond_kwargs,
- enable_temporal_attentions=enable_temporal_attentions,
- return_dict=False,
- )[0]
-
- # perform guidance
- if do_classifier_free_guidance:
- noise_pred_uncond, noise_pred_text = noise_pred.chunk(2)
- noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)
-
- # learned sigma
- if self.transformer.config.out_channels // 2 == latent_channels:
- noise_pred = noise_pred.chunk(2, dim=1)[0]
- else:
- noise_pred = noise_pred
-
- # compute previous image: x_t -> x_t-1
- latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs, return_dict=False)[0]
-
- # call the callback, if provided
- if i == len(timesteps) - 1 or ((i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0):
- progress_bar.update()
- if callback is not None and i % callback_steps == 0:
- step_idx = i // getattr(self.scheduler, "order", 1)
- callback(step_idx, t, latents)
-
- if not output_type == "latents":
- if enable_vae_temporal_decoder:
- video = self.decode_latents_with_temporal_decoder(latents)
- else:
- video = self.decode_latents(latents)
- else:
- video = latents
- return VideoPipelineOutput(video=video)
-
- # Offload all models
- self.maybe_free_model_hooks()
-
- if not return_dict:
- return (video,)
-
- return VideoPipelineOutput(video=video)
-
- def decode_latents(self, latents):
- video_length = latents.shape[2]
- latents = 1 / self.vae.config.scaling_factor * latents
- latents = einops.rearrange(latents, "b c f h w -> (b f) c h w")
- video = []
- for frame_idx in range(latents.shape[0]):
- video.append(self.vae.decode(latents[frame_idx : frame_idx + 1]).sample)
- video = torch.cat(video)
- video = einops.rearrange(video, "(b f) c h w -> b f h w c", f=video_length)
- video = ((video / 2.0 + 0.5).clamp(0, 1) * 255).to(dtype=torch.uint8).cpu().contiguous()
- # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
- return video
-
- def decode_latents_with_temporal_decoder(self, latents):
- video_length = latents.shape[2]
- latents = 1 / self.vae.config.scaling_factor * latents
- latents = einops.rearrange(latents, "b c f h w -> (b f) c h w")
- video = []
-
- decode_chunk_size = 14
- for frame_idx in range(0, latents.shape[0], decode_chunk_size):
- num_frames_in = latents[frame_idx : frame_idx + decode_chunk_size].shape[0]
-
- decode_kwargs = {}
- decode_kwargs["num_frames"] = num_frames_in
-
- video.append(self.vae.decode(latents[frame_idx : frame_idx + decode_chunk_size], **decode_kwargs).sample)
-
- video = torch.cat(video)
- video = einops.rearrange(video, "(b f) c h w -> b f h w c", f=video_length)
- video = ((video / 2.0 + 0.5).clamp(0, 1) * 255).to(dtype=torch.uint8).cpu().contiguous()
- # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
- return video
diff --git a/scripts/sample/latte/sample.sh b/scripts/sample/latte/sample.sh
deleted file mode 100644
index c85272f..0000000
--- a/scripts/sample/latte/sample.sh
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/usr/bin/env bash
-
-# get args
-GPUS=${1:-8}
-
-# get root dir
-FOLDER_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-ROOT_DIR=$FOLDER_DIR/../../..
-
-# go to root dir
-cd $ROOT_DIR
-
-
-export PYTHONPATH=$FOLDER_DIR:$PYTHONPATH
-python $FOLDER_DIR/sample_t2v.py \
- --checkpoint /home/lishenggui/projects/sora/hf-weights/models--maxin-cn--Latte/snapshots/8f0591220fa329f9d917086810b3c0f6544a87c7/t2v.pt \
- --model_path /home/lishenggui/projects/sora/hf-weights/models--maxin-cn--Latte/snapshots/8f0591220fa329f9d917086810b3c0f6544a87c7/t2v_required_models/ \
- --text_prompt "A dog in astronaut suit and sunglasses floating in space" \
- --output_path $ROOT_DIR/outputs/latte
diff --git a/scripts/sample/latte/sample_t2v.py b/scripts/sample/latte/sample_t2v.py
deleted file mode 100644
index c65d007..0000000
--- a/scripts/sample/latte/sample_t2v.py
+++ /dev/null
@@ -1,242 +0,0 @@
-# All rights reserved.
-# Copyright 2024 Vchitect/Latte
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# modified from https://github.com/Vchitect/Latte/blob/main/sample/sample_t2v.py
-
-import argparse
-import os
-import sys
-
-import torch
-from diffusers.models import AutoencoderKL, AutoencoderKLTemporalDecoder
-from diffusers.schedulers import (
- DDIMScheduler,
- DDPMScheduler,
- DEISMultistepScheduler,
- DPMSolverMultistepScheduler,
- EulerAncestralDiscreteScheduler,
- EulerDiscreteScheduler,
- HeunDiscreteScheduler,
- KDPM2AncestralDiscreteScheduler,
- PNDMScheduler,
-)
-from diffusers.schedulers.scheduling_dpmsolver_singlestep import DPMSolverSinglestepScheduler
-from transformers import T5EncoderModel, T5Tokenizer
-
-sys.path.append(os.path.split(sys.path[0])[0])
-import imageio
-from pipeline_videogen import VideoGenPipeline
-from utils import save_video_grid
-
-from download import find_model
-from open_sora.modeling import LatteT2V
-
-
-def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument("--model_path", type=str, required=True, help="The path to the pretrained model files")
- parser.add_argument("--checkpoint", type=str, required=True, help="The path to the t2v.pt file.")
- parser.add_argument("--output_path", type=str, required=True, help="The path to save the output")
-
- # generation configs
- parser.add_argument(
- "--text_prompt", type=str, nargs="+", required=True, help="The text prompt to generate the video."
- )
- parser.add_argument("--video_length", type=int, default=16, help="The number of frames in the video.")
- parser.add_argument("--image_height", type=int, default=256, help="The size of the generated images.")
- parser.add_argument("--image_width", type=int, default=256, help="The size of the generated images.")
- parser.add_argument("--guidance_scale", type=float, default=7.5, help="The scale of the guidance loss.")
- parser.add_argument("--sample_method", type=str, default="PNDM", help="The sampling method to use.")
- parser.add_argument("--num_sampling_steps", type=int, default=50, help="The number of sampling steps.")
- parser.add_argument(
- "--enable_temporal_attentions", action="store_true", default=True, help="Whether to enable temporal attentions."
- )
- parser.add_argument(
- "--enable_vae_temporal_decoder",
- action="store_true",
- default=True,
- help="Whether to enable the VAE temporal decoder.",
- )
-
- # Scheduler configs
- parser.add_argument("--beta_start", type=float, default=0.0001)
- parser.add_argument("--beta_end", type=float, default=0.02)
- parser.add_argument("--beta_schedule", type=str, default="linear")
- parser.add_argument("--variance_type", type=str, default="learned_range")
-
- args = parser.parse_args()
- return args
-
-
-def main(args):
- torch.set_grad_enabled(False)
- device = "cuda" if torch.cuda.is_available() else "cpu"
-
- transformer_model = LatteT2V.from_pretrained_2d(
- args.model_path, subfolder="transformer", video_length=args.video_length
- ).to(device, dtype=torch.float16)
- state_dict = find_model(args.checkpoint)
- transformer_model.load_state_dict(state_dict["model"])
-
- if args.enable_vae_temporal_decoder:
- vae = AutoencoderKLTemporalDecoder.from_pretrained(
- args.model_path, subfolder="vae_temporal_decoder", torch_dtype=torch.float16
- ).to(device)
- else:
- vae = AutoencoderKL.from_pretrained(args.model_path, subfolder="vae", torch_dtype=torch.float16).to(device)
- tokenizer = T5Tokenizer.from_pretrained(args.model_path, subfolder="tokenizer")
- text_encoder = T5EncoderModel.from_pretrained(
- args.model_path, subfolder="text_encoder", torch_dtype=torch.float16
- ).to(device)
-
- # set eval mode
- transformer_model.eval()
- vae.eval()
- text_encoder.eval()
-
- if args.sample_method == "DDIM":
- scheduler = DDIMScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "EulerDiscrete":
- scheduler = EulerDiscreteScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "DDPM":
- scheduler = DDPMScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "DPMSolverMultistep":
- scheduler = DPMSolverMultistepScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "DPMSolverSinglestep":
- scheduler = DPMSolverSinglestepScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "PNDM":
- scheduler = PNDMScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "HeunDiscrete":
- scheduler = HeunDiscreteScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "EulerAncestralDiscrete":
- scheduler = EulerAncestralDiscreteScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "DEISMultistep":
- scheduler = DEISMultistepScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
- elif args.sample_method == "KDPM2AncestralDiscrete":
- scheduler = KDPM2AncestralDiscreteScheduler.from_pretrained(
- args.model_path,
- subfolder="scheduler",
- beta_start=args.beta_start,
- beta_end=args.beta_end,
- beta_schedule=args.beta_schedule,
- variance_type=args.variance_type,
- )
-
- videogen_pipeline = VideoGenPipeline(
- vae=vae, text_encoder=text_encoder, tokenizer=tokenizer, scheduler=scheduler, transformer=transformer_model
- ).to(device)
- # videogen_pipeline.enable_xformers_memory_efficient_attention()
-
- if not os.path.exists(args.output_path):
- os.makedirs(args.output_path, exist_ok=True)
-
- video_grids = []
- for prompt in args.text_prompt:
- print("Processing the ({}) prompt".format(prompt))
- videos = videogen_pipeline(
- prompt,
- video_length=args.video_length,
- height=args.image_height,
- width=args.image_width,
- num_inference_steps=args.num_sampling_steps,
- guidance_scale=args.guidance_scale,
- enable_temporal_attentions=args.enable_temporal_attentions,
- num_images_per_prompt=1,
- mask_feature=True,
- enable_vae_temporal_decoder=args.enable_vae_temporal_decoder,
- ).video
- try:
- save_path = os.path.join(args.output_path, prompt.replace(" ", "_") + "_webv-imageio.mp4")
- imageio.mimwrite(save_path, videos[0], fps=8, quality=9) # highest quality is 10, lowest is 0
- except:
- print("Error when saving {}".format(prompt))
- video_grids.append(videos)
- video_grids = torch.cat(video_grids, dim=0)
-
- video_grids = save_video_grid(video_grids)
-
- # torchvision.io.write_video(args.output_path + '_%04d' % args.run_time + '-.mp4', video_grids, fps=6)
- save_path = os.path.join(args.output_path, "grid.mp4")
- imageio.mimwrite(save_path, video_grids, fps=8, quality=5)
- print("save path {}".format(abspath(args.output_path)))
-
- # save_videos_grid(video, f"./{prompt}.gif")
-
-
-if __name__ == "__main__":
- args = parse_args()
- main(args)
diff --git a/scripts/sample/latte/utils.py b/scripts/sample/latte/utils.py
deleted file mode 100644
index 3a9e4f1..0000000
--- a/scripts/sample/latte/utils.py
+++ /dev/null
@@ -1,471 +0,0 @@
-# All rights reserved.
-# Copyright 2024 Vchitect/Latte
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# copied from https://github.com/Vchitect/Latte/blob/main/utils.py
-
-import html
-import logging
-import math
-import os
-import re
-import subprocess
-import urllib.parse as ul
-from collections import OrderedDict
-from typing import Iterable, Union
-
-import torch
-import torch.distributed as dist
-from diffusers.utils import is_bs4_available, is_ftfy_available
-
-# from torch._six import inf
-from torch import inf
-from torch.utils.tensorboard import SummaryWriter
-
-if is_bs4_available():
- from bs4 import BeautifulSoup
-
-if is_ftfy_available():
- import ftfy
-
-_tensor_or_tensors = Union[torch.Tensor, Iterable[torch.Tensor]]
-
-
-#################################################################################
-# Training Clip Gradients #
-#################################################################################
-
-
-def get_grad_norm(parameters: _tensor_or_tensors, norm_type: float = 2.0) -> torch.Tensor:
- r"""
- Copy from torch.nn.utils.clip_grad_norm_
-
- Clips gradient norm of an iterable of parameters.
-
- The norm is computed over all gradients together, as if they were
- concatenated into a single vector. Gradients are modified in-place.
-
- Args:
- parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a
- single Tensor that will have gradients normalized
- max_norm (float or int): max norm of the gradients
- norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for
- infinity norm.
- error_if_nonfinite (bool): if True, an error is thrown if the total
- norm of the gradients from :attr:`parameters` is ``nan``,
- ``inf``, or ``-inf``. Default: False (will switch to True in the future)
-
- Returns:
- Total norm of the parameter gradients (viewed as a single vector).
- """
- if isinstance(parameters, torch.Tensor):
- parameters = [parameters]
- grads = [p.grad for p in parameters if p.grad is not None]
- norm_type = float(norm_type)
- if len(grads) == 0:
- return torch.tensor(0.0)
- device = grads[0].device
- if norm_type == inf:
- norms = [g.detach().abs().max().to(device) for g in grads]
- total_norm = norms[0] if len(norms) == 1 else torch.max(torch.stack(norms))
- else:
- total_norm = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)
- return total_norm
-
-
-def clip_grad_norm_(
- parameters: _tensor_or_tensors,
- max_norm: float,
- norm_type: float = 2.0,
- error_if_nonfinite: bool = False,
- clip_grad=True,
-) -> torch.Tensor:
- r"""
- Copy from torch.nn.utils.clip_grad_norm_
-
- Clips gradient norm of an iterable of parameters.
-
- The norm is computed over all gradients together, as if they were
- concatenated into a single vector. Gradients are modified in-place.
-
- Args:
- parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a
- single Tensor that will have gradients normalized
- max_norm (float or int): max norm of the gradients
- norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for
- infinity norm.
- error_if_nonfinite (bool): if True, an error is thrown if the total
- norm of the gradients from :attr:`parameters` is ``nan``,
- ``inf``, or ``-inf``. Default: False (will switch to True in the future)
-
- Returns:
- Total norm of the parameter gradients (viewed as a single vector).
- """
- if isinstance(parameters, torch.Tensor):
- parameters = [parameters]
- grads = [p.grad for p in parameters if p.grad is not None]
- max_norm = float(max_norm)
- norm_type = float(norm_type)
- if len(grads) == 0:
- return torch.tensor(0.0)
- device = grads[0].device
- if norm_type == inf:
- norms = [g.detach().abs().max().to(device) for g in grads]
- total_norm = norms[0] if len(norms) == 1 else torch.max(torch.stack(norms))
- else:
- total_norm = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)
- # print(total_norm)
-
- if clip_grad:
- if error_if_nonfinite and torch.logical_or(total_norm.isnan(), total_norm.isinf()):
- raise RuntimeError(
- f"The total norm of order {norm_type} for gradients from "
- "`parameters` is non-finite, so it cannot be clipped. To disable "
- "this error and scale the gradients by the non-finite norm anyway, "
- "set `error_if_nonfinite=False`"
- )
- clip_coef = max_norm / (total_norm + 1e-6)
- # Note: multiplying by the clamped coef is redundant when the coef is clamped to 1, but doing so
- # avoids a `if clip_coef < 1:` conditional which can require a CPU <=> device synchronization
- # when the gradients do not reside in CPU memory.
- clip_coef_clamped = torch.clamp(clip_coef, max=1.0)
- for g in grads:
- g.detach().mul_(clip_coef_clamped.to(g.device))
- # gradient_cliped = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)
- # print(gradient_cliped)
- return total_norm
-
-
-def get_experiment_dir(root_dir, args):
- # if args.pretrained is not None and 'Latte-XL-2-256x256.pt' not in args.pretrained:
- # root_dir += '-WOPRE'
- if args.use_compile:
- root_dir += "-Compile" # speedup by torch compile
- if args.fixed_spatial:
- root_dir += "-FixedSpa"
- if args.enable_xformers_memory_efficient_attention:
- root_dir += "-Xfor"
- if args.gradient_checkpointing:
- root_dir += "-Gc"
- if args.mixed_precision:
- root_dir += "-Amp"
- if args.image_size == 512:
- root_dir += "-512"
- return root_dir
-
-
-#################################################################################
-# Training Logger #
-#################################################################################
-
-
-def create_logger(logging_dir):
- """
- Create a logger that writes to a log file and stdout.
- """
- if dist.get_rank() == 0: # real logger
- logging.basicConfig(
- level=logging.INFO,
- # format='[\033[34m%(asctime)s\033[0m] %(message)s',
- format="[%(asctime)s] %(message)s",
- datefmt="%Y-%m-%d %H:%M:%S",
- handlers=[logging.StreamHandler(), logging.FileHandler(f"{logging_dir}/log.txt")],
- )
- logger = logging.getLogger(__name__)
-
- else: # dummy logger (does nothing)
- logger = logging.getLogger(__name__)
- logger.addHandler(logging.NullHandler())
- return logger
-
-
-def create_tensorboard(tensorboard_dir):
- """
- Create a tensorboard that saves losses.
- """
- if dist.get_rank() == 0: # real tensorboard
- # tensorboard
- writer = SummaryWriter(tensorboard_dir)
-
- return writer
-
-
-def write_tensorboard(writer, *args):
- """
- write the loss information to a tensorboard file.
- Only for pytorch DDP mode.
- """
- if dist.get_rank() == 0: # real tensorboard
- writer.add_scalar(args[0], args[1], args[2])
-
-
-#################################################################################
-# EMA Update/ DDP Training Utils #
-#################################################################################
-
-
-def update_ema(ema_model, model, decay=0.9999):
- """
- Step the EMA model towards the current model.
- """
- ema_params = OrderedDict(ema_model.named_parameters())
- model_params = OrderedDict(model.named_parameters())
-
- for name, param in model_params.items():
- # TODO: Consider applying only to params that require_grad to avoid small numerical changes of pos_embed
- ema_params[name].mul_(decay).add_(param.data, alpha=1 - decay)
-
-
-def requires_grad(model, flag=True):
- """
- Set requires_grad flag for all parameters in a model.
- """
- for p in model.parameters():
- p.requires_grad = flag
-
-
-def cleanup():
- """
- End DDP training.
- """
- dist.destroy_process_group()
-
-
-def setup_distributed(backend="nccl", port=None):
- """Initialize distributed training environment.
- support both slurm and torch.distributed.launch
- see torch.distributed.init_process_group() for more details
- """
- num_gpus = torch.cuda.device_count()
-
- if "SLURM_JOB_ID" in os.environ:
- rank = int(os.environ["SLURM_PROCID"])
- world_size = int(os.environ["SLURM_NTASKS"])
- node_list = os.environ["SLURM_NODELIST"]
- addr = subprocess.getoutput(f"scontrol show hostname {node_list} | head -n1")
- # specify master port
- if port is not None:
- os.environ["MASTER_PORT"] = str(port)
- elif "MASTER_PORT" not in os.environ:
- # os.environ["MASTER_PORT"] = "29566"
- os.environ["MASTER_PORT"] = str(29567 + num_gpus)
- if "MASTER_ADDR" not in os.environ:
- os.environ["MASTER_ADDR"] = addr
- os.environ["WORLD_SIZE"] = str(world_size)
- os.environ["LOCAL_RANK"] = str(rank % num_gpus)
- os.environ["RANK"] = str(rank)
- else:
- rank = int(os.environ["RANK"])
- world_size = int(os.environ["WORLD_SIZE"])
-
- # torch.cuda.set_device(rank % num_gpus)
-
- dist.init_process_group(
- backend=backend,
- world_size=world_size,
- rank=rank,
- )
-
-
-#################################################################################
-# Testing Utils #
-#################################################################################
-
-
-def save_video_grid(video, nrow=None):
- b, t, h, w, c = video.shape
-
- if nrow is None:
- nrow = math.ceil(math.sqrt(b))
- ncol = math.ceil(b / nrow)
- padding = 1
- video_grid = torch.zeros((t, (padding + h) * nrow + padding, (padding + w) * ncol + padding, c), dtype=torch.uint8)
-
- print(video_grid.shape)
- for i in range(b):
- r = i // ncol
- c = i % ncol
- start_r = (padding + h) * r
- start_c = (padding + w) * c
- video_grid[:, start_r : start_r + h, start_c : start_c + w] = video[i]
-
- return video_grid
-
-
-#################################################################################
-# MMCV Utils #
-#################################################################################
-
-
-def collect_env():
- # Copyright (c) OpenMMLab. All rights reserved.
- from mmcv.utils import collect_env as collect_base_env
- from mmcv.utils import get_git_hash
-
- """Collect the information of the running environments."""
-
- env_info = collect_base_env()
- env_info["MMClassification"] = get_git_hash()[:7]
-
- for name, val in env_info.items():
- print(f"{name}: {val}")
-
- print(torch.cuda.get_arch_list())
- print(torch.version.cuda)
-
-
-#################################################################################
-# Pixart-alpha Utils #
-#################################################################################
-
-bad_punct_regex = re.compile(
- r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"
-)
-
-
-def text_preprocessing(text, clean_caption=False):
- if clean_caption and not is_bs4_available():
- clean_caption = False
-
- if clean_caption and not is_ftfy_available():
- clean_caption = False
-
- if not isinstance(text, (tuple, list)):
- text = [text]
-
- def process(text: str):
- if clean_caption:
- text = clean_caption(text)
- text = clean_caption(text)
- else:
- text = text.lower().strip()
- return text
-
- return [process(t) for t in text]
-
-
-# Copied from diffusers.pipelines.deepfloyd_if.pipeline_if.IFPipeline._clean_caption
-def clean_caption(caption):
- caption = str(caption)
- caption = ul.unquote_plus(caption)
- caption = caption.strip().lower()
- caption = re.sub("<person>", "person", caption)
- # urls:
- caption = re.sub(
- r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
- "",
- caption,
- ) # regex for urls
- caption = re.sub(
- r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
- "",
- caption,
- ) # regex for urls
- # html:
- caption = BeautifulSoup(caption, features="html.parser").text
-
- # @<nickname>
- caption = re.sub(r"@[\w\d]+\b", "", caption)
-
- # 31C0—31EF CJK Strokes
- # 31F0—31FF Katakana Phonetic Extensions
- # 3200—32FF Enclosed CJK Letters and Months
- # 3300—33FF CJK Compatibility
- # 3400—4DBF CJK Unified Ideographs Extension A
- # 4DC0—4DFF Yijing Hexagram Symbols
- # 4E00—9FFF CJK Unified Ideographs
- caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)
- caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)
- caption = re.sub(r"[\u3200-\u32ff]+", "", caption)
- caption = re.sub(r"[\u3300-\u33ff]+", "", caption)
- caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)
- caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)
- caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)
- #######################################################
-
- # все виды тире / all types of dash --> "-"
- caption = re.sub(
- r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa
- "-",
- caption,
- )
-
- # кавычки к одному стандарту
- caption = re.sub(r"[`´«»“”¨]", '"', caption)
- caption = re.sub(r"[‘’]", "'", caption)
-
- # &quot;
- caption = re.sub(r"&quot;?", "", caption)
- # &amp
- caption = re.sub(r"&amp", "", caption)
-
- # ip adresses:
- caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)
-
- # article ids:
- caption = re.sub(r"\d:\d\d\s+$", "", caption)
-
- # \n
- caption = re.sub(r"\\n", " ", caption)
-
- # "#123"
- caption = re.sub(r"#\d{1,3}\b", "", caption)
- # "#12345.."
- caption = re.sub(r"#\d{5,}\b", "", caption)
- # "123456.."
- caption = re.sub(r"\b\d{6,}\b", "", caption)
- # filenames:
- caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)
-
- #
- caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""
- caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""
-
- caption = re.sub(bad_punct_regex, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT
- caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "
-
- # this-is-my-cute-cat / this_is_my_cute_cat
- regex2 = re.compile(r"(?:\-|\_)")
- if len(re.findall(regex2, caption)) > 3:
- caption = re.sub(regex2, " ", caption)
-
- caption = ftfy.fix_text(caption)
- caption = html.unescape(html.unescape(caption))
-
- caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640
- caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc
- caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231
-
- caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)
- caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)
- caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)
- caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)
- caption = re.sub(r"\bpage\s+\d+\b", "", caption)
-
- caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...
-
- caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)
-
- caption = re.sub(r"\b\s+\:\s+", r": ", caption)
- caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)
- caption = re.sub(r"\s+", " ", caption)
-
- caption.strip()
-
- caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)
- caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)
- caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)
- caption = re.sub(r"^\.\S+$", "", caption)
-
- return caption.strip()
diff --git a/scripts/train.sh b/scripts/train.sh
deleted file mode 100644
index b38de61..0000000
--- a/scripts/train.sh
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-
-# get args
-GPUS=${1:-8}
-
-# get root dir
-FOLDER_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
-ROOT_DIR=$FOLDER_DIR/..
-
-# go to root dir
-cd $ROOT_DIR
-
-# define dataset shards
-COLLATED_VIDEO_DIR=./dataset/MSRVTT-collated/train/videos
-PROCESSED_DATASET=(
- ./dataset/MSRVTT-processed/train/part-00000
- ./dataset/MSRVTT-processed/train/part-00001
- ./dataset/MSRVTT-processed/train/part-00002
- ./dataset/MSRVTT-processed/train/part-00003
- ./dataset/MSRVTT-processed/train/part-00004
- ./dataset/MSRVTT-processed/train/part-00005
- ./dataset/MSRVTT-processed/train/part-00006
- ./dataset/MSRVTT-processed/train/part-00007
- ./dataset/MSRVTT-processed/train/part-00008
- ./dataset/MSRVTT-processed/train/part-00009
-)
-
-# create timestamp to differentiate between runs
-timestamp=$(date +%Y-%m-%d-%H-%M)
-
-# run single node training
-torchrun --standalone \
- --nproc_per_node $GPUS \
- train.py \
- --epochs 1 \
- --batch_size 1 \
- --lr 1e-4 \
- --accumulation_steps 32 \
- --grad_checkpoint \
- --dataset "${PROCESSED_DATASET[@]}" \
- --video_dir $COLLATED_VIDEO_DIR \
- --save_interval 224 \
- --checkpoint_dir ./checkpoints/$timestamp \
- --tensorboard_dir ./runs/$timestamp
diff --git a/setup.py b/setup.py
deleted file mode 100644
index 582beef..0000000
--- a/setup.py
+++ /dev/null
@@ -1,82 +0,0 @@
-import os
-from typing import List
-
-from setuptools import find_packages, setup
-
-
-def fetch_requirements(path) -> List[str]:
- """
- This function reads the requirements file.
-
- Args:
- path (str): the path to the requirements file.
-
- Returns:
- The lines in the requirements file.
- """
- with open(path, "r") as fd:
- return [r.strip() for r in fd.readlines()]
-
-
-def fetch_readme() -> str:
- """
- This function reads the README.md file in the current directory.
-
- Returns:
- The lines in the README file.
- """
- with open("README.md", encoding="utf-8") as f:
- return f.read()
-
-
-def get_version() -> str:
- """
- This function reads the version.txt and generates the colossalai/version.py file.
-
- Returns:
- The library version stored in version.txt.
- """
-
- setup_file_path = os.path.abspath(__file__)
- project_path = os.path.dirname(setup_file_path)
- version_txt_path = os.path.join(project_path, "version.txt")
-
- with open(version_txt_path) as f:
- version = f.read().strip()
- return version
-
-
-setup(
- name="open-sora",
- version=get_version(),
- packages=find_packages(
- exclude=(
- "docker",
- "tests",
- "docs",
- "examples",
- "tests",
- "scripts",
- "requirements",
- "extensions",
- "*.egg-info",
- ),
- ),
- description="Unofficial implementation of OpenAI's Sora by the Colossal-AI Team",
- long_description=fetch_readme(),
- long_description_content_type="text/markdown",
- license="Apache Software License 2.0",
- url="https://www.colossalai.org",
- project_urls={
- "Github": "https://github.com/hpcaitech/Open-Sora",
- },
- install_requires=fetch_requirements("requirements.txt"),
- python_requires=">=3.6",
- classifiers=[
- "Programming Language :: Python :: 3",
- "License :: OSI Approved :: Apache Software License",
- "Environment :: GPU :: NVIDIA CUDA",
- "Topic :: Scientific/Engineering :: Artificial Intelligence",
- "Topic :: System :: Distributed Computing",
- ],
-)
diff --git a/tests/test_model.py b/tests/test_model.py
deleted file mode 100644
index 1c2462f..0000000
--- a/tests/test_model.py
+++ /dev/null
@@ -1,117 +0,0 @@
-import colossalai
-import pytest
-import torch
-import torch.distributed as dist
-import torch.nn.functional as F
-from colossalai.booster import Booster
-from colossalai.logging import disable_existing_loggers
-from colossalai.nn.optimizer import HybridAdam
-from colossalai.testing import parameterize, rerun_if_address_is_in_use, spawn
-from colossalai.utils import get_current_device
-from torch.testing import assert_close
-
-from open_sora.modeling import DiT_models
-from open_sora.utils.plugin import ZeroSeqParallelPlugin
-
-
-@parameterize("sp_size", [2, 4])
-@parameterize("sp_mode", ["ulysses", "fastseq"])
-@parameterize("sp_overlap", [True, False])
-@parameterize("cross_attn", [True, False])
-def check_dit_model_fwd_bwd(
- sp_size: int,
- sp_mode: str,
- sp_overlap: bool,
- cross_attn: bool,
- video_latent_states,
- text_latent_states,
- t,
- mask,
-):
- if sp_overlap and sp_mode == "ulysses":
- return
- if not cross_attn:
- mask = mask.new_ones(
- *mask.shape[:2],
- mask.shape[2] + mask.shape[3],
- mask.shape[2] + mask.shape[3],
- )
- plugin = ZeroSeqParallelPlugin(
- sp_size=sp_size, stage=2, precision="fp32", master_weights=False
- )
- booster = Booster(plugin=plugin)
- model = DiT_models["DiT-B/8"](text_dropout_prob=0.0, use_cross_attn=cross_attn).to(
- get_current_device()
- )
- parallel_model = DiT_models["DiT-B/8"](
- text_dropout_prob=0.0,
- use_cross_attn=cross_attn,
- seq_parallel_group=plugin.sp_group,
- seq_parallel_mode=sp_mode,
- seq_parallel_overlap=sp_overlap,
- ).to(get_current_device())
- parallel_model.load_state_dict(model.state_dict())
- opt = HybridAdam(parallel_model.parameters(), lr=1e-3)
- parallel_model, opt, *_ = booster.boost(parallel_model, opt)
-
- target = model(video_latent_states, t, text_latent_states, mask)
- noise = torch.randn_like(target)
- target_loss = F.mse_loss(target, noise)
- target_loss.backward()
-
- dp_video_latent_states = video_latent_states.chunk(plugin.dp_size)[plugin.dp_rank]
- dp_text_latent_states = text_latent_states.chunk(plugin.dp_size)[plugin.dp_rank]
- dp_t = t.chunk(plugin.dp_size)[plugin.dp_rank]
- dp_mask = mask.chunk(plugin.dp_size)[plugin.dp_rank]
- dp_noise = noise.chunk(plugin.dp_size)[plugin.dp_rank]
-
- output = parallel_model(
- dp_video_latent_states, dp_t, dp_text_latent_states, dp_mask
- )
- loss = F.mse_loss(output, dp_noise)
- booster.backward(loss, opt)
-
- if plugin.dp_size == 1:
- assert_close(target, output)
-
- for p1, p2 in zip(model.parameters(), opt._master_param_groups_of_current_rank[0]):
- working_p = opt._param_store.master_to_working_param[id(p2)]
- grads = opt._grad_store.get_partitioned_gradients_by_param_id(0, id(working_p))
- grad_index = 0 if opt._partition_grads else opt._local_rank
- grad = grads[grad_index]
- sharded_grad = p1.grad.view(-1).chunk(dist.get_world_size())[dist.get_rank()]
- assert_close(sharded_grad, grad[: sharded_grad.shape[0]])
-
-
-def run_dist(rank, world_size, port):
- disable_existing_loggers()
- colossalai.launch(
- config={},
- rank=rank,
- world_size=world_size,
- port=port,
- host="localhost",
- backend="nccl",
- )
- b, s, c, p = 4, 20, 3, 8
- dim_text, s_text = 512, 12
- video_latent_states = torch.rand(b, s, c, p, p, device=get_current_device())
- text_latent_states = torch.rand(b, s_text, dim_text, device=get_current_device())
- t = torch.randint(0, 1000, (b,), device=get_current_device())
- mask = torch.ones(b, 1, s, s_text, device=get_current_device(), dtype=torch.int)
- check_dit_model_fwd_bwd(
- video_latent_states=video_latent_states,
- text_latent_states=text_latent_states,
- t=t,
- mask=mask,
- )
-
-
-@rerun_if_address_is_in_use()
-def test_dit_model():
- spawn(run_dist, 4)
-
-
-if __name__ == "__main__":
- test_dit_model()
diff --git a/tests/test_sp_attn.py b/tests/test_sp_attn.py
deleted file mode 100644
index d62a4ab..0000000
--- a/tests/test_sp_attn.py
+++ /dev/null
@@ -1,102 +0,0 @@
-import colossalai
-import pytest
-import torch
-import torch.distributed as dist
-from colossalai.logging import disable_existing_loggers
-from colossalai.testing import parameterize, rerun_if_address_is_in_use, spawn
-from colossalai.utils import get_current_device
-from torch.testing import assert_close
-
-from open_sora.modeling.dit import (
- CrossAttention,
- FastSeqParallelCrossAttention,
- SeqParallelCrossAttention,
-)
-from open_sora.utils.comm import gather_seq, split_seq
-
-
-@parameterize("layer_cls", [SeqParallelCrossAttention, FastSeqParallelCrossAttention])
-@parameterize("overlap", [True, False])
-@parameterize("cross_attn", [True, False])
-def check_sp_attn(layer_cls, overlap, cross_attn):
- if overlap and layer_cls == SeqParallelCrossAttention:
- return
- model_kwargs = {}
- if layer_cls == FastSeqParallelCrossAttention:
- model_kwargs["overlap"] = overlap
- sp_size = dist.get_world_size()
- sp_rank = dist.get_rank()
- q_dim, context_dim = 8, 4
- num_heads = 4
- head_dim = 16
- bs = 2
- sq = 8
- skv = 4
- if not cross_attn:
- context_dim = None
- skv = sq
- attn = CrossAttention(q_dim, context_dim, num_heads, head_dim).to(
- get_current_device()
- )
- parallel_attn = layer_cls(
- q_dim,
- context_dim,
- num_heads,
- head_dim,
- seq_parallel_group=dist.group.WORLD,
- **model_kwargs,
- ).to(get_current_device())
- parallel_attn.load_state_dict(attn.state_dict())
- hidden_states = torch.rand(bs, sq, q_dim, device=get_current_device())
- if cross_attn:
- context = torch.rand(bs, skv, context_dim, device=get_current_device())
- else:
- context = None
- mask = torch.zeros(bs, 1, sq, skv, device=get_current_device())
- target = attn(hidden_states, context, mask)
- hidden_states_parallel = split_seq(hidden_states, sp_size, sp_rank)
- if cross_attn:
- context_parallel = split_seq(context, sp_size, sp_rank)
- else:
- context_parallel = None
- output_parallel = parallel_attn(
- hidden_states_parallel,
- context_parallel,
- mask,
- )
- assert torch.equal(target.chunk(sp_size, dim=1)[sp_rank], output_parallel)
- output = gather_seq(output_parallel, sp_size, sp_rank, dist.group.WORLD)
- assert torch.equal(target, output)
- target.mean().backward()
- output.mean().backward()
-
- # all-reduce mean of grads
- for p in parallel_attn.parameters():
- p.grad.data.div_(sp_size)
- dist.all_reduce(p.grad.data)
-
- for (n, p1), p2 in zip(attn.named_parameters(), parallel_attn.parameters()):
- assert_close(p1.grad, p2.grad, msg=lambda m: f"Check {n}\n{m}")
-
-
-def run_dist(rank, world_size, port):
- disable_existing_loggers()
- colossalai.launch(
- config={},
- rank=rank,
- world_size=world_size,
- port=port,
- host="localhost",
- backend="nccl",
- )
- check_sp_attn()
-
-
-@rerun_if_address_is_in_use()
-def test_seq_parallel_attn():
- spawn(run_dist, 2)
-
-
-if __name__ == "__main__":
- test_seq_parallel_attn()
diff --git a/train.py b/train.py
deleted file mode 100644
index 2d48f63..0000000
--- a/train.py
+++ /dev/null
@@ -1,278 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""
-A minimal training script for DiT using PyTorch DDP.
-"""
-import argparse
-import os
-from functools import partial
-
-import torch
-import torch.distributed as dist
-from colossalai import launch_from_torch
-from colossalai.accelerator import get_accelerator
-from colossalai.booster import Booster
-from colossalai.cluster import DistCoordinator
-from colossalai.logging import get_dist_logger
-from colossalai.nn.lr_scheduler import CosineAnnealingLR
-from colossalai.nn.optimizer import HybridAdam
-from colossalai.utils import get_current_device
-from torch.utils.tensorboard import SummaryWriter
-from tqdm import tqdm
-
-from open_sora.diffusion import create_diffusion
-from open_sora.modeling import DiT_models
-from open_sora.modeling.dit import SUPPORTED_MODEL_ARCH, SUPPORTED_SEQ_PARALLEL_MODES
-from open_sora.utils.data import (
- create_video_compressor,
- load_datasets,
- make_batch,
- preprocess_batch,
-)
-from open_sora.utils.plugin import ZeroSeqParallelPlugin
-
-#################################################################################
-# Training Helper Functions #
-#################################################################################
-
-
-def configure_backends():
- # the first flag below was False when we tested this script but True makes A100 training a lot faster:
- torch.backends.cuda.matmul.allow_tf32 = True
- torch.backends.cudnn.allow_tf32 = True
-
-
-def update_ema(ema_model, model, decay=0.9999):
- """
- Step the EMA model towards the current model.
- """
-
- for ema_p, p in zip(ema_model.parameters(), model.parameters()):
- ema_p.mul_(decay).add_(p.data, alpha=1 - decay)
-
-
-def requires_grad(model, flag=True):
- """
- Set requires_grad flag for all parameters in a model.
- """
- for p in model.parameters():
- p.requires_grad = flag
-
-
-def all_reduce_mean(tensor: torch.Tensor) -> torch.Tensor:
- dist.all_reduce(tensor=tensor, op=dist.ReduceOp.SUM)
- tensor.div_(dist.get_world_size())
- return tensor
-
-
-def save_checkpoints(booster, model, optimizer, ema, save_path, coordinator):
- os.makedirs(save_path, exist_ok=True)
- booster.save_model(model, os.path.join(save_path, "model"), shard=False)
- booster.save_optimizer(optimizer, os.path.join(save_path, "optimizer"), shard=True)
- if coordinator.is_master():
- ema_state_dict = ema.state_dict()
- for k, v in ema_state_dict.items():
- ema_state_dict[k] = v.cpu()
- torch.save(ema_state_dict, os.path.join(save_path, "ema.pt"))
- dist.barrier()
-
-
-#################################################################################
-# Training Loop #
-#################################################################################
-
-
-def main(args):
- """
- Trains a new DiT model.
- """
- # Step 1: init distributed environment
- launch_from_torch({})
- coordinator = DistCoordinator()
- logger = get_dist_logger()
- configure_backends()
-
- # Step 2: set up acceleration plugins
- plugin = ZeroSeqParallelPlugin(sp_size=args.sp_size, stage=2, precision="fp16")
- booster = Booster(plugin=plugin)
-
- if coordinator.is_master():
- os.makedirs(args.checkpoint_dir, exist_ok=True)
- os.makedirs(args.tensorboard_dir, exist_ok=True)
- writer = SummaryWriter(args.tensorboard_dir)
-
- # Step 3: Create video compressor
- video_compressor = create_video_compressor(args.compressor)
- model_kwargs = {
- "in_channels": video_compressor.out_channels,
- "seq_parallel_group": plugin.sp_group,
- "seq_parallel_mode": args.sp_mode,
- "seq_parallel_overlap": args.sp_overlap,
- "model_arch": args.model_arch,
- }
-
- # Step 4: Create DiT and EMA
- model = DiT_models[args.model](**model_kwargs).to(get_current_device())
- patch_size = model.patch_size
- ema = DiT_models[args.model](**model_kwargs).to(get_current_device())
- update_ema(ema, model, decay=0)
- requires_grad(ema, False)
-
- model.train() # important! This enables embedding dropout for classifier-free guidance
- ema.eval() # EMA model should always be in eval mode
-
- # configure gradient checkpointing
- if args.grad_checkpoint:
- model.enable_gradient_checkpointing()
-
- # Step 5: create diffusion pipeline
- diffusion = create_diffusion(
- timestep_respacing=""
- ) # default: 1000 steps, linear noise schedule
-
- # Step 6: setup optimizer (we used default Adam betas=(0.9, 0.999) and a constant learning rate of 1e-4 in our paper):
- opt = HybridAdam(model.parameters(), lr=args.lr, weight_decay=0)
-
- # Step 7: Setup dataloader
- dataset = load_datasets(args.dataset)
- dataloader = plugin.prepare_dataloader(
- dataset,
- batch_size=args.batch_size,
- collate_fn=partial(
- make_batch,
- video_dir=args.video_dir,
- pad_to_multiple=args.sp_size,
- use_pooled_text=args.model_arch == "adaln",
- ),
- shuffle=True,
- drop_last=True,
- )
- lr_scheduler = CosineAnnealingLR(
- opt, args.epochs * len(dataloader) // args.accumulation_steps
- )
- logger.info(f"Dataset contains {len(dataset)} samples", ranks=[0])
-
- # Step 8: setup booster
- model, opt, _, dataloader, lr_scheduler = booster.boost(
- model, opt, dataloader=dataloader, lr_scheduler=lr_scheduler
- )
- if args.load_model is not None:
- booster.load_model(model, args.load_model)
- if args.load_optimizer is not None:
- booster.load_optimizer(opt, args.load_optimizer)
- logger.info(
- f"Booster init max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",
- ranks=[0],
- )
-
- # Step 9: Train
- num_steps_per_epoch = len(dataloader) // args.accumulation_steps
-
- for epoch in range(args.epochs):
- dataloader.sampler.set_epoch(epoch)
- with tqdm(
- desc=f"Epoch {epoch}",
- disable=not coordinator.is_master(),
- total=num_steps_per_epoch,
- ) as pbar:
- total_loss = torch.tensor(0.0, device=get_current_device())
- for step, batch in enumerate(dataloader):
- batch = preprocess_batch(
- batch,
- patch_size,
- video_compressor,
- pad_to_multiple=args.sp_size,
- model_arch=args.model_arch,
- )
- video_inputs = batch.pop("video_latent_states")
- mask = batch.pop("video_padding_mask")
- t = torch.randint(
- 0,
- diffusion.num_timesteps,
- (video_inputs.shape[0],),
- device=video_inputs.device,
- )
- loss_dict = diffusion.training_losses(
- model, video_inputs, t, batch, mask=mask
- )
- loss = loss_dict["loss"].mean() / args.accumulation_steps
- total_loss.add_(loss.data)
- booster.backward(loss, opt)
-
- if (step + 1) % args.accumulation_steps == 0:
- opt.step()
- opt.zero_grad()
- lr_scheduler.step()
- update_ema(ema, model)
-
- all_reduce_mean(total_loss)
- pbar.set_postfix({"Loss": f"{total_loss.item():.4f}"})
- if coordinator.is_master():
- global_step = (epoch * num_steps_per_epoch) + (
- step + 1
- ) // args.accumulation_steps
- writer.add_scalar(
- tag="Loss",
- scalar_value=total_loss.item(),
- global_step=global_step,
- )
- pbar.update()
- total_loss.zero_()
-
- # Save DiT checkpoint:
- if args.save_interval > 0 and (
- (step + 1) % (args.save_interval * args.accumulation_steps) == 0
- or (step + 1) == len(dataloader)
- ):
- save_path = os.path.join(
- args.checkpoint_dir, f"epoch-{epoch}-step-{step}"
- )
- save_checkpoints(booster, model, opt, ema, save_path, coordinator)
- logger.info(f"Saved checkpoint to {save_path}", ranks=[0])
-
- get_accelerator().empty_cache()
- final_save_path = os.path.join(args.checkpoint_dir, "final")
- save_checkpoints(booster, model, opt, ema, final_save_path, coordinator)
- logger.info(f"Saved checkpoint to {final_save_path}", ranks=[0])
- logger.info(
- f"Training complete, max device memory: {get_accelerator().max_memory_allocated() / 1024 ** 2:.2f} MB",
- ranks=[0],
- )
-
-
-if __name__ == "__main__":
- # Default args here will train DiT-XL/2 with the hyperparameters we used in our paper (except training iters).
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "-m", "--model", type=str, choices=list(DiT_models.keys()), default="DiT-S/8"
- )
- parser.add_argument(
- "-x", "--model_arch", choices=SUPPORTED_MODEL_ARCH, default="cross-attn"
- )
- parser.add_argument("-d", "--dataset", nargs="+", default=[])
- parser.add_argument("-v", "--video_dir", type=str, required=True)
- parser.add_argument("-e", "--epochs", type=int, default=10)
- parser.add_argument("-b", "--batch_size", type=int, default=4)
- parser.add_argument("-g", "--grad_checkpoint", action="store_true", default=False)
- parser.add_argument("-a", "--accumulation_steps", default=1, type=int)
- parser.add_argument("--sp_size", type=int, default=1)
- parser.add_argument(
- "--sp_mode", type=str, default="ulysses", choices=SUPPORTED_SEQ_PARALLEL_MODES
- )
- parser.add_argument("--sp_overlap", action="store_true", default=False)
- parser.add_argument("--lr", type=float, default=1e-4)
- parser.add_argument("--save_interval", type=int, default=20)
- parser.add_argument("--checkpoint_dir", type=str, default="checkpoints")
- parser.add_argument("--tensorboard_dir", type=str, default="runs")
- parser.add_argument(
- "-c", "--compressor", choices=["raw", "vqvae", "vae"], default="raw"
- )
- parser.add_argument("--load_model", default=None)
- parser.add_argument("--load_optimizer", default=None)
- args = parser.parse_args()
- main(args)
diff --git a/version.txt b/version.txt
deleted file mode 100644
index 8acdd82..0000000
--- a/version.txt
+++ /dev/null
@@ -1 +0,0 @@
-0.0.1