Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

format (#69)

Zheng Zangwei (Alex Zheng)d851a855352b
diff --git a/.gitignore b/.gitignore
index b5fc089..e99a5d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -160,8 +160,14 @@ cython_debug/
.idea/
.vscode/
+# macos
+*.DS_Store
+
# misc files
dataset/
runs/
checkpoints/
outputs/
+
+# Secret files
+hostfile
diff --git a/README.md b/README.md
index 3b632d0..6b642da 100644
--- a/README.md
+++ b/README.md
@@ -1,149 +1,149 @@
-<p align="center">
- <img src="./assets/readme/icon_zw.png" width="250"/>
-<p>
-
-</p>
-<div align="center">
- <a href="https://github.com/hpcaitech/Open-Sora/stargazers"><img src="https://img.shields.io/github/stars/hpcaitech/Open-Sora?style=social"></a>
- <a href="https://github.com/hpcaitech/public_assets/tree/main/colossalai/contact/slack"><img src="https://img.shields.io/badge/Slack-join-blueviolet?logo=slack&amp"></a>
- <a href="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/WeChat.png"><img src="https://img.shields.io/badge/微信-加入-green?logo=wechat&amp"></a>
-</div>
-
-## Open-Sora: Towards Open Reproduction of Sora
-
-**Open-Sora** is an **open-source** initiative dedicated to **efficiently** reproducing OpenAI's Sora. Our project aims to cover **the full pipeline**, including video data preprocessing, training with acceleration, efficient inference and more. Operating on a limited budget, we prioritize the vibrant open-source community, providing access to text-to-image, image captioning, and language models. We hope to make a contribution to the community and make the project more accessible to everyone.
-
-## 📰 News
-
-* **[2024.03.18]** 🔥 We release **Open-Sora 1.0**, an open-source project to reproduce OpenAI Sora.
-Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with
-<a href="https://github.com/hpcaitech/ColossalAI"><img src="assets/readme/colossal_ai.png" width="8%" ></a> acceleration,
-inference, and more. Our provided checkpoint can produce 2s 512x512 videos.
-
-## 🎥 Latest Demo
-
-| **2s 512x512** | **2s 512x512** |
-| ----------------------------------------------- | ----------------------------------------------- |
-| <img src="assets/readme/sample_0.gif" width=""> | <img src="assets/readme/sample_0.gif" width=""> |
-
-## 🔆 New Features/Updates
-
-- 📍 Open-Sora-v1 is trained on xxx. We train the model in three stages. Model weights are available here. Training details can be found here.
-- ✅ Support training acceleration including flash-attention, accelerated T5, mixed precision, gradient checkpointing, splitted VAE, sequence parallelism, etc. XXX times. See more discussions [here]().
-- ✅ We provide video cutting and captioning tools for data preprocessing. Our data collection plan can be found [here]().
-- ✅ We find VQ-VAE from [] has a low quality and thus adopt a better VAE from []. We also find patching in the time dimension deteriorates the quality. See more discussions [here]().
-- ✅ We investigate different architectures including DiT, Latte, and our proposed STDiT. Our STDiT achieves a better trade-off between quality and speed. See more discussions [here]().
-- ✅ Support clip and t5 text conditioning.
-- ✅ By viewing images as one-frame videos, our project supports training DiT on both images and videos (e.g., ImageNet & UCF101).
-- ✅ Support inference with official weights from [DiT](https://github.com/facebookresearch/DiT), [Latte](https://github.com/Vchitect/Latte), and [PixArt](https://pixart-alpha.github.io/).
-
-
-
-### TODO list sorted by priority
-
-- [ ] Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, deduplication, etc.). See [datasets.md]() for more information. **[WIP]**
-- [ ] Training Video-VAE. **[WIP]**
-- [ ] Support image and video conditioning.
-- [ ] Evaluation pipeline.
-- [ ] Incoporate a better scheduler, e.g., rectified flow in SD3.
-- [ ] Support variable aspect ratios, resolutions, durations.
-- [ ] Support SD3 when released.
-
-
-## Contents
-
-- [Open-Sora: Towards Open Reproduction of Sora](#open-sora-towards-open-reproduction-of-sora)
-- [📰 News](#-news)
-- [🎥 Latest Demo](#-latest-demo)
-- [🔆 New Features/Updates](#-new-featuresupdates)
- - [TODO list sorted by priority](#todo-list-sorted-by-priority)
-- [Contents](#contents)
-- [Installation](#installation)
-- [Model Weights](#model-weights)
-- [Inference](#inference)
-- [Data Processing](#data-processing)
- - [Split video into clips](#split-video-into-clips)
- - [Generate video caption](#generate-video-caption)
-- [Training](#training)
-- [Acknowledgement](#acknowledgement)
-- [Citation](#citation)
-- [Star History](#star-history)
-- [TODO](#todo)
-
-## Installation
-
-```bash
-git clone https://github.com/hpcaitech/Open-Sora
-cd Open-Sora
-pip install xxx
-```
-
-After installation, to get fimilar with the project, you can check the [here]() for the project structure and how to use the config files.
-
-## Model Weights
-
-| Model | #Params | url |
-| ---------- | ------- | --- |
-| 16x256x256 | | |
-
-## Inference
-
-```bash
-python scripts/inference.py configs/opensora/inference/16x256x256.py
-```
-
-## Data Processing
-
-### Split video into clips
-
-We provide code to split a long video into separate clips efficiently using `multiprocessing`. See `tools/data/scene_detect.py`.
-
-### Generate video caption
-
-## Training
-
-## Acknowledgement
-
-* [DiT](https://github.com/facebookresearch/DiT): Scalable Diffusion Models with Transformers.
-* [OpenDiT](https://github.com/NUS-HPC-AI-Lab/OpenDiT): An acceleration for DiT training. OpenDiT's team provides valuable suggestions on acceleration of our training process.
-* [PixArt](https://github.com/PixArt-alpha/PixArt-alpha): An open-source DiT-based text-to-image model.
-* [Latte](https://github.com/Vchitect/Latte): An attempt to efficiently train DiT for video.
-* [StabilityAI VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse-original): A powerful image VAE model.
-* [CLIP](https://github.com/openai/CLIP): A powerful text-image embedding model.
-* [T5](https://github.com/google-research/text-to-text-transfer-transformer): The powerful text encoder.
-* [LLaVA](https://github.com/haotian-liu/LLaVA): A powerful image captioning model based on [LLaMA](https://github.com/meta-llama/llama) and [Yi-34B](https://huggingface.co/01-ai/Yi-34B).
-* [PySceneDetect](https://github.com/Breakthrough/PySceneDetect): A powerful tool to split video into clips.
-
-We are grateful for their exceptional work and generous contribution to open source.
-
-## Citation
-
-```bibtex
-@software{opensora,
- author = {Zangwei Zheng and Xiangyu Peng and Shenggui Li and Yang You},
- title = {Open-Sora: Towards Open Reproduction of Sora},
- month = {March},
- year = {2024},
- url = {https://github.com/hpcaitech/Open-Sora}
-}
-```
-
-Zangwei Zheng and Xiangyu Peng equally contributed to this work.
-
-## Star History
-
-[![Star History Chart](https://api.star-history.com/svg?repos=hpcaitech/Open-Sora&type=Date)](https://star-history.com/#hpcaitech/Open-Sora&Date)
-
-## TODO
-
-Modules for releasing:
-
-* `configs`
-* `opensora`
-* `assets`
-* `scripts`
-* `tools`
-
-packages for data processing
-
-put all outputs under ./checkpoints/, including pretrained_models, checkpoints, samples
+<p align="center">
+ <img src="./assets/readme/icon_zw.png" width="250"/>
+<p>
+
+</p>
+<div align="center">
+ <a href="https://github.com/hpcaitech/Open-Sora/stargazers"><img src="https://img.shields.io/github/stars/hpcaitech/Open-Sora?style=social"></a>
+ <a href="https://github.com/hpcaitech/public_assets/tree/main/colossalai/contact/slack"><img src="https://img.shields.io/badge/Slack-join-blueviolet?logo=slack&amp"></a>
+ <a href="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/WeChat.png"><img src="https://img.shields.io/badge/微信-加入-green?logo=wechat&amp"></a>
+</div>
+
+## Open-Sora: Towards Open Reproduction of Sora
+
+**Open-Sora** is an **open-source** initiative dedicated to **efficiently** reproducing OpenAI's Sora. Our project aims to cover **the full pipeline**, including video data preprocessing, training with acceleration, efficient inference and more. Operating on a limited budget, we prioritize the vibrant open-source community, providing access to text-to-image, image captioning, and language models. We hope to make a contribution to the community and make the project more accessible to everyone.
+
+## 📰 News
+
+* **[2024.03.18]** 🔥 We release **Open-Sora 1.0**, an open-source project to reproduce OpenAI Sora.
+Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with
+<a href="https://github.com/hpcaitech/ColossalAI"><img src="assets/readme/colossal_ai.png" width="8%" ></a> acceleration,
+inference, and more. Our provided checkpoint can produce 2s 512x512 videos.
+
+## 🎥 Latest Demo
+
+| **2s 512x512** | **2s 512x512** |
+| ----------------------------------------------- | ----------------------------------------------- |
+| <img src="assets/readme/sample_0.gif" width=""> | <img src="assets/readme/sample_0.gif" width=""> |
+
+## 🔆 New Features/Updates
+
+- 📍 Open-Sora-v1 is trained on xxx. We train the model in three stages. Model weights are available here. Training details can be found here.
+- ✅ Support training acceleration including flash-attention, accelerated T5, mixed precision, gradient checkpointing, splitted VAE, sequence parallelism, etc. XXX times. See more discussions [here]().
+- ✅ We provide video cutting and captioning tools for data preprocessing. Our data collection plan can be found [here]().
+- ✅ We find VQ-VAE from [] has a low quality and thus adopt a better VAE from []. We also find patching in the time dimension deteriorates the quality. See more discussions [here]().
+- ✅ We investigate different architectures including DiT, Latte, and our proposed STDiT. Our STDiT achieves a better trade-off between quality and speed. See more discussions [here]().
+- ✅ Support clip and t5 text conditioning.
+- ✅ By viewing images as one-frame videos, our project supports training DiT on both images and videos (e.g., ImageNet & UCF101).
+- ✅ Support inference with official weights from [DiT](https://github.com/facebookresearch/DiT), [Latte](https://github.com/Vchitect/Latte), and [PixArt](https://pixart-alpha.github.io/).
+
+
+
+### TODO list sorted by priority
+
+- [ ] Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, deduplication, etc.). See [datasets.md]() for more information. **[WIP]**
+- [ ] Training Video-VAE. **[WIP]**
+- [ ] Support image and video conditioning.
+- [ ] Evaluation pipeline.
+- [ ] Incoporate a better scheduler, e.g., rectified flow in SD3.
+- [ ] Support variable aspect ratios, resolutions, durations.
+- [ ] Support SD3 when released.
+
+
+## Contents
+
+- [Open-Sora: Towards Open Reproduction of Sora](#open-sora-towards-open-reproduction-of-sora)
+- [📰 News](#-news)
+- [🎥 Latest Demo](#-latest-demo)
+- [🔆 New Features/Updates](#-new-featuresupdates)
+ - [TODO list sorted by priority](#todo-list-sorted-by-priority)
+- [Contents](#contents)
+- [Installation](#installation)
+- [Model Weights](#model-weights)
+- [Inference](#inference)
+- [Data Processing](#data-processing)
+ - [Split video into clips](#split-video-into-clips)
+ - [Generate video caption](#generate-video-caption)
+- [Training](#training)
+- [Acknowledgement](#acknowledgement)
+- [Citation](#citation)
+- [Star History](#star-history)
+- [TODO](#todo)
+
+## Installation
+
+```bash
+git clone https://github.com/hpcaitech/Open-Sora
+cd Open-Sora
+pip install xxx
+```
+
+After installation, to get fimilar with the project, you can check the [here]() for the project structure and how to use the config files.
+
+## Model Weights
+
+| Model | #Params | url |
+| ---------- | ------- | --- |
+| 16x256x256 | | |
+
+## Inference
+
+```bash
+python scripts/inference.py configs/opensora/inference/16x256x256.py
+```
+
+## Data Processing
+
+### Split video into clips
+
+We provide code to split a long video into separate clips efficiently using `multiprocessing`. See `tools/data/scene_detect.py`.
+
+### Generate video caption
+
+## Training
+
+## Acknowledgement
+
+* [DiT](https://github.com/facebookresearch/DiT): Scalable Diffusion Models with Transformers.
+* [OpenDiT](https://github.com/NUS-HPC-AI-Lab/OpenDiT): An acceleration for DiT training. OpenDiT's team provides valuable suggestions on acceleration of our training process.
+* [PixArt](https://github.com/PixArt-alpha/PixArt-alpha): An open-source DiT-based text-to-image model.
+* [Latte](https://github.com/Vchitect/Latte): An attempt to efficiently train DiT for video.
+* [StabilityAI VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse-original): A powerful image VAE model.
+* [CLIP](https://github.com/openai/CLIP): A powerful text-image embedding model.
+* [T5](https://github.com/google-research/text-to-text-transfer-transformer): The powerful text encoder.
+* [LLaVA](https://github.com/haotian-liu/LLaVA): A powerful image captioning model based on [LLaMA](https://github.com/meta-llama/llama) and [Yi-34B](https://huggingface.co/01-ai/Yi-34B).
+* [PySceneDetect](https://github.com/Breakthrough/PySceneDetect): A powerful tool to split video into clips.
+
+We are grateful for their exceptional work and generous contribution to open source.
+
+## Citation
+
+```bibtex
+@software{opensora,
+ author = {Zangwei Zheng and Xiangyu Peng and Shenggui Li and Yang You},
+ title = {Open-Sora: Towards Open Reproduction of Sora},
+ month = {March},
+ year = {2024},
+ url = {https://github.com/hpcaitech/Open-Sora}
+}
+```
+
+Zangwei Zheng and Xiangyu Peng equally contributed to this work.
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=hpcaitech/Open-Sora&type=Date)](https://star-history.com/#hpcaitech/Open-Sora&Date)
+
+## TODO
+
+Modules for releasing:
+
+* `configs`
+* `opensora`
+* `assets`
+* `scripts`
+* `tools`
+
+packages for data processing
+
+put all outputs under ./checkpoints/, including pretrained_models, checkpoints, samples
diff --git a/assets/texts/t2i_samples.txt b/assets/texts/t2i_samples.txt
index 0fd1284..9b72952 100644
--- a/assets/texts/t2i_samples.txt
+++ b/assets/texts/t2i_samples.txt
@@ -1,8 +1,8 @@
-A small cactus with a happy face in the Sahara desert.
-Bright scene, aerial view,ancient city, fantasy, gorgeous light, mirror reflection, high detail, wide angle lens.
-Nature vs human nature, surreal, UHD, 8k, hyper details, rich colors, photograph.
-Poster of a mechanical cat, techical Schematics viewed from front.
-Luffy from ONEPIECE, handsome face, fantasy.
-Real beautiful woman.
-A alpaca made of colorful building blocks, cyberpunk.
-artistic
+A small cactus with a happy face in the Sahara desert.
+Bright scene, aerial view,ancient city, fantasy, gorgeous light, mirror reflection, high detail, wide angle lens.
+Nature vs human nature, surreal, UHD, 8k, hyper details, rich colors, photograph.
+Poster of a mechanical cat, techical Schematics viewed from front.
+Luffy from ONEPIECE, handsome face, fantasy.
+Real beautiful woman.
+A alpaca made of colorful building blocks, cyberpunk.
+artistic
diff --git a/configs/dit/train/1x256x256.py b/configs/dit/train/1x256x256.py
index ee9dde0..8cb8f9a 100644
--- a/configs/dit/train/1x256x256.py
+++ b/configs/dit/train/1x256x256.py
@@ -47,5 +47,5 @@ ckpt_every = 1000
load = None
batch_size = 128
-lr = 1e-4 # according to DiT repo
+lr = 1e-4 # according to DiT repo
grad_clip = 1.0
diff --git a/configs/opensora/inference/16x256x256.py b/configs/opensora/inference/16x256x256.py
index 1970635..8950f1a 100644
--- a/configs/opensora/inference/16x256x256.py
+++ b/configs/opensora/inference/16x256x256.py
@@ -23,7 +23,7 @@ text_encoder = dict(
scheduler = dict(
# type="iddpm",
# num_sampling_steps=250,
- type = "dpm-solver",
+ type="dpm-solver",
num_sampling_steps=20,
cfg_scale=7.0,
)
diff --git a/configs/pixart/inference/1x1024MS.py b/configs/pixart/inference/1x1024MS.py
index 5795fbd..6392c9e 100644
--- a/configs/pixart/inference/1x1024MS.py
+++ b/configs/pixart/inference/1x1024MS.py
@@ -22,7 +22,7 @@ text_encoder = dict(
model_max_length=120,
)
scheduler = dict(
- type = "dpm-solver",
+ type="dpm-solver",
num_sampling_steps=20,
cfg_scale=7.0,
)
diff --git a/configs/pixart/inference/1x256x256.py b/configs/pixart/inference/1x256x256.py
index fa89c56..5838c94 100644
--- a/configs/pixart/inference/1x256x256.py
+++ b/configs/pixart/inference/1x256x256.py
@@ -15,7 +15,7 @@ vae = dict(
# type="VideoAutoencoderKL",
# from_pretrained="stabilityai/sd-vae-ft-ema",
type="VideoAutoencoderKLTemporalDecoder",
- from_pretrained="pretrained_models/vae_temporal_decoder"
+ from_pretrained="pretrained_models/vae_temporal_decoder",
)
text_encoder = dict(
type="t5",
diff --git a/configs/pixart/inference/1x512x512.py b/configs/pixart/inference/1x512x512.py
index 6e4daa0..f22d545 100644
--- a/configs/pixart/inference/1x512x512.py
+++ b/configs/pixart/inference/1x512x512.py
@@ -21,7 +21,7 @@ text_encoder = dict(
model_max_length=120,
)
scheduler = dict(
- type = "dpm-solver",
+ type="dpm-solver",
num_sampling_steps=20,
cfg_scale=7.0,
)
diff --git a/opensora/__init__.py b/opensora/__init__.py
index 17a73fc..a3175b2 100644
--- a/opensora/__init__.py
+++ b/opensora/__init__.py
@@ -1,4 +1,4 @@
-from .acceleration import *
-from .datasets import *
-from .registry import *
-from .models import *
+from .acceleration import *
+from .datasets import *
+from .models import *
+from .registry import *
diff --git a/opensora/acceleration/communications.py b/opensora/acceleration/communications.py
index b07d4c0..d0900d2 100644
--- a/opensora/acceleration/communications.py
+++ b/opensora/acceleration/communications.py
@@ -12,13 +12,12 @@ def _all_to_all(
scatter_dim: int,
gather_dim: int,
):
- input_list = [
- t.contiguous() for t in torch.tensor_split(input_, world_size, scatter_dim)
- ]
+ input_list = [t.contiguous() for t in torch.tensor_split(input_, world_size, scatter_dim)]
output_list = [torch.empty_like(input_list[0]) for _ in range(world_size)]
dist.all_to_all(output_list, input_list, group=group)
return torch.cat(output_list, dim=gather_dim).contiguous()
+
class _AllToAll(torch.autograd.Function):
"""All-to-all communication.
@@ -35,20 +34,18 @@ class _AllToAll(torch.autograd.Function):
ctx.scatter_dim = scatter_dim
ctx.gather_dim = gather_dim
ctx.world_size = dist.get_world_size(process_group)
- output = _all_to_all(
- input_, ctx.world_size, process_group, scatter_dim, gather_dim
- )
+ output = _all_to_all(input_, ctx.world_size, process_group, scatter_dim, gather_dim)
return output
@staticmethod
def backward(ctx, grad_output):
grad_output = _all_to_all(
- grad_output,
- ctx.world_size,
- ctx.process_group,
- ctx.gather_dim,
- ctx.scatter_dim,
- )
+ grad_output,
+ ctx.world_size,
+ ctx.process_group,
+ ctx.gather_dim,
+ ctx.scatter_dim,
+ )
return (
grad_output,
None,
@@ -65,6 +62,7 @@ def all_to_all(
):
return _AllToAll.apply(input_, process_group, scatter_dim, gather_dim)
+
def _gather(
input_: torch.Tensor,
world_size: int,
@@ -76,10 +74,12 @@ def _gather(
dist.gather(input_, gather_list, group=group, gather_dim=gather_dim)
return gather_list
+
# ====================
# Gather-Split
# ====================
+
def _split(input_, pg: dist.ProcessGroup, dim=-1):
# skip if only one rank involved
world_size = dist.get_world_size(pg)
@@ -104,7 +104,7 @@ def _gather(input_, pg: dist.ProcessGroup, dim=-1):
# skip if only one rank involved
input_ = input_.contiguous()
world_size = dist.get_world_size(pg)
- rank = dist.get_rank(pg)
+ dist.get_rank(pg)
if world_size == 1:
return input_
@@ -119,6 +119,7 @@ def _gather(input_, pg: dist.ProcessGroup, dim=-1):
return output
+
class _GatherForwardSplitBackward(torch.autograd.Function):
"""Gather the input from model parallel region and concatenate.
@@ -130,7 +131,6 @@ class _GatherForwardSplitBackward(torch.autograd.Function):
@staticmethod
def symbolic(graph, input_):
-
return _gather(input_)
@staticmethod
@@ -183,5 +183,6 @@ class _SplitForwardGatherBackward(torch.autograd.Function):
def split_forward_gather_backward(input_, process_group, dim, grad_scale=1.0):
return _SplitForwardGatherBackward.apply(input_, process_group, dim, grad_scale)
+
def gather_forward_split_backward(input_, process_group, dim, grad_scale=None):
return _GatherForwardSplitBackward.apply(input_, process_group, dim, grad_scale)
diff --git a/opensora/acceleration/parallel_states.py b/opensora/acceleration/parallel_states.py
index a8d71e8..ff2893e 100644
--- a/opensora/acceleration/parallel_states.py
+++ b/opensora/acceleration/parallel_states.py
@@ -2,14 +2,18 @@ import torch.distributed as dist
_GLOBAL_PARALLEL_GROUPS = dict()
+
def set_data_parallel_group(group: dist.ProcessGroup):
_GLOBAL_PARALLEL_GROUPS["data"] = group
+
def get_data_parallel_group():
return _GLOBAL_PARALLEL_GROUPS.get("data", None)
+
def set_sequence_parallel_group(group: dist.ProcessGroup):
_GLOBAL_PARALLEL_GROUPS["sequence"] = group
+
def get_sequence_parallel_group():
- return _GLOBAL_PARALLEL_GROUPS.get("sequence", None)
\ No newline at end of file
+ return _GLOBAL_PARALLEL_GROUPS.get("sequence", None)
diff --git a/opensora/acceleration/plugin.py b/opensora/acceleration/plugin.py
index 7c9be5d..c657a95 100644
--- a/opensora/acceleration/plugin.py
+++ b/opensora/acceleration/plugin.py
@@ -75,13 +75,11 @@ class ZeroSeqParallelPlugin(LowLevelZeroPlugin):
pin_memory=False,
num_workers=0,
distributed_sampler_cls=None,
- **kwargs
+ **kwargs,
):
_kwargs = kwargs.copy()
distributed_sampler_cls = distributed_sampler_cls or DistributedSampler
- sampler = distributed_sampler_cls(
- dataset, num_replicas=self.dp_size, rank=self.dp_rank, shuffle=shuffle
- )
+ sampler = distributed_sampler_cls(dataset, num_replicas=self.dp_size, rank=self.dp_rank, shuffle=shuffle)
# Deterministic dataloader
def seed_worker(worker_id):
@@ -99,4 +97,4 @@ class ZeroSeqParallelPlugin(LowLevelZeroPlugin):
pin_memory=pin_memory,
num_workers=num_workers,
**_kwargs,
- )
\ No newline at end of file
+ )
diff --git a/opensora/acceleration/shardformer/modeling/t5.py b/opensora/acceleration/shardformer/modeling/t5.py
index d494c1c..9cfb808 100644
--- a/opensora/acceleration/shardformer/modeling/t5.py
+++ b/opensora/acceleration/shardformer/modeling/t5.py
@@ -28,9 +28,10 @@ class T5LayerNorm(nn.Module):
@staticmethod
def from_native_module(module, *args, **kwargs):
- assert module.__class__.__name__ == "FusedRMSNorm", \
- "Recovering T5LayerNorm requires the original layer to be apex's Fused RMS Norm." \
+ assert module.__class__.__name__ == "FusedRMSNorm", (
+ "Recovering T5LayerNorm requires the original layer to be apex's Fused RMS Norm."
"Apex's fused norm is automatically used by Hugging Face Transformers https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#L265C5-L265C48"
+ )
layer_norm = T5LayerNorm(module.normalized_shape, eps=module.eps)
layer_norm.weight.data.copy_(module.weight.data)
diff --git a/opensora/acceleration/shardformer/policy/t5_encoder.py b/opensora/acceleration/shardformer/policy/t5_encoder.py
index e7ce3ac..85c994e 100644
--- a/opensora/acceleration/shardformer/policy/t5_encoder.py
+++ b/opensora/acceleration/shardformer/policy/t5_encoder.py
@@ -1,12 +1,9 @@
from colossalai.shardformer.modeling.jit import get_jit_fused_dropout_add_func
-from colossalai.shardformer.modeling.t5 import (
- get_jit_fused_T5_layer_ff_forward,
- get_T5_layer_self_attention_forward,
-)
+from colossalai.shardformer.modeling.t5 import get_jit_fused_T5_layer_ff_forward, get_T5_layer_self_attention_forward
from colossalai.shardformer.policies.base_policy import Policy, SubModuleReplacementDescription
+
class T5EncoderPolicy(Policy):
-
def config_sanity_check(self):
assert not self.shard_config.enable_tensor_parallelism
assert not self.shard_config.enable_flash_attention
@@ -15,17 +12,12 @@ class T5EncoderPolicy(Policy):
return self.model
def module_policy(self):
- from transformers.models.t5.modeling_t5 import (
- T5LayerFF,
- T5LayerSelfAttention,
- T5Stack,
- )
+ from transformers.models.t5.modeling_t5 import T5LayerFF, T5LayerSelfAttention, T5Stack
policy = {}
# check whether apex is installed
try:
- from apex.normalization import FusedRMSNorm
from opensora.acceleration.shardformer.modeling.t5 import T5LayerNorm
# recover hf from fused rms norm to T5 norm which is faster
@@ -38,18 +30,12 @@ class T5EncoderPolicy(Policy):
target_key=T5LayerFF,
)
self.append_or_create_submodule_replacement(
- description=SubModuleReplacementDescription(
- suffix="layer_norm",
- target_module=T5LayerNorm
- ),
+ description=SubModuleReplacementDescription(suffix="layer_norm", target_module=T5LayerNorm),
policy=policy,
target_key=T5LayerSelfAttention,
)
self.append_or_create_submodule_replacement(
- description=SubModuleReplacementDescription(
- suffix="final_layer_norm",
- target_module=T5LayerNorm
- ),
+ description=SubModuleReplacementDescription(suffix="final_layer_norm", target_module=T5LayerNorm),
policy=policy,
target_key=T5Stack,
)
diff --git a/opensora/datasets/utils.py b/opensora/datasets/utils.py
index 00f8fe3..cd268ae 100644
--- a/opensora/datasets/utils.py
+++ b/opensora/datasets/utils.py
@@ -117,6 +117,7 @@ def prepare_dataloader(
**_kwargs,
)
+
def center_crop_arr(pil_image, image_size):
"""
Center cropping implementation from ADM.
diff --git a/opensora/models/__init__.py b/opensora/models/__init__.py
index 2aa793a..6025349 100644
--- a/opensora/models/__init__.py
+++ b/opensora/models/__init__.py
@@ -1,6 +1,6 @@
-from .dit import *
-from .latte import *
-from .stdit import *
-from .pixart import *
-from .text_encoder import *
-from .vae import *
+from .dit import *
+from .latte import *
+from .pixart import *
+from .stdit import *
+from .text_encoder import *
+from .vae import *
diff --git a/opensora/models/dit/__init__.py b/opensora/models/dit/__init__.py
index 2019831..94548a3 100644
--- a/opensora/models/dit/__init__.py
+++ b/opensora/models/dit/__init__.py
@@ -1 +1 @@
-from .dit import DiT, DiT_XL_2, DiT_XL_2x2
+from .dit import DiT, DiT_XL_2, DiT_XL_2x2
diff --git a/opensora/models/dit/dit.py b/opensora/models/dit/dit.py
index 106f660..a23dd7b 100644
--- a/opensora/models/dit/dit.py
+++ b/opensora/models/dit/dit.py
@@ -16,6 +16,7 @@ import torch.utils.checkpoint
from einops import rearrange
from timm.models.vision_transformer import Mlp
+from opensora.acceleration.checkpoint import auto_grad_checkpoint
from opensora.models.layers.blocks import (
Attention,
CaptionEmbedder,
@@ -31,7 +32,6 @@ from opensora.models.layers.blocks import (
)
from opensora.registry import MODELS
from opensora.utils.ckpt_utils import load_checkpoint
-from opensora.acceleration.checkpoint import auto_grad_checkpoint
class DiTBlock(nn.Module):
@@ -209,7 +209,7 @@ class DiT(nn.Module):
# blocks
for _, block in enumerate(self.blocks):
c = condition
- x = auto_grad_checkpoint(block, x, c) # (B, N, D)
+ x = auto_grad_checkpoint(block, x, c) # (B, N, D)
# final process
x = self.final_layer(x, condition) # (B, N, num_patches * out_channels)
diff --git a/opensora/models/latte/__init__.py b/opensora/models/latte/__init__.py
index 3115f47..f9d918a 100644
--- a/opensora/models/latte/__init__.py
+++ b/opensora/models/latte/__init__.py
@@ -1 +1 @@
-from .latte import Latte, Latte_XL_2, Latte_XL_2x2
+from .latte import Latte, Latte_XL_2, Latte_XL_2x2
diff --git a/opensora/models/latte/latte.py b/opensora/models/latte/latte.py
index 11259de..005cbac 100644
--- a/opensora/models/latte/latte.py
+++ b/opensora/models/latte/latte.py
@@ -11,10 +11,10 @@
import torch
from einops import rearrange, repeat
+from opensora.acceleration.checkpoint import auto_grad_checkpoint
from opensora.models.dit import DiT
from opensora.registry import MODELS
from opensora.utils.ckpt_utils import load_checkpoint
-from opensora.acceleration.checkpoint import auto_grad_checkpoint
@MODELS.register_module()
@@ -56,7 +56,7 @@ class Latte(DiT):
if i == 1:
x = x + self.pos_embed_temporal
- x = auto_grad_checkpoint(block, x, c) # (B, N, D)
+ x = auto_grad_checkpoint(block, x, c) # (B, N, D)
if i % 2 == 0:
x = rearrange(x, "(b t) s d -> b (t s) d", t=self.num_temporal, s=self.num_spatial)
diff --git a/opensora/models/layers/blocks.py b/opensora/models/layers/blocks.py
index f06d696..8a91a41 100644
--- a/opensora/models/layers/blocks.py
+++ b/opensora/models/layers/blocks.py
@@ -19,10 +19,9 @@ import torch.nn.functional as F
import torch.utils.checkpoint
import xformers.ops
from einops import rearrange
-from flash_attn import flash_attn_varlen_func
from timm.models.vision_transformer import Mlp
-from opensora.acceleration.communications import all_to_all, split_forward_gather_backward
+from opensora.acceleration.communications import all_to_all
from opensora.acceleration.parallel_states import get_sequence_parallel_group
approx_gelu = lambda: nn.GELU(approximate="tanh")
@@ -179,7 +178,6 @@ class Attention(nn.Module):
class SeqParallelAttention(Attention):
-
def __init__(
self,
dim: int,
@@ -293,7 +291,6 @@ class MultiHeadCrossAttention(nn.Module):
class SeqParallelMultiHeadCrossAttention(MultiHeadCrossAttention):
-
def __init__(
self,
d_model,
diff --git a/opensora/models/pixart/__init__.py b/opensora/models/pixart/__init__.py
index 622ad39..cf83202 100644
--- a/opensora/models/pixart/__init__.py
+++ b/opensora/models/pixart/__init__.py
@@ -1 +1 @@
-from .pixart import PixArt, PixArt_XL_2
+from .pixart import PixArt, PixArt_XL_2
diff --git a/opensora/models/pixart/pixart.py b/opensora/models/pixart/pixart.py
index daad7dc..32785ff 100644
--- a/opensora/models/pixart/pixart.py
+++ b/opensora/models/pixart/pixart.py
@@ -16,9 +16,7 @@ from timm.models.layers import DropPath
from timm.models.vision_transformer import Mlp
# from .builder import MODELS
-from torch.utils.checkpoint import checkpoint
from opensora.acceleration.checkpoint import auto_grad_checkpoint
-
from opensora.models.layers.blocks import (
Attention,
CaptionEmbedder,
diff --git a/opensora/models/stdit/__init__.py b/opensora/models/stdit/__init__.py
index 3b089af..5ca2cc9 100644
--- a/opensora/models/stdit/__init__.py
+++ b/opensora/models/stdit/__init__.py
@@ -1 +1 @@
-from .stdit import STDiT
+from .stdit import STDiT
diff --git a/opensora/models/stdit/stdit.py b/opensora/models/stdit/stdit.py
index c8a2e95..e9dcf15 100644
--- a/opensora/models/stdit/stdit.py
+++ b/opensora/models/stdit/stdit.py
@@ -1,4 +1,3 @@
-
import numpy as np
import torch
import torch.distributed as dist
@@ -42,7 +41,6 @@ class STDiTBlock(nn.Module):
enable_layernorm_kernel=False,
enable_sequence_parallelism=False,
):
-
super().__init__()
self.hidden_size = hidden_size
self.enable_flashattn = enable_flashattn
diff --git a/opensora/models/text_encoder/__init__.py b/opensora/models/text_encoder/__init__.py
index fa494a2..9fc6a99 100644
--- a/opensora/models/text_encoder/__init__.py
+++ b/opensora/models/text_encoder/__init__.py
@@ -1,3 +1,3 @@
-from .t5 import T5Encoder
-from .classes import ClassEncoder
-from .clip import ClipEncoder
+from .classes import ClassEncoder
+from .clip import ClipEncoder
+from .t5 import T5Encoder
diff --git a/opensora/models/vae/__init__.py b/opensora/models/vae/__init__.py
index 04a09c9..63510b0 100644
--- a/opensora/models/vae/__init__.py
+++ b/opensora/models/vae/__init__.py
@@ -1 +1 @@
-from .vae import VideoAutoencoderKL, VideoAutoencoderKLTemporalDecoder
+from .vae import VideoAutoencoderKL, VideoAutoencoderKLTemporalDecoder
diff --git a/opensora/registry.py b/opensora/registry.py
index af1bbc6..7797d36 100644
--- a/opensora/registry.py
+++ b/opensora/registry.py
@@ -1,39 +1,39 @@
-from copy import deepcopy
-
-import torch.nn as nn
-from mmengine.registry import Registry
-
-
-def build_module(module, builder, **kwargs):
- """Build module from config or return the module itself.
-
- Args:
- module (Union[dict, nn.Module]): The module to build.
- builder (Registry): The registry to build module.
- *args, **kwargs: Arguments passed to build function.
-
- Returns:
- Any: The built module.
- """
- if isinstance(module, dict):
- cfg = deepcopy(module)
- for k, v in kwargs.items():
- cfg[k] = v
- return builder.build(cfg)
- elif isinstance(module, nn.Module):
- return module
- elif module is None:
- return None
- else:
- raise TypeError(f"Only support dict and nn.Module, but got {type(module)}.")
-
-
-MODELS = Registry(
- "model",
- locations=["opensora.models"],
-)
-
-SCHEDULERS = Registry(
- "scheduler",
- locations=["opensora.schedulers"],
-)
+from copy import deepcopy
+
+import torch.nn as nn
+from mmengine.registry import Registry
+
+
+def build_module(module, builder, **kwargs):
+ """Build module from config or return the module itself.
+
+ Args:
+ module (Union[dict, nn.Module]): The module to build.
+ builder (Registry): The registry to build module.
+ *args, **kwargs: Arguments passed to build function.
+
+ Returns:
+ Any: The built module.
+ """
+ if isinstance(module, dict):
+ cfg = deepcopy(module)
+ for k, v in kwargs.items():
+ cfg[k] = v
+ return builder.build(cfg)
+ elif isinstance(module, nn.Module):
+ return module
+ elif module is None:
+ return None
+ else:
+ raise TypeError(f"Only support dict and nn.Module, but got {type(module)}.")
+
+
+MODELS = Registry(
+ "model",
+ locations=["opensora.models"],
+)
+
+SCHEDULERS = Registry(
+ "scheduler",
+ locations=["opensora.schedulers"],
+)
diff --git a/opensora/schedulers/__init__.py b/opensora/schedulers/__init__.py
index f64aa25..97ea76f 100644
--- a/opensora/schedulers/__init__.py
+++ b/opensora/schedulers/__init__.py
@@ -1,2 +1,2 @@
-from .iddpm import IDDPM
-from .dpms import DPMS
+from .dpms import DPMS
+from .iddpm import IDDPM
diff --git a/opensora/schedulers/dpms/dpm_solver.py b/opensora/schedulers/dpms/dpm_solver.py
index d887398..37a9399 100644
--- a/opensora/schedulers/dpms/dpm_solver.py
+++ b/opensora/schedulers/dpms/dpm_solver.py
@@ -700,7 +700,7 @@ class DPM_Solver:
x_t: A pytorch tensor. The approximated solution at time `t`.
"""
ns = self.noise_schedule
- dims = x.dim()
+ x.dim()
lambda_s, lambda_t = ns.marginal_lambda(s), ns.marginal_lambda(t)
h = lambda_t - lambda_s
log_alpha_s, log_alpha_t = ns.marginal_log_mean_coeff(s), ns.marginal_log_mean_coeff(t)
diff --git a/opensora/utils/misc.py b/opensora/utils/misc.py
index 056b6ae..9d89c10 100644
--- a/opensora/utils/misc.py
+++ b/opensora/utils/misc.py
@@ -1,339 +1,339 @@
-import collections
-import importlib
-import logging
-import os
-import time
-from collections import OrderedDict
-from collections.abc import Sequence
-from itertools import repeat
-
-import numpy as np
-import torch
-import torch.distributed as dist
-
-
-def print_rank(var_name, var_value, rank=0):
- if dist.get_rank() == rank:
- print(f"[Rank {rank}] {var_name}: {var_value}")
-
-
-def print_0(*args, **kwargs):
- if dist.get_rank() == 0:
- print(*args, **kwargs)
-
-
-def requires_grad(model: torch.nn.Module, flag: bool = True) -> None:
- """
- Set requires_grad flag for all parameters in a model.
- """
- for p in model.parameters():
- p.requires_grad = flag
-
-
-def format_numel_str(numel: int) -> str:
- B = 1024**3
- M = 1024**2
- K = 1024
- if numel >= B:
- return f"{numel / B:.2f} B"
- elif numel >= M:
- return f"{numel / M:.2f} M"
- elif numel >= K:
- return f"{numel / K:.2f} K"
- else:
- return f"{numel}"
-
-
-def all_reduce_mean(tensor: torch.Tensor) -> torch.Tensor:
- dist.all_reduce(tensor=tensor, op=dist.ReduceOp.SUM)
- tensor.div_(dist.get_world_size())
- return tensor
-
-
-def get_model_numel(model: torch.nn.Module) -> (int, int):
- num_params = 0
- num_params_trainable = 0
- for p in model.parameters():
- num_params += p.numel()
- if p.requires_grad:
- num_params_trainable += p.numel()
- return num_params, num_params_trainable
-
-
-def try_import(name):
- """Try to import a module.
-
- Args:
- name (str): Specifies what module to import in absolute or relative
- terms (e.g. either pkg.mod or ..mod).
- Returns:
- ModuleType or None: If importing successfully, returns the imported
- module, otherwise returns None.
- """
- try:
- return importlib.import_module(name)
- except ImportError:
- return None
-
-
-def transpose(x):
- """
- transpose a list of list
- Args:
- x (list[list]):
- """
- ret = list(map(list, zip(*x)))
- return ret
-
-
-def get_timestamp():
- timestamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(time.time()))
- return timestamp
-
-
-def format_time(seconds):
- days = int(seconds / 3600 / 24)
- seconds = seconds - days * 3600 * 24
- hours = int(seconds / 3600)
- seconds = seconds - hours * 3600
- minutes = int(seconds / 60)
- seconds = seconds - minutes * 60
- secondsf = int(seconds)
- seconds = seconds - secondsf
- millis = int(seconds * 1000)
-
- f = ""
- i = 1
- if days > 0:
- f += str(days) + "D"
- i += 1
- if hours > 0 and i <= 2:
- f += str(hours) + "h"
- i += 1
- if minutes > 0 and i <= 2:
- f += str(minutes) + "m"
- i += 1
- if secondsf > 0 and i <= 2:
- f += str(secondsf) + "s"
- i += 1
- if millis > 0 and i <= 2:
- f += str(millis) + "ms"
- i += 1
- if f == "":
- f = "0ms"
- return f
-
-
-def to_tensor(data):
- """Convert objects of various python types to :obj:`torch.Tensor`.
-
- Supported types are: :class:`numpy.ndarray`, :class:`torch.Tensor`,
- :class:`Sequence`, :class:`int` and :class:`float`.
-
- Args:
- data (torch.Tensor | numpy.ndarray | Sequence | int | float): Data to
- be converted.
- """
-
- if isinstance(data, torch.Tensor):
- return data
- elif isinstance(data, np.ndarray):
- return torch.from_numpy(data)
- elif isinstance(data, Sequence) and not isinstance(data, str):
- return torch.tensor(data)
- elif isinstance(data, int):
- return torch.LongTensor([data])
- elif isinstance(data, float):
- return torch.FloatTensor([data])
- else:
- raise TypeError(f"type {type(data)} cannot be converted to tensor.")
-
-
-def to_ndarray(data):
- if isinstance(data, torch.Tensor):
- return data.numpy()
- elif isinstance(data, np.ndarray):
- return data
- elif isinstance(data, Sequence):
- return np.array(data)
- elif isinstance(data, int):
- return np.ndarray([data], dtype=int)
- elif isinstance(data, float):
- return np.array([data], dtype=float)
- else:
- raise TypeError(f"type {type(data)} cannot be converted to ndarray.")
-
-
-def to_torch_dtype(dtype):
- if isinstance(dtype, torch.dtype):
- return dtype
- elif isinstance(dtype, str):
- dtype_mapping = {
- "float64": torch.float64,
- "float32": torch.float32,
- "float16": torch.float16,
- "fp32": torch.float32,
- "fp16": torch.float16,
- "half": torch.float16,
- "bf16": torch.bfloat16,
- }
- if dtype not in dtype_mapping:
- raise ValueError
- dtype = dtype_mapping[dtype]
- return dtype
- else:
- raise ValueError
-
-
-def count_params(model):
- return sum(p.numel() for p in model.parameters() if p.requires_grad)
-
-
-def _ntuple(n):
- def parse(x):
- if isinstance(x, collections.abc.Iterable) and not isinstance(x, str):
- return x
- return tuple(repeat(x, n))
-
- return parse
-
-
-to_1tuple = _ntuple(1)
-to_2tuple = _ntuple(2)
-to_3tuple = _ntuple(3)
-to_4tuple = _ntuple(4)
-to_ntuple = _ntuple
-
-
-def convert_SyncBN_to_BN2d(model_cfg):
- for k in model_cfg:
- v = model_cfg[k]
- if k == "norm_cfg" and v["type"] == "SyncBN":
- v["type"] = "BN2d"
- elif isinstance(v, dict):
- convert_SyncBN_to_BN2d(v)
-
-
-def get_topk(x, dim=4, k=5):
- x = to_tensor(x)
- inds = x[..., dim].topk(k)[1]
- return x[inds]
-
-
-def param_sigmoid(x, alpha):
- ret = 1 / (1 + (-alpha * x).exp())
- return ret
-
-
-def inverse_param_sigmoid(x, alpha, eps=1e-5):
- x = x.clamp(min=0, max=1)
- x1 = x.clamp(min=eps)
- x2 = (1 - x).clamp(min=eps)
- return torch.log(x1 / x2) / alpha
-
-
-def inverse_sigmoid(x, eps=1e-5):
- """Inverse function of sigmoid.
-
- Args:
- x (Tensor): The tensor to do the
- inverse.
- eps (float): EPS avoid numerical
- overflow. Defaults 1e-5.
- Returns:
- Tensor: The x has passed the inverse
- function of sigmoid, has same
- shape with input.
- """
- x = x.clamp(min=0, max=1)
- x1 = x.clamp(min=eps)
- x2 = (1 - x).clamp(min=eps)
- return torch.log(x1 / x2)
-
-
-def count_columns(df, columns):
- cnt_dict = OrderedDict()
- num_samples = len(df)
-
- for col in columns:
- d_i = df[col].value_counts().to_dict()
- for k in d_i:
- d_i[k] = (d_i[k], d_i[k] / num_samples)
- cnt_dict[col] = d_i
-
- return cnt_dict
-
-
-def build_logger(work_dir, cfgname):
- log_file = cfgname + ".log"
- log_path = os.path.join(work_dir, log_file)
-
- logger = logging.getLogger(cfgname)
- logger.setLevel(logging.INFO)
- # formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
- formatter = logging.Formatter("%(asctime)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S")
-
- handler1 = logging.FileHandler(log_path)
- handler1.setFormatter(formatter)
-
- handler2 = logging.StreamHandler()
- handler2.setFormatter(formatter)
-
- logger.addHandler(handler1)
- logger.addHandler(handler2)
- logger.propagate = False
-
- return logger
-
-
-timings = {}
-from contextlib import contextmanager
-
-
-@contextmanager
-def torch_timer(prefix):
- import time
-
- import torch
-
- torch.cuda.synchronize()
- start = time.time()
- try:
- yield
- finally:
- torch.cuda.synchronize()
- t_diff = (time.time() - start) * 1000
- if prefix not in timings:
- timings[prefix] = []
-
- timings[prefix].append(t_diff)
-
- num_ignored = 10
-
- if len(timings[prefix]) > num_ignored:
- # avg = sum(timings[prefix][num_ignored:]) / (len(timings[prefix]) - num_ignored)
- avg = sum(timings[prefix][-num_ignored:]) / num_ignored
- print("{}: {} ({})".format(prefix, t_diff, avg))
- else:
- print("{}: {}".format(prefix, t_diff))
-
-
-def strip_dc(x):
- """
- strip DataContainer
- """
- try:
- import mmcv
- except ImportError:
- pass
-
- if isinstance(x, dict):
- res = {}
- for k, v in x.items():
- res[k] = strip_dc(v)
- return res
- if isinstance(x, (list, tuple)) and isinstance(x[0], mmcv.parallel.DataContainer):
- return strip_dc(x[0])
- elif isinstance(x, mmcv.parallel.DataContainer):
- return strip_dc(x.data)
- return x
+import collections
+import importlib
+import logging
+import os
+import time
+from collections import OrderedDict
+from collections.abc import Sequence
+from itertools import repeat
+
+import numpy as np
+import torch
+import torch.distributed as dist
+
+
+def print_rank(var_name, var_value, rank=0):
+ if dist.get_rank() == rank:
+ print(f"[Rank {rank}] {var_name}: {var_value}")
+
+
+def print_0(*args, **kwargs):
+ if dist.get_rank() == 0:
+ print(*args, **kwargs)
+
+
+def requires_grad(model: torch.nn.Module, flag: bool = True) -> None:
+ """
+ Set requires_grad flag for all parameters in a model.
+ """
+ for p in model.parameters():
+ p.requires_grad = flag
+
+
+def format_numel_str(numel: int) -> str:
+ B = 1024**3
+ M = 1024**2
+ K = 1024
+ if numel >= B:
+ return f"{numel / B:.2f} B"
+ elif numel >= M:
+ return f"{numel / M:.2f} M"
+ elif numel >= K:
+ return f"{numel / K:.2f} K"
+ else:
+ return f"{numel}"
+
+
+def all_reduce_mean(tensor: torch.Tensor) -> torch.Tensor:
+ dist.all_reduce(tensor=tensor, op=dist.ReduceOp.SUM)
+ tensor.div_(dist.get_world_size())
+ return tensor
+
+
+def get_model_numel(model: torch.nn.Module) -> (int, int):
+ num_params = 0
+ num_params_trainable = 0
+ for p in model.parameters():
+ num_params += p.numel()
+ if p.requires_grad:
+ num_params_trainable += p.numel()
+ return num_params, num_params_trainable
+
+
+def try_import(name):
+ """Try to import a module.
+
+ Args:
+ name (str): Specifies what module to import in absolute or relative
+ terms (e.g. either pkg.mod or ..mod).
+ Returns:
+ ModuleType or None: If importing successfully, returns the imported
+ module, otherwise returns None.
+ """
+ try:
+ return importlib.import_module(name)
+ except ImportError:
+ return None
+
+
+def transpose(x):
+ """
+ transpose a list of list
+ Args:
+ x (list[list]):
+ """
+ ret = list(map(list, zip(*x)))
+ return ret
+
+
+def get_timestamp():
+ timestamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(time.time()))
+ return timestamp
+
+
+def format_time(seconds):
+ days = int(seconds / 3600 / 24)
+ seconds = seconds - days * 3600 * 24
+ hours = int(seconds / 3600)
+ seconds = seconds - hours * 3600
+ minutes = int(seconds / 60)
+ seconds = seconds - minutes * 60
+ secondsf = int(seconds)
+ seconds = seconds - secondsf
+ millis = int(seconds * 1000)
+
+ f = ""
+ i = 1
+ if days > 0:
+ f += str(days) + "D"
+ i += 1
+ if hours > 0 and i <= 2:
+ f += str(hours) + "h"
+ i += 1
+ if minutes > 0 and i <= 2:
+ f += str(minutes) + "m"
+ i += 1
+ if secondsf > 0 and i <= 2:
+ f += str(secondsf) + "s"
+ i += 1
+ if millis > 0 and i <= 2:
+ f += str(millis) + "ms"
+ i += 1
+ if f == "":
+ f = "0ms"
+ return f
+
+
+def to_tensor(data):
+ """Convert objects of various python types to :obj:`torch.Tensor`.
+
+ Supported types are: :class:`numpy.ndarray`, :class:`torch.Tensor`,
+ :class:`Sequence`, :class:`int` and :class:`float`.
+
+ Args:
+ data (torch.Tensor | numpy.ndarray | Sequence | int | float): Data to
+ be converted.
+ """
+
+ if isinstance(data, torch.Tensor):
+ return data
+ elif isinstance(data, np.ndarray):
+ return torch.from_numpy(data)
+ elif isinstance(data, Sequence) and not isinstance(data, str):
+ return torch.tensor(data)
+ elif isinstance(data, int):
+ return torch.LongTensor([data])
+ elif isinstance(data, float):
+ return torch.FloatTensor([data])
+ else:
+ raise TypeError(f"type {type(data)} cannot be converted to tensor.")
+
+
+def to_ndarray(data):
+ if isinstance(data, torch.Tensor):
+ return data.numpy()
+ elif isinstance(data, np.ndarray):
+ return data
+ elif isinstance(data, Sequence):
+ return np.array(data)
+ elif isinstance(data, int):
+ return np.ndarray([data], dtype=int)
+ elif isinstance(data, float):
+ return np.array([data], dtype=float)
+ else:
+ raise TypeError(f"type {type(data)} cannot be converted to ndarray.")
+
+
+def to_torch_dtype(dtype):
+ if isinstance(dtype, torch.dtype):
+ return dtype
+ elif isinstance(dtype, str):
+ dtype_mapping = {
+ "float64": torch.float64,
+ "float32": torch.float32,
+ "float16": torch.float16,
+ "fp32": torch.float32,
+ "fp16": torch.float16,
+ "half": torch.float16,
+ "bf16": torch.bfloat16,
+ }
+ if dtype not in dtype_mapping:
+ raise ValueError
+ dtype = dtype_mapping[dtype]
+ return dtype
+ else:
+ raise ValueError
+
+
+def count_params(model):
+ return sum(p.numel() for p in model.parameters() if p.requires_grad)
+
+
+def _ntuple(n):
+ def parse(x):
+ if isinstance(x, collections.abc.Iterable) and not isinstance(x, str):
+ return x
+ return tuple(repeat(x, n))
+
+ return parse
+
+
+to_1tuple = _ntuple(1)
+to_2tuple = _ntuple(2)
+to_3tuple = _ntuple(3)
+to_4tuple = _ntuple(4)
+to_ntuple = _ntuple
+
+
+def convert_SyncBN_to_BN2d(model_cfg):
+ for k in model_cfg:
+ v = model_cfg[k]
+ if k == "norm_cfg" and v["type"] == "SyncBN":
+ v["type"] = "BN2d"
+ elif isinstance(v, dict):
+ convert_SyncBN_to_BN2d(v)
+
+
+def get_topk(x, dim=4, k=5):
+ x = to_tensor(x)
+ inds = x[..., dim].topk(k)[1]
+ return x[inds]
+
+
+def param_sigmoid(x, alpha):
+ ret = 1 / (1 + (-alpha * x).exp())
+ return ret
+
+
+def inverse_param_sigmoid(x, alpha, eps=1e-5):
+ x = x.clamp(min=0, max=1)
+ x1 = x.clamp(min=eps)
+ x2 = (1 - x).clamp(min=eps)
+ return torch.log(x1 / x2) / alpha
+
+
+def inverse_sigmoid(x, eps=1e-5):
+ """Inverse function of sigmoid.
+
+ Args:
+ x (Tensor): The tensor to do the
+ inverse.
+ eps (float): EPS avoid numerical
+ overflow. Defaults 1e-5.
+ Returns:
+ Tensor: The x has passed the inverse
+ function of sigmoid, has same
+ shape with input.
+ """
+ x = x.clamp(min=0, max=1)
+ x1 = x.clamp(min=eps)
+ x2 = (1 - x).clamp(min=eps)
+ return torch.log(x1 / x2)
+
+
+def count_columns(df, columns):
+ cnt_dict = OrderedDict()
+ num_samples = len(df)
+
+ for col in columns:
+ d_i = df[col].value_counts().to_dict()
+ for k in d_i:
+ d_i[k] = (d_i[k], d_i[k] / num_samples)
+ cnt_dict[col] = d_i
+
+ return cnt_dict
+
+
+def build_logger(work_dir, cfgname):
+ log_file = cfgname + ".log"
+ log_path = os.path.join(work_dir, log_file)
+
+ logger = logging.getLogger(cfgname)
+ logger.setLevel(logging.INFO)
+ # formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
+ formatter = logging.Formatter("%(asctime)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S")
+
+ handler1 = logging.FileHandler(log_path)
+ handler1.setFormatter(formatter)
+
+ handler2 = logging.StreamHandler()
+ handler2.setFormatter(formatter)
+
+ logger.addHandler(handler1)
+ logger.addHandler(handler2)
+ logger.propagate = False
+
+ return logger
+
+
+timings = {}
+from contextlib import contextmanager
+
+
+@contextmanager
+def torch_timer(prefix):
+ import time
+
+ import torch
+
+ torch.cuda.synchronize()
+ start = time.time()
+ try:
+ yield
+ finally:
+ torch.cuda.synchronize()
+ t_diff = (time.time() - start) * 1000
+ if prefix not in timings:
+ timings[prefix] = []
+
+ timings[prefix].append(t_diff)
+
+ num_ignored = 10
+
+ if len(timings[prefix]) > num_ignored:
+ # avg = sum(timings[prefix][num_ignored:]) / (len(timings[prefix]) - num_ignored)
+ avg = sum(timings[prefix][-num_ignored:]) / num_ignored
+ print("{}: {} ({})".format(prefix, t_diff, avg))
+ else:
+ print("{}: {}".format(prefix, t_diff))
+
+
+def strip_dc(x):
+ """
+ strip DataContainer
+ """
+ try:
+ import mmcv
+ except ImportError:
+ pass
+
+ if isinstance(x, dict):
+ res = {}
+ for k, v in x.items():
+ res[k] = strip_dc(v)
+ return res
+ if isinstance(x, (list, tuple)) and isinstance(x[0], mmcv.parallel.DataContainer):
+ return strip_dc(x[0])
+ elif isinstance(x, mmcv.parallel.DataContainer):
+ return strip_dc(x.data)
+ return x
diff --git a/opensora/utils/train_utils.py b/opensora/utils/train_utils.py
index 8a5c90f..f846043 100644
--- a/opensora/utils/train_utils.py
+++ b/opensora/utils/train_utils.py
@@ -29,6 +29,3 @@ def update_ema(
else:
param_data = param.data
ema_params[name].mul_(decay).add_(param_data, alpha=1 - decay)
-
-
-
diff --git a/requirements.txt b/requirements.txt
index 6e8196d..d2e02b8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -26,4 +26,3 @@ pyav
# text
clip
ftfy
-
diff --git a/scripts/train.py b/scripts/train.py
index b750748..9f611b7 100644
--- a/scripts/train.py
+++ b/scripts/train.py
@@ -11,13 +11,13 @@ from colossalai.nn.optimizer import HybridAdam
from colossalai.utils import get_current_device
from tqdm import tqdm
+from opensora.acceleration.checkpoint import set_grad_checkpoint
from opensora.acceleration.parallel_states import (
get_data_parallel_group,
set_data_parallel_group,
set_sequence_parallel_group,
)
from opensora.acceleration.plugin import ZeroSeqParallelPlugin
-from opensora.acceleration.checkpoint import set_grad_checkpoint
from opensora.datasets import DatasetFromCSV, get_transforms_image, get_transforms_video, prepare_dataloader
from opensora.registry import MODELS, SCHEDULERS, build_module
from opensora.utils.ckpt_utils import create_logger, load, model_sharding, record_model_param_shape, save
diff --git a/setup.py b/setup.py
index e1717af..1ac64d2 100644
--- a/setup.py
+++ b/setup.py
@@ -39,7 +39,7 @@ setup(
"*.egg-info",
)
),
- description="OpenDiT",
+ description="OpenSora",
long_description=fetch_readme(),
long_description_content_type="text/markdown",
license="Apache Software License 2.0",
diff --git a/tests/test_seq_parallel_attention.py b/tests/test_seq_parallel_attention.py
index 67ce7dc..7cae1f0 100644
--- a/tests/test_seq_parallel_attention.py
+++ b/tests/test_seq_parallel_attention.py
@@ -18,12 +18,7 @@ def run_attention(rank, world_size):
torch.manual_seed(1024)
set_sequence_parallel_group(dist.group.WORLD)
- seq_parallel_attention = SeqParallelAttention(
- dim=256,
- num_heads=4,
- qkv_bias=True,
- enable_flashattn=False
- ).cuda()
+ seq_parallel_attention = SeqParallelAttention(dim=256, num_heads=4, qkv_bias=True, enable_flashattn=False).cuda()
torch.manual_seed(1024)
attention = Attention(
@@ -61,7 +56,7 @@ def run_attention(rank, world_size):
if p.grad is not None:
dist.all_reduce(p.grad, group=dist.group.WORLD)
p.grad.div_(world_size)
-
+
# check grad
for p1, p2 in zip(seq_parallel_attention.parameters(), attention.parameters()):
assert torch.allclose(p1.grad, p2.grad, atol=1e-7), f"{p1.grad}\nvs\n{p2.grad}"
@@ -126,27 +121,29 @@ def run_cross_attention(rank, world_size):
p.grad.div_(world_size)
else:
print(f"grad of {name} is None")
-
+
# # check grad
for p1, p2 in zip(seq_parallel_attention.named_parameters(), attention.named_parameters()):
# if not torch.allclose(p1[1].grad, p2[1].grad, atol=1e-7):
# print(p1[0], p2[0])
- assert torch.allclose(p1[1].grad, p2[1].grad, atol=1e-7), f"\n{p1[0]}\nvs\n{p2[0]}:\n{p1[1].grad}\nvs\n{p2[1].grad}"
-
+ assert torch.allclose(
+ p1[1].grad, p2[1].grad, atol=1e-7
+ ), f"\n{p1[0]}\nvs\n{p2[0]}:\n{p1[1].grad}\nvs\n{p2[1].grad}"
+
# # check input grad
assert torch.allclose(x.grad, seq_x.grad, atol=1e-7), f"{x.grad}\nvs\n{seq_x.grad}"
assert torch.allclose(y.grad, seq_y.grad, atol=1e-7), f"{y.grad}\nvs\n{seq_y.grad}"
-
-
def run_dist(rank, world_size, port):
colossalai.launch({}, rank=rank, world_size=world_size, host="localhost", port=port)
# run_attention(rank, world_size)
run_cross_attention(rank, world_size)
+
def test_seq_parallel_attention():
spawn(run_dist, nprocs=2)
-if __name__ == '__main__':
+
+if __name__ == "__main__":
test_seq_parallel_attention()
diff --git a/tests/test_t5_shardformer.py b/tests/test_t5_shardformer.py
index 7e32e98..68040ab 100644
--- a/tests/test_t5_shardformer.py
+++ b/tests/test_t5_shardformer.py
@@ -19,14 +19,16 @@ def run_t5_encoder(rank, world_size, port):
sf_t5 = deepcopy(hf_t5)
# create huggingface model as normal
- shard_config = ShardConfig(tensor_parallel_process_group=None,
- pipeline_stage_manager=None,
- enable_tensor_parallelism=False,
- enable_fused_normalization=False,
- enable_flash_attention=False,
- enable_jit_fused=True,
- enable_sequence_parallelism=False,
- enable_sequence_overlap=False)
+ shard_config = ShardConfig(
+ tensor_parallel_process_group=None,
+ pipeline_stage_manager=None,
+ enable_tensor_parallelism=False,
+ enable_fused_normalization=False,
+ enable_flash_attention=False,
+ enable_jit_fused=True,
+ enable_sequence_parallelism=False,
+ enable_sequence_overlap=False,
+ )
shard_former = ShardFormer(shard_config=shard_config)
sharded_model, _ = shard_former.optimize(sf_t5.model, policy=T5EncoderPolicy())
sf_t5.model = sharded_model
@@ -65,5 +67,5 @@ def test_t5_encoder():
spawn(run_t5_encoder)
-if __name__ == '__main__':
+if __name__ == "__main__":
test_t5_encoder()
diff --git a/tools/data/get_caption_llava.py b/tools/data/get_caption_llava.py
index bb54974..ce26e5b 100644
--- a/tools/data/get_caption_llava.py
+++ b/tools/data/get_caption_llava.py
@@ -6,7 +6,6 @@ import cv2
import torch
from llava.constants import DEFAULT_IMAGE_TOKEN, IGNORE_INDEX, IMAGE_TOKEN_INDEX
from llava.conversation import conv_templates
-from llava.eval.run_llava import eval_model
from llava.mm_utils import get_anyres_image_grid_shape, get_model_name_from_path, process_images, tokenizer_image_token
from llava.model.builder import load_pretrained_model
from llava.model.llava_arch import unpad_image
@@ -137,7 +136,6 @@ def prepare_inputs_labels_for_multimodal(
labels = torch.full_like(input_ids, IGNORE_INDEX)
# remove the padding using attention_mask -- FIXME
- _input_ids = input_ids
input_ids = [
cur_input_ids[cur_attention_mask] for cur_input_ids, cur_attention_mask in zip(input_ids, attention_mask)
]
diff --git a/tools/data/scene_detect.py b/tools/data/scene_detect.py
index 115f683..c46e59d 100644
--- a/tools/data/scene_detect.py
+++ b/tools/data/scene_detect.py
@@ -1,139 +1,138 @@
-import os
-from tqdm import tqdm
-from multiprocessing import Pool
-from mmengine.logging import MMLogger, print_log
-
-from scenedetect import detect, ContentDetector
-
-from .utils import check_mp4_integrity, split_video
-from .utils import clone_folder_structure, iterate_files, iterate_folders
-from opensora.utils.misc import get_timestamp
-
-
-# config
-target_fps = 30 # int
-shorter_size = 512 # int
-min_seconds = 1 # float
-max_seconds = 5 # float
-assert max_seconds > min_seconds
-cfg = dict(
- target_fps=target_fps,
- min_seconds=min_seconds,
- max_seconds=max_seconds,
- shorter_size=shorter_size,
-)
-
-
-def process_folder(root_src, root_dst):
- # create logger
- folder_path_log = os.path.dirname(root_dst)
- log_name = os.path.basename(root_dst)
- timestamp = get_timestamp()
- log_path = os.path.join(folder_path_log, f'{log_name}_{timestamp}.log')
- logger = MMLogger.get_instance(log_name, log_file=log_path)
-
- # clone folder structure
- clone_folder_structure(root_src, root_dst)
-
- # all source videos
- mp4_list = [x for x in iterate_files(root_src) if x.endswith('.mp4')]
- mp4_list = sorted(mp4_list)
-
- for idx, sample_path in tqdm(enumerate(mp4_list)):
- folder_src = os.path.dirname(sample_path)
- folder_dst = os.path.join(root_dst, os.path.relpath(folder_src, root_src))
-
- # check src video integrity
- if not check_mp4_integrity(sample_path, logger=logger):
- continue
-
- # detect scenes
- scene_list = detect(sample_path, ContentDetector(), start_in_scene=True)
-
- # split scenes
- save_path_list = split_video(sample_path, scene_list, save_dir=folder_dst, **cfg, logger=logger)
-
- # check integrity of generated clips
- for x in save_path_list:
- check_mp4_integrity(x, logger=logger)
-
-
-def scene_detect():
- """ detect & cut scenes using a single process
- Expected dataset structure:
- data/
- your_dataset/
- raw_videos/
- xxx.mp4
- yyy.mp4
-
- This function results in:
- data/
- your_dataset/
- raw_videos/
- xxx.mp4
- yyy.mp4
- zzz.mp4
- clips/
- xxx_scene-0.mp4
- yyy_scene-0.mp4
- yyy_scene-1.mp4
- """
- # TODO: specify your dataset root
- root_src = f'./data/your_dataset/raw_videos'
- root_dst = f'./data/your_dataset/clips'
-
- process_folder(root_src, root_dst)
-
-
-def scene_detect_mp():
- """ detect & cut scenes using multiple processes
- Expected dataset structure:
- data/
- your_dataset/
- raw_videos/
- split_0/
- xxx.mp4
- yyy.mp4
- split_1/
- xxx.mp4
- yyy.mp4
-
- This function results in:
- data/
- your_dataset/
- raw_videos/
- split_0/
- xxx.mp4
- yyy.mp4
- split_1/
- xxx.mp4
- yyy.mp4
- clips/
- split_0/
- xxx_scene-0.mp4
- yyy_scene-0.mp4
- split_1/
- xxx_scene-0.mp4
- yyy_scene-0.mp4
- yyy_scene-1.mp4
- """
- # TODO: specify your dataset root
- root_src = f'./data/your_dataset/raw_videos'
- root_dst = f'./data/your_dataset/clips'
-
- # TODO: specify your splits
- splits = ['split_0', 'split_1']
-
- # process folders
- root_src_list = [os.path.join(root_src, x) for x in splits]
- root_dst_list = [os.path.join(root_dst, x) for x in splits]
-
- with Pool(processes=len(splits)) as pool:
- pool.starmap(process_folder, list(zip(root_src_list, root_dst_list)))
-
-
-if __name__ == '__main__':
- # TODO: choose single process or multiprocessing
- scene_detect()
- # scene_detect_mp()
+import os
+from multiprocessing import Pool
+
+from mmengine.logging import MMLogger
+from scenedetect import ContentDetector, detect
+from tqdm import tqdm
+
+from opensora.utils.misc import get_timestamp
+
+from .utils import check_mp4_integrity, clone_folder_structure, iterate_files, split_video
+
+# config
+target_fps = 30 # int
+shorter_size = 512 # int
+min_seconds = 1 # float
+max_seconds = 5 # float
+assert max_seconds > min_seconds
+cfg = dict(
+ target_fps=target_fps,
+ min_seconds=min_seconds,
+ max_seconds=max_seconds,
+ shorter_size=shorter_size,
+)
+
+
+def process_folder(root_src, root_dst):
+ # create logger
+ folder_path_log = os.path.dirname(root_dst)
+ log_name = os.path.basename(root_dst)
+ timestamp = get_timestamp()
+ log_path = os.path.join(folder_path_log, f"{log_name}_{timestamp}.log")
+ logger = MMLogger.get_instance(log_name, log_file=log_path)
+
+ # clone folder structure
+ clone_folder_structure(root_src, root_dst)
+
+ # all source videos
+ mp4_list = [x for x in iterate_files(root_src) if x.endswith(".mp4")]
+ mp4_list = sorted(mp4_list)
+
+ for idx, sample_path in tqdm(enumerate(mp4_list)):
+ folder_src = os.path.dirname(sample_path)
+ folder_dst = os.path.join(root_dst, os.path.relpath(folder_src, root_src))
+
+ # check src video integrity
+ if not check_mp4_integrity(sample_path, logger=logger):
+ continue
+
+ # detect scenes
+ scene_list = detect(sample_path, ContentDetector(), start_in_scene=True)
+
+ # split scenes
+ save_path_list = split_video(sample_path, scene_list, save_dir=folder_dst, **cfg, logger=logger)
+
+ # check integrity of generated clips
+ for x in save_path_list:
+ check_mp4_integrity(x, logger=logger)
+
+
+def scene_detect():
+ """detect & cut scenes using a single process
+ Expected dataset structure:
+ data/
+ your_dataset/
+ raw_videos/
+ xxx.mp4
+ yyy.mp4
+
+ This function results in:
+ data/
+ your_dataset/
+ raw_videos/
+ xxx.mp4
+ yyy.mp4
+ zzz.mp4
+ clips/
+ xxx_scene-0.mp4
+ yyy_scene-0.mp4
+ yyy_scene-1.mp4
+ """
+ # TODO: specify your dataset root
+ root_src = f"./data/your_dataset/raw_videos"
+ root_dst = f"./data/your_dataset/clips"
+
+ process_folder(root_src, root_dst)
+
+
+def scene_detect_mp():
+ """detect & cut scenes using multiple processes
+ Expected dataset structure:
+ data/
+ your_dataset/
+ raw_videos/
+ split_0/
+ xxx.mp4
+ yyy.mp4
+ split_1/
+ xxx.mp4
+ yyy.mp4
+
+ This function results in:
+ data/
+ your_dataset/
+ raw_videos/
+ split_0/
+ xxx.mp4
+ yyy.mp4
+ split_1/
+ xxx.mp4
+ yyy.mp4
+ clips/
+ split_0/
+ xxx_scene-0.mp4
+ yyy_scene-0.mp4
+ split_1/
+ xxx_scene-0.mp4
+ yyy_scene-0.mp4
+ yyy_scene-1.mp4
+ """
+ # TODO: specify your dataset root
+ root_src = f"./data/your_dataset/raw_videos"
+ root_dst = f"./data/your_dataset/clips"
+
+ # TODO: specify your splits
+ splits = ["split_0", "split_1"]
+
+ # process folders
+ root_src_list = [os.path.join(root_src, x) for x in splits]
+ root_dst_list = [os.path.join(root_dst, x) for x in splits]
+
+ with Pool(processes=len(splits)) as pool:
+ pool.starmap(process_folder, list(zip(root_src_list, root_dst_list)))
+
+
+if __name__ == "__main__":
+ # TODO: choose single process or multiprocessing
+ scene_detect()
+ # scene_detect_mp()
diff --git a/tools/data/utils.py b/tools/data/utils.py
index b65270f..19eae31 100644
--- a/tools/data/utils.py
+++ b/tools/data/utils.py
@@ -1,148 +1,145 @@
-import os
-import cv2
-import subprocess
-from mmengine.logging import print_log
-
-from moviepy.editor import VideoFileClip
-from imageio_ffmpeg import get_ffmpeg_exe
-from scenedetect import FrameTimecode
-
-
-def iterate_files(folder_path):
- for root, dirs, files in os.walk(folder_path):
- # root contains the current directory path
- # dirs contains the list of subdirectories in the current directory
- # files contains the list of files in the current directory
-
- # Process files in the current directory
- for file in files:
- file_path = os.path.join(root, file)
- # print("File:", file_path)
- yield file_path
-
- # Process subdirectories and recursively call the function
- for subdir in dirs:
- subdir_path = os.path.join(root, subdir)
- # print("Subdirectory:", subdir_path)
- iterate_files(subdir_path)
-
-
-def iterate_folders(folder_path):
- for root, dirs, files in os.walk(folder_path):
- for subdir in dirs:
- subdir_path = os.path.join(root, subdir)
- yield subdir_path
- # print("Subdirectory:", subdir_path)
- iterate_folders(subdir_path)
-
-
-def clone_folder_structure(root_src, root_dst, verbose=False):
- src_path_list = iterate_folders(root_src)
- src_relpath_list = [os.path.relpath(x, root_src) for x in src_path_list]
-
- os.makedirs(root_dst, exist_ok=True)
- dst_path_list = [os.path.join(root_dst, x) for x in src_relpath_list]
- for folder_path in dst_path_list:
- os.makedirs(folder_path, exist_ok=True)
- if verbose:
- print(f'Create folder: \'{folder_path}\'')
-
-
-def count_files(root, suffix='.mp4'):
- files_list = iterate_files(root)
- cnt = len([x for x in files_list if x.endswith(suffix)])
- return cnt
-
-
-def check_mp4_integrity(file_path, verbose=True, logger=None):
- try:
- video_clip = VideoFileClip(file_path)
- if verbose:
- print_log(f'The MP4 file \'{file_path}\' is intact.', logger=logger)
- return True
- except Exception as e:
- if verbose:
- print_log(f'Error: {e}', logger=logger)
- print_log(f'The MP4 file \'{file_path}\' is not intact.', logger=logger)
- return False
-
-
-def count_frames(video_path):
- cap = cv2.VideoCapture(video_path)
-
- if not cap.isOpened():
- print(f"Error: Could not open video file '{video_path}'")
- return
-
- total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
- print(f"Total frames in the video '{video_path}': {total_frames}")
-
- cap.release()
-
-
-def split_video(sample_path,
- scene_list,
- save_dir,
- target_fps=30,
- min_seconds=1,
- max_seconds=10,
- shorter_size=512,
- verbose=False,
- logger=None,
- ):
- FFMPEG_PATH = get_ffmpeg_exe()
-
- save_path_list = []
- for idx, scene in enumerate(scene_list):
- s, t = scene # FrameTimecode
- fps = s.framerate
- max_duration = FrameTimecode(timecode='00:00:00', fps=fps)
- max_duration.frame_num = round(fps * max_seconds)
- duration = min(max_duration, t - s)
- if duration.get_frames() < round(min_seconds * fps):
- continue
-
- # save path
- fname = os.path.basename(sample_path)
- fname_wo_ext = os.path.splitext(fname)[0]
- # TODO: fname pattern
- save_path = os.path.join(save_dir, f'{fname_wo_ext}_scene-{idx}.mp4')
-
- # ffmpeg cmd
- cmd = [FFMPEG_PATH]
-
- # Only show ffmpeg output for the first call, which will display any
- # errors if it fails, and then break the loop. We only show error messages
- # for the remaining calls.
- # cmd += ['-v', 'error']
-
- # input path
- cmd += ['-i', sample_path]
-
- # clip to cut
- cmd += [
- '-nostdin', '-y',
- '-ss', str(s.get_seconds()),
- '-t', str(duration.get_seconds())
- ]
-
- # target fps
- # cmd += ['-vf', 'select=mod(n\,2)']
- cmd += ['-r', f'{target_fps}']
-
- # aspect ratio
- cmd += ['-vf', f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]
- # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]
-
- cmd += ['-map', '0', save_path]
-
- proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
- stdout, stderr = proc.communicate()
- if verbose:
- stdout = stdout.decode('utf-8')
- print_log(stdout, logger=logger)
-
- save_path_list.append(sample_path)
- print_log(f'Video clip saved to \'{save_path}\'', logger=logger)
-
- return save_path_list
+import os
+import subprocess
+
+import cv2
+from imageio_ffmpeg import get_ffmpeg_exe
+from mmengine.logging import print_log
+from moviepy.editor import VideoFileClip
+from scenedetect import FrameTimecode
+
+
+def iterate_files(folder_path):
+ for root, dirs, files in os.walk(folder_path):
+ # root contains the current directory path
+ # dirs contains the list of subdirectories in the current directory
+ # files contains the list of files in the current directory
+
+ # Process files in the current directory
+ for file in files:
+ file_path = os.path.join(root, file)
+ # print("File:", file_path)
+ yield file_path
+
+ # Process subdirectories and recursively call the function
+ for subdir in dirs:
+ subdir_path = os.path.join(root, subdir)
+ # print("Subdirectory:", subdir_path)
+ iterate_files(subdir_path)
+
+
+def iterate_folders(folder_path):
+ for root, dirs, files in os.walk(folder_path):
+ for subdir in dirs:
+ subdir_path = os.path.join(root, subdir)
+ yield subdir_path
+ # print("Subdirectory:", subdir_path)
+ iterate_folders(subdir_path)
+
+
+def clone_folder_structure(root_src, root_dst, verbose=False):
+ src_path_list = iterate_folders(root_src)
+ src_relpath_list = [os.path.relpath(x, root_src) for x in src_path_list]
+
+ os.makedirs(root_dst, exist_ok=True)
+ dst_path_list = [os.path.join(root_dst, x) for x in src_relpath_list]
+ for folder_path in dst_path_list:
+ os.makedirs(folder_path, exist_ok=True)
+ if verbose:
+ print(f"Create folder: '{folder_path}'")
+
+
+def count_files(root, suffix=".mp4"):
+ files_list = iterate_files(root)
+ cnt = len([x for x in files_list if x.endswith(suffix)])
+ return cnt
+
+
+def check_mp4_integrity(file_path, verbose=True, logger=None):
+ try:
+ VideoFileClip(file_path)
+ if verbose:
+ print_log(f"The MP4 file '{file_path}' is intact.", logger=logger)
+ return True
+ except Exception as e:
+ if verbose:
+ print_log(f"Error: {e}", logger=logger)
+ print_log(f"The MP4 file '{file_path}' is not intact.", logger=logger)
+ return False
+
+
+def count_frames(video_path):
+ cap = cv2.VideoCapture(video_path)
+
+ if not cap.isOpened():
+ print(f"Error: Could not open video file '{video_path}'")
+ return
+
+ total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+ print(f"Total frames in the video '{video_path}': {total_frames}")
+
+ cap.release()
+
+
+def split_video(
+ sample_path,
+ scene_list,
+ save_dir,
+ target_fps=30,
+ min_seconds=1,
+ max_seconds=10,
+ shorter_size=512,
+ verbose=False,
+ logger=None,
+):
+ FFMPEG_PATH = get_ffmpeg_exe()
+
+ save_path_list = []
+ for idx, scene in enumerate(scene_list):
+ s, t = scene # FrameTimecode
+ fps = s.framerate
+ max_duration = FrameTimecode(timecode="00:00:00", fps=fps)
+ max_duration.frame_num = round(fps * max_seconds)
+ duration = min(max_duration, t - s)
+ if duration.get_frames() < round(min_seconds * fps):
+ continue
+
+ # save path
+ fname = os.path.basename(sample_path)
+ fname_wo_ext = os.path.splitext(fname)[0]
+ # TODO: fname pattern
+ save_path = os.path.join(save_dir, f"{fname_wo_ext}_scene-{idx}.mp4")
+
+ # ffmpeg cmd
+ cmd = [FFMPEG_PATH]
+
+ # Only show ffmpeg output for the first call, which will display any
+ # errors if it fails, and then break the loop. We only show error messages
+ # for the remaining calls.
+ # cmd += ['-v', 'error']
+
+ # input path
+ cmd += ["-i", sample_path]
+
+ # clip to cut
+ cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())]
+
+ # target fps
+ # cmd += ['-vf', 'select=mod(n\,2)']
+ cmd += ["-r", f"{target_fps}"]
+
+ # aspect ratio
+ cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]
+ # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]
+
+ cmd += ["-map", "0", save_path]
+
+ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ stdout, stderr = proc.communicate()
+ if verbose:
+ stdout = stdout.decode("utf-8")
+ print_log(stdout, logger=logger)
+
+ save_path_list.append(sample_path)
+ print_log(f"Video clip saved to '{save_path}'", logger=logger)
+
+ return save_path_list