Open-Sora: Democratizing Efficient Video Production for All
Python
Update README.md
fastalgo126896afcbc3
diff --git a/README.md b/README.mdindex d0e0008..e356082 100644--- a/README.md+++ b/README.md@@ -26,11 +26,11 @@## π OverviewOpen-Sora is an open-source project that provides a high-performance implementation of the development pipeline that Sora might use powered by [Colossal-AI](https://github.com/hpcaitech/ColossalAI), including:-- Provide **a complete Sora reproduction architecture solution**, including the whole process from data processing to training and inference.+- Provides **a complete Sora reproduction architecture solution**, including the whole process from data processing to training and deployment.- Supports **dynamic resolution**, training can directly train any resolution of the video, without scaling.-- Supports **multiple model structures**. Since the actual model structure of Sora is unknown, we realize three common multimodal model structures such as adaLN-zero, cross attention, and in-context conditioning (token concat).-- Supports **multiple video compression methods**. Users can choose to use original video, VQVAE (video native model), SD-VAE (image native model) for training.-- Supports **multiple parallel training optimizations**. Including the AI large model system optimization capability combined with Colossal-AI, and hybrid sequence parallelism with Ulysses and FastSeq.+- Supports **multiple model structures**. Since the actual model structure of Sora is unknown, we implement three common multimodal model structures such as adaLN-zero, cross attention, and in-context conditioning (token concat).+- Supports **multiple video compression methods**. Users can choose to use original video, VQVAE (video native model), or SD-VAE (image native model) for training.+- Supports **multiple parallel training optimizations**. Including the AI large model system optimization capability compatible with Colossal-AI, and hybrid sequence parallelism with Ulysses and FastSeq.<p id="diffusion_demo" align="center"><img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/open-sora-1.png" width=800/>@@ -44,7 +44,7 @@ Open-Sora is an open-source project that provides a high-performance implementat### Use MSR-VTT-We use [MSR-VTT](https://cove.thecvf.com/datasets/839) dataset, which is a large-scale video description dataset. We should preprocess the raw videos before training the model. You can use the following scripts to perform data processing.+We use [MSR-VTT](https://cove.thecvf.com/datasets/839) dataset, which is a large-scale video description dataset. Users should preprocess the raw videos before training the model. You can use the following scripts to perform data processing.```bash@@ -61,7 +61,7 @@ python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/val/annotatipython scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/test/annotations.json -v ./dataset/MSRVTT-collated/test/videos -o ./dataset/MSRVTT-processed/test```-After completing the steps, you should have a processed MSR-VTT dataset in `./dataset/MSRVTT-processed`.+After completing these steps, you should have a processed MSR-VTT dataset in `./dataset/MSRVTT-processed`.### Use Customized Datasets@@ -92,7 +92,7 @@ Here is an example of the video directory:βββ ...```-Each video may have multiple captions. So the outputs are video-caption pairs. E.g., the first video has two captions, then the output will be two video-caption pairs.+Each video may have multiple captions. So the outputs are video-caption pairs. E.g., If the first video has two captions, then the output will be two video-caption pairs.We use [VQ-VAE](https://github.com/wilson1yan/VideoGPT/) to quantize the video frames. And we use [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#clip) to extract the text features.@@ -109,7 +109,7 @@ Note that this script needs to be run on a machine with a GPU. To avoid CUDA OOM## π Get Started-In this section, we will provide a guidance on how to run training and inference. Before that, make sure you installed the dependencies with the command below.+In this section, we will guide how to run training and inference. Before that, make sure you installed the dependencies with the command below.```bashpip install -r requirements.txt@@ -142,7 +142,7 @@ python sample.py -h## πͺ Acknowledgement-During development of the project, we learnt a lot from the following public materials:+During the development of the project, we learned a lot from the following information:- [OpenAI Sora Technical Report](https://openai.com/research/video-generation-models-as-world-simulators)- [VideoGPT Project](https://github.com/wilson1yan/VideoGPT)