Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

Update README.md

fastalgo126896afcbc3
diff --git a/README.md b/README.md
index d0e0008..e356082 100644
--- a/README.md
+++ b/README.md
@@ -26,11 +26,11 @@
## πŸ“ Overview
Open-Sora is an open-source project that provides a high-performance implementation of the development pipeline that Sora might use powered by [Colossal-AI](https://github.com/hpcaitech/ColossalAI), including:
-- Provide **a complete Sora reproduction architecture solution**, including the whole process from data processing to training and inference.
+- Provides **a complete Sora reproduction architecture solution**, including the whole process from data processing to training and deployment.
- Supports **dynamic resolution**, training can directly train any resolution of the video, without scaling.
-- Supports **multiple model structures**. Since the actual model structure of Sora is unknown, we realize three common multimodal model structures such as adaLN-zero, cross attention, and in-context conditioning (token concat).
-- Supports **multiple video compression methods**. Users can choose to use original video, VQVAE (video native model), SD-VAE (image native model) for training.
-- Supports **multiple parallel training optimizations**. Including the AI large model system optimization capability combined with Colossal-AI, and hybrid sequence parallelism with Ulysses and FastSeq.
+- Supports **multiple model structures**. Since the actual model structure of Sora is unknown, we implement three common multimodal model structures such as adaLN-zero, cross attention, and in-context conditioning (token concat).
+- Supports **multiple video compression methods**. Users can choose to use original video, VQVAE (video native model), or SD-VAE (image native model) for training.
+- Supports **multiple parallel training optimizations**. Including the AI large model system optimization capability compatible with Colossal-AI, and hybrid sequence parallelism with Ulysses and FastSeq.
<p id="diffusion_demo" align="center">
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/applications/sora/open-sora-1.png" width=800/>
@@ -44,7 +44,7 @@ Open-Sora is an open-source project that provides a high-performance implementat
### Use MSR-VTT
-We use [MSR-VTT](https://cove.thecvf.com/datasets/839) dataset, which is a large-scale video description dataset. We should preprocess the raw videos before training the model. You can use the following scripts to perform data processing.
+We use [MSR-VTT](https://cove.thecvf.com/datasets/839) dataset, which is a large-scale video description dataset. Users should preprocess the raw videos before training the model. You can use the following scripts to perform data processing.
```bash
@@ -61,7 +61,7 @@ python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/val/annotati
python scripts/data/preprocess_data.py -c ./dataset/MSRVTT-collated/test/annotations.json -v ./dataset/MSRVTT-collated/test/videos -o ./dataset/MSRVTT-processed/test
```
-After completing the steps, you should have a processed MSR-VTT dataset in `./dataset/MSRVTT-processed`.
+After completing these steps, you should have a processed MSR-VTT dataset in `./dataset/MSRVTT-processed`.
### Use Customized Datasets
@@ -92,7 +92,7 @@ Here is an example of the video directory:
└── ...
```
-Each video may have multiple captions. So the outputs are video-caption pairs. E.g., the first video has two captions, then the output will be two video-caption pairs.
+Each video may have multiple captions. So the outputs are video-caption pairs. E.g., If the first video has two captions, then the output will be two video-caption pairs.
We use [VQ-VAE](https://github.com/wilson1yan/VideoGPT/) to quantize the video frames. And we use [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#clip) to extract the text features.
@@ -109,7 +109,7 @@ Note that this script needs to be run on a machine with a GPU. To avoid CUDA OOM
## πŸš€ Get Started
-In this section, we will provide a guidance on how to run training and inference. Before that, make sure you installed the dependencies with the command below.
+In this section, we will guide how to run training and inference. Before that, make sure you installed the dependencies with the command below.
```bash
pip install -r requirements.txt
@@ -142,7 +142,7 @@ python sample.py -h
## πŸͺ„ Acknowledgement
-During development of the project, we learnt a lot from the following public materials:
+During the development of the project, we learned a lot from the following information:
- [OpenAI Sora Technical Report](https://openai.com/research/video-generation-models-as-world-simulators)
- [VideoGPT Project](https://github.com/wilson1yan/VideoGPT)