Open-Sora: Democratizing Efficient Video Production for All
Python
update docs
Zangwei Zhengf41ac612908a
diff --git a/README.md b/README.mdindex 9e3c35c..12a8f33 100644--- a/README.md+++ b/README.md@@ -16,21 +16,22 @@## π° News* **[2024.03.18]** π₯ We release **Open-Sora 1.0**, an open-source project to reproduce OpenAI Sora.-Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with+Open-Sora 1.0 supports a full pipeline including video data preprocessing, training with<a href="https://github.com/hpcaitech/ColossalAI"><img src="assets/readme/colossal_ai.png" width="8%" ></a> acceleration,-inference, and more. Our provided checkpoint can produce 2s 512x512 videos.+inference, and more. Our provided checkpoint can produce 2~5s 512x512 videos with only 3 days training.## π₯ Latest Demo-| **2s 512x512** | **2s 512x512** | **2s 512x512** |+| **2s 512Γ512** | **2s 512Γ512** | **2s 512Γ512** || ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- || [<img src="assets/readme/sample_0.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/de1963d3-b43b-4e68-a670-bb821ebb6f80) | [<img src="assets/readme/sample_1.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/13f8338f-3d42-4b71-8142-d234fbd746cc) | [<img src="assets/readme/sample_2.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/fa6a65a6-e32a-4d64-9a9e-eabb0ebb8c16) |-Videos are downsampled to `.gif`. Click the video for original ones.+Videos are downsampled to `.gif` for display. Click the video for original ones.## π New Features/Updates-* π Open-Sora-v1 is trained on xxx. We train the model in three stages. Model weights are available here. Training details can be found here. [WIP]+* π Open-Sora-v1 released. Model weights are available [here](#model-weights). With only 400K video clips and 200 H800 days, we are able to generate 2s 512Γ512 videos.+* β Three stages training from an image diffusion model to a video diffusion model. We provide the weights for each stage.* β Support training acceleration including accelerated transformer, faster T5 and VAE, and sequence parallelism. Open-Sora improve **55%** training speed when training on 64x512x512 videos. Details locates at [acceleration.md](docs/acceleration.md).* β We provide video cutting and captioning tools for data preprocessing. Instructions can be found [here](tools/data/README.md) and our data collection plan can be found at [datasets.md](docs/datasets.md).* β We find VQ-VAE from [VideoGPT](https://wilson1yan.github.io/videogpt/index.html) has a low quality and thus adopt a better VAE from [Stability-AI](https://huggingface.co/stabilityai/sd-vae-ft-mse-original). We also find patching in the time dimension deteriorates the quality. See our **[report](docs/report_v1.md)** for more discussions.@@ -103,9 +104,14 @@ After installation, we suggest reading [structure.md](docs/structure.md) to lear## Model Weights-| Model | #Params | url |-| ---------- | ------- | --- |-| 16x256x256 | | |+| Resoluion | Data | #iterations | Batch Size | GPU days (H800) | URL |+| ---------- | ------ | ----------- | ---------- | --------------- | --- |+| 16Γ256Γ256 | 366K | 80k | 8Γ64 | 117 | |+| 16Γ256Γ256 | 20K HQ | 24k | 8Γ64 | 45 | |+| 16Γ512Γ512 | 20K HQ | 20k | 2Γ64 | 35 | |+| 64Γ512Γ512 | 50K HQ | | 4Γ64 | | |++Our model's weight is partially initialized from [PixArt-Ξ±](https://github.com/PixArt-alpha/PixArt-alpha). The number of parameters is 724M. More information about training can be found in [report_v1.md](/docs/report_v1.md). More about dataset can be found in [dataset.md](/docs/dataset.md).## Inference@@ -122,7 +128,7 @@ python scripts/inference.py configs/opensora/inference/64x512x512.pyFor inference with other models, see [here](docs/commands.md) for more instructions.-## Data Processing+## Data Processing (WIP)### Split video into clipsdiff --git a/docs/datasets.md b/docs/datasets.mdindex e69de29..8c3b03b 100644--- a/docs/datasets.md+++ b/docs/datasets.md@@ -0,0 +1,3 @@+# Datasets++This page is for datasets. For data preprocessing, see [tools/README.md](/tools/README.md).diff --git a/docs/report_v1.md b/docs/report_v1.mdindex e69de29..cac8835 100644--- a/docs/report_v1.md+++ b/docs/report_v1.md@@ -0,0 +1,2 @@+# Open-Sora v1 Report+diff --git a/tools/data/README.md b/tools/data/README.mdindex e69de29..74658e5 100644--- a/tools/data/README.md+++ b/tools/data/README.md@@ -0,0 +1,2 @@+# Data Preprocessing Tools+