Open-Sora: Democratizing Efficient Video Production for All
Python
Docs/fix zw (#476)
Zheng Zangwei (Alex Zheng)eaa69025272e
diff --git a/README.md b/README.mdindex 42201f6..2144526 100644--- a/README.md+++ b/README.md@@ -89,6 +89,10 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.- β Trained our 3D-VAE for temporal dimension compression.- π **Open-Sora 1.1** released. Model weights are available [here](#model-weights). It is trained on **0s~15s, 144p to 720p, various aspect ratios** videos. See our **[report 1.1](/docs/report_02.md)** for more discussions.- π§ **Data processing pipeline v1.1** is released. An automatic [processing pipeline](#data-processing) from raw videos to (text, video clip) pairs is provided, including scene cutting $\rightarrow$ filtering(aesthetic, optical flow, OCR, etc.) $\rightarrow$ captioning $\rightarrow$ managing. With this tool, you can easily build your video dataset.++<details>+<summary>View more</summary>+- β Improved ST-DiT architecture includes rope positional encoding, qk norm, longer text length, etc.- β Support training with any resolution, aspect ratio, and duration (including images).- β Support image and video conditioning and video editing, and thus support animating images, connecting videos, etc.@@ -103,10 +107,6 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.including [downloading](tools/datasets/README.md), [video cutting](tools/scene_cut/README.md),and [captioning](tools/caption/README.md) tools. Our data collection plan can be foundat [datasets.md](docs/datasets.md).--<details>-<summary>View more</summary>-- β We find VQ-VAE from [VideoGPT](https://wilson1yan.github.io/videogpt/index.html) has a low quality and thus adopt abetter VAE from [Stability-AI](https://huggingface.co/stabilityai/sd-vae-ft-mse-original). We also find patching inthe time dimension deteriorates the quality. See our **[report](docs/report_01.md)** for more discussions.@@ -154,7 +154,11 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.Other useful documents and links are listed below.-- Report: [report 1.2](docs/report_03.md), [report 1.1](docs/report_02.md), [report 1.0](docs/report_01.md), [acceleration.md](docs/acceleration.md)+- Report: each version is trained from a image base seperately (not continuously trained), while a newer version will incorporate the techniques from the previous version.+ - [report 1.2](docs/report_03.md): rectified flow, 3d-VAE, score condition, evaluation, etc.+ - [report 1.1](docs/report_02.md): multi-resolution/length/aspect-ratio, image/video conditioning/editing, data preprocessing, etc.+ - [report 1.0](docs/report_01.md): architecture, captioning, etc.+ - [acceleration.md](docs/acceleration.md)- Repo structure: [structure.md](docs/structure.md)- Config file explanation: [config.md](docs/config.md)- Useful commands: [commands.md](docs/commands.md)@@ -494,7 +498,7 @@ For training other models and advanced usage, see [here](docs/commands.md) for mWe support evaluation based on:- Validation loss-- VBench score+- [VBench](https://github.com/Vchitect/VBench/tree/master) score- VBench-i2v score- Batch generation for human evaluationdiff --git a/docs/report_03.md b/docs/report_03.mdindex cb9677c..02c55b7 100644--- a/docs/report_03.md+++ b/docs/report_03.md@@ -63,7 +63,7 @@ Lastest diffusion model like Stable Diffusion 3 adopts the [rectified flow](httpFor the resolution-aware timestep sampling, we should use more noise for images with larger resolution. We extend this idea to video generation and use more noise for videos with longer length.-Open-Sora 1.2 starts from the [PixArt-Ξ£ 2K](https://github.com/PixArt-alpha/PixArt-sigma) checkpoint. Note that this model is trained with DDPM and SDXL VAE, also a much higher resolution. We find finetuning on a small dataset can easily adapt the model for our video generation setting. The adaptation process is as follows, all training is done on 8 GPUs:+Open-Sora 1.2 starts from the [PixArt-Ξ£ 2K](https://github.com/PixArt-alpha/PixArt-sigma) checkpoint. Note that this model is trained with DDPM and SDXL VAE, also a much higher resolution. We find finetuning on a small dataset can easily adapt the model for our video generation setting. The adaptation process is as follows, all training is done on 8 GPUs (the adaptation for the diffusion model is quite fast and straightforward):1. Multi-resolution image generation ability: we train the model to generate different resolution ranging from 144p to 2K for 20k steps.2. QK-norm: we add the QK-norm to the model and train for 18k steps.