Open-Sora: Democratizing Efficient Video Production for All
Python
Update README.md
Yanjia096b4df6be498
diff --git a/docs/zh_CN/README.md b/docs/zh_CN/README.mdindex 1ca8378..c2e8755 100644--- a/docs/zh_CN/README.md+++ b/docs/zh_CN/README.md@@ -12,150 +12,132 @@<a href="https://huggingface.co/spaces/hpcai-tech/open-sora"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Gradio Demo-blue"></a></div>-## Open-Sora: Democratizing Efficient Video Production for All+## Open-Sora: 让所有人都能轻松制作高效视频-We design and implement **Open-Sora**, an initiative dedicated to **efficiently** producing high-quality video. We hope to make the model,-tools and all details accessible to all. By embracing **open-source** principles,-Open-Sora not only democratizes access to advanced video generation techniques, but also offers a-streamlined and user-friendly platform that simplifies the complexities of video generation.-With Open-Sora, our goal is to foster innovation, creativity, and inclusivity within the field of content creation.+我们设计并实施了**Open-Sora**,这是一项致力于高效制作高质量视频的计划。我们希望让所有人都能使用模型、工具和所有细节。通过采用开源原则,Open-Sora 不仅使高级视频生成技术的使用变得民主化,而且还提供了一个简化且用户友好的平台,简化了视频生成的复杂性。借助 Open-Sora,我们的目标是在内容创作领域促进创新、创造力和包容性。[[中文文档]](/docs/zh_CN/README.md) [[潞晨云部署视频教程]](https://www.bilibili.com/video/BV141421R7Ag)-## 📰 News+## 📰 资讯-* **[2024.06.17]** 🔥 We released **Open-Sora 1.2**, which includes **3D-VAE**, **rectified flow**, and **score condition**. The video quality is greatly improved. [[checkpoints]](#open-sora-10-model-weights) [[report]](/docs/report_03.md)-* **[2024.04.25]** 🤗 We released the [Gradio demo for Open-Sora](https://huggingface.co/spaces/hpcai-tech/open-sora) on Hugging Face Spaces.-* **[2024.04.25]** We released **Open-Sora 1.1**, which supports **2s~15s, 144p to 720p, any aspect ratio** text-to-image, **text-to-video, image-to-video, video-to-video, infinite time** generation. In addition, a full video processing pipeline is released. [[checkpoints]]() [[report]](/docs/report_02.md)-* **[2024.03.18]** We released **Open-Sora 1.0**, a fully open-source project for video generation.- Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with+* **[2024.06.17]** 🔥+* 我们发布了**Open-Sora 1.2**,其中包括**3D-VAE**,**整流流**和**得分条件**。视频质量大大提高。[[checkpoints]](#open-sora-10-model-weights) [[report]](/docs/report_03.md)+* **[2024.04.25]** 🤗 我们在 Hugging Face Spaces 上发布了 [Open-Sora的Gradio演示](https://huggingface.co/spaces/hpcai-tech/open-sora)。+* **[2024.04.25]** 我们发布了**Open-Sora 1.1**,支持**2s~15s、144p 到 720p、任意比例的文本转图片、文本转视频、图片转视频、视频转视频、无限时间生成**。此外,还发布了完整的视频处理管道。 [[checkpoints]]() [[report]](/docs/report_02.md)+* **[2024.03.18]** 我们发布了 **Open-Sora 1.0**, 一个完全开源的视频生成项目。Open-Sora 1.0 支持完整的视频数据预处理流程、加速训练<a href="https://github.com/hpcaitech/ColossalAI"><img src="assets/readme/colossal_ai.png" width="8%" ></a>- acceleration,- inference, and more. Our model can produce 2s 512x512 videos with only 3 days training. [[checkpoints]](#open-sora-10-model-weights)+、推理等。我们的模型只需 3 天的训练就可以生成 2 秒的 512x512 视频。 [[checkpoints]](#open-sora-10-model-weights)[[blog]](https://hpc-ai.com/blog/open-sora-v1.0) [[report]](/docs/report_01.md)-* **[2024.03.04]** Open-Sora provides training with 46% cost reduction.+* **[2024.03.04]** Open-Sora 提供培训,成本降低 46%。[[blog]](https://hpc-ai.com/blog/open-sora)## 🎥 Latest Demo-🔥 You can experience Open-Sora on our [🤗 Gradio application on Hugging Face](https://huggingface.co/spaces/hpcai-tech/open-sora). More samples are available in our [Gallery](https://hpcaitech.github.io/Open-Sora/).+🔥 您可以在HuggingFace上的 [🤗 Gradio应用程序](https://huggingface.co/spaces/hpcai-tech/open-sora)上体验Open-Sora. 我们的[画廊](https://hpcaitech.github.io/Open-Sora/)中提供了更多示例.<details>-<summary>OpenSora 1.1 Demo</summary>+<summary>OpenSora 1.1 演示</summary>-| **2s 240×426** | **2s 240×426** |+| **2秒 240×426** | **2秒 240×426** || ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- || [<img src="assets/demo/sample_16x240x426_9.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) | [<img src="assets/demo/sora_16x240x426_26.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) || [<img src="assets/demo/sora_16x240x426_27.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/f7ce4aaa-528f-40a8-be7a-72e61eaacbbd) | [<img src="assets/demo/sora_16x240x426_40.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/5d58d71e-1fda-4d90-9ad3-5f2f7b75c6a9) |-| **2s 426×240** | **4s 480×854** |+| **2秒 426×240** | **4秒 480×854** || ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- || [<img src="assets/demo/sora_16x426x240_24.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/34ecb4a0-4eef-4286-ad4c-8e3a87e5a9fd) | [<img src="assets/demo/sample_32x480x854_9.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c1619333-25d7-42ba-a91c-18dbc1870b18) |-| **16s 320×320** | **16s 224×448** | **2s 426×240** |+| **16秒 320×320** | **16秒 224×448** | **2秒 426×240** || ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- || [<img src="assets/demo/sample_16s_320x320.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/3cab536e-9b43-4b33-8da8-a0f9cf842ff2) | [<img src="assets/demo/sample_16s_224x448.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/9fb0b9e0-c6f4-4935-b29e-4cac10b373c4) | [<img src="assets/demo/sora_16x426x240_3.gif" width="">](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/3e892ad2-9543-4049-b005-643a4c1bf3bf) |+</details><details><summary>OpenSora 1.0 Demo</summary>-| **2s 512×512** | **2s 512×512** | **2s 512×512** |+| **2秒 512×512** | **2秒 512×512** | **2秒 512×512** || ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- || [<img src="assets/readme/sample_0.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/de1963d3-b43b-4e68-a670-bb821ebb6f80) | [<img src="assets/readme/sample_1.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/13f8338f-3d42-4b71-8142-d234fbd746cc) | [<img src="assets/readme/sample_2.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/fa6a65a6-e32a-4d64-9a9e-eabb0ebb8c16) |-| A serene night scene in a forested area. [...] The video is a time-lapse, capturing the transition from day to night, with the lake and forest serving as a constant backdrop. | A soaring drone footage captures the majestic beauty of a coastal cliff, [...] The water gently laps at the rock base and the greenery that clings to the top of the cliff. | The majestic beauty of a waterfall cascading down a cliff into a serene lake. [...] The camera angle provides a bird's eye view of the waterfall. |+|森林地区宁静的夜景。 [...] 该视频是一段延时摄影,捕捉了白天到夜晚的转变,湖泊和森林始终作为背景。 | 无人机拍摄的镜头捕捉到了海岸悬崖的壮丽美景,[...] 海水轻轻地拍打着岩石底部和紧贴悬崖顶部的绿色植物。| 瀑布从悬崖上倾泻而下,流入宁静的湖泊,气势磅礴。[...] 摄像机角度提供了瀑布的鸟瞰图。 || [<img src="assets/readme/sample_3.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/64232f84-1b36-4750-a6c0-3e610fa9aa94) | [<img src="assets/readme/sample_4.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/983a1965-a374-41a7-a76b-c07941a6c1e9) | [<img src="assets/readme/sample_5.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/ec10c879-9767-4c31-865f-2e8d6cf11e65) |-| A bustling city street at night, filled with the glow of car headlights and the ambient light of streetlights. [...] | The vibrant beauty of a sunflower field. The sunflowers are arranged in neat rows, creating a sense of order and symmetry. [...] | A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-brown shell [...] |+| 夜晚繁华的城市街道,充满了汽车前灯的光芒和路灯的氛围光。 [...] | 向日葵田的生机勃勃,美不胜收。向日葵整齐排列,给人一种秩序感和对称感。 [...] |宁静的水下场景,一只海龟在珊瑚礁中游动。这只海龟的壳呈绿褐色 [...] |-Videos are downsampled to `.gif` for display. Click for original videos. Prompts are trimmed for display,-see [here](/assets/texts/t2v_samples.txt) for full prompts.+视频经过降采样以.gif用于显示。单击查看原始视频。提示经过修剪以用于显示,请参阅[此处](/assets/texts/t2v_samples.txt)查看完整提示。</details>-## 🔆 New Features/Updates--* 📍 **Open-Sora 1.2** released. Model weights are available [here](#model-weights). See our **[report 1.2](docs/report_03.md)** for more details.-* ✅ Support rectified flow scheduling.-* ✅ Trained our 3D-VAE for temporal dimension compression.-* 📍 **Open-Sora 1.1** released. Model weights are available [here](#model-weights). It is trained on **0s~15s, 144p to 720p, various aspect ratios** videos. See our **[report 1.1](/docs/report_02.md)** for more discussions.-* 🔧 **Data processing pipeline v1.1** is released. An automatic [processing pipeline](#data-processing) from raw videos to (text, video clip) pairs is provided, including scene cutting $\rightarrow$ filtering(aesthetic, optical flow, OCR, etc.) $\rightarrow$ captioning $\rightarrow$ managing. With this tool, you can easily build your video dataset.-* ✅ Improved ST-DiT architecture includes rope positional encoding, qk norm, longer text length, etc.-* ✅ Support training with any resolution, aspect ratio, and duration (including images).-* ✅ Support image and video conditioning and video editing, and thus support animating images, connecting videos, etc.-* 📍 **Open-Sora 1.0** released. Model weights are available [here](#model-weights). With only 400K video clips and 200 H800- days (compared with 152M samples in Stable Video Diffusion), we are able to generate 2s 512×512 videos. See our **[report 1.0](docs/report_01.md)** for more discussions.-* ✅ Three-stage training from an image diffusion model to a video diffusion model. We provide the weights for each- stage.-* ✅ Support training acceleration including accelerated transformer, faster T5 and VAE, and sequence parallelism.- Open-Sora improves **55%** training speed when training on 64x512x512 videos. Details locates- at [acceleration.md](docs/acceleration.md).-* 🔧 **Data preprocessing pipeline v1.0**,- including [downloading](tools/datasets/README.md), [video cutting](tools/scene_cut/README.md),- and [captioning](tools/caption/README.md) tools. Our data collection plan can be found- at [datasets.md](docs/datasets.md).+## 🔆 新功能/更新++* 📍 **Open-Sora 1.2** 发布。模型权重可在[此处](#model-weights)查看。有关更多详细信息,请参阅我们的**[技术报告 v1.2](docs/report_03.md)** 。+* ✅ 支持整流流调度。+* ✅ 训练我们的 3D-VAE 进行时间维度压缩。+* 📍 **Open-Sora 1.1**发布。模型权重可在[此处](#model-weights)获得。它针对**0s~15s、144p 到 720p、各种宽高比**的视频进行训练。有关更多讨论,请参阅我们的**[技术报告 v1.1](/docs/report_02.md)** 。+* 🔧 **数据处理流程** v1.1发布,提供从原始视频到(文本,视频片段)对的自动处理流程,包括场景剪切$\rightarrow$过滤(美学、光流、OCR 等)$\rightarrow$字幕$\rightarrow$管理。使用此工具,您可以轻松构建视频数据集。+* ✅ 改进的 ST-DiT 架构包括 rope 位置编码、qk 范数、更长的文本长度等。+* ✅ 支持任意分辨率、纵横比和时长(包括图像)的训练。+* ✅ 支持图像和视频调节以及视频编辑,从而支持动画图像,连接视频等。+* 📍 **Open-Sora 1.0**发布。模型权重可在[此处](#model-weights)获得。仅使用 400K 视频片段和 200 个 H800 天(相比稳定视频扩散中的 152M 样本),我们就能生成 2s 512×512 视频。有关更多讨论,请参阅我们的**[技术报告 v1.0](docs/report_01.md)**。+* ✅从图像扩散模型到视频扩散模型的三阶段训练。我们为每个阶段提供权重。+* ✅ 支持训练加速,包括加速 Transformer、更快的 T5 和 VAE 以及序列并行。Open-Sora 在 64x512x512 视频上训练时可将训练速度提高**55%**。详细信息位于[训练加速.md](docs/acceleration.md)。+* 🔧 **数据预处理流程 v1.0**,包括 [下载](tools/datasets/README.md), [视频剪辑](tools/scene_cut/README.md), 和 [字幕](tools/caption/README.md) 工具. 我们的数据收集计划可在 [数据集.md](docs/datasets.md)中找到.<details>-<summary>View more</summary>+<summary>查看更多</summary>-* ✅ We find VQ-VAE from [VideoGPT](https://wilson1yan.github.io/videogpt/index.html) has a low quality and thus adopt a- better VAE from [Stability-AI](https://huggingface.co/stabilityai/sd-vae-ft-mse-original). We also find patching in- the time dimension deteriorates the quality. See our **[report](docs/report_01.md)** for more discussions.-* ✅ We investigate different architectures including DiT, Latte, and our proposed STDiT. Our **STDiT** achieves a better- trade-off between quality and speed. See our **[report](docs/report_01.md)** for more discussions.-* ✅ Support clip and T5 text conditioning.-* ✅ By viewing images as one-frame videos, our project supports training DiT on both images and videos (e.g., ImageNet &- UCF101). See [commands.md](docs/commands.md) for more instructions.-* ✅ Support inference with official weights- from [DiT](https://github.com/facebookresearch/DiT), [Latte](https://github.com/Vchitect/Latte),- and [PixArt](https://pixart-alpha.github.io/).-* ✅ Refactor the codebase. See [structure.md](docs/structure.md) to learn the project structure and how to use the- config files.+✅ 我们发现[VideoGPT](https://wilson1yan.github.io/videogpt/index.html)的 VQ-VAE质量较低,因此采用了[Stability-AI](https://huggingface.co/stabilityai/sd-vae-ft-mse-original)中的更好的 VAE 。我们还发现时间维度的修补会降低质量。有关更多讨论,请参阅我们的**[技术报告v1.0](docs/report_01.md)**。+✅ 我们研究了不同的架构,包括 DiT、Latte 和我们提出的 **STDiT**。我们的STDiT在质量和速度之间实现了更好的平衡。请参阅我们的 **[技术报告v1.0](docs/report_01.md)**以了解更多讨论。+✅ 支持剪辑和T5文本调节。+✅ 通过将图像视为单帧视频,我们的项目支持在图像和视频上训练 DiT(例如 ImageNet 和 UCF101)。有关更多说明,请参阅[commands.md](docs/commands.md) 。+✅ 支持使用[DiT](https://github.com/facebookresearch/DiT), [Latte](https://github.com/Vchitect/Latte),+ 和 [PixArt](https://pixart-alpha.github.io/).的官方权重进行推理。+✅ 重构代码库。查看[structure.md](docs/structure.md)以了解项目结构以及如何使用配置文件。</details>-### TODO list sorted by priority+### 按优先级排序的 TODO 列表<details>-<summary>View more</summary>+<summary>查看更多</summary>-* [x] Training Video-VAE and adapt our model to new VAE.-* [x] Scaling model parameters and dataset size.-* [x] Incoporate a better scheduler (rectified flow).-* [x] Evaluation pipeline.-* [x] Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, etc.). See [the dataset](/docs/datasets.md) for more information-* [x] Support image and video conditioning.-* [x] Support variable aspect ratios, resolutions, durations.+* [x] 训练视频 VAE 并使我们的模型适应新的 VAE+* [x] 缩放模型参数和数据集大小+* [x] 纳入更好的调度程序(整流流程)+* [x] 评估流程+* [x] 完成数据处理流程(包括密集光流、美学评分、文本-图像相似度等)。有关更多信息,请参阅[数据集](/docs/datasets.md)+* [x] 支持图像和视频调节+* [x] 支持可变的纵横比、分辨率和持续时间</details>-## Contents+## 内容-* [Installation](#installation)-* [Model Weights](#model-weights)-* [Gradio Demo](#gradio-demo)-* [Inference](#inference)-* [Data Processing](#data-processing)-* [Training](#training)-* [Evaluation](#evaluation)-* [Contribution](#contribution)-* [Citation](#citation)-* [Acknowledgement](#acknowledgement)+* [安装](#安装)+* [模型权重](#模型权重)+* [Gradio演示](#gradio演示)+* [推理](#推理)+* [数据处理](#数据处理)+* [训练](#训练)+* [评估](#评估)+* [贡献](#贡献)+* [引用](#引用)+* [致谢](#致谢)-Other useful documents and links are listed below.+下面列出了其他有用的文档和链接。-* Report: [report 1.2](docs/report_03.md), [report 1.1](docs/report_02.md), [report 1.0](docs/report_01.md), [acceleration.md](docs/acceleration.md)-* Repo structure: [structure.md](docs/structure.md)-* Config file explanation: [config.md](docs/config.md)+* 报告: [技术报告 v1.2](docs/report_03.md), [技术报告 v1.1](docs/report_02.md), [技术报告 v1.0](docs/report_01.md), [训练加速.md](docs/acceleration.md)+* Repo 结构: [结构.md](docs/structure.md)+* 配置文件说明: [config.md](docs/config.md)* Useful commands: [commands.md](docs/commands.md)-* Data processing pipeline and dataset: [datasets.md](docs/datasets.md)-* Each data processing tool's README: [dataset conventions and management](/tools/datasets/README.md), [scene cutting](/tools/scene_cut/README.md), [scoring](/tools/scoring/README.md), [caption](/tools/caption/README.md)+* 数据处理管道和数据集: [datasets.md](docs/datasets.md)+* 每个数据处理工具的 README: [dataset conventions and management](/tools/datasets/README.md), [scene cutting](/tools/scene_cut/README.md), [scoring](/tools/scoring/README.md), [caption](/tools/caption/README.md)* Evaluation: [eval](/eval/README.md)* Gallery: [gallery](https://hpcaitech.github.io/Open-Sora/)-## Installation+## 安装-### Install from Source+### 从源头安装For CUDA 12.1, you can install the dependencies with the following commands. Otherwise, please refer to [Installation](docs/installation.md) for more instructions on different cuda version, and additional dependency for data preprocessing.