Open-Sora: Democratizing Efficient Video Production for All
Python
open-sora v1.3 code upload (#786)
Zheng Zangwei (Alex Zheng)f1c6b8b88e38
diff --git a/LICENSE b/LICENSEindex e7f2aa2..67a64c0 100644--- a/LICENSE+++ b/LICENSE@@ -678,3 +678,29 @@ Copyright 2024 HPC-AI Technology Inc. All rights reserved.WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.++ 6. Asymmetric_magvitv2 (https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/LICENSE)++ MIT License++ Copyright (c) 2024 bornfly-detachment++ Permission is hereby granted, free of charge, to any person obtaining a copy+ of this software and associated documentation files (the "Software"), to deal+ in the Software without restriction, including without limitation the rights+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+ copies of the Software, and to permit persons to whom the Software is+ furnished to do so, subject to the following conditions:++ The above copyright notice and this permission notice shall be included in all+ copies or substantial portions of the Software.++ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+ SOFTWARE.++diff --git a/README.md b/README.mdindex 38e4797..901ce78 100644--- a/README.md+++ b/README.md@@ -34,12 +34,13 @@ With Open-Sora, our goal is to foster innovation, creativity, and inclusivity wi<img src="https://github.com/hpcaitech/public_assets/blob/main/colossalai/img/1.gif" width="850" /></a></div>-## 📰 News+## 📰 News+- **[2024.01.20]** 🔥 We released **Open-Sora 1.3**. With the upgraded VAE and Transformer architecture, the quality of our generated videos has been greatly improved 🚀. [[checkpoints]](#open-sora-13-model-weights) [[report]](/docs/report_04.md) [[demo]](https://huggingface.co/spaces/hpcai-tech/open-sora)- **[2024.12.23]** 🔥 [The development cost of video generation models has saved by 50%! Open-source solutions are now available with H200 GPU vouchers](https://company.hpc-ai.com/blog/the-development-cost-of-video-generation-models-has-saved-by-50-open-source-solutions-are-now-available-with-h200-gpu-vouchers) [[code]](https://github.com/hpcaitech/Open-Sora/blob/main/scripts/train.py) [[vouchers]](https://colossalai.org/zh-Hans/docs/get_started/bonus/)-- **[2024.06.17]** 🔥 We released **Open-Sora 1.2**, which includes **3D-VAE**, **rectified flow**, and **score condition**. The video quality is greatly improved. [[checkpoints]](#open-sora-10-model-weights) [[report]](/docs/report_03.md) [[blog]](https://hpc-ai.com/blog/open-sora-from-hpc-ai-tech-team-continues-open-source-generate-any-16-second-720p-hd-video-with-one-click-model-weights-ready-to-use)+- **[2024.06.17]** 🔥 We released **Open-Sora 1.2**, which includes **3D-VAE**, **rectified flow**, and **score condition**. The video quality is greatly improved. [[checkpoints]](#open-sora-12-model-weights) [[report]](/docs/report_03.md) [[blog]](https://hpc-ai.com/blog/open-sora-from-hpc-ai-tech-team-continues-open-source-generate-any-16-second-720p-hd-video-with-one-click-model-weights-ready-to-use)- **[2024.04.25]** 🤗 We released the [Gradio demo for Open-Sora](https://huggingface.co/spaces/hpcai-tech/open-sora) on Hugging Face Spaces.-- **[2024.04.25]** We released **Open-Sora 1.1**, which supports **2s~15s, 144p to 720p, any aspect ratio** text-to-image, **text-to-video, image-to-video, video-to-video, infinite time** generation. In addition, a full video processing pipeline is released. [[checkpoints]]() [[report]](/docs/report_02.md)+- **[2024.04.25]** We released **Open-Sora 1.1**, which supports **2s~15s, 144p to 720p, any aspect ratio** text-to-image, **text-to-video, image-to-video, video-to-video, infinite time** generation. In addition, a full video processing pipeline is released. [[checkpoints]](#open-sora-11-model-weights) [[report]](/docs/report_02.md)- **[2024.03.18]** We released **Open-Sora 1.0**, a fully open-source project for video generation.Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with<a href="https://github.com/hpcaitech/ColossalAI"><img src="assets/readme/colossal_ai.png" width="8%" ></a>@@ -53,11 +54,23 @@ With Open-Sora, our goal is to foster innovation, creativity, and inclusivity wi🔥 You can experience Open-Sora on our [🤗 Gradio application on Hugging Face](https://huggingface.co/spaces/hpcai-tech/open-sora). More samples and corresponding prompts are available in our [Gallery](https://hpcaitech.github.io/Open-Sora/).+| **5s 720×1280** | **5s 720×1280** | **5s 720×1280** |+|------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|+| [<img src="assets/demo/open_sora_v1.3/demo_tomato.gif" width="">](https://streamable.com/e/r0imrp?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_fisherman.gif" width="">](https://streamable.com/e/hfvjkh?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_girl2.gif" width="">](https://streamable.com/e/kutmma?quality=highest&autoplay=1) |+| [<img src="assets/demo/open_sora_v1.3/demo_grape.gif" width="">](https://streamable.com/e/osn1la?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_mushroom.gif" width="">](https://streamable.com/e/l1pzws?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_parrot.gif" width="">](https://streamable.com/e/2vqari?quality=highest&autoplay=1) |+| [<img src="assets/demo/open_sora_v1.3/demo_trans.gif" width="">](https://streamable.com/e/1in7d6?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_bear.gif" width="">](https://streamable.com/e/e9bi4o?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_futureflower.gif" width="">](https://streamable.com/e/09z7xi?quality=highest&autoplay=1) |+| [<img src="assets/demo/open_sora_v1.3/demo_fire.gif" width="">](https://streamable.com/e/16c3hk?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_man.gif" width="">](https://streamable.com/e/wi250w?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_black.gif" width="">](https://streamable.com/e/vw5b64?quality=highest&autoplay=1) |+++<details>+<summary>OpenSora 1.2 Demo</summary>+| **4s 720×1280** | **4s 720×1280** | **4s 720×1280** ||------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|| [<img src="assets/demo/v1.2/sample_0013.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/7895aab6-ed23-488c-8486-091480c26327) | [<img src="assets/demo/v1.2/sample_1718.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/20f07c7b-182b-4562-bbee-f1df74c86c9a) | [<img src="assets/demo/v1.2/sample_0087.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/3d897e0d-dc21-453a-b911-b3bda838acc2) || [<img src="assets/demo/v1.2/sample_0052.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/644bf938-96ce-44aa-b797-b3c0b513d64c) | [<img src="assets/demo/v1.2/sample_1719.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/272d88ac-4b4a-484d-a665-8d07431671d0) | [<img src="assets/demo/v1.2/sample_0002.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/ebbac621-c34e-4bb4-9543-1c34f8989764) || [<img src="assets/demo/v1.2/sample_0011.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/a1e3a1a3-4abd-45f5-8df2-6cced69da4ca) | [<img src="assets/demo/v1.2/sample_0004.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/d6ce9c13-28e1-4dff-9644-cc01f5f11926) | [<img src="assets/demo/v1.2/sample_0061.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/561978f8-f1b0-4f4d-ae7b-45bec9001b4a) |+</details><details><summary>OpenSora 1.1 Demo</summary>@@ -93,12 +106,11 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.</details>## 🔆 New Features/Updates--- 📍 **Open-Sora 1.2** released. Model weights are available [here](#model-weights). See our **[report 1.2](/docs/report_03.md)** for more details.-- ✅ Support rectified flow scheduling.-- ✅ Support more conditioning including fps, aesthetic score, motion strength and camera motion.-- ✅ Trained our 3D-VAE for temporal dimension compression.-- 📍 **Open-Sora 1.1** released. Model weights are available [here](#model-weights). It is trained on **0s~15s, 144p to 720p, various aspect ratios** videos. See our **[report 1.1](/docs/report_02.md)** for more discussions.+- 📍 **Open-Sora 1.3** released. Model weights are available [here](#model-weights). See our **[report 1.3](docs/report_04.md)** for more infomation.+ - ✅ Upgraded model architecture, delivering enhanced performance and efficiency.+ - ✅ Incorporated more high-quality training data, significantly improving overall results.+- 📍 **Open-Sora 1.2** released. Model weights are available [here](#model-weights). See our **[report 1.2](docs/report_03.md)** for more details.+- 📍 **Open-Sora 1.1** released. Model weights are available [here](#model-weights). It is trained on **0s~15s, 144p to 720p, various aspect ratios** videos. See our **[report 1.1](docs/report_02.md)** for more discussions.- 🔧 **Data processing pipeline v1.1** is released. An automatic [processing pipeline](#data-processing) from raw videos to (text, video clip) pairs is provided, including scene cutting $\rightarrow$ filtering(aesthetic, optical flow, OCR, etc.) $\rightarrow$ captioning $\rightarrow$ managing. With this tool, you can easily build your video dataset.<details>@@ -165,7 +177,8 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.Other useful documents and links are listed below.-- Report: each version is trained from a image base separately (not continuously trained), while a newer version will incorporate the techniques from the previous version.+- Report: each version is trained from a image base seperately (not continuously trained), while a newer version will incorporate the techniques from the previous version.+ - [report 1.3](docs/report_04.md): shift-window attention, unified spatial-temporal VAE, etc.- [report 1.2](docs/report_03.md): rectified flow, 3d-VAE, score condition, evaluation, etc.- [report 1.1](docs/report_02.md): multi-resolution/length/aspect-ratio, image/video conditioning/editing, data preprocessing, etc.- [report 1.0](docs/report_01.md): architecture, captioning, etc.@@ -235,8 +248,24 @@ docker run -ti --gpus all -v .:/workspace/Open-Sora opensora## Model Weights+### Open-Sora 1.3 Model Weights++| Model | Model Size | Data | #iterations | Batch Size | URL |+|-----------|------------|------|-------------|------------|---------------------------------------------------------------|+| Diffusion | 1.1B | 60M | 70k | Dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v4) |+| Diffusion-360p | 1.1B | 60M | 70k | Dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v4-360p) |+| Diffusion-i2v | 1.1B | 60M | 70k | Dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3-i2v) |+| VAE | 259M | 3M | 1M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.3) |++See our **[report 1.3](docs/report_04.md)** for more infomation. Weight will be automatically downloaded when you run the inference script.++> For users from mainland China, try `export HF_ENDPOINT=https://hf-mirror.com` to successfully download the weights.+### Open-Sora 1.2 Model Weights+<details>+<summary>View more</summary>+| Model | Model Size | Data | #iterations | Batch Size | URL ||-----------|------------|------|-------------|------------|---------------------------------------------------------------|| Diffusion | 1.1B | 30M | 70k | Dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3) |@@ -245,6 +274,7 @@ docker run -ti --gpus all -v .:/workspace/Open-Sora opensoraSee our **[report 1.2](docs/report_03.md)** for more information. Weight will be automatically downloaded when you run the inference script.> For users from mainland China, try `export HF_ENDPOINT=https://hf-mirror.com` to successfully download the weights.+</details>### Open-Sora 1.1 Model Weights@@ -314,13 +344,12 @@ In the Gradio application, the basic options are as follows:The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button (scroll down if you cannot find). The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.-Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 80G H100 GPU, the generation speed (with `num_sampling_step=30`) and peak memory usage is:+Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 140G H200 GPU, the generation speed (with `num_sampling_step=30`) and peak memory usage is:-| | Image | 2s | 4s | 8s | 16s |-|------|---------|----------|-----------|-----------|-----------|-| 360p | 3s, 24G | 18s, 27G | 31s, 27G | 62s, 28G | 121s, 33G |-| 480p | 2s, 24G | 29s, 31G | 55s, 30G | 108s, 32G | 219s, 36G |-| 720p | 6s, 27G | 68s, 41G | 130s, 39G | 260s, 45G | 547s, 67G |+| | Image |49 Frames | 65 Frames | 81 Frames | 97 Frames |113 Frames |+|------|----------|----------|-----------|-----------|-----------|-----------|+| 360p | 10s, 23G | 16s, 23G | 21s, 24G | 27s, 24G | 33s, 25G | 39s, 25G |+| 720p | 13s, 24G | 61s, 28G | 83s, 29G | 106s, 31G | 130s, 33G | 157s, 35G |Note that besides text to video, you can also use **image to video generation**. You can upload an image and then click the "**Generate video**" button to generate a video with the image as the first frame. Or you can fill in the text prompt and click the "**Generate image**" button to generate an image with the text prompt, and then click the "**Generate video**" button to generate a video with the image generated with the same model.@@ -328,7 +357,7 @@ Note that besides text to video, you can also use **image to video generation**.Then you can specify more options, including "**Motion Strength**", "**Aesthetic**" and "**Camera Motion**". If "Enable" not checked or the choice is "none", the information is not passed to the model. Otherwise, the model will generate videos with the specified motion strength, aesthetic score, and camera motion.-For the **aesthetic score**, we recommend using values higher than 6. For **motion strength**, a smaller value will lead to a smoother but less dynamic video, while a larger value will lead to a more dynamic but likely more blurry video. Thus, you can try without it and then adjust it according to the generated video. For the **camera motion**, sometimes the model cannot follow the instruction well, and we are working on improving it.+For the **aesthetic score**, we recommend using values 'excellent'. For **motion strength**, a smaller value will lead to a smoother but less dynamic video, while a larger value will lead to a more dynamic but likely more blurry video. Thus, you can try without it and then adjust it according to the generated video. For the **camera motion**, sometimes the model cannot follow the instruction well, and we are working on improving it.You can also adjust the "**Sampling steps**", this is directly related to the generation speed as it is the number of denoising. A number smaller than 30 usually leads to a poor generation results, while a number larger than 100 usually has no significant improvement. The "**Seed**" is used for reproducibility, you can set it to a fixed number to generate the same video. The "**CFG Scale**" controls how much the model follows the text prompt, a smaller value will lead to a more random video, while a larger value will lead to a more text-following video (7 is recommended).@@ -336,7 +365,57 @@ For more advanced usage, you can refer to [Gradio README](./gradio/README.md#adv## Inference+### Open-Sora 1.3 Command Line Inference+The basic command line inference is as follows:++```bash+# text to video+python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 \+ --prompt "a beautiful waterfall"+```++You can add more options to the command line to customize the generation.++```bash+python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 --num-sample 1 \+ --aes "very good" --flow "fair" \+ --prompt "a beautiful waterfall"+```++For image to video generation and other functionalities, see [here](docs/commands.md) for more instructions.++### Sequence Parallelism Inference++To enable sequence parallelism, you need to use `torchrun` to run the inference script. The following command will run the inference with 2 GPUs.++```bash+# text to video+CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 \+ --prompt "a beautiful waterfall"+```++### GPT-4o Prompt Refinement++We find that GPT-4o can refine the prompt and improve the quality of the generated video. With this feature, you can also use other language (e.g., Chinese) as the prompt. To enable this feature, you need prepare your openai api key in the environment:++```bash+export OPENAI_API_KEY=YOUR_API_KEY+```++Then you can inference with `--llm-refine True` to enable the GPT-4o prompt refinement, or leave prompt empty to get a random prompt generated by GPT-4o.++```bash+python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --llm-refine True \+ --prompt "a beautiful waterfall"+```+### Open-Sora 1.2 Command Line Inference+<details>+<summary>View more</summary>The basic command line inference is as follows:@@ -395,6 +474,8 @@ python scripts/inference.py configs/opensora-v1-2/inference/sample.py \--num-frames 4s --resolution 720p --llm-refine True```+</details>+### Open-Sora 1.1 Command Line Inference<details>@@ -455,8 +536,22 @@ Also check out the [datasets](docs/datasets.md) we use.## Training+### Open-Sora 1.3 Training++```bash+# one node+torchrun --standalone --nproc_per_node 8 scripts/train.py \+ configs/opensora-v1-3/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT+# multiple nodes+colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py \+ configs/opensora-v1-3/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT+```+### Open-Sora 1.2 Training+<details>+<summary>View more</summary>+The training process is same as Open-Sora 1.1.```bash@@ -470,6 +565,7 @@ torchrun --standalone --nproc_per_node 8 scripts/train.py \colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py \configs/opensora-v1-2/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT```+</details>### Open-Sora 1.1 Training@@ -524,7 +620,7 @@ We support evaluation based on:- VBench-i2v score- Batch generation for human evaluation-All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details. Our [report](/docs/report_03.md#evaluation) also provides more information about the evaluation during training. The following table shows Open-Sora 1.2 greatly improves Open-Sora 1.0.+All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details. Our [report](/docs/report_04.md#evaluation) also provides more information about the evaluation during training. The following table shows Open-Sora 1.3 greatly improves Open-Sora 1.2.| Model | Total Score | Quality Score | Semantic Score ||----------------|-------------|---------------|----------------|@@ -533,6 +629,24 @@ All the evaluation code is released in `eval` folder. Check the [README](/eval/R## VAE Training & Evaluation+### Open-Sora 1.3 VAE Training+We implement and train a unified spatial-temporal VAE for better compress videos as well as preserve details. For more detailes, refer to [Open-Sora 1.3 VAE Documentation](docs/vae_v1_3.md). Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.+```bash+## Train VAE with 8 GPUs+colossalai run --hostfile hostfile --nproc_per_node 8 scripts/train_opensoravae_v1_3.py configs/vae_v1_3/train/video_16z.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT --wandb True > logs/train_opensoravae_v1_3.log 2>&1 &+```++### Evaluate VAE performance+You need to run VAE inference first to generate the videos, then calculate scores on the generated videos:+```bash+VID_PATH=YOUR_CSV_PATH CUDA_VISIBLE_DEVICES=0 bash eval/vae/launch.sh pretrained_models/OpenSoraVAE_V1_3/model.pt+```++### Open-Sora 1.2 VAE Training++<details>+<summary>View more</summary>+We train a VAE pipeline that consists of a spatial VAE followed by a temporal VAE.For more details, refer to [VAE Documentation](docs/vae.md).Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.@@ -562,6 +676,8 @@ torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py conpython eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips```+</details>+## ContributionThanks goes to these wonderful contributors:diff --git a/assets/texts/celebrities_interaction.txt b/assets/texts/celebrities_interaction.txtnew file mode 100644index 0000000..fa14926--- /dev/null+++ b/assets/texts/celebrities_interaction.txt@@ -0,0 +1,5 @@+Donald Trump and Kamala Harris are hugging each other and turning in rounds.+Donald Trump and Kamala Harris are fighting each other.+Kamala Harris slapped Donald Trump.+Donald Trump is kissing a random girl.+Donald Trump and Kamala Harris are kissing.diff --git a/assets/texts/i2v/celebrities_interaction.txt b/assets/texts/i2v/celebrities_interaction.txtnew file mode 100644index 0000000..522ff41--- /dev/null+++ b/assets/texts/i2v/celebrities_interaction.txt@@ -0,0 +1,14 @@+Harris and Trump shakes hands.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_1.jpeg"}+The female and male politician shakes hands.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_1.jpeg"}+Harris and Trump shakes hands.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_2.jpeg"}+The female and male politician shakes hands.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_2.jpeg"}+Harris and Trump shakes hands.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_3.jpeg"}+The female and male politician shakes hands.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_3.jpeg"}+As Harris and Trump shakes hands, they break into hugging each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_1.jpeg"}+As the female and male politician shakes hands, they break into hugging each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_1.jpeg"}+As Harris and Trump shakes hands, they break into hugging each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_2.jpeg"}+As the female and male politician shakes hands, they break into hugging each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_2.jpeg"}+As Harris and Trump shakes hands, they break into hugging each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_3.jpeg"}+As the female and male politician shakes hands, they break into hugging each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_shake_3.jpeg"}+As the woman speaks, the man from the back steps in front and hugs her.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_speech_1.jpeg"}+As the man on the left and woman on the left speak, they suddenly hug each other.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/harris_trump_speech_2.jpeg"}diff --git a/assets/texts/i2v/multi_human.txt b/assets/texts/i2v/multi_human.txtnew file mode 100644index 0000000..c1c1101--- /dev/null+++ b/assets/texts/i2v/multi_human.txt@@ -0,0 +1,10 @@+In a modern kitchen, two chefs are cutting onions on red plastic cutting boards placed on metallic work benches. Both chefs wear black working attire, black caps, and black gloves, meticulously chopping the onions into small pieces. Between them are large bowls filled with boiled, de-shelled eggs. In the background, neatly stacked raw eggs can be seen. A man, likely the supervisor, stands facing away, looking out an open-glass door while holding papers. The scene captures a moment of focused kitchen activity without transitions.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/chef_cut_onion.png"}+Three guys are standing on a side road in front of a Chinese rural village, engaging in an informal interview. The guy on the left, wearing a white T-shirt and blue jeans, is facing the other two, gesturing with his hands as he talks. The middle guy, dressed in a black polo shirt, is nodding and listening attentively. The rightmost guy, in a blue polo shirt, occasionally glances at the camera. Behind them is a large gray stone stacked on a red square pillar, resembling a landmark, with shrubs further in the background. The video focuses on the interaction among the three characters.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/rural_interview.png"}+The video captures a scene in a spacious, modern indoor sheep farm. A group of people stands in the middle passage, observing a black-and-white lamb. Fences dividing sheep into different areas are visible on the left side. The central figure is holding the lamb, with a man on the right trying to pat it, and another man on the left looking at it. A woman in a red-and-white dress is smiling at the lamb from a sideways angle, while another man's partial figure can be seen in the distance. All the men are wearing white long coats, and the lamb is kicking its hind legs while being held firmly by the central figure.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/sheep_farm.png"}+Two singers are walking slowly and singing on a stage during a performance. The male singer, wearing a beige scarf and white top, holds a microphone and sings attentively while holding the female singer's hand. The female singer, dressed in a white dress with long brown curly hair, smiles glamorously. The stage background features steel frames and is otherwise unclear, with an orange light in the background. The video is shot from the front at a slightly low angle, capturing both singers as they face slightly towards the left. The background is dark, but the spotlight illuminates the singers, making their faces and expressions clearly visible.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/couple_singers.png"}+A distinguished musical performance is in progress at a classical concert. In the center, a man in a formal navy coat, white shirt, and cravat holds a violin and addresses the audience. Behind him, a full orchestra of seated musicians prepares with their string and percussion instruments, some smiling and looking towards the central figure. The setting is elegant, with decorative music stands featuring intricate gold designs, indicating a prestigious event. The atmosphere exudes grandeur and professionalism, characteristic of a high-caliber classical music performance. The scene captures a moment of anticipation and readiness before the music begins.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/pre_ochestra.png"}+A dynamic scene at a musical concert featuring two women performing on stage. The woman on the left, with blonde wavy hair, is passionately singing into a golden microphone. She is dressed in a sparkly, embellished yellow outfit with dark glittering details, her eyes partially closed, and her hand raised expressively. The woman on the right, shown in profile, is wearing a sleeveless, glittering dark dress. Her long, wavy hair flows over her shoulders, and she appears composed, possibly preparing for her part or listening attentively. The background, slightly blurred, reveals musicians or orchestra members, enhancing the live performance atmosphere with focused and intense energy.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/dual_female_singers.png"}+A lively moment on a talk show set with the focus on a guest and the host. On the left, the guest, a woman in an elegant black outfit with sparkling details, is gesturing expressively with both hands as she passionately shares a story. Her hair is tied back, and she wears a necklace, adding to her stylish appearance. On the right, the host, in a suit and tie, sits behind a wooden desk, leaning slightly forward with a smile, appearing attentive and engaged. The backdrop showcases a nighttime city skyline, typical of late-night talk shows. The scene captures a warm interaction, highlighting the positive and relaxed mood of their conversation.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/talk_show.png"}+A vibrant scene featuring a talented girl group performing a synchronized dance routine against a backdrop screen displaying soft clouds and a moonlit sky. The group members are dressed in stylish pastel outfits and knee-high boots, radiating energy and confidence. Their dynamic choreography and intricate formations showcase their skill and dedication. Expressive gestures and movements highlight their commitment to the craft. The combination of music, dance, and visual elements creates an electrifying atmosphere that captivates the audience, retaining focus on their performance without scene transitions.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/girl_band_dance.png"}+A scene on a set of urban stairs where two girls are sitting, both intently focused on their mobile phones. The girl on the left is adjusting her hair with one hand while holding her phone with a protective case in the other, capturing a casual yet attentive demeanor. The girl on the right, dressed in a stylish outfit, is completely absorbed in her device. The background shows people walking by, emphasizing the bustling urban environment. Despite the activity around them, the girls are engaged in their own digital worlds. Their body language reflects a blend of relaxation and concentration as they interact with their screens.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/street_girls.png"}+A young girl in a colorful swimsuit and a cute sun hat is playfully covering a boy with sand on a beach. She is holding a small plastic shovel and has a joyful expression, clearly enjoying the activity. The boy, lying on a beach towel in a blue swimsuit and cap, appears relaxed and enjoying the sun. The sandy beach stretches out behind them, with the ocean visible in the background. The scene captures a moment of fun and carefree play, typical of a day at the beach. There is no scene transition, focusing on this single, joyous moment.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/kids_beach.png"}diff --git a/assets/texts/i2v/prompts_head.txt b/assets/texts/i2v/prompts_head.txtnew file mode 100644index 0000000..cd9e041--- /dev/null+++ b/assets/texts/i2v/prompts_head.txt@@ -0,0 +1,20 @@+An astronaut transitions from being underwater to floating in space.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/Astronaut_1.jpg"}+The Eiffel Tower transitions from a daytime scene to a nighttime scene.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/Effiel_1.jpeg"}+A woman is seamlessly moving through different scenes with a magical effect.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/MorphCharacters_1.jpg"}+A small kitten is playing in the snow.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/cat1.png"}+A person is giving water to a dog, and the dog tilts its head back to drink.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/dog1.png"}+A car is speeding down a narrow country road in a field.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/car1.jpeg"}+Three daffodils quietly bloom, their delicate petals unfurling in serene harmony.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/flower1.png"}+A butterfly, attracted by the fragrance of the flowers, flew over and landed on the scent.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/butterfly1.png"}+A cyclist rides along the waves as they crash against the shore.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/cyclist1.png"}+The earth is rotating.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/earth1.png"}+The bread is baking in the oven, slowly expanding as it rises.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/bread1.png"}+The little boy is celebrating his birthday and blowing out the candles.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/candle1.png"}+The airplane is landing at dusk.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/airplane1.png"}+Sharks swim in a circle.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/sea1.png"}+The man is walking along the path.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/walking1.png"}+The cat slipped on a pair of sunglasses.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/catwglass1.png"}+Attractive confident guy happily showing you gesture and winking on camera isolated.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/man1.png"}+Mother covering sleeping daughter with blanket.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/mother1.png"}+Pink Chocolate bar exploding midair against blue background.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/chocolate1.png"}+A guy is freeskate rollerblading.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/freeskate_1.png"}diff --git a/assets/texts/i2v/prompts_human_i2v_head.txt b/assets/texts/i2v/prompts_human_i2v_head.txtnew file mode 100644index 0000000..b7c50ea--- /dev/null+++ b/assets/texts/i2v/prompts_human_i2v_head.txt@@ -0,0 +1,10 @@+The young child with short hair, dressed in a vibrant green and white t-shirt, sits peacefully in a meditative pose, her hands gently clasped together. As she breathes deeply, a subtle sway in her posture hints at a gentle motion, like a leaf dancing in a light breeze. The serene expression on her face remains undisturbed, embodying a tranquil harmony with the natural world around her.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/girl_pray_1.jpeg"}+The video captures a charged moment at a political rally, with trump in the center raising his fist triumphantly against a backdrop of a waving American flag. He's flanked by serious-looking men in dark suits and sunglasses, suggesting a high-profile campaign event with security present. The scene exudes an atmosphere of patriotic fervor and political enthusiasm, with trump's gesture and the prominent flag symbolizing themes of American leadership and national pride.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/trump_1.jpeg"}+In this charming outdoor scene, a jubilant toddler revels in a makeshift bathtub on what appears to be a sunny deck or patio. The little one, adorned with a straw hat, sits gleefully in a large white tub, surrounded by potted plants and garden furniture. With mouth wide open in an expression of pure joy, the child seems to be thoroughly enjoying their unconventional bath time, complete with a blue beach ball as a bath toy. The video captures a moment of carefree childhood bliss, blending the fun of bath time with the pleasures of outdoor play on a beautiful day.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/baby_1.jpeg"}+Under the spotlight, Taylor Swift commands the stage with her signature blend of charm and musical prowess, her golden curls catching the light as she leans into the microphone. Her fingers dance across the strings of her ornate acoustic guitar, coaxing out melodies that have the audience hanging on every note. With a sparkle in her eye and a slight smile playing on her lips, she pours her heart into the performance, her voice soaring through the venue and weaving a spell of nostalgia and raw emotion that leaves the crowd breathless and yearning for more.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/swift_1.jpeg"}+Detective Chen prowled the shadowy streets of the city, his keen eyes scanning for clues behind the shield of his reflective aviators. The leather jacket and woolen scarf spoke of a man prepared for both action and the chill of long stakeouts, his composed demeanor masking the razor-sharp mind piecing together the puzzle of his latest case. As he leaned against the weathered stone wall, blending seamlessly into the urban landscape, Chen's senses remained on high alert, ready to spring into action at the slightest hint of his elusive quarry.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/huge_1.jpeg"}+In this brief black and white video clip, Albert Einstein sits at his desk, his piercing gaze fixed intently on a point off-camera. As the scene unfolds, Einstein makes a funny face.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/einstein_1.jpeg"}+In the video clip, a graceful empress, adorned in a golden robe, gently hands over a bamboo scroll to the young emperor, who is deep in thought. The emperor looks up, meeting her gaze with a soft smile, as they exchange a quiet moment of understanding in the grand, candle-lit royal chamber. Behind them, intricately carved wooden panels display ancient calligraphy, creating an atmosphere of wisdom and elegance as they prepare to discuss matters of state.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/royal_1.jpeg"}+In the video clip, the young emperor sits confidently on an intricately carved throne, his expression calm yet commanding as he reads a royal decree. The golden light from nearby lanterns illuminates the rich patterns on his robe, reflecting his authority and the weight of the decisions he must make. Behind him, a backdrop of ancient calligraphy hints at the wisdom of past dynasties, as he prepares to address his court with a decisive proclamation.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/ch_king_1.jpeg"}+In the video clip, Cheng Long stands confidently in front of an iconic cathedral, his fists raised in a poised stance, ready for action. The camera zooms in as he smiles, exuding both calm and determination, before swiftly transitioning into a series of dynamic, graceful moves. With the towering spires of the cathedral in the background, his martial arts sequence blends seamlessly with the historic architecture, creating a striking contrast between tradition and modern agility.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/chenglong_1.jpeg"}+The camera slowly pans around the figure in traditional attire, revealing intricate details of the red garment and gold embroidery. As the person turns, their black hat casts a dramatic shadow against the dimly lit backdrop, adding a mysterious aura to the scene. The soft sound of traditional music begins to play, transporting viewers into a captivating journey through cultural heritage and historical elegance.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/lizhongshua_1.jpeg"}diff --git a/assets/texts/i2v/prompts_loop.txt b/assets/texts/i2v/prompts_loop.txtnew file mode 100644index 0000000..1d3eddf--- /dev/null+++ b/assets/texts/i2v/prompts_loop.txt@@ -0,0 +1,20 @@+An astronaut transitions from being underwater to floating in space.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/Astronaut_1.jpg;/mnt/jfs-hdd2/sora/data/i2v_test_img/Astronaut_2.jpg"}+The Eiffel Tower transitions from a daytime scene to a nighttime scene.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/Effiel_1.jpeg;/mnt/jfs-hdd2/sora/data/i2v_test_img/Effiel_2.jpeg"}+A woman is seamlessly moving through different scenes with a magical effect.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/MorphCharacters_1.jpg;/mnt/jfs-hdd2/sora/data/i2v_test_img/MorphCharacters_2.jpg"}+A small kitten is playing in the snow.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/cat1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/cat2.png"}+A person is giving water to a dog, and the dog tilts its head back to drink.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/dog1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/dog2.png"}+A car is speeding down a narrow country road in a field.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/car1.jpeg;/mnt/jfs-hdd2/sora/data/i2v_test_img/car2.jpeg"}+Three daffodils quietly bloom, their delicate petals unfurling in serene harmony.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/flower1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/flower2.png"}+A butterfly, attracted by the fragrance of the flowers, flew over and landed on the scent.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/butterfly1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/butterfly2.png"}+A cyclist rides along the waves as they crash against the shore.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/cyclist1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/cyclist2.png"}+The earth is rotating.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/earth1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/earth2.png"}+The bread is baking in the oven, slowly expanding as it rises.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/bread1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/bread2.png"}+The little boy is celebrating his birthday and blowing out the candles.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/candle1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/candle2.png"}+The airplane is landing at dusk.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/airplane1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/airplane2.png"}+Sharks swim in a circle.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/sea1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/sea2.png"}+The man is walking along the path.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/walking1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/walking2.png"}+The cat slipped on a pair of sunglasses.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/catwglass1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/catwglass2.png"}+Attractive confident guy happily showing you gesture and winking on camera isolated.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/man1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/man2.png"}+Mother covering sleeping daughter with blanket.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/mother1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/mother2.png"}+Pink Chocolate bar exploding midair against blue background.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/chocolate1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/chocolate2.png"}+A guy is freeskate rollerblading.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/freeskate_1.png;/mnt/jfs-hdd2/sora/data/i2v_test_img/freeskate_2.png"}diff --git a/assets/texts/i2v/prompts_ori.txt b/assets/texts/i2v/prompts_ori.txtnew file mode 100644index 0000000..bca1097--- /dev/null+++ b/assets/texts/i2v/prompts_ori.txt@@ -0,0 +1,20 @@+An astronaut transitions from being underwater to floating in space.+The Eiffel Tower transitions from a daytime scene to a nighttime scene.+A woman is seamlessly moving through different scenes with a magical effect.+A small kitten is playing in the snow.+A person is giving water to a dog, and the dog tilts its head back to drink.+A car is speeding down a narrow country road in a field.+Three daffodils quietly bloom, their delicate petals unfurling in serene harmony.+A butterfly, attracted by the fragrance of the flowers, flew over and landed on the scent.+A cyclist rides along the waves as they crash against the shore.+The earth is rotating.+The bread is baking in the oven, slowly expanding as it rises.+The little boy is celebrating his birthday and blowing out the candles.+The airplane is landing at dusk.+Sharks swim in a circle.+The man is walking along the path.+The cat slipped on a pair of sunglasses.+Attractive confident guy happily showing you gesture and winking on camera isolated.+Mother covering sleeping daughter with blanket.+Pink Chocolate bar exploding midair against blue background.+A guy is freeskate rollerblading.diff --git a/assets/texts/i2v/prompts_tail.txt b/assets/texts/i2v/prompts_tail.txtnew file mode 100644index 0000000..5ad6540--- /dev/null+++ b/assets/texts/i2v/prompts_tail.txt@@ -0,0 +1,20 @@+An astronaut transitions from being underwater to floating in space.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/Astronaut_2.jpg"}+The Eiffel Tower transitions from a daytime scene to a nighttime scene.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/Effiel_2.jpeg"}+A woman is seamlessly moving through different scenes with a magical effect.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/MorphCharacters_2.jpg"}+A small kitten is playing in the snow.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/cat2.png"}+A person is giving water to a dog, and the dog tilts its head back to drink.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/dog2.png"}+A car is speeding down a narrow country road in a field.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/car2.jpeg"}+Three daffodils quietly bloom, their delicate petals unfurling in serene harmony.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/flower2.png"}+A butterfly, attracted by the fragrance of the flowers, flew over and landed on the scent.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/butterfly2.png"}+A cyclist rides along the waves as they crash against the shore.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/cyclist2.png"}+The earth is rotating.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/earth2.png"}+The bread is baking in the oven, slowly expanding as it rises.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/bread2.png"}+The little boy is celebrating his birthday and blowing out the candles.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/candle2.png"}+The airplane is landing at dusk.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/airplane2.png"}+Sharks swim in a circle.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/sea2.png"}+The man is walking along the path.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/walking2.png"}+The cat slipped on a pair of sunglasses.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/catwglass2.png"}+Attractive confident guy happily showing you gesture and winking on camera isolated.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/man2.png"}+Mother covering sleeping daughter with blanket.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/mother2.png"}+Pink Chocolate bar exploding midair against blue background.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/chocolate2.png"}+A guy is freeskate rollerblading.{"reference_path": "/mnt/jfs-hdd2/sora/data/i2v_test_img/freeskate_2.png"}diff --git a/assets/texts/imagenet_id.txt b/assets/texts/imagenet_id.txtdeleted file mode 100644index 9085aa0..0000000--- a/assets/texts/imagenet_id.txt+++ /dev/null@@ -1,8 +0,0 @@-207-360-387-974-88-979-417-279diff --git a/assets/texts/internal_prompts_1.txt b/assets/texts/internal_prompts_1.txtnew file mode 100644index 0000000..3fd2df5--- /dev/null+++ b/assets/texts/internal_prompts_1.txt@@ -0,0 +1,24 @@+A group of dolphins leaping out of the ocean, synchronized and playful, with crystal-clear blue water and a bright sunny sky.+A panda playing the violin on a grand stage, with an elegant backdrop and an attentive audience.+A dragon flying over a medieval village, breathing fire with villagers looking up in awe and fear.+A diverse group of people enjoying a picnic in the park, with a blonde baby playing on a blanket among them.+A child playing fetch with a golden retriever in a sunny backyard, both running and laughing with joy.+A couple dancing under the stars on a quiet beach, with gentle waves lapping at their feet and a full moon illuminating the scene.+A lone hiker standing on the edge of a cliff, looking out over a vast, misty valley at dawn.+A close-up of a person's face showing a range of emotions, with detailed expressions and lighting that highlights the contours of their face.+In ancient China, during the Han Dynasty, Zhaojun was leaving the Han capital of Chang'an, accompanied by a grand entourage. Dressed in luxurious attire, she wore a determined expression on her face. The procession moved along winding roads, heading towards the Xiongnu territory in the north. The atmosphere was solemn, filled with the sadness of parting.+In ancient China, the emperor attended court, accompanied by ministers and palace attendants. The emperor was discussing important state affairs with the ministers in the imperial court.+In early 20th century Paris, Marie Curie was conducting research on radioactivity in her laboratory. The lab was filled with various scientific instruments, and Curie was focused on her experiments. The atmosphere was charged with the rigor and passion of scientific exploration.+In a sunny city park, the camera captures a cat leisurely rolling around on the grass.+In a sunny city park, the camera captures a cat leisurely rolling around on the grass. In the distance, on a park bench, a little girl is reading a book.+In a sunny city park, the camera captures a cat leisurely rolling around on the grass. The camera slowly pans to a little girl dressed in colorful summer clothes, who is crouched beside the cat, happily playing with it.+The car moves along the road, with the wheels creating friction as they make contact with the ground.+The pendulum hanging from the clock swings periodically under the influence of gravity.+The two gases mix slowly and gracefully. The first gas is a light blue, while the second gas is bright green. As these gases converge, they create beautiful gradients and dynamic ripples in the air. The camera captures the details of the gas mixing, showcasing the various color transitions, bubbles, and flow effects.+On the vast prairie, an antelope is running at high speed. The camera focuses on its strong muscles, which rapidly shift between tension and relaxation with each stride. Its powerful limbs propel it forward. The wind on the prairie rustles the antelope's fur, and the entire scene is filled with natural vitality and a sense of freedom.+On a tranquil night, a night-blooming cereus quietly opens under the moonlight. In a softly focused shot, the petals slowly unfurl, revealing the pure white stamen. The fragrance of the flower fills the air, and the fleeting yet beautiful moment is captured with delicate detail. The atmosphere is serene and mysterious.+On a calm lake, an orange rubber duck floats gently, creating small ripples on the water's surface. Under the sunlight, the lake water displays a soft blue-green hue. The camera focuses on the small ripples around the rubber duck, showcasing a simple and relaxed dynamic effect.+In a volcanic eruption scene, lava erupts from the volcano's mouth, creating a spectacular fountain. The dynamic flow of the lava combines with the eruptive flames and smoke. The camera captures the high-speed movement of the lava and the dramatic changes in the surrounding environment, showcasing the awe-inspiring power of nature.+In a vibrant dance floor, several people are dancing. The camera moves with their steps, capturing the swift movements and jumps, with the background lights shifting and colors blending. The video showcases the fluidity of the dancers' movements and their sense of rhythm.+In a skydiving scene, the camera starts from the skydiver's perspective, rapidly descending through the clouds. It captures the free-fall motion of the skydiver, the changing airflow, and the passage through the mist. The skydiver performs spins and flips in the air, forming impressive formations and formations with other jumpers. The background music is exhilarating and dynamic, enhancing the thrill of the skydiving experience.+On an extreme mountain bike race course, the camera tracks a rider navigating through rugged, uneven terrain. The rider swiftly descends steep slopes, jumps, and makes sharp turns. The camera captures the details of each bump, leap, and turn, as well as the rider's interaction with the natural environment.diff --git a/assets/texts/internal_prompts_2.txt b/assets/texts/internal_prompts_2.txtnew file mode 100644index 0000000..4d801ee--- /dev/null+++ b/assets/texts/internal_prompts_2.txt@@ -0,0 +1,8 @@+In the center of the image, a woman is the main focus. She is elegantly dressed in a long, shimmering silver dress that cascades down to her feet. The dress is sleeveless, revealing her arms and shoulders. Her hair, styled in loose waves, frames her face as she turns her head to the side, her gaze directed towards the camera. She is standing in front of a black car, which is parked on a gray surface. The car is sleek and modern, with a distinctive design that includes a large rear window and a silver grille. The woman's position in front of the car suggests she is the center of attention, perhaps at a car show or a similar event. The background is blurred, but it appears to be an indoor setting with a high ceiling and bright lights. The focus is clearly on the woman and the car, with the background serving to highlight their features. The image captures a moment of elegance and sophistication, with the woman's dress and the car's design both drawing the eye.+In the video, a young woman is the main subject, standing in front of a bush of pink flowers. She is dressed in a traditional Japanese kimono, which is predominantly white with red and gold accents. The kimono features a floral pattern, adding to the overall aesthetic of the scene. The woman's hair is styled in a bun, adorned with a hair accessory that matches her kimono. She is holding a pink flower in her hand, which she is gently touching with her lips. Her gaze is directed off to the side, and her expression is one of contemplation. The background of the video is blurred, but it appears to be a garden or park, suggesting a peaceful and serene setting. The video is shot from a low angle, looking up at the woman, which adds a sense of grandeur to the scene. The camera angle also allows for a clear view of the woman's kimono and the flowers she is holding. The overall composition of the video suggests a theme of beauty and tranquility.+In the image, a young woman is the main subject. She is seated on a gray chair, her legs crossed in a relaxed manner. She is dressed in a gray dress that features a floral pattern, adding a touch of elegance to her appearance. The dress is complemented by a black choker necklace that she is wearing. She is holding a white phone in her hand, which she is using to take a selfie. The phone is held up in front of her, capturing her image in the mirror behind her. The mirror reflects the room she is in, which is dimly lit, creating an intimate atmosphere. The woman's pose and the way she is holding the phone suggest that she is in the middle of taking a selfie. The image captures a moment of self-expression and confidence. The woman's outfit, the chair she is sitting on, and the phone she is using are all clearly visible, providing a detailed view of the scene.+The video is a vibrant and colorful music video featuring a female performer. The performer is dressed in a white dress with pink and white accents, and she has long blonde hair. She is dancing in front of a large clock with a red and gold design, which is set against a backdrop of red and white flowers. The performer is holding a pink flower in her hand and appears to be in motion, suggesting she is dancing. The overall style of the video is reminiscent of a fairy tale or fantasy theme, with the use of the clock and flowers adding to the whimsical atmosphere. The performer's outfit and the setting suggest that the video may be related to a specific theme or concept, but the details of this are not provided in the description. The video does not contain any text or additional elements that can be confidently described.+In the image, a young woman is the main subject. She is seated on a wooden bench, her body oriented towards the camera. She is wearing a white top and a blue hat adorned with bunny ears, adding a playful touch to her outfit. In her hands, she holds a white bowl and a spoon, suggesting she might be enjoying a meal or a snack. The bench she is sitting on is positioned in front of a wooden wall, which provides a simple and uncluttered background. The overall scene is casual and relaxed, with the woman appearing to be at ease in her surroundings.+A close-up shot of a couple sharing a gentle kiss in a softly lit room. The couple’s faces are the focus, with their eyes closed and lips meeting in a tender, affectionate moment. The background is softly blurred, with warm, ambient lighting enhancing the intimate atmosphere. The scene captures the connection between the couple, with no other objects or distractions in the frame, emphasizing the simplicity and emotion of the kiss. The overall mood is peaceful and loving, highlighting the deep affection between them.+A serene scene of a couple walking hand in hand along a quiet beach at sunset. The couple is silhouetted against the warm, golden light of the setting sun, which casts long shadows on the sand. The ocean waves gently lap at the shore, creating a rhythmic, soothing background. The sky is a mix of soft oranges, pinks, and purples, adding to the tranquil atmosphere. The couple’s steps are in sync, their hands clasped as they stroll along the water’s edge. The focus is on the connection between them, with no other people or distractions in the frame, emphasizing the peacefulness of their shared moment.+A playful scene featuring a Golden Retriever and a white cat in a cozy living room. The Golden Retriever, with its fluffy golden fur, is lying down on a soft rug, looking up with a friendly expression. Nearby, the white cat is perched gracefully on the armrest of a sofa, its sleek fur contrasting with the dog’s. The two animals are close to each other, with the cat gently reaching out a paw towards the dog, suggesting a moment of gentle interaction. The background includes a few scattered toys and a warm, homey ambiance created by soft lighting. The scene captures the peaceful coexistence and budding friendship between the two pets.diff --git a/assets/texts/internal_test.txt b/assets/texts/internal_test.txtnew file mode 100644index 0000000..d44e695--- /dev/null+++ b/assets/texts/internal_test.txt@@ -0,0 +1,32 @@+A group of dolphins leaping out of the ocean, synchronized and playful, with crystal-clear blue water and a bright sunny sky.+A panda playing the violin on a grand stage, with an elegant backdrop and an attentive audience.+A dragon flying over a medieval village, breathing fire with villagers looking up in awe and fear.+A diverse group of people enjoying a picnic in the park, with a blonde baby playing on a blanket among them.+A child playing fetch with a golden retriever in a sunny backyard, both running and laughing with joy.+A couple dancing under the stars on a quiet beach, with gentle waves lapping at their feet and a full moon illuminating the scene.+A lone hiker standing on the edge of a cliff, looking out over a vast, misty valley at dawn.+A close-up of a person's face with changing emotions, with detailed expressions and lighting that highlights the contours of his or her face.+In ancient China, during the Han Dynasty, Zhaojun was leaving the Han capital of Chang'an, accompanied by a grand entourage. Dressed in luxurious attire, she wore a determined expression on her face. The procession moved along winding roads, heading towards the Xiongnu territory in the north. The atmosphere was solemn, filled with the sadness of parting.+In ancient China, the emperor attended court, accompanied by ministers and palace attendants. The emperor was discussing important state affairs with the ministers in the imperial court.+In early 20th century Paris, Marie Curie was conducting research on radioactivity in her laboratory. The lab was filled with various scientific instruments, and Curie was focused on her experiments. The atmosphere was charged with the rigor and passion of scientific exploration.+In a sunny city park, the camera captures a cat leisurely rolling around on the grass.+In a sunny city park, the camera captures a cat leisurely rolling around on the grass. In the distance, on a park bench, a little girl is reading a book.+In a sunny city park, the camera captures a cat leisurely rolling around on the grass. The camera slowly pans to a little girl dressed in colorful summer clothes, who is crouched beside the cat, happily playing with it.+The car moves along the road, with the wheels creating friction as they make contact with the ground.+The pendulum hanging from the clock swings periodically under the influence of gravity.+The two gases mix slowly and gracefully. The first gas is a light blue, while the second gas is bright green. As these gases converge, they create beautiful gradients and dynamic ripples in the air. The camera captures the details of the gas mixing, showcasing the various color transitions, bubbles, and flow effects.+On the vast prairie, an antelope is running at high speed. The camera focuses on its strong muscles, which rapidly shift between tension and relaxation with each stride. Its powerful limbs propel it forward. The wind on the prairie rustles the antelope's fur, and the entire scene is filled with natural vitality and a sense of freedom.+On a tranquil night, a night-blooming cereus quietly opens under the moonlight. In a softly focused shot, the petals slowly unfurl, revealing the pure white stamen. The fragrance of the flower fills the air, and the fleeting yet beautiful moment is captured with delicate detail. The atmosphere is serene and mysterious.+On a calm lake, an orange rubber duck floats gently, creating small ripples on the water's surface. Under the sunlight, the lake water displays a soft blue-green hue. The camera focuses on the small ripples around the rubber duck, showcasing a simple and relaxed dynamic effect.+In a volcanic eruption scene, lava erupts from the volcano's mouth, creating a spectacular fountain. The dynamic flow of the lava combines with the eruptive flames and smoke. The camera captures the high-speed movement of the lava and the dramatic changes in the surrounding environment, showcasing the awe-inspiring power of nature.+In a vibrant dance floor, several people are dancing. The camera moves with their steps, capturing the swift movements and jumps, with the background lights shifting and colors blending. The video showcases the fluidity of the dancers' movements and their sense of rhythm.+In a skydiving scene, the camera starts from the skydiver's perspective, rapidly descending through the clouds. It captures the free-fall motion of the skydiver, the changing airflow, and the passage through the mist. The skydiver performs spins and flips in the air, forming impressive formations and formations with other jumpers. The background music is exhilarating and dynamic, enhancing the thrill of the skydiving experience.+On an extreme mountain bike race course, the camera tracks a rider navigating through rugged, uneven terrain. The rider swiftly descends steep slopes, jumps, and makes sharp turns. The camera captures the details of each bump, leap, and turn, as well as the rider's interaction with the natural environment.+In the center of the image, a woman is the main focus. She is elegantly dressed in a long, shimmering silver dress that cascades down to her feet. The dress is sleeveless, revealing her arms and shoulders. Her hair, styled in loose waves, frames her face as she turns her head to the side, her gaze directed towards the camera. She is standing in front of a black car, which is parked on a gray surface. The car is sleek and modern, with a distinctive design that includes a large rear window and a silver grille. The woman's position in front of the car suggests she is the center of attention, perhaps at a car show or a similar event. The background is blurred, but it appears to be an indoor setting with a high ceiling and bright lights. The focus is clearly on the woman and the car, with the background serving to highlight their features. The image captures a moment of elegance and sophistication, with the woman's dress and the car's design both drawing the eye.+In the video, a young woman is the main subject, standing in front of a bush of pink flowers. She is dressed in a traditional Japanese kimono, which is predominantly white with red and gold accents. The kimono features a floral pattern, adding to the overall aesthetic of the scene. The woman's hair is styled in a bun, adorned with a hair accessory that matches her kimono. She is holding a pink flower in her hand, which she is gently touching with her lips. Her gaze is directed off to the side, and her expression is one of contemplation. The background of the video is blurred, but it appears to be a garden or park, suggesting a peaceful and serene setting. The video is shot from a low angle, looking up at the woman, which adds a sense of grandeur to the scene. The camera angle also allows for a clear view of the woman's kimono and the flowers she is holding. The overall composition of the video suggests a theme of beauty and tranquility.+In the image, a young woman is the main subject. She is seated on a gray chair, her legs crossed in a relaxed manner. She is dressed in a gray dress that features a floral pattern, adding a touch of elegance to her appearance. The dress is complemented by a black choker necklace that she is wearing. She is holding a white phone in her hand, which she is using to take a selfie. The phone is held up in front of her, capturing her image in the mirror behind her. The mirror reflects the room she is in, which is dimly lit, creating an intimate atmosphere. The woman's pose and the way she is holding the phone suggest that she is in the middle of taking a selfie. The image captures a moment of self-expression and confidence. The woman's outfit, the chair she is sitting on, and the phone she is using are all clearly visible, providing a detailed view of the scene.+The video is a vibrant and colorful music video featuring a female performer. The performer is dressed in a white dress with pink and white accents, and she has long blonde hair. She is dancing in front of a large clock with a red and gold design, which is set against a backdrop of red and white flowers. The performer is holding a pink flower in her hand and appears to be in motion, suggesting she is dancing. The overall style of the video is reminiscent of a fairy tale or fantasy theme, with the use of the clock and flowers adding to the whimsical atmosphere. The performer's outfit and the setting suggest that the video may be related to a specific theme or concept, but the details of this are not provided in the description. The video does not contain any text or additional elements that can be confidently described.+In the image, a young woman is the main subject. She is seated on a wooden bench, her body oriented towards the camera. She is wearing a white top and a blue hat adorned with bunny ears, adding a playful touch to her outfit. In her hands, she holds a white bowl and a spoon, suggesting she might be enjoying a meal or a snack. The bench she is sitting on is positioned in front of a wooden wall, which provides a simple and uncluttered background. The overall scene is casual and relaxed, with the woman appearing to be at ease in her surroundings.+A close-up shot of a couple sharing a gentle kiss in a softly lit room. The couple’s faces are the focus, with their eyes closed and lips meeting in a tender, affectionate moment. The background is softly blurred, with warm, ambient lighting enhancing the intimate atmosphere. The scene captures the connection between the couple, with no other objects or distractions in the frame, emphasizing the simplicity and emotion of the kiss. The overall mood is peaceful and loving, highlighting the deep affection between them.+A serene scene of a couple walking hand in hand along a quiet beach at sunset. The couple is silhouetted against the warm, golden light of the setting sun, which casts long shadows on the sand. The ocean waves gently lap at the shore, creating a rhythmic, soothing background. The sky is a mix of soft oranges, pinks, and purples, adding to the tranquil atmosphere. The couple’s steps are in sync, their hands clasped as they stroll along the water’s edge. The focus is on the connection between them, with no other people or distractions in the frame, emphasizing the peacefulness of their shared moment.+A playful scene featuring a Golden Retriever and a white cat in a cozy living room. The Golden Retriever, with its fluffy golden fur, is lying down on a soft rug, looking up with a friendly expression. Nearby, the white cat is perched gracefully on the armrest of a sofa, its sleek fur contrasting with the dog’s. The two animals are close to each other, with the cat gently reaching out a paw towards the dog, suggesting a moment of gentle interaction. The background includes a few scattered toys and a warm, homey ambiance created by soft lighting. The scene captures the peaceful coexistence and budding friendship between the two pets.diff --git a/assets/texts/multi_human.txt b/assets/texts/multi_human.txtnew file mode 100644index 0000000..5ded64d--- /dev/null+++ b/assets/texts/multi_human.txt@@ -0,0 +1,10 @@+In a modern kitchen, two chefs are cutting onions on red plastic cutting boards placed on metallic work benches. Both chefs wear black working attire, black caps, and black gloves, meticulously chopping the onions into small pieces. Between them are large bowls filled with boiled, de-shelled eggs. In the background, neatly stacked raw eggs can be seen. A man, likely the supervisor, stands facing away, looking out an open-glass door while holding papers. The scene captures a moment of focused kitchen activity without transitions.+Three guys are standing on a side road in front of a Chinese rural village, engaging in an informal interview. The guy on the left, wearing a white T-shirt and blue jeans, is facing the other two, gesturing with his hands as he talks. The middle guy, dressed in a black polo shirt, is nodding and listening attentively. The rightmost guy, in a blue polo shirt, occasionally glances at the camera. Behind them is a large gray stone stacked on a red square pillar, resembling a landmark, with shrubs further in the background. The video focuses on the interaction among the three characters.+The video captures a scene in a spacious, modern indoor sheep farm. A group of people stands in the middle passage, observing a black-and-white lamb. Fences dividing sheep into different areas are visible on the left side. The central figure is holding the lamb, with a man on the right trying to pat it, and another man on the left looking at it. A woman in a red-and-white dress is smiling at the lamb from a sideways angle, while another man's partial figure can be seen in the distance. All the men are wearing white long coats, and the lamb is kicking its hind legs while being held firmly by the central figure.+Two singers are walking slowly and singing on a stage during a performance. The male singer, wearing a beige scarf and white top, holds a microphone and sings attentively while holding the female singer's hand. The female singer, dressed in a white dress with long brown curly hair, smiles glamorously. The stage background features steel frames and is otherwise unclear, with an orange light in the background. The video is shot from the front at a slightly low angle, capturing both singers as they face slightly towards the left. The background is dark, but the spotlight illuminates the singers, making their faces and expressions clearly visible.+A distinguished musical performance is in progress at a classical concert. In the center, a man in a formal navy coat, white shirt, and cravat holds a violin and addresses the audience. Behind him, a full orchestra of seated musicians prepares with their string and percussion instruments, some smiling and looking towards the central figure. The setting is elegant, with decorative music stands featuring intricate gold designs, indicating a prestigious event. The atmosphere exudes grandeur and professionalism, characteristic of a high-caliber classical music performance. The scene captures a moment of anticipation and readiness before the music begins.+A dynamic scene at a musical concert featuring two women performing on stage. The woman on the left, with blonde wavy hair, is passionately singing into a golden microphone. She is dressed in a sparkly, embellished yellow outfit with dark glittering details, her eyes partially closed, and her hand raised expressively. The woman on the right, shown in profile, is wearing a sleeveless, glittering dark dress. Her long, wavy hair flows over her shoulders, and she appears composed, possibly preparing for her part or listening attentively. The background, slightly blurred, reveals musicians or orchestra members, enhancing the live performance atmosphere with focused and intense energy.+A lively moment on a talk show set with the focus on a guest and the host. On the left, the guest, a woman in an elegant black outfit with sparkling details, is gesturing expressively with both hands as she passionately shares a story. Her hair is tied back, and she wears a necklace, adding to her stylish appearance. On the right, the host, in a suit and tie, sits behind a wooden desk, leaning slightly forward with a smile, appearing attentive and engaged. The backdrop showcases a nighttime city skyline, typical of late-night talk shows. The scene captures a warm interaction, highlighting the positive and relaxed mood of their conversation.+A vibrant scene featuring a talented girl group performing a synchronized dance routine against a backdrop screen displaying soft clouds and a moonlit sky. The group members are dressed in stylish pastel outfits and knee-high boots, radiating energy and confidence. Their dynamic choreography and intricate formations showcase their skill and dedication. Expressive gestures and movements highlight their commitment to the craft. The combination of music, dance, and visual elements creates an electrifying atmosphere that captivates the audience, retaining focus on their performance without scene transitions.+A scene on a set of urban stairs where two girls are sitting, both intently focused on their mobile phones. The girl on the left is adjusting her hair with one hand while holding her phone with a protective case in the other, capturing a casual yet attentive demeanor. The girl on the right, dressed in a stylish outfit, is completely absorbed in her device. The background shows people walking by, emphasizing the bustling urban environment. Despite the activity around them, the girls are engaged in their own digital worlds. Their body language reflects a blend of relaxation and concentration as they interact with their screens.+A young girl in a colorful swimsuit and a cute sun hat is playfully covering a boy with sand on a beach. She is holding a small plastic shovel and has a joyful expression, clearly enjoying the activity. The boy, lying on a beach towel in a blue swimsuit and cap, appears relaxed and enjoying the sun. The sandy beach stretches out behind them, with the ocean visible in the background. The scene captures a moment of fun and carefree play, typical of a day at the beach. There is no scene transition, focusing on this single, joyous moment.diff --git a/assets/texts/rand_types.txt b/assets/texts/rand_types.txtindex bd4b5d8..2f32c18 100644--- a/assets/texts/rand_types.txt+++ b/assets/texts/rand_types.txt@@ -1,40 +1,8 @@+随机风景镜头随机电影镜头-随机电影镜头-随机电影镜头-随机电影镜头-随机电影镜头-随机任务镜头-随机任务镜头-随机任务镜头-随机任务镜头-随机任务镜头-随机游戏镜头-随机游戏镜头-随机游戏镜头随机游戏镜头-随机游戏镜头-随机开车镜头-随机开车镜头-随机开车镜头随机开车镜头-随机开车镜头-随机动物镜头-随机动物镜头随机动物镜头-随机动物镜头-随机动物镜头-随机森林镜头-随机森林镜头随机森林镜头-随机森林镜头-随机森林镜头-随机动漫镜头随机动漫镜头-随机动漫镜头-随机动漫镜头-随机动漫镜头-随机舞蹈镜头-随机舞蹈镜头-随机舞蹈镜头-随机舞蹈镜头随机舞蹈镜头diff --git a/assets/texts/t2v_demo.txt b/assets/texts/t2v_demo.txtnew file mode 100644index 0000000..39419fc--- /dev/null+++ b/assets/texts/t2v_demo.txt@@ -0,0 +1,12 @@+a heartfelt moment in a church, where a bride and groom are standing in front of a priest. the bride, clad in a white dress, is holding a bouquet of flowers, while the groom is dressed in a black suit. the priest, standing between them, is holding a book, possibly a bible or a prayer book. the church is adorned with stained glass windows, adding a touch of color to the otherwise solemn setting. the couple, the priest, and the church form the main subjects of the video. the bride and groom are positioned in the foreground, while the priest is slightly behind them. the stained glass windows are located in the background, providing a beautiful backdrop to the scene. the relative positions of the objects suggest a typical church setting, with the priest officiating the ceremony. the bride and groom are the focal point of the video, indicating the importance of their union in this setting. the stained glass windows add a touch of color and artistry to the scene, enhancing the overall aesthetic of the video.+the breathtaking view of the vigelandsparken sculpture park in oslo, norway. the park is nestled amidst a lush forest of trees and is surrounded by towering mountains. the camera angle is from a low angle, providing a unique perspective of the park and its surroundings. the park is bustling with people, some of whom are seen walking on the paths, while others are admiring the sculptures. the sculptures themselves are made of wood and are scattered throughout the park, adding an artistic touch to the natural beauty of the place. the colors in the video are predominantly green and brown, reflecting the natural environment of the park. the overall scene is a harmonious blend of nature and art, with the park serving as a perfect spot for relaxation and contemplation.+a 3d animated scene featuring three animated characters. in the foreground, there is a character with green skin, wearing a red and green striped hat, a brown vest, and a pair of boots. this character is holding a mug and appears to be in a cheerful mood. to the right, there is a character with pink hair and a green crown, wearing a pink and green striped dress and holding a small card. this character is smiling and seems to be in a good mood. on the left, there is a character with brown skin, wearing a green and brown striped hat, a brown vest, and a pair of boots. this character is holding a mug and appears to be in a relaxed state. in the background, there is a colorful and whimsical setting that resembles a cozy room with a window, a fireplace, and a bookshelf. the room has a warm and inviting atmosphere, with a variety of objects and colors that add to the fantastical and playful nature of the scene.+a young man is the central figure, standing in a room with a wooden floor and a window in the background. he is dressed in a blue and white striped shirt, a blue bandana wrapped around his head, and a pair of black gloves. in his hands, he holds a small bottle of medicine, which he appears to be examining closely. his gaze is focused on the bottle, suggesting a sense of curiosity or concern. behind him, a woman is seated on a chair, her attention directed towards the man. she is wearing a white dress adorned with red flowers, and a white hat with a red ribbon. her posture and expression suggest she is observing the man's actions with interest. the room they are in is simple and uncluttered, with the wooden floor and window providing a natural contrast to the indoor setting. the window allows a glimpse of the outside world, where a tree can be seen, adding a touch of nature to the scene. the video captures a moment of quiet interaction between the two characters, with the man's examination of the medicine bottle as the main action.+a bustling scene in a commercial kitchen where a group of chefs are preparing a meal. the kitchen is well equipped with stainless steel appliances and counters, and the chefs are dressed in orange aprons, indicating their professional roles. in the foreground, a chef is seen stirring a large pot filled with a dark liquid, possibly a sauce or soup. the pot is placed on a stove, and the chef is using a long handled spoon to stir the contents. the pot is placed on a stove, and the chef is using a long handled spoon to stir the contents. in the background, another chef is seen preparing a dish on a tray. the dish appears to be a type of sushi, with various ingredients neatly arranged on a bed of rice. the chef is using a pair of chopsticks to pick up a piece of sushi, presumably to inspect its quality or to serve it to a customer. the kitchen is well lit, with bright lights illuminating the workspace and highlighting the chefs' focused expressions.+a bustling scene from a souk in marrakesh, morocco. the perspective is from a low angle, looking up at the vibrant market stalls and the people moving about. the stalls are filled with a variety of goods, including colorful textiles, leather bags, and intricately designed lanterns. the people, dressed in traditional moroccan clothing, add to the lively atmosphere of the souk. the sun is shining brightly, casting a warm glow on the scene and highlighting the rich colors of the market. the video is a lively depiction of daily life in marrakesh, capturing the energy and vibrancy of the city's markets.+we see a young man sitting in the backseat of a car. he is wearing a green shirt and has brown hair. the car is in motion, as suggested by the blurred background. the man is looking out of the window, his gaze directed towards something outside the frame. the interior of the car is visible, with the backseat where the man is sitting and the front seat in the foreground. the car appears to be a sedan, and the man is seated in the middle of the backseat.+a close-up shot of an older man wearing a military uniform, including a cap with a badge, a jacket with epaulettes, and a tie. he appears to be in a room with various mechanical parts and gauges in the background, suggesting a setting related to aviation or machinery. the man has a stern expression on his face and is looking directly at the camera. the style of the video is realistic, with a focus on the man's face and upper body, and the lighting is even, without harsh shadows. the background is out of focus, which draws attention to the man. the overall tone of the video is serious and intense.+a moment in a restaurant where a young woman is seated at a table, engrossed in her meal. she is dressed in a black dress, which contrasts with the vibrant yellow of the booth she is seated on. the table in front of her is adorned with a white tablecloth and holds a plate of food, from which she is eating with chopsticks. the restaurant itself is elegantly decorated with a chandelier hanging from the ceiling, casting a warm glow over the scene. a large fish tank is also visible in the background, adding to the ambiance of the place. the woman appears to be enjoying her meal, her attention fully on the food in front of her. the overall scene paints a picture of a quiet, intimate dining experience.+a vibrant scene of a flowering plant in full bloom. the plant is adorned with pink flowers that are in the process of opening, revealing their delicate petals. the flowers are not alone; they are accompanied by a group of bees, their black bodies contrasting with the pink hues of the flowers. the bees are engaged in the act of pollination, their movements creating a sense of life and activity. the background is a blur of green, suggesting a lush garden or park setting. the overall scene is a beautiful representation of nature in action, with the bees playing a crucial role in the life cycle of the plant.+a close-up shot of a woman sitting on a bed, holding a pastry with both hands. the pastry appears to be a round, golden-brown item with a cross-section design on top. the woman is wearing a white sweater and has long, wavy brown hair. she is looking at the pastry with a smile on her face. the background is blurred but suggests an indoor setting with warm lighting. the style of the video is casual and personal, likely intended for a lifestyle or food blog.+a serene beach scene under a cloudy sky. the sky is filled with clouds, casting a soft light over the scene. the water is a light blue color, gently lapping against the sandy beach. the beach is populated with a few birds, their small forms scattered across the sand. the waves are small, rolling onto the shore in a rhythmic pattern. the overall atmosphere of the video is peaceful and calm, a typical day at the beach.diff --git a/assets/texts/t2v_pllava.txt b/assets/texts/t2v_pllava.txtindex f0ab0f5..6003c27 100644--- a/assets/texts/t2v_pllava.txt+++ b/assets/texts/t2v_pllava.txt@@ -8,3 +8,5 @@ a scene where a person is examining a dog. the person is wearing a blue shirt wia close-up shot of a woman applying makeup. she is using a black brush to apply a dark powder to her face. the woman has blonde hair and is wearing a black top. the background is black, which contrasts with her skin tone and the makeup. the focus is on her face and the brush, with the rest of her body and the background being out of focus. the lighting is soft and even, highlighting the texture of the makeup and the woman's skin. there are no texts or other objects in the video. the woman's expression is neutral, and she is looking directly at the camera. the video does not contain any action, as it is a still shot of a woman applying makeup. the relative position of the woman and the brush is such that the brush is in her hand and is being used to apply the makeup to her face. the video does not contain any other objects or actions. the woman is the only person in the video, and she is the main subject. the video does not contain any sound. the description is based on the visible content of the video and does not include any assumptions or interpretations.a young woman is seated in a black gaming chair in a room filled with computer monitors and other gaming equipment. she is wearing a red tank top and black pants, and her hair is styled in loose waves. the room is dimly lit, with the glow of the monitors casting a soft light on her face. she is holding a black game controller in her hands, and her attention is focused on the screen in front of her. the room is filled with other gaming equipment, including keyboards and mice, and there are other chairs and desks scattered around the room. the woman appears to be engrossed in her game, her posture relaxed yet focused. the room is quiet, the only sound coming from the beeps and boops of the game. the woman is the only person in the room, adding a sense of solitude to the scene. the video does not contain any text. the relative positions of the objects suggest a well-organized gaming setup, with the woman at the center, surrounded by her gaming equipment. the video does not contain any action, but the woman's focused expression suggests that she is in the middle of an intense ga breathtaking aerial view of a coastal landscape at sunset. the sky, painted in hues of orange and pink, serves as a stunning backdrop to the scene. the sun, partially obscured by the horizon, casts a warm glow on the landscape below. the foreground of the image is dominated by a rocky cliff, its rugged surface adding a touch of raw beauty to the scene. the cliff's edge is adorned with patches of green vegetation, providing a stark contrast to the otherwise barren landscape. the middle ground of the image reveals a winding road that hugs the coastline. the road, appearing as a thin line against the vast expanse of the landscape, guides the viewer's eye towards the horizon. in the background, the silhouette of mountains can be seen, their peaks shrouded in a light mist. the mountains, along with the road, add depth to the image, creating a sense of distance and scale. overall, the video presents a serene and majestic coastal landscape, captured at the perfect moment of sunset. the colors+A close-up shot of a person eating noodles in a simple, well-lit kitchen. The person is seated at a wooden table with a bowl of steaming noodles in front of them, using chopsticks to lift the noodles to their mouth. The camera captures the motion of the noodles as they are lifted, with the focus on the person’s hand and the noodles. The background features a plain white wall and a few kitchen utensils hanging on the wall. The lighting is natural, coming from a window to the right, casting soft shadows. The overall atmosphere is warm and homey, emphasizing the simplicity and comfort of the meal.+A tender moment of a man and woman embracing in a quiet, softly lit room. The couple stands close, their arms wrapped around each other, with the man gently resting his chin on the woman’s shoulder. The background is blurred, highlighting the intimacy of their embrace, with warm, diffused lighting creating a cozy and serene atmosphere. The focus is on the expressions of comfort and connection between them, as they hold each other in a quiet moment of affection. There are no other objects or distractions in the frame, allowing the viewer to feel the depth of their bond.⬤diff --git a/configs/opensora-v1-2/train/stage1.py b/configs/opensora-v1-2/train/stage1.pyindex cbaa20c..e1f1e7c 100644--- a/configs/opensora-v1-2/train/stage1.py+++ b/configs/opensora-v1-2/train/stage1.py@@ -108,6 +108,3 @@ lr = 1e-4ema_decay = 0.99adam_eps = 1e-15warmup_steps = 1000--cache_pin_memory = True-pin_memory_cache_pre_alloc_numels = [(290 + 20) * 1024**2] * (2 * 8 + 4)diff --git a/configs/opensora-v1-3/inference/t2v.py b/configs/opensora-v1-3/inference/t2v.pynew file mode 100644index 0000000..8d3ee7c--- /dev/null+++ b/configs/opensora-v1-3/inference/t2v.py@@ -0,0 +1,49 @@+num_frames = 113+fps = 24+frame_interval = 1+resolution = "720p"+aspect_ratio = "9:16"++save_dir = "./samples/samples/"+multi_resolution = "STDiT2"+seed = 42+batch_size = 1+dtype = "bf16"++model = dict(+ type="STDiT3-XL/2",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-STDiT-v4",+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ kernel_size=(8, 8, -1),+ use_spatial_rope=True,+ force_huggingface=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="/mnt/jfs-hdd/sora/checkpoints/pretrained_models/t5-v1_1-xxl",+ model_max_length=300,+)+scheduler = dict(+ type="rflow",+ use_timestep_transform=True,+ num_sampling_steps=30,+ cfg_scale=7.5,+ use_oscillation_guidance=True,+ use_flaw_fix=True,+ # transform_scale=2.0,+)diff --git a/configs/opensora-v1-3/inference/v2v.py b/configs/opensora-v1-3/inference/v2v.pynew file mode 100644index 0000000..949d91f--- /dev/null+++ b/configs/opensora-v1-3/inference/v2v.py@@ -0,0 +1,58 @@+num_frames = 113+resolution = "360p"+aspect_ratio = "9:16"+fps = 24+frame_interval = 1++save_dir = "samples/v2v/debug"+multi_resolution = "STDiT2"+seed = 42+batch_size = 1+dtype = "bf16"++cond_type = "i2v_head" # v2v_head, i2v_head, i2v_loop, or None+condition_frame_length = 5 # condition on the first n frames, latent size+use_sdedit = True+use_oscillation_guidance_for_text = True+use_oscillation_guidance_for_image = True++model = dict(+ type="STDiT3-XL/2",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-STDiT-v4",+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ kernel_size=(8, 8, -1), # H W T+ use_spatial_rope=True,+ class_dropout_prob=0.0,+ force_huggingface=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="/mnt/jfs-hdd/sora/checkpoints/pretrained_models/t5-v1_1-xxl",+ model_max_length=300,+)+scheduler = dict(+ type="rflow",+ use_timestep_transform=True,+ num_sampling_steps=30,+ cfg_scale=7.5,+ scale_image_weight=True,+ initial_image_scale=1.0,+)+image_cfg_scale = 5.0 # cfg, increase to force follow+aes = 7.0+flow = Nonediff --git a/configs/opensora-v1-3/misc/eval_loss.py b/configs/opensora-v1-3/misc/eval_loss.pynew file mode 100644index 0000000..0da015e--- /dev/null+++ b/configs/opensora-v1-3/misc/eval_loss.py@@ -0,0 +1,46 @@+num_workers = 8+dtype = "bf16"+seed = 42+num_eval_timesteps = 10++# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)++bucket_config = {+ "360p": {113: (None, 6)},+}++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained=None,+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ kernel_size=(8, 8, -1),+ use_spatial_rope=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+)+scheduler = dict(+ type="rflow",+)diff --git a/configs/opensora-v1-3/train/adapt.py b/configs/opensora-v1-3/train/adapt.pynew file mode 100644index 0000000..d8ff572--- /dev/null+++ b/configs/opensora-v1-3/train/adapt.py@@ -0,0 +1,78 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)+bucket_config = {+ "360p": {1: (1.0, 8)},+ # ---+ "480p": {1: (0.3, 4)},+ # ---+ "720p": {1: (0.2, 2)},+ # ---+ "1080p": {1: (1.0, 1)},+ # ---+ "2048": {1: (0.7, 1)},+}+grad_checkpoint = False+warmup = 1000++# Acceleration settings+num_workers = 8+num_bucket_build_workers = 16+dtype = "bf16"+plugin = "zero2"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained="outputs/0373-STDiT3-XL-2/epoch3-global_step36000/ema.pt",+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ adapt_16ch=True,+ skip_temporal=True,+)+# vae = dict(+# type="VideoAutoencoderKL",+# from_pretrained="PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers",+# subfolder="vae",+# scaling_factor=0.13025,+# )+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ use_timestep_transform=True,+ sample_method="logit-normal",+)++# Log settings+seed = 42+outputs = "outputs"+wandb = False+epochs = 1000+log_every = 10+ckpt_every = 500++# optimization settings+load = None+grad_clip = 1.0+lr = 1e-4+adam_eps = 1e-8+ema_decay = 0.99diff --git a/configs/opensora-v1-3/train/adapt_i2v.py b/configs/opensora-v1-3/train/adapt_i2v.pynew file mode 100644index 0000000..4723536--- /dev/null+++ b/configs/opensora-v1-3/train/adapt_i2v.py@@ -0,0 +1,106 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)++bucket_config = {+ "360p": {+ 49: (1.0, 10), # 15+ 65: (1.0, 7), # 20+ 81: (1.0, 6), # 25+ 97: (1.0, 5), # 30+ 113: (1.0, 4), # 35+ },+ "720p": {+ 49: (0.25, 2), # 15+ 65: (0.25, 1), # 20+ 81: (0.25, 1), # 25+ 97: (0.25, 1), # 30+ 113: (0.25, 1), # 35+ },+}++# i2v & v2v condition+drop_condition = {+ "cond": 0.05,+ "text": 0.05,+ "null": 0.05,+ "keep": 0.85, # 85% of the time don't drop anything+}++# i2v & v2v condition+mask_types = {+ "i2v_head": 5,+ "i2v_tail": 2,+ "i2v_loop": 2,+ "v2v_head": 1,+ "v2v_head_noisy": 2,+ "v2v_tail": 1,+ "other": 1,+ "none": 2,+}++grad_checkpoint = True++# Acceleration settings+num_workers = 8 # 8+num_bucket_build_workers = 16 # 16+dtype = "bf16"+plugin = "zero2"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained=None,+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ freeze_y_embedder=True,+ class_dropout_prob=0.1, # i2v & v2v change+ kernel_size=(8, 8, -1), # H W T+ use_spatial_rope=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="pretrained_models/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ use_timestep_transform=True,+ sample_method="logit-normal",+ drop_condition=drop_condition, # i2v & v2v random drop text/image/video condition+)++# Log settings+seed = 4207+outputs = "outputs/"+wandb = False+epochs = 1000+log_every = 8+ckpt_every = 200++# optimization settings+grad_clip = 1.0+lr = 1e-4+ema_decay = 0.99+adam_eps = 1e-15+warmup_steps = 1000++# v2v params+accumulation_steps = 1diff --git a/configs/opensora-v1-3/train/stage1.py b/configs/opensora-v1-3/train/stage1.pynew file mode 100644index 0000000..56f3195--- /dev/null+++ b/configs/opensora-v1-3/train/stage1.py@@ -0,0 +1,74 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)+bucket_config = {+ "360p": {+ 1: (1.0, 60),+ 49: (1.0, 5), # 15+ 65: (1.0, 3), # 20+ 81: (1.0, 3), # 25+ 97: (1.0, 2), # 30+ 113: (1.0, 2), # 35+ },+}+grad_checkpoint = True++# Acceleration settings+num_workers = 8+num_bucket_build_workers = 16+dtype = "bf16"+plugin = "zero2"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained=None,+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ kernel_size=(8, 8, -1), # H W T+ use_spatial_rope=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ sample_method="logit-normal",+ use_timestep_transform=True,+)++# Log settings+seed = 42+outputs = "outputs"+wandb = False+epochs = 2+log_every = 10+ckpt_every = 250++# optimization settings+lr = 1e-4+warmup_steps = 0+use_cosine_scheduler = True+grad_clip = 1.0+adam_eps = 1e-15+ema_decay = 0.99+accumulation_steps = 4diff --git a/configs/opensora-v1-3/train/stage1_hq.py b/configs/opensora-v1-3/train/stage1_hq.pynew file mode 100644index 0000000..bb06888--- /dev/null+++ b/configs/opensora-v1-3/train/stage1_hq.py@@ -0,0 +1,95 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)+bucket_config = {+ "360p": {+ 1: (1.0, 60),+ 49: (1.0, 5), # 15+ 65: (1.0, 3), # 20+ 81: (1.0, 3), # 25+ 97: (1.0, 2), # 30+ 113: (1.0, 2), # 35+ },+}+grad_checkpoint = True++# i2v & v2v condition+mask_types = {+ "none": 8, # t2v+ "i2v_head": 2,+ "i2v_tail": 2,+ "i2v_loop": 2,+ "v2v_head": 1,+ "v2v_tail": 1,+ "other": 1,+}+drop_condition = {+ "cond": 0.05,+ "text": 0.05,+ "null": 0.05,+ "keep": 0.85, # 85% of the time don't drop anything+}++# Acceleration settings+num_workers = 8+num_bucket_build_workers = 16+dtype = "bf16"+plugin = "zero2"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained=None,+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ freeze_y_embedder=True,+ class_dropout_prob=0.1,+ kernel_size=(8, 8, -1), # H W T+ use_spatial_rope=True,+ use_cond=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ sample_method="logit-normal",+ use_timestep_transform=True,+ drop_condition=drop_condition, # i2v & v2v random drop text/image/video condition+)++# Log settings+seed = 42+outputs = "outputs"+wandb = False+epochs = 1+log_every = 10+ckpt_every = 250++# optimization settings+lr = 1e-4+warmup_steps = 1000+use_cosine_scheduler = True+grad_clip = 1.0+adam_eps = 1e-15+ema_decay = 0.99+accumulation_steps = 2diff --git a/configs/opensora-v1-3/train/stage1_nk.py b/configs/opensora-v1-3/train/stage1_nk.pynew file mode 100644index 0000000..3fed68e--- /dev/null+++ b/configs/opensora-v1-3/train/stage1_nk.py@@ -0,0 +1,68 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)+bucket_config = {+ "360p": {+ 1: (1.0, 60),+ 51: (1.0, 5), # 15+ 68: (1.0, 3), # 20+ 85: (1.0, 3), # 25+ 102: (1.0, 2), # 30+ 119: (1.0, 2), # 35+ },+}+grad_checkpoint = True++# Acceleration settings+num_workers = 8+num_bucket_build_workers = 16+dtype = "bf16"+plugin = "zero2"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained="outputs/0461-STDiT3-XL-2/epoch3-global_step40000/ema.pt",+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ use_timestep_transform=True,+ sample_method="logit-normal",+)++# Log settings+seed = 42+outputs = "outputs"+wandb = False+epochs = 1000+log_every = 10+ckpt_every = 200++# optimization settings+lr = 1e-4+warmup = 100+grad_clip = 1.0+adam_eps = 1e-8+ema_decay = 0.99diff --git a/configs/opensora-v1-3/train/stage1_noise.py b/configs/opensora-v1-3/train/stage1_noise.pynew file mode 100644index 0000000..100771d--- /dev/null+++ b/configs/opensora-v1-3/train/stage1_noise.py@@ -0,0 +1,86 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)+bucket_config = {+ "360p": {+ 1: (1.0, 60),+ 49: (1.0, 5), # 15+ 65: (1.0, 3), # 20+ 81: (1.0, 3), # 25+ 97: (1.0, 2), # 30+ 113: (1.0, 2), # 35+ },+}+grad_checkpoint = True++# Acceleration settings+num_workers = 8+num_bucket_build_workers = 16+dtype = "bf16"+plugin = "zero2"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained=None,+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ kernel_size=(8, 8, -1), # H W T+ use_spatial_rope=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ sample_method="logit-normal",+ use_timestep_transform=True,+)++# Log settings+seed = 42+outputs = "outputs"+wandb = False+epochs = 1000+log_every = 10+ckpt_every = 250++# optimization settings+lr = 1e-4+warmup = 100+grad_clip = 1.0+adam_eps = 1e-15+ema_decay = 0.99+accumulation_steps = 1++# Noise injection settings+noise_augmentation = True+noise_prob = {+ "gaussian_pixel": 0.5,+ "downsample_spatial": 0.2,+ "downsample_temporal": 0.2,+}+noise_strength = {+ "gaussian_pixel": 0.4,+ "downsample_spatial": [2, 4, 4, 8, 8, 8, 8],+ "downsample_temporal": [2, 4, 8],+}diff --git a/configs/opensora-v1-3/train/stage2.py b/configs/opensora-v1-3/train/stage2.pynew file mode 100644index 0000000..7269439--- /dev/null+++ b/configs/opensora-v1-3/train/stage2.py@@ -0,0 +1,73 @@+# Dataset settings+dataset = dict(+ type="VariableVideoTextDataset",+ transform_name="resize_crop",+)+bucket_config = {+ "720p": {+ 1: (1.0, 30),+ 49: (1.0, 2), # 15+ 65: (1.0, 1), # 20+ 81: (1.0, 1), # 25+ 97: (1.0, 1), # 30+ 113: (1.0, 1), # 35+ },+}+grad_checkpoint = True++# Acceleration settings+num_workers = 8+num_bucket_build_workers = 16+dtype = "bf16"+plugin = "zero1"++# Model settings+model = dict(+ type="STDiT3-XL/2",+ from_pretrained=None,+ qk_norm=True,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ kernel_size=(8, 8, -1), # H W T+ use_spatial_rope=True,+)+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained="/home/guoxinying/open_source_video_ocean_V1/OpenSora-VAE-v1.3",+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+ normalization="video",+ temporal_overlap=True,+ force_huggingface=True,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=300,+ shardformer=True,+)+scheduler = dict(+ type="rflow",+ sample_method="logit-normal",+ use_timestep_transform=True,+)++# Log settings+seed = 42+outputs = "outputs"+wandb = False+epochs = 1000+log_every = 10+ckpt_every = 250++# optimization settings+lr = 5e-5+warmup_steps = 1000+grad_clip = 1.0+adam_eps = 1e-15+ema_decay = 0.99+accumulation_steps = 2diff --git a/configs/vae/inference/video.py b/configs/vae/inference/video.pyindex 904de9a..8e94c85 100644--- a/configs/vae/inference/video.py+++ b/configs/vae/inference/video.py@@ -1,6 +1,7 @@image_size = (256, 256)-num_frames = 51+num_frames = 17+fps = 24dtype = "bf16"batch_size = 1seed = 42@@ -14,6 +15,7 @@ dataset = dict(data_path=None,num_frames=num_frames,image_size=image_size,+ transform_name="resize_crop",)num_samples = 100num_workers = 4diff --git a/configs/vae_v1_3/inference/image_16z.py b/configs/vae_v1_3/inference/image_16z.pynew file mode 100644index 0000000..26c32e7--- /dev/null+++ b/configs/vae_v1_3/inference/image_16z.py@@ -0,0 +1,40 @@+image_size = (1024, 1024)+num_frames = 1++dtype = "bf16"+batch_size = 1+seed = 42+save_dir = "samples/OpenSoraVAE_V1_3_vae/image"++# Define dataset+dataset = dict(+ type="VideoTextDataset",+ data_path=None,+ num_frames=num_frames,+ image_size=image_size,+ transform_name="resize_crop",+)+num_samples = 1000+num_workers = 4+cal_stats = True++# Define model+model = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained=None,+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ use_tiled_conv3d=True,+ tile_size=16,+)+is_init_image = True++vae_loss_config = dict(perceptual_loss_weight=1.0, kl_loss_weight=1e-6, logvar_init=0.0)++gan_loss_confg = dict(+ disc_start=0,+ disc_factor=1.0,+ disc_weight=0.5,+ disc_loss_type="hinge",+)diff --git a/configs/vae_v1_3/inference/video_16z_256x256.py b/configs/vae_v1_3/inference/video_16z_256x256.pynew file mode 100644index 0000000..9a9bcd8--- /dev/null+++ b/configs/vae_v1_3/inference/video_16z_256x256.py@@ -0,0 +1,42 @@+image_size = (256, 256)+num_frames = 51++fps = 24+dtype = "bf16"+batch_size = 1+seed = 42+save_dir = "samples/OpenSoraVAE_V1_3_vae_16z"++# Define dataset+dataset = dict(+ type="VideoTextDataset",+ data_path=None,+ num_frames=num_frames,+ image_size=image_size,+ transform_name="resize_crop",+)+num_samples = 100+num_workers = 4+cal_stats = True++# Define model+model = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained=None,+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=16,+)+is_init_image = True++vae_loss_config = dict(perceptual_loss_weight=1.0, kl_loss_weight=1e-6, logvar_init=0.0)++gan_loss_confg = dict(+ disc_start=0,+ disc_factor=1.0,+ disc_weight=0.5,+ disc_loss_type="hinge",+)diff --git a/configs/vae_v1_3/inference/video_16z_512x512.py b/configs/vae_v1_3/inference/video_16z_512x512.pynew file mode 100644index 0000000..37a3ae4--- /dev/null+++ b/configs/vae_v1_3/inference/video_16z_512x512.py@@ -0,0 +1,42 @@+image_size = (512, 512)+num_frames = 51++fps = 24+dtype = "bf16"+batch_size = 1+seed = 42+save_dir = "samples/OpenSoraVAE_V1_3_16z"++# Define dataset+dataset = dict(+ type="VideoTextDataset",+ data_path=None,+ num_frames=num_frames,+ image_size=image_size,+ transform_name="resize_crop",+)+num_samples = 100+num_workers = 4+cal_stats = True++# Define model+model = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained=None,+ z_channels=16,+ micro_batch_size=1,+ micro_batch_size_2d=4,+ micro_frame_size=17,+ use_tiled_conv3d=True,+ tile_size=4,+)+is_init_image = True++vae_loss_config = dict(perceptual_loss_weight=1.0, kl_loss_weight=1e-6, logvar_init=0.0)++gan_loss_confg = dict(+ disc_start=0,+ disc_factor=1.0,+ disc_weight=0.5,+ disc_loss_type="hinge",+)diff --git a/configs/vae_v1_3/train/video_16z.py b/configs/vae_v1_3/train/video_16z.pynew file mode 100644index 0000000..6a10d4a--- /dev/null+++ b/configs/vae_v1_3/train/video_16z.py@@ -0,0 +1,63 @@+num_frames = 17+image_size = (256, 256)++# Define dataset+dataset = dict(+ type="VideoTextDataset",+ data_path=None,+ num_frames=num_frames,+ frame_interval=1,+ image_size=image_size,+ transform_name="resize_crop",+)++# Define acceleration+num_workers = 16+dtype = "bf16"+grad_checkpoint = True+plugin = "zero2"++# Define model+model = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained=None,+ z_channels=16,+ # use_tiled_conv3d=True,+ # tile_size=16,+)++discriminator = dict(+ type="N_LAYER_DISCRIMINATOR_3D",+ from_pretrained="/mnt/jfs-hdd/sora/checkpoints/pretrained_models/causalvae_v1.1.0/discriminator/model.pt",+ input_nc=3,+ n_layers=3,+ use_actnorm=False,+)++# training+mixed_strategy = "mixed_video_image"+mixed_image_ratio = 0.2++vae_loss_config = dict(perceptual_loss_weight=1.0, kl_loss_weight=1e-6, logvar_init=0.0)++gan_loss_confg = dict(+ disc_start=0,+ disc_factor=1.0,+ disc_weight=0.5,+ disc_loss_type="hinge",+)++# Others+seed = 42+outputs = "outputs/OpenSoraVAE_V1_3"+wandb = True++epochs = 100 # NOTE: adjust accordingly w.r.t dataset size+log_every = 1+ckpt_every = 500+load = None++batch_size = 1+lr = 5e-6+grad_clip = 1.0+warmup_steps = 1000diff --git a/docs/commands.md b/docs/commands.mdindex 92ff5e6..d36c6ab 100644--- a/docs/commands.md+++ b/docs/commands.md@@ -2,6 +2,7 @@- [Config](#Config)- [Inference](#inference)+ - [Inference with Open-Sora 1.3](#inference-with-open-sora-13)- [Inference with Open-Sora 1.2](#inference-with-open-sora-12)- [Inference with Open-Sora 1.1](#inference-with-open-sora-11)- [Inference with DiT pretrained on ImageNet](#inference-with-dit-pretrained-on-imagenet)@@ -47,6 +48,49 @@ However, if you want to load a self-trained model, do not set `force_huggingfaceYou can modify corresponding config files to change the inference settings. See more details [here](/docs/structure.md#inference-config-demos).+### Inference with Open-Sora 1.3+```bash+# t2v++python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 --llm-refine True \+ --aes "very good" --flow "fair" \+ --prompt "a beautiful waterfall"+```++```bash+# i2v & v2v++# first-frame condition generation+python scripts/inference_i2v.py configs/opensora-v1-3/inference/v2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio "9:16" --cond-type i2v_head --use-sdedit True \+ --use-oscillation-guidance-for-image True --image-cfg-scale 2.0 \+ --use-oscillation-guidance-for-text True --cfg-scale 7.5 \+ --prompt 'A breathtaking sunrise scene.{"reference_path": "assets/images/condition/wave.png"}'++# last-frame condition generation+python scripts/inference_i2v.py configs/opensora-v1-3/inference/v2v.py \+ --save-dir samples/i2v/i2v_tail --seed 42 --ckpt-path path/to/your/ckpt \ # You can also load your self-trained checkpoints+ --prompt-path assets/texts/i2v/prompts_human_i2v_head.txt \ # Support for inputting multiple prompts via a text or csv file+ --num-frames 113 --resolution 360p --aspect-ratio 9:16 \+ --use-sdedit False --cond-type i2v_tail \+ --use-oscillation-guidance-for-image True --image-cfg-scale 2.5 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# video extending+python scripts/inference_i2v.py configs/opensora-v1-3/inference/v2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio "9:16" --cond-type v2v_head --use-sdedit True \+ --use-oscillation-guidance-for-image True --image-cfg-scale 2.0 \+ --use-oscillation-guidance-for-text True --cfg-scale 7.5 \+ --prompt 'A car driving on the ocean.{"reference_path": "https://cdn.openai.com/tmp/s/interp/d0.mp4"}'++# video connecting+python scripts/inference_i2v.py configs/opensora-v1-3/inference/v2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio "9:16" --cond-type i2v_loop --use-sdedit True \+ --use-oscillation-guidance-for-image True --image-cfg-scale 2.0 \+ --use-oscillation-guidance-for-text True --cfg-scale 7.5 \+ --prompt 'A breathtaking sunrise scene.{"reference_path": "assets/images/condition/sunset1.png;assets/images/condition/sunset2.png"}'+```+### Inference with Open-Sora 1.2The inference API is compatible with Open-Sora 1.1. To ease users' experience, we add support to `--resolution` and `--aspect-ratio` options, which is a more user-friendly way to specify the image size.diff --git a/docs/report_04.md b/docs/report_04.mdnew file mode 100644index 0000000..4d025f1--- /dev/null+++ b/docs/report_04.md@@ -0,0 +1,116 @@+# Open-Sora 1.3 Report++- [Video compression network](#video-compression-network)+- [Upgraded STDiT with shifted-window attention](#upgraded-stdit-with-shifted-window-attention)+- [Easy and effective model conditioning](#easy-and-effective-model-conditioning)+- [Evaluation](#evaluation)++In Open-Sora 1.3 release, we train a 1.1B models on >60M data (\~85k hours), with training cost 35k H100 GPU hours, supporting 0s\~113 frames, 360p & 720p, various aspect ratios video generation. Our configurations is listed below. Following our 1.2 version, Open-Sora 1.3 can also do image-to-video generation and video extension.++| | image | 49 frames | 65 frames | 81 frames | 97 frames | 113 frames |+| ---- | ----- | ---------- | ---------- | ---------- | --------- | ---------- |+| 360p | ✅ | ✅ | ✅ | ✅ | ✅ |✅ |+| 720p | ✅ | ✅ | ✅ | ✅ | ✅ |✅ |++Here ✅ means that the data is seen during training.++Besides features introduced in Open-Sora 1.2, Open-Sora 1.3 highlights:++- Video compression network+- Upgraded STDiT with shifted-window attention+- More data and better multi-stage training+- Easy and effective model conditioning+- Better evaluation metrics++All implementations (both training and inference) of the above improvements are available in the Open-Sora 1.3 release. The following sections will introduce the details of the improvements. We also refine our codebase and documentation to make it easier to use and develop, and add a LLM refiner to [refine input prompts](/README.md#gpt-4o-prompt-refinement) and support more languages.++## Video compression network++In Open-Sora 1.2, the video compression architecture employed a modular approach, where spatial and temporal dimensions were handled separately. The spatial VAE, based on Stability AI's SDXL VAE, compressed individual frames along the spatial dimensions. The temporal VAE then processed the latent representations from the spatial VAE to handle temporal compression. This two-stage design allowed effective spatial and temporal compression but introduced limitations. These included inefficiencies in handling long videos due to fixed-length input frames, a lack of seamless integration between spatial and temporal features, and higher memory requirements during both training and inference.++Open-Sora 1.3 introduces a unified approach to video compression. By combining spatial and temporal processing into a single framework and leveraging advanced features like tiled 3D convolutions and dynamic frame support, Open-Sora 1.3 achieves improved better efficiency, scalability, and reconstruction quality. Here are the key improvements in Open-Sora 1.3 VAE:++**1. Unified Spatial-Temporal Processing:** Instead of using separate VAEs for spatial and temporal compression, Open-Sora 1.3 adopts a single encoder-decoder structure that simultaneously handles both dimensions. This approach eliminates the need for intermediate representations and redundant data transfers between spatial and temporal modules.++**2. Tiled 3D Convolutions:** Open-Sora 1.3 incorporates tiled 3D convolution support for the temporal dimension. By breaking down videos into smaller temporal tiles, this feature enables efficient encoding and decoding of longer video sequences without increasing memory overhead. This improvement addresses the limitations of Open-Sora 1.2 in handling large frame counts and ensures higher flexibility in temporal compression.++**3. Dynamic Micro-Batch and Micro-Frame Processing:** Open-Sora 1.3 introduces a new micro-batch and micro-frame processing mechanism. This allows for: (1) Adaptive temporal overlap: Overlapping frames during temporal encoding and decoding help reduce discontinuities at tile boundaries. (2) Dynamic frame size support: Instead of being restricted to fixed-length sequences (e.g., 17 frames in Open-Sora 1.2), Open-Sora 1.3 supports dynamic sequence lengths, making it robust for varied video lengths.++**4. Unified Normalization Mechanism:** The normalization process in Open-Sora 1.3 has been refined with tunable scaling (scale) and shifting (shift) parameters that ensure consistent latent space distributions across diverse datasets. Unlike Open-Sora 1.2, where normalization was specific to fixed datasets, this version introduces more generalized parameters and support for frame-specific normalization strategies.+++#### Summary of Improvements++| Feature | Open-Sora 1.2 | Open-Sora 1.3 |+|------------------------|-----------------------------------------|-----------------------------------------|+| **Architecture** | Separate spatial and temporal VAEs | Unified spatial-temporal VAE |+| **Tiled Processing** | Not supported | Supported (Tiled 3D Convolutions) |+| **Frame Length Support**| Fixed (17 frames) | Dynamic frame support with overlap |+| **Normalization** | Fixed parameters | Tunable scaling and shifting |+++## Upgraded STDiT with shifted-window attention++Following the success of OpenSora 1.2, version 1.3 introduces several architectural improvements and new capabilities to enhance video generation quality and flexibility. This section outlines the key improvements and differences between these two versions.++Latest diffusion models like Stable Diffusion 3 adopt the [rectified flow](https://github.com/gnobitab/RectifiedFlow) instead of DDPM for better performance. While SD3's rectified flow training code is not open-sourced, OpenSora provides the training code following SD3's paper. OpenSora 1.2 introduced several key strategies from SD3:++1. Basic rectified flow training, which enables continuous-time diffusion+2. Logit-norm sampling for training acceleration (following SD3 paper Section 3.1), preferentially sampling timesteps at middle noise levels+3. Resolution and video length aware timestep sampling (following SD3 paper Section 5.3.2), using more noise for larger resolutions and longer videos++For OpenSora 1.3, we further enhance the model with significant improvements in architecture, capabilities, and performance:++#### 1. Shift-Window Attention Mechanism+- Introduced kernel-based local attention with configurable kernel_size for efficient computation+- Implemented shift-window partitioning strategy similar to Swin Transformer+- Added padding mask handling for window boundaries with extra_pad_on_dims support+- Extended position encoding with 3D relative positions within local windows (temporal, height, width)+#### 2. Enhanced Position Encoding+- Improved RoPE implementation with reduced rotation_dim (1/3 of original) for 3D scenarios+- Added separate rotary embeddings for temporal, height, and width dimensions+- Implemented resolution-adaptive scaling for position encodings+- Optional spatial RoPE for better spatial relationship modeling+#### 3. Flexible Generation+- Added I2V and V2V capabilities with dedicated conditioning mechanisms+- Introduced conditional embedding modules (x_embedder_cond and x_embedder_cond_mask)+- Zero-initialized condition embeddings for stable training+- Flexible temporal modeling with skip_temporal option+#### 4. Performance Optimization+- Refined Flash Attention triggering conditions (N > 128) for better efficiency+- Added support for torch.scaled_dot_product_attention (SDPA) as an alternative backend+- Optimized memory usage through improved padding and window partitioning+- Enhanced sequence parallelism with adaptive height padding++The adaptation process from [PixArt-Σ 2K](https://github.com/PixArt-alpha/PixArt-sigma) remains similar but with additional steps:+1-7. [Same as v1.2: multi-resolution training, QK-norm, rectified flow, logit-norm sampling, smaller AdamW epsilon, new VAE, and basic temporal attention]+#### 8. Enhanced temporal blocks+ - Added kernel-based local attention with shift-window support+ - Implemented 3D relative position encoding with resolution-adaptive scaling+ - Zero-initialized projection layers with improved initialization strategy++Compared to v1.2 which focused on basic video generation, v1.3 brings substantial improvements in three key areas: **1. Quality**: Enhanced spatial-temporal modeling through shift-window attention and 3D position encoding. **2. Flexibility**: Support for I2V/V2V tasks and configurable temporal modeling. **3. Efficiency**: Optimized attention computation and memory usage++These improvements maintain backward compatibility with v1.2's core features while extending the model's capabilities for real-world applications. The model retains its ability to generate high-quality images and videos using rectified flow, while gaining new strengths in conditional generation and long sequence modeling.++## Easy and effective model conditioning++We calculate the aesthetic score and motion score for each video clip, and filter out those clips with low scores, which leads to a dataset with better video quality. Additionally, we append the scores to the captions and use them as conditioning. Specifically, we convert numerical scores into descriptive language based on predefined ranges. The aesthetic score transformation function converts numerical aesthetic scores into descriptive labels based on predefined ranges: scores below 4 are labeled "terrible," progressing through "very poor," "poor," "fair," "good," and "very good," with scores of 6.5 or higher labeled as "excellent." Similarly, the motion score transformation function maps motion intensity scores to descriptors: scores below 0.5 are labeled "very low," progressing through "low," "fair," "high," and "very high," with scores of 20 or more labeled as "extremely high." We find this method can make model aware of the scores and follows the scores to generate videos with better quality.++For example, a video with aesthetic score 5.5, motion score 10, and a detected camera motion pan left, the caption will be:++```plaintext+[Original Caption] The aesthetic score is good, the motion strength is high, camera motion: pan left.+```++During inference, we can also use the scores to condition the model. For camera motion, we only label 13k clips with high confidence, and the camera motion detection module is released in our tools.++## Evaluation++Previously, we monitor the training process only by human evaluation, as DDPM traning loss is not well correlated with the quality of generated videos. However, for rectified flow, we find the training loss is well correlated with the quality of generated videos as stated in SD3. Thus, we keep track of rectified flow evaluation loss on 100 images and 1k videos.++We sampled 1k videos from pixabay as validation dataset. We calculate the evaluation loss for image and different lengths of videos (49 frames, 65 frames, 81 frames, 97 frames, 113 frames) for different resolution (360p, 720p). For each setting, we equidistantly sample 10 timesteps. Then all the losses are averaged.++In addition, we also keep track of [VBench](https://vchitect.github.io/VBench-project/) scores during training. VBench is an automatic video evaluation benchmark for short video generation. We calcuate the vbench score with 360p 49-frame videos. The two metrics verify that our model continues to improve during training.++All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details.diff --git a/docs/structure.md b/docs/structure.mdindex 322b730..3c38d8d 100644--- a/docs/structure.md+++ b/docs/structure.md@@ -21,24 +21,27 @@ Open-Sora│ ├── report_01.md -> Report for Open-Sora 1.0│ ├── report_02.md -> Report for Open-Sora 1.1│ ├── report_03.md -> Report for Open-Sora 1.2+│ ├── report_04.md -> Report for Open-Sora 1.3│ ├── vae.md -> our VAE report│ └── zh_CN -> Chinese version of the above├── eval -> Evaluation scripts│ ├── README.md -> Evaluation documentation| ├── human_eval -> for human eval-| ├── launch.sh -> script for launching 8 cards sampling+| ├── I2V -> for image to video human eval| ├── loss -> eval loss| ├── sample.sh -> script for quickly launching inference on predefined prompts| ├── vae -> for vae eval| ├── vbench -> for VBench evaluation│ └── vbench_i2v -> for VBench i2v evaluation├── gradio -> Gradio demo related code-├── notebooks -> Jupyter notebooks for generating commands to run├── scripts│ ├── train.py -> diffusion training script+│ ├── train_opensoravae_v1_3.py -> vae v1.3 training script│ ├── train_vae.py -> vae training script│ ├── inference.py -> diffusion inference script+│ ├── inference_opensoravae_v1_3.py -> vae v1.3 training script│ ├── inference_vae.py -> vae inference script+│ ├── inference_i2v.py -> image to video inference script│ └── misc -> misc scripts, including batch size search├── opensora│ ├── __init__.py@@ -49,6 +52,7 @@ Open-Sora│ │ ├── dit -> DiT│ │ ├── layers -> Common layers│ │ ├── vae -> VAE as image encoder+│ │ ├── vae_v1_3 -> VAE V1.3 as image encoder│ │ ├── text_encoder -> Text encoder│ │ │ ├── classes.py -> Class id encoder (inference only)│ │ │ ├── clip.py -> CLIP encoder@@ -71,7 +75,7 @@ Our config files follows [MMEgine](https://github.com/open-mmlab/mmengine). MMEn```plaintextOpen-Sora-└── configs -> Configs for training & inference+└── configs -> Configs for training & inferences├── opensora-v1-1 -> STDiT2 related configs│ ├── inference│ │ ├── sample.py -> Sample videos and imagesdiff --git a/docs/vae_v1_3.md b/docs/vae_v1_3.mdnew file mode 100644index 0000000..5ff07d5--- /dev/null+++ b/docs/vae_v1_3.md@@ -0,0 +1,59 @@+# Commands++## Train VAE with 8 machines++```bash+colossalai run --hostfile hostfile --nproc_per_node 8 scripts/train_opensoravae_v1_3.py configs/vae_v1_3/train/video_16z.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT --wandb True > logs/train_opensoravae_v1_3.log 2>&1 &+```++## Evaluate VAE performance++* If ``VID_PATH`` is not specified, use the default uses vid100 used in `eval/loss`;+* We use image1k used in `eval/loss` for image evaluation;+* eval stats are saved to `${CKPT_PATH}/eval/ folder`.++```bash+VID_PATH=/home/shenchenhui/data/eval_loss/forest_vid_100/pixabay_forest_vid_100.csv CUDA_VISIBLE_DEVICES=0 bash eval/vae/launch.sh pretrained_models/OpenSoraVAE_V1_3/model.pt+```++## Inference++We can set optimization options in vae config:+```bash+vae = dict(+ type="OpenSoraVAE_V1_3",+ from_pretrained=None,+ z_channels=16,+ shift=[...],+ scale=[...],+ micro_batch_size=1, # DON'T set during training of vae+ micro_batch_size_2d=4, # DON'T set during training of vae+ micro_frame_size=17, # DON'T set during training of vae+ use_tiled_conv3d=True,+ tile_size=4,+)+```++### Inference with VAE+set force-huggingface to True if loading the original pretrained huggingface model `pretrained_models/OpenSoraVAE_V1_3`.++```bash+# video+CUDA_VISIBLE_DEVICES=0 torchrun --standalone --nproc_per_node=1 scripts/inference_opensoravae_v1_3.py configs/vae_v1_3/inference/video_16z_512x512.py --data-path YOUR_CSV_PATH --save-dir ./samples/vae_16z/videos --ckpt-path YOUR_PRETRAINED_CKPT --force-huggingface False+++# image+CUDA_VISIBLE_DEVICES=0 torchrun --standalone --nproc_per_node=1 scripts/inference_opensoravae_v1_3.py configs/vae_v1_3/inference/image_16z.py --data-path YOUR_CSV_PATH --save-dir ./samples/vae_16z/images/ --ckpt-path YOUR_PRETRAINED_CKPT --force-huggingface False+```++### Train DiT with freezed VAE++```bash+CUDA_VISIBLE_DEVICES=0 torchrun --standalone --nproc_per_node 1 scripts/train.py configs/opensora-v1-3/train/stage1.py --data-path /mnt/ddn/sora/meta/pro_1_0_ddn/internvid_first_quarter_ext.csv+```++### Inference DiT with VAE++```bash+CUDA_VISIBLE_DEVICES=7 python scripts/inference.py configs/opensora-v1-3/train/stage1.py --ckpt-path /mnt/ddn/sora/checkpoints/outputs/0245-STDiT3-XL-2/epoch0-global_step13000 --prompt-path assets/texts/t2v_samples.txt --save-dir samples/debug --num-frames 51 --resolution 360p --aspect-ratio 9:16 --sample-name sample_2s_360p --batch-size 1+```diff --git a/docs/zh_CN/README.md b/docs/zh_CN/README.mdindex e6df4ac..6812e90 100644--- a/docs/zh_CN/README.md+++ b/docs/zh_CN/README.md@@ -19,7 +19,7 @@[[中文文档](/docs/zh_CN/README.md)] [[潞晨云](https://cloud.luchentech.com/)|[OpenSora镜像](https://cloud.luchentech.com/doc/docs/image/open-sora/)|[视频教程](https://www.bilibili.com/video/BV1ow4m1e7PX/?vd_source=c6b752764cd36ff0e535a768e35d98d2)]## 📰 资讯-+**[2024.01.20]** 🔥 我们发布了 **Open-Sora 1.3**。通过升级的VAE和Transformer架构,我们生成的视频质量得到了大幅提升 🚀。[[模型权重]](#open-sora-13-model-weights) [[技术报告]](/report_v4.md) [[演示]](https://huggingface.co/spaces/hpcai-tech/open-sora)* **[2024.12.23]** 🔥[视频生成模型开发成本直降50%,开源解决方案来了,还能白嫖GPU算力](https://company.hpc-ai.com/blog/the-development-cost-of-video-generation-models-has-saved-by-50-open-source-solutions-are-now-available-with-h200-gpu-vouchers) [[代码]](https://github.com/hpcaitech/Open-Sora/blob/main/scripts/train.py) [[代金券]](https://colossalai.org/zh-Hans/docs/get_started/bonus/)* **[2024.06.22]** 🔥我们在[潞晨云](https://cloud.luchentech.com/)上发布了Open-Sora1.2镜像,并在B站上传了详细的[使用教程](https://www.bilibili.com/video/BV1ow4m1e7PX/)* **[2024.06.17]** 🔥我们发布了**Open-Sora 1.2**,其中包括**3D-VAE**,**整流流**和**得分条件**。视频质量大大提高。[[模型权重]](#模型权重) [[技术报告]](report_v3.md) [[公众号文章]](https://mp.weixin.qq.com/s/QHq2eItZS9e00BVZnivdjg)@@ -36,11 +36,22 @@🔥 您可以在HuggingFace上的 [🤗 Gradio应用程序](https://huggingface.co/spaces/hpcai-tech/open-sora)上体验Open-Sora. 我们的[画廊](https://hpcaitech.github.io/Open-Sora/)中提供了更多示例.+| **5s 720×1280** | **5s 720×1280** | **5s 720×1280** |+|------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|+| [<img src="assets/demo/open_sora_v1.3/demo_tomato.gif" width="">](https://streamable.com/e/r0imrp?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_fisherman.gif" width="">](https://streamable.com/e/hfvjkh?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_girl2.gif" width="">](https://streamable.com/e/kutmma?quality=highest&autoplay=1) |+| [<img src="assets/demo/open_sora_v1.3/demo_grape.gif" width="">](https://streamable.com/e/osn1la?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_mushroom.gif" width="">](https://streamable.com/e/l1pzws?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_parrot.gif" width="">](https://streamable.com/e/2vqari?quality=highest&autoplay=1) |+| [<img src="assets/demo/open_sora_v1.3/demo_trans.gif" width="">](https://streamable.com/e/1in7d6?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_bear.gif" width="">](https://streamable.com/e/e9bi4o?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_futureflower.gif" width="">](https://streamable.com/e/09z7xi?quality=highest&autoplay=1) |+| [<img src="assets/demo/open_sora_v1.3/demo_fire.gif" width="">](https://streamable.com/e/16c3hk?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_man.gif" width="">](https://streamable.com/e/wi250w?quality=highest&autoplay=1) | [<img src="assets/demo/open_sora_v1.3/demo_black.gif" width="">](https://streamable.com/e/vw5b64?quality=highest&autoplay=1) |++<details>+<summary>OpenSora 1.2 演示</summary>+| **4s 720×1280** | **4s 720×1280** | **4s 720×1280** || ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- || [<img src="/assets/demo/v1.2/sample_0013.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/7895aab6-ed23-488c-8486-091480c26327) | [<img src="/assets/demo/v1.2/sample_1718.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/20f07c7b-182b-4562-bbee-f1df74c86c9a) | [<img src="/assets/demo/v1.2/sample_0087.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/3d897e0d-dc21-453a-b911-b3bda838acc2) || [<img src="/assets/demo/v1.2/sample_0052.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/644bf938-96ce-44aa-b797-b3c0b513d64c) | [<img src="/assets/demo/v1.2/sample_1719.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/272d88ac-4b4a-484d-a665-8d07431671d0) | [<img src="/assets/demo/v1.2/sample_0002.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/ebbac621-c34e-4bb4-9543-1c34f8989764) || [<img src="/assets/demo/v1.2/sample_0011.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/a1e3a1a3-4abd-45f5-8df2-6cced69da4ca) | [<img src="/assets/demo/v1.2/sample_0004.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/d6ce9c13-28e1-4dff-9644-cc01f5f11926) | [<img src="/assets/demo/v1.2/sample_0061.gif" width="">](https://github.com/hpcaitech/Open-Sora/assets/99191637/561978f8-f1b0-4f4d-ae7b-45bec9001b4a) |+</details><details><summary>OpenSora 1.1 演示</summary>@@ -76,10 +87,10 @@</details>## 🔆 新功能/更新-+* 📍 **Open-Sora 1.3** 已发布。模型权重可在[此处](#model-weights)获取。更多详细信息请参见我们的**[版本报告 v1.3](report_v4.md)**。+* ✅ 升级的模型架构,提供更强的性能和效率。+* ✅ 引入更多高质量训练数据,显著提升整体效果。* 📍 **Open-Sora 1.2** 发布。模型权重可在[此处](#model-weights)查看。有关更多详细信息,请参阅我们的**[技术报告 v1.2](docs/report_03.md)** 。-* ✅ 支持整流流调度。-* ✅ 训练我们的 3D-VAE 进行时间维度压缩。* 📍 **Open-Sora 1.1**发布。模型权重可在[此处](#model-weights)获得。它针对**0s~15s、144p 到 720p、各种宽高比**的视频进行训练。有关更多讨论,请参阅我们的**[技术报告 v1.1](/docs/report_02.md)** 。* 🔧 **数据处理流程** v1.1发布,提供从原始视频到(文本,视频片段)对的自动处理流程,包括场景剪切$\rightarrow$过滤(美学、光流、OCR 等)$\rightarrow$字幕$\rightarrow$管理。使用此工具,您可以轻松构建视频数据集。* ✅ 改进的 ST-DiT 架构包括 rope 位置编码、qk 范数、更长的文本长度等。@@ -133,7 +144,7 @@下面列出了其他有用的文档和链接。-* 报告: [技术报告 v1.2](docs/report_v3.md), [技术报告 v1.1](/docs/report_v2.md), [技术报告 v1.0](/docs/report_v1.md), [训练加速.md](docs/acceleration.md)+* 报告: [技术报告 v1.3](report_v4.md), [技术报告 v1.2](report_v3.md), [技术报告 v1.1](report_v2.md), [技术报告 v1.0](report_v1.md), [训练加速.md](acceleration.md)* Repo 结构: [结构.md](docs/structure.md)* 配置文件说明: [config.md](docs/config.md)* Useful commands: [commands.md](docs/commands.md)@@ -193,13 +204,21 @@ docker run -ti --gpus all -v .:/workspace/Open-Sora opensora## 模型权重+### Open-Sora 1.3 模型权重+| 分辨率 | 模型大小 | 数据 | 迭代次数 | 批次大小 | 网址 |+| ---------- | ---------- | ---- | ----------- | ---------- | --- |+| Diffusion | 1.1B | 30M | 70k | 动态大小 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v4) |+| VAE | 259M | 3M | 1M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.3) |++请参阅我们的 **[report 1.3](docs/report_v4.md)**以了解更多信息。+### Open-Sora 1.2 模型权重| 分辨率 | 模型大小 | 数据 | 迭代次数 | 批次大小 | 网址 || ---------- | ---------- | ---- | ----------- | ---------- | --- || Diffusion | 1.1B | 30M | 70k | 动态大小 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3) || VAE | 384M | 3M | 1M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2) |-请参阅我们的**[report 1.2](docs/report_v3.md)**以了解更多信息。+请参阅我们的 **[report 1.2](docs/report_v3.md)**以了解更多信息。### Open-Sora 1.1 模型权重@@ -264,13 +283,12 @@ export OPENAI_API_KEY=YOUR_API_KEY生成视频最简单的方式是输入文本提示,然后点击“**生成视频**”按钮(如果找不到,请向下滚动)。生成的视频将显示在右侧面板中。勾选“**使用 GPT4o 增强提示**”将使用 GPT-4o 来细化提示,而“**随机提示**”按钮将由 GPT-4o 为您生成随机提示。由于 OpenAI 的 API 限制,提示细化结果具有一定的随机性。-然后,你可以选择生成视频的**分辨率**、**时长**、**长宽比**。不同的分辨率和视频长度会影响视频生成速度。在 80G H100 GPU 上,生成速度和峰值内存使用量为:+然后,你可以选择生成视频的**分辨率**、**时长**、**长宽比**。不同的分辨率和视频长度会影响视频生成速度。在 140G H200 GPU 上,生成速度和峰值内存使用量为:-| 分辨率 | 图像 | 2秒 | 4秒 | 8秒 | 16秒 |-| ---- | ------- | -------- | --------- | --------- | --------- |-| 360p | 3s, 24G | 18s, 27G | 31s, 27G | 62s, 28G | 121s, 33G |-| 480p | 2s, 24G | 29s, 31G | 55s, 30G | 108s, 32G | 219s, 36G |-| 720p | 6s, 27G | 68s, 41G | 130s, 39G | 260s, 45G | 547s, 67G |+| | Image |49 帧 | 65 帧 | 81 帧 | 97 帧 |113 帧 |+|------|----------|----------|-----------|-----------|-----------|-----------|+| 360p | 10s, 23G | 16s, 23G | 21s, 24G | 27s, 24G | 33s, 25G | 39s, 25G |+| 720p | 13s, 24G | 61s, 28G | 83s, 29G | 106s, 31G | 130s, 33G | 157s, 35G |注意,除了文本转视频,你还可以使用图片转视频。你可以上传图片,然后点击“**生成视频**”按钮,生成以图片为第一帧的视频。或者,你可以填写文本提示,然后点击“**生成图片**”按钮,根据文本提示生成图片,然后点击“**生成视频**”按钮,根据同一模型生成的图片生成视频。@@ -278,7 +296,7 @@ export OPENAI_API_KEY=YOUR_API_KEY然后您可以指定更多选项,包括“**运动强度**”、“**美学**”和“**相机运动**”。如果未选中“启用”或选择“无”,则不会将信息传递给模型。否则,模型将生成具有指定运动强度、美学分数和相机运动的视频。-对于**美学分数**,我们建议使用高于 6 的值。对于**运动强度**,较小的值将导致更平滑但动态性较差的视频,而较大的值将导致更动态但可能更模糊的视频。因此,您可以尝试不使用它,然后根据生成的视频进行调整。对于**相机运动**,有时模型无法很好地遵循指令,我们正在努力改进它。+对于**美学分数**,我们建议使用'excellent'。对于**运动强度**,较小的值将导致更平滑但动态性较差的视频,而较大的值将导致更动态但可能更模糊的视频。因此,您可以尝试不使用它,然后根据生成的视频进行调整。对于**相机运动**,有时模型无法很好地遵循指令,我们正在努力改进它。您还可以调整“**采样步数**”,这是去噪的次数,与生成速度直接相关。小于 30 的数字通常会导致较差的生成结果,而大于 100 的数字通常不会有明显的改善。“种子”用于可重复性,您可以将其设置为固定数字以生成相同的视频。“**CFG 比例**”控制模型遵循文本提示的程度,较小的值会导致视频更随机,而较大的值会导致视频更遵循文本(建议为 7)。@@ -286,7 +304,53 @@ export OPENAI_API_KEY=YOUR_API_KEY## 推理+### Open-Sora 1.3 命令行推理+基础的命令行推理:++```bash+# text to video+python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 \+ --prompt "a beautiful waterfall"+```+您可以向命令行添加更多选项来定制生成。++```bash+python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 --num-sample 1 \+ --aes "very good" --flow "fair" \+ --prompt "a beautiful waterfall"+```+对于图像到视频生成和其他功能,请参阅[此处]](docs/commands.md)了解更多说明。++### 序列并行推理+```bash+# text to video+CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --aspect-ratio 9:16 \+ --prompt "a beautiful waterfall"+```++## GPT-4o 快速细化++我们发现 GPT-4o 可以细化提示并提高生成视频的质量。利用此功能,您还可以使用其他语言(例如中文)作为提示。要启用此功能,您需要在环境中准备您的 openai api 密钥:++```bash+export OPENAI_API_KEY=YOUR_API_KEY+```++然后您可以用 `--llm-refine True` 启用GPT-4o进行提示细化以完成推理。++```bash+python scripts/inference.py configs/opensora-v1-3/inference/t2v.py \+ --num-frames 97 --resolution 720p --llm-refine True \+ --prompt "a beautiful waterfall"+```++### Open-Sora 1.2 命令行推理+<details>+<summary>View more</summary>基础的命令行推理:@@ -340,6 +404,7 @@ export OPENAI_API_KEY=YOUR_API_KEY```然后您可以用 `--llm-refine True` 启用GPT-4o进行提示细化以完成推理。+</details>### Open-Sora 1.1 命令行推理<details>@@ -396,8 +461,20 @@ torchrun --standalone --nproc_per_node 2 scripts/inference.py configs/opensora/i## 训练-### Open-Sora 1.2 训练+### Open-Sora 1.3 训练+```bash+# one node+torchrun --standalone --nproc_per_node 8 scripts/train.py \+ configs/opensora-v1-3/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT+# multiple nodes+colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py \+ configs/opensora-v1-3/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT+```++### Open-Sora 1.2 训练+<details>+<summary>View more</summary>训练过程与Open-Sora 1.1相同。```bash@@ -408,6 +485,7 @@ torchrun --standalone --nproc_per_node 8 scripts/train.py \colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py \configs/opensora-v1-2/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT```+</details>### Open-Sora 1.1 训练@@ -464,9 +542,26 @@ colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py configs/o| -------------- | ----------- | ------------- | -------------- || Open-Sora V1.0 | 75.91% | 78.81% | 64.28% || Open-Sora V1.2 | 79.23% | 80.71% | 73.30% |+| Open-Sora V1.3 | ??.??% | ??.??% | ??.??% |## VAE 训练与评估+### Open-Sora 1.3 VAE 训练+我们实现并训练了一个统一的时空VAE,以更好地压缩视频并保留细节。更多详情,请参阅[Open-Sora 1.3 VAE文档](/docs/vae_v1_3.md)。在运行以下命令之前,请按照我们的安装文档安装VAE和评估所需的依赖项。+``bash+## Train VAE with 8 GPUs+colossalai run --hostfile hostfile --nproc_per_node 8 scripts/train_opensoravae_v1_3.py configs/vae_v1_3/train/video_16z.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT --wandb True > logs/train_opensoravae_v1_3.log 2>&1 &+```++### Open-Sora 1.3 VAE 评估+您需要先运行VAE推理来生成视频,然后计算生成视频的评分:+```bash+VID_PATH=YOUR_CSV_PATH CUDA_VISIBLE_DEVICES=0 bash eval/vae/launch.sh pretrained_models/OpenSoraVAE_V1_3/model.pt+```++### Open-Sora 1.2 VAE 训练+<details>+<summary>View more</summary>我们训练一个由空间 VAE 和时间 VAE 组成的 VAE 管道。有关更多详细信息,请参阅[VAE 文档](vae.md)。在运行以下命令之前,请按照我们的[安装文档](installation.md)安装 VAE 和评估所需的依赖项。如果您想训练自己的 VAE,我们需要按照[数据处理](#data-processing)流程在 csv 中准备数据,然后运行以下命令。请注意,您需要根据自己的 csv 数据大小相应地调整配置文件中的训练`epochs`数量。@@ -493,7 +588,7 @@ torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py con# score calculationpython eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips```-+</details>## 贡献diff --git a/docs/zh_CN/report_v4.md b/docs/zh_CN/report_v4.mdnew file mode 100644index 0000000..56a833e--- /dev/null+++ b/docs/zh_CN/report_v4.md@@ -0,0 +1,117 @@+# Open-Sora 1.3 报告++- [视频压缩网络](#视频压缩网络)+- [升级版带位移窗口注意力的STDiT](#升级版带位移窗口注意力的STDiT)+- [简单有效的模型条件控制](#简单有效的模型条件控制)+- [评估方法](#评估方法)++在Open-Sora 1.3版本中,我们在超过60M(约85k小时)的数据上训练了一个1.1B参数的模型,训练耗时35k H100 GPU小时,支持0~113帧、360p和720p分辨率以及多种宽高比的视频生成。我们的配置如下。延续1.2版本的特性,Open-Sora 1.3同样支持图像到视频的生成和视频延展。++| | image | 49 frames | 65 frames | 81 frames | 97 frames | 113 frames |+| ---- | ----- | ---------- | ---------- | ---------- | --------- | ---------- |+| 360p | ✅ | ✅ | ✅ | ✅ | ✅ |✅ |+| 720p | ✅ | ✅ | ✅ | ✅ | ✅ |✅ |++这里✅表示在训练过程中已经见过的数据。++除了Open-Sora 1.2中引入的特性外,Open-Sora 1.3的亮点包括:++- 视频压缩网络+- 升级版带位移窗口注意力的STDiT+- 更多数据和更好的多阶段训练+- 简单有效的模型条件控制+- 更好的评估指标++以上所有改进的实现(包括训练和推理)都在Open-Sora 1.3版本中提供。以下部分将详细介绍这些改进。我们还优化了代码库和文档以使其更易于使用和开发,并添加了LLM优化器来[优化输入提示词](/README.md#gpt-4o-prompt-refinement)并支持更多语言。++## 视频压缩网络++在Open-Sora 1.2中,视频压缩架构采用了模块化方法,分别处理空间和时间维度。基于Stability AI的SDXL VAE的空间VAE压缩单个帧的空间维度。时间VAE则处理来自空间VAE的潜在表示以实现时间压缩。这种两阶段设计实现了有效的空间和时间压缩,但也带来了一些限制。这些限制包括由于固定长度输入帧而导致的长视频处理效率低下、空间和时间特征之间缺乏无缝集成,以及在训练和推理过程中更高的内存需求。++Open-Sora 1.3引入了统一的视频压缩方法。通过将空间和时间处理结合到单一框架中,并利用诸如分块3D卷积和动态帧支持等高级特性,Open-Sora 1.3实现了更好的效率、可扩展性和重建质量。以下是Open-Sora 1.3 VAE的主要改进:++**1. 统一的时空处理:** 不同于使用独立的VAE进行空间和时间压缩,Open-Sora 1.3采用单一的编码器-解码器结构同时处理这两个维度。这种方法消除了中间表示和空间-时间模块之间的冗余数据传输的需求。++**2. 分块3D卷积:** Open-Sora 1.3在时间维度上引入了分块3D卷积支持。通过将视频分解成更小的时间块,该特性实现了对更长视频序列的高效编码和解码,而不会增加内存开销。这一改进解决了Open-Sora 1.2在处理大量帧时的限制,确保了更高的时间压缩灵活性。++**3. 动态微批次和微帧处理:** Open-Sora 1.3引入了新的微批次和微帧处理机制。这实现了:(1) 自适应时间重叠:时间编码和解码过程中的重叠帧帮助减少块边界的不连续性。(2) 动态帧大小支持:不再局限于固定长度序列(如Open-Sora 1.2中的17帧),Open-Sora 1.3支持动态序列长度,使其能够适应不同的视频长度。++**4. 统一的归一化机制:** Open-Sora 1.3中的归一化过程通过可调的缩放(scale)和平移(shift)参数得到了改进,确保了不同数据集间潜在空间分布的一致性。与Open-Sora 1.2特定于固定数据集的归一化不同,这个版本引入了更通用的参数并支持特定于帧的归一化策略。++#### 改进总结++| 特性 | Open-Sora 1.2 | Open-Sora 1.3 |+|---------------|---------------------------------|----------------------------------|+| **架构** | 独立的空间和时间VAE | 统一的时空VAE |+| **分块处理** | 不支持 | 支持(分块3D卷积) |+| **帧长度支持** | 固定(17帧) | 支持动态帧长度和重叠 |+| **归一化** | 固定参数 | 可调的缩放和平移参数 |++## 包含滑动窗口注意力的STDiT++在Open-Sora 1.2取得成功的基础上,1.3版本引入了多项架构改进和新功能,以提升视频生成的质量和灵活性。本节概述了这两个版本之间的主要改进和差异。++最新的扩散模型(如Stable Diffusion 3)采用[rectified flow](https://github.com/gnobitab/RectifiedFlow)代替DDPM以获得更好的性能。虽然SD3的rectified flow训练代码未开源,但OpenSora按照SD3论文提供了训练代码实现。OpenSora 1.2从SD3引入了几个关键策略:++1. 基础的rectified flow训练,实现连续时间扩散+2. Logit-norm采样用于加速训练(遵循SD3论文第3.1节),优先在中等噪声水平采样时间步+3. 分辨率和视频长度感知的时间步采样(遵循SD3论文第5.3.2节),对更大分辨率和更长视频使用更多噪声++在OpenSora 1.3中,我们在架构、功能和性能方面进行了显著改进:++#### 1. 位移窗口注意力机制+- 引入可配置kernel_size的基于核的局部注意力,提高计算效率+- 实现类似Swin Transformer的位移窗口分区策略+- 增加带extra_pad_on_dims支持的窗口边界填充掩码处理+- 在局部窗口(时间、高度、宽度)内扩展3D相对位置编码++#### 2. 增强的位置编码+- 改进RoPE实现,将rotation_dim降至原来的1/3以适应3D场景+- 为时间、高度和宽度维度添加独立的旋转嵌入+- 实现分辨率自适应的位置编码缩放+- 可选的空间RoPE以更好地建模空间关系++#### 3. 灵活的生成能力+- 添加I2V和V2V功能,配备专门的条件控制机制+- 引入条件嵌入模块(x_embedder_cond和x_embedder_cond_mask)+- 零初始化条件嵌入以实现稳定训练+- 通过skip_temporal选项实现灵活的时序建模++#### 4. 性能优化+- 改进Flash Attention触发条件(N > 128)以提高效率+- 添加torch.scaled_dot_product_attention (SDPA)作为替代后端+- 通过改进的填充和窗口分区优化内存使用+- 通过自适应高度填充增强序列并行性++从[PixArt-Σ 2K](https://github.com/PixArt-alpha/PixArt-sigma)的适应过程保持相似,但增加了额外步骤:+[第1-7点与v1.2相同:多分辨率训练、QK-norm、rectified flow、logit-norm采样、更小的AdamW epsilon、新VAE和基础时序注意力]+#### 8. 增强的时序模块+ - 添加带位移窗口支持的基于核的局部注意力+ - 实现带分辨率自适应缩放的3D相对位置编码+ - 采用改进的初始化策略进行投影层零初始化++相比专注于基础视频生成的v1.2,v1.3在三个关键领域带来了实质性改进:**1. 质量**:通过位移窗口注意力和3D位置编码增强时空建模。**2. 灵活性**:支持I2V/V2V任务和可配置的时序建模。**3. 效率**:优化注意力计算和内存使用++这些改进在保持v1.2核心功能的同时,扩展了模型在实际应用中的能力。模型保留了使用rectified flow生成高质量图像和视频的能力,同时在条件生成和长序列建模方面获得了新的优势。++## 简单有效的模型条件控制++我们对每个视频片段计算美学分数和运动分数,并过滤掉得分较低的片段,从而得到一个视频质量更好的数据集。此外,我们将这些分数附加到标题中并用作条件控制。具体来说,我们基于预定义的范围将数值分数转换为描述性语言。美学分数转换函数基于预定义范围将数值美学分数转换为描述标签:低于4分标记为"terrible",依次通过"very poor"、"poor"、"fair"、"good"和"very good",6.5分或更高标记为"excellent"。同样,运动分数转换函数将运动强度分数映射为描述符:低于0.5分标记为"very low",依次通过"low"、"fair"、"high"和"very high",20分或更高标记为"extremely high"。我们发现这种方法可以使模型意识到这些分数并遵循分数来生成更高质量的视频。++例如,对于一个美学分数为5.5,运动分数为10,检测到的相机运动为向左平移的视频,其标题将是:++```plaintext+[Original Caption] The aesthetic score is good, the motion strength is high, camera motion: pan left.+```++在推理过程中,我们也可以使用这些分数来控制模型。对于相机运动,我们只标记了13k个高置信度的片段,相机运动检测模块已在我们的工具中发布。++## 评估方法++此前,我们仅通过人工评估来监控训练过程,因为DDPM训练损失与生成视频的质量相关性不高。然而,对于rectified flow,我们发现正如SD3所述,训练损失与生成视频的质量有很好的相关性。因此,我们持续跟踪100张图像和1k个视频的rectified flow评估损失。++我们从pixabay采样了1k个视频作为验证数据集。我们计算了不同分辨率(360p,720p)下图像和不同长度视频(49帧、65帧、81帧、97帧、113帧)的评估损失。对于每种设置,我们等距采样10个时间步。然后对所有损失取平均值。++此外,我们还在训练期间跟踪[VBench](https://vchitect.github.io/VBench-project/)分数。VBench是一个用于短视频生成的自动视频评估基准。我们使用360p 49帧视频计算vbench分数。这两个指标验证了我们的模型在训练过程中持续改进。++所有评估代码都在`eval`文件夹中发布。查看[README](/eval/README.md)获取更多详细信息。\ No newline at end of filediff --git a/docs/zh_CN/structure.md b/docs/zh_CN/structure.mdindex 1e3460e..fca78b9 100644--- a/docs/zh_CN/structure.md+++ b/docs/zh_CN/structure.md@@ -5,24 +5,56 @@```plaintextOpen-Sora├── README.md+├── assets+│ ├── images -> images used for image-conditioned generation+│ ├── demo -> images used for demo+│ ├── texts -> prompts used for text-conditioned generation+│ └── readme -> images used in README+├── configs -> Configs for training & inference+├── docker -> dockerfile for Open-Sora├── docs-│ ├── acceleration.md -> Acceleration & Speed benchmark-│ ├── command.md -> Commands for training & inference+│ ├── acceleration.md -> Report on acceleration & speed benchmark+│ ├── commands.md -> Commands for training & inference│ ├── datasets.md -> Datasets used in this project+| ├── data_processing.md -> Data pipeline documents+| ├── installation.md -> Data pipeline documents│ ├── structure.md -> This file-│ └── report_v1.md -> Report for Open-Sora v1+│ ├── config.md -> Configs for training and inference+│ ├── report_01.md -> Report for Open-Sora 1.0+│ ├── report_02.md -> Report for Open-Sora 1.1+│ ├── report_03.md -> Report for Open-Sora 1.2+│ ├── report_04.md -> Report for Open-Sora 1.3+│ ├── vae.md -> our VAE report+│ └── zh_CN -> Chinese version of the above+├── eval -> Evaluation scripts+│ ├── README.md -> Evaluation documentation+| ├── human_eval -> for human eval+| ├── I2V -> for image to video human eval+| ├── loss -> eval loss+| ├── sample.sh -> script for quickly launching inference on predefined prompts+| ├── vae -> for vae eval+| ├── vbench -> for VBench evaluation+│ └── vbench_i2v -> for VBench i2v evaluation+├── gradio -> Gradio demo related code├── scripts│ ├── train.py -> diffusion training script-│ └── inference.py -> Report for Open-Sora v1-├── configs -> Configs for training & inference+│ ├── train_opensoravae_v1_3.py -> vae v1.3 training script+│ ├── train_vae.py -> vae training script+│ ├── inference.py -> diffusion inference script+│ ├── inference_opensoravae_v1_3.py -> vae v1.3 training script+│ ├── inference_vae.py -> vae inference script+│ ├── inference_i2v.py -> image to video inference script+│ └── misc -> misc scripts, including batch size search├── opensora│ ├── __init__.py│ ├── registry.py -> Registry helper│ ├── acceleration -> Acceleration related code-│ ├── dataset -> Dataset related code+│ ├── datasets -> Dataset related code│ ├── models+│ │ ├── dit -> DiT│ │ ├── layers -> Common layers│ │ ├── vae -> VAE as image encoder+│ │ ├── vae_v1_3 -> VAE V1.3 as image encoder│ │ ├── text_encoder -> Text encoder│ │ │ ├── classes.py -> Class id encoder (inference only)│ │ │ ├── clip.py -> CLIP encoder@@ -35,6 +67,7 @@ Open-Sora│ │ ├── iddpm -> IDDPM for training and inference│ │ └── dpms -> DPM-Solver for fast inference│ └── utils+├── tests -> Tests for the project└── tools -> Tools for data processing and more```@@ -45,7 +78,18 @@ Open-Sora```plaintextOpen-Sora-└── configs -> Configs for training & inference+└── configs -> Configs for training & inferences+ ├── opensora-v1-1 -> STDiT2 related configs+ │ ├── inference+ │ │ ├── sample.py -> Sample videos and images+ │ │ └── sample-ref.py -> Sample videos with image/video condition+ │ └── train+ │ ├── stage1.py -> Stage 1 training config+ │ ├── stage2.py -> Stage 2 training config+ │ ├── stage3.py -> Stage 3 training config+ │ ├── image.py -> Illustration of image training config+ │ ├── video.py -> Illustration of video training config+ │ └── benchmark.py -> For batch size searching├── opensora -> STDiT related configs│ ├── inference│ │ ├── 16x256x256.py -> Sample videos 16 frames 256x256diff --git a/eval/I2V/launch.sh b/eval/I2V/launch.shnew file mode 100644index 0000000..c8bdd9b--- /dev/null+++ b/eval/I2V/launch.sh@@ -0,0 +1,106 @@+BASE_MODEL_PATH=$1+TRAINED_MODEL_PATH=$2+I2V_HEAD_PATH=$3+I2V_TAIL_PATH=$4+I2V_LOOP_PATH=$5+I2V_ORI_PATH=$6+++if [ -z $I2V_ORI_PATH ]; then+ I2V_ORI_PATH="assets/texts/i2v/prompts_ori.txt"+fi++if [ -z $I2V_HEAD_PATH ]; then+ I2V_HEAD_PATH="assets/texts/i2v/prompts_head.txt"+fi++if [ -z $I2V_TAIL_PATH ]; then+ I2V_TAIL_PATH="assets/texts/i2v/prompts_tail.txt"+fi++if [ -z $I2V_LOOP_PATH ]; then+ I2V_LOOP_PATH="assets/texts/i2v/prompts_loop.txt"+fi++STEP_RECORD=$(basename $TRAINED_MODEL_PATH)+if [ -z $SAVE_DIR ]; then+ SAVE_DIR="samples/i2v/test/${STEP_RECORD}"+fi+echo "save dir: ${SAVE_DIR}"++if [ -z $NUM_FRAMES ]; then+ NUM_FRAMES=49+fi+echo "num frames: ${NUM_FRAMES}"+++command="python scripts/inference_i2v.py configs/opensora-v1-3/inference/v2v.py"++# # original uncond+# ${command} --ckpt-path ${BASE_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_ORI_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name base_uncond --use-sdedit False++# # trained uncond+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_ORI_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_uncond --use-sdedit False++# trained uncond+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_ORI_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_cond_none_image1osci --use-sdedit False --use-oscillation-guidance-for-image True --image-cfg-scale 1 --cond-type "none" --start-index 1 --end-index 2++# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_ORI_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_cond_none_image1osci_bias0 --use-sdedit False --use-oscillation-guidance-for-image True --image-cfg-scale 1 --cond-type "none" --start-index 0 --end-index 2+++# trained cond: i2v_head+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_HEAD_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_head_image1.5osci_text7.5osci --use-sdedit False --cond-type i2v_head --use-oscillation-guidance-for-image True --image-cfg-scale 1.5 --use-oscillation-guidance-for-text True --cfg-scale 7.5+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_HEAD_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_head_image1.5osci_text7.5osci_bias0 --use-sdedit False --cond-type i2v_head --use-oscillation-guidance-for-image True --image-cfg-scale 1.5 --use-oscillation-guidance-for-text True --cfg-scale 7.5+++# trained cond: i2v_tail+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_TAIL_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_tail_image1.5osci_text7.5osci --use-sdedit False --cond-type i2v_tail --use-oscillation-guidance-for-image True --image-cfg-scale 1.5 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# trained cond: i2v_head+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_HEAD_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_head_image2osci_text7.5osci --use-sdedit False --cond-type i2v_head --use-oscillation-guidance-for-image True --image-cfg-scale 2 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# trained cond: i2v_tail+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_TAIL_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_tail_image2osci_text7.5osci --use-sdedit False --cond-type i2v_tail --use-oscillation-guidance-for-image True --image-cfg-scale 2 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# trained cond: i2v_head+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_HEAD_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_head_image2.5osci_text7.5osci --use-sdedit False --cond-type i2v_head --use-oscillation-guidance-for-image True --image-cfg-scale 2.5 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# trained cond: i2v_tail+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_TAIL_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_tail_image2.5osci_text7.5osci --use-sdedit False --cond-type i2v_tail --use-oscillation-guidance-for-image True --image-cfg-scale 2.5 --use-oscillation-guidance-for-text True --cfg-scale 7.5++++# trained cond: i2v_loop+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop --use-sdedit False --cond-type i2v_loop --loop 2++# # traind cond: i2v_loop+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop_image1osci_text7.5osci --use-sdedit False --cond-type i2v_loop --use-oscillation-guidance-for-image True --image-cfg-scale 1 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# # traind cond: i2v_loop+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop_image2osci_text7.5osci --use-sdedit False --cond-type i2v_loop --use-oscillation-guidance-for-image True --image-cfg-scale 2 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# # traind cond: i2v_loop+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop_image3osci_text7.5osci --use-sdedit False --cond-type i2v_loop --use-oscillation-guidance-for-image True --image-cfg-scale 3 --use-oscillation-guidance-for-text True --cfg-scale 7.5++# # traind cond: i2v_loop, cfg text osci+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop_text7osci --use-sdedit False --cond-type i2v_loop --use-oscillation-guidance-for-text True --cfg-scale 7++# # trained cond: i2v_loop, image text osci+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop_text3.5osci_image3.5osci --use-sdedit False --cond-type i2v_loop --use-oscillation-guidance-for-text True --cfg-scale 3.5 --use-oscillation-guidance-for-image Tru++# # trained cond: i2v_loop, image text oscie --image-cfg-scale 3.5+# ${command} --ckpt-path ${TRAINED_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name trained_i2v_loop_text7osci_image3.5osci --use-sdedit False --cond-type i2v_loop --use-oscillation-guidance-for-text True --cfg-scale 7 --use-oscillation-guidance-for-image True --image-cfg-scale 3.5+++++++# # base cond: i2v_head+# ${command} --ckpt-path ${BASE_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_HEAD_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name base_i2v_head --use-sdedit False --cond-type i2v_head++# # base cond: i2v_tail+# ${command} --ckpt-path ${BASE_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_TAIL_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name base_i2v_tail --use-sdedit False --cond-type i2v_tail++# # base cond: i2v_loop+# ${command} --ckpt-path ${BASE_MODEL_PATH} --save-dir ${SAVE_DIR} --prompt-path ${I2V_LOOP_PATH} --num-frames ${NUM_FRAMES} --resolution 360p --aspect-ratio 9:16 --sample-name base_i2v_loop --use-sdedit False --cond-type i2v_loopdiff --git a/eval/README.md b/eval/README.mdindex 261c21b..1689ffd 100644--- a/eval/README.md+++ b/eval/README.md@@ -19,13 +19,13 @@ Evaluate the rectified flow loss with the following commands.```bash# image-torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-2/misc/eval_loss.py --data-path /path/to/img.csv --ckpt-path /path/to/ckpt+torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-3/misc/eval_loss.py --data-path /path/to/img.csv --ckpt-path /path/to/ckpt# video-torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-2/misc/eval_loss.py --data-path /path/to/vid.csv --ckpt-path /path/to/ckpt+torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-3/misc/eval_loss.py --data-path /path/to/vid.csv --ckpt-path /path/to/ckpt# select resolution-torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-2/misc/eval_loss.py --data-path /path/to/vid.csv --ckpt-path /path/to/ckpt --resolution 720p+torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-3/misc/eval_loss.py --data-path /path/to/vid.csv --ckpt-path /path/to/ckpt --resolution 720p```To launch multiple jobs at once, use the following script.@@ -47,7 +47,7 @@ First, generate the relevant videos with the following commands:```bash# vbench task, if evaluation all set start_index to 0, end_index to 2000-bash eval/sample.sh /path/to/ckpt num_frames model_name_for_log -4 start_index end_index+bash eval/sample.sh /path/to/ckpt num_frames model_name_for_log -4 start_index end_index# Alternatively, launch 8 jobs at once (you must read the script to understand the details)bash eval/vbench/launch.sh /path/to/ckpt num_frames model_name@@ -100,7 +100,7 @@ Similarly as VBench, you can specify resolution, aspect ratio, sampling steps, f```bashbash eval/vbench_i2v/launch.sh /path/to/ckpt num_frames model_name_for_log res_value aspect_ratio_value steps_value flow_value llm_refine_value# for example-# bash eval/vbench_i2v/launch.sh /mnt/jfs-hdd/sora/checkpoints/outputs/042-STDiT3-XL-2/epoch1-global_step16200_llm_refine/ema.pt 51 042-STDiT3-XL-2 240p 9:16 30 2 True+# bash eval/vbench_i2v/launch.sh /mnt/jfs-hdd/sora/checkpoints/outputs/042-STDiT3-XL-2/epoch1-global_step16200_llm_refine/ema.pt 51 042-STDiT3-XL-2 360p 9:16 30 2 True# if no flow control, use "None" instead```@@ -110,5 +110,5 @@ Install the dependencies package following our [installation](../docs/installati```bash# metric can any one or list of: ssim, psnr, lpips, flolpips-python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir path/to/original/videos --generated_video_dir path/to/generated/videos --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips+python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir path/to/original/videos --generated_video_dir path/to/generated/videos --device cuda --sample_fps 24 --crop_size 360 --resolution 360p --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips```diff --git a/eval/loss/eval_loss.py b/eval/loss/eval_loss.pyindex f37f895..0f0746d 100644--- a/eval/loss/eval_loss.py+++ b/eval/loss/eval_loss.py@@ -47,6 +47,14 @@ def main():logger.info("Building models...")# == build text-encoder and vae ==text_encoder = build_module(cfg.text_encoder, MODELS, device=device)+ if text_encoder is not None:+ text_encoder_output_dim = text_encoder.output_dim+ text_encoder_model_max_length = text_encoder.model_max_length+ cfg.dataset.tokenize_fn = text_encoder.tokenize_fn+ else:+ text_encoder_output_dim = cfg.get("text_encoder_output_dim", 4096)+ text_encoder_model_max_length = cfg.get("text_encoder_model_max_length", 300)+vae = build_module(cfg.vae, MODELS).to(device, dtype).eval()# == build diffusion model ==@@ -58,8 +66,9 @@ def main():MODELS,input_size=latent_size,in_channels=vae.out_channels,- caption_channels=text_encoder.output_dim,- model_max_length=text_encoder.model_max_length,+ caption_channels=text_encoder_output_dim,+ model_max_length=text_encoder_model_max_length,+ enable_sequence_parallelism=cfg.get("sp_size", 1) > 1,).to(device, dtype).eval()@@ -103,13 +112,22 @@ def main():start = cfg.start_index if "start_index" in cfg else 0end = cfg.end_index if "end_index" in cfg else len(bucket_config)for i, res in enumerate(bucket_config):- if i < start or i >= end: # skip task+ if len(bucket_config) > 1 and (i < start or i >= end): # skip task+ print("skipping:", bucket_config[res])continuet_bucket = bucket_config[res]+ num_frames_index = 0for num_frames, (_, batch_size) in t_bucket.items():if batch_size is None:continue++ if len(bucket_config) == 1 and (num_frames_index < start or num_frames_index >= end): # skip task+ print("skipping:", num_frames)+ num_frames_index += 1+ continue+ else:+ num_frames_index += 1logger.info("Evaluating resolution: %s, num_frames: %s", res, num_frames)dataloader, num_steps_per_epoch, num_batch = build_dataset(res, num_frames, batch_size)if num_batch == 0:@@ -124,9 +142,11 @@ def main():for _ in tqdm(range(num_steps_per_epoch), desc=f"res: {res}, num_frames: {num_frames}, t: {t:.2f}"):batch = next(dataloader_iter)x = batch.pop("video").to(device, dtype)- y = batch.pop("text")+ batch.pop("text")x = vae.encode(x)- model_args = text_encoder.encode(y)+ input_ids = batch.pop("input_ids")+ attention_mask = batch.pop("attention_mask")+ model_args = text_encoder.encode(input_ids, attention_mask=attention_mask)# == mask ==mask = Nonediff --git a/eval/loss/launch_single.sh b/eval/loss/launch_single.shnew file mode 100644index 0000000..dfabe01--- /dev/null+++ b/eval/loss/launch_single.sh@@ -0,0 +1,10 @@+#!/bin/bash++CMD_FILE_CONFIG="eval/loss/eval_loss.py configs/opensora-pro/misc/eval_loss.py"+PORTS=$1+CKPT_PATH=$2+VID_PATH=$3++# only evaluate for 360p, 102f+RESOLUTION=360p+torchrun --master-port ${PORTS} --nproc_per_node 1 $CMD_FILE_CONFIG --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution ${RESOLUTION}diff --git a/eval/sample.sh b/eval/sample.shindex 61df175..0533d30 100644--- a/eval/sample.sh+++ b/eval/sample.sh@@ -14,7 +14,7 @@ FLOW=${10}LLM_REFINE=${11}BASE_ASPECT_RATIO=360p-ASPECT_RATIOS=(144p 240p 360p 480p 720p 1080p)+ASPECT_RATIOS=(360p 720p)# Loop through the list of aspect ratiosi=0for r in "${ASPECT_RATIOS[@]}"; do@@ -55,7 +55,10 @@ echo "DOUBLE_FRAMES=${DOUBLE_FRAMES}"echo "QUAD_FRAMES=${QUAD_FRAMES}"echo "OCT_FRAMES=${OCT_FRAMES}"-CMD="python scripts/inference.py configs/opensora-v1-2/inference/sample.py"+# CMD="python scripts/inference.py configs/opensora-v1-2/inference/sample.py"+CMD="python scripts/inference.py configs/opensora-v1-3/inference/t2v.py"+CMD_I2V="python scripts/inference_i2v.py configs/opensora-v1-3/inference/v2v.py"+if [[ $CKPT == *"ema"* ]]; thenparentdir=$(dirname $CKPT)CKPT_BASE=$(basename $parentdir)_ema@@ -69,20 +72,14 @@ DEFAULT_BS=1### Functions# called inside run_video_b-function run_image() { # 14min- # 1.1 1024x1024- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2i_samples.txt --save-dir $OUTPUT --num-frames 1 --resolution 1024 --aspect-ratio 1:1 --sample-name image_1024_1_1 --batch-size $DEFAULT_BS-- # 1.2 240x426- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2i_samples.txt --save-dir $OUTPUT --num-frames 1 --resolution 240p --aspect-ratio 9:16 --sample-name image_240p_9_16 --end-index 3 --batch-size $DEFAULT_BS-- # 1.3 512x512- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2i_samples.txt --save-dir $OUTPUT --num-frames 1 --resolution 512 --aspect-ratio 1:1 --sample-name image_t2i_512_1_1 --end-index 3 --batch-size $DEFAULT_BS- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 1 --resolution 512 --aspect-ratio 1:1 --sample-name image_t2v_512_1_1 --end-index 3 --batch-size $DEFAULT_BS- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 1 --resolution 512 --aspect-ratio 1:1 --sample-name image_short_512_1_1 --end-index 3 --batch-size $DEFAULT_BS- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 1 --resolution 512 --aspect-ratio 1:1 --sample-name image_sora_512_1_1 --end-index 3 --batch-size $DEFAULT_BS-- # 1.4 720p multi-resolution+function run_image() {+ # 360p multi-sample+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 1 --resolution 360p --aspect-ratio 1:1 --sample-name image_sora_360p_1_1 --end-index 3 --batch-size $DEFAULT_BS+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 1 --resolution 360p --aspect-ratio 1:1 --sample-name image_short_360p_1_1 --end-index 3 --batch-size $DEFAULT_BS+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 1 --resolution 360p --aspect-ratio 1:1 --sample-name image_t2v_360p_1_1 --end-index 3 --batch-size $DEFAULT_BS+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2i_samples.txt --save-dir $OUTPUT --num-frames 1 --resolution 360p --aspect-ratio 1:1 --sample-name image_t2i_360p_1_1 --end-index 3 --batch-size $DEFAULT_BS++ # 720p multi-resolution# 1:1PROMPT="Bright scene, aerial view,ancient city, fantasy, gorgeous light, mirror reflection, high detail, wide angle lens."eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 1 --resolution 720p --aspect-ratio 1:1 --sample-name image_720p_1_1@@ -100,113 +97,53 @@ function run_image() { # 14mineval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 1 --resolution 720p --aspect-ratio 2:1 --sample-name image_720p_2_1}-# for (sample, short, sora)-# for ( (4s, 720p), (8s, 480p), (16s, 360p) )--function run_video_a() { # ~ 30min ?- ### previous cmds # 42min, sample & multi-resolution- # # sample, 144p, 9:16, 2s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 2s --resolution 144p --aspect-ratio 9:16 --sample-name sample_2s_144p_9_16 --batch-size $DEFAULT_BS- # # sample, 240p, 9:16, 2s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 2s --resolution 240p --aspect-ratio 9:16 --sample-name sample_2s_240p_9_16 --batch-size $DEFAULT_BS- # # sample, 240p, 9:16, 4s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 4s --resolution 240p --aspect-ratio 9:16 --sample-name sample_4s_240p_9_16 --batch-size $DEFAULT_BS- # # sample, 240p, 9:16, 8s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 8s --resolution 240p --aspect-ratio 9:16 --sample-name sample_8s_240p_9_16 --batch-size $DEFAULT_BS- # # sample, 480p, 9:16, 2s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 2s --resolution 480p --aspect-ratio 9:16 --sample-name sample_2s_480p_9_16 --batch-size $DEFAULT_BS- # # sample, 480p, 9:16, 4s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 4s --resolution 480p --aspect-ratio 9:16 --sample-name sample_4s_480p_9_16 --batch-size $DEFAULT_BS- # # sample, 720p, 9:16, 2s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 2s --resolution 720p --aspect-ratio 9:16 --sample-name sample_2s_720p_9_16 --batch-size $DEFAULT_BS-- # sample, 720p, 9:16, 2s- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 4s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 9:16 --sample-name sample_4s_${ASPECT_RATIO_INCR_2} --batch-size $DEFAULT_BS-- # sample, 480p, 9:16, 8s- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 8s --resolution ${ASPECT_RATIO_INCR_1} --aspect-ratio 9:16 --sample-name sample_8s_${ASPECT_RATIO_INCR_1} --batch-size $DEFAULT_BS-- # sample, 360p, 9:16, 16s- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 16s --resolution ${BASE_ASPECT_RATIO} --aspect-ratio 9:16 --sample-name sample_16s_${BASE_ASPECT_RATIO} --batch-size $DEFAULT_BS+function run_video_a() {+ # sample, 720p, 9:16+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 97 --resolution 720p --aspect-ratio 9:16 --sample-name sample_97_720p --batch-size $DEFAULT_BS++ # sample, 360p, 9:16+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 97 --resolution 360p --aspect-ratio 9:16 --sample-name sample_97_360p --batch-size $DEFAULT_BS++ # sample random type, 720p, 9:16+ if [[ -z "${OPENAI_API_KEY}" ]];+ then+ echo "Error: Required environment variable 'OPENAI_API_KEY' is not set."+ exit 1+ else+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/rand_types.txt --save-dir $OUTPUT --num-frames 97 --resolution 720p --aspect-ratio 9:16 --sample-name rand_types_2s_720p --batch-size $DEFAULT_BS --llm-refine True+ fi}-function run_video_b() { # 18min + 14min = 32min, short 16x240p & 64x240p- # run image, 14min+function run_video_b() {echo "Inside run_video_b, running image samples..."run_imageecho "Inside run_video_b, running video samples..."- ### previous cmds, 18min- # # short, 240p, 9:16, 4s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 4s --resolution 240p --aspect-ratio 9:16 --sample-name short_4s_240p_9_16 --batch-size $DEFAULT_BS- # # short, 240p, 9:16, 8s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 8s --resolution 240p --aspect-ratio 9:16 --sample-name short_8s_240p_9_16 --batch-size $DEFAULT_BS-- # short, 480p, 9:16, 8s: ~24min- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 8s --resolution ${ASPECT_RATIO_INCR_1} --aspect-ratio 9:16 --sample-name short_8s_${ASPECT_RATIO_INCR_1} --batch-size $DEFAULT_BS-- # short, 360p, 9:16, 16s: ~24min- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 16s --resolution ${BASE_ASPECT_RATIO} --aspect-ratio 9:16 --sample-name short_16s_${BASE_ASPECT_RATIO} --batch-size $DEFAULT_BS+ # short, 720p, 9:16+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 97 --resolution 720p --aspect-ratio 9:16 --sample-name short_97_720p --batch-size $DEFAULT_BS+ # short, 360p, 9:16+ eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 97 --resolution 360p --aspect-ratio 9:16 --sample-name short_97_360p --batch-size $DEFAULT_BS}function run_video_c() {- ### previous cmds, 60min- # # sora, 240p, 16:9, 2s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 2s --resolution 240p --aspect-ratio 16:9 --sample-name sora_2s_240p_16_9 --batch-size $DEFAULT_BS- # # sora, 240p, 9:16, 2s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 2s --resolution 240p --aspect-ratio 9:16 --sample-name sora_2s_240p_9_16 --batch-size $DEFAULT_BS- # # sora, 240p, 9:16, 16s- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 16s --resolution 240p --aspect-ratio 9:16 --sample-name sora_16s_240p_9_16 --batch-size $DEFAULT_BS-- # short, 720p, 9:16, 2s: ~9min- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 4s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 9:16 --sample-name short_4s_${ASPECT_RATIO_INCR_2} --batch-size $DEFAULT_BS-- # sora, 360p, 9:16, 16s: ~40min- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 16s --resolution ${BASE_ASPECT_RATIO} --aspect-ratio 9:16 --sample-name sora_16s_${BASE_ASPECT_RATIO} --batch-size $DEFAULT_BS-}--function run_video_d() {- ### previous cmds, 21min + 30min = 51min- # # short, 480p, 9:16, 4s: 21min- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 4s --resolution 480p --aspect-ratio 9:16 --sample-name short_4s_480p_9_16 --batch-size $DEFAULT_BS- # # sora, 480p, 9:16, 8s, 1/3 # moved from run_video_e, 30min- # eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 8s --resolution 480p --aspect-ratio 9:16 --sample-name sora_8s_480p_9_16 --batch-size $DEFAULT_BS --start-index 0 --end-index 16-- # sora, 480p, 9:16, 8s, 1/3 # moved from run_video_e, 30min- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 8s --resolution ${ASPECT_RATIO_INCR_1} --aspect-ratio 9:16 --sample-name sora_8s_${ASPECT_RATIO_INCR_1} --batch-size $DEFAULT_BS --start-index 0 --end-index 16-}--function run_video_e() { # 90min * 2/3 = 60min- # sora, 480p, 9:16, 8s, 2/3- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 8s --resolution ${ASPECT_RATIO_INCR_1} --aspect-ratio 9:16 --sample-name sora_8s_${ASPECT_RATIO_INCR_1} --batch-size $DEFAULT_BS --start-index 16 --end-index 100-}--function run_video_f() { # 60min- # sora, 720p, 9:16, 2s- eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 4s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 9:16 --sample-name sora_4s_${ASPECT_RATIO_INCR_2} --batch-size $DEFAULT_BS-}--# --resolution 720p --aspect-ratio [16:9, 9:16, ...]--function run_video_g() { # 15min- # 720p, 2s multi-resolution+ # 720p, multi-resolution# 1:1PROMPT="A soaring drone footage captures the majestic beauty of a coastal cliff, its red and yellow stratified rock faces rich in color and against the vibrant turquoise of the sea. Seabirds can be seen taking flight around the cliff's precipices. As the drone slowly moves from different angles, the changing sunlight casts shifting shadows that highlight the rugged textures of the cliff and the surrounding calm sea. The water gently laps at the rock base and the greenery that clings to the top of the cliff, and the scene gives a sense of peaceful isolation at the fringes of the ocean. The video captures the essence of pristine natural beauty untouched by human structures."- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 1:1 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_1_1+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 1:1 --sample-name drone_cliff_prompt_720p_49_1_1# 16:9- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 16:9 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_16_9+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 16:9 --sample-name drone_cliff_prompt_720p_49_16_9# 9:16- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 9:16 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_9_16+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 9:16 --sample-name drone_cliff_prompt_720p_49_9_16# 4:3- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 4:3 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_4_3+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 4:3 --sample-name drone_cliff_prompt_720p_49_4_3# 3:4- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 3:4 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_3_4+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 3:4 --sample-name drone_cliff_prompt_720p_49_3_4# 1:2- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 1:2 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_1_2+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 1:2 --sample-name drone_cliff_prompt_720p_49_1_2# 2:1- eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --aspect-ratio 2:1 --sample-name drone_cliff_prompt_${ASPECT_RATIO_INCR_2}_2s_2_1+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 49 --resolution 720p --aspect-ratio 2:1 --sample-name drone_cliff_prompt_720p_49_2_1# add motion scoreeval $CMD --ckpt-path $CKPT --save-dir $OUTPUT --num-frames 2s --resolution ${ASPECT_RATIO_INCR_2} --sample-name motion_2s_${ASPECT_RATIO_INCR_2} --prompt \@@ -230,38 +167,11 @@ function run_video_g() { # 15min\"A stylish woman walking in the street of Tokyo. aesthetic score: 7.0\"}-# resolution -> 480p--function run_video_h() { # 61min- # 3.1 image-conditioned long video generation- eval $CMD --ckpt-path $CKPT --save-dir $OUTPUT --sample-name ref_L5C5_2s_${BASE_ASPECT_RATIO}_9_16 \- --prompt-path assets/texts/t2v_ref.txt --start-index 0 --end-index 3 \- --num-frames 2s --resolution ${BASE_ASPECT_RATIO} --aspect-ratio 9:16 \- --loop 5 --condition-frame-length 5 \- --reference-path assets/images/condition/cliff.png assets/images/condition/wave.png assets/images/condition/ship.png \- --mask-strategy "0" "0" "0" --batch-size $DEFAULT_BS-- eval $CMD --ckpt-path $CKPT --save-dir $OUTPUT --sample-name ref_L5C10_16s_${BASE_ASPECT_RATIO}_9_16 \- --prompt-path assets/texts/t2v_ref.txt --start-index 0 --end-index 3 \- --num-frames 16s --resolution ${BASE_ASPECT_RATIO} --aspect-ratio 9:16 \- --loop 5 --condition-frame-length 10 \- --reference-path assets/images/condition/cliff.png assets/images/condition/wave.png assets/images/condition/ship.png \- --mask-strategy "0" "0" "0" --batch-size $DEFAULT_BS-- # 3.2- eval $CMD --ckpt-path $CKPT --save-dir $OUTPUT --sample-name ref_L1_16s_${BASE_ASPECT_RATIO}_9_16 \- --prompt-path assets/texts/t2v_ref.txt --start-index 3 --end-index 6 \- --num-frames 16s --resolution ${BASE_ASPECT_RATIO} --aspect-ratio 9:16 \- --loop 1 \- --reference-path assets/images/condition/cliff.png "assets/images/condition/cactus-sad.png\;assets/images/condition/cactus-happy.png" https://cdn.openai.com/tmp/s/interp/d0.mp4 \- --mask-strategy "0" "0\;0,1,0,-1,1" "0,0,0,0,${QUAD_FRAMES},0.5" --batch-size $DEFAULT_BS-}-# vbench has 950 samples-VBENCH_BS=1 # 80GB-VBENCH_H=240-VBENCH_W=426+VBENCH_BS=1+VBENCH_H=360+VBENCH_W=640function run_vbench() {if [ -z ${VBENCH_RES} ] || [ -z ${VBENCH_ASP_RATIO} ]; then@@ -279,24 +189,24 @@ function run_vbench() {if [ -z ${FLOW} ]; theneval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_dimension.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2elseif [ -z ${LLM_REFINE} ]; theneval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_dimension.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} --flow ${FLOW} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --flow ${FLOW} \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2elseif [ "${FLOW}" = "None" ]; theneval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_dimension.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} --llm-refine ${LLM_REFINE} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --llm-refine ${LLM_REFINE} \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2elseeval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_dimension.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} --flow ${FLOW} --llm-refine ${LLM_REFINE} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATI --flow ${FLOW} --llm-refine ${LLM_REFINE} \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2fifi@@ -307,8 +217,8 @@ function run_vbench() {# vbench-i2v has 1120 samples-VBENCH_I2V_H=256-VBENCH_I2V_W=256+VBENCH_I2V_H=360+VBENCH_I2V_W=360function run_vbench_i2v() {if [ -z ${VBENCH_RES} ] || [ -z ${VBENCH_ASP_RATIO} ]; then@@ -326,24 +236,24 @@ function run_vbench_i2v() {if [ -z ${FLOW} ]; theneval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench_i2v --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_i2v.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2elseif [ -z ${LLM_REFINE} ]; theneval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench_i2v --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_i2v.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} --flow ${FLOW} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --flow ${FLOW} \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2elseif [ "${FLOW}" = "None" ]; theneval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench_i2v --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_i2v.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} --llm-refine ${LLM_REFINE} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --llm-refine ${LLM_REFINE} \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2elseeval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench_i2v --prompt-as-path --num-sample 5 \--prompt-path assets/texts/VBench/all_i2v.txt \- --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --num-sampling-steps ${NUM_SAMPLING_STEPS} --flow ${FLOW} --llm-refine ${LLM_REFINE} \+ --resolution $VBENCH_RES --aspect-ratio $VBENCH_ASP_RATIO --flow ${FLOW} --llm-refine ${LLM_REFINE} \--batch-size $VBENCH_BS --num-frames $NUM_FRAMES --start-index $1 --end-index $2fifi@@ -372,26 +282,6 @@ for arg in "$@"; doecho "Running video samples c..."run_video_cfi- if [[ "$arg" = -2d ]] || [[ "$arg" = --video ]]; then- echo "Running video samples d..."- run_video_d- fi- if [[ "$arg" = -2e ]] || [[ "$arg" = --video ]]; then- echo "Running video samples e..."- run_video_e- fi- if [[ "$arg" = -2f ]] || [[ "$arg" = --video ]]; then- echo "Running video samples f..."- run_video_f- fi- if [[ "$arg" = -2g ]] || [[ "$arg" = --video ]]; then- echo "Running video samples g..."- run_video_g- fi- if [[ "$arg" = -2h ]] || [[ "$arg" = --video ]]; then- echo "Running video samples h..."- run_video_h- fi# vbenchif [[ "$arg" = -4 ]] || [[ "$arg" = --vbench ]]; thenecho "Running vbench samples ..."diff --git a/eval/vae/eval_common_metric.py b/eval/vae/eval_common_metric.pyindex e2fdc00..f76bc9b 100644--- a/eval/vae/eval_common_metric.py+++ b/eval/vae/eval_common_metric.py@@ -27,7 +27,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License."""-+import jsonimport osimport os.path as ospimport sys@@ -35,9 +35,11 @@ from argparse import ArgumentDefaultsHelpFormatter, ArgumentParserimport numpy as npimport torch+import torchvision.transforms as transformsfrom decord import VideoReader, cpufrom pytorchvideo.transforms import ShortSideScalefrom torch.utils.data import DataLoader, Dataset, Subset+from torchvision.datasets.folder import pil_loaderfrom torchvision.transforms import Compose, Lambdafrom torchvision.transforms._transforms_video import CenterCropVideo@@ -58,6 +60,7 @@ except ImportError:class VideoDataset(Dataset):def __init__(self,+ type, # image or videoreal_video_dir,generated_video_dir,num_frames,@@ -66,6 +69,7 @@ class VideoDataset(Dataset):resolution=128,) -> None:super().__init__()+ self.type = typeself.real_video_files = self._combine_without_prefix(real_video_dir)self.generated_video_files = self._combine_without_prefix(generated_video_dir)self.num_frames = num_frames@@ -79,13 +83,28 @@ class VideoDataset(Dataset):def __getitem__(self, index):if index >= len(self):raise IndexError+real_video_file = self.real_video_files[index]generated_video_file = self.generated_video_files[index]print(real_video_file, generated_video_file)- real_video_tensor = self._load_video(real_video_file)- generated_video_tensor = self._load_video(generated_video_file)++ if self.type == "video":+ real_video_tensor = self._load_video(real_video_file)+ generated_video_tensor = self._load_video(generated_video_file)+ else:+ real_video_tensor = self._load_image(real_video_file)+ generated_video_tensor = self._load_image(generated_video_file)+return {"real": real_video_tensor, "generated": generated_video_tensor}+ def _load_image(self, image_path):+ image = pil_loader(image_path)+ transform = transforms.Compose([transforms.ToTensor()])+ image = transform(image)+ video = image.unsqueeze(0)+ video = video.permute(1, 0, 2, 3) # TCHW -> CTHW+ return _preprocess(video, short_size=self.short_size, crop_size=self.crop_size)+def _load_video(self, video_path):num_frames = self.num_framessample_rate = self.sample_rate@@ -169,6 +188,9 @@ def calculate_common_metric(args, dataloader, device):def main():parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)+ parser.add_argument(+ "--type", type=str, choices=["video", "image"], default="video", help="whether evaluating images or videos"+ )parser.add_argument("--batch_size", type=int, default=2, help="Batch size to use")parser.add_argument("--real_video_dir", type=str, help=("the path of real videos`"))parser.add_argument("--generated_video_dir", type=str, help=("the path of generated videos`"))@@ -188,7 +210,7 @@ def main():# parser.add_argument("--metric", type=str, default="fvd",choices=['fvd','psnr','ssim','lpips', 'flolpips'])parser.add_argument("--metric", nargs="+", default=[])parser.add_argument("--fvd_method", type=str, default="styleganv", choices=["styleganv", "videogpt"])-+ parser.add_argument("--res_dir", type=str, default=None, help="dir to save result json")args = parser.parse_args()if args.device is None:@@ -210,6 +232,7 @@ def main():num_workers = args.num_workersdataset = VideoDataset(+ args.type,args.real_video_dir,args.generated_video_dir,num_frames=args.num_frames,@@ -225,8 +248,18 @@ def main():dataloader = DataLoader(dataset, args.batch_size, num_workers=num_workers, pin_memory=True)metric_score = calculate_common_metric(args, dataloader, device)+ for k, v in metric_score.items():+ metric_score[k] = round(v, 3)print("metric: ", args.metric, " ", metric_score)+ if args.res_dir:+ output_file_path = os.path.join(+ args.res_dir, "metric_" + str(args.num_frames) + "f_" + str(args.resolution) + "res.json"+ )+ with open(output_file_path, "w") as outfile:+ json.dump(metric_score, outfile, indent=4, sort_keys=True)+ print(f"metric results saved to: {output_file_path}")+if __name__ == "__main__":main()diff --git a/eval/vae/launch.sh b/eval/vae/launch.shnew file mode 100644index 0000000..9d71de8--- /dev/null+++ b/eval/vae/launch.sh@@ -0,0 +1,55 @@+#!/bin/bash+CKPT_PATH=$1++if [ -z $IMG_PATH ]; then+ IMG_PATH="/mnt/jfs-hdd/sora/meta/validation/img_1k.csv"+fi++if [ -z $VID_PATH ]; then+ VID_PATH="/mnt/jfs-hdd/sora/meta/validation/vid_100.csv"+fi++if [ -z $NUM_FRAMES ]; then+ NUM_FRAMES=17+fi++if [ -z $FORCE_HUGGINGFACE ]; then+ FORCE_HUGGINGFACE=False+fi++if [ -z $CKPT_PATH ]; then # huggingface model+ STEP_RECORD=epoch0-global_step0+ LOG_DIR=outputs/OpenSoraVAE_V1_3/eval+ FORCE_HUGGINGFACE=True+ CKPT_PATH=pretrained_models/OpenSoraVAE_V1_3+else+ if [[ -d $CKPT_PATH ]] ; then+ STEP_RECORD=$(basename $CKPT_PATH)+ elif [[ -f $CKPT_PATH ]]; then+ STEP_RECORD=$(basename $(dirname $CKPT_PATH))+ else+ echo "$CKPT_PATH is not valid";+ exit 1+ fi+ LOG_DIR=$(dirname $CKPT_PATH)/eval+fi+++echo "saving losses and metrics to $LOG_DIR"+echo "video path: ${VID_PATH}"+mkdir -p $LOG_DIR++# generate video, 256x256+torchrun --standalone --nproc_per_node=1 scripts/inference_opensoravae_v1_3.py configs/vae_v1_3/inference/video_16z_256x256.py --data-path $VID_PATH --save-dir samples/opensoravae_v1_3/${STEP_RECORD}/${NUM_FRAMES}x256x256 --ckpt-path ${CKPT_PATH} --num-frames $NUM_FRAMES --force-huggingface ${FORCE_HUGGINGFACE}+# calc metrics, 17x256x256+python eval/vae/eval_common_metric.py --batch_size 4 --real_video_dir samples/opensoravae_v1_3/${STEP_RECORD}/${NUM_FRAMES}x256x256_ori --generated_video_dir samples/opensoravae_v1_3/${STEP_RECORD}/${NUM_FRAMES}x256x256_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames $NUM_FRAMES --sample_rate 1 --metric ssim psnr lpips flolpips --type video --res_dir ${LOG_DIR}++# # generate video, 512x512+torchrun --standalone --nproc_per_node=1 scripts/inference_opensoravae_v1_3.py configs/vae_v1_3/inference/video_16z_512x512.py --data-path $VID_PATH --save-dir samples/opensoravae_v1_3/${STEP_RECORD}/${NUM_FRAMES}x512x512 --ckpt-path ${CKPT_PATH} --num-frames $NUM_FRAMES --force-huggingface ${FORCE_HUGGINGFACE}+# # calc metrics, 17x512x512+python eval/vae/eval_common_metric.py --batch_size 4 --real_video_dir samples/opensoravae_v1_3/${STEP_RECORD}/${NUM_FRAMES}x512x512_ori --generated_video_dir samples/opensoravae_v1_3/${STEP_RECORD}/${NUM_FRAMES}x512x512_rec --device cuda --sample_fps 24 --crop_size 512 --resolution 512 --num_frames $NUM_FRAMES --sample_rate 1 --metric ssim psnr lpips flolpips --type video --res_dir ${LOG_DIR}++# # generate image, 1024x1024+torchrun --standalone --nproc_per_node=1 scripts/inference_opensoravae_v1_3.py configs/vae_v1_3/inference/image_16z.py --data-path $IMG_PATH --save-dir samples/opensoravae_v1_3/${STEP_RECORD}/1x1024x1024 --ckpt-path ${CKPT_PATH} --num-frames 1 --force-huggingface ${FORCE_HUGGINGFACE}+# # calc metrics, 1x1024x1024+python eval/vae/eval_common_metric.py --batch_size 4 --real_video_dir samples/opensoravae_v1_3/${STEP_RECORD}/1x1024x1024_ori --generated_video_dir samples/opensoravae_v1_3/${STEP_RECORD}/1x1024x1024_rec --device cuda --sample_fps 1 --crop_size 1024 --resolution 1024 --num_frames 1 --sample_rate 1 --metric ssim psnr lpips --type image --res_dir ${LOG_DIR}diff --git a/eval/vbench/calc_vbench.py b/eval/vbench/calc_vbench.pyindex bce29a5..e5570a1 100644--- a/eval/vbench/calc_vbench.py+++ b/eval/vbench/calc_vbench.py@@ -3,6 +3,7 @@ import osimport timeimport torch+from vbench import VBenchfull_info_path = "eval/vbench/VBench_full_info.json"diff --git a/eval/vbench/launch.sh b/eval/vbench/launch.shindex df4d06d..fa0aba0 100644--- a/eval/vbench/launch.sh+++ b/eval/vbench/launch.sh@@ -16,7 +16,9 @@ if [[ $CKPT == *"ema"* ]]; thenelseCKPT_BASE=$(basename $CKPT)fi-LOG_BASE=$(dirname $CKPT)/eval+# LOG_BASE=$(dirname $CKPT)/eval+LOG_BASE=./sample/eval+echo "Logging to $LOG_BASE"GPUS=(0 1 2 3 4 5 6 7)diff --git a/eval/vbench_i2v/calc_vbench_i2v.py b/eval/vbench_i2v/calc_vbench_i2v.pyindex e838c2d..b8d5901 100644--- a/eval/vbench_i2v/calc_vbench_i2v.py+++ b/eval/vbench_i2v/calc_vbench_i2v.py@@ -3,6 +3,7 @@ import osimport timeimport torch+from vbench import VBenchfrom vbench2_beta_i2v import VBenchI2Vdiff --git a/gradio/README.md b/gradio/README.mdindex 5dddd47..9c600a1 100644--- a/gradio/README.md+++ b/gradio/README.md@@ -9,8 +9,8 @@ app_file: app.pypinned: falselicense: apache-2.0preload_from_hub:- - hpcai-tech/OpenSora-STDiT-v3- - hpcai-tech/OpenSora-VAE-v1.2+ - hpcai-tech/OpenSora-STDiT-v4+ - hpcai-tech/OpenSora-VAE-v1.3- DeepFloyd/t5-v1_1-xxl---diff --git a/gradio/app.py b/gradio/app.pyindex da5b188..3351f8a 100644--- a/gradio/app.py+++ b/gradio/app.py@@ -19,12 +19,19 @@ import torchimport gradio as gr-MODEL_TYPES = ["v1.2-stage3"]+MODEL_TYPES = ["v1.3"]WATERMARK_PATH = "./assets/images/watermark/watermark.png"CONFIG_MAP = {- "v1.2-stage3": "configs/opensora-v1-2/inference/sample.py",+ "v1.3": "configs/opensora-v1-3/inference/t2v.py",+ "v1.3_i2v": "configs/opensora-v1-3/inference/v2v.py",+}+HF_STDIT_MAP = {+ "t2v": {+ "360p": "/home/guoxinying/open_source_video_ocean_V1/OpenSora-STDiT-v4-360p",+ "720p": "/home/guoxinying/open_source_video_ocean_V1/OpenSora-STDiT-v4",+ },+ "i2v": "/home/guoxinying/open_source_video_ocean_V1/OpenSora-STDiT-v4-i2v",}-HF_STDIT_MAP = {"v1.2-stage3": "hpcai-tech/OpenSora-STDiT-v3"}# ============================@@ -82,27 +89,39 @@ def read_config(config_path):return Config.fromfile(config_path)-def build_models(model_type, config, enable_optimization=False):+def build_models(mode, resolution, enable_optimization=False):"""- Build the models for the given model type and configuration.+ Build the models for the given mode, resolution, and configuration."""# build vaefrom opensora.registry import MODELS, build_module+ if mode == "i2v":+ config = read_config(CONFIG_MAP["v1.3_i2v"])+ else:+ config = read_config(CONFIG_MAP["v1.3"])+vae = build_module(config.vae, MODELS).cuda()# build text encodertext_encoder = build_module(config.text_encoder, MODELS) # T5 must be fp32text_encoder.t5.model = text_encoder.t5.model.cuda()+ # Determine model weights based on mode and resolution+ if mode == "i2v":+ weight_path = HF_STDIT_MAP["i2v"]+ else: # t2v+ weight_path = HF_STDIT_MAP["t2v"].get(resolution, None)+ if not weight_path:+ raise ValueError(f"Unsupported resolution {resolution} for mode {mode}")+# build stdit- # we load model from HuggingFace directly so that we don't need to- # handle model download logic in HuggingFace Spacefrom opensora.models.stdit.stdit3 import STDiT3model_kwargs = {k: v for k, v in config.model.items() if k not in ("type", "from_pretrained", "force_huggingface")}- stdit = STDiT3.from_pretrained(HF_STDIT_MAP[model_type], **model_kwargs)- stdit = stdit.cuda()++ print("Load STDIT3 from ", weight_path)+ stdit = STDiT3.from_pretrained(weight_path, **model_kwargs).cuda()# build schedulerfrom opensora.registry import SCHEDULERS@@ -112,21 +131,21 @@ def build_models(model_type, config, enable_optimization=False):# hack for classifier-free guidancetext_encoder.y_embedder = stdit.y_embedder- # move modelst to device+ # move models to devicevae = vae.to(torch.bfloat16).eval()text_encoder.t5.model = text_encoder.t5.model.eval() # t5 must be in fp32stdit = stdit.to(torch.bfloat16).eval()# clear cudatorch.cuda.empty_cache()- return vae, text_encoder, stdit, scheduler+ return vae, text_encoder, stdit, scheduler, configdef parse_args():parser = argparse.ArgumentParser()parser.add_argument("--model-type",- default="v1.2-stage3",+ default="v1.3",choices=MODEL_TYPES,help=f"The type of model to run for the Gradio App, can only be {MODEL_TYPES}",)@@ -184,6 +203,8 @@ from opensora.utils.inference_utils import (prepare_multi_resolution_info,refine_prompts_by_openai,split_prompt,+ prep_ref_and_update_mask_in_loop,+ prep_ref_and_mask)from opensora.utils.misc import to_torch_dtype@@ -192,9 +213,8 @@ dtype = to_torch_dtype(config.dtype)device = torch.device("cuda")# build model-vae, text_encoder, stdit, scheduler = build_models(- args.model_type, config, enable_optimization=args.enable_optimization-)+def initialize_models(mode, resolution):+ return build_models(mode, resolution, enable_optimization=args.enable_optimization)def run_inference(@@ -220,6 +240,13 @@ def run_inference(gr.Warning("Your prompt is empty, please enter a valid prompt")return None+ # Dynamically choose mode based on reference image+ if reference_image is not None and mode != "Text2Image":+ mode = "i2v"++ # Initialize models+ vae, text_encoder, stdit, scheduler, config = initialize_models(mode, resolution)+torch.manual_seed(seed)with torch.inference_mode():# ======================@@ -229,6 +256,16 @@ def run_inference(# frame_interval must be 1 so we ignore it hereimage_size = get_image_size(resolution, aspect_ratio)+ use_sdedit = config.get("use_sdedit", False)+ use_oscillation_guidance_for_text = config.get("use_oscillation_guidance_for_text", None)+ use_oscillation_guidance_for_image = config.get("use_oscillation_guidance_for_image", None)++ cond_type = config.get("cond_type", None)+ cond_type = None if cond_type == "none" else cond_type+ mask_index = None+ ref = None+ image_cfg_scale = None+# compute generation parametersif mode == "Text2Image":num_frames = 1@@ -237,8 +274,8 @@ def run_inference(num_frames = config.num_framesnum_frames = get_num_frames(length)- condition_frame_length = int(num_frames / 17 * 5 / 3)- condition_frame_edit = 0.0+ condition_frame_length = config.get("condition_frame_length", 5)+ condition_frame_edit = config.get("condition_frame_edit", 0.0)input_size = (num_frames, *image_size)latent_size = vae.get_latent_size(input_size)@@ -248,11 +285,17 @@ def run_inference(# == prepare mask strategy ==if mode == "Text2Image":mask_strategy = [None]+ mask_index = []elif mode == "Text2Video":if reference_image is not None:mask_strategy = ["0"]+ mask_index = [0]else:mask_strategy = [None]+ mask_index = []+ elif mode == "i2v":+ mask_strategy = ["0"]+ mask_index = [0]else:raise ValueError(f"Invalid mode: {mode}")@@ -270,6 +313,17 @@ def run_inference(refs = [temp_file.name]else:refs = [""]+ elif mode == "i2v":+ if reference_image is not None:+ # save image to disk+ from PIL import Image++ im = Image.fromarray(reference_image)+ temp_file = NamedTemporaryFile(suffix=".png")+ im.save(temp_file.name)+ refs = [temp_file.name]+ else:+ refs = [""]else:raise ValueError(f"Invalid mode: {mode}")@@ -280,6 +334,13 @@ def run_inference(# == get reference for condition ==refs = collect_references_batch(refs, vae, image_size)+ target_shape = [len(batch_prompts), vae.out_channels, *latent_size]+ if mode == "i2v":+ image_cfg_scale = config.get("image_cfg_scale", 7.5)+ ref, mask_index = prep_ref_and_mask(+ cond_type, condition_frame_length, refs, target_shape, num_loop, device, dtype+ )+# == multi-resolution info ==model_args = prepare_multi_resolution_info(multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype@@ -330,20 +391,22 @@ def run_inference(# Generate image/video# =========================video_clips = []-for loop_i in range(num_loop):# 4.4 sample in hidden spacebatch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)# == loop ==- if loop_i > 0:- refs, mask_strategy = append_generated(- vae, video_clips[-1], refs, mask_strategy, loop_i, condition_frame_length, condition_frame_edit- )+ # if loop_i > 0:+ # refs, mask_strategy = append_generated(+ # vae, video_clips[-1], refs, mask_strategy, loop_i, condition_frame_length, condition_frame_edit+ # )# == sampling ==z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)- masks = apply_mask_strategy(z, refs, mask_strategy, loop_i, align=align)+ masks = apply_mask_strategy(z, refs, mask_strategy, loop_i, align=align) if mask_index is None else None+ x_cond_mask = torch.zeros(len(batch_prompts), vae.out_channels, *latent_size, device=device).to(dtype) if mask_index is not None else None+ if x_cond_mask is not None and mask_index is not None:+ x_cond_mask[:, :, mask_index, :, :] = 1.0# 4.6. diffusion sampling# hack to update num_sampling_steps and cfg_scale@@ -357,13 +420,47 @@ def run_inference(stdit,text_encoder,z=z,+ z_cond=ref,+ z_cond_mask=x_cond_mask,prompts=batch_prompts_loop,device=device,additional_args=model_args,progress=True,mask=masks,+ mask_index=mask_index,+ image_cfg_scale=image_cfg_scale,+ use_sdedit=use_sdedit,+ use_oscillation_guidance_for_text=use_oscillation_guidance_for_text,+ use_oscillation_guidance_for_image=use_oscillation_guidance_for_image,)- samples = vae.decode(samples.to(dtype), num_frames=num_frames)++ if loop_i > 1: # process conditions for subsequent loop+ if cond_type is not None: # i2v or v2v+ is_last_loop = loop_i == loop_i - 1+ ref, mask_index = prep_ref_and_update_mask_in_loop(+ cond_type,+ condition_frame_length,+ samples,+ refs,+ target_shape,+ is_last_loop,+ device,+ dtype,+ )++ else:+ refs, mask_strategy = append_generated(+ vae,+ samples,+ refs,+ mask_strategy,+ loop_i,+ condition_frame_length,+ condition_frame_edit,+ is_latent=True,+ )++ # samples = vae.decode(samples.to(dtype), num_frames=num_frames)video_clips.append(samples)# =========================@@ -371,16 +468,23 @@ def run_inference(# =========================video_clips = [val[0] for val in video_clips]for i in range(1, num_loop):- video_clips[i] = video_clips[i][:, dframe_to_frame(condition_frame_length) :]+ video_clips[i] = video_clips[i][:, condition_frame_length:]video = torch.cat(video_clips, dim=1)++ t_cut = max(video.size(1) // 5 * 5, 1)+ if t_cut < video.size(1):+ video = video[:, :t_cut]++ video = vae.decode(video.to(dtype), num_frames=t_cut * 17 // 5).squeeze(0)+current_datetime = datetime.datetime.now()timestamp = current_datetime.timestamp()+save_path = os.path.join(args.output, f"output_{timestamp}")saved_path = save_sample(video, save_path=save_path, fps=24)torch.cuda.empty_cache()# add watermark- # all watermarked videos should have a _watermarked suffixif mode != "Text2Image" and os.path.exists(WATERMARK_PATH):watermarked_path = saved_path.replace(".mp4", "_watermarked.mp4")success = add_watermark(saved_path, WATERMARK_PATH, watermarked_path)@@ -392,6 +496,7 @@ def run_inference(return saved_path+@spaces.GPU(duration=200)def run_image_inference(prompt_text,@@ -412,24 +517,24 @@ def run_image_inference(cfg_scale,):return run_inference(- "Text2Image",- prompt_text,- resolution,- aspect_ratio,- length,- motion_strength,- aesthetic_score,- use_motion_strength,- use_aesthetic_score,- camera_motion,- reference_image,- refine_prompt,- fps,- num_loop,- seed,- sampling_steps,- cfg_scale,- )+ "Text2Image",+ prompt_text,+ resolution,+ aspect_ratio,+ length,+ motion_strength,+ aesthetic_score,+ use_motion_strength,+ use_aesthetic_score,+ camera_motion,+ reference_image,+ refine_prompt,+ fps,+ num_loop,+ seed,+ sampling_steps,+ cfg_scale,+ )@spaces.GPU(duration=200)@@ -520,8 +625,8 @@ def main():gr.Markdown("## Basic Settings")resolution = gr.Radio(- choices=["144p", "240p", "360p", "480p", "720p"],- value="480p",+ choices=["360p", "720p"],+ value="720p",label="Resolution",)aspect_ratio = gr.Radio(@@ -530,10 +635,10 @@ def main():label="Aspect Ratio (H:W)",)length = gr.Radio(- choices=["2s", "4s", "8s", "16s"],- value="2s",- label="Video Length",- info="only effective for video generation, 8s may fail as Hugging Face ZeroGPU has the limitation of max 200 seconds inference time.",+ choices=[1, 49, 65, 81, 97, 113],+ value=97,+ label="Video Length (Number of Frames)",+ info="Setting the number of frames to 1 indicates image generation instead of video generation.",)with gr.Row():@@ -544,24 +649,20 @@ def main():with gr.Row():with gr.Column():- motion_strength = gr.Slider(- value=5,- minimum=0,- maximum=100,- step=1,+ motion_strength = gr.Radio(+ choices=["very low", "low", "fair", "high", "very high", "extremely high"],+ value="fair",label="Motion Strength",- info="only effective for video generation",+ info="Only effective for video generation",)- use_motion_strength = gr.Checkbox(value=False, label="Enable")+ use_motion_strength = gr.Checkbox(value=True, label="Enable")with gr.Column():- aesthetic_score = gr.Slider(- value=6.5,- minimum=4,- maximum=7,- step=0.1,+ aesthetic_score = gr.Radio(+ choices=["terrible", "very poor", "poor", "fair", "good", "very good", "excellent"],+ value="excellent",label="Aesthetic",- info="effective for text & video generation",+ info="Effective for text & video generation",)use_aesthetic_score = gr.Checkbox(value=True, label="Enable")@@ -623,6 +724,7 @@ def main():],outputs=reference_image,)+video_gen_button.click(fn=run_video_inference,inputs=[diff --git a/notebooks/inference.ipynb b/notebooks/inference.ipynbdeleted file mode 100644index 170d55b..0000000--- a/notebooks/inference.ipynb+++ /dev/null@@ -1,478 +0,0 @@-{- "cells": [- {- "cell_type": "markdown",- "metadata": {},- "source": [- "# Inference for OpenSora"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Define global variables. You should change the following variables according to your setting."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "# global variables\n",- "ROOT = \"..\"\n",- "cfg_path = f\"{ROOT}/configs/opensora-v1-2/inference/sample.py\"\n",- "ckpt_path = \"/home/lishenggui/projects/sora/Open-Sora-dev/outputs/207-STDiT3-XL-2/epoch0-global_step9000/\"\n",- "vae_path = f\"{ROOT}/pretrained_models/vae-pipeline\"\n",- "save_dir = f\"{ROOT}/samples/samples_notebook/\"\n",- "device = \"cuda:0\""- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Import necessary libraries and load the models."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "import os\n",- "from pprint import pformat\n",- "\n",- "import colossalai\n",- "import torch\n",- "import torch.distributed as dist\n",- "from colossalai.cluster import DistCoordinator\n",- "from mmengine.runner import set_random_seed\n",- "from tqdm.notebook import tqdm\n",- "\n",- "from opensora.acceleration.parallel_states import set_sequence_parallel_group\n",- "from opensora.datasets import save_sample, is_img\n",- "from opensora.datasets.aspect import get_image_size, get_num_frames\n",- "from opensora.models.text_encoder.t5 import text_preprocessing\n",- "from opensora.registry import MODELS, SCHEDULERS, build_module\n",- "from opensora.utils.config_utils import read_config\n",- "from opensora.utils.inference_utils import (\n",- " append_generated,\n",- " apply_mask_strategy,\n",- " collect_references_batch,\n",- " extract_json_from_prompts,\n",- " extract_prompts_loop,\n",- " get_save_path_name,\n",- " load_prompts,\n",- " prepare_multi_resolution_info,\n",- ")\n",- "from opensora.utils.misc import all_exists, create_logger, is_distributed, is_main_process, to_torch_dtype"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "torch.set_grad_enabled(False)\n",- "\n",- "# == parse configs ==\n",- "cfg = read_config(cfg_path)\n",- "cfg.model.from_pretrained = ckpt_path\n",- "cfg.vae.from_pretrained = vae_path\n",- "\n",- "# == device and dtype ==\n",- "cfg_dtype = cfg.get(\"dtype\", \"fp32\")\n",- "assert cfg_dtype in [\"fp16\", \"bf16\", \"fp32\"], f\"Unknown mixed precision {cfg_dtype}\"\n",- "dtype = to_torch_dtype(cfg.get(\"dtype\", \"bf16\"))\n",- "torch.backends.cuda.matmul.allow_tf32 = True\n",- "torch.backends.cudnn.allow_tf32 = True\n",- "\n",- "set_random_seed(seed=cfg.get(\"seed\", 1024))"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "# == build text-encoder and vae ==\n",- "text_encoder = build_module(cfg.text_encoder, MODELS, device=device)\n",- "vae = build_module(cfg.vae, MODELS).to(device, dtype).eval()\n",- "\n",- "# == build diffusion model ==\n",- "input_size = (None, None, None)\n",- "latent_size = vae.get_latent_size(input_size)\n",- "model = (\n",- " build_module(\n",- " cfg.model,\n",- " MODELS,\n",- " input_size=latent_size,\n",- " in_channels=vae.out_channels,\n",- " caption_channels=text_encoder.output_dim,\n",- " model_max_length=text_encoder.model_max_length,\n",- " )\n",- " .to(device, dtype)\n",- " .eval()\n",- ")\n",- "text_encoder.y_embedder = model.y_embedder # HACK: for classifier-free guidance"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Define inference function."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "start_idx = 0\n",- "multi_resolution = cfg.get(\"multi_resolution\", None)\n",- "batch_size = cfg.get(\"batch_size\", 1)\n",- "\n",- "\n",- "def inference(\n",- " prompts=cfg.get(\"prompt\", None),\n",- " image_size=None,\n",- " num_frames=None,\n",- " resolution=None,\n",- " aspect_ratio=None,\n",- " mask_strategy=None,\n",- " reference_path=None,\n",- " num_sampling_steps=None,\n",- " cfg_scale=None,\n",- " seed=None,\n",- " fps=cfg.fps,\n",- " num_sample=cfg.get(\"num_sample\", 1),\n",- " loop=cfg.get(\"loop\", 1),\n",- " condition_frame_length=cfg.get(\"condition_frame_length\", 5),\n",- " align=cfg.get(\"align\", None),\n",- " sample_name=cfg.get(\"sample_name\", None),\n",- " prompt_as_path=cfg.get(\"prompt_as_path\", False),\n",- " disable_progress=False,\n",- "):\n",- " global start_idx\n",- " os.makedirs(save_dir, exist_ok=True)\n",- " if seed is not None:\n",- " set_random_seed(seed=seed)\n",- " if not isinstance(prompts, list):\n",- " prompts = [prompts]\n",- " if mask_strategy is None:\n",- " mask_strategy = [\"\"] * len(prompts)\n",- " if reference_path is None:\n",- " reference_path = [\"\"] * len(prompts)\n",- " save_fps = cfg.fps // cfg.get(\"frame_interval\", 1)\n",- " if num_sampling_steps is not None:\n",- " cfg.scheduler[\"num_sampling_steps\"] = num_sampling_steps\n",- " if cfg_scale is not None:\n",- " cfg.scheduler[\"scale\"] = cfg_scale\n",- " scheduler = build_module(cfg.scheduler, SCHEDULERS)\n",- " ret_path = []\n",- "\n",- " # == prepare video size ==\n",- " if image_size is None:\n",- " assert (\n",- " resolution is not None and aspect_ratio is not None\n",- " ), \"resolution and aspect_ratio must be provided if image_size is not provided\"\n",- " image_size = get_image_size(resolution, aspect_ratio)\n",- " num_frames = get_num_frames(num_frames)\n",- " input_size = (num_frames, *image_size)\n",- " latent_size = vae.get_latent_size(input_size)\n",- "\n",- " # == Iter over all samples ==\n",- " for i in tqdm(range(0, len(prompts), batch_size), disable=disable_progress):\n",- " # == prepare batch prompts ==\n",- " batch_prompts = prompts[i : i + batch_size]\n",- " ms = mask_strategy[i : i + batch_size]\n",- " refs = reference_path[i : i + batch_size]\n",- "\n",- " batch_prompts, refs, ms = extract_json_from_prompts(batch_prompts, refs, ms)\n",- " refs = collect_references_batch(refs, vae, image_size)\n",- "\n",- " # == multi-resolution info ==\n",- " model_args = prepare_multi_resolution_info(\n",- " multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype\n",- " )\n",- "\n",- " # == Iter over number of sampling for one prompt ==\n",- " for k in range(num_sample):\n",- " # == prepare save paths ==\n",- " save_paths = [\n",- " get_save_path_name(\n",- " save_dir,\n",- " sample_name=sample_name,\n",- " sample_idx=start_idx + idx,\n",- " prompt=batch_prompts[idx],\n",- " prompt_as_path=prompt_as_path,\n",- " num_sample=num_sample,\n",- " k=k,\n",- " )\n",- " for idx in range(len(batch_prompts))\n",- " ]\n",- "\n",- " # NOTE: Skip if the sample already exists\n",- " # This is useful for resuming sampling VBench\n",- " if prompt_as_path and all_exists(save_paths):\n",- " continue\n",- "\n",- " # == Iter over loop generation ==\n",- " video_clips = []\n",- " for loop_i in range(loop):\n",- " batch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)\n",- " batch_prompts_cleaned = [text_preprocessing(prompt) for prompt in batch_prompts_loop]\n",- "\n",- " # == loop ==\n",- " if loop_i > 0:\n",- " refs, ms = append_generated(vae, video_clips[-1], refs, ms, loop_i, condition_frame_length)\n",- "\n",- " # == sampling ==\n",- " z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)\n",- " masks = apply_mask_strategy(z, refs, ms, loop_i, align=align)\n",- " samples = scheduler.sample(\n",- " model,\n",- " text_encoder,\n",- " z=z,\n",- " prompts=batch_prompts_cleaned,\n",- " device=device,\n",- " additional_args=model_args,\n",- " progress=False,\n",- " mask=masks,\n",- " )\n",- " samples = vae.decode(samples.to(dtype), num_frames=num_frames)\n",- " video_clips.append(samples)\n",- "\n",- " # == save samples ==\n",- " if is_main_process():\n",- " for idx, batch_prompt in enumerate(batch_prompts):\n",- " save_path = save_paths[idx]\n",- " video = [video_clips[i][idx] for i in range(loop)]\n",- " for i in range(1, loop):\n",- " video[i] = video[i][:, condition_frame_length:]\n",- " video = torch.cat(video, dim=1)\n",- " path = save_sample(\n",- " video,\n",- " fps=save_fps,\n",- " save_path=save_path,\n",- " verbose=False,\n",- " )\n",- " ret_path.append(path)\n",- " start_idx += len(batch_prompts)\n",- " return ret_path"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "from IPython.display import Video, Image, display\n",- "\n",- "\n",- "def display_results(paths):\n",- " for path in paths:\n",- " if is_img(path):\n",- " display(Image(path))\n",- " else:\n",- " display(Video(path, embed=True))\n",- "\n",- "\n",- "def reset_start_idx():\n",- " global start_idx\n",- " start_idx = 0\n",- "\n",- "\n",- "ALL_ASPECT_RATIO = [\"1:1\", \"16:9\", \"9:16\", \"3:4\", \"4:3\", \"1:2\", \"2:1\"]\n",- "\n",- "\n",- "def inference_all_aspects(prompts, resolution, num_frames, *args, **kwargs):\n",- " paths = []\n",- " for aspect_ratio in tqdm(ALL_ASPECT_RATIO):\n",- " paths.extend(\n",- " inference(\n",- " prompts,\n",- " resolution=resolution,\n",- " num_frames=num_frames,\n",- " aspect_ratio=aspect_ratio,\n",- " disable_progress=True,\n",- " *args,\n",- " **kwargs\n",- " )\n",- " )\n",- " return paths"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": []- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "## Inference for OpenSora"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Sample code for inference for OpenSora."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "paths = inference(\n",- " [\"a man.\", \"a woman\"],\n",- " resolution=\"240p\",\n",- " aspect_ratio=\"1:1\",\n",- " num_frames=\"1x\",\n",- " num_sampling_steps=30,\n",- " cfg_scale=7.0,\n",- ")\n",- "display_results(paths)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Sample all aspect ratios."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "PROMPT = \"a boy.\"\n",- "paths = inference_all_aspects(\n",- " PROMPT,\n",- " resolution=\"240p\",\n",- " num_frames=\"1x\",\n",- " num_sampling_steps=30,\n",- " cfg_scale=7.0,\n",- ")\n",- "display_results(paths)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Sample all resolution and length."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "PROMPT = \"a boy.\"\n",- "sample_cfg = {\n",- " \"144p\": [1, \"1x\", \"2x\", \"4x\", \"8x\"],\n",- " \"240p\": [1, \"1x\", \"2x\", \"4x\", \"8x\"],\n",- " \"360p\": [1, \"1x\", \"2x\", \"4x\"],\n",- " \"480p\": [1, \"1x\", \"2x\", \"4x\"],\n",- " \"720p\": [1, \"1x\", \"2x\"],\n",- "}\n",- "all_paths = []\n",- "for resolution, num_frames in sample_cfg.items():\n",- " for num_frame in num_frames:\n",- " print(f\"Resolution: {resolution}, Num Frames: {num_frame}\")\n",- " paths = inference(\n",- " PROMPT,\n",- " resolution=resolution,\n",- " num_frames=num_frame,\n",- " aspect_ratio=\"9:16\",\n",- " num_sampling_steps=30,\n",- " cfg_scale=7.0,\n",- " disable_progress=True,\n",- " )\n",- " display_results(paths)\n",- " all_paths.extend(paths)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Sample all resolution, length, and aspect ratios."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "PROMPT = \"a boy.\"\n",- "sample_cfg = {\n",- " \"144p\": [1, \"1x\", \"2x\", \"4x\", \"8x\"],\n",- " \"240p\": [1, \"1x\", \"2x\", \"4x\", \"8x\"],\n",- " \"360p\": [1, \"1x\", \"2x\", \"4x\"],\n",- " \"480p\": [1, \"1x\", \"2x\", \"4x\"],\n",- " \"720p\": [1, \"1x\", \"2x\"],\n",- "}\n",- "all_paths = []\n",- "for resolution, num_frames in sample_cfg.items():\n",- " for num_frame in num_frames:\n",- " paths = inference_all_aspects(\n",- " PROMPT,\n",- " resolution=resolution,\n",- " num_frames=num_frames,\n",- " num_sampling_steps=30,\n",- " cfg_scale=7.0,\n",- " )\n",- " display_results(paths)\n",- " all_paths.extend(paths)"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": []- }- ],- "metadata": {- "kernelspec": {- "display_name": "opensora",- "language": "python",- "name": "python3"- },- "language_info": {- "codemirror_mode": {- "name": "ipython",- "version": 3- },- "file_extension": ".py",- "mimetype": "text/x-python",- "name": "python",- "nbconvert_exporter": "python",- "pygments_lexer": "ipython3",- "version": "3.10.14"- }- },- "nbformat": 4,- "nbformat_minor": 2-}diff --git a/notebooks/launch.ipynb b/notebooks/launch.ipynbdeleted file mode 100644index 309d7e4..0000000--- a/notebooks/launch.ipynb+++ /dev/null@@ -1,448 +0,0 @@-{- "cells": [- {- "cell_type": "markdown",- "metadata": {},- "source": [- "# Data Process Pipeline"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "### Data Process Commands"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "import os\n",- "\n",- "# TODO: change to your own project path!!!\n",- "OPEN_SORA_HOME = \"/path/to/Open-Sora/\"\n",- "\n",- "\n",- "def convert_dataset_cmd(input_dir, output_file, datatype=\"video\"):\n",- " commands = []\n",- " commands.append(f'echo \"Converting {input_dir} to {output_file}\"')\n",- " output_dir = os.path.dirname(output_file)\n",- "\n",- " commands.append(f\"mkdir -p {output_dir}\")\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"python -m tools.datasets.convert {datatype} {input_dir} --output {output_file}\")\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_video_info(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_info{ext}\"\n",- " output_format = ext[1:]\n",- "\n",- " commands.append(f'echo \"Getting info of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(\n",- " f\"python -m tools.datasets.datautil {input_file} --output {output_file} --format {output_format} --info --fmin 1\"\n",- " )\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_video_info_torchvision(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_info{ext}\"\n",- " output_format = ext[1:]\n",- "\n",- " commands.append(f'echo \"Getting info of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(\n",- " f\"python -m tools.datasets.datautil {input_file} --output {output_file} --format {output_format} --video-info --fmin 1\"\n",- " )\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_caption_llava7b_video(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_caption{ext}\"\n",- " output_format = ext[1:]\n",- "\n",- " commands.append(f'echo \"Getting info of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"conda activate llava2\")\n",- " commands.append(\n",- " f\"torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava {input_file} --dp-size 8 --tp-size 1 --model-path liuhaotian/llava-v1.6-mistral-7b --prompt video\"\n",- " )\n",- " commands.append(f\"conda activate opensora\")\n",- " commands.append(\n",- " f\"python -m tools.datasets.datautil {base}_caption_part*{ext} --output {output_file} --format {output_format} --intersection {input_file} --clean-caption --refine-llm-caption --remove-empty-caption\"\n",- " )\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_caption_load(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_caption{ext}\"\n",- " output_format = ext[1:]\n",- "\n",- " commands.append(f'echo \"Getting caption of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(\n",- " f\"python -m tools.datasets.datautil {input_file} --output {output_file} --format {output_format} --load-caption json --remove-empty-caption --clean-caption\"\n",- " )\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_aesthetic_score(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_aes{ext}\"\n",- " output_format = ext[1:]\n",- "\n",- " commands.append(f'echo \"Getting aesthetic score of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"torchrun --standalone --nproc_per_node 8 -m tools.scoring.aesthetic.inference {input_file}\")\n",- " commands.append(\n",- " f\"python -m tools.datasets.datautil {base}_aes_part*{ext} --output {output_file} --format {output_format} --sort aes\"\n",- " )\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_flow_score(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_flow{ext}\"\n",- "\n",- " commands.append(f'echo \"Getting flow score of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"torchrun --standalone --nproc_per_node 8 -m tools.scoring.optical_flow.inference {input_file}\")\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_ocr(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_match{ext}\"\n",- "\n",- " commands.append(f'echo \"Getting match score of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"torchrun --standalone --nproc_per_node 8 -m tools.scoring.ocr.inference {input_file}\")\n",- " return \" && \".join(commands), output_file\n",- "\n",- " \n",- "def get_match_score(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_match{ext}\"\n",- "\n",- " commands.append(f'echo \"Getting match score of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"torchrun --standalone --nproc_per_node 8 -m tools.scoring.matching.inference {input_file}\")\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_cmotion_score(input_file):\n",- " commands = []\n",- " base, ext = os.path.splitext(input_file)\n",- " output_file = f\"{base}_cmotion{ext}\"\n",- "\n",- " commands.append(f'echo \"Getting cmotion score of {input_file} to {output_file}\"')\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append(f\"python -m tools.caption.camera_motion_detect {input_file}\")\n",- " return \" && \".join(commands), output_file\n",- "\n",- "\n",- "def get_commands(job_list):\n",- " commands = []\n",- " output_file = None\n",- " for job in job_list:\n",- " cmd = job.pop(\"cmd\")\n",- " if output_file is None:\n",- " command, output_file = cmd(**job)\n",- " commands.append(command)\n",- " else:\n",- " job[\"input_file\"] = output_file\n",- " command, output_file = cmd(**job)\n",- " commands.append(command)\n",- " commands.append(f'echo \"All Done!\"')\n",- " return \" && \".join(commands), output_file"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "### Remote Launch via Paramiko\n",- "\n",- "First, add hosts to `~/.ssh/config`"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "import paramiko\n",- "\n",- "HOSTS = [\"host-0\", \"host-1\", \"host-2\", \"host-3\", \"host-4\", \"host-5\", \"host-6\", \"host-7\"]\n",- "\n",- "# load from ~/.ssh/config\n",- "ssh_config = paramiko.SSHConfig()\n",- "user_config_file = os.path.expanduser(\"~/.ssh/config\")\n",- "if os.path.exists(user_config_file):\n",- " with open(user_config_file) as f:\n",- " ssh_config.parse(f)\n",- "\n",- "\n",- "def get_ssh_config(hostname):\n",- " # get the configuration for the host\n",- " user_config = ssh_config.lookup(hostname)\n",- " cfg = {\n",- " \"hostname\": user_config[\"hostname\"],\n",- " \"username\": user_config[\"user\"],\n",- " \"port\": int(user_config[\"port\"]),\n",- " \"key_filename\": user_config[\"identityfile\"],\n",- " }\n",- " return cfg\n",- "\n",- "\n",- "def connect(hostname):\n",- " cfg = get_ssh_config(hostname)\n",- " # connect\n",- " client = paramiko.SSHClient()\n",- " client.set_missing_host_key_policy(paramiko.AutoAddPolicy())\n",- " client.connect(**cfg)\n",- " return client\n",- "\n",- "\n",- "def run_command(command, hostname, nohup=False, log_file=None, sleep=None):\n",- " client = connect(hostname)\n",- " print(\"HOST:\", hostname)\n",- " if sleep:\n",- " command = f\"sleep {sleep}; {command}\"\n",- " command = f\"bash -ic '{command}'\"\n",- " if log_file:\n",- " command = f\"{command} >> {log_file} 2>&1\"\n",- " if nohup:\n",- " command = f\"nohup {command} &\"\n",- " print(\"COMMAND:\", command)\n",- " stdin, stdout, stderr = client.exec_command(command, get_pty=False)\n",- "\n",- " stdout_str = stdout.read().decode()\n",- " stderr_str = stderr.read().decode()\n",- " if stdout_str:\n",- " print(\"==== STDOUT ====\\n\", stdout_str)\n",- " if stderr_str:\n",- " print(\"==== STDERR ====\\n\", stderr_str)\n",- "\n",- " client.close()\n",- "\n",- "\n",- "def run_command_all_hosts(command, hosts=HOSTS):\n",- " for hostname in hosts:\n",- " run_command(command, hostname)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Here are tools to examine machine's status."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "def nvidia_smi(host):\n",- " if host:\n",- " run_command(\"nvidia-smi\", host)\n",- " else:\n",- " run_command_all_hosts(\"nvidia-smi\")\n",- "\n",- "\n",- "def nvitop(host=None):\n",- " if host:\n",- " run_command(f\"/home/user/.local/bin/nvitop -1\", host)\n",- " else:\n",- " run_command_all_hosts(\"/home/user/.local/bin/nvitop -1\")\n",- "\n",- "\n",- "def ps(host=None, interest=\"python|sleep|torchrun|colossal\", all=True):\n",- " cmd = \"ps aux\" if all else \"ps ux\"\n",- " if host:\n",- " if interest is None:\n",- " run_command(f\"{cmd} | cat\", host)\n",- " else:\n",- " run_command(f'{cmd} | cat | grep --color=never -E \"{interest}\"', host)\n",- " else:\n",- " if interest is None:\n",- " run_command_all_hosts(f\"{cmd} | cat\")\n",- " else:\n",- " run_command_all_hosts(f'{cmd} | cat | grep --color=never -E \"{interest}\"')\n",- "\n",- "\n",- "def kill(pid, host):\n",- " run_command(f\"kill -KILL {pid}\", host)\n",- "\n",- "\n",- "def pkill(interest, host):\n",- " run_command(f'pkill -9 -f \"{interest}\"', host)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "### Example\n",- "\n",- "Remote launch via paramiko."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "sleep = None\n",- "run_command(cmd, host, log_file=log_file, nohup=True, sleep=sleep)\n",- "ps(host)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Using following commands to monitor the status of the jobs."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "ps()"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "nvitop(host)"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "kill(, host)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "# Training"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "def colossal_run(data_path, load_path=None):\n",- " commands = []\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " command = f\"colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py configs/opensora-v1-1/train/video.py --wandb True --data-path {data_path}\"\n",- " if load_path:\n",- " command = f\"{command} --load-path {load_path}\"\n",- " commands.append(command)\n",- " cmd = \" && \".join(commands)\n",- " return cmd\n",- "\n",- "\n",- "def kill_all():\n",- " commands = []\n",- " commands.append(f\"cd {OPEN_SORA_HOME}\")\n",- " commands.append('cat hostfile | xargs -I \"{}\" ssh \"{}\" pkill -9 python')\n",- " cmd = \" && \".join(commands)\n",- " return cmd"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "### Examples"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "host = \"host-0\"\n",- "log_file = os.path.join(OPEN_SORA_HOME, \"logs/train.log\")\n",- "data_path = \"/path/to/meta.csv\"\n",- "cmd = colossal_run(data_path)\n",- "print(cmd)"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "run_command(cmd, host, log_file=log_file, nohup=True)"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "cmd = kill_all()\n",- "run_command(cmd, host)"- ]- }- ],- "metadata": {- "kernelspec": {- "display_name": "Python 3 (ipykernel)",- "language": "python",- "name": "python3"- },- "language_info": {- "codemirror_mode": {- "name": "ipython",- "version": 3- },- "file_extension": ".py",- "mimetype": "text/x-python",- "name": "python",- "nbconvert_exporter": "python",- "pygments_lexer": "ipython3",- "version": "3.9.18"- }- },- "nbformat": 4,- "nbformat_minor": 4-}diff --git a/opensora/acceleration/plugin.py b/opensora/acceleration/plugin.pyindex c657a95..7cb0395 100644--- a/opensora/acceleration/plugin.py+++ b/opensora/acceleration/plugin.py@@ -32,6 +32,8 @@ class ZeroSeqParallelPlugin(LowLevelZeroPlugin):cpu_offload: bool = False,master_weights: bool = True,verbose: bool = False,+ overlap_allgather: bool = False,+ cast_inputs: bool = False,) -> None:super().__init__(stage=stage,@@ -51,6 +53,8 @@ class ZeroSeqParallelPlugin(LowLevelZeroPlugin):cpu_offload=cpu_offload,master_weights=master_weights,verbose=verbose,+ overlap_allgather=overlap_allgather,+ cast_inputs=cast_inputs,)self.sp_size = sp_sizeassert self.world_size % sp_size == 0, "world_size must be divisible by sp_size"diff --git a/opensora/datasets/aspect.py b/opensora/datasets/aspect.pyindex f6defb7..501fc7f 100644--- a/opensora/datasets/aspect.py+++ b/opensora/datasets/aspect.py@@ -1,4 +1,5 @@import math+from copy import deepcopy# computation@@ -443,7 +444,7 @@ def get_closest_ratio(height: float, width: float, ratios: dict):return closest_ratio-ASPECT_RATIOS = {+OLD_ASPECT_RATIOS = {"144p": (36864, ASPECT_RATIO_144P),"256": (65536, ASPECT_RATIO_256),"240p": (102240, ASPECT_RATIO_240P),@@ -459,6 +460,15 @@ ASPECT_RATIOS = {"4k": (8294400, ASPECT_RATIO_4K),}+ASPECT_RATIOS = {}+for name, aspect_ratio in OLD_ASPECT_RATIOS.items():+ aspect_ratio = deepcopy(aspect_ratio)+ for ap_key, ap_value in aspect_ratio[1].items():+ h, w = ap_value[0] // 16 * 16, ap_value[1] // 16 * 16+ aspect_ratio[1][ap_key] = (h, w)+ ASPECT_RATIOS[f"{name}_16d"] = aspect_ratio+ASPECT_RATIOS.update(OLD_ASPECT_RATIOS)+def get_num_pixels(name):return ASPECT_RATIOS[name][0]diff --git a/opensora/datasets/bucket.py b/opensora/datasets/bucket.pyindex 8b0230f..b5f3747 100644--- a/opensora/datasets/bucket.py+++ b/opensora/datasets/bucket.py@@ -1,3 +1,4 @@+import bisectfrom collections import OrderedDictimport numpy as np@@ -140,3 +141,83 @@ def closet_smaller_bucket(value, bucket):if value < bucket[i]:return bucket[i - 1]return bucket[-1]+++def merge_dicts(dict_list, maps):+ result = {}+ for index, d in enumerate(dict_list):+ for key, value in d.items():+ # Create the key with an empty set if it doesn't exist+ if key not in result:+ result[key] = []+ # Add the value and the tuple (value, index) to the set+ if value not in result[key]:+ result[key].append((value, maps[index]))+ return result+++def find_proper_res(x1, y1, points):+ result_point = []+ points = sorted(points, key=lambda x: -x[0][0])+ for k, res in points:+ x2 = k[0]+ y2 = k[1]+ if x2 <= x1 and y2 <= y1:+ result_point.append(res)+ return result_point+++class Bucket_ar_first(Bucket):+ def __init__(self, bucket_config):+ super(Bucket_ar_first, self).__init__(bucket_config)+ all_aspects = []+ maps = []+ for x in bucket_config.keys():+ maps.append(x)+ all_aspects.append(ASPECT_RATIOS[x][1])+ all_aspects = merge_dicts(all_aspects, maps)+ all_ars = [float(i) for i in all_aspects.keys()]++ self.all_aspects = all_aspects+ self.all_ars = sorted(all_ars)++ def get_bucket_id(self, T, H, W, frame_interval=1, seed=None):+ ar = H / W++ # binary search+ ind = bisect.bisect_left(self.all_ars, ar)+ if ind == 0:+ ar_id = str(self.all_ars[0])+ if ind == len(self.all_ars):+ ar_id = str(self.all_ars[-1])+ else:+ before = self.all_ars[ind - 1]+ after = self.all_ars[ind]+ ar_id = f"{before:.2f}" if abs(ar - before) < abs(ar - after) else f"{after:.2f}"++ # find proper hw_id+ hw_ids = find_proper_res(H, W, self.all_aspects[ar_id])+ fail = True+ for hw_id in hw_ids:+ t_fail = True+ for t_id, prob in self.bucket_probs[hw_id].items():+ if isinstance(prob, tuple):+ rng = np.random.default_rng(seed + self.bucket_id[hw_id][t_id])+ prob_t = prob[1]+ if rng.random() > prob_t:+ continue+ if T > t_id * frame_interval and t_id != 1:+ t_fail = False+ break+ if t_fail:+ continue++ if isinstance(prob, tuple):+ prob = prob[0]+ if prob >= 1 or rng.random() < prob:+ fail = False+ break+ if fail:+ return None++ return hw_id, t_id, ar_iddiff --git a/opensora/datasets/dataloader.py b/opensora/datasets/dataloader.pyindex 97df0a1..5d7d24e 100644--- a/opensora/datasets/dataloader.py+++ b/opensora/datasets/dataloader.py@@ -1,216 +1,17 @@import collections-import functools-import queueimport random-import threadingfrom typing import Optionalimport numpy as npimport torch-import torch.multiprocessing as multiprocessing-from torch._utils import ExceptionWrapperfrom torch.distributed import ProcessGroupfrom torch.distributed.distributed_c10d import _get_default_group-from torch.utils.data import DataLoader, _utils-from torch.utils.data._utils import MP_STATUS_CHECK_INTERVAL-from torch.utils.data.dataloader import (- IterDataPipe,- MapDataPipe,- _BaseDataLoaderIter,- _MultiProcessingDataLoaderIter,- _sharding_worker_init_fn,- _SingleProcessDataLoaderIter,-)+from torch.utils.data import DataLoader-from .datasets import BatchFeatureDataset, VariableVideoTextDataset, VideoTextDataset-from .pin_memory_cache import PinMemoryCache+from .datasets import BatchFeatureDataset, VariableVideoTextDataset, VideoClasssificationDataset, VideoTextDatasetfrom .sampler import BatchDistributedSampler, StatefulDistributedSampler, VariableVideoBatchSampler-def _pin_memory_loop(- in_queue, out_queue, device_id, done_event, device, pin_memory_cache: PinMemoryCache, pin_memory_key: str-):- # This setting is thread local, and prevents the copy in pin_memory from- # consuming all CPU cores.- torch.set_num_threads(1)-- if device == "cuda":- torch.cuda.set_device(device_id)- elif device == "xpu":- torch.xpu.set_device(device_id) # type: ignore[attr-defined]- elif device == torch._C._get_privateuse1_backend_name():- custom_device_mod = getattr(torch, torch._C._get_privateuse1_backend_name())- custom_device_mod.set_device(device_id)-- def do_one_step():- try:- r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)- except queue.Empty:- return- idx, data = r- if not done_event.is_set() and not isinstance(data, ExceptionWrapper):- try:- assert isinstance(data, dict)- if pin_memory_key in data:- val = data[pin_memory_key]- pin_memory_value = pin_memory_cache.get(val)- pin_memory_value.copy_(val)- data[pin_memory_key] = pin_memory_value- except Exception:- data = ExceptionWrapper(where=f"in pin memory thread for device {device_id}")- r = (idx, data)- while not done_event.is_set():- try:- out_queue.put(r, timeout=MP_STATUS_CHECK_INTERVAL)- break- except queue.Full:- continue-- # See NOTE [ Data Loader Multiprocessing Shutdown Logic ] for details on the- # logic of this function.- while not done_event.is_set():- # Make sure that we don't preserve any object from one iteration- # to the next- do_one_step()---class _MultiProcessingDataLoaderIterForVideo(_MultiProcessingDataLoaderIter):- pin_memory_key: str = "video"-- def __init__(self, loader):- _BaseDataLoaderIter.__init__(self, loader)- self.pin_memory_cache = PinMemoryCache()-- self._prefetch_factor = loader.prefetch_factor-- assert self._num_workers > 0- assert self._prefetch_factor > 0-- if loader.multiprocessing_context is None:- multiprocessing_context = multiprocessing- else:- multiprocessing_context = loader.multiprocessing_context-- self._worker_init_fn = loader.worker_init_fn-- # Adds forward compatibilities so classic DataLoader can work with DataPipes:- # Additional worker init function will take care of sharding in MP and Distributed- if isinstance(self._dataset, (IterDataPipe, MapDataPipe)):- self._worker_init_fn = functools.partial(- _sharding_worker_init_fn, self._worker_init_fn, self._world_size, self._rank- )-- # No certainty which module multiprocessing_context is- self._worker_result_queue = multiprocessing_context.Queue() # type: ignore[var-annotated]- self._worker_pids_set = False- self._shutdown = False- self._workers_done_event = multiprocessing_context.Event()-- self._index_queues = []- self._workers = []- for i in range(self._num_workers):- # No certainty which module multiprocessing_context is- index_queue = multiprocessing_context.Queue() # type: ignore[var-annotated]- # Need to `cancel_join_thread` here!- # See sections (2) and (3b) above.- index_queue.cancel_join_thread()- w = multiprocessing_context.Process(- target=_utils.worker._worker_loop,- args=(- self._dataset_kind,- self._dataset,- index_queue,- self._worker_result_queue,- self._workers_done_event,- self._auto_collation,- self._collate_fn,- self._drop_last,- self._base_seed,- self._worker_init_fn,- i,- self._num_workers,- self._persistent_workers,- self._shared_seed,- ),- )- w.daemon = True- # NB: Process.start() actually take some time as it needs to- # start a process and pass the arguments over via a pipe.- # Therefore, we only add a worker to self._workers list after- # it started, so that we do not call .join() if program dies- # before it starts, and __del__ tries to join but will get:- # AssertionError: can only join a started process.- w.start()- self._index_queues.append(index_queue)- self._workers.append(w)-- if self._pin_memory:- self._pin_memory_thread_done_event = threading.Event()-- # Queue is not type-annotated- self._data_queue = queue.Queue() # type: ignore[var-annotated]- if self._pin_memory_device == "xpu":- current_device = torch.xpu.current_device() # type: ignore[attr-defined]- elif self._pin_memory_device == torch._C._get_privateuse1_backend_name():- custom_device_mod = getattr(torch, torch._C._get_privateuse1_backend_name())- current_device = custom_device_mod.current_device()- else:- current_device = torch.cuda.current_device() # choose cuda for default- pin_memory_thread = threading.Thread(- target=_pin_memory_loop,- args=(- self._worker_result_queue,- self._data_queue,- current_device,- self._pin_memory_thread_done_event,- self._pin_memory_device,- self.pin_memory_cache,- self.pin_memory_key,- ),- )- pin_memory_thread.daemon = True- pin_memory_thread.start()- # Similar to workers (see comment above), we only register- # pin_memory_thread once it is started.- self._pin_memory_thread = pin_memory_thread- else:- self._data_queue = self._worker_result_queue # type: ignore[assignment]-- # In some rare cases, persistent workers (daemonic processes)- # would be terminated before `__del__` of iterator is invoked- # when main process exits- # It would cause failure when pin_memory_thread tries to read- # corrupted data from worker_result_queue- # atexit is used to shutdown thread and child processes in the- # right sequence before main process exits- if self._persistent_workers and self._pin_memory:- import atexit-- for w in self._workers:- atexit.register(_MultiProcessingDataLoaderIter._clean_up_worker, w)-- # .pid can be None only before process is spawned (not the case, so ignore)- _utils.signal_handling._set_worker_pids(id(self), tuple(w.pid for w in self._workers)) # type: ignore[misc]- _utils.signal_handling._set_SIGCHLD_handler()- self._worker_pids_set = True- self._reset(loader, first_iter=True)-- def remove_cache(self, output_tensor: torch.Tensor):- self.pin_memory_cache.remove(output_tensor)-- def get_cache_info(self) -> str:- return str(self.pin_memory_cache)---class DataloaderForVideo(DataLoader):- def _get_iterator(self) -> "_BaseDataLoaderIter":- if self.num_workers == 0:- return _SingleProcessDataLoaderIter(self)- else:- self.check_worker_number_rationality()- return _MultiProcessingDataLoaderIterForVideo(self)--# Deterministic dataloaderdef get_seed_worker(seed):def seed_worker(worker_id):@@ -234,7 +35,6 @@ def prepare_dataloader(bucket_config=None,num_bucket_build_workers=1,prefetch_factor=None,- cache_pin_memory=False,**kwargs,):_kwargs = kwargs.copy()@@ -250,9 +50,8 @@ def prepare_dataloader(verbose=True,num_bucket_build_workers=num_bucket_build_workers,)- dl_cls = DataloaderForVideo if cache_pin_memory else DataLoaderreturn (- dl_cls(+ DataLoader(dataset,batch_sampler=batch_sampler,worker_init_fn=get_seed_worker(seed),@@ -264,7 +63,7 @@ def prepare_dataloader(),batch_sampler,)- elif isinstance(dataset, VideoTextDataset):+ elif isinstance(dataset, VideoTextDataset) or isinstance(dataset, VideoClasssificationDataset):process_group = process_group or _get_default_group()sampler = StatefulDistributedSampler(dataset,@@ -272,9 +71,8 @@ def prepare_dataloader(rank=process_group.rank(),shuffle=shuffle,)- dl_cls = DataloaderForVideo if cache_pin_memory else DataLoaderreturn (- dl_cls(+ DataLoader(dataset,batch_size=batch_size,sampler=sampler,diff --git a/opensora/datasets/datasets.py b/opensora/datasets/datasets.pyindex b148268..f375584 100644--- a/opensora/datasets/datasets.py+++ b/opensora/datasets/datasets.py@@ -3,9 +3,12 @@ from glob import globimport numpy as npimport torch+import torchvision.transforms as transformsfrom PIL import ImageFilefrom torchvision.datasets.folder import IMG_EXTENSIONS, pil_loader+from torchvision.transforms import Compose, RandomHorizontalFlip+from opensora.datasets.utils import video_transformsfrom opensora.registry import DATASETSfrom .read_video import read_video@@ -18,7 +21,6 @@ IMG_FPS = 120@DATASETS.register_module()class VideoTextDataset(torch.utils.data.Dataset):"""load video according to the csv file.-Args:target_video_len (int): the number of video frames will be load.align_transform (callable): Align different videos in a specified size.@@ -32,6 +34,9 @@ class VideoTextDataset(torch.utils.data.Dataset):frame_interval=1,image_size=(256, 256),transform_name="center",+ tokenize_fn=None,+ bucket_class="Bucket",+ return_path=False,):self.data_path = data_pathself.data = read_file(data_path)@@ -43,6 +48,9 @@ class VideoTextDataset(torch.utils.data.Dataset):"image": get_transforms_image(transform_name, image_size),"video": get_transforms_video(transform_name, image_size),}+ self.tokenize_fn = tokenize_fn+ self.bucket_class = bucket_class+ self.return_path = return_pathdef _print_data_number(self):num_videos = 0@@ -74,6 +82,8 @@ class VideoTextDataset(torch.utils.data.Dataset):# Sampling video framesvideo = temporal_random_crop(vframes, self.num_frames, self.frame_interval)+ video = video.clone()+ del vframes# transformtransform = self.transforms["video"]@@ -93,9 +103,20 @@ class VideoTextDataset(torch.utils.data.Dataset):# TCHW -> CTHWvideo = video.permute(1, 0, 2, 3)- ret = {"video": video, "fps": video_fps}+ ret = {+ "video": video,+ "num_frames": self.num_frames,+ "height": self.image_size[0],+ "width": self.image_size[1],+ "ar": 1.0,+ "fps": video_fps,+ }if self.get_text:ret["text"] = sample["text"]+ if self.tokenize_fn is not None:+ ret.update({k: v.squeeze(0) for k, v in self.tokenize_fn(ret["text"]).items()})+ if self.return_path:+ ret["path"] = pathreturn retdef __getitem__(self, index):@@ -122,8 +143,11 @@ class VariableVideoTextDataset(VideoTextDataset):image_size=(None, None),transform_name=None,dummy_text_feature=False,+ tokenize_fn=None,):- super().__init__(data_path, num_frames, frame_interval, image_size, transform_name=None)+ super().__init__(+ data_path, num_frames, frame_interval, image_size, transform_name=None, tokenize_fn=tokenize_fn+ )self.transform_name = transform_nameself.data["id"] = np.arange(len(self.data))self.dummy_text_feature = dummy_text_feature@@ -134,6 +158,14 @@ class VariableVideoTextDataset(VideoTextDataset):W = self.data.iloc[index]["width"]return T, H, W+ def get_path(self, index):+ try:+ index, num_frames, height, width = [int(val) for val in index.split("-")]+ return self.data.iloc[index]["path"]+ except Exception as e:+ print(f"data {index}: {e}")+ return index+def getitem(self, index):# a hack to pass in the (time, height, width) info from samplerindex, num_frames, height, width = [int(val) for val in index.split("-")]@@ -180,9 +212,12 @@ class VariableVideoTextDataset(VideoTextDataset):"width": width,"ar": ar,"fps": video_fps,+ "path": path,}if self.get_text:ret["text"] = sample["text"]+ if self.tokenize_fn is not None:+ ret.update({k: v.squeeze(0) for k, v in self.tokenize_fn(ret["text"]).items()})if self.dummy_text_feature:text_len = 50ret["text"] = torch.zeros((1, text_len, 1152))@@ -192,7 +227,9 @@ class VariableVideoTextDataset(VideoTextDataset):def __getitem__(self, index):try:return self.getitem(index)- except:+ except Exception as e:+ path = self.get_path(index)+ print(f"data {path}: {e}")return None@@ -248,3 +285,101 @@ class BatchFeatureDataset(torch.utils.data.Dataset):"num_frames": batch["num_frames"],}return ret++[email protected]_module()+class VideoClasssificationDataset(torch.utils.data.Dataset):+ """load video according to the csv file.++ Args:+ target_video_len (int): the number of video frames will be load.+ align_transform (callable): Align different videos in a specified size.+ temporal_sample (callable): Sample the target length of a video.+ """++ def __init__(+ self,+ data_path=None,+ num_frames=16,+ frame_interval=1,+ image_size=(224, 224),+ data_split="train",+ label_key="preference_mean",+ problem_type="regression",+ ):+ self.data_path = data_path+ self.data = read_file(data_path)+ self.get_text = "text" in self.data.columns+ self.num_frames = num_frames+ self.frame_interval = frame_interval+ self.image_size = image_size+ self.label_key = label_key+ self.problem_type = problem_type++ mean = [0.485, 0.456, 0.406]+ std = [0.229, 0.224, 0.225]+ if data_split == "train":+ self.transforms = Compose(+ [+ video_transforms.ToTensorVideo(), # TCHW+ video_transforms.ResizeCrop(image_size),+ transforms.Normalize(mean, std, inplace=True),+ RandomHorizontalFlip(p=0.5),+ ]+ )+ else:+ self.transforms = Compose(+ [+ video_transforms.ToTensorVideo(), # TCHW+ video_transforms.ResizeCrop(image_size),+ transforms.Normalize(mean, std, inplace=True),+ ]+ )++ def getitem(self, index):+ sample = self.data.iloc[index]+ path = sample["path"]++ label = None+ if sample.get(self.label_key): # # label processing, if test won't have it+ label = sample[self.label_key]+ if self.label_key != "label_idx":+ label = label - 1 # normalize score to 0 - 4+ if self.problem_type == "single_label_classification":+ label = round(label)++ # loading+ vframes, vinfo = read_video(path, backend="av")+ video_fps = vinfo["video_fps"] if "video_fps" in vinfo else 24++ if (+ self.frame_interval == -1+ ): # instead of random sampling, we use all of the video by uniformly sampling num_frames from the video+ frame_indices = np.linspace(0, vframes[0] - 1, self.num_frames)+ video = vframes[frame_indices]+ else:+ # Sampling video frames+ video = temporal_random_crop(vframes, self.num_frames, self.frame_interval) # Sampling video frames, TCHW++ # transform+ video = self.transforms(video) # T C H W++ if label is not None:+ ret = {"video": video, "label": label, "fps": video_fps, "path": path, "index": index}+ else:+ ret = {"video": video, "fps": video_fps, "path": path, "index": index}++ return ret++ def __getitem__(self, index):+ for _ in range(10):+ try:+ return self.getitem(index)+ except Exception as e:+ path = self.data.iloc[index]["path"]+ print(f"data {path}: {e}")+ index = np.random.randint(len(self))+ raise RuntimeError("Too many bad data.")++ def __len__(self):+ return len(self.data)diff --git a/opensora/datasets/pin_memory_cache.py b/opensora/datasets/pin_memory_cache.pydeleted file mode 100644index 3f6e755..0000000--- a/opensora/datasets/pin_memory_cache.py+++ /dev/null@@ -1,76 +0,0 @@-import threading-from typing import Dict, List, Optional--import torch---class PinMemoryCache:- force_dtype: Optional[torch.dtype] = None- min_cache_numel: int = 0- pre_alloc_numels: List[int] = []-- def __init__(self):- self.cache: Dict[int, torch.Tensor] = {}- self.output_to_cache: Dict[int, int] = {}- self.cache_to_output: Dict[int, int] = {}- self.lock = threading.Lock()- self.total_cnt = 0- self.hit_cnt = 0-- if len(self.pre_alloc_numels) > 0 and self.force_dtype is not None:- for n in self.pre_alloc_numels:- cache_tensor = torch.empty(n, dtype=self.force_dtype, device="cpu", pin_memory=True)- with self.lock:- self.cache[id(cache_tensor)] = cache_tensor-- def get(self, tensor: torch.Tensor) -> torch.Tensor:- """Receive a cpu tensor and return the corresponding pinned tensor. Note that this only manage memory allocation, doesn't copy content.-- Args:- tensor (torch.Tensor): The tensor to be pinned.-- Returns:- torch.Tensor: The pinned tensor.- """- self.total_cnt += 1- with self.lock:- # find free cache- for cache_id, cache_tensor in self.cache.items():- if cache_id not in self.cache_to_output and cache_tensor.numel() >= tensor.numel():- target_cache_tensor = cache_tensor[: tensor.numel()].view(tensor.shape)- out_id = id(target_cache_tensor)- self.output_to_cache[out_id] = cache_id- self.cache_to_output[cache_id] = out_id- self.hit_cnt += 1- return target_cache_tensor- # no free cache, create a new one- dtype = self.force_dtype if self.force_dtype is not None else tensor.dtype- cache_numel = max(tensor.numel(), self.min_cache_numel)- cache_tensor = torch.empty(cache_numel, dtype=dtype, device="cpu", pin_memory=True)- target_cache_tensor = cache_tensor[: tensor.numel()].view(tensor.shape)- out_id = id(target_cache_tensor)- with self.lock:- self.cache[id(cache_tensor)] = cache_tensor- self.output_to_cache[out_id] = id(cache_tensor)- self.cache_to_output[id(cache_tensor)] = out_id- return target_cache_tensor-- def remove(self, output_tensor: torch.Tensor) -> None:- """Release corresponding cache tensor.-- Args:- output_tensor (torch.Tensor): The tensor to be released.- """- out_id = id(output_tensor)- with self.lock:- if out_id not in self.output_to_cache:- raise ValueError("Tensor not found in cache.")- cache_id = self.output_to_cache.pop(out_id)- del self.cache_to_output[cache_id]-- def __str__(self):- with self.lock:- num_cached = len(self.cache)- num_used = len(self.output_to_cache)- total_cache_size = sum([v.numel() * v.element_size() for v in self.cache.values()])- return f"PinMemoryCache(num_cached={num_cached}, num_used={num_used}, total_cache_size={total_cache_size / 1024**3:.2f} GB, hit rate={self.hit_cnt / self.total_cnt:.2f})"diff --git a/opensora/datasets/sampler.py b/opensora/datasets/sampler.pyindex 511579f..e0291a0 100644--- a/opensora/datasets/sampler.py+++ b/opensora/datasets/sampler.py@@ -10,7 +10,7 @@ from torch.utils.data import Dataset, DistributedSamplerfrom opensora.utils.misc import format_numel_str, get_loggerfrom .aspect import get_num_pixels-from .bucket import Bucket+from .bucket import Bucket, Bucket_ar_firstfrom .datasets import VariableVideoTextDataset@@ -75,7 +75,12 @@ class VariableVideoBatchSampler(DistributedSampler):dataset=dataset, num_replicas=num_replicas, rank=rank, shuffle=shuffle, seed=seed, drop_last=drop_last)self.dataset = dataset- self.bucket = Bucket(bucket_config)+ if self.dataset.bucket_class == "Bucket":+ self.bucket = Bucket(bucket_config)+ elif self.dataset.bucket_class == "Bucket_ar_first":+ self.bucket = Bucket_ar_first(bucket_config)+ else:+ raise ValueError(f"Invalid bucket class: {self.dataset.bucket_class}")self.verbose = verboseself.last_micro_batch_access_index = 0self.approximate_num_batch = Nonediff --git a/opensora/datasets/utils.py b/opensora/datasets/utils.pyindex b7083a3..ed874ff 100644--- a/opensora/datasets/utils.py+++ b/opensora/datasets/utils.py@@ -1,6 +1,8 @@import os+import randomimport re+import cv2import numpy as npimport pandas as pdimport requests@@ -70,6 +72,7 @@ def temporal_random_crop(vframes, num_frames, frame_interval):end_frame_ind - start_frame_ind >= num_frames), f"Not enough frames to sample, {end_frame_ind} - {start_frame_ind} < {num_frames}"frame_indice = np.linspace(start_frame_ind, end_frame_ind - 1, num_frames, dtype=int)+ # print(frame_indice)video = vframes[frame_indice]return video@@ -95,6 +98,14 @@ def get_transforms_video(name="center", image_size=(256, 256)):transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5], inplace=True),])+ elif name == "rand_size_crop":+ transform_video = transforms.Compose(+ [+ video_transforms.ToTensorVideo(), # TCHW+ video_transforms.RandomSizedCrop(image_size),+ transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5], inplace=True),+ ]+ )else:raise NotImplementedError(f"Transform {name} not implemented")return transform_video@@ -121,6 +132,14 @@ def get_transforms_image(name="center", image_size=(256, 256)):transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5], inplace=True),])+ elif name == "rand_size_crop":+ transform = transforms.Compose(+ [+ transforms.Lambda(lambda pil_image: rand_size_crop_arr(pil_image, image_size)),+ transforms.ToTensor(),+ transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5], inplace=True),+ ]+ )else:raise NotImplementedError(f"Transform {name} not implemented")return transform@@ -156,7 +175,33 @@ def read_from_path(path, image_size, transform_name="center"):return read_image_from_path(path, image_size=image_size, transform_name=transform_name)-def save_sample(x, save_path=None, fps=8, normalize=True, value_range=(-1, 1), force_video=False, verbose=True):+def write_video_cv2(+ filename: str,+ video: torch.Tensor,+ fps: float,+):+ image_size = (video.size(2), video.size(1))+ fourcc = cv2.VideoWriter_fourcc(+ *"avc1"+ ) # NOTE: your opencv must be installed with `conda install -c conda-forge opencv` else the video may not display on browser+ output = cv2.VideoWriter(filename, fourcc, fps, image_size)+ for frame_idx in range(video.size(0)):+ frame = np.array(video[frame_idx]) # H,W,C+ frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)+ output.write(frame)+ output.release()+++def save_sample(+ x,+ save_path=None,+ fps=8,+ normalize=True,+ value_range=(-1, 1),+ force_video=False,+ verbose=True,+ write_video_backend="cv2",+):"""Args:x (Tensor): shape [C, T, H, W]@@ -164,18 +209,35 @@ def save_sample(x, save_path=None, fps=8, normalize=True, value_range=(-1, 1), fassert x.ndim == 4if not force_video and x.shape[1] == 1: # T = 1: save as image- save_path += ".png"+ with_ext = False+ for ext in IMG_EXTENSIONS:+ if save_path.endswith(ext):+ with_ext = True+ break+ if not with_ext:+ save_path += ".png"+x = x.squeeze(1)save_image([x], save_path, normalize=normalize, value_range=value_range)else:- save_path += ".mp4"+ with_ext = False+ for ext in VID_EXTENSIONS:+ if save_path.endswith(ext):+ with_ext = True+ break+ if not with_ext:+ save_path += ".mp4"+if normalize:low, high = value_rangex.clamp_(min=low, max=high)x.sub_(low).div_(max(high - low, 1e-5))x = x.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 3, 0).to("cpu", torch.uint8)- write_video(save_path, x, fps=fps, video_codec="h264")+ if write_video_backend == "cv2":+ write_video_cv2(save_path, x, fps=fps)+ else:+ write_video(save_path, x, fps=fps, video_codec="h264")if verbose:print(f"Saved to {save_path}")return save_path@@ -198,6 +260,27 @@ def center_crop_arr(pil_image, image_size):return Image.fromarray(arr[crop_y : crop_y + image_size, crop_x : crop_x + image_size])+def rand_size_crop_arr(pil_image, image_size):+ """+ Randomly crop image for height and width, ranging from image_size[0] to image_size[1]+ """+ arr = np.array(pil_image)++ # get random target h w+ height = random.randint(image_size[0], image_size[1])+ width = random.randint(image_size[0], image_size[1])+ # ensure that h w are factors of 8+ height = height - height % 8+ width = width - width % 8++ # get random start pos+ h_start = random.randint(0, max(len(arr) - height, 0))+ w_start = random.randint(0, max(len(arr[0]) - height, 0))++ # crop+ return Image.fromarray(arr[h_start : h_start + height, w_start : w_start + width])++def resize_crop_to_fill(pil_image, image_size):w, h = pil_image.size # PIL is (W, H)th, tw = image_size@@ -215,3 +298,27 @@ def resize_crop_to_fill(pil_image, image_size):arr = np.array(image)assert i + th <= arr.shape[0] and j + tw <= arr.shape[1]return Image.fromarray(arr[i : i + th, j : j + tw])+++def rand_size_crop_arr(pil_image, image_size):+ w, h = pil_image.size # PIL is (W, H)+ th = random.randint(self.size[0], self.size[1])+ tw = random.randint(self.size[0], self.size[1])+ # ensure that h w are factors of 8+ th = th - th % multiples_of+ tw = tw - tw % multiples_of++ if h < th:+ th = h - h % multiples_of+ if w < tw:+ tw = w - w % multiples_of++ if w == tw and h == th:+ i = j = 0++ else:+ # get random start pos+ i = random.randint(0, h - th)+ j = random.randint(0, w - tw)++ return Image.fromarray(arr[i : i + th, j : j + tw])diff --git a/opensora/datasets/video_transforms.py b/opensora/datasets/video_transforms.pyindex 8cf5046..5da870d 100644--- a/opensora/datasets/video_transforms.py+++ b/opensora/datasets/video_transforms.py@@ -12,10 +12,9 @@# See the License for the specific language governing permissions and# limitations under the License.# Modified from Latte-# - This file is adapted from https://github.com/Vchitect/Latte/blob/main/datasets/video_transforms.py--import numbers++# - This file is adapted from https://github.com/Vchitect/Latte/blob/main/datasets/video_transforms.pyimport randomimport numpy as np@@ -53,7 +52,8 @@ def resize_scale(clip, target_size, interpolation_mode):raise ValueError(f"target size should be tuple (height, width), instead got {target_size}")H, W = clip.size(-2), clip.size(-1)scale_ = target_size[0] / min(H, W)- return torch.nn.functional.interpolate(clip, scale_factor=scale_, mode=interpolation_mode, align_corners=False)+ th, tw = int(round(H * scale_)), int(round(W * scale_))+ return torch.nn.functional.interpolate(clip, size=(th, tw), mode=interpolation_mode, align_corners=False)def resized_crop(clip, i, j, h, w, size, interpolation_mode="bilinear"):@@ -124,6 +124,40 @@ def resize_crop_to_fill(clip, target_size):return crop(clip, i, j, th, tw)+# def rand_crop_h_w(clip, target_size_range, multiples_of: int = 8):+# # NOTE: for some reason, if don't re-import, gives same randint results+# import sys++# del sys.modules["random"]+# import random++# if not _is_tensor_video_clip(clip):+# raise ValueError("clip should be a 4D torch.tensor")+# h, w = clip.size(-2), clip.size(-1)++# # get random target h w+# th = random.randint(target_size_range[0], target_size_range[1])+# tw = random.randint(target_size_range[0], target_size_range[1])++# # ensure that h w are factors of 8+# th = th - th % multiples_of+# tw = tw - tw % multiples_of++# # get random start pos+# i = random.randint(0, h-th) if h > th else 0+# j = random.randint(0, w-tw) if w > tw else 0++# th = th if th < h else h+# tw = tw if tw < w else w++# # print("target size range:",target_size_range)+# # print("original size:", h, w)+# # print("crop size:", th, tw)+# # print(f"crop:{i}-{i+th}, {j}-{j+tw}")++# return (crop(clip, i, j, th, tw), th, tw)++def random_shift_crop(clip):"""Slide along the long edge, with the short edge as crop size@@ -207,6 +241,47 @@ class ResizeCrop:return f"{self.__class__.__name__}(size={self.size})"+class RandomSizedCrop:+ def __init__(self, size):+ if isinstance(size, numbers.Number):+ self.size = (int(size), int(size))+ else:+ self.size = size++ def __call__(self, clip):+ i, j, h, w = self.get_params(clip)+ # self.size = (h, w)+ return crop(clip, i, j, h, w)++ def __repr__(self) -> str:+ return f"{self.__class__.__name__}(size={self.size})"++ def get_params(self, clip, multiples_of=8):+ h, w = clip.shape[-2:]++ # get random target h w+ th = random.randint(self.size[0], self.size[1])+ tw = random.randint(self.size[0], self.size[1])+ # ensure that h w are factors of 8+ th = th - th % multiples_of+ tw = tw - tw % multiples_of++ if h < th:+ th = h - h % multiples_of+ if w < tw:+ tw = w - w % multiples_of++ if w == tw and h == th:+ return 0, 0, h, w++ else:+ # get random start pos+ i = random.randint(0, h - th)+ j = random.randint(0, w - tw)++ return i, j, th, tw++class RandomCropVideo:def __init__(self, size):if isinstance(size, numbers.Number):diff --git a/opensora/models/__init__.py b/opensora/models/__init__.pyindex 6025349..e33df46 100644--- a/opensora/models/__init__.py+++ b/opensora/models/__init__.py@@ -1,3 +1,4 @@+from .vae_v1_3 import *from .dit import *from .latte import *from .pixart import *diff --git a/opensora/models/layers/blocks.py b/opensora/models/layers/blocks.pyindex e28ef34..0e9bfff 100644--- a/opensora/models/layers/blocks.py+++ b/opensora/models/layers/blocks.py@@ -11,7 +11,9 @@import functoolsimport math-from typing import Optional+import warnings+from itertools import chain+from typing import Optional, Sequenceimport numpy as npimport torch@@ -71,6 +73,84 @@ def t2i_modulate(x, shift, scale):return x * (1 + scale) + shift+def pad_to_multiples(tensor: torch.Tensor, multiples, dims, value=0, extra_pad_on_dims=None):+ """+ Pad the input tensor to the multiples of the given values. Right padding+ """+ assert len(multiples) == len(dims) and len(multiples) > 0 and len(multiples) <= tensor.dim()+ if extra_pad_on_dims is not None:+ assert len(extra_pad_on_dims) == len(multiples) * 2+ n_pads = [0] * tensor.dim() * 2+ for i, (dim, multiple) in enumerate(zip(dims, multiples)):+ n_pads[dim * 2] = (multiple - tensor.size(dim) % multiple) % multiple+ if extra_pad_on_dims is not None:+ n_pads[dim * 2 + 1] += extra_pad_on_dims[i * 2]+ n_pads[dim * 2] += extra_pad_on_dims[i * 2 + 1]+ n_pads = n_pads[::-1]+ output_tensor = F.pad(tensor, n_pads, value=value)+ if output_tensor.shape == tensor.shape:+ padding_mask = None+ else:+ padding_mask = torch.ones_like(tensor, dtype=torch.int)+ padding_mask = F.pad(padding_mask, n_pads, value=0)+ return output_tensor, padding_mask+++def remove_padding_nd(tensor: torch.Tensor, origin_sizes, dims):+ """+ Remove padding from the input tensor+ """+ assert len(origin_sizes) == len(dims) and len(origin_sizes) > 0 and len(origin_sizes) <= tensor.dim()+ for size, dim in zip(origin_sizes, dims):+ tensor = tensor.narrow(dim, 0, size)+ return tensor+++def split_seq_cat_batch(tensor: torch.Tensor, split_sizes, dims, batch_dim: int = 0) -> torch.Tensor:+ """split tensor in sequence dimension and cat in batch dimension++ Args:+ tensor (torch.Tensor): [B, *N, C]+ split_sizes (_type_): kernel size+ dims (_type_): dim index of "N"+ batch_dim (int, optional): dim of batch. Defaults to 0.++ Returns:+ torch.Tensor: [MB, *K, C], K is kernel size. Total number of dim is the same as input tensor.+ """+ chunks = [tensor]+ for dim, split_size in zip(dims, split_sizes):+ new_chunks = []+ for t in chunks:+ new_chunks.extend(t.split(split_size, dim))+ chunks = new_chunks+ return torch.cat(chunks, batch_dim)+++def split_batch_cat_seq(tensor: torch.Tensor, batch_size: int, num_splits, dims, batch_dim: int = 0) -> torch.Tensor:+ """split tensor in batch dimension and cat in sequence dimension++ Args:+ tensor (torch.Tensor): [MB, *K, C]+ batch_size (int): original batch size+ num_splits (_type_): number of splits of each dim+ dims (_type_): dim index of "K"+ batch_dim (int, optional): dim of batch. Defaults to 0.++ Returns:+ torch.Tensor: [B, *N, C]+ """+ chunks = tensor.split(batch_size, batch_dim)+ for dim, num_split in zip(dims[::-1], num_splits[::-1]):+ new_chunks = []+ for i in range(0, len(chunks), num_split):+ group = chunks[i : i + num_split]+ new_chunks.append(torch.cat(group, dim))+ chunks = new_chunks+ assert len(chunks) == 1+ return chunks[0]++# ===============================================# General-purpose Layers# ===============================================@@ -142,6 +222,8 @@ class Attention(nn.Module):enable_flash_attn: bool = False,rope=None,qk_norm_legacy: bool = False,+ kernel_size: Optional[Sequence[int]] = None,+ shift_window: bool = False,) -> None:super().__init__()assert dim % num_heads == 0, "dim should be divisible by num_heads"@@ -163,15 +245,69 @@ class Attention(nn.Module):if rope is not None:self.rope = Trueself.rotary_emb = rope-+self.is_causal = False- def forward(self, x: torch.Tensor) -> torch.Tensor:- B, N, C = x.shape- # flash attn is not memory efficient for small sequences, this is empirical- enable_flash_attn = self.enable_flash_attn and (N > B)+ self.is_causal = False+ self.kernel_size = kernel_size+ self.shift_window = shift_window+ if shift_window and kernel_size is None:+ warnings.warn(f"shift_window is enabled but kernel_size is not set, this may not work as expected")+ if kernel_size is not None and shift_window:+ self.extra_pad_on_dims = tuple(+ chain.from_iterable((k // 2, k - k // 2) if k >= 0 else (0, 0) for k in kernel_size)+ )+ else:+ self.extra_pad_on_dims = None++ def rotate_3d(self, k, ks1, ks2, scale=None):+ k_t, k_h, k_w = k.chunk(3, dim=-1)+ # temporal+ k_t = rearrange(k_t, "b h (k1 k2 T) d -> b h k1 k2 T d", k1=ks1, k2=ks2)+ k_t = self.rotary_emb(k_t)+ k_t = rearrange(k_t, "b h k1 k2 T d -> b h (k1 k2 T) d", k1=ks1, k2=ks2)+ # height+ k_h = rearrange(k_h, "b h (k1 k2 T) d -> b h k2 T k1 d", k1=ks1, k2=ks2)+ k_h = self.rotary_emb(k_h, scale=scale)+ k_h = rearrange(k_h, "b h k2 T k1 d -> b h (k1 k2 T) d", k1=ks1, k2=ks2)+ # width+ k_w = rearrange(k_w, "b h (k1 k2 T) d -> b h T k1 k2 d", k1=ks1, k2=ks2)+ k_w = self.rotary_emb(k_w, scale=scale)+ k_w = rearrange(k_w, "b h T k1 k2 d -> b h (k1 k2 T) d", k1=ks1, k2=ks2)+ # combine+ k = torch.cat([k_t, k_h, k_w], dim=-1)+ return k++ def forward(self, x: torch.Tensor, scale=None, H=None, W=None) -> torch.Tensor:+ attn_mask = None+ if self.kernel_size is not None:+ B, *dims, C = x.shape+ assert len(dims) == len(+ self.kernel_size+ ), f"input shape and kernel size mismatch, {dims} vs {self.kernel_size}"+ kernel_size = [(k if k >= 0 else s) for s, k in zip(dims, self.kernel_size)]+ N = np.prod(kernel_size)+ indices = list(range(1, x.dim() - 1))++ x, padding_mask = pad_to_multiples(x, kernel_size, indices, extra_pad_on_dims=self.extra_pad_on_dims)+ num_splits = [x.size(dim) // k for dim, k in zip(indices, kernel_size)]+ x = split_seq_cat_batch(x, kernel_size, indices)+ qkv_b = x.shape[0]+ if padding_mask is not None:+ attn_mask = padding_mask.narrow(-1, 0, 1).squeeze(-1)+ attn_mask = 1.0 - split_seq_cat_batch(attn_mask, kernel_size, indices).to(x.dtype)+ attn_mask.masked_fill_(attn_mask.bool(), float("-inf"))+ attn_mask = attn_mask.view(qkv_b, N)+ attn_mask = attn_mask[:, None, :].expand(qkv_b, N, N).unsqueeze(1)+ else:+ B, N, C = x.shape+ qkv_b = B+qkv = self.qkv(x)- qkv_shape = (B, N, 3, self.num_heads, self.head_dim)+ # flash attn is not memory efficient for small sequences, this is empirical+ enable_flash_attn = self.enable_flash_attn and (N > 128) and self.kernel_size is None+ enable_sdpa = self.enable_flash_attn and (N > 128) and self.kernel_size is not None+ qkv_shape = (qkv_b, N, 3, self.num_heads, self.head_dim)qkv = qkv.view(qkv_shape).permute(2, 0, 3, 1, 4)q, k, v = qkv.unbind(0)@@ -184,8 +320,15 @@ class Attention(nn.Module):else:q, k = self.q_norm(q), self.k_norm(k)if self.rope:- q = self.rotary_emb(q)- k = self.rotary_emb(k)+ if self.kernel_size is not None:+ q = self.rotate_3d(q, self.kernel_size[0], self.kernel_size[1], scale=scale)+ k = self.rotate_3d(k, self.kernel_size[0], self.kernel_size[1], scale=scale)+ elif H is not None and W is not None:+ q = self.rotate_3d(q, H, W, scale=scale)+ k = self.rotate_3d(k, H, W, scale=scale)+ else:+ q = self.rotary_emb(q)+ k = self.rotary_emb(k)if enable_flash_attn:from flash_attn import flash_attn_func@@ -202,6 +345,16 @@ class Attention(nn.Module):softmax_scale=self.scale,causal=self.is_causal,)+ elif enable_sdpa:+ x = F.scaled_dot_product_attention(+ q,+ k,+ v,+ attn_mask=attn_mask,+ dropout_p=self.attn_drop.p if self.training else 0.0,+ scale=self.scale,+ is_causal=self.is_causal,+ )else:dtype = q.dtypeq = q * self.scale@@ -209,19 +362,28 @@ class Attention(nn.Module):attn = attn.to(torch.float32)if self.is_causal:causal_mask = torch.tril(torch.ones_like(attn), diagonal=0)- causal_mask = torch.where(causal_mask.bool(), 0, float('-inf'))+ causal_mask = torch.where(causal_mask.bool(), 0, float("-inf"))attn += causal_mask+ if attn_mask is not None:+ attn += attn_maskattn = attn.softmax(dim=-1)attn = attn.to(dtype) # cast back attn to original dtypeattn = self.attn_drop(attn)x = attn @ v- x_output_shape = (B, N, C)+ x_output_shape = (qkv_b, N, C)if not enable_flash_attn:x = x.transpose(1, 2)x = x.reshape(x_output_shape)+x = self.proj(x)x = self.proj_drop(x)++ if self.kernel_size is not None:+ x = x.reshape(qkv_b, *kernel_size, C)+ x = split_batch_cat_seq(x, B, num_splits, indices)+ x = remove_padding_nd(x, dims, indices)+ assert x.shape == (B, *dims, C)return x@@ -367,8 +529,11 @@ class SeqParallelAttention(Attention):norm_layer: nn.Module = LlamaRMSNorm,enable_flash_attn: bool = False,rope=None,+ qk_norm_legacy: bool = False,+ kernel_size: Optional[Sequence[int]] = None,+ shift_window: bool = False,+ temporal: bool = False,) -> None:- assert rope is None, "Rope is not supported in SeqParallelAttention"super().__init__(dim=dim,num_heads=num_heads,@@ -378,71 +543,164 @@ class SeqParallelAttention(Attention):proj_drop=proj_drop,norm_layer=norm_layer,enable_flash_attn=enable_flash_attn,+ rope=rope,+ qk_norm_legacy=qk_norm_legacy,+ kernel_size=kernel_size,+ shift_window=shift_window,)+ self.temporal = temporal- def forward(self, x: torch.Tensor) -> torch.Tensor:- B, N, C = x.shape # for sequence parallel here, the N is a local sequence length+ def forward(self, x: torch.Tensor, scale=None, H=None, W=None) -> torch.Tensor:+ sp_group = get_sequence_parallel_group()+ sp_size = dist.get_world_size(sp_group)qkv = self.qkv(x)- qkv_shape = (B, N, 3, self.num_heads, self.head_dim)- qkv = qkv.view(qkv_shape)- sp_group = get_sequence_parallel_group()+ if self.kernel_size is not None:+ B, *dims, C = x.shape+ N = np.prod(dims)+ qkv_shape = (B, N, 3, self.num_heads, self.head_dim)+ qkv = qkv.view(qkv_shape)- # apply all_to_all to gather sequence and split attention heads- # [B, SUB_N, 3, NUM_HEAD, HEAD_DIM] -> [B, N, 3, NUM_HEAD_PER_DEVICE, HEAD_DIM]- qkv = all_to_all(qkv, sp_group, scatter_dim=3, gather_dim=1)-- if self.enable_flash_attn:- qkv_permute_shape = (- 2,- 0,- 1,- 3,- 4,- ) # [3, B, N, NUM_HEAD_PER_DEVICE, HEAD_DIM]+ # apply all_to_all to gather sequence and split attention heads+ # [B, SUB_N, 3, NUM_HEAD, HEAD_DIM] -> [B, N, 3, NUM_HEAD_PER_DEVICE, HEAD_DIM]+ qkv = all_to_all(qkv, sp_group, scatter_dim=3, gather_dim=1)++ dims_sp = [dims[0] * sp_size, *dims[1:]]+ C_sp = C // sp_size+ qkv = qkv.view([qkv.shape[0], *dims_sp, 3 * C_sp])++ assert len(dims_sp) == len(+ self.kernel_size+ ), f"input shape and kernel size mismatch, {dims_sp} vs {self.kernel_size}"+ kernel_size = [(k if k >= 0 else s) for s, k in zip(dims_sp, self.kernel_size)]++ indices = list(range(1, x.dim() - 1))+ qkv, qkv_padding_mask = pad_to_multiples(+ qkv, kernel_size, indices, extra_pad_on_dims=self.extra_pad_on_dims+ )+ num_splits = [qkv.size(dim) // k for dim, k in zip(indices, kernel_size)]++ qkv = split_seq_cat_batch(qkv, kernel_size, indices)+ qkv_b = qkv.shape[0]+ qkv_n = np.prod(kernel_size)++ if qkv_padding_mask is not None:+ qkv_padding_mask = qkv_padding_mask.narrow(-1, 0, 1).squeeze(-1)+ qkv_padding_mask = split_seq_cat_batch(qkv_padding_mask, kernel_size, indices).to(qkv.dtype)+ qkv_padding_mask.masked_fill_(qkv_padding_mask.logical_not(), float("-inf"))+ qkv_padding_mask = qkv_padding_mask.view(qkv_b, qkv_n)+ qkv_padding_mask = qkv_padding_mask[:, None, :].expand(qkv_b, qkv_n, qkv_n).unsqueeze(1)++ qkv_shape = (qkv_b, qkv_n, 3, self.num_heads // sp_size, self.head_dim)+ qkv = qkv.view(qkv_shape)else:- qkv_permute_shape = (- 2,- 0,- 3,- 1,- 4,- ) # [3, B, NUM_HEAD_PER_DEVICE, N, HEAD_DIM]- qkv = qkv.permute(qkv_permute_shape)+ B, N, C = x.shape+ qkv_shape = (B, N, 3, self.num_heads, self.head_dim)+ qkv = qkv.view(qkv_shape)+ if not self.temporal:+ # apply all_to_all to gather sequence and split attention heads+ # [B, SUB_N, 3, NUM_HEAD, HEAD_DIM] -> [B, N, 3, NUM_HEAD_PER_DEVICE, HEAD_DIM]+ qkv = all_to_all(qkv, sp_group, scatter_dim=3, gather_dim=1)+ else:+ sp_size = 1+ qkv_b, qkv_n = qkv.shape[0], qkv.shape[1]+ qkv_padding_mask = None++ # flash attn is not memory efficient for small sequences, this is empirical+ enable_flash_attn = self.enable_flash_attn and (N > qkv_b) and self.kernel_size is None+ enable_sdpa = self.enable_flash_attn and (N > qkv_b) and self.kernel_size is not None+ qkv = qkv.permute(2, 0, 3, 1, 4)# ERROR: Should qk_norm firstq, k, v = qkv.unbind(0)- q, k = self.q_norm(q), self.k_norm(k)- if self.enable_flash_attn:+ if self.qk_norm_legacy:+ # WARNING: this may be a bug+ if self.rope:+ q = self.rotary_emb(q)+ k = self.rotary_emb(k)+ q, k = self.q_norm(q), self.k_norm(k)+ else:+ q, k = self.q_norm(q), self.k_norm(k)+ if self.rope:+ if self.kernel_size is not None:+ q = self.rotate_3d(q, self.kernel_size[0], self.kernel_size[1], scale=scale)+ k = self.rotate_3d(k, self.kernel_size[0], self.kernel_size[1], scale=scale)+ elif H is not None and W is not None:+ if self.temporal is False and self.kernel_size is None:+ H = H * sp_size+ q = self.rotate_3d(q, H, W, scale=scale)+ k = self.rotate_3d(k, H, W, scale=scale)+ else:+ q = self.rotary_emb(q)+ k = self.rotary_emb(k)++ if enable_flash_attn:from flash_attn import flash_attn_func+ # (B, #heads, N, #dim) -> (B, N, #heads, #dim)+ q = q.permute(0, 2, 1, 3)+ k = k.permute(0, 2, 1, 3)+ v = v.permute(0, 2, 1, 3)x = flash_attn_func(q,k,v,dropout_p=self.attn_drop.p if self.training else 0.0,softmax_scale=self.scale,+ causal=self.is_causal,+ )+ elif enable_sdpa:+ x = F.scaled_dot_product_attention(+ q,+ k,+ v,+ attn_mask=qkv_padding_mask,+ dropout_p=self.attn_drop.p if self.training else 0.0,+ scale=self.scale,+ is_causal=self.is_causal,)else:dtype = q.dtypeq = q * self.scaleattn = q @ k.transpose(-2, -1) # translate attn to float32attn = attn.to(torch.float32)+ if self.is_causal:+ causal_mask = torch.tril(torch.ones_like(attn), diagonal=0)+ causal_mask = torch.where(causal_mask.bool(), 0, float("-inf"))+ attn += causal_mask+ if qkv_padding_mask is not None:+ attn += qkv_padding_maskattn = attn.softmax(dim=-1)attn = attn.to(dtype) # cast back attn to original dtypeattn = self.attn_drop(attn)x = attn @ v- if not self.enable_flash_attn:- x = x.transpose(1, 2)+ if not enable_flash_attn:+ x = x.transpose(1, 2) # (B, #heads, N, #dim) -> (B, N, #heads, #dim)- # apply all to all to gather back attention heads and split sequence- # [B, N, NUM_HEAD_PER_DEVICE, HEAD_DIM] -> [B, SUB_N, NUM_HEAD, HEAD_DIM]- x = all_to_all(x, sp_group, scatter_dim=1, gather_dim=2)+ if self.kernel_size is not None:+ x = x.reshape(qkv_b, *kernel_size, C_sp)+ x = split_batch_cat_seq(x, B, num_splits, indices)+ x = remove_padding_nd(x, dims_sp, indices)++ x_shape = (B, N * sp_size, self.num_heads // sp_size, self.head_dim)+ x = x.reshape(x_shape)+ # apply all to all to gather back attention heads and split sequence+ # [B, N, NUM_HEAD_PER_DEVICE, HEAD_DIM] -> [B, SUB_N, NUM_HEAD, HEAD_DIM]+ x = all_to_all(x, sp_group, scatter_dim=1, gather_dim=2)++ x = x.view(B, *dims, C)++ assert x.shape == (B, *dims, C)+ else:+ if not self.temporal:+ # apply all to all to gather back attention heads and split sequence+ # [B, N, NUM_HEAD_PER_DEVICE, HEAD_DIM] -> [B, SUB_N, NUM_HEAD, HEAD_DIM]+ x = all_to_all(x, sp_group, scatter_dim=1, gather_dim=2)++ x_output_shape = (B, N, C)+ x = x.reshape(x_output_shape)- # reshape outputs back to [B, N, C]- x_output_shape = (B, N, C)- x = x.reshape(x_output_shape)x = self.proj(x)x = self.proj_drop(x)return x@@ -487,6 +745,40 @@ class MultiHeadCrossAttention(nn.Module):return x+class MultiHeadCrossAttentionForCondition(nn.Module):+ def __init__(self, d_model, num_heads, attn_drop=0.0, proj_drop=0.0):+ super().__init__()+ assert d_model % num_heads == 0, "d_model must be divisible by num_heads"++ self.d_model = d_model+ self.num_heads = num_heads+ self.head_dim = d_model // num_heads++ self.q_linear = nn.Linear(d_model, d_model)+ self.kv_linear = nn.Linear(d_model, d_model * 2)+ self.attn_drop = nn.Dropout(attn_drop)+ self.proj = nn.Linear(d_model, d_model)+ self.proj_drop = nn.Dropout(proj_drop)++ def forward(self, x, cond, mask=None):+ # query/value: img tokens; key: condition; mask: if padding tokens+ B, N, C = x.shape++ q = self.q_linear(cond).view(1, -1, self.num_heads, self.head_dim)+ kv = self.kv_linear(x).view(1, -1, 2, self.num_heads, self.head_dim)+ k, v = kv.unbind(2)++ attn_bias = None+ if mask is not None:+ attn_bias = xformers.ops.fmha.BlockDiagonalMask.from_seqlens(mask, [N] * B)+ cond = xformers.ops.memory_efficient_attention(q, k, v, p=self.attn_drop.p, attn_bias=attn_bias)++ cond = cond.view(1, -1, C)+ cond = self.proj(cond)+ cond = self.proj_drop(cond)+ return cond++class SeqParallelMultiHeadCrossAttention(MultiHeadCrossAttention):def __init__(self,@@ -740,23 +1032,25 @@ class CaptionEmbedder(nn.Module):)self.uncond_prob = uncond_prob- def token_drop(self, caption, force_drop_ids=None):+ def token_drop(self, caption, force_drop_ids=None, uncond_prob=None):"""Drops labels to enable classifier-free guidance."""+ uncond_prob = uncond_prob if uncond_prob is not None else self.uncond_probif force_drop_ids is None:- drop_ids = torch.rand(caption.shape[0]).cuda() < self.uncond_prob+ drop_ids = torch.rand(caption.shape[0]).cuda() < uncond_probelse:drop_ids = force_drop_ids == 1caption = torch.where(drop_ids[:, None, None, None], self.y_embedding, caption)return caption- def forward(self, caption, train, force_drop_ids=None):+ def forward(self, caption, train, force_drop_ids=None, uncond_prob=None):if train:assert caption.shape[2:] == self.y_embedding.shape- use_dropout = self.uncond_prob > 0+ uncond_prob = uncond_prob if uncond_prob is not None else self.uncond_prob+ use_dropout = uncond_prob > 0if (train and use_dropout) or (force_drop_ids is not None):- caption = self.token_drop(caption, force_drop_ids)+ caption = self.token_drop(caption, force_drop_ids=force_drop_ids, uncond_prob=uncond_prob)caption = self.y_proj(caption)return captiondiff --git a/opensora/models/layers/rotary_embedding_torch.py b/opensora/models/layers/rotary_embedding_torch.pynew file mode 100644index 0000000..ea5f823--- /dev/null+++ b/opensora/models/layers/rotary_embedding_torch.py@@ -0,0 +1,272 @@+from math import pi++import torch+from beartype import beartype+from beartype.typing import Literal, Optional, Union+from einops import rearrange, repeat+from torch import Tensor, broadcast_tensors, einsum, nn+from torch.cuda.amp import autocast+from torch.nn import Module++# helper functions+++def exists(val):+ return val is not None+++def default(val, d):+ return val if exists(val) else d+++# broadcat, as tortoise-tts was using it+++def broadcat(tensors, dim=-1):+ broadcasted_tensors = broadcast_tensors(*tensors)+ return torch.cat(broadcasted_tensors, dim=dim)+++# rotary embedding helper functions+++def rotate_half(x):+ x = rearrange(x, "... (d r) -> ... d r", r=2)+ x1, x2 = x.unbind(dim=-1)+ x = torch.stack((-x2, x1), dim=-1)+ return rearrange(x, "... d r -> ... (d r)")+++@autocast(enabled=False)+def apply_rotary_emb(freqs, t, start_index=0, scale=1.0, seq_dim=-2):+ if t.ndim == 3:+ seq_len = t.shape[seq_dim]+ freqs = freqs[-seq_len:].to(t)++ rot_dim = freqs.shape[-1]+ end_index = start_index + rot_dim++ assert (+ rot_dim <= t.shape[-1]+ ), f"feature dimension {t.shape[-1]} is not of sufficient size to rotate in all the positions {rot_dim}"++ t_left, t, t_right = t[..., :start_index], t[..., start_index:end_index], t[..., end_index:]+ t = (t * freqs.cos() * scale) + (rotate_half(t) * freqs.sin() * scale)+ return torch.cat((t_left, t, t_right), dim=-1)+++# learned rotation helpers+++def apply_learned_rotations(rotations, t, start_index=0, freq_ranges=None):+ if exists(freq_ranges):+ rotations = einsum("..., f -> ... f", rotations, freq_ranges)+ rotations = rearrange(rotations, "... r f -> ... (r f)")++ rotations = repeat(rotations, "... n -> ... (n r)", r=2)+ return apply_rotary_emb(rotations, t, start_index=start_index)+++# classes+++class RotaryEmbedding(Module):+ @beartype+ def __init__(+ self,+ dim,+ custom_freqs: Optional[Tensor] = None,+ freqs_for: Union[Literal["lang"], Literal["pixel"], Literal["constant"]] = "lang",+ theta=10000,+ max_freq=10,+ num_freqs=1,+ learned_freq=False,+ use_xpos=False,+ xpos_scale_base=512,+ interpolate_factor=1.0,+ theta_rescale_factor=1.0,+ seq_before_head_dim=False,+ cache_if_possible=True,+ ):+ super().__init__()+ # proposed by reddit user bloc97, to rescale rotary embeddings to longer sequence length without fine-tuning+ # has some connection to NTK literature+ # https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/++ theta *= theta_rescale_factor ** (dim / (dim - 2))++ self.freqs_for = freqs_for++ if exists(custom_freqs):+ freqs = custom_freqs+ elif freqs_for == "lang":+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))+ elif freqs_for == "pixel":+ freqs = torch.linspace(1.0, max_freq / 2, dim // 2) * pi+ elif freqs_for == "constant":+ freqs = torch.ones(num_freqs).float()++ self.cache_if_possible = cache_if_possible++ self.tmp_store("cached_freqs", None)+ self.tmp_store("cached_scales", None)++ self.freqs = nn.Parameter(freqs, requires_grad=learned_freq)++ self.learned_freq = learned_freq++ # dummy for device++ self.tmp_store("dummy", torch.tensor(0))++ # default sequence dimension++ self.seq_before_head_dim = seq_before_head_dim+ self.default_seq_dim = -3 if seq_before_head_dim else -2++ # interpolation factors++ assert interpolate_factor >= 1.0+ self.interpolate_factor = interpolate_factor++ # xpos++ self.use_xpos = use_xpos+ if not use_xpos:+ self.tmp_store("scale", None)+ return++ scale = (torch.arange(0, dim, 2) + 0.4 * dim) / (1.4 * dim)+ self.scale_base = xpos_scale_base+ self.tmp_store("scale", scale)++ @property+ def device(self):+ return self.dummy.device++ def tmp_store(self, key, value):+ self.register_buffer(key, value, persistent=False)++ def get_seq_pos(self, seq_len, device, dtype, offset=0, scale=None):+ interpolate_factor = self.interpolate_factor if scale is None else scale+ return (torch.arange(seq_len, device=device, dtype=dtype) + offset) / interpolate_factor++ def rotate_queries_or_keys(self, t, seq_dim=None, offset=0, freq_seq_len=None, scale=None):+ seq_dim = default(seq_dim, self.default_seq_dim)++ assert (+ not self.use_xpos+ ), "you must use `.rotate_queries_and_keys` method instead and pass in both queries and keys, for length extrapolatable rotary embeddings"++ device, dtype, seq_len = t.device, t.dtype, t.shape[seq_dim]++ if exists(freq_seq_len):+ assert freq_seq_len >= seq_len+ seq_len = freq_seq_len++ freqs = self.forward(+ self.get_seq_pos(seq_len, device=device, dtype=dtype, offset=offset, scale=scale),+ seq_len=seq_len,+ offset=offset,+ )++ if seq_dim == -3:+ freqs = rearrange(freqs, "n d -> n 1 d")++ return apply_rotary_emb(freqs, t, seq_dim=seq_dim)++ def rotate_queries_with_cached_keys(self, q, k, seq_dim=None, offset=0):+ seq_dim = default(seq_dim, self.default_seq_dim)++ q_len, k_len = q.shape[seq_dim], k.shape[seq_dim]+ assert q_len <= k_len+ rotated_q = self.rotate_queries_or_keys(q, seq_dim=seq_dim, freq_seq_len=k_len)+ rotated_k = self.rotate_queries_or_keys(k, seq_dim=seq_dim)++ rotated_q = rotated_q.type(q.dtype)+ rotated_k = rotated_k.type(k.dtype)++ return rotated_q, rotated_k++ def rotate_queries_and_keys(self, q, k, seq_dim=None):+ seq_dim = default(seq_dim, self.default_seq_dim)++ assert self.use_xpos+ device, dtype, seq_len = q.device, q.dtype, q.shape[seq_dim]++ seq = self.get_seq_pos(seq_len, dtype=dtype, device=device)++ freqs = self.forward(seq, seq_len=seq_len)+ scale = self.get_scale(seq, seq_len=seq_len).to(dtype)++ if seq_dim == -3:+ freqs = rearrange(freqs, "n d -> n 1 d")+ scale = rearrange(scale, "n d -> n 1 d")++ rotated_q = apply_rotary_emb(freqs, q, scale=scale, seq_dim=seq_dim)+ rotated_k = apply_rotary_emb(freqs, k, scale=scale**-1, seq_dim=seq_dim)++ rotated_q = rotated_q.type(q.dtype)+ rotated_k = rotated_k.type(k.dtype)++ return rotated_q, rotated_k++ @beartype+ def get_scale(self, t: Tensor, seq_len: Optional[int] = None, offset=0):+ assert self.use_xpos++ should_cache = self.cache_if_possible and exists(seq_len)++ if should_cache and exists(self.cached_scales) and (seq_len + offset) <= self.cached_scales.shape[0]:+ return self.cached_scales[offset : (offset + seq_len)]++ scale = 1.0+ if self.use_xpos:+ power = (t - len(t) // 2) / self.scale_base+ scale = self.scale ** rearrange(power, "n -> n 1")+ scale = torch.cat((scale, scale), dim=-1)++ if should_cache:+ self.tmp_store("cached_scales", scale)++ return scale++ def get_axial_freqs(self, *dims):+ Colon = slice(None)+ all_freqs = []++ for ind, dim in enumerate(dims):+ if self.freqs_for == "pixel":+ pos = torch.linspace(-1, 1, steps=dim, device=self.device)+ else:+ pos = torch.arange(dim, device=self.device)++ freqs = self.forward(pos, seq_len=dim)++ all_axis = [None] * len(dims)+ all_axis[ind] = Colon++ new_axis_slice = (Ellipsis, *all_axis, Colon)+ all_freqs.append(freqs[new_axis_slice])++ all_freqs = broadcast_tensors(*all_freqs)+ return torch.cat(all_freqs, dim=-1)++ @autocast(enabled=False)+ def forward(self, t: Tensor, seq_len=None, offset=0):+ should_cache = (+ self.cache_if_possible and not self.learned_freq and exists(seq_len) and self.freqs_for != "pixel"+ )++ if should_cache and exists(self.cached_freqs) and (offset + seq_len) <= self.cached_freqs.shape[0]:+ return self.cached_freqs[offset : (offset + seq_len)].detach()++ freqs = self.freqs++ freqs = einsum("..., f -> ... f", t.type(freqs.dtype), freqs)+ freqs = repeat(freqs, "... n -> ... (n r)", r=2)++ if should_cache:+ self.tmp_store("cached_freqs", freqs.detach())++ return freqsdiff --git a/opensora/models/layers/tiled_conv3d.py b/opensora/models/layers/tiled_conv3d.pynew file mode 100644index 0000000..f595bf9--- /dev/null+++ b/opensora/models/layers/tiled_conv3d.py@@ -0,0 +1,165 @@+import math+from typing import Union++import numpy as np+import torch+import torch.nn as nn+import torch.nn.functional as F+from torch.nn.modules.conv import _size_3_t+++class TiledConv3d(nn.Conv3d):+ """+ This is a PyTorch module for 3D Convolution with tiling.++ Args:+ num_tiles (int, optional): The number of tiles to partition the input tensor. If not given, the input tensor will be partitioned into tiles of size 16.+ tiled_dim (int, optional): The dimension to be tiled. If not given, the dimension with the largest size will be chosen.+ """++ def __init__(+ self,+ in_channels: int,+ out_channels: int,+ kernel_size: _size_3_t,+ stride: _size_3_t = 1,+ padding: Union[str, _size_3_t] = 0,+ dilation: _size_3_t = 1,+ groups: int = 1,+ bias: bool = True,+ padding_mode: str = "zeros",+ device=None,+ dtype=None,+ num_tiles=None, # if have num_tiles, use num_tiles, else use tile_size to determine+ tile_size=16,+ tiled_dim=None,+ exclude_temporal_dim=False,+ ):+ super(TiledConv3d, self).__init__(+ in_channels=in_channels,+ out_channels=out_channels,+ kernel_size=kernel_size,+ stride=stride,+ padding=padding,+ dilation=dilation,+ groups=groups,+ bias=bias,+ padding_mode=padding_mode,+ device=device,+ dtype=dtype,+ )++ self.num_tiles = num_tiles+ self.tile_size = tile_size+ self.exclude_temporal_dim = exclude_temporal_dim++ self.tiled_dim = tiled_dim+ if self.tiled_dim is not None:+ assert self.tiled_dim in [2, 3, 4], "tiled_dim should be 2, 3, or 4"++ def forward(self, x):+ assert self.padding_mode == "zeros", "Only support zero padding"++ # 1. determinte the tiled dim and num_tiles+ if self.tiled_dim is None:+ # if not given, choose the dim with th largest dimension+ B, C, T, H, W = x.size()+ if not self.exclude_temporal_dim:+ tiled_dim = np.argmax([T, H, W]).item() + 2+ else:+ tiled_dim = np.argmax([H, W]).item() + 3+ else:+ tiled_dim = self.tiled_dim++ if self.num_tiles is None:+ # if num_tiles is not given, tile the dim into 16+ num_tiles = x.size(tiled_dim) // self.tile_size+ else:+ num_tiles = self.num_tiles++ assert num_tiles < x.size(tiled_dim), "num_tiles should be less than the size of the tiled dimension"++ # we compute the number of conv ops in total+ # and compute the number of tiles+ # we then allocate the tiles based on the number of conv ops+ tiled_dim_size = x.size(tiled_dim)+ kernel_size = self.kernel_size[tiled_dim - 2]+ num_slibing_elements = (self.kernel_size[tiled_dim - 2] - 1) // 2+ stride = self.stride[tiled_dim - 2]+ padding = self.padding[tiled_dim - 2]+ dilation = self.dilation[tiled_dim - 2]+ assert dilation == 1, "Only support dilation=1"+ num_conv_ops = math.floor((tiled_dim_size + 2 * padding - dilation * (kernel_size - 1) - 1) / stride + 1)++ # distribute the conv ops into tiles+ conv_ops_per_partition = num_conv_ops // num_tiles+ remainder = num_conv_ops % num_tiles+ num_conv_ops_per_tile = [conv_ops_per_partition] * num_tiles+ for i in range(remainder):+ num_conv_ops_per_tile[i] += 1++ # 2. compute with tiling+ out_list = []+ conv_ops_processed = 0++ # we apply padding ahead of computation to make the tiling sharding easier+ padding_3d = [0, 0, 0, 0, 0, 0]+ padding_on_tiled_dim = self.padding[tiled_dim - 2]+ if padding_on_tiled_dim > 0:+ padding_3d[2 * (tiled_dim - 2)] = padding_on_tiled_dim+ padding_3d[2 * (tiled_dim - 2) + 1] = padding_on_tiled_dim+ x = F.pad(x, padding_3d, "constant", 0)+ padding_list = list(self.padding)+ padding_list[tiled_dim - 2] = 0++ for i in range(num_tiles):+ # comput the center point where convolution is applied+ start_idx = num_slibing_elements + stride * conv_ops_processed+ end_idx = start_idx + stride * num_conv_ops_per_tile[i]+ conv_ops_processed += num_conv_ops_per_tile[i]++ # the actual boundary requires considering the silbling elements+ start_idx -= num_slibing_elements+ end_idx += num_slibing_elements++ cur_partition = x.narrow(tiled_dim, start_idx, end_idx - start_idx)+ out_partition = F.conv3d(+ cur_partition, self.weight, self.bias, self.stride, padding_list, self.dilation, self.groups+ )+ out_list.append(out_partition)++ merged_out = torch.cat(out_list, dim=tiled_dim)++ return merged_out++ @staticmethod+ def from_native_conv3d(module: nn.Conv3d, num_tiles=None, tile_size=16, tiled_dim=None, exclude_temporal_dim=False):+ """+ This is a static method for easy conversion from the PyTorch native Conv3d module to the TiledConv3d module.++ Examples:+ tiled_conv3d = TiledConv3d.from_native_conv3d(conv3d, num_tiles=32, tiled_dim=3)+ """+ auto_tiled_conv3d = TiledConv3d(+ in_channels=module.in_channels,+ out_channels=module.out_channels,+ kernel_size=module.kernel_size,+ stride=module.stride,+ padding=module.padding,+ dilation=module.dilation,+ groups=module.groups,+ bias=module.bias is not None,+ padding_mode=module.padding_mode,+ device=module.weight.device,+ dtype=module.weight.dtype,+ num_tiles=num_tiles,+ tiled_dim=tiled_dim,+ tile_size=tile_size,+ exclude_temporal_dim=exclude_temporal_dim,+ )+ auto_tiled_conv3d.weight = module.weight+ auto_tiled_conv3d.bias = module.bias+ return auto_tiled_conv3d++ def __repr__(self):+ return f"{self.__class__.__name__}({self.in_channels}, {self.out_channels}, kernel_size={self.kernel_size}, stride={self.stride}, padding={self.padding}, dilation={self.dilation}, num_tiles={self.num_tiles}, tile_size={self.tile_size}, tiled_dim={self.tiled_dim}, exclude_temporal_dim={self.exclude_temporal_dim})"diff --git a/opensora/models/pixart/__init__.py b/opensora/models/pixart/__init__.pyindex 8f359d2..a069441 100644--- a/opensora/models/pixart/__init__.py+++ b/opensora/models/pixart/__init__.py@@ -1,2 +1,3 @@+from .head import PixArtHead_XL_2from .pixart import PixArt, PixArt_1B_2, PixArt_XL_2from .pixart_sigma import PixArt_Sigma_XL_2diff --git a/opensora/models/pixart/head.py b/opensora/models/pixart/head.pynew file mode 100644index 0000000..a37ed76--- /dev/null+++ b/opensora/models/pixart/head.py@@ -0,0 +1,218 @@+import numpy as np+import torch+import torch.nn as nn+from einops import rearrange++from opensora.acceleration.checkpoint import auto_grad_checkpoint+from opensora.models.layers.blocks import (+ CaptionEmbedder,+ PatchEmbed3D,+ PositionEmbedding2D,+ T2IFinalLayer,+ TimestepEmbedder,+ approx_gelu,+)+from opensora.models.pixart.pixart import PixArtBlock+from opensora.registry import MODELS+from opensora.utils.ckpt_utils import load_checkpoint++[email protected]_module()+class PixArtHead(nn.Module):+ """+ Diffusion model with a Transformer backbone.+ """++ def __init__(+ self,+ input_size=(None, None, None),+ input_sq_size=512,+ in_channels=4,+ patch_size=(1, 2, 2),+ hidden_size=1152,+ depth=28,+ num_heads=16,+ mlp_ratio=4.0,+ class_dropout_prob=0.1,+ pred_sigma=True,+ drop_path: float = 0.0,+ caption_channels=4096,+ model_max_length=120,+ enable_flash_attn=True,+ enable_layernorm_kernel=True,+ enable_sequence_parallelism=False,+ ):+ super().__init__()+ assert enable_sequence_parallelism is False, "Sequence parallelism is not supported in this version."+ self.pred_sigma = pred_sigma+ self.in_channels = in_channels+ self.out_channels = in_channels * 2 if pred_sigma else in_channels++ # model size related+ self.depth = depth+ self.mlp_ratio = mlp_ratio+ self.hidden_size = hidden_size+ self.num_heads = num_heads++ # computation related+ self.drop_path = drop_path+ self.enable_flash_attn = enable_flash_attn+ self.enable_layernorm_kernel = enable_layernorm_kernel++ # input size related+ self.patch_size = patch_size+ self.input_sq_size = input_sq_size+ self.pos_embed = PositionEmbedding2D(hidden_size)++ # embedding+ self.x_embedder = PatchEmbed3D(patch_size, in_channels, hidden_size)+ self.t_embedder = TimestepEmbedder(hidden_size)+ self.t_block = nn.Sequential(nn.SiLU(), nn.Linear(hidden_size, 6 * hidden_size, bias=True))+ self.y_embedder = CaptionEmbedder(+ in_channels=caption_channels,+ hidden_size=hidden_size,+ uncond_prob=class_dropout_prob,+ act_layer=approx_gelu,+ token_num=model_max_length,+ )++ # spatial blocks+ drop_path = [x.item() for x in torch.linspace(0, drop_path, depth)] # stochastic depth decay rule+ self.blocks = nn.ModuleList(+ [+ PixArtBlock(+ hidden_size,+ num_heads,+ mlp_ratio=mlp_ratio,+ drop_path=drop_path[i],+ enable_flash_attn=enable_flash_attn,+ enable_layernorm_kernel=enable_layernorm_kernel,+ )+ for i in range(depth)+ ]+ )++ # final layer+ self.final_layer = T2IFinalLayer(hidden_size, np.prod(self.patch_size), self.out_channels)++ @property+ def dtype(self):+ return self.x_embedder.proj.weight.dtype++ def get_dynamic_size(self, x):+ _, _, T, H, W = x.size()+ if T % self.patch_size[0] != 0:+ T += self.patch_size[0] - T % self.patch_size[0]+ if H % self.patch_size[1] != 0:+ H += self.patch_size[1] - H % self.patch_size[1]+ if W % self.patch_size[2] != 0:+ W += self.patch_size[2] - W % self.patch_size[2]+ T = T // self.patch_size[0]+ H = H // self.patch_size[1]+ W = W // self.patch_size[2]+ return (T, H, W)++ def get_base_size(self, S):+ base_sizes = torch.tensor([1, 2, 4, 8, 16, 32, 64, 128, 256, 512])+ value = S**0.5+ base_size = base_sizes[(base_sizes - value).abs().argmin()]+ return base_size++ def get_scale(self, height, width):+ scales = [0.5, 1, 2, 4, 8]+ resolution_sq = (height[0].item() * width[0].item()) ** 0.5+ scale = resolution_sq / self.input_sq_size+ scale = scales[(torch.tensor(scales) - scale).abs().argmin()]+ return scale++ def encode_text(self, y, mask=None):+ y = self.y_embedder(y, self.training) # [B, 1, N_token, C]+ if mask is not None:+ if mask.shape[0] != y.shape[0]:+ mask = mask.repeat(y.shape[0] // mask.shape[0], 1)+ mask = mask.squeeze(1).squeeze(1)+ y = y.squeeze(1).masked_select(mask.unsqueeze(-1) != 0).view(1, -1, self.hidden_size)+ y_lens = mask.sum(dim=1).tolist()+ else:+ y_lens = [y.shape[2]] * y.shape[0]+ y = y.squeeze(1).view(1, -1, self.hidden_size)+ return y, y_lens++ def forward(self, x, timestep, y, mask=None, height=None, width=None, cond=None, **kwargs):+ x, timestep, y = x.to(self.dtype), timestep.to(self.dtype), y.to(self.dtype)++ # === get pos embed ===+ _, _, Tx, Hx, Wx = x.size()+ assert Tx == 1, "T must be 1"+ T, H, W = self.get_dynamic_size(x)+ S = H * W+ base_size = self.get_base_size(S)+ scale = self.get_scale(height, width)+ pos_embed = self.pos_embed(x, H, W, scale=scale, base_size=base_size)++ # === get timestep embed ===+ t = self.t_embedder(timestep, dtype=x.dtype) # (N, D)+ t0 = self.t_block(t)++ # === get y embed ===+ if y.size(0) != x.size(0):+ t_mult = x.size(0) // y.size(0)+ y = y.repeat_interleave(t_mult, dim=0)+ mask = mask.repeat_interleave(t_mult, dim=0)+ y, y_lens = self.encode_text(y, mask)++ # === get x embed ===+ x = self.x_embedder(x) # (B, N, D)+ x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)+ x = x + pos_embed+ x = rearrange(x, "B T S C -> B (T S) C", T=T, S=S)++ # === add condition ===+ if cond is not None:+ x = x + cond++ # === blocks ===+ for block in self.blocks:+ x = auto_grad_checkpoint(block, x, y, t0, y_lens)++ # final process+ x = self.final_layer(x, t, T=T, S=S)+ x = self.unpatchify(x, T, H, W, Tx, Hx, Wx)++ # cast to float32 for better accuracy+ x = x.to(torch.float32)+ return x++ def unpatchify(self, x, N_t, N_h, N_w, R_t, R_h, R_w):+ """+ Args:+ x (torch.Tensor): of shape [B, N, C]++ Return:+ x (torch.Tensor): of shape [B, C_out, T, H, W]+ """++ # N_t, N_h, N_w = [self.input_size[i] // self.patch_size[i] for i in range(3)]+ T_p, H_p, W_p = self.patch_size+ x = rearrange(+ x,+ "B (N_t N_h N_w) (T_p H_p W_p C_out) -> B C_out (N_t T_p) (N_h H_p) (N_w W_p)",+ N_t=N_t,+ N_h=N_h,+ N_w=N_w,+ T_p=T_p,+ H_p=H_p,+ W_p=W_p,+ C_out=self.out_channels,+ )+ # unpad+ x = x[:, :, :R_t, :R_h, :R_w]+ return x++[email protected]_module("PixArt-HEAD-XL/2")+def PixArtHead_XL_2(from_pretrained=None, **kwargs):+ model = PixArtHead(depth=28, hidden_size=1152, patch_size=(1, 2, 2), num_heads=16, **kwargs)+ if from_pretrained is not None:+ load_checkpoint(model, from_pretrained)+ return modeldiff --git a/opensora/models/pixart/pixart.py b/opensora/models/pixart/pixart.pyindex d99c572..66c7ada 100644--- a/opensora/models/pixart/pixart.py+++ b/opensora/models/pixart/pixart.py@@ -205,7 +205,7 @@ class PixArt(nn.Module):y: (N, 1, 120, C) tensor of class labels"""dtype = self.x_embedder.proj.weight.dtype- B = x.size(0)+ x.size(0)x = x.to(dtype)timestep = timestep.to(dtype)y = y.to(dtype)diff --git a/opensora/models/stdit/stdit3.py b/opensora/models/stdit/stdit3.pyindex b0c046a..c17fe8b 100644--- a/opensora/models/stdit/stdit3.py+++ b/opensora/models/stdit/stdit3.py@@ -1,4 +1,5 @@import os+from typing import Optional, Sequenceimport numpy as npimport torch@@ -6,7 +7,6 @@ import torch.distributed as distimport torch.nn as nnimport torch.nn.functional as Ffrom einops import rearrange-from rotary_embedding_torch import RotaryEmbeddingfrom timm.models.layers import DropPathfrom timm.models.vision_transformer import Mlpfrom transformers import PretrainedConfig, PreTrainedModel@@ -29,6 +29,7 @@ from opensora.models.layers.blocks import (get_layernorm,t2i_modulate,)+from opensora.models.layers.rotary_embedding_torch import RotaryEmbeddingfrom opensora.registry import MODELSfrom opensora.utils.ckpt_utils import load_checkpoint@@ -46,6 +47,7 @@ class STDiT3Block(nn.Module):enable_flash_attn=False,enable_layernorm_kernel=False,enable_sequence_parallelism=False,+ kernel_size: Optional[Sequence[int]] = None,):super().__init__()self.temporal = temporal@@ -53,12 +55,14 @@ class STDiT3Block(nn.Module):self.enable_flash_attn = enable_flash_attnself.enable_sequence_parallelism = enable_sequence_parallelism- if self.enable_sequence_parallelism and not temporal:+ if self.enable_sequence_parallelism:attn_cls = SeqParallelAttentionmha_cls = SeqParallelMultiHeadCrossAttention+ attn_kwargs = dict(kernel_size=kernel_size, temporal=temporal)else:attn_cls = Attentionmha_cls = MultiHeadCrossAttention+ attn_kwargs = dict(kernel_size=kernel_size)self.norm1 = get_layernorm(hidden_size, eps=1e-6, affine=False, use_kernel=enable_layernorm_kernel)self.attn = attn_cls(@@ -68,6 +72,7 @@ class STDiT3Block(nn.Module):qk_norm=qk_norm,rope=rope,enable_flash_attn=enable_flash_attn,+ **attn_kwargs,)self.cross_attn = mha_cls(hidden_size, num_heads)self.norm2 = get_layernorm(hidden_size, eps=1e-6, affine=False, use_kernel=enable_layernorm_kernel)@@ -76,6 +81,7 @@ class STDiT3Block(nn.Module):)self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()self.scale_shift_table = nn.Parameter(torch.randn(6, hidden_size) / hidden_size**0.5)+ self.enable_2d_rope = rope is not None and not temporaldef t_mask_select(self, x_mask, x, masked_x, T, S):# x: [B, (T, S), C]@@ -97,6 +103,11 @@ class STDiT3Block(nn.Module):t0=None, # t with timestamp=0T=None, # number of framesS=None, # number of pixel patches+ H=None,+ W=None,+ scale=None,+ mask_index=None, # for i2v and v2v+ y_null=None, # for cfg training):# prepare modulate parametersB, N, C = x.shape@@ -116,13 +127,23 @@ class STDiT3Block(nn.Module):# attentionif self.temporal:- x_m = rearrange(x_m, "B (T S) C -> (B S) T C", T=T, S=S)- x_m = self.attn(x_m)- x_m = rearrange(x_m, "(B S) T C -> B (T S) C", T=T, S=S)+ if getattr(self.attn, "kernel_size", None) is not None:+ x_m = rearrange(x_m, "B (T H W) C -> B H W T C", T=T, H=H, W=W)+ x_m = self.attn(x_m, scale=scale)+ x_m = rearrange(x_m, "B H W T C -> B (T H W) C", T=T, H=H, W=W)+ else:+ x_m = rearrange(x_m, "B (T S) C -> (B S) T C", T=T, S=S)+ x_m = self.attn(x_m)+ x_m = rearrange(x_m, "(B S) T C -> B (T S) C", T=T, S=S)else:- x_m = rearrange(x_m, "B (T S) C -> (B T) S C", T=T, S=S)- x_m = self.attn(x_m)- x_m = rearrange(x_m, "(B T) S C -> B (T S) C", T=T, S=S)+ if self.enable_2d_rope:+ x_m = rearrange(x_m, "B (T S) C -> (B T) S C", T=T, S=S)+ x_m = self.attn(x_m, H=H, W=W, scale=scale)+ x_m = rearrange(x_m, "(B T) S C -> B (T S) C", T=T, S=S)+ else:+ x_m = rearrange(x_m, "B (T S) C -> (B T) S C", T=T, S=S)+ x_m = self.attn(x_m)+ x_m = rearrange(x_m, "(B T) S C -> B (T S) C", T=T, S=S)# modulate (attention)x_m_s = gate_msa * x_m@@ -134,7 +155,26 @@ class STDiT3Block(nn.Module):x = x + self.drop_path(x_m_s)# cross attention- x = x + self.cross_attn(x, y, mask)+ if mask_index is not None and len(mask_index) > 0: # i2v, v2v+ x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)+ x_rec = x[:, mask_index, :, :]+ assert len([i for i in mask_index if i < 0]) == 0 # mask_index cannot contain negative values+ rest_mask = [i for i in range(x.shape[1]) if i not in mask_index]+ x_pred = x[:, rest_mask, :, :]+ rec_frames = len(mask_index) # reconstruction+ pred_frames = len(rest_mask) # prediction+ x_rec = rearrange(x_rec, "B T S C -> B (T S) C", T=rec_frames, S=S)+ x_pred = rearrange(x_pred, "B T S C -> B (T S) C", T=pred_frames, S=S)+ x_rec = x_rec + self.cross_attn(x_rec, y_null, mask) # given video, remove text influence+ x_pred = x_pred + self.cross_attn(x_pred, y, mask)+ x_rec = rearrange(x_rec, "B (T S) C -> B T S C", T=rec_frames, S=S)+ x_pred = rearrange(x_pred, "B (T S) C -> B T S C", T=pred_frames, S=S)+ # concat according to mask_index and rest_mask positions, allow mask_index to be non_continuous+ x[:, mask_index] = x_rec+ x[:, rest_mask] = x_pred+ x = rearrange(x, "B T S C -> B (T S) C", T=T, S=S)+ else:+ x = x + self.cross_attn(x, y, mask)# modulate (MLP)x_m = t2i_modulate(self.norm2(x), shift_mlp, scale_mlp)@@ -164,7 +204,7 @@ class STDiT3Config(PretrainedConfig):self,input_size=(None, None, None),input_sq_size=512,- in_channels=4,+ in_channels=16,patch_size=(1, 2, 2),hidden_size=1152,depth=28,@@ -181,7 +221,10 @@ class STDiT3Config(PretrainedConfig):enable_sequence_parallelism=False,only_train_temporal=False,freeze_y_embedder=False,+ skip_temporal=False,skip_y_embedder=False,+ kernel_size=None,+ use_spatial_rope=False,**kwargs,):self.input_size = input_size@@ -204,6 +247,9 @@ class STDiT3Config(PretrainedConfig):self.only_train_temporal = only_train_temporalself.freeze_y_embedder = freeze_y_embedderself.skip_y_embedder = skip_y_embedder+ self.skip_temporal = skip_temporal+ self.kernel_size = kernel_size+ self.use_spatial_rope = use_spatial_ropesuper().__init__(**kwargs)@@ -232,10 +278,18 @@ class STDiT3(PreTrainedModel):self.patch_size = config.patch_sizeself.input_sq_size = config.input_sq_sizeself.pos_embed = PositionEmbedding2D(config.hidden_size)- self.rope = RotaryEmbedding(dim=self.hidden_size // self.num_heads)+ self.kernel_size = config.kernel_size+ rotation_dim = self.hidden_size // self.num_heads+ if self.kernel_size is not None:+ rotation_dim = rotation_dim // 3 # use 3D rotation+ self.rope = RotaryEmbedding(dim=rotation_dim, cache_if_possible=False)# embeddingself.x_embedder = PatchEmbed3D(config.patch_size, config.in_channels, config.hidden_size)+ # i2v/v2v cond embeddings+ self.x_embedder_cond = PatchEmbed3D(config.patch_size, config.in_channels, config.hidden_size)+ self.x_embedder_cond_mask = PatchEmbed3D(config.patch_size, config.in_channels, config.hidden_size)+self.t_embedder = TimestepEmbedder(config.hidden_size)self.fps_embedder = SizeEmbedder(self.hidden_size)self.t_block = nn.Sequential(@@ -252,6 +306,7 @@ class STDiT3(PreTrainedModel):# spatial blocksdrop_path = [x.item() for x in torch.linspace(0, self.drop_path, config.depth)]+ assert not config.use_spatial_rope or self.kernel_size is not None, "Spatial rope requires kernel_size"self.spatial_blocks = nn.ModuleList([STDiT3Block(@@ -263,31 +318,36 @@ class STDiT3(PreTrainedModel):enable_flash_attn=config.enable_flash_attn,enable_layernorm_kernel=config.enable_layernorm_kernel,enable_sequence_parallelism=config.enable_sequence_parallelism,+ # spatial+ rope=self.rope.rotate_queries_or_keys if config.use_spatial_rope else None,)for i in range(config.depth)])# temporal blocks- drop_path = [x.item() for x in torch.linspace(0, self.drop_path, config.depth)]- self.temporal_blocks = nn.ModuleList(- [- STDiT3Block(- hidden_size=config.hidden_size,- num_heads=config.num_heads,- mlp_ratio=config.mlp_ratio,- drop_path=drop_path[i],- qk_norm=config.qk_norm,- enable_flash_attn=config.enable_flash_attn,- enable_layernorm_kernel=config.enable_layernorm_kernel,- enable_sequence_parallelism=config.enable_sequence_parallelism,- # temporal- temporal=True,- rope=self.rope.rotate_queries_or_keys,- )- for i in range(config.depth)- ]- )+ self.skip_temporal = config.skip_temporal+ if not self.skip_temporal:+ drop_path = [x.item() for x in torch.linspace(0, self.drop_path, config.depth)]+ self.temporal_blocks = nn.ModuleList(+ [+ STDiT3Block(+ hidden_size=config.hidden_size,+ num_heads=config.num_heads,+ mlp_ratio=config.mlp_ratio,+ drop_path=drop_path[i],+ qk_norm=config.qk_norm,+ enable_flash_attn=config.enable_flash_attn,+ enable_layernorm_kernel=config.enable_layernorm_kernel,+ enable_sequence_parallelism=config.enable_sequence_parallelism,+ # temporal+ temporal=True,+ rope=self.rope.rotate_queries_or_keys,+ kernel_size=config.kernel_size,+ )+ for i in range(config.depth)+ ]+ )# final layerself.final_layer = T2IFinalLayer(config.hidden_size, np.prod(self.patch_size), self.out_channels)@@ -314,6 +374,12 @@ class STDiT3(PreTrainedModel):self.apply(_basic_init)+ # Initialize condition embedding+ torch.nn.init.zeros_(self.x_embedder_cond.proj.weight)+ torch.nn.init.zeros_(self.x_embedder_cond.proj.bias)+ torch.nn.init.zeros_(self.x_embedder_cond_mask.proj.weight)+ torch.nn.init.zeros_(self.x_embedder_cond_mask.proj.bias)+# Initialize fps_embeddernn.init.normal_(self.fps_embedder.mlp[0].weight, std=0.02)nn.init.constant_(self.fps_embedder.mlp[0].bias, 0)@@ -321,10 +387,11 @@ class STDiT3(PreTrainedModel):nn.init.constant_(self.fps_embedder.mlp[2].bias, 0)# Initialize timporal blocks- for block in self.temporal_blocks:- nn.init.constant_(block.attn.proj.weight, 0)- nn.init.constant_(block.cross_attn.proj.weight, 0)- nn.init.constant_(block.mlp.fc2.weight, 0)+ if not self.skip_temporal:+ for block in self.temporal_blocks:+ nn.init.constant_(block.attn.proj.weight, 0)+ nn.init.constant_(block.cross_attn.proj.weight, 0)+ nn.init.constant_(block.mlp.fc2.weight, 0)def get_dynamic_size(self, x):_, _, T, H, W = x.size()@@ -339,24 +406,41 @@ class STDiT3(PreTrainedModel):W = W // self.patch_size[2]return (T, H, W)- def encode_text(self, y, mask=None):- y = self.y_embedder(y, self.training) # [B, 1, N_token, C]+ def encode_text(self, y, mask=None, uncond_prob=None):+ y = self.y_embedder(y, self.training, uncond_prob=uncond_prob) # [B, 1, N_token, C]if mask is not None:if mask.shape[0] != y.shape[0]:mask = mask.repeat(y.shape[0] // mask.shape[0], 1)mask = mask.squeeze(1).squeeze(1)- y = y.squeeze(1).masked_select(mask.unsqueeze(-1) != 0).view(1, -1, self.hidden_size)+ y = (+ y.squeeze(1).masked_select(mask.unsqueeze(-1) != 0).view(1, -1, self.hidden_size)+ ) # [B, valid_id_count, 1152] --> [1, B*valid_id_count, 1152]y_lens = mask.sum(dim=1).tolist()else:y_lens = [y.shape[2]] * y.shape[0]y = y.squeeze(1).view(1, -1, self.hidden_size)return y, y_lens- def forward(self, x, timestep, y, mask=None, x_mask=None, fps=None, height=None, width=None, **kwargs):+ def forward(+ self,+ x,+ timestep,+ y,+ mask=None,+ x_mask=None,+ fps=None,+ height=None,+ width=None,+ cond=None,+ cond_mask=None,+ mask_index=None,+ y_null=None,+ text_uncond_prob=None,+ **kwargs,+ ):dtype = self.x_embedder.proj.weight.dtypeB = x.size(0)x = x.to(dtype)- timestep = timestep.to(dtype)y = y.to(dtype)# === get pos embed ===@@ -404,10 +488,17 @@ class STDiT3(PreTrainedModel):if isinstance(y_lens, torch.Tensor):y_lens = y_lens.long().tolist()else:- y, y_lens = self.encode_text(y, mask)+ y, y_lens = self.encode_text(y, mask, uncond_prob=text_uncond_prob)+ if y_null is not None:+ y_null, _ = self.encode_text(y_null, mask, uncond_prob=text_uncond_prob)# === get x embed ===x = self.x_embedder(x) # [B, N, C]++ # i2v, v2v+ if mask_index is not None:+ x = x + self.x_embedder_cond(cond) + self.x_embedder_cond_mask(cond_mask)+x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)x = x + pos_emb@@ -415,18 +506,69 @@ class STDiT3(PreTrainedModel):if self.enable_sequence_parallelism:x = split_forward_gather_backward(x, get_sequence_parallel_group(), dim=2, grad_scale="down")S = S // dist.get_world_size(get_sequence_parallel_group())+ H = H // dist.get_world_size(get_sequence_parallel_group())x = rearrange(x, "B T S C -> B (T S) C", T=T, S=S)# === blocks ===- for spatial_block, temporal_block in zip(self.spatial_blocks, self.temporal_blocks):- x = auto_grad_checkpoint(spatial_block, x, y, t_mlp, y_lens, x_mask, t0_mlp, T, S)- x = auto_grad_checkpoint(temporal_block, x, y, t_mlp, y_lens, x_mask, t0_mlp, T, S)+ if not self.skip_temporal:+ for spatial_block, temporal_block in zip(self.spatial_blocks, self.temporal_blocks):+ x = auto_grad_checkpoint(+ spatial_block,+ x,+ y,+ t_mlp,+ y_lens,+ x_mask,+ t0_mlp,+ T,+ S,+ H,+ W,+ scale,+ mask_index=mask_index,+ y_null=y_null,+ )+ x = auto_grad_checkpoint(+ temporal_block,+ x,+ y,+ t_mlp,+ y_lens,+ x_mask,+ t0_mlp,+ T,+ S,+ H,+ W,+ scale,+ mask_index=mask_index,+ y_null=y_null,+ )+ else:+ for spatial_block in self.spatial_blocks:+ x = auto_grad_checkpoint(+ spatial_block,+ x,+ y,+ t_mlp,+ y_lens,+ x_mask,+ t0_mlp,+ T,+ S,+ H,+ W,+ scale,+ mask_index=mask_index,+ y_null=y_null,+ )if self.enable_sequence_parallelism:x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)x = gather_forward_split_backward(x, get_sequence_parallel_group(), dim=2, grad_scale="up")S = S * dist.get_world_size(get_sequence_parallel_group())+ H = H * dist.get_world_size(get_sequence_parallel_group())x = rearrange(x, "B T S C -> B (T S) C", T=T, S=S)# === final layer ===@@ -467,13 +609,14 @@ class STDiT3(PreTrainedModel):@MODELS.register_module("STDiT3-XL/2")def STDiT3_XL_2(from_pretrained=None, **kwargs):force_huggingface = kwargs.pop("force_huggingface", False)+ adapt_16ch = kwargs.pop("adapt_16ch", False)if force_huggingface or from_pretrained is not None and not os.path.exists(from_pretrained):model = STDiT3.from_pretrained(from_pretrained, **kwargs)else:config = STDiT3Config(depth=28, hidden_size=1152, patch_size=(1, 2, 2), num_heads=16, **kwargs)model = STDiT3(config)if from_pretrained is not None:- load_checkpoint(model, from_pretrained)+ load_checkpoint(model, from_pretrained, adapt_16ch=adapt_16ch)return modeldiff --git a/opensora/models/text_encoder/classes.py b/opensora/models/text_encoder/classes.pyindex f02c9f2..86b6590 100644--- a/opensora/models/text_encoder/classes.py+++ b/opensora/models/text_encoder/classes.py@@ -12,9 +12,10 @@ class ClassEncoder:self.model_max_length = model_max_lengthself.output_dim = Noneself.device = device+ self.tokenize_fn = None- def encode(self, text):- return dict(y=torch.tensor([int(t) for t in text]).to(self.device))+ def encode(self, input_ids, attention_mask=None):+ return dict(y=input_ids.to(self.device))def null(self, n):return torch.tensor([self.num_classes] * n).to(self.device)diff --git a/opensora/models/text_encoder/clip.py b/opensora/models/text_encoder/clip.pyindex c628d02..c4a2c2b 100644--- a/opensora/models/text_encoder/clip.py+++ b/opensora/models/text_encoder/clip.py@@ -23,6 +23,8 @@# --------------------------------------------------------+from functools import partial+import torchimport torch.nn as nnimport transformers@@ -51,15 +53,8 @@ class FrozenCLIPEmbedder(AbstractEncoder):self.device = deviceself.max_length = max_lengthself._freeze()-- def _freeze(self):- self.transformer = self.transformer.eval()- for param in self.parameters():- param.requires_grad = False-- def forward(self, text):- batch_encoding = self.tokenizer(- text,+ self.tokenize_fn = partial(+ self.tokenizer,truncation=True,max_length=self.max_length,return_length=True,@@ -67,7 +62,14 @@ class FrozenCLIPEmbedder(AbstractEncoder):padding="max_length",return_tensors="pt",)- tokens = batch_encoding["input_ids"].to(self.device)++ def _freeze(self):+ self.transformer = self.transformer.eval()+ for param in self.parameters():+ param.requires_grad = False++ def forward(self, input_ids):+ tokens = input_ids.to(self.device)outputs = self.transformer(input_ids=tokens)z = outputs.last_hidden_state@@ -100,8 +102,12 @@ class ClipEncoder:self.model_max_length = model_max_lengthself.output_dim = self.text_encoder.transformer.config.hidden_size- def encode(self, text):- _, pooled_embeddings = self.text_encoder.encode(text)+ @property+ def tokenize_fn(self):+ return self.text_encoder.tokenize_fn++ def encode(self, input_ids, attention_mask=None):+ _, pooled_embeddings = self.text_encoder.encode(input_ids)y = pooled_embeddings.unsqueeze(1).unsqueeze(1)return dict(y=y)diff --git a/opensora/models/text_encoder/t5.py b/opensora/models/text_encoder/t5.pyindex 6378c34..c05abcb 100644--- a/opensora/models/text_encoder/t5.py+++ b/opensora/models/text_encoder/t5.py@@ -23,6 +23,7 @@import htmlimport re+from functools import partialimport ftfyimport torch@@ -109,10 +110,8 @@ class T5Embedder:**t5_model_kwargs,).eval()self.model_max_length = model_max_length-- def get_text_embeddings(self, texts):- text_tokens_and_mask = self.tokenizer(- texts,+ self.tokenize_fn = partial(+ self.tokenizer,max_length=self.model_max_length,padding="max_length",truncation=True,@@ -121,8 +120,10 @@ class T5Embedder:return_tensors="pt",)- input_ids = text_tokens_and_mask["input_ids"].to(self.device)- attention_mask = text_tokens_and_mask["attention_mask"].to(self.device)+ def get_text_embeddings(self, input_ids: torch.Tensor, attention_mask):+ input_ids = input_ids.to(self.device)+ if attention_mask is not None:+ attention_mask = attention_mask.to(self.device)with torch.no_grad():text_encoder_embs = self.model(input_ids=input_ids,@@ -170,8 +171,14 @@ class T5Encoder:from opensora.utils.misc import requires_gradshard_config = ShardConfig(+ tensor_parallel_process_group=None,+ pipeline_stage_manager=None,enable_tensor_parallelism=False,+ enable_fused_normalization=False,+ enable_flash_attention=False,enable_jit_fused=True,+ enable_sequence_parallelism=False,+ enable_sequence_overlap=False,)shard_former = ShardFormer(shard_config=shard_config)optim_model, _ = shard_former.optimize(self.t5.model, policy=T5EncoderPolicy())@@ -180,8 +187,12 @@ class T5Encoder:# ensure the weights are frozenrequires_grad(self.t5.model, False)- def encode(self, text):- caption_embs, emb_masks = self.t5.get_text_embeddings(text)+ @property+ def tokenize_fn(self):+ return self.t5.tokenize_fn++ def encode(self, input_ids, attention_mask=None):+ caption_embs, emb_masks = self.t5.get_text_embeddings(input_ids, attention_mask)caption_embs = caption_embs[:, None]return dict(y=caption_embs, mask=emb_masks)diff --git a/opensora/models/vae/video_sdxl/blocks.py b/opensora/models/vae/video_sdxl/blocks.pyindex 0ed4973..239733e 100644--- a/opensora/models/vae/video_sdxl/blocks.py+++ b/opensora/models/vae/video_sdxl/blocks.py@@ -3,7 +3,7 @@ Adapted from SDXL VAE (https://huggingface.co/stabilityai/sdxl-vae/blob/main/conAll default values of kwargs are the same as SDXL"""-from typing import Optional, Tuple, Union+from typing import Optionalimport torchimport torch.nn as nn@@ -16,20 +16,21 @@ def video_to_image(func):def wrapper(self, x, *args, **kwargs):if x.ndim == 5:B = x.shape[0]- x = rearrange(x, 'B C T H W -> (B T) C H W')+ x = rearrange(x, "B C T H W -> (B T) C H W")- if hasattr(self, 'micro_batch_size') and self.micro_batch_size is None:+ if hasattr(self, "micro_batch_size") and self.micro_batch_size is None:x = func(self, x, *args, **kwargs)else:bs = self.micro_batch_sizex_out = []for i in range(0, x.shape[0], bs):- x_i = func(self, x[i:i + bs], *args, **kwargs)+ x_i = func(self, x[i : i + bs], *args, **kwargs)x_out.append(x_i)x = torch.cat(x_out, dim=0)- x = rearrange(x, '(B T) C H W -> B C T H W', B=B)+ x = rearrange(x, "(B T) C H W -> B C T H W", B=B)return x+return wrapper@@ -45,11 +46,12 @@ class VideoConv2d(nn.Conv2d):class ResnetBlock2D(nn.Module):"""- Use nn.Conv2d- Default activation is nn.SiLU()- Make sure input tensor is of shape [B, C, T, H, W] or [B, C, H, W]- Support micro_batch_size+ Use nn.Conv2d+ Default activation is nn.SiLU()+ Make sure input tensor is of shape [B, C, T, H, W] or [B, C, H, W]+ Support micro_batch_size"""+def __init__(self,in_channels: int,@@ -104,10 +106,11 @@ class ResnetBlock2D(nn.Module):class ResnetBlock3D(nn.Module):"""- Use nn.Conv3d- Default activation is nn.SiLU()- Make sure input tensor is of shape [B, C, T, H, W]+ Use nn.Conv3d+ Default activation is nn.SiLU()+ Make sure input tensor is of shape [B, C, T, H, W]"""+def __init__(self,in_channels: int,@@ -140,7 +143,7 @@ class ResnetBlock3D(nn.Module):stride=1,padding=0,)-+def forward(self, x):res = self.norm1(x)res = self.act(res)@@ -159,10 +162,11 @@ class ResnetBlock3D(nn.Module):class SpatialDownsample2x(nn.Module):"""- Default downsample is Conv2d(stride=2)- Make sure input tensor is of shape [B, C, T, H, W]- Support micro_batch_size+ Default downsample is Conv2d(stride=2)+ Make sure input tensor is of shape [B, C, T, H, W]+ Support micro_batch_size"""+def __init__(self,channels: int,@@ -176,7 +180,11 @@ class SpatialDownsample2x(nn.Module):if use_conv:self.downsample = nn.Conv2d(- self.channels, self.channels, kernel_size=3, stride=2, padding=0,+ self.channels,+ self.channels,+ kernel_size=3,+ stride=2,+ padding=0,)else:self.downsample = nn.AvgPool2d(kernel_size=2, stride=2)@@ -193,10 +201,11 @@ class SpatialDownsample2x(nn.Module):class SpatialUpsample2x(nn.Module):"""- Default upsample is F.interpolate(scale_factor=2) + Conv2d(stride=1)- Make sure input tensor is of shape [B, C, T, H, W]- Support micro_batch_size+ Default upsample is F.interpolate(scale_factor=2) + Conv2d(stride=1)+ Make sure input tensor is of shape [B, C, T, H, W]+ Support micro_batch_size"""+def __init__(self,channels: int,@@ -213,10 +222,10 @@ class SpatialUpsample2x(nn.Module):else:raise NotImplementedErrorself.upsample = nn.ConvTranspose2d(channels, self.channels, kernel_size=4, stride=2, padding=1)-+def forward(self, x):B = x.shape[0]- x = rearrange(x, 'B C T H W -> (B T) C H W')+ x = rearrange(x, "B C T H W -> (B T) C H W")if self.micro_batch_size is None:x = self.forward_BCHW(x)@@ -224,11 +233,11 @@ class SpatialUpsample2x(nn.Module):bs = self.micro_batch_sizex_out = []for i in range(0, x.shape[0], bs):- x_i = self.forward_BCHW(x[i:i + bs])+ x_i = self.forward_BCHW(x[i : i + bs])x_out.append(x_i)x = torch.cat(x_out, dim=0)- x = rearrange(x, '(B T) C H W -> B C T H W', B=B)+ x = rearrange(x, "(B T) C H W -> B C T H W", B=B)return xdef forward_BCHW(self, x):@@ -248,9 +257,10 @@ class SpatialUpsample2x(nn.Module):class TemporalDownsample2x(nn.Module):"""- Default downsample is Conv3d(stride=(2, 1, 1))- Make sure input tensor is of shape [B, C, T, H, W]+ Default downsample is Conv3d(stride=(2, 1, 1))+ Make sure input tensor is of shape [B, C, T, H, W]"""+def __init__(self,channels: int,@@ -262,8 +272,12 @@ class TemporalDownsample2x(nn.Module):if use_conv:self.downsample = nn.Conv3d(- self.channels, self.channels, kernel_size=(3, 3, 3), stride=(2, 1, 1), padding=(1, 1, 1),- )+ self.channels,+ self.channels,+ kernel_size=(3, 3, 3),+ stride=(2, 1, 1),+ padding=(1, 1, 1),+ )else:self.downsample = nn.AvgPool3d(kernel_size=(3, 1, 1), stride=(2, 1, 1))@@ -274,10 +288,11 @@ class TemporalDownsample2x(nn.Module):class TemporalUpsample2x(nn.Module):"""- Default upsample is F.interpolate(scale_factor=(2, 1, 1)) + Conv3d(stride=1)- Make sure input tensor is of shape [B, C, T, H, W]- Support micro_batch_size+ Default upsample is F.interpolate(scale_factor=(2, 1, 1)) + Conv3d(stride=1)+ Make sure input tensor is of shape [B, C, T, H, W]+ Support micro_batch_size"""+def __init__(self,channels,@@ -296,9 +311,10 @@ class TemporalUpsample2x(nn.Module):class UNetMidBlock2D(nn.Module):"""- default is ResnetBlock2D + Spatial Attention + ResnetBlock2D- Make sure input tensor is of shape [B, C, T, H, W] or [B, C, H, W]+ default is ResnetBlock2D + Spatial Attention + ResnetBlock2D+ Make sure input tensor is of shape [B, C, T, H, W] or [B, C, H, W]"""+def __init__(self,in_channels: int,@@ -364,7 +380,7 @@ class UNetMidBlock2D(nn.Module):has_T = x.ndim == 5if has_T:B = x.shape[0]- x = rearrange(x, 'B C T H W -> (B T) C H W')+ x = rearrange(x, "B C T H W -> (B T) C H W")x = self.res_blocks[0](x)for attn, res_block in zip(self.attn_blocks, self.res_blocks[1:]):@@ -373,15 +389,16 @@ class UNetMidBlock2D(nn.Module):x = res_block(x)if has_T:- x = rearrange(x, '(B T) C H W -> B C T H W', B=B)+ x = rearrange(x, "(B T) C H W -> B C T H W", B=B)return xclass Encoder(nn.Module):"""- default arch is conv_in + blocks + mid_block + out_block- Make sure input tensor is of shape [B, C, T, H, W]+ default arch is conv_in + blocks + mid_block + out_block+ Make sure input tensor is of shape [B, C, T, H, W]"""+def __init__(self,in_channels=3,@@ -431,7 +448,7 @@ class Encoder(nn.Module):SpatialDownsample2x(channels=out_channels,use_conv=True,- micro_batch_size=micro_batch_size,+ micro_batch_size=micro_batch_size,),))@@ -458,12 +475,12 @@ class Encoder(nn.Module):SpatialDownsample2x(channels=out_channels,use_conv=True,- micro_batch_size=micro_batch_size,+ micro_batch_size=micro_batch_size,),TemporalDownsample2x(channels=out_channels,use_conv=True,- )+ ),))@@ -491,7 +508,7 @@ class Encoder(nn.Module):TemporalDownsample2x(channels=out_channels,use_conv=True,- )+ ),))@@ -517,7 +534,6 @@ class Encoder(nn.Module):self.blocks = nn.ModuleList(blocks)-# mid_blockin_channels = block_out_channels[-1]self.mid_block = UNetMidBlock2D(@@ -537,7 +553,7 @@ class Encoder(nn.Module):nn.SiLU(),nn.Conv3d(in_channels, out_channels, kernel_size=3, padding=1),)-+def forward(self, x):x = self.conv_in(x)@@ -552,9 +568,10 @@ class Encoder(nn.Module):class Decoder(nn.Module):"""- default arch is conv_in + mid_block + blocks + out_block- Make sure input tensor is of shape [B, C, T, H, W]+ default arch is conv_in + mid_block + blocks + out_block+ Make sure input tensor is of shape [B, C, T, H, W]"""+def __init__(self,in_channels=4,@@ -596,7 +613,7 @@ class Decoder(nn.Module):out_channels = block_out_channels[0]seq = [ResnetBlock3D(- in_channels=in_channels if idx ==0 else out_channels,+ in_channels=in_channels if idx == 0 else out_channels,out_channels=out_channels,norm_groups=norm_groups,norm_eps=norm_eps,@@ -618,7 +635,7 @@ class Decoder(nn.Module):out_channels = block_out_channels[1]seq = [ResnetBlock3D(- in_channels=in_channels if idx ==0 else out_channels,+ in_channels=in_channels if idx == 0 else out_channels,out_channels=out_channels,norm_groups=norm_groups,norm_eps=norm_eps,@@ -640,7 +657,7 @@ class Decoder(nn.Module):out_channels = block_out_channels[2]seq = [ResnetBlock3D(- in_channels=in_channels if idx ==0 else out_channels,+ in_channels=in_channels if idx == 0 else out_channels,out_channels=out_channels,norm_groups=norm_groups,norm_eps=norm_eps,@@ -659,7 +676,7 @@ class Decoder(nn.Module):out_channels = block_out_channels[3]seq = [ResnetBlock2D(- in_channels=in_channels if idx ==0 else out_channels,+ in_channels=in_channels if idx == 0 else out_channels,out_channels=out_channels,norm_groups=norm_groups,norm_eps=norm_eps,@@ -681,44 +698,56 @@ class Decoder(nn.Module):def forward(self, x):x = self.conv_in(x)- print(torch.cuda.memory_allocated() / 1024 ** 3)+ print(torch.cuda.memory_allocated() / 1024**3)x = self.mid_block(x)- print(torch.cuda.memory_allocated() / 1024 ** 3)+ print(torch.cuda.memory_allocated() / 1024**3)for block in self.blocks:x = block(x)- print(torch.cuda.memory_allocated() / 1024 ** 3)+ print(torch.cuda.memory_allocated() / 1024**3)x = self.out_block(x)- print(torch.cuda.memory_allocated() / 1024 ** 3)+ print(torch.cuda.memory_allocated() / 1024**3)return x-if __name__ == '__main__':++if __name__ == "__main__":from opensora.utils.misc import count_params- device = 'cuda'- dtype = torch.bfloat16- encoder = Encoder(- in_channels=3,- out_channels=4,- double_z=False,- micro_batch_size=4,- ).to(torch.bfloat16).to(device, dtype).eval()+ device = "cuda"+ dtype = torch.bfloat16- decoder = Decoder(- in_channels=4,- out_channels=3,- ).to(torch.bfloat16).to(device, dtype).eval()+ encoder = (+ Encoder(+ in_channels=3,+ out_channels=4,+ double_z=False,+ micro_batch_size=4,+ )+ .to(torch.bfloat16)+ .to(device, dtype)+ .eval()+ )++ decoder = (+ Decoder(+ in_channels=4,+ out_channels=3,+ )+ .to(torch.bfloat16)+ .to(device, dtype)+ .eval()+ )num_params_enc = count_params(encoder)num_params_dec = count_params(decoder)- print(f'Encoder #params: {num_params_enc}')- print(f'Decoder #params: {num_params_dec}')+ print(f"Encoder #params: {num_params_enc}")+ print(f"Decoder #params: {num_params_dec}")# inferencex = torch.rand(1, 3, 51, 720, 1080).to(device, dtype)with torch.inference_mode():x_enc = encoder(x)x_dec = decoder(x_enc)- print(torch.cuda.memory_allocated() / 1024 ** 3)+ print(torch.cuda.memory_allocated() / 1024**3)breakpoint()diff --git a/opensora/models/vae_v1_3/__init__.py b/opensora/models/vae_v1_3/__init__.pynew file mode 100644index 0000000..13ec032--- /dev/null+++ b/opensora/models/vae_v1_3/__init__.py@@ -0,0 +1,4 @@+from .decoder import VideoDecoder+from .discriminator import N_LAYER_DISCRIMINATOR_3D+from .encoder import VideoEncoder+from .vae import OpenSoraVAE_V1_3, OpenSoraVAE_V1_3_Piplinediff --git a/opensora/models/vae_v1_3/constants.py b/opensora/models/vae_v1_3/constants.pynew file mode 100644index 0000000..b07d628--- /dev/null+++ b/opensora/models/vae_v1_3/constants.py@@ -0,0 +1,300 @@+import torch++VAE_STAT_DICT = {+ "image": {+ "mean": [+ 0.10250157117843628,+ 0.016017071902751923,+ -0.0014998418046161532,+ -0.12508268654346466,+ 0.07124370336532593,+ 0.01971251890063286,+ 0.09438388794660568,+ -0.053667355328798294,+ 0.02179020643234253,+ -0.059147659689188004,+ -0.10370352864265442,+ -0.023039299994707108,+ -0.039374228566884995,+ 0.25110629200935364,+ 0.009911992587149143,+ 0.024221234023571014,+ ],+ "std": [+ 0.9946285486221313,+ 1.0125455856323242,+ 0.9564010500907898,+ 0.982365608215332,+ 1.0083516836166382,+ 1.0457563400268555,+ 1.005329966545105,+ 0.9874873757362366,+ 1.0161429643630981,+ 0.9727572798728943,+ 0.9623528718948364,+ 0.9576594829559326,+ 0.9431302547454834,+ 0.9041504859924316,+ 0.9532864093780518,+ 1.0081173181533813,+ ],+ },+ "video_5frames": {+ "mean": [+ [+ -0.31038138270378113,+ -0.17488302290439606,+ -0.13900649547576904,+ -0.11228419095277786,+ -0.21430839598178864,+ ],+ [+ 0.08975527435541153,+ -0.055954232811927795,+ -0.07017382979393005,+ -0.1284228265285492,+ -0.059222422540187836,+ ],+ [+ 0.009729883633553982,+ 0.02230699174106121,+ 0.023845115676522255,+ 0.023336032405495644,+ 0.024730036035180092,+ ],+ [+ 0.106182761490345,+ -0.10902953892946243,+ 0.06071978434920311,+ 0.10005339235067368,+ -0.06466034799814224,+ ],+ [+ 0.029384655877947807,+ 0.05276906490325928,+ -0.012805480509996414,+ -0.030368609353899956,+ -0.09690208733081818,+ ],+ [+ 0.017169909551739693,+ -0.007233283948153257,+ 0.0011372794397175312,+ -0.0029931864701211452,+ -0.002679176162928343,+ ],+ [+ 0.1150113120675087,+ 0.08740745484828949,+ 0.07578480243682861,+ 0.07428467273712158,+ 0.06640278548002243,+ ],+ [+ 0.04360577464103699,+ 0.12531928718090057,+ 0.0574030801653862,+ 0.09716417640447617,+ 0.12161172926425934,+ ],+ [+ -0.0038005930837243795,+ 0.0669507160782814,+ 0.0037457006983458996,+ 0.0006462433375418186,+ -0.02049373835325241,+ ],+ [+ 0.09442657232284546,+ 0.04747205227613449,+ 0.04356396198272705,+ 0.05506575107574463,+ 0.16380564868450165,+ ],+ [+ -0.17690078914165497,+ -0.1373550444841385,+ -0.022591449320316315,+ 0.033208176493644714,+ -0.0589708648622036,+ ],+ [+ 0.049464937299489975,+ 0.06297765672206879,+ 0.04696730151772499,+ 0.04245228320360184,+ 0.05074438825249672,+ ],+ [+ -0.052013952285051346,+ -0.03836681321263313,+ -0.0018303691176697612,+ -0.005724515300244093,+ -0.0069338483735919,+ ],+ [+ -0.0682193860411644,+ -0.13791534304618835,+ -0.14595246315002441,+ -0.1195555180311203,+ -0.13990911841392517,+ ],+ [+ -0.009703801944851875,+ -0.014363707974553108,+ -0.005133059341460466,+ 0.006809945683926344,+ 0.0030800539534538984,+ ],+ [+ -0.14393238723278046,+ -0.009123840369284153,+ -0.01863870956003666,+ -0.005709421820938587,+ 0.062058091163635254,+ ],+ ],+ "std": [+ [0.9969105124473572, 0.9459409117698669, 0.9167423248291016, 0.9288502335548401, 0.8993298411369324],+ [1.0192935466766357, 0.941669225692749, 0.9346942901611328, 0.9353693723678589, 0.9003037214279175],+ [0.9833933711051941, 0.9082298278808594, 0.9308426976203918, 0.9624825716018677, 0.9349731802940369],+ [0.9718613624572754, 0.9105200171470642, 0.8878195881843567, 0.8698861598968506, 0.8630944490432739],+ [1.0202057361602783, 0.9447319507598877, 0.9330360889434814, 0.9632452130317688, 0.9210478663444519],+ [1.0385587215423584, 0.8876767754554749, 0.7368916869163513, 0.7580698728561401, 0.7508725523948669],+ [1.0592126846313477, 0.8657042980194092, 0.8215179443359375, 0.8476718068122864, 0.820949375629425],+ [0.9920971989631653, 0.9271690249443054, 0.8988684415817261, 0.9289981722831726, 0.8869122266769409],+ [1.035900592803955, 0.9599395394325256, 0.9465612769126892, 0.9801756739616394, 0.934220016002655],+ [0.9621859788894653, 0.8828722238540649, 0.8710391521453857, 0.880846381187439, 0.8516324758529663],+ [0.9560437202453613, 0.8453353643417358, 0.8308913111686707, 0.8258389830589294, 0.8010841012001038],+ [0.9879955649375916, 0.8114659786224365, 0.75618976354599, 0.7889909744262695, 0.7724364995956421],+ [0.958446204662323, 0.885802686214447, 0.8992094397544861, 0.9229688048362732, 0.884797990322113],+ [0.9060379266738892, 0.7863245606422424, 0.7374365329742432, 0.7614732980728149, 0.7428738474845886],+ [0.9648597836494446, 0.8712865114212036, 0.8902931213378906, 0.8947092890739441, 0.8709816336631775],+ [1.0170196294784546, 0.9330487847328186, 0.9144400954246521, 0.9294652938842773, 0.8871583342552185],+ ],+ },+ "video": {+ "mean": [+ -0.19016949832439423,+ -0.04480062797665596,+ 0.020789705216884613,+ 0.01865335740149021,+ -0.011585569009184837,+ 0.0010865575168281794,+ 0.08378933370113373,+ 0.08902134001255035,+ 0.00940945465117693,+ 0.08086848258972168,+ -0.07251627743244171,+ 0.05052144080400467,+ -0.0209724809974432,+ -0.1223079189658165,+ -0.003862356534227729,+ -0.023072388023138046,+ ],+ "std": [+ 0.9490776062011719,+ 0.9594117999076843,+ 0.9444302916526794,+ 0.9103284478187561,+ 0.959351658821106,+ 0.8430351614952087,+ 0.8888124823570251,+ 0.9291136860847473,+ 0.972594141960144,+ 0.8917953968048096,+ 0.8630853891372681,+ 0.8279891610145569,+ 0.9114300012588501,+ 0.7922844886779785,+ 0.8991407155990601,+ 0.9473513960838318,+ ],+ },+ "video_head": {+ "mean": [+ -0.31038138270378113,+ 0.08975527435541153,+ 0.009729883633553982,+ 0.106182761490345,+ 0.029384655877947807,+ 0.017169909551739693,+ 0.1150113120675087,+ 0.04360577464103699,+ -0.0038005930837243795,+ 0.09442657232284546,+ -0.17690078914165497,+ 0.049464937299489975,+ -0.052013952285051346,+ -0.0682193860411644,+ -0.009703801944851875,+ -0.14393238723278046,+ ],+ "std": [+ 0.9969105124473572,+ 1.0192935466766357,+ 0.9833933711051941,+ 0.9718613624572754,+ 1.0202057361602783,+ 1.0385587215423584,+ 1.0592126846313477,+ 0.9920971989631653,+ 1.035900592803955,+ 0.9621859788894653,+ 0.9560437202453613,+ 0.9879955649375916,+ 0.958446204662323,+ 0.9060379266738892,+ 0.9648597836494446,+ 1.0170196294784546,+ ],+ },+ "video_tail_mean": {+ "mean": [+ -0.16012459993362427,+ -0.07843801379203796,+ 0.023553527891635895,+ -0.0032353713177144527,+ -0.02182665653526783,+ -0.0029409679118543863,+ 0.0759628489613533,+ 0.1003696396946907,+ 0.012710502371191978,+ 0.07747535407543182,+ -0.04642125591635704,+ 0.05078204721212387,+ -0.013212214224040508,+ -0.13583600521087646,+ -0.0023993775248527527,+ 0.007145590148866177,+ ],+ "std": [+ 0.9236865043640137,+ 0.9286295175552368,+ 0.9343323707580566,+ 0.8872344493865967,+ 0.9421619772911072,+ 0.7857372164726257,+ 0.8392113447189331,+ 0.9110831022262573,+ 0.955938994884491,+ 0.8731299638748169,+ 0.8282777667045593,+ 0.7825783491134644,+ 0.8984475135803223,+ 0.7573372721672058,+ 0.8819227814674377,+ 0.9167794585227966,+ ],+ },+}++for key in VAE_STAT_DICT.keys():+ VAE_STAT_DICT[key]["mean"] = torch.tensor(VAE_STAT_DICT[key]["mean"])+ VAE_STAT_DICT[key]["std"] = torch.tensor(VAE_STAT_DICT[key]["std"])+++def get_vae_stats(name, z):+ mean = VAE_STAT_DICT[name]["mean"].to(z.device)+ std = VAE_STAT_DICT[name]["std"].to(z.device)+ return mean, stddiff --git a/opensora/models/vae_v1_3/decoder.py b/opensora/models/vae_v1_3/decoder.pynew file mode 100644index 0000000..b119018--- /dev/null+++ b/opensora/models/vae_v1_3/decoder.py@@ -0,0 +1,145 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/vae/decoder.py+++import torch.nn as nn+import torch.nn.functional as F++from opensora.models.layers.tiled_conv3d import TiledConv3d+from opensora.registry import MODELS++from .modules.attention import AttnBlock+from .modules.conv import CausalConv3dPlainAR+from .modules.ops import Normalize, nonlinearity+from .modules.resblock import Resnet3DBlock+from .modules.updownsample import Upsample2D, Upsample3D++[email protected]_module()+class VideoDecoder(nn.Module):+ def __init__(+ self,+ ch,+ out_ch,+ in_channels,+ z_channels,+ ch_mult=(1, 2, 4, 8),+ num_res_blocks=2,+ dropout=0.0,+ resamp_with_conv=True,+ temporal_up_layers=[2, 3],+ temporal_downsample=4,+ micro_batch_size_2d=None,+ **ignorekwargs,+ ):+ super().__init__()+ self.ch = ch+ self.temb_ch = 0+ self.num_resolutions = len(ch_mult)+ self.num_res_blocks = num_res_blocks+ self.in_channels = in_channels+ self.temporal_up_layers = temporal_up_layers+ self.temporal_downsample = temporal_downsample++ (1,) + tuple(ch_mult)+ block_in = ch * ch_mult[self.num_resolutions - 1]+ self.conv_in = CausalConv3dPlainAR(z_channels, block_in, kernel_size=3)+ # middle+ self.mid = nn.Module()+ self.mid.block_1 = Resnet3DBlock(+ in_channels=block_in,+ out_channels=block_in,+ dropout=dropout,+ )+ self.mid.attn_1 = AttnBlock(block_in)+ self.mid.block_2 = Resnet3DBlock(+ in_channels=block_in,+ out_channels=block_in,+ dropout=dropout,+ )++ # upsampling+ self.up_id = len(self.temporal_up_layers)+ self.up = nn.ModuleList()+ for i_level in reversed(range(self.num_resolutions)):+ block = nn.ModuleList()+ block_out = ch * ch_mult[i_level]+ for i_block in range(self.num_res_blocks + 1):+ block.append(+ Resnet3DBlock(+ in_channels=block_in,+ out_channels=block_out,+ dropout=dropout,+ )+ )+ block_in = block_out+ up = nn.Module()+ up.block = block+ if i_level != 0:+ if i_level in self.temporal_up_layers:+ up.upsample = Upsample3D(block_in, resamp_with_conv)+ else:+ up.upsample = Upsample2D(block_in, resamp_with_conv, micro_batch_size_2d=micro_batch_size_2d)+ self.up.insert(0, up) # prepend to get consistent order++ # end+ self.norm_out = Normalize(block_in)+ self.time_causal_padding = (1, 1, 1, 1, 2, 0)+ self.time_uncausal_padding = (1, 1, 1, 1, 0, 0)+ self.conv_out = nn.Conv3d(block_in, out_ch, kernel_size=3)++ def get_last_layer(self):+ return self.conv_out.weight++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ if hasattr(self.conv_in, "_enable_tiled_conv3d"):+ self.conv_in._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ if hasattr(self.mid.block_1, "_enable_tiled_conv3d"):+ self.mid.block_1._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self.mid.attn_1, "_enable_tiled_conv3d"):+ self.mid.attn_1._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self.mid.block_2, "_enable_tiled_conv3d"):+ self.mid.block_2._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ for i_level in reversed(range(self.num_resolutions)):+ for i_block in range(self.num_res_blocks + 1):+ if hasattr(self.up[i_level].block[i_block], "_enable_tiled_conv3d"):+ self.up[i_level].block[i_block]._enable_tiled_conv3d(+ tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles+ )+ if i_level != 0:+ if hasattr(self.up[i_level].upsample, "_enable_tiled_conv3d"):+ self.up[i_level].upsample._enable_tiled_conv3d(+ tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles+ )++ self.conv_out = TiledConv3d.from_native_conv3d(+ self.conv_out,+ num_tiles=num_tiles,+ tile_size=tile_size,+ tiled_dim=tiled_dim,+ )++ def forward(self, z, is_training=False):+ self.last_z_shape = z.shape++ h = self.conv_in(z)+ h = self.mid.block_1(h, is_training=is_training)+ h = self.mid.attn_1(h)+ h = self.mid.block_2(h, is_training=is_training)++ # upsampling+ for i_level in reversed(range(self.num_resolutions)):+ for i_block in range(self.num_res_blocks + 1):+ h = self.up[i_level].block[i_block](h, is_training=is_training)+ if i_level != 0:+ h = self.up[i_level].upsample(h, is_split=True, is_training=is_training)++ h = self.norm_out(h)+ h = nonlinearity(h, is_training=is_training)+ h = F.pad(h, self.time_causal_padding, mode="constant")+ h = self.conv_out(h)+ h = h[:, :, (self.temporal_downsample - 1) :]++ return hdiff --git a/opensora/models/vae_v1_3/discriminator.py b/opensora/models/vae_v1_3/discriminator.pynew file mode 100644index 0000000..6c08fb1--- /dev/null+++ b/opensora/models/vae_v1_3/discriminator.py@@ -0,0 +1,104 @@+import functools+import os++import torch.nn as nn++from opensora.registry import MODELS+from opensora.utils.ckpt_utils import load_checkpoint+++def weights_init(m):+ classname = m.__class__.__name__+ if classname.find("Conv") != -1:+ nn.init.normal_(m.weight.data, 0.0, 0.02)+ elif classname.find("BatchNorm") != -1:+ nn.init.normal_(m.weight.data, 1.0, 0.02)+ nn.init.constant_(m.bias.data, 0)+++def weights_init_conv(m):+ if hasattr(m, "conv"):+ m = m.conv+ classname = m.__class__.__name__+ if classname.find("Conv") != -1:+ nn.init.normal_(m.weight.data, 0.0, 0.02)+ elif classname.find("BatchNorm") != -1:+ nn.init.normal_(m.weight.data, 1.0, 0.02)+ nn.init.constant_(m.bias.data, 0)+++class NLayerDiscriminator3D(nn.Module):+ """Defines a 3D PatchGAN discriminator as in Pix2Pix but for 3D inputs."""++ def __init__(self, input_nc=1, ndf=64, n_layers=3, use_actnorm=False):+ """+ Construct a 3D PatchGAN discriminator++ Parameters:+ input_nc (int) -- the number of channels in input volumes+ ndf (int) -- the number of filters in the last conv layer+ n_layers (int) -- the number of conv layers in the discriminator+ use_actnorm (bool) -- flag to use actnorm instead of batchnorm+ """+ super(NLayerDiscriminator3D, self).__init__()+ if not use_actnorm:+ norm_layer = nn.BatchNorm3d+ else:+ raise NotImplementedError("Not implemented.")+ if type(norm_layer) == functools.partial:+ use_bias = norm_layer.func != nn.BatchNorm3d+ else:+ use_bias = norm_layer != nn.BatchNorm3d++ kw = 3+ padw = 1+ sequence = [nn.Conv3d(input_nc, ndf, kernel_size=kw, stride=2, padding=padw), nn.LeakyReLU(0.2, True)]+ nf_mult = 1+ nf_mult_prev = 1+ for n in range(1, n_layers): # gradually increase the number of filters+ nf_mult_prev = nf_mult+ nf_mult = min(2**n, 8)+ sequence += [+ nn.Conv3d(+ ndf * nf_mult_prev,+ ndf * nf_mult,+ kernel_size=(kw, kw, kw),+ stride=(2 if n == 1 else 1, 2, 2),+ padding=padw,+ bias=use_bias,+ ),+ norm_layer(ndf * nf_mult),+ nn.LeakyReLU(0.2, True),+ ]++ nf_mult_prev = nf_mult+ nf_mult = min(2**n_layers, 8)+ sequence += [+ nn.Conv3d(+ ndf * nf_mult_prev, ndf * nf_mult, kernel_size=(kw, kw, kw), stride=1, padding=padw, bias=use_bias+ ),+ norm_layer(ndf * nf_mult),+ nn.LeakyReLU(0.2, True),+ ]++ sequence += [+ nn.Conv3d(ndf * nf_mult, 1, kernel_size=kw, stride=1, padding=padw)+ ] # output 1 channel prediction map+ self.main = nn.Sequential(*sequence)++ def forward(self, input):+ """Standard forward."""+ return self.main(input)++[email protected]_module()+def N_LAYER_DISCRIMINATOR_3D(from_pretrained=None, force_huggingface=None, **kwargs):+ model = NLayerDiscriminator3D(**kwargs).apply(weights_init)+ if from_pretrained is not None:+ if force_huggingface or from_pretrained is not None and not os.path.exists(from_pretrained):+ # TODO+ raise NotImplementedError+ else:+ load_checkpoint(model, from_pretrained)+ print(f"loaded model from: {from_pretrained}")+ return modeldiff --git a/opensora/models/vae_v1_3/encoder.py b/opensora/models/vae_v1_3/encoder.pynew file mode 100644index 0000000..d4126e7--- /dev/null+++ b/opensora/models/vae_v1_3/encoder.py@@ -0,0 +1,139 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/vae/encoder.py+++import torch.nn as nn++from opensora.registry import MODELS++from .modules.attention import AttnBlock+from .modules.conv import CausalConv3dPlainAR+from .modules.ops import Normalize, nonlinearity+from .modules.resblock import Resnet3DBlock+from .modules.updownsample import Downsample2D, Downsample3D+from .utils import is_odd++[email protected]_module()+class VideoEncoder(nn.Module):+ def __init__(+ self,+ *,+ ch,+ ch_mult=(1, 2, 4, 8),+ num_res_blocks,+ dropout=0.0,+ resamp_with_conv=True,+ in_channels,+ z_channels,+ double_z=True,+ down_sampling_layer=[1, 2],+ micro_batch_size_2d=None,+ **ignore_kwargs,+ ):+ super().__init__()+ self.ch = ch+ self.num_resolutions = len(ch_mult)+ self.num_res_blocks = num_res_blocks+ self.in_channels = in_channels+ self.down_sampling_layer = down_sampling_layer++ # downsampling+ self.conv_in = CausalConv3dPlainAR(in_channels, self.ch, kernel_size=3)++ in_ch_mult = (1,) + tuple(ch_mult)+ self.in_ch_mult = in_ch_mult+ self.down = nn.ModuleList()+ for i_level in range(self.num_resolutions):+ block = nn.ModuleList()+ block_in = ch * in_ch_mult[i_level]+ block_out = ch * ch_mult[i_level]+ for i_block in range(self.num_res_blocks):+ block.append(+ Resnet3DBlock(+ in_channels=block_in,+ out_channels=block_out,+ dropout=dropout,+ )+ )+ block_in = block_out+ down = nn.Module()+ down.block = block+ if i_level != self.num_resolutions - 1:+ if i_level in self.down_sampling_layer:+ down.downsample = Downsample3D(block_in, resamp_with_conv, stride=(2, 2, 2))+ else:+ down.downsample = Downsample2D(block_in, resamp_with_conv, micro_batch_size_2d=micro_batch_size_2d)+ self.down.append(down)++ # middle+ self.mid = nn.Module()+ self.mid.block_1 = Resnet3DBlock(+ in_channels=block_in,+ out_channels=block_in,+ dropout=dropout,+ )+ self.mid.attn_1 = AttnBlock(block_in)+ self.mid.block_2 = Resnet3DBlock(+ in_channels=block_in,+ out_channels=block_in,+ dropout=dropout,+ )+ # end+ self.norm_out = Normalize(block_in)+ self.conv_out = CausalConv3dPlainAR(block_in, 2 * z_channels if double_z else z_channels, kernel_size=3)++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ if hasattr(self.conv_in, "_enable_tiled_conv3d"):+ self.conv_in._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ for i_level in range(self.num_resolutions):+ for i_block in range(self.num_res_blocks):+ if hasattr(self.down[i_level].block[i_block], "_enable_tiled_conv3d"):+ self.down[i_level].block[i_block]._enable_tiled_conv3d(+ tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles+ )+ if i_level != self.num_resolutions - 1:+ if hasattr(self.down[i_level].downsample, "_enable_tiled_conv3d"):+ self.down[i_level].downsample._enable_tiled_conv3d(+ tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles+ )++ if hasattr(self.mid.block_1, "_enable_tiled_conv3d"):+ self.mid.block_1._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self.mid.attn_1, "_enable_tiled_conv3d"):+ self.mid.attn_1._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self.mid.block_2, "_enable_tiled_conv3d"):+ self.mid.block_2._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ if hasattr(self.conv_out, "_enable_tiled_conv3d"):+ self.conv_out._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ def forward(self, x, is_training=False):+ # timestep embedding+ t = x.size(2)+ if is_odd(t):+ temporal_length = max(t // (2 ** len(self.down_sampling_layer)) + 1, 1)+ else:+ temporal_length = t // (2 ** len(self.down_sampling_layer))++ # downsampling+ h = self.conv_in(x)+ for i_level in range(self.num_resolutions):+ for i_block in range(self.num_res_blocks):+ h = self.down[i_level].block[i_block](h, is_training=is_training)++ if i_level != self.num_resolutions - 1:+ h = self.down[i_level].downsample(h, is_training=is_training)++ h = self.mid.block_1(h, is_training=is_training)+ h = self.mid.attn_1(h)+ h = self.mid.block_2(h, is_training=is_training)++ h = self.norm_out(h)+ h = nonlinearity(h, is_training=is_training)+ h = self.conv_out(h)++ if h.shape[2] != temporal_length:+ print("shape strange: ", h.shape, x.shape)+ return hdiff --git a/opensora/models/vae_v1_3/losses.py b/opensora/models/vae_v1_3/losses.pynew file mode 100644index 0000000..13aa226--- /dev/null+++ b/opensora/models/vae_v1_3/losses.py@@ -0,0 +1,205 @@+import torch+import torch.nn as nn+import torch.nn.functional as F+from einops import rearrange++from opensora.models.vae.lpips import LPIPS+++def hinge_d_loss(logits_real, logits_fake):+ loss_real = torch.mean(F.relu(1.0 - logits_real))+ loss_fake = torch.mean(F.relu(1.0 + logits_fake))+ d_loss = 0.5 * (loss_real + loss_fake)+ return d_loss+++def vanilla_d_loss(logits_real, logits_fake):+ d_loss = 0.5 * (+ torch.mean(torch.nn.functional.softplus(-logits_real)) + torch.mean(torch.nn.functional.softplus(logits_fake))+ )+ return d_loss+++def adopt_weight(weight, global_step, threshold=0, value=0.0):+ if global_step < threshold:+ weight = value+ return weight+++def measure_perplexity(predicted_indices, n_embed):+ # src: https://github.com/karpathy/deep-vector-quantization/blob/main/model.py+ # eval cluster perplexity. when perplexity == num_embeddings then all clusters are used exactly equally+ encodings = F.one_hot(predicted_indices, n_embed).float().reshape(-1, n_embed)+ avg_probs = encodings.mean(0)+ perplexity = (-(avg_probs * torch.log(avg_probs + 1e-10)).sum()).exp()+ cluster_use = torch.sum(avg_probs > 0)+ return perplexity, cluster_use+++def l1(x, y):+ return torch.abs(x - y)+++def l2(x, y):+ return torch.pow((x - y), 2)+++def sigmoid_cross_entropy_with_logits(labels, logits):+ # The final formulation is: max(x, 0) - x * z + log(1 + exp(-abs(x)))+ zeros = torch.zeros_like(logits, dtype=logits.dtype)+ condition = logits >= zeros+ relu_logits = torch.where(condition, logits, zeros)+ neg_abs_logits = torch.where(condition, -logits, logits)+ return relu_logits - logits * labels + torch.log1p(torch.exp(neg_abs_logits))+++def lecam_reg(real_pred, fake_pred, ema_real_pred, ema_fake_pred):+ assert real_pred.ndim == 0 and ema_fake_pred.ndim == 0+ lecam_loss = torch.mean(torch.pow(nn.ReLU()(real_pred - ema_fake_pred), 2))+ lecam_loss += torch.mean(torch.pow(nn.ReLU()(ema_real_pred - fake_pred), 2))+ return lecam_loss+++def gradient_penalty_fn(images, output):+ gradients = torch.autograd.grad(+ outputs=output,+ inputs=images,+ grad_outputs=torch.ones(output.size(), device=images.device),+ create_graph=True,+ retain_graph=True,+ only_inputs=True,+ )[0]++ gradients = rearrange(gradients, "b ... -> b (...)")+ return ((gradients.norm(2, dim=1) - 1) ** 2).mean()+++class VAELoss(nn.Module):+ def __init__(+ self,+ logvar_init=0.0,+ perceptual_loss_weight=1.0,+ kl_loss_weight=1e-6,+ device="cpu",+ dtype="bf16",+ ):+ super().__init__()++ if type(dtype) == str:+ if dtype == "bf16":+ dtype = torch.bfloat16+ elif dtype == "fp16":+ dtype = torch.float16+ else:+ raise NotImplementedError(f"dtype: {dtype}")++ # KL Loss+ self.kl_weight = kl_loss_weight+ # Perceptual Loss+ self.perceptual_loss_fn = LPIPS().eval().to(device, dtype)+ self.perceptual_loss_weight = perceptual_loss_weight+ self.logvar = nn.Parameter(torch.ones(size=()) * logvar_init)++ def forward(+ self,+ video,+ recon_video,+ posterior,+ nll_weights=None,+ ):+ video = rearrange(video, "b c t h w -> (b t) c h w").contiguous()+ recon_video = rearrange(recon_video, "b c t h w -> (b t) c h w").contiguous()+ # reconstruction loss+ recon_loss = torch.abs(video - recon_video)++ # perceptual loss+ if self.perceptual_loss_weight is not None and self.perceptual_loss_weight > 0.0:+ perceptual_loss = self.perceptual_loss_fn(video, recon_video)+ recon_loss = recon_loss + self.perceptual_loss_weight * perceptual_loss+ nll_loss = recon_loss / torch.exp(self.logvar) + self.logvar+ weighted_nll_loss = nll_loss+ if nll_weights is not None:+ weighted_nll_loss = nll_weights * nll_loss+ weighted_nll_loss = torch.sum(weighted_nll_loss) / weighted_nll_loss.shape[0]+ nll_loss = torch.sum(nll_loss) / nll_loss.shape[0]++ weighted_kl_loss = 0+ if self.kl_weight is not None and self.kl_weight > 0.0:+ kl_loss = posterior.kl()+ kl_loss = torch.sum(kl_loss) / kl_loss.shape[0]+ weighted_kl_loss = kl_loss * self.kl_weight++ return nll_loss, weighted_nll_loss, weighted_kl_loss+++class GeneratorLoss(nn.Module):+ def __init__(self, disc_start=2001, disc_factor=1.0, disc_weight=0.5, **kwargs):+ super().__init__()+ self.disc_factor = disc_factor+ self.disc_start = disc_start+ self.disc_weight = disc_weight++ def calculate_adaptive_weight(self, nll_loss, g_loss, last_layer):+ nll_grads = torch.autograd.grad(nll_loss, last_layer, retain_graph=True)[0]+ g_grads = torch.autograd.grad(g_loss, last_layer, retain_graph=True)[0]+ d_weight = torch.norm(nll_grads) / (torch.norm(g_grads) + 1e-4)+ d_weight = torch.clamp(d_weight, 0.0, 1e4).detach()+ d_weight = d_weight * self.disc_weight+ return d_weight++ def forward(+ self,+ logits_fake,+ nll_loss,+ last_layer,+ global_step,+ is_training=True,+ ):+ g_loss = -torch.mean(logits_fake)++ if self.disc_factor is not None and self.disc_factor > 0.0:+ try:+ d_weight = self.calculate_adaptive_weight(nll_loss, g_loss, last_layer)+ except RuntimeError:+ assert not is_training+ d_weight = torch.tensor(0.0)+ else:+ d_weight = torch.tensor(0.0)++ disc_factor = adopt_weight(self.disc_factor, global_step, threshold=self.disc_start)+ weighted_gen_loss = d_weight * disc_factor * g_loss++ return weighted_gen_loss+++class DiscriminatorLoss(nn.Module):+ def __init__(self, disc_start=2001, disc_factor=1.0, disc_loss_type="hinge", **kwargs):+ super().__init__()++ assert disc_loss_type in ["hinge", "vanilla"]+ self.disc_factor = disc_factor+ self.disc_start = disc_start+ self.disc_loss_type = disc_loss_type++ def forward(+ self,+ real_logits,+ fake_logits,+ global_step,+ ):+ if self.disc_factor is not None and self.disc_factor > 0.0:+ disc_factor = adopt_weight(self.disc_factor, global_step, threshold=self.disc_start)++ if self.disc_loss_type == "hinge":+ disc_loss = hinge_d_loss(real_logits, fake_logits)+ elif self.disc_loss_type == "vanilla":+ disc_loss = vanilla_d_loss(real_logits, fake_logits)+ else:+ raise ValueError(f"Unknown GAN loss '{self.disc_loss_type}'.")++ weighted_discriminator_loss = disc_factor * disc_loss++ else:+ weighted_discriminator_loss = 0++ return weighted_discriminator_lossdiff --git a/opensora/models/vae_v1_3/modules/__init__.py b/opensora/models/vae_v1_3/modules/__init__.pynew file mode 100644index 0000000..4fb289c--- /dev/null+++ b/opensora/models/vae_v1_3/modules/__init__.py@@ -0,0 +1 @@+from .conv import CausalConv3dPlainARdiff --git a/opensora/models/vae_v1_3/modules/attention.py b/opensora/models/vae_v1_3/modules/attention.pynew file mode 100644index 0000000..856d8f2--- /dev/null+++ b/opensora/models/vae_v1_3/modules/attention.py@@ -0,0 +1,46 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/modules/attention.py+++import torch+import torch.nn as nn+from einops import rearrange++from .conv import CausalConv3dPlainAR+from .ops import Normalize+++class AttnBlock(nn.Module):+ def __init__(self, in_channels):+ super().__init__()+ self.in_channels = in_channels++ self.norm = Normalize(in_channels)+ self.q = CausalConv3dPlainAR(in_channels, in_channels, kernel_size=1, stride=1, padding=0)+ self.k = CausalConv3dPlainAR(in_channels, in_channels, kernel_size=1, stride=1, padding=0)+ self.v = CausalConv3dPlainAR(in_channels, in_channels, kernel_size=1, stride=1, padding=0)+ self.proj_out = CausalConv3dPlainAR(in_channels, in_channels, kernel_size=1, stride=1, padding=0)++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ self.q._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ self.k._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ self.v._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ self.proj_out._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ def attention(self, h_) -> torch.Tensor:+ h_ = self.norm(h_)+ q = self.q(h_)+ k = self.k(h_)+ v = self.v(h_)+ # print('h_', h_.shape)++ b, c, t, h, w = q.shape+ q, k, v = map(lambda x: rearrange(x, "b c t h w -> b 1 (t h w) c").contiguous(), (q, k, v))+ h_ = torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=True)+ return rearrange(h_, "b 1 (t h w) c -> b c t h w", t=t, h=h, w=w, c=c, b=b)++ def forward(self, x):+ h_ = x+ h_ = self.attention(h_)+ h_ = self.proj_out(h_)+ return x + h_diff --git a/opensora/models/vae_v1_3/modules/conv.py b/opensora/models/vae_v1_3/modules/conv.pynew file mode 100644index 0000000..50b3d09--- /dev/null+++ b/opensora/models/vae_v1_3/modules/conv.py@@ -0,0 +1,61 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/modules/conv.py++import logging+from typing import Tuple, Union++import torch.nn as nn+import torch.nn.functional as F+from torch.nn import Module++from opensora.models.vae_v1_3.utils import cast_tuple, is_odd+from opensora.models.layers.tiled_conv3d import TiledConv3d++logpy = logging.getLogger(__name__)+++class CausalConv3dPlainAR(Module):+ def __init__(+ self,+ chan_in,+ chan_out,+ kernel_size: Union[int, Tuple[int, int, int]],+ pad_mode="constant",+ **kwargs,+ ):+ super().__init__()+ if isinstance(kernel_size, int):+ kernel_size = cast_tuple(kernel_size, 3)+ time_kernel_size, height_kernel_size, width_kernel_size = kernel_size+ self.time_kernel_size = time_kernel_size+ assert is_odd(height_kernel_size) and is_odd(width_kernel_size)+ dilation = kwargs.pop("dilation", 1)+ stride = kwargs.pop("stride", 1)+ self.pad_mode = pad_mode+ if isinstance(stride, int):+ stride = (stride, 1, 1)+ time_pad = dilation * (time_kernel_size - 1) + max((1 - stride[0]), 0)+ height_pad = height_kernel_size // 2+ width_pad = width_kernel_size // 2+ self.time_pad = time_pad+ self.time_causal_padding = (width_pad, width_pad, height_pad, height_pad, time_pad, 0)++ self.time_uncausal_padding = (width_pad, width_pad, height_pad, height_pad, 0, 0)+ self.conv = nn.Conv3d(chan_in, chan_out, kernel_size, stride=stride, dilation=dilation, **kwargs)++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ # NOTE: currently not enable for stride != 1+ if self.conv.stride == (1, 1, 1):+ self.conv = TiledConv3d.from_native_conv3d(+ self.conv,+ num_tiles=num_tiles,+ tile_size=tile_size,+ tiled_dim=tiled_dim,+ )++ def forward(self, x):+ pad_mode = self.pad_mode if self.time_pad < x.shape[2] else "constant"+ x = F.pad(x, self.time_causal_padding, mode=pad_mode)++ x = self.conv(x)+ return xdiff --git a/opensora/models/vae_v1_3/modules/ops.py b/opensora/models/vae_v1_3/modules/ops.pynew file mode 100644index 0000000..fb1a3b7--- /dev/null+++ b/opensora/models/vae_v1_3/modules/ops.py@@ -0,0 +1,20 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/modules/ops.py++import torch+++def Normalize(in_channels, num_groups=32):+ return torch.nn.GroupNorm(num_groups=num_groups, num_channels=in_channels, eps=1e-6, affine=True)+++def nonlinearity(x, is_training=False):+ if is_training:+ return x * torch.sigmoid(x)+ else:+ x.mul_(torch.sigmoid(x))+ return x+++def cast_tuple(t, length=1):+ return t if isinstance(t, tuple) else ((t,) * length)diff --git a/opensora/models/vae_v1_3/modules/resblock.py b/opensora/models/vae_v1_3/modules/resblock.pynew file mode 100644index 0000000..9f13564--- /dev/null+++ b/opensora/models/vae_v1_3/modules/resblock.py@@ -0,0 +1,67 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/modules/resblock.py+++import logging++import torch+import torch.nn as nn++logpy = logging.getLogger(__name__)+from .conv import CausalConv3dPlainAR+from .ops import Normalize, nonlinearity+++class Resnet3DBlock(nn.Module):+ def __init__(self, *, in_channels, out_channels=None, conv_shortcut=False, dropout, empty_cache=False):+ super().__init__()+ self.in_channels = in_channels+ out_channels = in_channels if out_channels is None else out_channels+ self.out_channels = out_channels+ self.use_conv_shortcut = conv_shortcut+ self.norm1 = Normalize(in_channels)+ self.conv1 = CausalConv3dPlainAR(in_channels, out_channels, kernel_size=3)++ self.norm2 = Normalize(out_channels)+ self.dropout = torch.nn.Dropout(dropout)+ self.conv2 = CausalConv3dPlainAR(out_channels, out_channels, kernel_size=3)+ self.empty_cache = empty_cache+ if self.in_channels != self.out_channels:+ if self.use_conv_shortcut:+ self.conv_shortcut = CausalConv3dPlainAR(in_channels, out_channels, kernel_size=3)+ else:+ self.nin_shortcut = CausalConv3dPlainAR(in_channels, out_channels, kernel_size=1)++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ self.conv1._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ self.conv2._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self, "conv_shortcut"):+ self.conv_shortcut._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self, "nin_shortcut"):+ self.nin_shortcut._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ def forward(self, x, is_training=False):+ h = x.clone()+ h = self.norm1(h)++ if self.empty_cache and not is_training:+ torch.cuda.empty_cache()++ h = nonlinearity(h)+ h = self.conv1(h)++ h = self.norm2(h)+ h = nonlinearity(h)+ h = self.dropout(h)+ h = self.conv2(h)++ if self.in_channels != self.out_channels:+ if self.use_conv_shortcut:+ x = self.conv_shortcut(x)+ else:+ x = self.nin_shortcut(x)+ if is_training:+ x = x + h+ else:+ x.add_(h)+ return xdiff --git a/opensora/models/vae_v1_3/modules/updownsample.py b/opensora/models/vae_v1_3/modules/updownsample.pynew file mode 100644index 0000000..f965240--- /dev/null+++ b/opensora/models/vae_v1_3/modules/updownsample.py@@ -0,0 +1,208 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/modules/updownsample.py++import logging++import torch+import torch.nn as nn++from opensora.models.vae_v1_3.utils import video_to_image++logpy = logging.getLogger(__name__)+from .conv import CausalConv3dPlainAR+++class Downsample2D(nn.Module):+ def __init__(self, in_channels, with_conv, micro_batch_size_2d=None):+ super().__init__()+ self.with_conv = with_conv+ if self.with_conv:+ self.conv = torch.nn.Conv2d(in_channels, in_channels, kernel_size=3, stride=2, padding=0)+ self.micro_batch_size_2d = micro_batch_size_2d++ @video_to_image+ def forward(self, x, is_training=False):+ if self.with_conv:+ pad = (0, 1, 0, 1)+ x = torch.nn.functional.pad(x, pad, mode="constant", value=0)+ x = self.conv(x)+ else:+ x = torch.nn.functional.avg_pool2d(x, kernel_size=2, stride=2)+ return x+++class Downsample3D(nn.Module):+ def __init__(self, in_channels, with_conv, stride):+ super().__init__()+ self.with_conv = with_conv+ if self.with_conv:+ self.conv = CausalConv3dPlainAR(in_channels, in_channels, kernel_size=3, stride=stride)++ def forward(self, x, is_training=False):+ if self.with_conv:+ x = self.conv(x)+ else:+ x = torch.nn.functional.avg_pool3d(x, kernel_size=2, stride=2)+ return x+++class Res3DBlockUpsample(nn.Module):+ def __init__(+ self,+ input_filters,+ num_filters,+ base_filters,+ down_sampling_stride,+ down_sampling=False,+ down_sampling_temporal=None,+ is_real_3d=True,+ with_norm=True,+ ):+ super(Res3DBlockUpsample, self).__init__()+ self.num_filters = num_filters+ self.base_filters = base_filters+ self.input_filters = input_filters+ self.with_norm = with_norm+ if down_sampling:+ if is_real_3d and down_sampling_temporal:+ self.down_sampling_stride = down_sampling_stride+ else:+ self.down_sampling_stride = down_sampling_stride+ else:+ self.down_sampling_stride = [1, 1, 1]++ self.down_sampling = down_sampling++ self.act = nn.SiLU()+ self.conv1 = CausalConv3dPlainAR(num_filters, num_filters, kernel_size=[3, 3, 3], stride=[1, 1, 1])+ if self.with_norm:+ self.norm1 = nn.GroupNorm(32, num_filters)+ self.conv2 = CausalConv3dPlainAR(num_filters, num_filters, kernel_size=[3, 3, 3], stride=[1, 1, 1])+ if self.with_norm:+ self.norm2 = nn.GroupNorm(32, num_filters)+ if num_filters != input_filters or down_sampling:+ self.conv3 = CausalConv3dPlainAR(+ input_filters, num_filters, kernel_size=[1, 1, 1], stride=self.down_sampling_stride+ )+ if self.with_norm:+ self.norm3 = nn.GroupNorm(32, num_filters)++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ self.conv1._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ self.conv2._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ if hasattr(self, "conv3"):+ self.conv3._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ def forward(self, x, is_training=False):+ identity = x+ out = self.conv1(x)+ if self.with_norm:+ out = self.norm1(out)+ out = self.act(out)++ out = self.conv2(out)+ if self.with_norm:+ out = self.norm2(out)+ if self.down_sampling or self.num_filters != self.input_filters:+ identity = self.conv3(identity)+ if self.with_norm:+ identity = self.norm3(identity)+ if is_training:+ out += identity+ else:+ out.add_(identity)+ out = self.act(out)+ return out+++class Upsample3D(nn.Module):+ def __init__(self, in_channels, with_conv, scale_factor=2):+ super().__init__()+ self.with_conv = with_conv+ self.scale_factor = scale_factor++ self.conv3d = Res3DBlockUpsample(+ input_filters=in_channels,+ num_filters=in_channels,+ base_filters=in_channels,+ down_sampling_stride=(1, 1, 1),+ down_sampling=False,+ )++ def _split_by_channel(self, x, split_size):+ slices = torch.split(x, split_size, dim=1)+ return slices++ def _split_by_batch(self, x, split_size):+ slices = torch.split(x, split_size, dim=0)+ return slices++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ self.conv3d._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ def forward(self, x, is_split=False, is_training=False):+ b, c, t, h, w = x.shape+ if is_split and not is_training:+ split_size = c // 8+ x_slices = self._split_by_channel(x, split_size)++ x = [+ torch.nn.functional.interpolate(+ x,+ (+ x.shape[2] * self.scale_factor,+ x.shape[3] * self.scale_factor,+ x.shape[4] * self.scale_factor,+ ),+ mode="nearest",+ )+ for x in x_slices+ ]+ x = torch.cat(x, dim=1)+ identity = x+ if b > 2 and b % 2 == 0:+ split_size = b // 2+ x_slices = self._split_by_batch(x, split_size)+ x = [self.conv3d(b_x) for b_x in x_slices]+ x = torch.cat(x, dim=0)+ else:+ x = self.conv3d(x)++ if is_training:+ x += identity+ else:+ x.add_(identity)+ return x+ else:+ x = torch.nn.functional.interpolate(+ x,+ (+ x.shape[2] * self.scale_factor,+ x.shape[3] * self.scale_factor,+ x.shape[4] * self.scale_factor,+ ),+ mode="nearest",+ )+ identity = x+ x = self.conv3d(x)+ if is_training:+ x += identity+ else:+ x.add_(identity)+ return x+++class Upsample2D(nn.Module):+ def __init__(self, in_channels, with_conv, micro_batch_size_2d=None):+ super().__init__()+ self.with_conv = with_conv+ if self.with_conv:+ self.conv = torch.nn.Conv2d(in_channels, in_channels, kernel_size=3, stride=1, padding=1)+ self.micro_batch_size_2d = micro_batch_size_2d++ @video_to_image+ def forward(self, x, is_split=False, is_training=False):+ x = torch.nn.functional.interpolate(x, scale_factor=2.0, mode="nearest")+ if self.with_conv:+ x = self.conv(x)+ return xdiff --git a/opensora/models/vae_v1_3/utils.py b/opensora/models/vae_v1_3/utils.pynew file mode 100644index 0000000..602a106--- /dev/null+++ b/opensora/models/vae_v1_3/utils.py@@ -0,0 +1,61 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/utils/util.py+++import torch+from einops import rearrange+++def to_torch_dtype(dtype):+ if isinstance(dtype, torch.dtype):+ return dtype+ elif isinstance(dtype, str):+ dtype_mapping = {+ "float64": torch.float64,+ "float32": torch.float32,+ "float16": torch.float16,+ "fp32": torch.float32,+ "fp16": torch.float16,+ "half": torch.float16,+ "bf16": torch.bfloat16,+ }+ if dtype not in dtype_mapping:+ raise ValueError+ dtype = dtype_mapping[dtype]+ return dtype+ else:+ raise ValueError+++def divisible_by(num, den):+ return (num % den) == 0+++def is_odd(n):+ return not divisible_by(n, 2)+++def cast_tuple(t, length=1):+ return t if isinstance(t, tuple) else ((t,) * length)+++def video_to_image(func):+ def wrapper(self, x, *args, **kwargs):+ if x.dim() == 5:+ t = x.shape[2]+ x = rearrange(x, "b c t h w -> (b t) c h w")+ micro_batch_size_2d = self.micro_batch_size_2d if hasattr(self, "micro_batch_size_2d") else None+ if micro_batch_size_2d is None:+ x = func(self, x, *args, **kwargs)+ else:+ bs = micro_batch_size_2d+ x_out = []+ for i in range(0, x.shape[0], bs):+ x_bs = x[i : i + bs]+ x_bs = func(self, x_bs, *args, **kwargs)+ x_out.append(x_bs)+ x = torch.cat(x_out, dim=0)+ x = rearrange(x, "(b t) c h w -> b c t h w", t=t)+ return x++ return wrapperdiff --git a/opensora/models/vae_v1_3/vae.py b/opensora/models/vae_v1_3/vae.pynew file mode 100644index 0000000..342367a--- /dev/null+++ b/opensora/models/vae_v1_3/vae.py@@ -0,0 +1,320 @@+# modified from+# https://github.com/bornfly-detachment/asymmetric_magvitv2/blob/main/models/vae/vae.py+++import os+from typing import Tuple, Union++import torch+from transformers import PretrainedConfig, PreTrainedModel++from opensora.models.vae.utils import DiagonalGaussianDistribution+from opensora.registry import MODELS, build_module+from opensora.utils.ckpt_utils import load_checkpoint++from .constants import get_vae_stats+from .utils import to_torch_dtype+++class OpenSoraVAE_V1_3_PiplineConfig(PretrainedConfig):+ model_type = "OpenSoraVAE_V1_3_Pipline"++ def __init__(+ self,+ **kwargs,+ ):+ super().__init__(**kwargs)+++class OpenSoraVAE_V1_3_Pipline(PreTrainedModel):+ config_class = OpenSoraVAE_V1_3_PiplineConfig++ def __init__(+ self,+ config: OpenSoraVAE_V1_3_PiplineConfig,+ micro_batch_size=None,+ micro_batch_size_2d=None,+ micro_frame_size=None,+ use_tiled_conv3d=False,+ tile_size=16,+ tiled_dim=None,+ num_tiles=None,+ temporal_overlap=False,+ normalization=None,+ ):+ super().__init__(config=config)+ print("OpenSoraVAE_V1_3_Pipline config", config)+ self.micro_batch_size = micro_batch_size+ self.micro_batch_size_2d = micro_batch_size_2d+ self.micro_frame_size = micro_frame_size+ self.encoder = build_module(config.encoder, MODELS)+ self.decoder = build_module(config.decoder, MODELS)++ self.out_channels = config.encoder["z_channels"] # for dit training+ if use_tiled_conv3d:+ self._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ self.temporal_overlap = temporal_overlap+ self.normalization = normalization++ def _enable_tiled_conv3d(self, tile_size=16, tiled_dim=None, num_tiles=None):+ self.encoder._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ self.decoder._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)++ # for dit training+ def get_latent_size(self, input_size, patch_size=[4, 8, 8]):+ latent_size = []+ # temporal+ if input_size[0] is None:+ latent_size.append(None)+ elif input_size[0] == 1:+ latent_size.append(1)+ elif self.micro_frame_size is None:+ latent_size.append((input_size[0] - 1) // patch_size[0] + 1)+ elif not self.temporal_overlap:+ micro_z_frame_size = int((self.micro_frame_size - 1) / 4 + 1)+ latent_size.append(input_size[0] // self.micro_frame_size * micro_z_frame_size)+ else:+ micro_z_frame_size = int((self.micro_frame_size - 1) / 4 + 1)+ latent_size.append((input_size[0] - 1) // (self.micro_frame_size - 1) * micro_z_frame_size)+ # spatial+ for i in range(1, 3):+ latent_size.append((input_size[i] - 1) // patch_size[i] + 1 if input_size[i] is not None else None)+ return latent_size++ def get_last_layer(self):+ return self.decoder.get_last_layer()++ @property+ def micro_z_frame_size(self):+ return int((self.micro_frame_size - 1) / 4 + 1) if self.micro_frame_size else None++ def encode_micro_frame(self, x):+ z_list = []+ # x.shape[2] = n * self.micro_frame_size+ if not self.temporal_overlap:+ for i in range(0, x.shape[2], self.micro_frame_size):+ x_bs = x[:, :, i : i + self.micro_frame_size]+ z_list.append(self.encoder(x_bs))+ else:+ for i in range(0, x.shape[2] - 1, self.micro_frame_size - 1):+ x_bs = x[:, :, i : i + self.micro_frame_size]+ z_list.append(self.encoder(x_bs))+ z = torch.cat(z_list, dim=2)+ return z++ def encode_micro_batch(self, x):+ z_list = []+ # x.shape[0] = n * self.micro_batch_size+ for i in range(0, x.shape[0], self.micro_batch_size):+ x_bs = x[i : i + self.micro_batch_size]+ if self.micro_frame_size is None or x_bs.size(2) == 1:+ z_list.append(self.encoder(x_bs))+ else:+ z_list.append(self.encode_micro_frame(x_bs))+ z = torch.cat(z_list, dim=0)+ return z++ def normalize_z(self, z):+ if self.normalization is None:+ return z+ elif self.normalization == "video":+ mean, std = get_vae_stats("video", z)+ mean = mean[None, :, None, None, None]+ std = std[None, :, None, None, None]+ z = ((z - mean) / std).to(z.dtype)+ elif self.normalization == "video_5frames":+ mean, std = get_vae_stats("video_5frames", z)+ if z.size(2) == 1:+ mean = mean[None, :, :1, None, None]+ std = std[None, :, :1, None, None]+ else:+ num_replica = z.size(2) // mean.size(1)+ mean = mean.repeat(1, num_replica)[None, :, :, None, None]+ std = std.repeat(1, num_replica)[None, :, :, None, None]+ z = ((z - mean) / std).to(z.dtype)+ else:+ raise NotImplementedError+ return z++ def encode(+ self,+ x: torch.Tensor,+ is_training=False,+ sample_posterior=True,+ return_posterior=False,+ **kwargs,+ ) -> Union[torch.Tensor, Tuple[torch.Tensor, dict]]:+ if is_training: # only use micro_batch_size during inference+ self.micro_batch_size = None+ self.micro_batch_size_2d = None++ if self.micro_batch_size is None:+ z = self.encoder(x, is_training=is_training)+ else:+ z = self.encode_micro_batch(x)++ posterior = DiagonalGaussianDistribution(z)+ if sample_posterior:+ z = posterior.sample()+ else:+ z = posterior.mode()++ # scale for dit training+ z = self.normalize_z(z)++ if return_posterior:+ return z, posterior+ else:+ return z++ def decode_micro_frame(self, z):+ x_list = []+ # z.shape[2] = n * micro_z_frame_size+ micro_z_frame_size = int((self.micro_frame_size - 1) / 4 + 1) if self.micro_frame_size else None+ for i in range(0, z.size(2), micro_z_frame_size):+ z_bs = z[:, :, i : i + micro_z_frame_size]+ x_bs = self.decoder(z_bs)+ if self.temporal_overlap and i > 0:+ x_list[-1][:, :, -1:, :, :] = 0.5 * (x_list[-1][:, :, -1:, :, :] + x_bs[:, :, :1, :, :])+ x_bs = x_bs[:, :, 1:, :, :]+ x_list.append(x_bs)+ x = torch.cat(x_list, dim=2)+ return x++ def decode_micro_batch(self, z):+ x_list = []+ # x.shape[0] = n * self.micro_batch_size+ for i in range(0, z.shape[0], self.micro_batch_size):+ z_bs = z[i : i + self.micro_batch_size]+ if self.micro_frame_size is None:+ x_list.append(self.decoder(z_bs))+ else:+ x_list.append(self.decode_micro_frame(z_bs))+ x = torch.cat(x_list, dim=0)+ return x++ def un_normalize_z(self, z):+ if self.normalization is None:+ return z+ elif self.normalization == "video":+ mean, std = get_vae_stats("video", z)+ mean = mean[None, :, None, None, None]+ std = std[None, :, None, None, None]+ z = (z * std + mean).to(z.dtype)+ elif self.normalization == "video_5frames":+ mean, std = get_vae_stats("video_5frames", z)+ if z.size(2) == 1:+ mean = mean[None, :, :1, None, None]+ std = std[None, :, :1, None, None]+ else:+ num_replica = z.size(2) // mean.size(1)+ mean = mean.repeat(1, num_replica)[None, :, :, None, None]+ std = std.repeat(1, num_replica)[None, :, :, None, None]+ z = (z * std + mean).to(z.dtype)+ else:+ raise NotImplementedError+ return z++ def decode(self, z: torch.Tensor, is_training=False, **kwargs) -> torch.Tensor:+ # revert back for vae inference+ z = self.un_normalize_z(z)++ if is_training:+ self.micro_batch_size = None+ self.micro_batch_size_2d = None++ if self.micro_batch_size is None:+ x = self.decoder(z, is_training=is_training)+ else:+ x = self.decode_micro_batch(z)+ return x++ def forward(self, x: torch.Tensor, is_training=False) -> Tuple[torch.Tensor, torch.Tensor, dict]:+ z, posterior = self.encode(x, is_training=is_training, return_posterior=True)+ dec = self.decode(z.to(x.dtype), is_training=is_training)+ return z, dec, posterior++[email protected]_module()+def OpenSoraVAE_V1_3(+ from_pretrained=None,+ force_huggingface=False,+ dtype="bf16",+ z_channels=16,+ micro_batch_size=None,+ micro_batch_size_2d=None,+ micro_frame_size=None,+ use_tiled_conv3d=False,+ tile_size=16,+ tiled_dim=None,+ num_tiles=None,+ temporal_overlap=False,+ normalization=None,+):+ encoder_config = dict(+ type="VideoEncoder",+ double_z=True,+ z_channels=z_channels,+ in_channels=3,+ out_ch=3,+ ch=128,+ ch_mult=[1, 2, 4, 4],+ num_res_blocks=2,+ down_sampling_layer=[1, 2],+ micro_batch_size_2d=micro_batch_size_2d,+ )++ decoder_config = dict(+ type="VideoDecoder",+ z_channels=z_channels,+ in_channels=3,+ out_ch=3,+ ch=128,+ ch_mult=[1, 2, 4, 4],+ num_res_blocks=2,+ temporal_up_layers=[2, 3],+ micro_batch_size_2d=micro_batch_size_2d,+ )++ dtype = to_torch_dtype(dtype)++ kwargs = dict(+ encoder=encoder_config,+ decoder=decoder_config,+ )++ if force_huggingface or (from_pretrained is not None and not os.path.exists(from_pretrained)):+ model = OpenSoraVAE_V1_3_Pipline.from_pretrained(from_pretrained, **kwargs)+ model.micro_batch_size = micro_batch_size+ model.micro_batch_size_2d = micro_batch_size_2d+ model.micro_frame_size = micro_frame_size+ model.encoder.micro_batch_size = micro_batch_size+ model.decoder.micro_batch_size = micro_batch_size+ model.normalization = normalization+ model.temporal_overlap = temporal_overlap++ if use_tiled_conv3d:+ model._enable_tiled_conv3d(tile_size=tile_size, tiled_dim=tiled_dim, num_tiles=num_tiles)+ print(f"huggingface model loaded: {from_pretrained}")+ else:+ config = OpenSoraVAE_V1_3_PiplineConfig(**kwargs)+ model = OpenSoraVAE_V1_3_Pipline(+ config,+ use_tiled_conv3d=use_tiled_conv3d,+ micro_batch_size=micro_batch_size,+ micro_batch_size_2d=micro_batch_size_2d,+ micro_frame_size=micro_frame_size,+ tile_size=16,+ tiled_dim=None,+ num_tiles=None,+ temporal_overlap=temporal_overlap,+ normalization=normalization,+ )+ if from_pretrained:+ load_checkpoint(model, from_pretrained)+ print(f"local model loaded: {from_pretrained}")+ model.encoder.to(dtype)+ model.decoder.to(dtype)++ return modeldiff --git a/opensora/schedulers/dpms/__init__.py b/opensora/schedulers/dpms/__init__.pyindex 111e97b..3ba9f64 100644--- a/opensora/schedulers/dpms/__init__.py+++ b/opensora/schedulers/dpms/__init__.py@@ -23,11 +23,12 @@ class DPM_SOLVER:additional_args=None,mask=None,progress=True,+ **kwargs,):if mask is not None:print("[WARNING] mask is not supported in dpm-solver, it will be ignored")n = len(prompts)- model_args = text_encoder.encode(prompts)+ model_args = text_encoder.encode(**text_encoder.tokenize_fn(prompts))y = model_args.pop("y")null_y = text_encoder.null(n)if additional_args is not None:diff --git a/opensora/schedulers/iddpm/__init__.py b/opensora/schedulers/iddpm/__init__.pyindex 7b65cad..444ea9c 100644--- a/opensora/schedulers/iddpm/__init__.py+++ b/opensora/schedulers/iddpm/__init__.py@@ -62,10 +62,11 @@ class IDDPM(SpacedDiffusion):additional_args=None,mask=None,progress=True,+ **kwargs,):n = len(prompts)z = torch.cat([z, z], 0)- model_args = text_encoder.encode(prompts)+ model_args = text_encoder.encode(**text_encoder.tokenize_fn(prompts))y_null = text_encoder.null(n)model_args["y"] = torch.cat([model_args["y"], y_null], 0)if additional_args is not None:diff --git a/opensora/schedulers/rf/__init__.py b/opensora/schedulers/rf/__init__.pyindex e5931ec..a0326b9 100644--- a/opensora/schedulers/rf/__init__.py+++ b/opensora/schedulers/rf/__init__.py@@ -3,7 +3,33 @@ from tqdm import tqdmfrom opensora.registry import SCHEDULERS-from .rectified_flow import RFlowScheduler, timestep_transform+from .rectified_flow import RFlowScheduler+from .time_sampler import timestep_transform+++def dynamic_thresholding(x, ratio=0.995, base=6.0):+ s = torch.quantile(x.abs().flatten(), ratio)+ s = max(s, base)+ x = x.clip(-s, s) * base / s+ return x+++def get_oscillation_gs(guidance_scale, i, force_num=10):+ if i < force_num or (i >= force_num and i % 2 == 0):+ gs = guidance_scale+ else:+ gs = 1.0+ return gs+++def fix_guidance_flaw(v_pred, pred_cond):+ # common diffision noise schedules+ std_cond = pred_cond.std(dim=list(range(1, pred_cond.dim())), keepdim=True)+ std_cfg = v_pred.std(dim=list(range(1, v_pred.dim())), keepdim=True)+ factor = std_cond / std_cfg+ factor = 0.7 * factor + (1 - 0.7)+ v_pred = v_pred * factor+ return v_pred@SCHEDULERS.register_module("rflow")@@ -13,8 +39,17 @@ class RFLOW:num_sampling_steps=10,num_timesteps=1000,cfg_scale=4.0,+ # time sampleruse_discrete_timesteps=False,use_timestep_transform=False,+ transform_scale=1.0,+ scale_temporal=True,+ # guidance+ use_oscillation_guidance=False,+ use_flaw_fix=False,+ scale_image_weight=False,+ initial_image_scale=1.0,+ force_num=10,**kwargs,):self.num_sampling_steps = num_sampling_steps@@ -22,15 +57,53 @@ class RFLOW:self.cfg_scale = cfg_scaleself.use_discrete_timesteps = use_discrete_timestepsself.use_timestep_transform = use_timestep_transform+ self.transform_scale = transform_scale+ self.scale_temporal = scale_temporal+ self.use_oscillation_guidance = use_oscillation_guidance+ self.use_flaw_fix = use_flaw_fix++ self.scale_image_weight = scale_image_weight+ self.initial_image_scale = initial_image_scale+ self.force_num = force_numself.scheduler = RFlowScheduler(num_timesteps=num_timesteps,num_sampling_steps=num_sampling_steps,use_discrete_timesteps=use_discrete_timesteps,use_timestep_transform=use_timestep_transform,+ transform_scale=1.0,+ scale_temporal=scale_temporal,**kwargs,)+ def scale_weight(+ self,+ scale,+ init_scale,+ step_idx,+ total_steps,+ scale_method="decr",+ ):+ if scale_method == "incr":+ res_scale = torch.linspace(init_scale, scale, total_steps)[step_idx]+ elif scale_method == "decr":+ res_scale = torch.linspace(scale, init_scale, total_steps)[step_idx]+ else:+ raise NotImplementedError++ return res_scale++ def scale_temporal_weight(+ self,+ z,+ upper_scale,+ lower_scale,+ ):+ b, c, t, h, w = z.size()+ res_scale = torch.linspace(lower_scale, upper_scale, t)[None, None, :, None, None].repeat(b, c, 1, h, w)+ res_scale = res_scale.to(z.device, z.dtype)+ return res_scale+def sample(self,model,@@ -40,18 +113,51 @@ class RFLOW:device,additional_args=None,mask=None,- guidance_scale=None,+ guidance_scale=None, # text+ image_cfg_scale=None, # imageprogress=True,+ neg_prompts=None,+ z_cond=None,+ z_cond_mask=None,+ mask_index=None,+ use_sdedit=False,+ use_oscillation_guidance_for_text=None,+ use_oscillation_guidance_for_image=None,):+ if use_oscillation_guidance_for_text is None:+ use_oscillation_guidance_for_text = self.use_oscillation_guidance+ if use_oscillation_guidance_for_image is None:+ use_oscillation_guidance_for_image = self.use_oscillation_guidance++ if z_cond is not None and z_cond_mask is not None:+ z_cond = z_cond * z_cond_mask+# if no specific guidance scale is provided, use the default scale when initializing the schedulerif guidance_scale is None:guidance_scale = self.cfg_scale+ if image_cfg_scale is None:+ image_cfg_scale = self.cfg_scalen = len(prompts)# text encoding- model_args = text_encoder.encode(prompts)- y_null = text_encoder.null(n)- model_args["y"] = torch.cat([model_args["y"], y_null], 0)+ model_args = text_encoder.encode(**text_encoder.tokenize_fn(prompts))+ y_null = text_encoder.null(n) # [n, 1, 300, 4096] where n is batch size+ if neg_prompts is None:+ if mask_index is not None and len(mask_index) > 0:+ model_args["y"] = torch.cat([model_args["y"], y_null, y_null], 0)+ else:+ model_args["y"] = torch.cat([model_args["y"], y_null], 0)+ else:+ y_null_model_args = text_encoder.encode(**text_encoder.tokenize_fn(neg_prompts))+ if mask_index is not None and len(mask_index) > 0:+ model_args["y"] = torch.cat([model_args["y"], y_null_model_args["y"], y_null_model_args["y"]], 0)+ model_args["mask"] = torch.cat(+ [model_args["mask"], y_null_model_args["mask"], y_null_model_args["mask"]], 0+ )+ else:+ model_args["y"] = torch.cat([model_args["y"], y_null_model_args["y"]], 0)+ model_args["mask"] = torch.cat([model_args["mask"], y_null_model_args["mask"]], 0)+if additional_args is not None:model_args.update(additional_args)@@ -61,16 +167,26 @@ class RFLOW:timesteps = [int(round(t)) for t in timesteps]timesteps = [torch.tensor([t] * z.shape[0], device=device) for t in timesteps]if self.use_timestep_transform:- timesteps = [timestep_transform(t, additional_args, num_timesteps=self.num_timesteps) for t in timesteps]+ timesteps = [+ timestep_transform(+ t,+ additional_args,+ scale=self.transform_scale,+ num_timesteps=self.num_timesteps,+ scale_temporal=self.scale_temporal,+ )+ for t in timesteps+ ]if mask is not None:noise_added = torch.zeros_like(mask, dtype=torch.bool)noise_added = noise_added | (mask == 1)progress_wrap = tqdm if progress else (lambda x: x)+for i, t in progress_wrap(enumerate(timesteps)):# mask for adding noise- if mask is not None:+ if mask is not None: # not for i2v and v2v, need to force mask=Nonemask_t = mask * self.num_timestepsx0 = z.clone()x_noise = self.scheduler.add_noise(x0, torch.randn_like(x0), t)@@ -82,6 +198,132 @@ class RFLOW:z = torch.where(mask_add_noise[:, None, :, None, None], x_noise, x0)noise_added = mask_t_upper+ ## SDEdit+ if use_sdedit == True:+ if mask_index is not None and len(mask_index) > 0: # use condition instead of noise for i2v and v2v+ # NOTE: sdedit should add frames regardless of cfg to provide the same initial starting point+ z_noise = self.scheduler.add_noise(z_cond, torch.randn_like(z_cond), t)+ z = torch.where(z_cond_mask == 1, z_noise, z)+ text_gs = guidance_scale+ if use_oscillation_guidance_for_text:+ text_gs = get_oscillation_gs(guidance_scale, i, force_num=self.force_num)++ # classifier-free guidance+ if mask_index is not None and len(mask_index) > 0:+ # image_gs+ image_gs = image_cfg_scale+ if use_oscillation_guidance_for_image:+ image_gs = get_oscillation_gs(image_cfg_scale, i, force_num=self.force_num)+ if self.scale_image_weight and image_gs > self.initial_image_scale:+ upper_weight = self.scale_weight(+ image_gs,+ self.initial_image_scale,+ i,+ len(timesteps),+ scale_method="decr",+ )+ image_gs = self.scale_temporal_weight(z, upper_weight, self.initial_image_scale)++ # if type(image_gs) is not float: # dev debug message+ # print(f"step {i}, image_gs:{image_gs[0,0,:,0,0]}")++ z_in = torch.cat([z, z, z], 0)+ t = torch.cat([t, t, t], 0)++ # cfg, text+image, image only, nothing+ z_cond_in = torch.cat([z_cond, z_cond, torch.zeros_like(z_cond).to(z_cond.device).to(z_cond.dtype)], 0)+ z_cond_mask_in = torch.cat([z_cond_mask, z_cond_mask, z_cond_mask], 0)++ pred = model(+ z_in,+ t,+ cond=z_cond_in,+ cond_mask=z_cond_mask_in,+ mask_index=mask_index,+ y_null=y_null.repeat(+ z_in.shape[0], 1, 1, 1+ ), # NOTE: this shall not contain neg prompt info, strictly null+ **model_args,+ ).chunk(2, dim=1)[0]+ pred_cond, pred_uncond_text, pred_uncond_all = pred.chunk(3, dim=0)+ v_pred = (+ pred_uncond_all+ + image_gs * (pred_uncond_text - pred_uncond_all)+ + text_gs * (pred_cond - pred_uncond_text)+ )+ else:+ z_in = torch.cat([z, z], 0)+ t = torch.cat([t, t], 0)+ pred = model(z_in, t, **model_args).chunk(2, dim=1)[0]+ pred_cond, pred_uncond = pred.chunk(2, dim=0)+ v_pred = pred_uncond + text_gs * (pred_cond - pred_uncond)+ if self.use_flaw_fix:+ v_pred = fix_guidance_flaw(v_pred, pred_cond)++ # update z+ dt = timesteps[i] - timesteps[i + 1] if i < len(timesteps) - 1 else timesteps[i]+ dt = dt / self.num_timesteps+ z = z + v_pred * dt[:, None, None, None, None]++ if mask is not None:+ z = torch.where(mask_t_upper[:, None, :, None, None], z, x0)++ return z++ def sample_debug(+ self,+ model,+ text_encoder,+ z,+ prompts,+ device,+ additional_args=None,+ mask=None,+ guidance_scale=None,+ progress=True,+ neg_prompts=None,+ ):+ # if no specific guidance scale is provided, use the default scale when initializing the scheduler+ if guidance_scale is None:+ guidance_scale = self.cfg_scale++ n = len(prompts)+ # text encoding+ model_args = text_encoder.encode(**text_encoder.tokenize_fn(prompts))++ # use "" as negative prompts if not provided+ if neg_prompts is None:+ y_null = text_encoder.null(n)+ model_args["y"] = torch.cat([model_args["y"], y_null], 0)+ else:+ y_null_model_args = text_encoder.encode(neg_prompts)+ model_args["y"] = torch.cat([model_args["y"], y_null_model_args["y"]], 0)+ model_args["mask"] = torch.cat([model_args["mask"], y_null_model_args["mask"]], 0)++ if additional_args is not None:+ model_args.update(additional_args)++ # prepare timesteps+ timesteps = [(1.0 - i / self.num_sampling_steps) * self.num_timesteps for i in range(self.num_sampling_steps)]+ if self.use_discrete_timesteps:+ timesteps = [int(round(t)) for t in timesteps]+ timesteps = [torch.tensor([t] * z.shape[0], device=device) for t in timesteps]+ if self.use_timestep_transform:+ timesteps = [+ timestep_transform(+ t,+ additional_args,+ scale=self.transform_scale,+ num_timesteps=self.num_timesteps,+ scale_temporal=self.scale_temporal,+ )+ for t in timesteps+ ]++ infos = []+ progress_wrap = tqdm if progress else (lambda x: x)+ for i, t in progress_wrap(enumerate(timesteps)):+ info = dict(i=i, t=t.cpu().item())# classifier-free guidancez_in = torch.cat([z, z], 0)t = torch.cat([t, t], 0)@@ -94,10 +336,33 @@ class RFLOW:dt = dt / self.num_timestepsz = z + v_pred * dt[:, None, None, None, None]- if mask is not None:- z = torch.where(mask_t_upper[:, None, :, None, None], z, x0)+ info["z"] = z.clone()+ infos.append(info)- return z+ return infos- def training_losses(self, model, x_start, model_kwargs=None, noise=None, mask=None, weights=None, t=None):- return self.scheduler.training_losses(model, x_start, model_kwargs, noise, mask, weights, t)+ def training_losses(+ self,+ model,+ x_start,+ model_args=None,+ noise=None,+ mask=None,+ weights=None,+ t=None,+ mask_index=None,+ text_uncond_prob=None,+ **kwargs,+ ):+ return self.scheduler.training_losses(+ model,+ x_start,+ model_args,+ noise,+ mask,+ weights,+ t,+ mask_index=mask_index,+ text_uncond_prob=text_uncond_prob,+ **kwargs,+ )diff --git a/opensora/schedulers/rf/rectified_flow.py b/opensora/schedulers/rf/rectified_flow.pyindex f35bebb..0004ea4 100644--- a/opensora/schedulers/rf/rectified_flow.py+++ b/opensora/schedulers/rf/rectified_flow.py@@ -1,96 +1,101 @@+import random+import torch-from torch.distributions import LogisticNormalfrom ..iddpm.gaussian_diffusion import _extract_into_tensor, mean_flat+from .time_sampler import TimeSampler# some code are inspired by https://github.com/magic-research/piecewise-rectified-flow/blob/main/scripts/train_perflow.py# and https://github.com/magic-research/piecewise-rectified-flow/blob/main/src/scheduler_perflow.py-def timestep_transform(- t,- model_kwargs,- base_resolution=512 * 512,- base_num_frames=1,- scale=1.0,- num_timesteps=1,-):- # Force fp16 input to fp32 to avoid nan output- for key in ["height", "width", "num_frames"]:- if model_kwargs[key].dtype == torch.float16:- model_kwargs[key] = model_kwargs[key].float()-- t = t / num_timesteps- resolution = model_kwargs["height"] * model_kwargs["width"]- ratio_space = (resolution / base_resolution).sqrt()- # NOTE: currently, we do not take fps into account- # NOTE: temporal_reduction is hardcoded, this should be equal to the temporal reduction factor of the vae- if model_kwargs["num_frames"][0] == 1:- num_frames = torch.ones_like(model_kwargs["num_frames"])- else:- num_frames = model_kwargs["num_frames"] // 17 * 5- ratio_time = (num_frames / base_num_frames).sqrt()-- ratio = ratio_space * ratio_time * scale- new_t = ratio * t / (1 + (ratio - 1) * t)-- new_t = new_t * num_timesteps- return new_t--class RFlowScheduler:def __init__(self,num_timesteps=1000,num_sampling_steps=10,- use_discrete_timesteps=False,+ # time samplersample_method="uniform",- loc=0.0,- scale=1.0,+ use_discrete_timesteps=False,use_timestep_transform=False,transform_scale=1.0,+ scale_temporal=True,+ uniform_over_threshold=None,+ drop_condition=None,+ x_cond_weight=1,):self.num_timesteps = num_timestepsself.num_sampling_steps = num_sampling_steps- self.use_discrete_timesteps = use_discrete_timesteps-- # sample method- assert sample_method in ["uniform", "logit-normal"]- assert (- sample_method == "uniform" or not use_discrete_timesteps- ), "Only uniform sampling is supported for discrete timesteps"- self.sample_method = sample_method- if sample_method == "logit-normal":- self.distribution = LogisticNormal(torch.tensor([loc]), torch.tensor([scale]))- self.sample_t = lambda x: self.distribution.sample((x.shape[0],))[:, 0].to(x.device)-- # timestep transform- self.use_timestep_transform = use_timestep_transform- self.transform_scale = transform_scale-- def training_losses(self, model, x_start, model_kwargs=None, noise=None, mask=None, weights=None, t=None):+ self.time_sampler = TimeSampler(+ sample_method=sample_method,+ use_discrete_timesteps=use_discrete_timesteps,+ use_timestep_transform=use_timestep_transform,+ transform_scale=transform_scale,+ scale_temporal=scale_temporal,+ uniform_over_threshold=uniform_over_threshold,+ )+ self.drop_condition = drop_condition+ self.x_cond_weight = x_cond_weight # the weight to use for i2v and v2v condition++ def training_losses(+ self,+ model,+ x_start,+ model_kwargs=None,+ noise=None,+ mask=None,+ weights=None,+ t=None,+ x_gt=None,+ mask_index=None,+ noise_disable_threshold=None,+ text_uncond_prob=None,+ x_noisy_ref=None,+ ):"""Compute training losses for a single timestep.Arguments format copied from opensora/schedulers/iddpm/gaussian_diffusion.py/training_lossesNote: t is int tensor and should be rescaled from [0, num_timesteps-1] to [1,0]"""- if t is None:- if self.use_discrete_timesteps:- t = torch.randint(0, self.num_timesteps, (x_start.shape[0],), device=x_start.device)- elif self.sample_method == "uniform":- t = torch.rand((x_start.shape[0],), device=x_start.device) * self.num_timesteps- elif self.sample_method == "logit-normal":- t = self.sample_t(x_start) * self.num_timesteps- if self.use_timestep_transform:- t = timestep_transform(t, model_kwargs, scale=self.transform_scale, num_timesteps=self.num_timesteps)+ y_null = None+ if mask_index is not None and len(mask_index) > 0: # i2v and v2v+ num_frames = x_start.shape[2]+ x_cond_mask = torch.zeros_like(x_start, device=x_start.device)+ x_cond_mask[:, :, mask_index, :, :] = 1.0+ x_noisy_ref = x_noisy_ref if x_noisy_ref is not None else x_start+ x_cond = x_noisy_ref * x_cond_mask+ y_null = (+ model.module.y_embedder.y_embedding[None]+ .repeat(model_kwargs["y"].shape[1], 1, 1)[:, None]+ .expand_as(model_kwargs["y"])+ )+ condition = random.choices(+ list(self.drop_condition.keys()), weights=list(self.drop_condition.values()), k=1+ )[0]+ if condition == "null": # no text, no x_cond+ x_cond = torch.zeros_like(x_cond).to(x_cond.device).to(x_cond.dtype)+ model_kwargs["y"] = y_null+ elif condition == "cond": # no text+ model_kwargs["y"] = y_null+ elif condition == "text": # no x_cond+ x_cond = torch.zeros_like(x_cond).to(x_cond.device).to(x_cond.dtype)+ else:+ x_cond_mask = torch.zeros_like(x_start, device=x_start.device)+ x_cond = x_start * x_cond_mask+ if t is None:+ t = self.time_sampler.sample(x_start, self.num_timesteps, model_kwargs)if model_kwargs is None:model_kwargs = {}if noise is None:noise = torch.randn_like(x_start)assert noise.shape == x_start.shape+ if noise_disable_threshold is not None:+ no_noise_mask = t > noise_disable_threshold+ x_start[no_noise_mask] = x_gt[no_noise_mask]+x_t = self.add_noise(x_start, noise, t)if mask is not None:t0 = torch.zeros_like(t)@@ -98,13 +103,44 @@ class RFlowScheduler:x_t = torch.where(mask[:, None, :, None, None], x_t, x_t0)terms = {}- model_output = model(x_t, t, **model_kwargs)+ # model_output = model(x_t, t, **model_kwargs)+ model_output = model(+ x_t,+ t,+ cond=x_cond,+ cond_mask=x_cond_mask,+ mask_index=mask_index,+ y_null=y_null,+ text_uncond_prob=text_uncond_prob,+ **model_kwargs,+ )velocity_pred = model_output.chunk(2, dim=1)[0]++ res_weights = None+ if mask_index is not None and len(mask_index) > 0:+ mask_nums = len(mask_index)+ new_weight = (num_frames - mask_nums * self.x_cond_weight) / (+ num_frames - mask_nums+ ) # scale to ensure sum is same+ res_weights = (+ torch.ones_like(velocity_pred).to(velocity_pred.device).to(velocity_pred.dtype) * new_weight+ ) # set non-masked frame weight+ res_weights[:, :, mask_index, :, :] = 1 * self.x_cond_weight # set masked frame weight++ x_target = x_gt if x_gt is not None else x_startif weights is None:- loss = mean_flat((velocity_pred - (x_start - noise)).pow(2), mask=mask)+ loss = (+ mean_flat(res_weights * (velocity_pred - (x_target - noise)).pow(2), mask=mask)+ if res_weights is not None+ else mean_flat((velocity_pred - (x_target - noise)).pow(2), mask=mask)+ )else:- weight = _extract_into_tensor(weights, t, x_start.shape)- loss = mean_flat(weight * (velocity_pred - (x_start - noise)).pow(2), mask=mask)+ weight = _extract_into_tensor(weights, t, x_target.shape)+ loss = (+ mean_flat(res_weights * weight * (velocity_pred - (x_target - noise)).pow(2), mask=mask)+ if res_weights is not None+ else mean_flat(weight * (velocity_pred - (x_target - noise)).pow(2), mask=mask)+ )terms["loss"] = lossreturn termsdiff --git a/opensora/schedulers/rf/time_sampler.py b/opensora/schedulers/rf/time_sampler.pynew file mode 100644index 0000000..93bf415--- /dev/null+++ b/opensora/schedulers/rf/time_sampler.py@@ -0,0 +1,193 @@+import matplotlib.pyplot as plt+import numpy as np+import torch+from torch.distributions import LogisticNormal+++def extract_hwt(model_kwargs, scale_temporal=True):+ # Force fp16 input to fp32 to avoid nan output+ for key in ["height", "width", "num_frames"]:+ assert key in model_kwargs, f"model_kwargs must contain key {key}"+ if isinstance(model_kwargs[key], torch.Tensor) and model_kwargs[key].dtype == torch.float16:+ model_kwargs[key] = model_kwargs[key].float()++ resolution = model_kwargs["height"] * model_kwargs["width"]+ # NOTE: currently, we do not take fps into account+ # NOTE: temporal_reduction is hardcoded, this should be equal to the temporal reduction factor of the vae+ if model_kwargs["num_frames"][0] == 1 or not scale_temporal:+ num_frames = torch.ones_like(model_kwargs["num_frames"])+ is_image = True+ else:+ # num_frames = model_kwargs["num_frames"] // 17 * 5+ num_frames = (model_kwargs["num_frames"] - 1) // 4+ is_image = False+ return resolution, num_frames, is_image+++def timestep_transform(+ t,+ model_kwargs,+ base_resolution=512 * 512,+ base_num_frames=1,+ scale=1.0,+ num_timesteps=1,+ scale_temporal=True,+ ret_ratio=False,+ uniform_over_threshold=None,+):+ t = t / num_timesteps++ resolution, num_frames, is_image = extract_hwt(model_kwargs, scale_temporal)++ ratio_space = (resolution / base_resolution).sqrt()+ ratio_time = (num_frames / base_num_frames).sqrt()+ ratio = ratio_space * ratio_time+ if not is_image:+ ratio = ratio * scale+ t = ratio * t / (1 + (ratio - 1) * t)++ if uniform_over_threshold is not None and not is_image:+ t_over = t > uniform_over_threshold+ t_resample = torch.rand_like(t) * (1 - uniform_over_threshold) + uniform_over_threshold+ t = t_over * t_resample + ~t_over * t++ t = t * num_timesteps+ if ret_ratio:+ return t, ratio+ return t+++class TimeSampler:+ def __init__(+ self,+ sample_method="uniform",+ use_discrete_timesteps=False,+ use_timestep_transform=False,+ transform_scale=1.0,+ scale_temporal=True,+ loc=0.0,+ scale=1.0,+ uniform_over_threshold=None,+ ):+ assert sample_method in ["uniform", "logit-normal"]+ assert (+ sample_method == "uniform" or not use_discrete_timesteps+ ), "Only uniform sampling is supported for discrete timesteps"++ self.sample_method = sample_method+ self.use_discrete_timesteps = use_discrete_timesteps+ self.use_timestep_transform = use_timestep_transform+ self.transform_scale = transform_scale+ self.scale_temporal = scale_temporal+ self.uniform_over_threshold = uniform_over_threshold+ if sample_method == "logit-normal":+ self.distribution = LogisticNormal(torch.tensor([loc]), torch.tensor([scale]))+ self.sample_t = lambda x: self.distribution.sample((x.shape[0],))[:, 0].to(x.device)++ def sample(self, x_start, num_timesteps, model_kwargs=None):+ if self.use_discrete_timesteps:+ t = torch.randint(0, num_timesteps, (x_start.shape[0],), device=x_start.device)+ elif self.sample_method == "uniform":+ t = torch.rand((x_start.shape[0],), device=x_start.device) * num_timesteps+ elif self.sample_method == "logit-normal":+ t = self.sample_t(x_start) * num_timesteps++ if not self.use_timestep_transform:+ return t++ t = timestep_transform(+ t,+ model_kwargs,+ scale=self.transform_scale,+ num_timesteps=num_timesteps,+ scale_temporal=self.scale_temporal,+ uniform_over_threshold=self.uniform_over_threshold,+ )++ return t++ def visualize(self, height=360, width=640, num_frames=113, num_timesteps=1000):+ bs = 10000+ x_start = torch.randn(bs)+ self.use_timestep_transform = False+ original_t_values = self.sample(x_start, num_timesteps)+ self.use_timestep_transform = True++ color = "#8E7CC3"+ label = f"({height}x{width})"+ bins = np.linspace(0, 1000, 1000)+ fig, axes = plt.subplots(2, 1, figsize=(18, 18))++ model_kwargs = {+ "height": torch.full((bs,), height),+ "width": torch.full((bs,), width),+ "num_frames": torch.full((bs,), num_frames),+ }+ transformed_t_values, ratio = timestep_transform(+ original_t_values,+ model_kwargs,+ scale=2,+ num_timesteps=num_timesteps,+ scale_temporal=self.scale_temporal,+ ret_ratio=True,+ uniform_over_threshold=0.95,+ )++ # Scatter with lines+ axes[0].scatter(+ original_t_values.numpy(),+ transformed_t_values.numpy(),+ label=f"{label} (scale={ratio[0].item()})",+ s=10,+ color=color,+ )+ for j in range(len(original_t_values)):+ axes[0].plot(+ [original_t_values[j], original_t_values[j]],+ [original_t_values[j], transformed_t_values[j]],+ color="gray",+ alpha=0.5,+ )++ axes[0].scatter(+ original_t_values.numpy(), original_t_values.numpy(), label="Original $t$", color="#C27BA0", s=10+ ) # Soft Rose+ axes[0].set_xlabel("Original $t$")+ axes[0].set_ylabel("Transformed $new_t$")+ axes[0].set_title(f"{label} Transformation")+ axes[0].legend()+ axes[0].grid(True)++ # Histogram+ axes[1].hist(original_t_values.numpy(), bins=bins, alpha=0.6, label="Original", color="#C27BA0")+ axes[1].hist(transformed_t_values.numpy(), bins=bins, alpha=0.6, label=f"Transformed ({label})", color=color)++ axes[1].set_xlabel("Adjusted $new_t$")+ axes[1].set_ylabel("Number of Points")+ axes[1].set_title(f"Distribution of Original and Transformed $t$ ({label})")+ axes[1].legend()++ plt.tight_layout()+ path = "./samples/infos/timestep_sampling_comparison.png"+ plt.savefig(path)+ print(f"Saved figure to {path}")++ prob_80 = ((transformed_t_values > 800).sum() / len(transformed_t_values)).item()+ prob_85 = ((transformed_t_values > 850).sum() / len(transformed_t_values)).item()+ prob_90 = ((transformed_t_values > 900).sum() / len(transformed_t_values)).item()+ prob_95 = ((transformed_t_values > 950).sum() / len(transformed_t_values)).item()+ prob_99 = ((transformed_t_values > 990).sum() / len(transformed_t_values)).item()+ print(f"Ratio: {ratio[0].item()}")+ print(f"Probability of t > 800: {prob_80}")+ print(f"Probability of t > 850: {prob_85}")+ print(f"Probability of t > 900: {prob_90}")+ print(f"Probability of t > 950: {prob_95}")+ print(f"Probability of t > 990: {prob_99}")+++if __name__ == "__main__":+ time_sampler = TimeSampler(+ use_timestep_transform=True,+ sample_method="logit-normal",+ )+ time_sampler.visualize()diff --git a/opensora/utils/ckpt_utils.py b/opensora/utils/ckpt_utils.pyindex 78b1631..fb3d266 100644--- a/opensora/utils/ckpt_utils.py+++ b/opensora/utils/ckpt_utils.py@@ -2,16 +2,18 @@ import functoolsimport jsonimport operatorimport os-from typing import Dict, Optional, Tuple+import re+import shutil+from glob import glob+from typing import Tupleimport torchimport torch.distributed as distimport torch.nn as nnfrom colossalai.booster import Boosterfrom colossalai.checkpoint_io import GeneralCheckpointIO-from colossalai.utils.safetensors import save as async_save+from huggingface_hub import hf_hub_downloadfrom safetensors.torch import load_file-from tensornvme.async_file_io import AsyncFileWriterfrom torch.optim import Optimizerfrom torch.optim.lr_scheduler import _LRSchedulerfrom torchvision.datasets.utils import download_url@@ -103,7 +105,7 @@ def reparameter(ckpt, name=None, model=None):)ckpt["y_embedder.y_embedding"] = ckpt["y_embedder.y_embedding"][: model.y_embedder.y_embedding.shape[0]]# stdit3 special case- if type(model).__name__ == "STDiT3" and "PixArt-Sigma" in name:+ if type(model).__name__ == "STDiT3" and "PixArt" in model_name:ckpt_keys = list(ckpt.keys())for key in ckpt_keys:if "blocks." in key:@@ -153,14 +155,7 @@ def load_from_sharded_state_dict(model, ckpt_path, model_name="model", strict=Fackpt_io.load_model(model, os.path.join(ckpt_path, model_name), strict=strict)-def model_sharding(model: torch.nn.Module, device: torch.device = None):- """- Sharding the model parameters across multiple GPUs.-- Args:- model (torch.nn.Module): The model to shard.- device (torch.device): The device to shard the model to.- """+def model_sharding(model: torch.nn.Module, device=None):global_rank = dist.get_rank()world_size = dist.get_world_size()for _, param in model.named_parameters():@@ -176,31 +171,17 @@ def model_sharding(model: torch.nn.Module, device: torch.device = None):param.data = splited_params.to(device)-def model_gathering(model: torch.nn.Module, model_shape_dict: dict, pinned_state_dict: dict) -> None:- """- Gather the model parameters from multiple GPUs.-- Args:- model (torch.nn.Module): The model to gather.- model_shape_dict (dict): The shape of the model parameters.- device (torch.device): The device to gather the model to.- """+def model_gathering(model: torch.nn.Module, model_shape_dict: dict, device=None):global_rank = dist.get_rank()global_size = dist.get_world_size()- params = set()for name, param in model.named_parameters():- params.add(name)+ if device is None:+ device = param.deviceall_params = [torch.empty_like(param.data) for _ in range(global_size)]dist.all_gather(all_params, param.data, group=dist.group.WORLD)if int(global_rank) == 0:all_params = torch.cat(all_params)- gathered_param = remove_padding(all_params, model_shape_dict[name]).view(model_shape_dict[name])- pinned_state_dict[name].copy_(gathered_param)- if int(global_rank) == 0:- for k, v in model.state_dict(keep_vars=True).items():- if k not in params:- pinned_state_dict[k].copy_(v)-+ param.data = remove_padding(all_params, model_shape_dict[name]).view(model_shape_dict[name]).to(device)dist.barrier()@@ -215,28 +196,84 @@ def record_model_param_shape(model: torch.nn.Module) -> dict:return param_shape-def load_checkpoint(model, ckpt_path, save_as_pt=False, model_name="model", strict=False):- if ckpt_path.endswith(".pt") or ckpt_path.endswith(".pth"):- state_dict = find_model(ckpt_path, model=model)- missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=strict)+def adapt_16ch_vae(state_dict):+ if "x_embedder.proj.weight" in state_dict:+ # state_dict.pop("x_embedder.proj.weight")+ # state_dict.pop("final_layer.linear.bias")+ # state_dict.pop("final_layer.linear.weight")++ state_dict["x_embedder.proj.weight"] = state_dict["x_embedder.proj.weight"].repeat(1, 4, 1, 1, 1) / 4+ state_dict["x_embedder.proj.bias"] = state_dict["x_embedder.proj.bias"]+ state_dict["final_layer.linear.weight"] = (+ state_dict["final_layer.linear.weight"].reshape(4, 2, 4, -1).repeat(1, 1, 4, 1).reshape(128, -1)+ )+ state_dict["final_layer.linear.bias"] = (+ state_dict["final_layer.linear.bias"].reshape(4, 2, 4).repeat(1, 1, 4).reshape(128)+ )+ return state_dict+++# def adapt_v2v(state_dict, first_adapter = False):+# if first_adapter:+# ori_dim = state_dict['final_layer.linear.weight'].shape[0]+# state_dict['final_layer.linear.weight'] = state_dict['final_layer.linear.weight'][:ori_dim//2,:]+# state_dict['final_layer.linear.bias'] = state_dict['final_layer.linear.bias'][:ori_dim//2]+# else:+# pass+# return state_dict+++def load_from_hf_hub(repo_path, cache_dir):+ repo_id = "/".join(repo_path.split("/")[:-1])+ repo_file = repo_path.split("/")[-1]+ try:+ ckpt_path = hf_hub_download(repo_id=repo_id, filename=repo_file, cache_dir=cache_dir)+ return ckpt_path+ except Exception as e:+ print(f"Error downloading from Hugging Face Hub: {e}")+ return None+++def load_checkpoint(+ model,+ ckpt_path,+ save_as_pt=False,+ model_name="model",+ strict=False,+ adapt_16ch=False,+ cache_dir=None,+ device: torch.device | str = "cpu",+):+ if not os.path.exists(ckpt_path):+ get_logger().info(f"Checkpoint not found at {ckpt_path} trying to download from Hugging Face Hub")+ ckpt_path = load_from_hf_hub(ckpt_path, cache_dir)++ get_logger().info(f"Loading checkpoint from {ckpt_path}")++ if ckpt_path.endswith(".safetensors"):+ ckpt = load_file(ckpt_path, device=str(device))+ missing_keys, unexpected_keys = model.load_state_dict(ckpt, strict=strict)get_logger().info("Missing keys: %s", missing_keys)get_logger().info("Unexpected keys: %s", unexpected_keys)- elif ckpt_path.endswith(".safetensors"):- from safetensors.torch import load_file- state_dict = load_file(ckpt_path)- missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=False)+ elif ckpt_path.endswith(".pt") or ckpt_path.endswith(".pth"):+ # state_dict = find_model(ckpt_path, model=model)+ state_dict = torch.load(ckpt_path, map_location=device)+ if adapt_16ch:+ state_dict = adapt_16ch_vae(state_dict)+ missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=strict)get_logger().info("Missing keys: %s", missing_keys)get_logger().info("Unexpected keys: %s", unexpected_keys)- elif os.path.isdir(ckpt_path):+ else:+ assert os.path.isdir(ckpt_path), f"Invalid checkpoint path: {ckpt_path}"load_from_sharded_state_dict(model, ckpt_path, model_name, strict=strict)get_logger().info("Model checkpoint loaded from %s", ckpt_path)if save_as_pt:save_path = os.path.join(ckpt_path, model_name + "_ckpt.pt")torch.save(model.state_dict(), save_path)get_logger().info("Model checkpoint saved to %s", save_path)- else:- raise ValueError(f"Invalid checkpoint path: {ckpt_path}")++ return modeldef load_json(file_path: str):@@ -252,130 +289,113 @@ def save_json(data, file_path: str):# save and load for training-def _prepare_ema_pinned_state_dict(model: nn.Module, ema_shape_dict: dict):- ema_pinned_state_dict = dict()- for name, p in model.named_parameters():- ema_pinned_state_dict[name] = torch.empty(ema_shape_dict[name], pin_memory=True, device="cpu", dtype=p.dtype)- sd = model.state_dict(keep_vars=True)- # handle buffers- for k, v in sd.items():- if k not in ema_pinned_state_dict:- ema_pinned_state_dict[k] = torch.empty(v.shape, pin_memory=True, device="cpu", dtype=v.dtype)-- return ema_pinned_state_dict---class CheckpointIO:- def __init__(self, n_write_entries: int = 32):- self.n_write_entries = n_write_entries- self.writer: Optional[AsyncFileWriter] = None- self.pinned_state_dict: Optional[Dict[str, torch.Tensor]] = None-- def _sync_io(self):- if self.writer is not None:- self.writer = None-- def __del__(self):- self._sync_io()-- def _prepare_pinned_state_dict(self, ema: nn.Module, ema_shape_dict: dict):- if self.pinned_state_dict is None and dist.get_rank() == 0:- self.pinned_state_dict = _prepare_ema_pinned_state_dict(ema, ema_shape_dict)-- def save(- self,- booster: Booster,- save_dir: str,- model: nn.Module = None,- ema: nn.Module = None,- optimizer: Optimizer = None,- lr_scheduler: _LRScheduler = None,- sampler=None,- epoch: int = None,- step: int = None,- global_step: int = None,- batch_size: int = None,- ema_shape_dict: dict = None,- async_io: bool = True,- ):- self._sync_io()- save_dir = os.path.join(save_dir, f"epoch{epoch}-global_step{global_step}")- os.environ["TENSORNVME_DEBUG_LOG"] = os.path.join(save_dir, "async_file_io.log")+def save(+ booster: Booster,+ save_dir: str,+ model: nn.Module = None,+ ema: nn.Module = None,+ optimizer: Optimizer = None,+ lr_scheduler: _LRScheduler = None,+ sampler=None,+ epoch: int = None,+ step: int = None,+ global_step: int = None,+ batch_size: int = None,+ is_lora_train: bool = False,+ is_save_both_lora_and_main: bool = False,+):+ save_dir = os.path.join(save_dir, f"epoch{epoch}-global_step{global_step}")+ if is_lora_train:+ os.makedirs(os.path.join(save_dir, "lora"), exist_ok=True)+ else:os.makedirs(os.path.join(save_dir, "model"), exist_ok=True)- if model is not None:- booster.save_model(- model,- os.path.join(save_dir, "model"),- shard=True,- use_safetensors=True,- size_per_shard=4096,- use_async=async_io,- )- if optimizer is not None:- booster.save_optimizer(- optimizer, os.path.join(save_dir, "optimizer"), shard=True, size_per_shard=4096, use_async=async_io- )- if lr_scheduler is not None:- booster.save_lr_scheduler(lr_scheduler, os.path.join(save_dir, "lr_scheduler"))- if ema is not None:- self._prepare_pinned_state_dict(ema, ema_shape_dict)- model_gathering(ema, ema_shape_dict, self.pinned_state_dict)- if dist.get_rank() == 0:- running_states = {- "epoch": epoch,- "step": step,- "global_step": global_step,- "batch_size": batch_size,- }- save_json(running_states, os.path.join(save_dir, "running_states.json"))-- if ema is not None:- if async_io:- self.writer = async_save(os.path.join(save_dir, "ema.safetensors"), self.pinned_state_dict)- else:- torch.save(self.pinned_state_dict, os.path.join(save_dir, "ema.pt"))-- if sampler is not None:- # only for VariableVideoBatchSampler- torch.save(sampler.state_dict(step), os.path.join(save_dir, "sampler"))- dist.barrier()- return save_dir-- def load(- self,- booster: Booster,- load_dir: str,- model: nn.Module = None,- ema: nn.Module = None,- optimizer: Optimizer = None,- lr_scheduler: _LRScheduler = None,- sampler=None,- ) -> Tuple[int, int, int]:- assert os.path.exists(load_dir), f"Checkpoint directory {load_dir} does not exist"- assert os.path.exists(os.path.join(load_dir, "running_states.json")), "running_states.json does not exist"- running_states = load_json(os.path.join(load_dir, "running_states.json"))- if model is not None:- booster.load_model(model, os.path.join(load_dir, "model"))- if ema is not None:- # ema is not boosted, so we don't use booster.load_model- if os.path.exists(os.path.join(load_dir, "ema.safetensors")):- ema_state_dict = load_file(os.path.join(load_dir, "ema.safetensors"))+ if model is not None:+ if not is_lora_train:+ booster.save_model(model, os.path.join(save_dir, "model"), shard=True)+ else:+ if is_save_both_lora_and_main:+ booster.save_model(model, os.path.join(save_dir, "model"), shard=True)+ booster.save_lora_as_pretrained(model, os.path.join(save_dir, "lora"))else:- ema_state_dict = torch.load(os.path.join(load_dir, "ema.pt"), map_location=torch.device("cpu"))- ema.load_state_dict(- ema_state_dict,- strict=False,- )- if optimizer is not None:- booster.load_optimizer(optimizer, os.path.join(load_dir, "optimizer"))- if lr_scheduler is not None:- booster.load_lr_scheduler(lr_scheduler, os.path.join(load_dir, "lr_scheduler"))- if sampler is not None:- sampler.load_state_dict(torch.load(os.path.join(load_dir, "sampler")))- dist.barrier()+ booster.save_lora_as_pretrained(model, os.path.join(save_dir, "lora"))++ if optimizer is not None:+ booster.save_optimizer(optimizer, os.path.join(save_dir, "optimizer"), shard=True, size_per_shard=4096)+ if lr_scheduler is not None:+ booster.save_lr_scheduler(lr_scheduler, os.path.join(save_dir, "lr_scheduler"))+ if dist.get_rank() == 0:+ running_states = {+ "epoch": epoch,+ "step": step,+ "global_step": global_step,+ "batch_size": batch_size,+ }+ save_json(running_states, os.path.join(save_dir, "running_states.json"))- return (- running_states["epoch"],- running_states["step"],+ if ema is not None:+ torch.save(ema.state_dict(), os.path.join(save_dir, "ema.pt"))++ if sampler is not None:+ # only for VariableVideoBatchSampler+ torch.save(sampler.state_dict(step), os.path.join(save_dir, "sampler"))+ dist.barrier()+ return save_dir+++def load(+ booster: Booster,+ load_dir: str,+ model: nn.Module = None,+ ema: nn.Module = None,+ optimizer: Optimizer = None,+ lr_scheduler: _LRScheduler = None,+ sampler=None,+ is_lora_train: bool = False,+ is_load_both_lora_and_main: bool = False,+) -> Tuple[int, int, int]:+ assert os.path.exists(load_dir), f"Checkpoint directory {load_dir} does not exist"+ assert os.path.exists(os.path.join(load_dir, "running_states.json")), "running_states.json does not exist"+ running_states = load_json(os.path.join(load_dir, "running_states.json"))+ if model is not None:+ if is_lora_train:+ if is_load_both_lora_and_main:+ booster.load_model(model, os.path.join(load_dir, "lora", "adapter_model.bin"), strict=False)+ booster.load_model(model, os.path.join(load_dir, "model"), strict=False)+ else:+ booster.load_model(model, os.path.join(load_dir, "lora", "adapter_model.bin"), strict=False)+ else:+ booster.load_model(model, os.path.join(load_dir, "model"), strict=False)+ if ema is not None:+ # ema is not boosted, so we don't use booster.load_model+ ema.load_state_dict(+ torch.load(os.path.join(load_dir, "ema.pt"), map_location=torch.device("cpu")),+ strict=False,)+ if optimizer is not None:+ booster.load_optimizer(optimizer, os.path.join(load_dir, "optimizer"))+ if lr_scheduler is not None:+ booster.load_lr_scheduler(lr_scheduler, os.path.join(load_dir, "lr_scheduler"))+ if sampler is not None:+ sampler.load_state_dict(torch.load(os.path.join(load_dir, "sampler")))+ dist.barrier()++ return (+ running_states["epoch"],+ running_states["step"],+ )+++def rm_checkpoints(+ save_dir: str,+ keep_n_latest: int = 0,+):+ if keep_n_latest <= 0 or dist.get_rank() != 0:+ return+ files = glob(os.path.join(save_dir, "epoch*-global_step*"))+ files = sorted(+ files, key=lambda s: tuple(map(int, re.search(r"epoch(\d+)-global_step(\d+)", s).groups())), reverse=True+ )+ to_remove = files[keep_n_latest:]+ for f in to_remove:+ shutil.rmtree(f)diff --git a/opensora/utils/config_utils.py b/opensora/utils/config_utils.pyindex 0333bec..5ff17ba 100644--- a/opensora/utils/config_utils.py+++ b/opensora/utils/config_utils.py@@ -22,6 +22,12 @@ def parse_args(training=False):type=str,help="path to model ckpt; will overwrite cfg.model.from_pretrained if specified",)+ parser.add_argument(+ "--ckpt-path-lora",+ default=None,+ type=str,+ help="path to lora model ckpt; will overwrite cfg.from_pretrained_lora if specified",+ )parser.add_argument("--batch-size", default=None, type=int, help="batch size")parser.add_argument("--outputs", default=None, type=str, help="the dir to save model weights")parser.add_argument("--flash-attn", default=None, type=str2bool, help="enable flash attention")@@ -29,6 +35,12 @@ def parse_args(training=False):parser.add_argument("--resolution", default=None, type=str, help="multi resolution")parser.add_argument("--data-path", default=None, type=str, help="path to data csv")parser.add_argument("--dtype", default=None, type=str, help="data type")+ parser.add_argument(+ "--force-huggingface",+ default=None,+ type=str2bool,+ help="set to true for huggingface format model, if local, set to false",+ )# ======================================================# Inference@@ -46,8 +58,17 @@ def parse_args(training=False):# promptparser.add_argument("--prompt-path", default=None, type=str, help="path to prompt txt file")parser.add_argument("--prompt", default=None, type=str, nargs="+", help="prompt list")+ parser.add_argument("--image-prompt-path", default=None, type=str, help="path to image prompt txt file")+ parser.add_argument("--image-prompt", default=None, type=str, nargs="+", help="image prompt list")+ parser.add_argument(+ "--csv-reference-column-name",+ default=None,+ type=str,+ help="column name of the reference path if prompt path is an csv",+ )parser.add_argument("--llm-refine", default=None, type=str2bool, help="enable LLM refine")parser.add_argument("--prompt-generator", default=None, type=str, help="prompt generator")+ parser.add_argument("--num-prompt-generator", default=None, type=int, help="number of prompts to generate")# image/videoparser.add_argument("--num-frames", default=None, type=str, help="number of frames")@@ -61,22 +82,51 @@ def parse_args(training=False):# hyperparametersparser.add_argument("--num-sampling-steps", default=None, type=int, help="sampling steps")parser.add_argument("--cfg-scale", default=None, type=float, help="balance between cond & uncond")+ parser.add_argument(+ "--image-cfg-scale", default=None, type=float, help="balance between cond & uncond for image and video"+ )# referenceparser.add_argument("--loop", default=None, type=int, help="loop")+ parser.add_argument(+ "--cond-type",+ default=None,+ type=str,+ choices=["v2v_head", "i2v_head", "i2v_loop", "i2v_tail"],+ help="how to condition on the reference",+ )+ parser.add_argument("--use-sdedit", default=None, type=str2bool, help="whether use sdeit during inference")+ parser.add_argument(+ "--use-oscillation-guidance-for-text",+ default=None,+ type=str2bool,+ help="whether use oscillated weights for text cfg guidance",+ )+ parser.add_argument(+ "--use-oscillation-guidance-for-image",+ default=None,+ type=str2bool,+ help="whether use oscillated weights for image cfg guidance",+ )+parser.add_argument("--condition-frame-length", default=None, type=int, help="condition frame length")parser.add_argument("--reference-path", default=None, type=str, nargs="+", help="reference path")+ parser.add_argument("--reference-txt-path", default=None, type=str, help="path to reference txt file")parser.add_argument("--mask-strategy", default=None, type=str, nargs="+", help="mask strategy")- parser.add_argument("--aes", default=None, type=float, help="aesthetic score")- parser.add_argument("--flow", default=None, type=float, help="flow score")+ parser.add_argument("--neg-prompt", default=None, type=str, nargs="+", help="negative prompt")+ parser.add_argument("--aes", default=None, type=str, help="aesthetic score")+ parser.add_argument("--flow", default=None, type=str, help="flow score")parser.add_argument("--camera-motion", default=None, type=str, help="camera motion")+ parser.add_argument("--super-resolution", default=None, type=float, help="super resolution")+ parser.add_argument("--deflicker", default=None, type=str2bool, help="deflicker")# ======================================================# Training# ======================================================else:parser.add_argument("--lr", default=None, type=float, help="learning rate")- parser.add_argument("--wandb", default=None, type=bool, help="enable wandb")+ parser.add_argument("--wandb", default=None, type=str2bool, help="enable wandb")parser.add_argument("--load", default=None, type=str, help="path to continue training")+ parser.add_argument("--load-lora", default=None, type=str, help="path to continue training LoRA")parser.add_argument("--start-from-scratch", action="store_true", help="start training from scratch")parser.add_argument("--warmup-steps", default=None, type=int, help="warmup steps")parser.add_argument("--record-time", default=False, action="store_true", help="record time of each part")@@ -90,6 +140,11 @@ def merge_args(cfg, args, training=False):if cfg.get("discriminator") is not None:cfg.discriminator["from_pretrained"] = args.ckpt_pathargs.ckpt_path = None+ if args.ckpt_path_lora is not None:+ cfg.from_pretrained_lora = args.ckpt_path_lora+ args.ckpt_path_lora = None+ if args.force_huggingface is not None:+ cfg.model["force_huggingface"] = args.force_huggingfaceif args.flash_attn is not None:cfg.model["enable_flash_attn"] = args.flash_attnargs.enable_flash_attn = None@@ -112,6 +167,8 @@ def merge_args(cfg, args, training=False):if args.num_sampling_steps is not None:cfg.scheduler["num_sampling_steps"] = args.num_sampling_stepsargs.num_sampling_steps = None+ if args.num_frames is not None and args.num_frames.isdigit():+ assert int(args.num_frames) in [49, 65, 81, 97, 113], f"--num-frames must be any of [49, 65, 81, 97, 113]"for k, v in vars(args).items():if v is not None:@@ -150,7 +207,7 @@ def define_experiment_workspace(cfg, get_last_workspace=False):# Create an experiment foldermodel_name = cfg.model["type"].replace("/", "-")- exp_name = f"{experiment_index:03d}-{model_name}"+ exp_name = f"{experiment_index:04d}-{model_name}"exp_dir = f"{cfg.outputs}/{exp_name}"return exp_name, exp_dirdiff --git a/opensora/utils/inference_utils.py b/opensora/utils/inference_utils.pyindex b9536a7..1a95b32 100644--- a/opensora/utils/inference_utils.py+++ b/opensora/utils/inference_utils.py@@ -1,8 +1,16 @@+import base64import jsonimport osimport re+import shutil+import tempfile+from io import BytesIO+import cv2+import numpy as np+import pandas as pdimport torch+from torchvision import transformsfrom opensora.datasets import IMG_FPSfrom opensora.datasets.utils import read_from_path@@ -27,9 +35,21 @@ def prepare_multi_resolution_info(info_type, batch_size, image_size, num_frames,raise NotImplementedError-def load_prompts(prompt_path, start_idx=None, end_idx=None):- with open(prompt_path, "r") as f:- prompts = [line.strip() for line in f.readlines()]+def load_prompts(prompt_path, start_idx=None, end_idx=None, csv_ref_column_name=None):+ if prompt_path.endswith(".txt"):+ with open(prompt_path, "r") as f:+ prompts = [line.strip() for line in f.readlines()]+ elif prompt_path.endswith(".csv"):+ df = pd.read_csv(prompt_path)+ prompts = df["text"].tolist()+ if csv_ref_column_name is not None:+ assert (+ csv_ref_column_name in df+ ), f"column {csv_ref_column_name} for reference paths not found in {prompt_path}"+ reference_paths = df[csv_ref_column_name]+ return (prompts, reference_paths)+ else:+ raise ValueErrorprompts = prompts[start_idx:end_idx]return prompts@@ -52,17 +72,63 @@ def get_save_path_name(return save_path+def transform_aes(aes):+ # < 4 filter out+ if aes < 4:+ return "terrible"+ elif aes < 4.5:+ return "very poor"+ elif aes < 5:+ return "poor"+ elif aes < 5.5:+ return "fair"+ elif aes < 6:+ return "good"+ elif aes < 6.5:+ return "very good"+ else:+ return "excellent"+++def transform_motion(motion):+ # < 0.3 filter out+ if motion < 0.5:+ return "very low"+ elif motion < 2:+ return "low"+ elif motion < 5:+ return "fair"+ elif motion < 10:+ return "high"+ elif motion < 20:+ return "very high"+ else:+ return "extremely high"++def append_score_to_prompts(prompts, aes=None, flow=None, camera_motion=None):new_prompts = []for prompt in prompts:new_prompt = prompt- if aes is not None and "aesthetic score:" not in prompt:- new_prompt = f"{new_prompt} aesthetic score: {aes:.1f}."- if flow is not None and "motion score:" not in prompt:- new_prompt = f"{new_prompt} motion score: {flow:.1f}."+ if aes is not None and "aesthetic score is" not in prompt:+ try:+ aes = float(aes)+ aes = transform_aes(aes)+ except ValueError:+ pass # already in text format+ new_prompt = f"{new_prompt} The aesthetic score is {aes}."++ if flow is not None and "motion strength is" not in prompt:+ try:+ flow = float(flow)+ flow = transform_motion(flow)+ except ValueError:+ pass # already in text format+ new_prompt = f"{new_prompt} The motion strength is {flow}."if camera_motion is not None and "camera motion:" not in prompt:new_prompt = f"{new_prompt} camera motion: {camera_motion}."new_prompts.append(new_prompt)+ print("processed prompt:\n", new_prompts)return new_prompts@@ -83,6 +149,25 @@ def extract_json_from_prompts(prompts, reference, mask_strategy):return ret_prompts, reference, mask_strategy+def extract_json_from_prompts_new(prompts_with_json):+ prompts = []+ reference = []+ mask_strategy = []+ for i, prompt in enumerate(prompts_with_json):+ parts = re.split(r"(?=[{])", prompt)+ assert len(parts) <= 2, f"Invalid prompt: {prompt}"+ prompts.append(parts[0])+ if len(parts) > 1:+ additional_info = json.loads(parts[1])+ for key in additional_info:+ assert key in ["reference_path", "mask_strategy"], f"Invalid key: {key}"+ if key == "reference_path":+ reference[i] = additional_info[key]+ elif key == "mask_strategy":+ mask_strategy[i] = additional_info[key]+ return prompts, reference, mask_strategy++def collect_references_batch(reference_paths, vae, image_size):refs_x = [] # refs_x: [batch, ref_num, C, T, H, W]for reference_path in reference_paths:@@ -93,6 +178,17 @@ def collect_references_batch(reference_paths, vae, image_size):ref = []for r_path in ref_path:r = read_from_path(r_path, image_size, transform_name="resize_crop")++ # need to ensure r has length accepted by vae+ actual_t = r.size(1)+ if vae.micro_frame_size is None:+ target_t = (actual_t - 1) // 4 * 4 + 1+ elif not vae.temporal_overlap:+ target_t = actual_t // vae.micro_frame_size * vae.micro_frame_size+ else:+ target_t = (actual_t - 1) // (vae.micro_frame_size - 1) * (vae.micro_frame_size - 1) + 1+ r = r[:, :target_t]+r_x = vae.encode(r.unsqueeze(0).to(vae.device, vae.dtype))r_x = r_x.squeeze(0)ref.append(r_x)@@ -100,6 +196,21 @@ def collect_references_batch(reference_paths, vae, image_size):return refs_x+def extract_images_from_ref_paths(reference_paths, image_size):+ refs_images = [] # refs_images: [batch, ref_num, C, T, H, W]+ for reference_path in reference_paths:+ if reference_path == "":+ refs_images.append([])+ continue+ ref_path = reference_path.split(";")+ ref = []+ for r_path in ref_path:+ r = read_from_path(r_path, image_size, transform_name="resize_crop")+ ref.append(r.squeeze(1))+ refs_images.append(ref)+ return refs_images++def extract_prompts_loop(prompts, num_loop):ret_prompts = []for prompt in prompts:@@ -204,8 +315,11 @@ def apply_mask_strategy(z, refs_x, mask_strategys, loop_i, align=None):return masks-def append_generated(vae, generated_video, refs_x, mask_strategy, loop_i, condition_frame_length, condition_frame_edit):- ref_x = vae.encode(generated_video)+def append_generated(+ vae, generated_video, refs_x, mask_strategy, loop_i, condition_frame_length, condition_frame_edit, is_latent=False+):+ ref_x = vae.encode(generated_video) if not is_latent else generated_video+for j, refs in enumerate(refs_x):if refs is None:refs_x[j] = [ref_x[j]]@@ -228,19 +342,20 @@ def dframe_to_frame(num):OPENAI_CLIENT = NoneREFINE_PROMPTS = None-REFINE_PROMPTS_PATH = "assets/texts/t2v_pllava.txt"+# REFINE_PROMPTS_PATH = "assets/texts/t2v_pllava.txt"+REFINE_PROMPTS_PATH = "assets/texts/t2v_demo.txt"REFINE_PROMPTS_TEMPLATE = """You need to refine user's input prompt. The user's input prompt is used for video generation task. You need to refine the user's prompt to make it more suitable for the task. Here are some examples of refined prompts:{}-The refined prompt should pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. The refined prompt should be in English.+The refined prompt should pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. The video should not have any scene transitions and must be on the same scene. The refined prompt should be in English."""RANDOM_PROMPTS = NoneRANDOM_PROMPTS_TEMPLATE = """You need to generate one input prompt for video generation task. The prompt should be suitable for the task. Here are some examples of refined prompts:{}-The prompt should pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. The prompt should be in English.+The prompt should pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. The video should not have any scene transitions and must be on the same scene. The prompt should be in English."""@@ -309,6 +424,128 @@ def refine_prompts_by_openai(prompts):return new_prompts+FIRST_FRAME_PROMPT_TEMPLATE_WITH_INFO = """+Given the first frame of the video, describe this video and its style in a very detailed manner. Some information about the video is:+'{}'.++Describe the video and its style in a very detailed manner. Pay attention to all objects in the video. You must describe what the human character is doing with action in the video, for instance, talk, walk, blink, laugh, sing or anything else that involves movements in the video. Your description must make it easy for this vide to have human movements, instead of being motionless. The description should be useful for AI to re-generate the video. The description should be no more than six sentences.++"""++FIRST_FRAME_PROMPT_TEMPLATE = """+Given the first frame of the video, you need to generate one input prompt for video generation task. The prompt should be suitable for generating a video using the given image as the first frame.++Describe the video and its style in a very detailed manner. Pay attention to all objects in the video. You must describe what the human character is doing with action in the video, for instance, talk, walk, blink, laugh, sing or anything else that involves movements in the video. Your description must make it easy for this vide to have human movements, instead of being motionless. The description should be useful for AI to re-generate the video. The description should be no more than six sentences.+"""+++def to_base64(image_tensor):+ buffer = BytesIO()+ pil_image = transforms.ToPILImage()(image_tensor)+ pil_image.save(buffer, format="JPEG")+ return base64.b64encode(buffer.getvalue()).decode("utf-8")+++LLAVA_PREFIX = [+ "The video shows ",+ "The video captures ",+ "The video features ",+ "The video depicts ",+ "The video presents ",+ "The video features ",+ "The video is ",+ "In the video, ",+ "The image shows ",+ "The image captures ",+ "The image features ",+ "The image depicts ",+ "The image presents ",+ "The image features ",+ "The image is ",+ "The image portrays ",+ "In the image, ",+]+++def remove_caption_prefix(caption):+ for prefix in LLAVA_PREFIX:+ if caption.startswith(prefix) or caption.startswith(prefix.lower()):+ caption = caption[len(prefix) :].strip()+ if caption[0].islower():+ caption = caption[0].upper() + caption[1:]+ return caption+ return caption+++def get_caption(frame, prompt):+ global OPENAI_CLIENT+ if OPENAI_CLIENT is None:+ from openai import OpenAI++ OPENAI_CLIENT = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))++ response = client.chat.completions.create(+ model="gpt-4o-2024-08-06",+ messages=[+ {"role": "system", "content": prompt},+ {+ "role": "user",+ "content": [+ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame}", "detail": "low"}},+ ],+ },+ ],+ max_tokens=300,+ top_p=0.1,+ )+ caption = response.choices[0].message.content+ caption = caption.replace("\n", " ")+ caption = remove_caption_prefix(caption).replace(" image ", " video ")+ return caption+++def refine_batched_prompts_with_images(prompts, images):+ new_prompts = []+ for prompt, image in zip(prompts, images):+ try:+ if prompt.strip() == "":+ new_prompt = get_random_prompt_with_image(image)+ print(f"[Info] Empty prompt detected, generate random prompt: {new_prompt}")+ else:+ new_prompt = refine_prompt_with_image(prompt, image)+ print(f"[Info] Refine prompt: {prompt} -> {new_prompt}")+ new_prompts.append(new_prompt)+ except Exception as e:+ print(f"[Warning] Failed to refine prompt: {prompt} due to {e}")+ new_prompts.append(prompt)+ return new_prompts+++def refine_prompt_with_image(prompt, image):+ # check api keys+ if has_openai_key():+ os.environ.get("OPENAI_API_KEY")+ else:+ print("no openai api key found, prompt not refined")+ return prompt+ frame = to_base64(image)+ caption = get_caption(frame, FIRST_FRAME_PROMPT_TEMPLATE_WITH_INFO.format(prompt))+ return caption+++def get_random_prompt_with_image(image):+ # check api keys+ if has_openai_key():+ os.environ.get("OPENAI_API_KEY")+ else:+ print("no openai api key found, prompt not refined")+ return prompt++ frame = to_base64(image)+ caption = get_caption(frame, FIRST_FRAME_PROMPT_TEMPLATE)+ return caption++def add_watermark(input_video_path, watermark_image_path="./assets/images/watermark/watermark.png", output_video_path=None):@@ -320,3 +557,205 @@ def add_watermark(exit_code = os.system(cmd)is_success = exit_code == 0return is_success+++def super_resolution(input_video_path, sr=2):+ temp_dir = tempfile.TemporaryDirectory()+ cmd = f"python -m tools.repair.inference_realesrgan_video -n RealESRGAN_x4plus -s 2 -i {input_video_path} -o {temp_dir.name}"+ os.system(cmd)+ output_video_path = os.path.join(temp_dir.name, os.path.basename(input_video_path).split(".")[0] + "_out.mp4")+ dst_video_path = os.path.join(+ os.path.dirname(input_video_path), os.path.basename(input_video_path).split(".")[0] + f"_sr{sr:.0f}.mp4"+ )+ shutil.copyfile(output_video_path, dst_video_path)+ temp_dir.cleanup()+ return dst_video_path+++def deflicker_video_local_brightness(input_video_path, output_video_path, smoothing_window=30, block_size=32):+ # 打开输入视频+ cap = cv2.VideoCapture(input_video_path)++ # 获取视频的基本信息+ width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))+ height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))+ fps = cap.get(cv2.CAP_PROP_FPS)+ total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))++ # 设置输出视频的编解码器和格式+ fourcc = cv2.VideoWriter_fourcc(*"h264")+ out = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))++ # 初始化存储每帧局部亮度信息的数组+ local_brightness_list = []++ # 读取视频帧并计算局部亮度+ frames = []+ while True:+ ret, frame = cap.read()+ if not ret:+ break+ frames.append(frame)++ # 转换为灰度图像+ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)++ # 将图像划分为小块,并计算每块的平均亮度+ local_brightness = []+ for y in range(0, height, block_size):+ for x in range(0, width, block_size):+ block = gray[y : y + block_size, x : x + block_size]+ mean_brightness = np.mean(block)+ local_brightness.append(mean_brightness)++ local_brightness_list.append(local_brightness)++ # 将局部亮度进行平滑处理+ local_brightness_array = np.array(local_brightness_list)++ # 在前后添加填充+ assert total_frames % 2 == 1, "The number of frames should be odd."+ pad_width = smoothing_window // 2+ padded_local_brightness = np.pad(local_brightness_array, ((pad_width, pad_width - 1), (0, 0)), mode="edge")++ # 创建一个存储平滑亮度的数组,大小与local_brightness_array相同+ smoothed_local_brightness = np.zeros_like(local_brightness_array)++ # 对每个局部块的亮度进行平滑处理+ for i in range(local_brightness_array.shape[1]):+ # 使用卷积进行平滑处理+ smoothed_brightness = np.convolve(+ padded_local_brightness[:, i], np.ones(smoothing_window) / smoothing_window, mode="valid"+ )+ smoothed_local_brightness[:, i] = smoothed_brightness++ # 对每一帧应用局部亮度校正+ for i in range(len(frames)):+ frame = frames[i].copy()+ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)++ # 逐块调整亮度+ idx = 0+ for y in range(0, height, block_size):+ for x in range(0, width, block_size):+ block = gray[y : y + block_size, x : x + block_size]+ current_brightness = np.mean(block)+ brightness_ratio = smoothed_local_brightness[i, idx] / current_brightness+ frame[y : y + block_size, x : x + block_size] = np.clip(+ frame[y : y + block_size, x : x + block_size] * brightness_ratio, 0, 255+ ).astype(np.uint8)+ idx += 1++ out.write(frame)++ # 释放资源+ cap.release()+ out.release()+ print(f"Deflickered video saved as {output_video_path}")+++def deflicker(input_video_path):+ deflicker_video_local_brightness(input_video_path, input_video_path.replace(".mp4", "_deflicker.mp4"))+ return input_video_path+++GIGABYTE = 1024**3+++def print_memory_usage(prefix: str, device: torch.device):+ torch.cuda.synchronize()+ max_memory_allocated = torch.cuda.max_memory_allocated(device)+ max_memory_reserved = torch.cuda.max_memory_reserved(device)+ print(f"{prefix}: max memory allocated: {max_memory_allocated / GIGABYTE:.4f} GB")+ print(f"{prefix}: max memory reserved: {max_memory_reserved / GIGABYTE:.4f} GB")+++def easy_data(csv):+ from opensora.registry import DATASETS, build_module++ dataset = build_module(+ {+ "type": "VariableVideoTextDataset",+ "transform_name": "resize_crop",+ "data_path": csv,+ },+ DATASETS,+ )++ return dataset["0-113-360-640"]+++def prep_ref_and_mask(cond_type, condition_frame_length, refs, target_shape, loop, device, dtype):+ """+ prepare the mask_index and reference for the 1st loop+ Input:+ loop: total number of loops to do+ """+ latent_t = target_shape[2]++ if cond_type is None:+ mask_index = []++ elif cond_type == "v2v_head":+ min_ref_length = min([ref[0].shape[1] for ref in refs])+ condition_frame_length = min(+ min(condition_frame_length, min_ref_length), latent_t+ ) # ensure condition frame is no more than generated length+ mask_index = [i for i in range(condition_frame_length)]+ elif cond_type == "i2v_head" or cond_type == "i2v_loop":+ mask_index = [0] # update mask on last frame lfor i2v_loop+ elif cond_type == "i2v_tail":+ if loop == 1:+ mask_index = [-1] # update mask to be positive later+ else:+ mask_index = [] # cond on last frame in final loop+ else:+ raise NotImplementedError++ # prep ref in correct shape+ ref = torch.zeros(target_shape, device=device, dtype=dtype)++ if len(mask_index) > 0:+ b = target_shape[0]+ for b_i in range(b):+ ref[b_i, :, mask_index] = refs[b_i][0][:, mask_index].unsqueeze(0)++ # get finalized mask_index, except i2v_tail and i2v_loop intermediate loops will update later+ if cond_type == "i2v_loop" and loop <= 1:+ b = target_shape[0]+ for b_i in range(b):+ if len(refs[0]) == 1: # if only 1 ref, use last frame+ ref[b_i, :, -1] = refs[b_i][0][:, -1].unsqueeze(0)+ else:+ ref[b_i, :, -1] = refs[b_i][1][:, 0].unsqueeze(0) # CHANGED TO USE IMAGE+ mask_index.append(latent_t - 1)+ if cond_type == "i2v_tail" and loop <= 1:+ mask_index = [latent_t - 1]++ return ref, mask_index+++def prep_ref_and_update_mask_in_loop(+ cond_type, condition_frame_length, samples, refs, target_shape, is_last_loop, device, dtype+):+ latent_t = target_shape[2]+ # cond frames from last generation+ loop_cond_index = [i for i in range(-condition_frame_length, 0)]++ # get ref in correct shape+ ref = torch.zeros(target_shape, device=device, dtype=dtype)+ ref_cut = samples[:, :, loop_cond_index].to(device=device, dtype=dtype)+ mask_index = [i for i in range(condition_frame_length)]+ ref[:, :, mask_index] = ref_cut++ if cond_type == "i2v_loop" or cond_type == "i2v_tail" and is_last_loop:+ b = target_shape[0]+ for b_i in range(b):+ if len(refs[b_i]) == 1: # if only 1 reference passed, use last frame+ ref[b_i, :, -1] = refs[b_i][0][:, -1].unsqueeze(0).to(device=device, dtype=dtype)+ else: # use the last frame (either video or image) of second reference+ ref[b_i, :, -1] = refs[b_i][1][:, 0].unsqueeze(0).to(device=device, dtype=dtype)++ mask_index.append(latent_t - 1) # mask_index for final loop++ return ref, mask_indexdiff --git a/opensora/utils/lr_scheduler.py b/opensora/utils/lr_scheduler.pyindex e0f75f5..0de60d7 100644--- a/opensora/utils/lr_scheduler.py+++ b/opensora/utils/lr_scheduler.py@@ -11,12 +11,13 @@ class LinearWarmupLR(_LRScheduler):the schedule is started from the beginning or When last_step=-1, sets initial lr as lr."""- def __init__(self, optimizer, warmup_steps: int = 0, last_epoch: int = -1):+ def __init__(self, optimizer, initial_lr=0, warmup_steps: int = 0, last_epoch: int = -1):+ self.initial_lr = initial_lrself.warmup_steps = warmup_stepssuper().__init__(optimizer, last_epoch=last_epoch)def get_lr(self):if self.last_epoch < self.warmup_steps:- return [(self.last_epoch + 1) / (self.warmup_steps + 1) * lr for lr in self.base_lrs]+ return [self.initial_lr + (self.last_epoch + 1) / (self.warmup_steps + 1) * lr for lr in self.base_lrs]else:return self.base_lrsdiff --git a/opensora/utils/train_utils.py b/opensora/utils/train_utils.pyindex e542686..4b2d2c9 100644--- a/opensora/utils/train_utils.py+++ b/opensora/utils/train_utils.py@@ -4,7 +4,9 @@ from collections import OrderedDictimport torchimport torch.distributed as dist+import torch.nn.functional as Ffrom colossalai.booster.plugin import LowLevelZeroPlugin+from einops import rearrangefrom opensora.acceleration.parallel_states import set_data_parallel_group, set_sequence_parallel_groupfrom opensora.acceleration.plugin import ZeroSeqParallelPlugin@@ -13,25 +15,29 @@ from .misc import get_loggerdef create_colossalai_plugin(plugin, dtype, grad_clip, sp_size, reduce_bucket_size_in_m: int = 20):- if plugin == "zero2":- assert sp_size == 1, "Zero2 plugin does not support sequence parallelism"+ plugin_kwargs = dict(+ precision=dtype,+ initial_scale=2**16,+ max_norm=grad_clip,+ reduce_bucket_size_in_m=reduce_bucket_size_in_m,+ overlap_allgather=True,+ cast_inputs=False,+ )+ if plugin == "zero1" or plugin == "zero2":+ assert sp_size == 1, "Zero plugin does not support sequence parallelism"+ stage = 1 if plugin == "zero1" else 2plugin = LowLevelZeroPlugin(- stage=2,- precision=dtype,- initial_scale=2**16,- max_norm=grad_clip,- reduce_bucket_size_in_m=reduce_bucket_size_in_m,+ stage=stage,+ **plugin_kwargs,)set_data_parallel_group(dist.group.WORLD)- elif plugin == "zero2-seq":- assert sp_size > 1, "Zero2-seq plugin requires sequence parallelism"+ elif plugin == "zero1-seq" or plugin == "zero2-seq":+ assert sp_size > 1, "Zero-seq plugin requires sequence parallelism"+ stage = 1 if plugin == "zero1-seq" else 2plugin = ZeroSeqParallelPlugin(sp_size=sp_size,- stage=2,- precision=dtype,- initial_scale=2**16,- max_norm=grad_clip,- reduce_bucket_size_in_m=reduce_bucket_size_in_m,+ stage=stage,+ **plugin_kwargs,)set_sequence_parallel_group(plugin.sp_group)set_data_parallel_group(plugin.dp_group)@@ -166,3 +172,120 @@ class MaskGenerator:masks.append(mask)masks = torch.stack(masks, dim=0)return masks+++def class_dropout(text_list, drop_ratio):+ # replace text with "" in text_list with probability drop_ratio+ text_ret = []+ for i in range(len(text_list)):+ if random.random() < drop_ratio:+ text_ret.append("")+ else:+ text_ret.append(text_list[i])+ return text_ret+++# Noise augmentation+++def add_noise(x, noise_level: float):+ noise = torch.randn_like(x)+ x_noise = (1 - noise_level) * x + noise_level * noise+ return x_noise+++def downsample_spatial(x, downsample_ratio: int):+ assert downsample_ratio in [1, 2, 4, 8], f"downsample_ratio should be one of [1, 2, 4, 8], got {downsample_ratio}"+ bs = x.shape[0]+ x = rearrange(x, "b c t h w -> (b t) c h w")++ downsampled_image = F.interpolate(x, scale_factor=1 / downsample_ratio, mode="bilinear", align_corners=False)+ upsampled_image = F.interpolate(downsampled_image, size=x.shape[2:], mode="bilinear", align_corners=False)++ x = rearrange(upsampled_image, "(b t) c h w -> b c t h w", b=bs)++ return x+++def downsample_temporal(x, downsample_ratio: int):+ assert downsample_ratio in [1, 2, 4, 8], f"downsample_ratio should be one of [1, 2, 4, 8], got {downsample_ratio}"++ downsampled_video = F.interpolate(+ x,+ scale_factor=(1 / downsample_ratio, 1, 1),+ mode="trilinear",+ align_corners=False,+ )+ upsampled_video = F.interpolate(+ downsampled_video,+ size=x.shape[2:],+ mode="trilinear",+ align_corners=False,+ )++ return upsampled_video+++# noise_injection_prob = cfg.get("noise_injection_prob", 0.0)+# if noise_injection_prob > 0 and random.random() < noise_injection_prob:+# noise_upper_bound = cfg.get("noise_upper_bound", 0.3)+++def aug_x(x, vae, prob_dict, strength_dict):+ x_gt = vae.encode(x)++ # downsample_spatial+ if random.random() < prob_dict.get("downsample_spatial", 0.0):+ st = random.choice(strength_dict["downsample_spatial"])+ x = downsample_spatial(x, st)++ # downsample_temporal+ if random.random() < prob_dict.get("downsample_temporal", 0.0):+ st = random.choice(strength_dict["downsample_temporal"])+ x = downsample_temporal(x, st)++ # gaussian_pixel+ if random.random() < prob_dict.get("gaussian_pixel", 0.0):+ st = random.random() * strength_dict["gaussian_pixel"]+ x = add_noise(x, st)++ x = vae.encode(x)++ # gaussian_feature+ if random.random() < prob_dict.get("gaussian_feature", 0.0):+ st = random.random() * strength_dict["gaussian_feature"]+ x = add_noise(x, st)++ return x, x_gt+++def get_mask_cond(randgen, mask_types) -> str:+ mask_cond = randgen.choices(list(mask_types.keys()), weights=list(mask_types.values()), k=1)[0]+ return mask_cond+++def get_mask_index(mask_cond, latent_t):+ if mask_cond == "v2v_head" or mask_cond == "v2v_head_noisy":+ mask_index = [k for k in range(latent_t // 2)]+ elif mask_cond == "v2v_tail":+ mask_index = [k for k in range(latent_t // 2, latent_t)]+ elif mask_cond == "i2v" or mask_cond == "i2v_head": # equivalent+ mask_index = [0]+ elif mask_cond == "i2v_loop":+ mask_index = [0, latent_t - 1]+ elif mask_cond == "i2v_tail":+ mask_index = [latent_t - 1]+ elif mask_cond == "other" or mask_cond == "other_noisy":+ edge = random.choices([0, latent_t - 1], k=1)[0]+ if edge == 0:+ mask_index = [+ k for k in range(random.randint(1, latent_t - 2))+ ] # IMPORTANT: don't allow full mask as not useful and x_pred will have zero T size+ else:+ mask_index = [k for k in range(random.randint(1, latent_t - 2), latent_t)]+ elif mask_cond == "none":+ mask_index = []+ else:+ raise NotImplementedError++ return mask_indexdiff --git a/pyproject.toml b/pyproject.tomldeleted file mode 100644index 8cb3c76..0000000--- a/pyproject.toml+++ /dev/null@@ -1,16 +0,0 @@-[tool.autoflake]-remove-unused-variables = true-remove-all-unused-imports = true-ignore-init-module-imports = true--[tool.isort]-line_length = 120-multi_line_output = 3-include_trailing_comma = true-ignore_comments = true-profile = "black"-honor_noqa = true--[tool.black]-line-length = 120-target-version = ["py37", "py38", "py39", "py310"]diff --git a/requirements/requirements.txt b/requirements/requirements.txtindex 92d9620..20a3fa0 100644--- a/requirements/requirements.txt+++ b/requirements/requirements.txt@@ -1,10 +1,10 @@-colossalai>=0.4.0+colossalai>=0.4.1mmengine>=0.10.3pandas>=2.0.3timm==0.9.16rotary_embedding_torch==0.5.3ftfy>=6.2.0 # for t5-diffusers==0.27.2 # for vae+diffusers==0.29.0 # for vaeaccelerate==0.29.2 # for t5av>=12.0.0 # for video loadingnumpy<2.0.0diff --git a/scripts/inference.py b/scripts/inference.pyindex c4578a7..2d6eace 100644--- a/scripts/inference.py+++ b/scripts/inference.py@@ -21,15 +21,19 @@ from opensora.utils.inference_utils import (append_score_to_prompts,apply_mask_strategy,collect_references_batch,- dframe_to_frame,+ deflicker,extract_json_from_prompts,extract_prompts_loop,get_save_path_name,load_prompts,merge_prompt,+ prep_ref_and_mask,+ prep_ref_and_update_mask_in_loop,prepare_multi_resolution_info,+ print_memory_usage,refine_prompts_by_openai,split_prompt,+ super_resolution,)from opensora.utils.misc import all_exists, create_logger, is_distributed, is_main_process, to_torch_dtype@@ -121,6 +125,7 @@ def main():prompts = [cfg.get("prompt_generator", "")] * 1_000_000 # endless loop# == prepare reference ==+ neg_prompts = cfg.get("neg_prompt", [None] * len(prompts))reference_path = cfg.get("reference_path", [""] * len(prompts))mask_strategy = cfg.get("mask_strategy", [""] * len(prompts))assert len(reference_path) == len(prompts), "Length of reference must be the same as prompts"@@ -142,12 +147,17 @@ def main():sample_name = cfg.get("sample_name", None)prompt_as_path = cfg.get("prompt_as_path", False)+ use_sdedit = cfg.get("use_sdedit", False)+ use_oscillation_guidance_for_text = cfg.get("use_oscillation_guidance_for_text", None)+ use_oscillation_guidance_for_image = cfg.get("use_oscillation_guidance_for_image", None)+# == Iter over all samples ==for i in progress_wrap(range(0, len(prompts), batch_size)):# == prepare batch prompts ==batch_prompts = prompts[i : i + batch_size]ms = mask_strategy[i : i + batch_size]refs = reference_path[i : i + batch_size]+ neg_prompts_batch = neg_prompts[i : i + batch_size]# == get json from prompts ==batch_prompts, refs, ms = extract_json_from_prompts(batch_prompts, refs, ms)@@ -241,6 +251,10 @@ def main():# 3. clean prompt with T5for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):batched_prompt_segment_list[idx] = [text_preprocessing(prompt) for prompt in prompt_segment_list]+ if neg_prompts_batch[0] is None:+ neg_prompts_batch_cl = None+ else:+ neg_prompts_batch_cl = [text_preprocessing(prompt) for prompt in neg_prompts_batch]# 4. merge to obtain the final promptbatch_prompts = []@@ -248,32 +262,81 @@ def main():batch_prompts.append(merge_prompt(prompt_segment_list, loop_idx_list))# == Iter over loop generation ==+ mask_index = None+ ref = None+ cond_type = cfg.get("cond_type", None)+ cond_type = None if cond_type == "none" else cond_type+ image_cfg_scale = None+ target_shape = [len(batch_prompts), vae.out_channels, *latent_size]+ if cond_type is not None: # i2v or v2v+ image_cfg_scale = cfg.get("image_cfg_scale", 7.5)+ ref, mask_index = prep_ref_and_mask(+ cond_type, condition_frame_length, refs, target_shape, loop, device, dtype+ )+video_clips = []for loop_i in range(loop):# == get prompt for loop i ==batch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)-- # == add condition frames for loop ==- if loop_i > 0:- refs, ms = append_generated(- vae, video_clips[-1], refs, ms, loop_i, condition_frame_length, condition_frame_edit- )-# == sampling ==torch.manual_seed(1024)z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)- masks = apply_mask_strategy(z, refs, ms, loop_i, align=align)+ masks = (+ apply_mask_strategy(z, refs, ms, loop_i, align=align) if mask_index is None else None+ ) # no mask for i2v and v2v+ x_cond_mask = (+ torch.zeros(len(batch_prompts), vae.out_channels, *latent_size, device=device).to(dtype)+ if mask_index is not None+ else None+ )+ if x_cond_mask is not None and mask_index is not None:+ x_cond_mask[:, :, mask_index, :, :] = 1.0+samples = scheduler.sample(model,text_encoder,z=z,+ z_cond=ref,+ z_cond_mask=x_cond_mask,prompts=batch_prompts_loop,device=device,additional_args=model_args,progress=verbose >= 2,mask=masks,+ mask_index=mask_index,+ image_cfg_scale=image_cfg_scale,+ neg_prompts=neg_prompts_batch_cl if mask_index is None else None, # no mask for i2v and v2v+ use_sdedit=use_sdedit,+ use_oscillation_guidance_for_text=use_oscillation_guidance_for_text,+ use_oscillation_guidance_for_image=use_oscillation_guidance_for_image,)- samples = vae.decode(samples.to(dtype), num_frames=num_frames)++ if loop > 1: # process conditions for subsequent loop+ if cond_type is not None: # i2v or v2v+ is_last_loop = loop_i == loop - 1+ ref, mask_index = prep_ref_and_update_mask_in_loop(+ cond_type,+ condition_frame_length,+ samples,+ refs,+ target_shape,+ is_last_loop,+ device,+ dtype,+ )+ else:+ refs, ms = append_generated(+ vae,+ samples,+ refs,+ ms,+ loop_i,+ condition_frame_length,+ condition_frame_edit,+ is_latent=True,+ )++ # samples = vae.decode(samples.to(dtype), num_frames=num_frames)video_clips.append(samples)# == save samples ==@@ -284,20 +347,36 @@ def main():save_path = save_paths[idx]video = [video_clips[i][idx] for i in range(loop)]for i in range(1, loop):- video[i] = video[i][:, dframe_to_frame(condition_frame_length) :]- video = torch.cat(video, dim=1)+ # video[i] = video[i][:, dframe_to_frame(condition_frame_length) :]+ video[i] = video[i][:, condition_frame_length:] # latent video concat+ video = torch.cat(video, dim=1) # latent [C, T, H, W]+ # ensure latent frame size is multiples of 5+ t_cut = video.size(1) // 5 * 5+ if t_cut < video.size(1):+ video = video[:, :t_cut]++ video = vae.decode(video.to(dtype), num_frames=t_cut * 17 // 5).squeeze(0)++ print("video size:", video.size())save_path = save_sample(video,fps=save_fps,save_path=save_path,verbose=verbose >= 2,)+ if save_path.endswith(".mp4") and cfg.get("deflicker", False):+ time.sleep(1)+ save_path = deflicker(save_path)+ if save_path.endswith(".mp4") and cfg.get("super_resolution", False):+ time.sleep(1)+ save_path = super_resolution(save_path, cfg.get("super_resolution"))if save_path.endswith(".mp4") and cfg.get("watermark", False):time.sleep(1) # prevent loading previous generated videoadd_watermark(save_path)start_idx += len(batch_prompts)logger.info("Inference finished.")logger.info("Saved %s samples to %s", start_idx, save_dir)+ print_memory_usage("After inference", device)if __name__ == "__main__":diff --git a/scripts/inference_i2v.py b/scripts/inference_i2v.pynew file mode 100644index 0000000..9c4f3a9--- /dev/null+++ b/scripts/inference_i2v.py@@ -0,0 +1,390 @@+import os+import time+from pprint import pformat++import colossalai+import torch+import torch.distributed as dist+from colossalai.cluster import DistCoordinator+from mmengine.runner import set_random_seed+from tqdm import tqdm++from opensora.acceleration.parallel_states import set_sequence_parallel_group+from opensora.datasets import save_sample+from opensora.datasets.aspect import get_image_size, get_num_frames+from opensora.models.text_encoder.t5 import text_preprocessing+from opensora.registry import MODELS, SCHEDULERS, build_module+from opensora.utils.config_utils import parse_configs+from opensora.utils.inference_utils import (+ add_watermark,+ append_score_to_prompts,+ apply_mask_strategy,+ collect_references_batch,+ deflicker,+ extract_images_from_ref_paths,+ extract_json_from_prompts,+ extract_prompts_loop,+ get_save_path_name,+ load_prompts,+ merge_prompt,+ prep_ref_and_mask,+ prep_ref_and_update_mask_in_loop,+ prepare_multi_resolution_info,+ print_memory_usage,+ refine_batched_prompts_with_images,+ refine_prompts_by_openai,+ split_prompt,+ super_resolution,+)+from opensora.utils.misc import all_exists, create_logger, is_distributed, is_main_process, to_torch_dtype+++def main():+ torch.set_grad_enabled(False)+ # ======================================================+ # configs & runtime variables+ # ======================================================+ # == parse configs ==+ cfg = parse_configs(training=False)++ # == device and dtype ==+ device = "cuda" if torch.cuda.is_available() else "cpu"+ cfg_dtype = cfg.get("dtype", "fp32")+ assert cfg_dtype in ["fp16", "bf16", "fp32"], f"Unknown mixed precision {cfg_dtype}"+ dtype = to_torch_dtype(cfg.get("dtype", "bf16"))+ torch.backends.cuda.matmul.allow_tf32 = True+ torch.backends.cudnn.allow_tf32 = True++ # == init distributed env ==+ if is_distributed():+ colossalai.launch_from_torch({})+ coordinator = DistCoordinator()+ enable_sequence_parallelism = coordinator.world_size > 1+ if enable_sequence_parallelism:+ set_sequence_parallel_group(dist.group.WORLD)+ else:+ coordinator = None+ enable_sequence_parallelism = False+ set_random_seed(seed=cfg.get("seed", 1024))++ # == init logger ==+ logger = create_logger()+ logger.info("Inference configuration:\n %s", pformat(cfg.to_dict()))+ verbose = cfg.get("verbose", 1)+ progress_wrap = tqdm if verbose == 1 else (lambda x: x)++ # ======================================================+ # build model & load weights+ # ======================================================+ logger.info("Building models...")+ # == build text-encoder and vae ==+ text_encoder = build_module(cfg.text_encoder, MODELS, device=device)+ vae = build_module(cfg.vae, MODELS).to(device, dtype).eval()++ # == prepare video size ==+ image_size = cfg.get("image_size", None)+ if image_size is None:+ resolution = cfg.get("resolution", None)+ aspect_ratio = cfg.get("aspect_ratio", None)+ assert (+ resolution is not None and aspect_ratio is not None+ ), "resolution and aspect_ratio must be provided if image_size is not provided"+ image_size = get_image_size(resolution, aspect_ratio)+ num_frames = get_num_frames(cfg.num_frames)++ # == build diffusion model ==+ input_size = (num_frames, *image_size)+ latent_size = vae.get_latent_size(input_size)+ model = (+ build_module(+ cfg.model,+ MODELS,+ input_size=latent_size,+ in_channels=vae.out_channels,+ caption_channels=text_encoder.output_dim,+ model_max_length=text_encoder.model_max_length,+ enable_sequence_parallelism=enable_sequence_parallelism,+ )+ .to(device, dtype)+ .eval()+ )+ text_encoder.y_embedder = model.y_embedder # HACK: for classifier-free guidance++ # == build scheduler ==+ scheduler = build_module(cfg.scheduler, SCHEDULERS)++ # ======================================================+ # inference+ # ======================================================+ # == load prompts ==+ prompts = cfg.get("prompt", None)+ start_idx = cfg.get("start_index", 0)+ obtained_ref_from_csv = False+ if prompts is None:+ if cfg.get("prompt_path", None) is not None:+ if cfg.get("csv_reference_column_name") is not None and cfg.prompt_path.endswith("csv"):+ prompts, reference_path = load_prompts(+ cfg.prompt_path,+ start_idx,+ cfg.get("end_index", None),+ csv_ref_column_name=cfg.csv_reference_column_name,+ )+ obtained_ref_from_csv = True+ else:+ prompts = load_prompts(cfg.prompt_path, start_idx, cfg.get("end_index", None))+ else:+ prompts = [cfg.get("prompt_generator", "")] * 1_000_000 # endless loop++ # == prepare reference ==+ neg_prompts = cfg.get("neg_prompt", [None] * len(prompts))++ if not obtained_ref_from_csv:+ reference_path = cfg.get("reference_path", [""] * len(prompts))++ mask_strategy = cfg.get("mask_strategy", [""] * len(prompts))+ assert len(reference_path) == len(prompts), "Length of reference must be the same as prompts"+ assert len(mask_strategy) == len(prompts), "Length of mask_strategy must be the same as prompts"++ # == prepare arguments ==+ fps = cfg.fps+ save_fps = cfg.get("save_fps", fps // cfg.get("frame_interval", 1))+ multi_resolution = cfg.get("multi_resolution", None)+ batch_size = cfg.get("batch_size", 1)+ num_sample = cfg.get("num_sample", 1)+ loop = cfg.get("loop", 1)+ condition_frame_length = cfg.get("condition_frame_length", 5)+ cfg.get("condition_frame_edit", 0.0)+ align = cfg.get("align", None)++ save_dir = cfg.save_dir+ os.makedirs(save_dir, exist_ok=True)+ sample_name = cfg.get("sample_name", None)+ prompt_as_path = cfg.get("prompt_as_path", False)++ use_sdedit = cfg.get("use_sdedit", False)+ use_oscillation_guidance_for_text = cfg.get("use_oscillation_guidance_for_text", None)+ use_oscillation_guidance_for_image = cfg.get("use_oscillation_guidance_for_image", None)++ # == Iter over all samples ==+ for i in progress_wrap(range(0, len(prompts), batch_size)):+ # == prepare batch prompts ==+ batch_prompts = prompts[i : i + batch_size]+ ms = mask_strategy[i : i + batch_size]+ batch_reference_paths = reference_path[i : i + batch_size]+ neg_prompts_batch = neg_prompts[i : i + batch_size]++ # == get json from prompts ==+ if not obtained_ref_from_csv:+ batch_prompts, batch_reference_paths, ms = extract_json_from_prompts(+ batch_prompts, batch_reference_paths, ms+ )++ original_batch_prompts = batch_prompts++ # == get reference for condition ==+ refs = collect_references_batch(batch_reference_paths, vae, image_size)++ # == multi-resolution info ==+ model_args = prepare_multi_resolution_info(+ multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype+ )++ # == Iter over number of sampling for one prompt ==+ for k in range(num_sample):+ # == prepare save paths ==+ save_paths = [+ get_save_path_name(+ save_dir,+ sample_name=sample_name,+ sample_idx=start_idx + idx,+ prompt=original_batch_prompts[idx],+ prompt_as_path=prompt_as_path,+ num_sample=num_sample,+ k=k,+ )+ for idx in range(len(batch_prompts))+ ]++ # NOTE: Skip if the sample already exists+ # This is useful for resuming sampling VBench+ if prompt_as_path and all_exists(save_paths):+ continue++ # == process prompts step by step ==+ # 0. split prompt+ # each element in the list is [prompt_segment_list, loop_idx_list]+ batched_prompt_segment_list = []+ batched_loop_idx_list = []+ for prompt in batch_prompts:+ prompt_segment_list, loop_idx_list = split_prompt(prompt)+ batched_prompt_segment_list.append(prompt_segment_list)+ batched_loop_idx_list.append(loop_idx_list)++ # 1. refine prompt by openai+ if cfg.get("llm_refine", False):+ # only call openai API when+ # 1. seq parallel is not enabled+ # 2. seq parallel is enabled and the process is rank 0+ if not enable_sequence_parallelism or (enable_sequence_parallelism and is_main_process()):+ for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):+ # batched_prompt_segment_list[idx] = refine_prompts_by_openai(prompt_segment_list)+ images = extract_images_from_ref_paths(batch_reference_paths, image_size)+ images = [img[0] for img in images]+ batched_prompt_segment_list[idx] = refine_batched_prompts_with_images(+ prompt_segment_list, images+ )+ batched_prompt_segment_list[idx] = refine_prompts_by_openai(batched_prompt_segment_list[idx])++ # sync the prompt if using seq parallel+ if enable_sequence_parallelism:+ coordinator.block_all()+ prompt_segment_length = [+ len(prompt_segment_list) for prompt_segment_list in batched_prompt_segment_list+ ]++ # flatten the prompt segment list+ batched_prompt_segment_list = [+ prompt_segment+ for prompt_segment_list in batched_prompt_segment_list+ for prompt_segment in prompt_segment_list+ ]++ # create a list of size equal to world size+ broadcast_obj_list = [batched_prompt_segment_list] * coordinator.world_size+ dist.broadcast_object_list(broadcast_obj_list, 0)++ # recover the prompt list+ batched_prompt_segment_list = []+ segment_start_idx = 0+ all_prompts = broadcast_obj_list[0]+ for num_segment in prompt_segment_length:+ batched_prompt_segment_list.append(+ all_prompts[segment_start_idx : segment_start_idx + num_segment]+ )+ segment_start_idx += num_segment++ # 2. append score+ for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):+ batched_prompt_segment_list[idx] = append_score_to_prompts(+ prompt_segment_list,+ aes=cfg.get("aes", None),+ flow=cfg.get("flow", None),+ camera_motion=cfg.get("camera_motion", None),+ )++ # 3. clean prompt with T5+ for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):+ batched_prompt_segment_list[idx] = [text_preprocessing(prompt) for prompt in prompt_segment_list]+ if neg_prompts_batch[0] is None:+ neg_prompts_batch_cl = None+ else:+ neg_prompts_batch_cl = [text_preprocessing(prompt) for prompt in neg_prompts_batch]++ # 4. merge to obtain the final prompt+ batch_prompts = []+ for prompt_segment_list, loop_idx_list in zip(batched_prompt_segment_list, batched_loop_idx_list):+ batch_prompts.append(merge_prompt(prompt_segment_list, loop_idx_list))++ # == Iter over loop generation ==+ ref = None+ cond_type = cfg.get("cond_type", None)+ image_cfg_scale = None+ image_cfg_scale = cfg.get("image_cfg_scale", 7.5)+ target_shape = [len(batch_prompts), vae.out_channels, *latent_size]+ ref_len = [len(ref) for ref in refs]+ if 0 in ref_len:+ print(f"refenrece not provided for {cond_type}, will default to t2v generation")+ cond_type = None++ ref, mask_index = prep_ref_and_mask(+ cond_type, condition_frame_length, refs, target_shape, loop, device, dtype+ )++ video_clips = []+ for loop_i in range(loop):+ # == get prompt for loop i ==+ batch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)+ # == sampling ==+ z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)+ masks = (+ apply_mask_strategy(z, refs, ms, loop_i, align=align) if len(mask_index) == 0 else None+ ) # no mask for i2v and v2v+ x_cond_mask = torch.zeros(len(batch_prompts), vae.out_channels, *latent_size, device=device).to(dtype)+ if len(mask_index) > 0:+ x_cond_mask[:, :, mask_index, :, :] = 1.0++ samples = scheduler.sample(+ model,+ text_encoder,+ z=z,+ z_cond=ref,+ z_cond_mask=x_cond_mask,+ prompts=batch_prompts_loop,+ device=device,+ additional_args=model_args,+ progress=verbose >= 2,+ mask=masks,+ mask_index=mask_index,+ image_cfg_scale=image_cfg_scale,+ neg_prompts=neg_prompts_batch_cl if len(mask_index) > 0 else None, # no mask for i2v and v2v+ use_sdedit=use_sdedit,+ use_oscillation_guidance_for_text=use_oscillation_guidance_for_text,+ use_oscillation_guidance_for_image=use_oscillation_guidance_for_image,+ )++ is_last_loop = loop_i == loop - 1+ if loop > 1 and not is_last_loop: # process conditions for subsequent loop+ ref, mask_index = prep_ref_and_update_mask_in_loop(+ cond_type,+ condition_frame_length,+ samples,+ refs,+ target_shape,+ is_last_loop,+ device,+ dtype,+ )++ video_clips.append(samples)++ # == save samples ==+ if is_main_process():+ for idx, batch_prompt in enumerate(batch_prompts):+ if verbose >= 2:+ logger.info("Prompt: %s", batch_prompt)+ save_path = save_paths[idx]+ video = [video_clips[i][idx] for i in range(loop)]+ for i in range(1, loop):+ video[i] = video[i][:, condition_frame_length:] # latent video concat+ video = torch.cat(video, dim=1) # latent [C, T, H, W]+ # ensure latent frame size is multiples of 5+ t_cut = video.size(1) // 5 * 5+ if t_cut < video.size(1):+ video = video[:, :t_cut]++ video = vae.decode(video.to(dtype), num_frames=t_cut * 17 // 5).squeeze(0)++ print("video size:", video.size())+ save_path = save_sample(+ video,+ fps=save_fps,+ save_path=save_path,+ verbose=verbose >= 2,+ )+ if save_path.endswith(".mp4") and cfg.get("deflicker", False):+ time.sleep(1)+ save_path = deflicker(save_path)+ if save_path.endswith(".mp4") and cfg.get("super_resolution", False):+ time.sleep(1)+ save_path = super_resolution(save_path, cfg.get("super_resolution"))+ if save_path.endswith(".mp4") and cfg.get("watermark", False):+ time.sleep(1) # prevent loading previous generated video+ add_watermark(save_path)+ start_idx += len(batch_prompts)+ logger.info("Inference finished.")+ logger.info("Saved %s samples to %s", start_idx, save_dir)+ print_memory_usage("After inference", device)+++if __name__ == "__main__":+ main()diff --git a/scripts/inference_opensoravae_v1_3.py b/scripts/inference_opensoravae_v1_3.pynew file mode 100644index 0000000..49390cc--- /dev/null+++ b/scripts/inference_opensoravae_v1_3.py@@ -0,0 +1,243 @@+import json+import os+import resource+from pprint import pformat++import colossalai+import torch+from mmengine.runner import set_random_seed+from tqdm import tqdm++from opensora.acceleration.parallel_states import get_data_parallel_group+from opensora.datasets import save_sample+from opensora.datasets.dataloader import prepare_dataloader+from opensora.models.vae_v1_3.losses import VAELoss+from opensora.registry import DATASETS, MODELS, build_module+from opensora.utils.config_utils import parse_configs+from opensora.utils.misc import (+ create_logger,+ format_numel_str,+ get_model_numel,+ get_world_size,+ is_distributed,+ is_main_process,+ to_torch_dtype,+)+++def get_memory_usage():+ rusage = resource.getrusage(resource.RUSAGE_SELF)+ return rusage.ru_maxrss # Memory usage in kilobytes+++def main():+ torch.set_grad_enabled(False)+ # ======================================================+ # configs & runtime variables+ # ======================================================+ # == parse configs ==+ cfg = parse_configs(training=False)++ # == device and dtype ==+ device = "cuda" if torch.cuda.is_available() else "cpu"+ cfg_dtype = cfg.get("dtype", "fp32")+ assert cfg_dtype in ["fp16", "bf16", "fp32"], f"Unknown mixed precision {cfg_dtype}"+ dtype = to_torch_dtype(cfg.get("dtype", "bf16"))+ torch.backends.cuda.matmul.allow_tf32 = True+ torch.backends.cudnn.allow_tf32 = True++ # == init distributed env ==+ if is_distributed():+ colossalai.launch_from_torch({})+ set_random_seed(seed=cfg.get("seed", 1024))++ # == init logger ==+ logger = create_logger()+ logger.info("Inference configuration:\n %s", pformat(cfg.to_dict()))+ verbose = cfg.get("verbose", 1)++ # ======================================================+ # build dataset and dataloader+ # ======================================================+ logger.info("Building reconstruction dataset...")+ dataset = build_module(cfg.dataset, DATASETS)+ batch_size = cfg.get("batch_size", 1)+ dataloader, _ = prepare_dataloader(+ dataset,+ batch_size=batch_size,+ num_workers=cfg.get("num_workers", 4),+ shuffle=False,+ drop_last=False,+ pin_memory=True,+ process_group=get_data_parallel_group(),+ )+ logger.info("Dataset %s contains %s videos.", cfg.dataset.data_path, len(dataset))+ total_batch_size = batch_size * get_world_size()+ logger.info("Total batch size: %s", total_batch_size)++ total_steps = len(dataloader)+ if cfg.get("num_samples", None) is not None:+ total_steps = min(int(cfg.num_samples // cfg.batch_size), total_steps)+ logger.info("limiting test dataset to %s", int(cfg.num_samples // cfg.batch_size) * cfg.batch_size)+ dataiter = iter(dataloader)++ # ======================================================+ # build model & loss+ # ======================================================+ logger.info("Building models...")+ model = build_module(cfg.model, MODELS).to(device, dtype).eval()+ model_numel, model_numel_trainable = get_model_numel(model)+ print("model parameters:", format_numel_str(model_numel))+ max_mem_alloc = torch.cuda.max_memory_allocated(device) / (1024**3)+ max_mem_resv = torch.cuda.max_memory_reserved(device) / (1024**3)+ logger.info("After building CausalVAE model; Max memory allocated: %.4f GiB", max_mem_alloc)+ logger.info("After building CausalVAE model; Max memory reserved: %.4f GiB", max_mem_resv)++ cal_stats = cfg.get("cal_stats", None)+ if cal_stats:+ # == setup loss functions ==++ vae_loss_fn = (+ VAELoss(**cfg.vae_loss_config, device=device, dtype=dtype) if cfg.get("vae_loss_config", None) else None+ )++ # # 显存开销+ # mod_name_mapping = {}+ # for name, mod in model.named_modules():+ # mod_name_mapping[mod] = name++ # def hook(module, input, output):+ # print(f"{mod_name_mapping[module]}: {torch.cuda.memory_allocated() / 1024 ** 2} MB")++ # for mod in model.modules():+ # mod.register_forward_hook(hook)++ # ======================================================+ # inference+ # ======================================================+ # == global variables ==+ running_loss = running_gen_loss = running_disc_loss = 0.0+ loss_info = {}+ if cal_stats:+ num_samples = 0+ running_sum = running_var = 0.0+ running_sum_c_head = torch.zeros(model.out_channels, dtype=torch.float, device=device)+ running_var_c_head = torch.zeros(model.out_channels, dtype=torch.float, device=device)+ running_sum_c_tail = torch.zeros(model.out_channels, dtype=torch.float, device=device)+ running_var_c_tail = torch.zeros(model.out_channels, dtype=torch.float, device=device)++ running_sum_c = torch.zeros(model.out_channels, dtype=torch.float, device=device)+ running_var_c = torch.zeros(model.out_channels, dtype=torch.float, device=device)++ # prepare arguments+ save_fps = cfg.get("fps", 24) // cfg.get("frame_interval", 1)++ # Iter over the dataset+ with tqdm(+ range(total_steps),+ disable=not is_main_process() or verbose < 1,+ total=total_steps,+ initial=0,+ ) as pbar:+ for step in pbar:+ batch = next(dataiter)+ x = batch["video"].to(device, dtype) # [B, C, T, H, W]+ z, x_rec, posterior = model(x, is_training=False)++ # == save samples ==+ save_dir = cfg.get("save_dir", None)+ if is_main_process() and save_dir is not None:+ ori_dir = f"{save_dir}_ori"+ rec_dir = f"{save_dir}_rec"+ os.makedirs(ori_dir, exist_ok=True)+ os.makedirs(rec_dir, exist_ok=True)+ for idx, vid in enumerate(x):+ pos = step * cfg.batch_size + idx+ save_sample(vid, fps=save_fps, save_path=f"{ori_dir}/{pos:03d}", verbose=verbose >= 2)+ save_sample(x_rec[idx], fps=save_fps, save_path=f"{rec_dir}/{pos:03d}", verbose=verbose >= 2)++ if cal_stats:+ num_samples += 1+ running_sum += z.mean().item()+ running_var += (z - running_sum / num_samples).pow(2).mean().item()++ running_sum_c += z.mean(dim=(0, 2, 3, 4)).float()+ running_var_c += (+ (z - running_sum_c_head[None, :, None, None, None] / num_samples)+ .pow(2)+ .mean(dim=(0, 2, 3, 4))+ .float()+ )++ running_sum_c_head += z[:, :, :1, :, :].mean(dim=(0, 2, 3, 4)).float()+ running_var_c_head += (+ (z[:, :, :1, :, :] - running_sum_c_head[None, :, None, None, None] / num_samples)+ .pow(2)+ .mean(dim=(0, 2, 3, 4))+ .float()+ )++ running_sum_c_tail += z[:, :, 1:, :, :].mean(dim=(0, 2, 3, 4)).float()+ running_var_c_tail += (+ (z[:, :, 1:, :, :] - running_sum_c_tail[None, :, None, None, None] / num_samples)+ .pow(2)+ .mean(dim=(0, 2, 3, 4))+ .float()+ )++ if vae_loss_fn:+ # == reconstruction loss ==+ nll_loss, weighted_nll_loss, weighted_kl_loss = vae_loss_fn(x, x_rec, posterior)+ vae_loss = weighted_kl_loss + weighted_nll_loss++ running_loss = vae_loss.item() / num_samples + running_loss * ((num_samples - 1) / num_samples)++ if verbose >= 1:+ pbar.set_postfix(+ {+ "mean": running_sum / num_samples,+ "std": (running_var / num_samples) ** 0.5,+ }+ )+ if num_samples % cfg.get("log_stats_every", 100) == 0:+ logger.info(+ "[OVERALL] VAE feature per channel stats: mean %s, std %s",+ (running_sum_c / num_samples).cpu().tolist(),+ (running_var_c / num_samples).sqrt().cpu().tolist(),+ )++ logger.info(+ "[HEAD] VAE feature per channel stats: mean %s, std %s",+ (running_sum_c_head / num_samples).cpu().tolist(),+ (running_var_c_head / num_samples).sqrt().cpu().tolist(),+ )+ logger.info(+ "[TAIL] VAE feature per channel stats: mean %s, std %s",+ (running_sum_c_tail / num_samples).cpu().tolist(),+ (running_var_c_tail / num_samples).sqrt().cpu().tolist(),+ )++ if cal_stats:+ logger.info("total vae loss: %s", running_loss)+ loss_info["total_vae_loss"] = running_loss+ if cfg.model.from_pretrained:+ if os.path.isdir(cfg.model.from_pretrained):+ loss_dir = os.path.join(cfg.model.from_pretrained, "eval")+ else:+ loss_dir = os.path.join(os.path.dirname(cfg.model.from_pretrained), "eval")+ os.makedirs(loss_dir, exist_ok=True)+ output_file_path = os.path.join(+ loss_dir, "loss_" + str(cfg.num_frames) + "f_" + str(cfg.image_size[0]) + "res.json"+ )+ with open(output_file_path, "w") as outfile:+ json.dump(loss_info, outfile, indent=4, sort_keys=True)+ print(f"results saved to: {output_file_path}")++ max_mem_alloc = torch.cuda.max_memory_allocated(device) / (1024**3)+ max_mem_resv = torch.cuda.max_memory_reserved(device) / (1024**3)+ logger.info("Max memory allocated: %.4f GiB", max_mem_alloc)+ logger.info("Max memory reserved: %.4f GiB", max_mem_resv)+++if __name__ == "__main__":+ main()diff --git a/scripts/misc/merge_i2v_text_ref.py b/scripts/misc/merge_i2v_text_ref.pynew file mode 100644index 0000000..12eb144--- /dev/null+++ b/scripts/misc/merge_i2v_text_ref.py@@ -0,0 +1,26 @@+import argparse+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("--text_path", type=str)+ parser.add_argument("--ref_path", type=str)+ parser.add_argument("--output", type=str)+ args = parser.parse_args()+ return args+++args = parse_args()++with open(args.text_path, "r", encoding="utf-8") as text_f, open(args.ref_path, "r", encoding="utf-8") as ref_f, open(+ args.output, "w", encoding="utf-8"+) as f:+ texts = text_f.readlines()+ refs = ref_f.readlines()+ assert len(texts[:25]) == len(refs[:25])++ for text, ref in zip(texts, refs):+ text = text.strip()+ ref = ref.strip()+ formated_prompt = text + '{"reference_path": "' + ref + '"}'+ f.write(formated_prompt + "\n")diff --git a/scripts/misc/search_bs.py b/scripts/misc/search_bs.pyindex 8efdcca..c296143 100644--- a/scripts/misc/search_bs.py+++ b/scripts/misc/search_bs.py@@ -64,6 +64,7 @@ def main():logger.info("Building models...")# == build text-encoder and vae ==text_encoder = build_module(cfg.text_encoder, MODELS, device=device)+ cfg.dataset.tokenize_fn = text_encoder.tokenize_fnvae = build_module(cfg.vae, MODELS).to(device, dtype).eval()# == build diffusion model ==@@ -132,15 +133,6 @@ def main():def reset_optimizer():# this is essential for the first iteration after OOM- optimizer._grad_store.reset_all_gradients()- optimizer._bucket_store.reset_num_elements_in_bucket()- optimizer._bucket_store.grad_to_param_mapping = dict()- optimizer._bucket_store._grad_in_bucket = dict()- optimizer._bucket_store._param_list = []- optimizer._bucket_store._padding_size = []- for rank in range(optimizer._bucket_store._world_size):- optimizer._bucket_store._grad_in_bucket[rank] = []- optimizer._bucket_store.offset_list = [0]optimizer.zero_grad()def build_dataset(resolution, num_frames, batch_size):@@ -196,14 +188,16 @@ def main():start = time.time()x = batch.pop("video").to(device, dtype) # [B, C, T, H, W]- y = batch.pop("text")+ batch.pop("text")+ input_ids = batch.pop("input_ids")+ attention_mask = batch.pop("attention_mask")# == visual and text encoding ==with torch.no_grad():# Prepare visual inputsx = vae.encode(x) # [B, C, T, H/P, W/P]# Prepare text inputs- model_args = text_encoder.encode(y)+ model_args = text_encoder.encode(input_ids, attention_mask=attention_mask)# == mask ==mask = Nonediff --git a/scripts/misc/watermark_test.sh b/scripts/misc/watermark_test.shnew file mode 100644index 0000000..e166605--- /dev/null+++ b/scripts/misc/watermark_test.sh@@ -0,0 +1,32 @@+CKPT=$1+RES=$2+OUTPUT=$3+TIME=$4+LOG_BASE=$5++# RAND_TYPES=(随机风景镜头 随机森林镜头 随机梯田镜头 随机河流镜头 随机沼泽镜头 随机热气球镜头 随机海底镜头 随机动物镜头 随机老虎镜头 随机候鸟迁徙镜头)++# RAND_TYPES=(随机新闻播报镜头 随机直播镜头 随机社交媒体镜头 随机电视剧镜头 随机网课镜头 随机广告镜头 随机宣传片镜头 随机电影预告片镜头 随机Vlog镜头 随机MV镜头 随机电竞镜头 随机奥运会镜头 随机足球镜头 随机乒乓球镜头)++# RAND_TYPES=(随机新闻播报镜头 随机直播镜头 随机社交媒体镜头)+# RAND_TYPES=(随机电视剧镜头 随机网课镜头 随机广告镜头)+# RAND_TYPES=(随机宣传片镜头 随机电影预告片镜头 随机Vlog镜头)+# RAND_TYPES=(随机MV镜头 随机电竞镜头 随机奥运会镜头)+++RAND_REPEAT=10 # each generates 5 random samples++CMD="python scripts/inference.py configs/opensora-pro/inference/stage1.py"++if [[ -z "${OPENAI_API_KEY}" ]];+ then+ echo "Error: Required environment variable 'OPENAI_API_KEY' is not set."+ exit 1+ else+ for rand_idx in "${!RAND_TYPES[@]}"; do+ PROMPT=${RAND_TYPES[rand_idx]}+ for ((rand_count=1;rand_count<=${RAND_REPEAT};rand_count++)); do+ eval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames ${TIME} --resolution ${RES} --aspect-ratio 9:16 --sample-name ${PROMPT}_${TIME}_${RES}_${rand_count} --batch-size 1 --llm-refine True > ${LOG_BASE}/${PROMPT}_${TIME}_${RES}_${rand_count}.log 2>&1+ done+ done+fidiff --git a/scripts/train.py b/scripts/train.pyindex e41857f..4d78396 100644--- a/scripts/train.py+++ b/scripts/train.py@@ -1,5 +1,5 @@import os-import subprocess+import randomfrom contextlib import nullcontextfrom copy import deepcopyfrom datetime import timedelta@@ -10,6 +10,7 @@ import torch.distributed as distimport wandbfrom colossalai.booster import Boosterfrom colossalai.cluster import DistCoordinator+from colossalai.nn.lr_scheduler import CosineAnnealingWarmupLRfrom colossalai.nn.optimizer import HybridAdamfrom colossalai.utils import get_current_device, set_seedfrom tqdm import tqdm@@ -17,9 +18,8 @@ from tqdm import tqdmfrom opensora.acceleration.checkpoint import set_grad_checkpointfrom opensora.acceleration.parallel_states import get_data_parallel_groupfrom opensora.datasets.dataloader import prepare_dataloader-from opensora.datasets.pin_memory_cache import PinMemoryCachefrom opensora.registry import DATASETS, MODELS, SCHEDULERS, build_module-from opensora.utils.ckpt_utils import CheckpointIO, model_sharding, record_model_param_shape+from opensora.utils.ckpt_utils import load, model_gathering, model_sharding, record_model_param_shape, savefrom opensora.utils.config_utils import define_experiment_workspace, parse_configs, save_training_configfrom opensora.utils.lr_scheduler import LinearWarmupLRfrom opensora.utils.misc import (@@ -32,7 +32,14 @@ from opensora.utils.misc import (requires_grad,to_torch_dtype,)-from opensora.utils.train_utils import MaskGenerator, create_colossalai_plugin, update_ema+from opensora.utils.train_utils import (+ MaskGenerator,+ aug_x,+ create_colossalai_plugin,+ get_mask_cond,+ get_mask_index,+ update_ema,+)def main():@@ -48,16 +55,12 @@ def main():cfg_dtype = cfg.get("dtype", "bf16")assert cfg_dtype in ["fp16", "bf16"], f"Unknown mixed precision {cfg_dtype}"dtype = to_torch_dtype(cfg.get("dtype", "bf16"))- checkpoint_io = CheckpointIO()# == colossalai init distributed training ==# NOTE: A very large timeout is set to avoid some processes exit earlydist.init_process_group(backend="nccl", timeout=timedelta(hours=24))torch.cuda.set_device(dist.get_rank() % torch.cuda.device_count())set_seed(cfg.get("seed", 1024))- PinMemoryCache.force_dtype = dtype- pin_memory_cache_pre_alloc_numels = cfg.get("pin_memory_cache_pre_alloc_numels", [])- PinMemoryCache.pre_alloc_numels = pin_memory_cache_pre_alloc_numelscoordinator = DistCoordinator()device = get_current_device()@@ -76,7 +79,7 @@ def main():if coordinator.is_master():tb_writer = create_tensorboard_writer(exp_dir)if cfg.get("wandb", False):- wandb.init(project="Open-Sora", name=exp_name, config=cfg.to_dict(), dir="./outputs/wandb")+ wandb.init(project="Open-Sora", name=exp_name, config=cfg.to_dict(), dir=exp_dir)# == init ColossalAI booster ==plugin = create_colossalai_plugin(@@ -89,6 +92,16 @@ def main():booster = Booster(plugin=plugin)torch.set_num_threads(1)+ # == build text-encoder ==+ text_encoder = build_module(cfg.get("text_encoder", None), MODELS, device=device, dtype=dtype)+ if text_encoder is not None:+ text_encoder_output_dim = text_encoder.output_dim+ text_encoder_model_max_length = text_encoder.model_max_length+ cfg.dataset.tokenize_fn = text_encoder.tokenize_fn+ else:+ text_encoder_output_dim = cfg.get("text_encoder_output_dim", 4096)+ text_encoder_model_max_length = cfg.get("text_encoder_model_max_length", 300)+# ======================================================# 2. build dataset and dataloader# ======================================================@@ -98,7 +111,6 @@ def main():logger.info("Dataset contains %s samples.", len(dataset))# == build dataloader ==- cache_pin_memory = cfg.get("cache_pin_memory", False)dataloader_args = dict(dataset=dataset,batch_size=cfg.get("batch_size", None),@@ -109,7 +121,6 @@ def main():pin_memory=True,process_group=get_data_parallel_group(),prefetch_factor=cfg.get("prefetch_factor", None),- cache_pin_memory=cache_pin_memory,)dataloader, sampler = prepare_dataloader(bucket_config=cfg.get("bucket_config", None),@@ -122,14 +133,6 @@ def main():# 3. build model# ======================================================logger.info("Building models...")- # == build text-encoder and vae ==- text_encoder = build_module(cfg.get("text_encoder", None), MODELS, device=device, dtype=dtype)- if text_encoder is not None:- text_encoder_output_dim = text_encoder.output_dim- text_encoder_model_max_length = text_encoder.model_max_length- else:- text_encoder_output_dim = cfg.get("text_encoder_output_dim", 4096)- text_encoder_model_max_length = cfg.get("text_encoder_model_max_length", 300)# == build vae ==vae = build_module(cfg.get("vae", None), MODELS)@@ -165,10 +168,11 @@ def main():)# == build ema for diffusion model ==- ema = deepcopy(model).cpu().to(torch.float32)+ ema = deepcopy(model).to(torch.float32).to(device)requires_grad(ema, False)ema_shape_dict = record_model_param_shape(ema)ema.eval()+ update_ema(ema, model, decay=0, sharded=False)# == setup loss function, build scheduler ==scheduler = build_module(cfg.scheduler, SCHEDULERS)@@ -183,11 +187,18 @@ def main():)warmup_steps = cfg.get("warmup_steps", None)+ use_cosine_scheduler = cfg.get("use_cosine_scheduler", False)- if warmup_steps is None:+ if warmup_steps is None and not use_cosine_scheduler:lr_scheduler = None+ elif use_cosine_scheduler:+ lr_scheduler = CosineAnnealingWarmupLR(+ optimizer,+ total_steps=num_steps_per_epoch * cfg.get("epochs", 1000),+ warmup_steps=cfg.get("warmup_steps", 0),+ )else:- lr_scheduler = LinearWarmupLR(optimizer, warmup_steps=cfg.get("warmup_steps"))+ lr_scheduler = LinearWarmupLR(optimizer, initial_lr=1e-6, warmup_steps=cfg.get("warmup_steps"))# == additional preparation ==if cfg.get("grad_checkpoint", False):@@ -220,7 +231,7 @@ def main():# == resume ==if cfg.get("load", None) is not None:logger.info("Loading checkpoint")- ret = checkpoint_io.load(+ ret = load(booster,cfg.load,model=model,@@ -233,7 +244,12 @@ def main():start_epoch, start_step = retlogger.info("Loaded checkpoint %s at epoch %s step %s", cfg.load, start_epoch, start_step)- model_sharding(ema, device=device)+ model_sharding(ema)++ # == mask ==+ mask_types = cfg.get("mask_types", None)+ if mask_types is not None:+ mask_randgen = random.Random(dist.get_rank())# =======================================================# 5. training loop@@ -242,20 +258,26 @@ def main():timers = {}timer_keys = ["move_data",+ "mask_index","encode","mask",+ "move_args","diffusion","backward","update_ema","reduce_loss",- "optim",- "ckpt",+ "log",+ "checkpoint",]for key in timer_keys:if record_time:timers[key] = Timer(key, coordinator=coordinator)else:timers[key] = nullcontext()+ if record_time:+ record_file = open(os.path.join(exp_dir, f"record_time_r{dist.get_rank()}.txt"), "w")++ accumulation_steps = cfg.get("accumulation_steps", 1)for epoch in range(start_epoch, cfg_epochs):# == set dataloader to new epoch ==sampler.set_epoch(epoch)@@ -271,24 +293,63 @@ def main():total=num_steps_per_epoch,) as pbar:for step, batch in pbar:- # if cache_pin_memory:- # print(f"==debug== rank{dist.get_rank()} {dataloader_iter.get_cache_info()}")timer_list = []+ paths = batch.pop("path")with timers["move_data"] as move_data_t:- pinned_video = batch.pop("video")- x = pinned_video.to(device, dtype, non_blocking=True) # [B, C, T, H, W]+ x = batch.pop("video").to(device, dtype) # [B, C, T, H, W]y = batch.pop("text")+ input_ids = batch.pop("input_ids")+ attention_mask = batch.pop("attention_mask")if record_time:timer_list.append(move_data_t)+ # == prepare i2v&v2v mask_index ==+ with timers["mask_index"] as mask_index_t:+ num_frames = x.shape[2]+ latent_t = vae.get_latent_size(x.shape[2:])[0]+ mask_index = []+ text_uncond_prob = cfg.model.get("class_dropout_prob", 0.1)+ if mask_types is not None:+ mask_cond = get_mask_cond(mask_randgen, mask_types)+ if num_frames > 1: # NOTE: only use mask_indx for video+ mask_index = get_mask_index(mask_cond, latent_t)+ if len(mask_index) > 0:+ text_uncond_prob = 0.0+ if record_time:+ timer_list.append(mask_index_t)+# == visual and text encoding ==with timers["encode"] as encode_t:+ x_noisy_ref = None # for v2v, add a little noise to video's referenced partwith torch.no_grad():# Prepare visual inputsif cfg.get("load_video_features", False):x = x.to(device, dtype)+ x_gt = x+ # NOTE: x_noisy_ref is skipped for now+ elif cfg.get("noise_augmentation", False) and x.shape[2] > 1:+ x, x_gt = aug_x(+ x,+ vae,+ cfg.get("noise_prob", {}),+ cfg.get("noise_strength", {}),+ )+ # NOTE: x_noisy_ref is skipped for nowelse:- x = vae.encode(x) # [B, C, T, H/P, W/P]+ if 0 in mask_index and "noisy" in mask_cond:+ v2v_noise_min_weight = cfg.model.get("v2v_noise_min_weight", 0.1)+ v2v_noise_max_weight = cfg.model.get("v2v_noise_max_weight", 0.3)+ v2v_noise_ratio = v2v_noise_min_weight + random.uniform(0, 1) * (+ v2v_noise_max_weight - v2v_noise_min_weight+ )+ x_noisy = v2v_noise_ratio * torch.randn_like(x) + (1 - v2v_noise_ratio) * x+ else:+ x_noisy = x++ x_noisy_ref = vae.encode(x_noisy)++ x_gt = x = vae.encode(x) # [B, C, T, H/P, W/P]+# Prepare text inputsif cfg.get("load_text_features", False):model_args = {"y": y.to(device, dtype)}@@ -297,7 +358,7 @@ def main():mask = mask.to(device, dtype)model_args["mask"] = maskelse:- model_args = text_encoder.encode(y)+ model_args = text_encoder.encode(input_ids, attention_mask=attention_mask)if record_time:timer_list.append(encode_t)@@ -311,35 +372,52 @@ def main():timer_list.append(mask_t)# == video meta info ==- for k, v in batch.items():- if isinstance(v, torch.Tensor):- model_args[k] = v.to(device, dtype)-- if cache_pin_memory:- dataloader_iter.remove_cache(pinned_video)+ with timers["move_args"] as move_args_t:+ for k, v in batch.items():+ if isinstance(v, torch.Tensor):+ model_args[k] = v.to(device, dtype)+ if record_time:+ timer_list.append(move_args_t)# == diffusion loss computation ==with timers["diffusion"] as loss_t:- loss_dict = scheduler.training_losses(model, x, model_args, mask=mask)+ if len(mask_index) > 0: # i2v and v2v training+ model_args["x_mask"] = None # Don't use any other input masks+ mask = None+ loss_dict = scheduler.training_losses(+ model,+ x,+ model_args=model_args,+ mask=mask,+ mask_index=mask_index,+ x_gt=x_gt,+ noise_disable_threshold=cfg.get("noise_disable_threshold", None),+ text_uncond_prob=text_uncond_prob,+ x_noisy_ref=x_noisy_ref,+ )if record_time:timer_list.append(loss_t)# == backward & update ==with timers["backward"] as backward_t:loss = loss_dict["loss"].mean()- booster.backward(loss=loss, optimizer=optimizer)- if record_time:- timer_list.append(backward_t)-- with timers["optim"] as optim_t:- optimizer.step()- optimizer.zero_grad()+ loss = loss / accumulation_steps+ ctx = (+ booster.no_sync(model, optimizer)+ if cfg.get("plugin", "zero2") in ("zero1", "zero1-seq") and (step + 1) % accumulation_steps != 0+ else nullcontext()+ )+ with ctx:+ booster.backward(loss=loss, optimizer=optimizer)+ if (step + 1) % accumulation_steps == 0:+ optimizer.step()+ optimizer.zero_grad()# update learning rateif lr_scheduler is not None:lr_scheduler.step()if record_time:- timer_list.append(optim_t)+ timer_list.append(backward_t)# == update EMA ==with timers["update_ema"] as ema_t:@@ -349,57 +427,57 @@ def main():# == update log info ==with timers["reduce_loss"] as reduce_loss_t:- all_reduce_mean(loss)- running_loss += loss.item()+ all_reduce_mean(loss.data)+ running_loss += loss.item() * accumulation_stepsglobal_step = epoch * num_steps_per_epoch + steplog_step += 1acc_step += 1if record_time:timer_list.append(reduce_loss_t)- # == logging ==- if coordinator.is_master() and (global_step + 1) % cfg.get("log_every", 1) == 0:- avg_loss = running_loss / log_step- # progress bar- pbar.set_postfix({"loss": avg_loss, "step": step, "global_step": global_step})- # tensorboard- tb_writer.add_scalar("loss", loss.item(), global_step)- # wandb- if cfg.get("wandb", False):- wandb_dict = {- "iter": global_step,- "acc_step": acc_step,- "epoch": epoch,- "loss": loss.item(),- "avg_loss": avg_loss,- "lr": optimizer.param_groups[0]["lr"],- }- if record_time:- wandb_dict.update(- {- "debug/move_data_time": move_data_t.elapsed_time,- "debug/encode_time": encode_t.elapsed_time,- "debug/mask_time": mask_t.elapsed_time,- "debug/diffusion_time": loss_t.elapsed_time,- "debug/backward_time": backward_t.elapsed_time,- "debug/update_ema_time": ema_t.elapsed_time,- "debug/reduce_loss_time": reduce_loss_t.elapsed_time,- }- )- wandb.log(wandb_dict, step=global_step)-- running_loss = 0.0- log_step = 0-- # == uncomment to clear ram cache ==- # if ckpt_every > 0 and (global_step + 1) % ckpt_every == 0 and coordinator.is_master():- # subprocess.run("sync && sudo sh -c \"echo 3 > /proc/sys/vm/drop_caches\"", shell=True)+ with timers["log"] as log_t:+ # == logging ==+ if coordinator.is_master() and (global_step + 1) % cfg.get("log_every", 1) == 0:+ avg_loss = running_loss / log_step+ # progress bar+ pbar.set_postfix({"loss": avg_loss, "step": step, "global_step": global_step})+ # tensorboard+ tb_writer.add_scalar("loss", loss.item() * accumulation_steps, global_step)+ # wandb+ if cfg.get("wandb", False):+ wandb_dict = {+ "iter": global_step,+ "acc_step": acc_step,+ "epoch": epoch,+ "loss": loss.item() * accumulation_steps,+ "avg_loss": avg_loss,+ "lr": optimizer.param_groups[0]["lr"],+ }+ if record_time:+ wandb_dict.update(+ {+ "debug/move_data_time": move_data_t.elapsed_time,+ "debug/encode_time": encode_t.elapsed_time,+ "debug/mask_time": mask_t.elapsed_time,+ "debug/diffusion_time": loss_t.elapsed_time,+ "debug/backward_time": backward_t.elapsed_time,+ "debug/update_ema_time": ema_t.elapsed_time,+ "debug/reduce_loss_time": reduce_loss_t.elapsed_time,+ }+ )+ wandb.log(wandb_dict, step=global_step)++ running_loss = 0.0+ log_step = 0+ if record_time:+ timer_list.append(log_t)# == checkpoint saving ==- ckpt_every = cfg.get("ckpt_every", 0)- with timers["ckpt"] as ckpt_t:+ with timers["checkpoint"] as checkpoint_t:+ ckpt_every = cfg.get("ckpt_every", 0)if ckpt_every > 0 and (global_step + 1) % ckpt_every == 0:- save_dir = checkpoint_io.save(+ model_gathering(ema, ema_shape_dict)+ save_dir = save(booster,exp_dir,model=model,@@ -411,9 +489,9 @@ def main():step=step + 1,global_step=global_step + 1,batch_size=cfg.get("batch_size", None),- ema_shape_dict=ema_shape_dict,- async_io=True,)+ if dist.get_rank() == 0:+ model_sharding(ema)logger.info("Saved checkpoint at epoch %s, step %s, global_step %s to %s",epoch,@@ -422,17 +500,18 @@ def main():save_dir,)if record_time:- timer_list.append(ckpt_t)- # uncomment below 3 lines to benchmark checkpoint- # if ckpt_every > 0 and (global_step + 1) % ckpt_every == 0:- # booster.checkpoint_io._sync_io()- # checkpoint_io._sync_io()+ timer_list.append(checkpoint_t)if record_time:- log_str = f"Rank {dist.get_rank()} | Epoch {epoch} | Step {step} | "+ total_step_time = sum([timer.elapsed_time for timer in timer_list])+ log_str = (+ f"Rank {dist.get_rank()} | Epoch {epoch} | Step {step} | Step time: {total_step_time:.3f}s | "+ )for timer in timer_list:log_str += f"{timer.name}: {timer.elapsed_time:.3f}s | "- print(log_str)-+ # print(log_str)+ log_str += f"path: {paths}"+ record_file.write(log_str + "\n")+ record_file.flush()sampler.reset()start_step = 0diff --git a/scripts/train_opensoravae_v1_3.py b/scripts/train_opensoravae_v1_3.pynew file mode 100644index 0000000..8dba224--- /dev/null+++ b/scripts/train_opensoravae_v1_3.py@@ -0,0 +1,366 @@+import os+import random+from datetime import timedelta+from pprint import pformat++import torch+import torch.distributed as dist+import wandb+from colossalai.booster import Booster+from colossalai.cluster import DistCoordinator+from colossalai.nn.optimizer import HybridAdam+from colossalai.utils import get_current_device, set_seed+from tqdm import tqdm++from opensora.acceleration.checkpoint import set_grad_checkpoint+from opensora.acceleration.parallel_states import get_data_parallel_group+from opensora.datasets.dataloader import prepare_dataloader+from opensora.models.vae_v1_3.losses import DiscriminatorLoss, GeneratorLoss, VAELoss+from opensora.registry import DATASETS, MODELS, build_module+from opensora.utils.ckpt_utils import load, save+from opensora.utils.config_utils import define_experiment_workspace, parse_configs, save_training_config+from opensora.utils.lr_scheduler import LinearWarmupLR+from opensora.utils.misc import (+ all_reduce_mean,+ create_logger,+ create_tensorboard_writer,+ format_numel_str,+ get_model_numel,+ to_torch_dtype,+)+from opensora.utils.train_utils import create_colossalai_plugin+++def main():+ # ======================================================+ # 1. configs & runtime variables+ # ======================================================+ # == parse configs ==+ cfg = parse_configs(training=True)++ # == device and dtype ==+ assert torch.cuda.is_available(), "Training currently requires at least one GPU."+ cfg_dtype = cfg.get("dtype", "bf16")+ assert cfg_dtype in ["fp16", "bf16"], f"Unknown mixed precision {cfg_dtype}"+ dtype = to_torch_dtype(cfg.get("dtype", "bf16"))++ # == colossalai init distributed training ==+ # NOTE: A very large timeout is set to avoid some processes exit early+ dist.init_process_group(backend="nccl", timeout=timedelta(hours=24))+ torch.cuda.set_device(dist.get_rank() % torch.cuda.device_count())+ set_seed(cfg.get("seed", 1024))+ coordinator = DistCoordinator()+ device = get_current_device()++ # == init exp_dir ==+ exp_name, exp_dir = define_experiment_workspace(cfg)+ coordinator.block_all()+ if coordinator.is_master():+ os.makedirs(exp_dir, exist_ok=True)+ save_training_config(cfg.to_dict(), exp_dir)+ coordinator.block_all()++ # == init logger, tensorboard & wandb ==+ logger = create_logger(exp_dir)+ logger.info("Experiment directory created at %s", exp_dir)+ logger.info("Training configuration:\n %s", pformat(cfg.to_dict()))+ if coordinator.is_master():+ tb_writer = create_tensorboard_writer(exp_dir)+ if cfg.get("wandb", False):+ wandb.init(project="causalvae", name=exp_name, config=cfg.to_dict(), dir="./outputs/wandb")++ # == init ColossalAI booster ==+ plugin = create_colossalai_plugin(+ plugin=cfg.get("plugin", "zero2"),+ dtype=cfg_dtype,+ grad_clip=cfg.get("grad_clip", 0),+ sp_size=cfg.get("sp_size", 1),+ )+ booster = Booster(plugin=plugin)++ # ======================================================+ # 2. build dataset and dataloader+ # ======================================================+ logger.info("Building dataset...")+ # == build dataset ==+ assert cfg.dataset.type == "VideoTextDataset", "Only support VideoTextDataset for vae training"+ dataset = build_module(cfg.dataset, DATASETS)+ logger.info("Dataset contains %s samples.", len(dataset))++ # == build dataloader ==+ dataloader_args = dict(+ dataset=dataset,+ batch_size=cfg.batch_size,+ num_workers=cfg.get("num_workers", 4),+ seed=cfg.get("seed", 1024),+ shuffle=True,+ drop_last=True,+ pin_memory=True,+ process_group=get_data_parallel_group(),+ )+ dataloader, sampler = prepare_dataloader(**dataloader_args)+ total_batch_size = cfg.batch_size * dist.get_world_size() // cfg.get("sp_size", 1)+ logger.info("Total batch size: %s", total_batch_size)+ num_steps_per_epoch = len(dataloader)++ # ======================================================+ # 3. build model+ # ======================================================+ logger.info("Building models...")+ # == build vae model ==+ model = build_module(cfg.model, MODELS).to(device, dtype).train()+ model_numel, model_numel_trainable = get_model_numel(model)+ logger.info(+ "[VAE] Trainable model params: %s, Total model params: %s",+ format_numel_str(model_numel_trainable),+ format_numel_str(model_numel),+ )++ # == build discriminator model ==+ use_discriminator = cfg.get("discriminator", None) is not None+ if use_discriminator:+ discriminator = build_module(cfg.discriminator, MODELS).to(device, dtype).train()+ discriminator_numel, discriminator_numel_trainable = get_model_numel(discriminator)+ logger.info(+ "[Discriminator] Trainable model params: %s, Total model params: %s",+ format_numel_str(discriminator_numel_trainable),+ format_numel_str(discriminator_numel),+ )++ # == setup loss functions ==+ vae_loss_fn = VAELoss(**cfg.vae_loss_config, device=device, dtype=dtype)++ if use_discriminator:+ generator_loss_fn = GeneratorLoss(**cfg.gan_loss_confg)+ discriminator_loss_fn = DiscriminatorLoss(**cfg.gan_loss_confg)++ # == setup vae optimizer ==+ optimizer = HybridAdam(+ filter(lambda p: p.requires_grad, model.parameters()),+ adamw_mode=True,+ lr=cfg.get("lr", 1e-5),+ weight_decay=cfg.get("weight_decay", 0),+ )++ warmup_steps = cfg.get("warmup_steps", None)+ if warmup_steps is None:+ lr_scheduler = None+ else:+ lr_scheduler = LinearWarmupLR(optimizer, warmup_steps=cfg.get("warmup_steps"))++ # == setup discriminator optimizer ==+ if use_discriminator:+ disc_optimizer = HybridAdam(+ filter(lambda p: p.requires_grad, discriminator.parameters()),+ adamw_mode=True,+ lr=cfg.get("lr", 1e-5),+ weight_decay=cfg.get("weight_decay", 0),+ )+ disc_lr_scheduler = None++ # == additional preparation ==+ if cfg.get("grad_checkpoint", False):+ set_grad_checkpoint(model)++ # =======================================================+ # 4. distributed training preparation with colossalai+ # =======================================================+ logger.info("Preparing for distributed training...")+ # == boosting ==+ # NOTE: we set dtype first to make initialization of model consistent with the dtype; then reset it to the fp32 as we make diffusion scheduler in fp32+ torch.set_default_dtype(dtype)+ model, optimizer, _, dataloader, lr_scheduler = booster.boost(+ model=model,+ optimizer=optimizer,+ lr_scheduler=lr_scheduler,+ dataloader=dataloader,+ )+ if use_discriminator:+ discriminator, disc_optimizer, _, _, disc_lr_scheduler = booster.boost(+ model=discriminator,+ optimizer=disc_optimizer,+ lr_scheduler=disc_lr_scheduler,+ )+ torch.set_default_dtype(torch.float)+ logger.info("Boosting model for distributed training")++ # == global variables ==+ cfg_epochs = cfg.get("epochs", 1000)+ start_epoch = start_step = log_step = sampler_start_idx = acc_step = 0+ running_loss = running_disc_loss = running_gen_loss = 0.0+ logger.info("Training for %s epochs with %s steps per epoch", cfg_epochs, num_steps_per_epoch)++ # == resume == # TODO+ if cfg.get("load", None) is not None:+ logger.info("Loading checkpoint")+ start_epoch, start_step = load(+ booster,+ cfg.load,+ model=model,+ optimizer=optimizer,+ lr_scheduler=lr_scheduler,+ sampler=sampler,+ )+ if use_discriminator and os.path.exists(os.path.join(cfg.load, "discriminator")):+ booster.load_model(discriminator, os.path.join(cfg.load, "discriminator"))+ booster.load_optimizer(disc_optimizer, os.path.join(cfg.load, "disc_optimizer"))+ dist.barrier()+ logger.info("Loaded checkpoint %s at epoch %s step %s", cfg.load, start_epoch, start_step)++ # =======================================================+ # 5. training loop+ # =======================================================+ dist.barrier()+ for epoch in range(start_epoch, cfg_epochs):+ # == set dataloader to new epoch ==+ sampler.set_epoch(epoch)+ dataiter = iter(dataloader)+ logger.info("Beginning epoch %s...", epoch)++ # == training loop in an epoch ==+ with tqdm(+ enumerate(dataiter, start=start_step),+ desc=f"Epoch {epoch}",+ disable=not coordinator.is_master(),+ total=num_steps_per_epoch,+ initial=start_step,+ ) as pbar:+ for step, batch in pbar:+ # == log config ==+ global_step = epoch * num_steps_per_epoch + step+ log_step += 1+ acc_step += 1++ # == vae encoding & decoding ===+ x = batch["video"].to(device, dtype) # [B, C, T, H, W]+ # mixed training setting+ mixed_strategy = cfg.get("mixed_strategy", None)+ if mixed_strategy == "mixed_video_image":+ if random.random() < cfg.get("mixed_image_ratio", 0.0):+ x = x[:, :, :1, :, :]+ elif mixed_strategy == "mixed_video_random":+ length = random.randint(1, x.size(2))+ x = x[:, :, :length, :, :]+ z, x_rec, posterior = model(x, is_training=True)++ # == loss initialization ==+ vae_loss = torch.tensor(0.0, device=device, dtype=dtype)+ disc_loss = torch.tensor(0.0, device=device, dtype=dtype)+ log_dict = {}++ # == reconstruction loss ==+ nll_loss, weighted_nll_loss, weighted_kl_loss = vae_loss_fn(x, x_rec, posterior)+ log_dict["kl_loss"] = weighted_kl_loss.item()+ log_dict["nll_loss"] = weighted_nll_loss.item()+ vae_loss += weighted_nll_loss + weighted_kl_loss++ # == generator loss ==+ if use_discriminator:+ fake_logits = discriminator(x_rec.contiguous())+ generator_loss = generator_loss_fn(+ fake_logits,+ nll_loss,+ model.module.get_last_layer(),+ global_step,+ is_training=model.training,+ )+ log_dict["generator_loss"] = generator_loss.item()+ vae_loss += generator_loss+ running_gen_loss += generator_loss.item()++ # == generator backward & update ==+ optimizer.zero_grad()+ booster.backward(loss=vae_loss, optimizer=optimizer)+ optimizer.step()+ all_reduce_mean(vae_loss)+ running_loss += vae_loss.item()++ # == loss: discriminator adversarial ==+ if use_discriminator:+ real_logits = discriminator(x.contiguous().detach())+ fake_logits = discriminator(x_rec.contiguous().detach())+ disc_loss = discriminator_loss_fn(+ real_logits,+ fake_logits,+ global_step,+ )+ log_dict["disc_loss"] = disc_loss.item()++ # == discriminator backward & update ==+ disc_optimizer.zero_grad()+ booster.backward(loss=disc_loss, optimizer=disc_optimizer)+ disc_optimizer.step()+ all_reduce_mean(disc_loss)+ running_disc_loss += disc_loss.item()++ # == logging ==+ if coordinator.is_master() and (global_step + 1) % cfg.get("log_every", 1) == 0:+ avg_loss = running_loss / log_step+ avg_disc_loss = running_disc_loss / log_step+ avg_gen_loss = running_gen_loss / log_step+ # progress bar+ pbar.set_postfix(+ {+ "vae_loss": avg_loss,+ "gen_loss": avg_gen_loss,+ "disc_loss": avg_disc_loss,+ "step": step,+ "global_step": global_step,+ }+ )+ # tensorboard+ tb_writer.add_scalar("loss", vae_loss.item(), global_step)+ # wandb+ if cfg.wandb:+ wandb.log(+ {+ "iter": global_step,+ "num_samples": global_step * total_batch_size,+ "epoch": epoch,+ "loss": vae_loss.item(),+ "avg_loss": avg_loss,+ **log_dict,+ },+ step=global_step,+ )+ running_loss = running_disc_loss = 0.0+ log_step = 0++ # == checkpoint saving ==+ ckpt_every = cfg.get("ckpt_every", 0)+ if ckpt_every > 0 and (global_step + 1) % ckpt_every == 0:+ save(+ booster,+ exp_dir,+ model=model,+ optimizer=optimizer,+ lr_scheduler=lr_scheduler,+ epoch=epoch,+ step=step + 1,+ global_step=global_step + 1,+ batch_size=cfg.get("batch_size", None),+ sampler=sampler,+ )++ save_dir = os.path.join(exp_dir, f"epoch{epoch}-global_step{global_step+1}")+ if use_discriminator:+ booster.save_model(discriminator, os.path.join(save_dir, "discriminator"), shard=True)+ booster.save_optimizer(+ disc_optimizer, os.path.join(save_dir, "disc_optimizer"), shard=True, size_per_shard=4096+ )+ dist.barrier()++ logger.info(+ "Saved checkpoint at epoch %s step %s global_step %s to %s",+ epoch,+ step + 1,+ global_step + 1,+ exp_dir,+ )++ sampler.reset()+ start_step = 0+++if __name__ == "__main__":+ main()diff --git a/tests/test_attn.py b/tests/test_attn.pyindex 4cf5844..930237a 100644--- a/tests/test_attn.py+++ b/tests/test_attn.py@@ -1,9 +1,12 @@+from itertools import product++import pytestimport torchfrom colossalai.accelerator import get_acceleratorfrom colossalai.utils import get_current_device-from rotary_embedding_torch import RotaryEmbedding-from opensora.models.layers.blocks import Attention+from opensora.models.layers.blocks import Attention, split_batch_cat_seq, split_seq_cat_batch+from opensora.models.layers.rotary_embedding_torch import RotaryEmbedding# B, S, H = 7488, 1, 1152# B, S, H = 32, 234, 1152@@ -27,6 +30,186 @@ def run_attn(enable_flash_attn: bool):print(f"Peak memory: {get_accelerator().max_memory_allocated() / 1024**2:.2f} MB")+def test_block_transform():+ b, h, w, c = 8, 12, 4, 3+ x = torch.randn(b, h, w, c)+ kernel_sizes = (3, 2)+ dims = (1, 2)+ num_splits = [x.size(d) // k for d, k in zip(dims, kernel_sizes)]+ y = split_seq_cat_batch(x, kernel_sizes, dims)+ z = split_batch_cat_seq(y, b, num_splits, dims)+ assert torch.equal(x, z)+++ "shape, kernel_sizes",+ [+ [(8, 12, 4, 1), (2, 2, -1)], # divisible + N<B + 3D+ [(1, 5, 2, 5), (2, 2, -1)], # undivisible + N>B + 3D+ ],+)[email protected]("shift_window", [False, True])+def test_block_attn_nd(shape, kernel_sizes, shift_window):+ hidden_size = 96+ num_heads = 4+ head_dim = hidden_size // num_heads+ rope = RotaryEmbedding(head_dim // 3).to(device=get_current_device(), dtype=torch.bfloat16)+ attn = Attention(+ hidden_size,+ num_heads,+ qkv_bias=True,+ qk_norm=True,+ enable_flash_attn=True,+ rope=rope.rotate_queries_or_keys,+ kernel_size=kernel_sizes,+ shift_window=shift_window,+ ).to(device=get_current_device(), dtype=torch.bfloat16)+ # [B, H, W, C]+ x = torch.rand(*shape, hidden_size, device=get_current_device(), dtype=torch.bfloat16).requires_grad_()+ y = attn(x)+ assert x.shape == y.shape+ loss = y.mean()+ loss.backward()+++ "shape, kernel_sizes",+ [+ [(8, 12, 4, 1), (2, 2, -1)], # divisible + N<B + 3D+ [(8, 12, 4, 6), (2, 2, -1)], # divisible + N<B + 3D+ [(8, 12, 3, 6), (2, 2, -1)], # divisible + N<B + 3D+ [(8, 90, 60, 13), (8, 8, -1)], # 480p video+ [(8, 160, 90, 1), (8, 8, -1)], # 720p image+ ],+)+def test_block_attn_3d(shape, kernel_sizes):+ hidden_size = 96+ num_heads = 4+ head_dim = hidden_size // num_heads+ rope = RotaryEmbedding(head_dim // 3).to(device=get_current_device(), dtype=torch.bfloat16)+ attn = Attention(+ hidden_size,+ num_heads,+ qkv_bias=True,+ qk_norm=True,+ enable_flash_attn=False,+ rope=rope.rotate_queries_or_keys,+ kernel_size=kernel_sizes,+ ).to(device=get_current_device(), dtype=torch.bfloat16)+ # [B, H, W, T, C]+ x = torch.rand(*shape, hidden_size, device=get_current_device(), dtype=torch.bfloat16)+ y = attn(x)++ split_size = [k if k > 0 else x.size(i + 1) for i, k in enumerate(kernel_sizes)]+ for start_indices in product(*[range(0, x.size(i + 1), s) for i, s in enumerate(split_size)]):+ piece = x[+ :,+ start_indices[0] : start_indices[0] + split_size[0],+ start_indices[1] : start_indices[1] + split_size[1],+ start_indices[2] : start_indices[2] + split_size[2],+ :,+ ]+ piece_z = attn(piece)+ piece_y = y[+ :,+ start_indices[0] : start_indices[0] + split_size[0],+ start_indices[1] : start_indices[1] + split_size[1],+ start_indices[2] : start_indices[2] + split_size[2],+ :,+ ]+ assert piece_y.shape == piece_z.shape+ assert torch.equal(+ piece_z,+ piece_y,+ )+++ "shape, kernel_sizes, kernel_sizes2",+ [+ [(2, 80, 60, 1), (8, 8, 4), (8, 8, -1)], # 720p image+ [(2, 4, 4, 6), (4, 4, -1), (8, 8, -1)], # 720p image+ ],+)+def test_block_attn_3d_var_kernel(shape, kernel_sizes, kernel_sizes2):+ hidden_size = 24+ num_heads = 2+ head_dim = hidden_size // num_heads+ rope = RotaryEmbedding(head_dim // 3).to(device=get_current_device(), dtype=torch.bfloat16)+ attn = Attention(+ hidden_size,+ num_heads,+ qkv_bias=True,+ qk_norm=True,+ enable_flash_attn=False,+ rope=rope.rotate_queries_or_keys,+ kernel_size=kernel_sizes,+ ).to(device=get_current_device(), dtype=torch.bfloat16)+ # [B, H, W, T, C]+ x = torch.rand(*shape, hidden_size, device=get_current_device(), dtype=torch.bfloat16)+ y = attn(x)+ attn2 = Attention(+ hidden_size,+ num_heads,+ qkv_bias=True,+ qk_norm=True,+ enable_flash_attn=False,+ rope=rope.rotate_queries_or_keys,+ kernel_size=kernel_sizes2,+ ).to(device=get_current_device(), dtype=torch.bfloat16)+ attn2.load_state_dict(attn.state_dict())+ y2 = attn2(x)+ assert y.shape == y2.shape+ torch.testing.assert_close(y, y2)+++def test_block_attn_3d_overlap():+ kernel_sizes = (8, 8, -1)+ hidden_size = 24+ num_heads = 2+ head_dim = hidden_size // num_heads+ rope = RotaryEmbedding(head_dim // 3).to(device=get_current_device(), dtype=torch.bfloat16)+ attn = Attention(+ hidden_size,+ num_heads,+ qkv_bias=True,+ qk_norm=True,+ enable_flash_attn=False,+ rope=rope.rotate_queries_or_keys,+ kernel_size=kernel_sizes,+ ).to(device=get_current_device(), dtype=torch.bfloat16)+ # [B, H, W, T, C]+ x = torch.rand(2, 40, 40, 6, hidden_size, device=get_current_device(), dtype=torch.bfloat16)+ y = attn(x)+ x2 = torch.rand(2, 48, 48, 6, hidden_size, device=get_current_device(), dtype=torch.bfloat16)+ x2[:, :40, :40] = x+ y2 = attn(x2)+ torch.testing.assert_close(y, y2[:, :40, :40])+++def test_block_transform_3d():+ b, h, w, t, c = 8, 12, 4, 6, 3+ x = torch.randn(b, h, w, t, c)+ kernel_sizes = (3, 2, 6)+ dims = (1, 2, 3)+ num_splits = [x.size(d) // k for d, k in zip(dims, kernel_sizes)]+ y = split_seq_cat_batch(x, kernel_sizes, dims)+ split_size = [k if k > 0 else x.size(i + 1) for i, k in enumerate(kernel_sizes)]+ for i, start_indices in enumerate(product(*[range(0, x.size(i + 1), s) for i, s in enumerate(split_size)])):+ piece = x[+ :,+ start_indices[0] : start_indices[0] + split_size[0],+ start_indices[1] : start_indices[1] + split_size[1],+ start_indices[2] : start_indices[2] + split_size[2],+ :,+ ]+ y_piece = y[i * b : (i + 1) * b]+ assert torch.equal(y_piece, piece), f"{y_piece.shape} vs {piece.shape}"++ z = split_batch_cat_seq(y, b, num_splits, dims)+ assert torch.equal(x, z)++if __name__ == "__main__":print("Use flashattn")run_attn(True)diff --git a/tests/test_seq_parallel_attention.py b/tests/test_seq_parallel_attention.pyindex 841115a..1a06eba 100644--- a/tests/test_seq_parallel_attention.py+++ b/tests/test_seq_parallel_attention.py@@ -13,24 +13,32 @@ from opensora.models.layers.blocks import ()-def run_attention(rank, world_size):+def run_attention(rank, world_size, kernel_size=None, temporal=False):# create modeltorch.manual_seed(1024)set_sequence_parallel_group(dist.group.WORLD)- seq_parallel_attention = SeqParallelAttention(dim=256, num_heads=4, qkv_bias=True, enable_flash_attn=False).cuda()+ seq_parallel_attention = SeqParallelAttention(+ dim=1152, num_heads=16, qkv_bias=True, enable_flash_attn=False, kernel_size=kernel_size, temporal=temporal+ ).cuda()+ # seq_parallel_attention = SeqParallelAttention(dim=256, num_heads=4, qkv_bias=True, enable_flash_attn=False, kernel_size=kernel_size).cuda()torch.manual_seed(1024)attention = Attention(- dim=256,- num_heads=4,+ dim=1152,+ num_heads=16,qkv_bias=True,enable_flash_attn=False,+ kernel_size=kernel_size,).cuda()# create inputstorch.manual_seed(1024)- x = torch.randn(4, 64, 256).cuda()+ # x = torch.randn(4, 64, 256).cuda()+ if kernel_size:+ x = torch.randn(3, 12, 40, 30, 1152).cuda()+ else:+ x = torch.randn(3, 16 * 8, 1152).cuda()seq_x = x.clone().detach()x.requires_grad = True@@ -38,14 +46,33 @@ def run_attention(rank, world_size):seq_x.requires_grad = Trueseq_x.retain_grad()- sub_seq_x = split_forward_gather_backward(seq_x, dist.group.WORLD, dim=1, grad_scale="down")+ if kernel_size is None and temporal is True:+ from einops import rearrange- # run model- out = attention(x)- sub_seq_out = seq_parallel_attention(sub_seq_x)- seq_out = gather_forward_split_backward(sub_seq_out, dist.group.WORLD, dim=1, grad_scale="up")+ seq_x_ = rearrange(seq_x, "B (T S) C -> B T S C", T=16, S=8)+ sub_seq_x = split_forward_gather_backward(seq_x_, dist.group.WORLD, dim=2, grad_scale="down")+ sub_seq_x = rearrange(sub_seq_x, "B T S C -> (B S) T C", T=16, S=8 // world_size)++ sub_seq_out = seq_parallel_attention(sub_seq_x)++ sub_seq_out = rearrange(sub_seq_out, "(B S) T C -> B T S C", T=16, S=8 // world_size)+ seq_out = gather_forward_split_backward(sub_seq_out, dist.group.WORLD, dim=2, grad_scale="up")+ seq_out = rearrange(seq_out, "B T S C -> B (T S) C", T=16, S=8)++ x_ = rearrange(x, "B (T S) C -> (B S) T C", T=16, S=8)+ out = attention(x_)+ out = rearrange(out, "(B S) T C -> B (T S) C", T=16, S=8)++ else:+ sub_seq_x = split_forward_gather_backward(seq_x, dist.group.WORLD, dim=1, grad_scale="down")+ sub_seq_out = seq_parallel_attention(sub_seq_x)+ seq_out = gather_forward_split_backward(sub_seq_out, dist.group.WORLD, dim=1, grad_scale="up")++ # run model+ out = attention(x)+ seq_out = seq_out.view(out.shape)- assert torch.allclose(seq_out, out, atol=1e-7), f"{seq_out}\nvs\n{out}"+ assert torch.allclose(seq_out, out, atol=1e-6), f"{seq_out.view(-1)[:10]}\nvs\n{out.view(-1)[:10]}"# run backwardseq_out.mean().backward()@@ -146,13 +173,16 @@ def run_cross_attention(rank, world_size):def run_dist(rank, world_size, port):- colossalai.launch({}, rank=rank, world_size=world_size, host="localhost", port=port)- # run_attention(rank, world_size)- run_cross_attention(rank, world_size)+ colossalai.launch(rank=rank, world_size=world_size, host="localhost", port=port)+ run_attention(rank, world_size, temporal=True)+ run_attention(rank, world_size, temporal=False)+ run_attention(rank, world_size, kernel_size=(8, 8, -1), temporal=True)+ run_attention(rank, world_size, kernel_size=(8, 8, -1), temporal=False)+ # run_cross_attention(rank, world_size)def test_seq_parallel_attention():- spawn(run_dist, nprocs=2)+ spawn(run_dist, nprocs=4)if __name__ == "__main__":diff --git a/tests/test_tiled_conv3d.py b/tests/test_tiled_conv3d.pynew file mode 100644index 0000000..852810e--- /dev/null+++ b/tests/test_tiled_conv3d.py@@ -0,0 +1,52 @@+import torch++from opensora.models.layers.tiled_conv3d import TiledConv3d++torch.backends.cudnn.benchmark = True+torch.backends.cudnn.deterministic = True+++def test_tiled_conv3d():+ data = torch.rand(1, 128, 51, 256, 256).cuda().to(torch.bfloat16)++ exclude_temporal_dim_options = [True, False]+ padding_options = [+ (0, 0, 0),+ (0, 1, 1),+ (1, 1, 1),+ ]+ stride_options = [(1, 1, 1), (2, 1, 1)]+ kernel_size_options = [(1, 1, 1), (3, 3, 3)]+ tile_size = 16++ for padding in padding_options:+ for stride in stride_options:+ for kernel_size in kernel_size_options:+ for exclude_temporal_dim in exclude_temporal_dim_options:+ conv3d = (+ torch.nn.Conv3d(128, 128, kernel_size=kernel_size, stride=stride, padding=padding)+ .cuda()+ .to(torch.bfloat16)+ )+ auto_tiled_conv3d = TiledConv3d.from_native_conv3d(+ conv3d, tile_size=tile_size, exclude_temporal_dim=exclude_temporal_dim+ )++ # compare+ with torch.inference_mode():+ out = conv3d(data)+ merged_out = auto_tiled_conv3d(data)++ print(f"max allocated: {torch.cuda.max_memory_allocated() / 1024 / 1024 / 1024:.2f} GB")+ print(f"max reserved: {torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024:.2f} GB")++ try:+ torch.testing.assert_close(out, merged_out)+ except Exception as e:+ print(+ f"Failed with padding={padding}, stride={stride}, kernel_size={kernel_size}, exclude_temporal_dim={exclude_temporal_dim}, error: {e}"+ )+++if __name__ == "__main__":+ test_tiled_conv3d()diff --git a/tests/test_video_io.py b/tests/test_video_io.pynew file mode 100644index 0000000..73c3ecd--- /dev/null+++ b/tests/test_video_io.py@@ -0,0 +1,163 @@+import os++import cv2+import numpy as np+import torch+from mmengine.runner import set_random_seed+from tqdm import tqdm++from opensora.datasets import video_transforms+from opensora.datasets.utils import read_file+from opensora.utils.misc import to_torch_dtype++# data_path = "~/data/issue.csv"+data_path = "~/data/test.csv"+save_dir = "samples/debug_original_video_read_write"+num_frames = 17+frame_interval = 1+image_size = 1024++set_random_seed(1024)+os.makedirs(save_dir, exist_ok=True)+data = read_file(data_path)++device = "cuda" if torch.cuda.is_available() else "cpu"+dtype = to_torch_dtype("bf16")+++def temporal_random_crop(vframes, num_frames, frame_interval):+ temporal_sample = video_transforms.TemporalRandomCrop(num_frames * frame_interval)+ total_frames = len(vframes)+ start_frame_ind, end_frame_ind = temporal_sample(total_frames)+ assert (+ end_frame_ind - start_frame_ind >= num_frames+ ), f"Not enough frames to sample, {end_frame_ind} - {start_frame_ind} < {num_frames}"+ frame_indice = np.linspace(start_frame_ind, end_frame_ind - 1, num_frames, dtype=int)+ video = vframes[frame_indice]+ return video+++def to_tensor(clip):+ """+ Convert tensor data type from uint8 to float, divide value by 255.0 and+ permute the dimensions of clip tensor+ Args:+ clip (torch.tensor, dtype=torch.uint8): Size is (T, C, H, W)+ Return:+ clip (torch.tensor, dtype=torch.float): Size is (T, C, H, W)+ """+ if not clip.dtype == torch.uint8:+ raise TypeError("clip tensor should have data type uint8. Got %s" % str(clip.dtype))+ # breakpoint()+ return clip.float() / 255.0+++def read_video_cv2(video_path):+ cap = cv2.VideoCapture(video_path)++ if not cap.isOpened():+ raise ValueError+ else:+ fps = cap.get(cv2.CAP_PROP_FPS)+ vinfo = {+ "video_fps": fps,+ }++ frames = []+ while True:+ # Read a frame from the video+ ret, frame = cap.read()++ # If frame is not read correctly, break the loop+ if not ret:+ break++ # frames.append(frame[:, :, ::-1]) # BGR to RGB+ frames.append(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))+ # Exit if 'q' is pressed+ if cv2.waitKey(25) & 0xFF == ord("q"):+ break++ # Release the video capture object and close all windows+ cap.release()+ cv2.destroyAllWindows()++ frames = np.stack(frames)+ frames = torch.from_numpy(frames) # [T, H, W, C=3]+ frames = frames.permute(0, 3, 1, 2)+ return frames, vinfo+++def write_video_cv2(path, video, fps=24, image_size=(1920, 1080)):+ # Set the video codec and create a VideoWriter object+ fourcc = cv2.VideoWriter_fourcc(*"mp4v")+ output = cv2.VideoWriter(path, fourcc, fps, image_size)++ for frame_idx in range(video.size(0)):+ frame = np.array(video[frame_idx].permute(1, 2, 0))+ frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)+ output.write(frame)++ output.release()+++for video_path in tqdm(data["path"]):+ name = os.path.basename(video_path)++ # # DEBUG: read image and save as if video: no issue+ # image = cv2.imread('/home/shenchenhui/data/ship-in-coffee-image.png')+ # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)+ # image = torch.Tensor(image)+ # fake_vid = image.repeat(48, 1, 1, 1)+ # write_video(f"{save_dir}/fake.mp4", fake_vid, fps=24, video_codec="h264")++ # # ===== data loading ====== #+ # # loading+ # vframes, vinfo = read_video(video_path, backend="cv2")+ vframes, vinfo = read_video_cv2(video_path)+ video_fps = vinfo["video_fps"] if "video_fps" in vinfo else 24+ print("fps:", video_fps)+ # # Sampling video frames+ video = vframes+ video = temporal_random_crop(vframes, num_frames, frame_interval) # not this issue+ # # breakpoint()++ # video = to_tensor(video)++ # # # transform+ # # transform_video = transforms.Compose(+ # # [+ # # # video_transforms.ToTensorVideo(), # moved up+ # # # video_transforms.UCFCenterCropVideo(image_size), # not this issue+ # # transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5], inplace=True), # not this issues+ # # ]+ # # )+ # # video = transform_video(video) # T C H W++ ### write each frame as image+ # for frame_idx in range(video.size(0)):+ # frame = np.array(video[frame_idx].permute(1,2,0))+ # frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)+ # cv2.imwrite(f'{save_dir}/{frame_idx}.jpg', frame)++ # # # TCHW -> CTHW+ # video = video.permute(1, 0, 2, 3)++ # # # # ===== model training ====== #+ # # # video = video.to(device, dtype)++ # # # ===== data saving ====== #+ # # # # Normalize+ # # # low, high = -1,1+ # # # video.clamp_(min=low, max=high)+ # # # video.sub_(low).div_(max(high - low, 1e-5))++ # # # video = video.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 3, 0).to("cpu", torch.uint8)+ # # # breakpoint()+ # # # video = video.permute(1, 2, 3, 0).to("cpu", torch.uint8)+ # video = video.permute(1, 2, 3, 0)++ #+ write_video_cv2(f"{save_dir}/{name}", video, fps=video_fps)+ # # prep to [T, H, W, C] in order to write+ # write_video(f"{save_dir}/{name}", video, fps=video_fps, video_codec="h264")diff --git a/tools/caption/README.md b/tools/caption/README.mdindex 8f7dfed..e7519af 100644--- a/tools/caption/README.md+++ b/tools/caption/README.md@@ -49,7 +49,7 @@ cd .. # step back to pllava_dirCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \PYTHONPATH='$PYTHONPATH:OPEN_SORA_HOME/tools/caption/pllava_dir/PLLaVA' \-nohup python caption_pllava.py \+nohup torchrun --nproc_per_node 8 --standalone caption_pllava.py \--pretrained_model_name_or_path PLLaVA/MODELS/pllava-13b \--use_lora \--lora_alpha 4 \@@ -71,10 +71,10 @@ Qualitatively speaking, we observe that PLLaVA has a somewhat higher chance of a-| LLaVA | PLLaVA |-|----------|----------|-| The video is a close-up shot of two gold wedding rings. The rings are placed on a (black surface)✅, casting a soft shadow beneath them. The rings are positioned in such a way that (they are facing each other)❌, creating a mirror image effect. The rings are (identical in size and design)✅, suggesting they are a pair. The lighting in the video is soft and diffused, highlighting the gold color of the rings and creating a warm and inviting atmosphere. The overall style of the video is minimalist and elegant, focusing solely on the rings and their reflection. | The video shows a pair of gold wedding rings on a (reflective surface)✅. The rings are placed one on top of the other, (with the top ring slightly tilted to the left)✅. The rings have a (shiny, metallic finish)✅ and are the main focus of the image. The background is a gradient of dark to light gray, providing a neutral backdrop that highlights the rings. There are no texts or other objects in the image. The style of the video is a simple product display with a focus on the rings, likely intended for promotional or sales purposes. The lighting and shadows suggest a soft, even light source, (possibly a studio light)✅, which creates a reflective surface beneath the rings.|-<!-- |Row2Cell1|Row2Cell2| -->+ | LLaVA | PLLaVA |+-----|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----+ | The video is a close-up shot of two gold wedding rings. The rings are placed on a (black surface)✅, casting a soft shadow beneath them. The rings are positioned in such a way that (they are facing each other)❌, creating a mirror image effect. The rings are (identical in size and design)✅, suggesting they are a pair. The lighting in the video is soft and diffused, highlighting the gold color of the rings and creating a warm and inviting atmosphere. The overall style of the video is minimalist and elegant, focusing solely on the rings and their reflection. | The video shows a pair of gold wedding rings on a (reflective surface)✅. The rings are placed one on top of the other, (with the top ring slightly tilted to the left)✅. The rings have a (shiny, metallic finish)✅ and are the main focus of the image. The background is a gradient of dark to light gray, providing a neutral backdrop that highlights the rings. There are no texts or other objects in the image. The style of the video is a simple product display with a focus on the rings, likely intended for promotional or sales purposes. The lighting and shadows suggest a soft, even light source, (possibly a studio light)✅, which creates a reflective surface beneath the rings. |+<!-- | Row2Cell1 | Row2Cell2 | -->@@ -211,3 +211,43 @@ torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 meta.cs```This will generate tags based on the `text` column of `meta.csv` and put the results to `output_prefix + key.csv`. Currently the prompts for `objects` and `actions` are supported.++++## LLaVA-Next Captioning+LLaVA-NeXT-Video-32B-Qwen, based on Qwen-32B, improves captioning accuracy and linguistic diversity through enhanced language understanding and better visual-text alignment. Using 32 video frames, it captures detailed spatial information for more contextually accurate captions. To optimize efficiency, we batch frames for GPU utilization and employ multi-threaded frame extraction to run in parallel with GPU computations, preventing bottlenecks. Loading in 8-bits is currently buggy and needs to be fixed.++### Installation++You need to install VllaVA++```+# install VLLaVA+git clone https://github.com/LLaVA-VL/LLaVA-NeXT+cd LLaVA-NeXT+conda create -n llava python=3.10 -y+conda activate llava+pip install --upgrade pip # Enable PEP 660 support.+pip install -e ".[train]"++# we use fixed transformers version+pip install transformers==0.40.0+```++### Usage++The script takes one csv dataset file and lauch World_Size processes, each handle one split of the caption jobs.++```+CUDA_VISIBLE_DEVICES=1,2,3,4 python3 caption_llava_next.py \+ --model-path /PATH/TO/LLaVA-NeXT-Video-32B-Qwen \+ --data_file /PATH/TO/data.csv \+ --output_folder /PATH/TO/output_dir \+ --overwrite true \+ --mm_spatial_pool_stride 2 \+ --for_get_frames_num 32 \+ --conv-mode qwen_2 \+ --mm_spatial_pool_mode average \+ --mm_newline_position grid \+ --prompt "Please provide a detailed description of the video, focusing on the main subjects, their actions, the background scenes."+```\ No newline at end of filediff --git a/tools/caption/caption_gpt4.py b/tools/caption/caption_gpt4.pyindex f22c296..b896480 100644--- a/tools/caption/caption_gpt4.py+++ b/tools/caption/caption_gpt4.py@@ -4,11 +4,13 @@ import csvimport osfrom io import BytesIO-import requestsimport tqdm+from openai import OpenAIfrom .utils import IMG_EXTENSIONS, PROMPTS, VID_EXTENSIONS, VideoTextDataset+client = OpenAI()+def to_base64(image):buffer = BytesIO()@@ -16,29 +18,26 @@ def to_base64(image):return base64.b64encode(buffer.getvalue()).decode("utf-8")-def get_caption(frame, prompt, api_key):- headers = {"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"}- payload = {- "model": "gpt-4-vision-preview",- "messages": [+def get_caption(frame, prompt):+ response = client.chat.completions.create(+ model="gpt-4o-2024-08-06",+ messages=[+ {"role": "system", "content": prompt},{"role": "user","content": [- {- "type": "text",- "text": prompt,- },- {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame[0]}"}},- {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame[1]}"}},- {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame[2]}"}},+ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame[0]}", "detail": "low"}},+ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame[1]}", "detail": "low"}},+ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame[2]}", "detail": "low"}},],- }+ },],- "max_tokens": 300,- }- response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload, timeout=60)- caption = response.json()["choices"][0]["message"]["content"]+ max_tokens=300,+ top_p=0.1,+ )+ caption = response.choices[0].message.contentcaption = caption.replace("\n", " ")+return caption@@ -46,7 +45,7 @@ def main(args):# ======================================================# 1. read video list# ======================================================- dataset = VideoTextDataset(args.input)+ dataset = VideoTextDataset(args.input, resize=360)output_file = os.path.splitext(args.input)[0] + "_caption.csv"f = open(output_file, "w")writer = csv.writer(f)@@ -75,7 +74,7 @@ def main(args):prompt = prompt.format(sample["text"])frames = sample["image"]frames = [to_base64(frame) for frame in frames]- caption = get_caption(frames, prompt, args.key)+ caption = get_caption(frames, prompt)writer.writerow((sample["path"], caption))f.close()@@ -84,8 +83,7 @@ def main(args):if __name__ == "__main__":parser = argparse.ArgumentParser()parser.add_argument("input", type=str, help="Path to the input CSV file")- parser.add_argument("--prompt", type=str, default="video-f3-detail-3ex")- parser.add_argument("--key", type=str)+ parser.add_argument("--prompt", type=str, default="video-template") # 1k/1hargs = parser.parse_args()main(args)diff --git a/tools/caption/caption_llama3.py b/tools/caption/caption_llama3.pyindex 754f5a6..5337c0e 100644--- a/tools/caption/caption_llama3.py+++ b/tools/caption/caption_llama3.py@@ -10,8 +10,7 @@ import torch.distributed as distfrom torch.utils.data import Datasetfrom tqdm import tqdmfrom transformers import AutoModelForCausalLM, AutoTokenizer--from .utils import read_file+from utils import read_fileos.system(f"cp {__file__} ~/backup/") # optionally backup the scriptwarnings.filterwarnings("ignore")@@ -96,10 +95,26 @@ def main(args):ptvsd.enable_attach(address=("localhost", int(os.getenv("DEBUG_ADDRESS"))), redirect_output=True)ptvsd.wait_for_attach()- output_file = args.output_prefix + f"_rank{dist.get_rank()}" + f"_{args.key}.csv"+ # output_file = output_prefix + f"_rank{dist.get_rank()}" + f"_{args.key}.csv"+ # 假设你已经有了 output_prefix 和 parts+ output_prefix = os.path.splitext(args.input)[0]+ parts = "parts"++ # 找到最后一个 '/'+ last_slash_index = output_prefix.rfind("/")++ # 如果找到了 '/', 在其后插入 parts+ if last_slash_index != -1:+ output_prefix_total = output_prefix[: last_slash_index + 1] + "/" + output_prefix[last_slash_index + 1 :]+ output_prefix = output_prefix[: last_slash_index + 1] + parts + "/" + output_prefix[last_slash_index + 1 :]++ output_file = output_prefix + f"_rank{dist.get_rank()}" + f"_objects_actions.csv"+ # 确保目录存在+ os.makedirs(os.path.dirname(output_file), exist_ok=True)output_file_handle = open(output_file, "w")writer = csv.writer(output_file_handle)- columns = list(dataframe.columns) + [args.key]+ # columns = list(dataframe.columns) + [args.key]+ columns = list(dataframe.columns) + ["objects"] + ["actions"]writer.writerow(columns)@@ -185,62 +200,77 @@ def main(args):return responsesprint("Processing starting...")- if args.prompt == "" and args.key == "objects":- prompt = (- "You are a AI assistant to extract objects from user's text. "- "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of objects separated by ',' and wrapped by '[' and ']': '[dog, person]' "- )- elif args.prompt == "" and args.key == "actions":- prompt = (- "You are a AI assistant to extract actions from user's text. "- "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of actions separated by ',' and wrapped by '[' and ']': '[run, laugh]' "- )- else:- prompt = args.prompt+ # prompt_objects = args.prompt_objects+ # prompt_actions = args.prompt_actions+ # if args.prompt_objects == "":+ prompt_objects = (+ "You are a AI assistant to extract objects from user's text. "+ "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of objects separated by ',' and wrapped by '[' and ']': '[dog, person]' "+ )+ # if args.prompt_actions == "":+ prompt_actions = (+ "You are a AI assistant to extract actions from user's text. "+ "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of actions separated by ',' and wrapped by '[' and ']': '[run, laugh]' "+ )- print("Prompt: {}".format(prompt))+ print("prompt_objects: {}".format(prompt_objects))+ print("prompt_actions: {}".format(prompt_actions))args.batch_size# for i in tqdm(range(0, len(dataframe), batch_size)):for _, batch in enumerate(tqdm(dataloader)):# get the text column from the batchtexts = [batch[i]["text"] for i in range(len(batch))]- list_keywords = extract_batch(texts, prompt)+ # Extract keywords using both prompts+ list_actions = extract_batch(texts, prompt_actions)+ list_objects = extract_batch(texts, prompt_objects)- for idx, keywords in enumerate(list_keywords):+ for idx, (actions, objects) in enumerate(zip(list_actions, list_objects)):try:- keywords_start = keywords.find("[")- keywords_end = keywords.find("]")- keywords = keywords[keywords_start + 1 : keywords_end]- if (- "\n" in keywords or len(keywords.strip()) == 0- ): # we empirically observe that it produces newlines when no keywords are found- keywords = "NONE_FOUND"+ # Process actions+ if actions.startswith("'") and actions.endswith("'"):+ actions = actions[1:-1]+ actions_start = actions.find("[")+ actions_end = actions.find("]")+ actions = actions[actions_start + 1 : actions_end]+ actions = [act.strip() for act in actions.split(",") if act.strip()]++ # Process objects+ if objects.startswith("'") and objects.endswith("'"):+ objects = objects[1:-1]+ objects_start = objects.find("[")+ objects_end = objects.find("]")+ objects = objects[objects_start + 1 : objects_end]+ objects = [obj.strip() for obj in objects.split(",") if obj.strip()]except:- keywords = "NONE_FOUND"- row = batch[idx]- writer.writerow([*row, keywords])+ actions = "NONE_FOUND"+ objects = "NONE_FOUND"++ row = batch[idx]+ writer.writerow([*row, objects, actions]) # Adjusted to include both objects and actionsoutput_file_handle.close()dist.barrier()if dist.get_rank() == 0:- collated_file = args.output_prefix + f"_{args.key}.csv"+ # collated_file = output_prefix + f"_{args.key}.csv"+ collated_file = output_prefix_total + f"_objects_actions.csv"print("All ranks are finished. Collating the processed data to {}".format(collated_file))import pandas as pd- csv_files = [args.output_prefix + f"_rank{i}" + f"_{args.key}.csv" for i in range(dist.get_world_size())]+ csv_files = [output_prefix + f"_rank{i}" + f"_objects_actions.csv" for i in range(dist.get_world_size())]# List to hold DataFramesdataframes = []# Read each CSV into a DataFrame and append to listfor file in csv_files:df = pd.read_csv(file)# scan each line in the df, if the ``key`` column is NaN, replace it with "NONE_FOUND"- df[args.key] = df[args.key].fillna("NONE_FOUND")+ # df[args.key] = df[args.key].fillna("NONE_FOUND")+ df["objects"] = df["objects"].fillna("NONE_FOUND")+ df["actions"] = df["actions"].fillna("NONE_FOUND")dataframes.append(df)# Concatenate all DataFramescombined_df = pd.concat(dataframes, ignore_index=True)-# Save the combined DataFrame to a new CSV filecombined_df.to_csv(collated_file, index=False)print("Collated data saved to {}".format(collated_file))@@ -252,10 +282,11 @@ if __name__ == "__main__":parser = argparse.ArgumentParser()parser.add_argument("--model-id", default="meta-llama/Meta-Llama-3-8B-Instruct")parser.add_argument("input", type=str, help="Path to the input CSV file")- parser.add_argument("--output_prefix", type=str, help="Path to the output CSV file")- parser.add_argument("--prompt", type=str, default="")+ # parser.add_argument("--output_prefix", type=str, help="Path to the output CSV file")+ # parser.add_argument("--prompt_objects", type=str, default="")+ # parser.add_argument("--prompt_actions", type=str, default="")parser.add_argument("--batch_size", type=int, default=32)- parser.add_argument("--key", type=str)+ # parser.add_argument("--key", type=str)args = parser.parse_args()main(args)diff --git a/tools/caption/caption_llava.py b/tools/caption/caption_llava.pyindex ab69145..1674a1b 100644--- a/tools/caption/caption_llava.py+++ b/tools/caption/caption_llava.py@@ -22,6 +22,9 @@ from .acceleration.llava.policies import LlavaLlamaForCausalLMPolicy, LlavaMistrfrom .utils import PROMPTS, Timer, VideoTextDataset, collate_fndisable_torch_init()+import transformers++transformers.logging.set_verbosity_error()class NoPaddingDistributedSampler(DistributedSampler):@@ -67,10 +70,10 @@ def main(args):set_seed(1024)coordinator = DistCoordinator()- # prepare the dp and tp groupsassert (args.dp_size * args.tp_size == coordinator.world_size), f"DP size {args.dp_size} * TP size {args.tp_size} must equal to world size {coordinator.world_size}"+mesh = ProcessGroupMesh(args.dp_size, args.tp_size)dp_group = mesh.get_group_along_axis(0)tp_group = mesh.get_group_along_axis(1)@@ -80,7 +83,7 @@ def main(args):# ======================================================model_path = args.model_pathwith warnings.catch_warnings():- warnings.simplefilter("ignore") # Pytorch non-meta copying warning fills out the console+ warnings.simplefilter("ignore")tokenizer, model, image_processor, context_len = load_pretrained_model(model_path=model_path,model_base=None,@@ -127,7 +130,6 @@ def main(args):query_text = query.format(text)conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\n" + query_text)prompt = conv.get_prompt()- # add num_frames imagest = prompt.split("<image>")prompt = t[0] + "<image>" * args.num_frames + t[1]input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt")@@ -138,7 +140,6 @@ def main(args):conv = conv_templates["chatml_direct"].copy()conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\n" + query)prompt = conv.get_prompt()- # add num_frames imagest = prompt.split("<image>")prompt = t[0] + "<image>" * args.num_frames + t[1]input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt")@@ -167,11 +168,11 @@ def main(args):if prompt_type == "image":assert (data_extension.lower() in IMG_EXTENSIONS- ), f"The prompt is suitable for an image dataset but the data is not image. The first data is of format {data_extension}"+ ), f"The prompt is suitable for an image dataset but the data is not image."elif prompt_type == "video":assert (data_extension.lower() in VID_EXTENSIONS- ), f"The prompt is suitable for a video dataset but the data is not video. The first data is of format {data_extension}"+ ), f"The prompt is suitable for a video dataset but the data is not video."else:raise ValueError(f"Found invalid prompt type {prompt_type}")@@ -182,7 +183,6 @@ def main(args):dp_size = dist.get_world_size(dp_group)sampler = NoPaddingDistributedSampler(dataset, rank=dp_rank, num_replicas=dp_size)- # build dataloaderdataloader = torch.utils.data.DataLoader(dataset,batch_size=args.bs,@@ -194,24 +194,18 @@ def main(args):collate_fn=collate_fn,)+ with open(args.input, "r") as f:+ reader = csv.DictReader(f)+ original_data = [row for row in reader]+ headers = reader.fieldnames+# prepare output file readeroutput_file = args.input.replace(".csv", "_caption.csv")- # create csv writer- has_dp_writter = dist.get_rank(tp_group) == 0-- if has_dp_writter:- # the dp writer takes care of the files processed on the current dp rank- # so we use write mode- output_file_split = output_file.replace(".csv", f"_part{dp_rank}.csv")- dp_file = open(output_file_split, "w")- dp_writer = csv.writer(dp_file)- dp_writer.writerow(["path", "text", "num_frames"])-# ======================================================# 5. generate captions# ======================================================- if dist.get_rank(tp_group) == 0:+ if dist.get_rank() == 0:pbar = tqdm(dataloader, position=dp_rank, desc=f"Data Parallel Rank {dist.get_rank(dp_group)}")else:pbar = dataloader@@ -222,6 +216,7 @@ def main(args):output_length = []total_time = []+ results = []for i, batch in enumerate(pbar):# measure timeif args.profile:@@ -262,16 +257,14 @@ def main(args):]inputs_embeds = torch.cat(inputs_embeds, dim=0)- # generate outputswith Timer() as generate_timer:output_ids = super(type(model), model).generate(inputs_embeds=inputs_embeds,attention_mask=attention_mask,- do_sample=False, # sampling is not deterministic and may cause TP to hang+ do_sample=False,max_new_tokens=args.max_tokens,use_cache=True,)-# skip warmup and add profiling dataif args.profile and i >= args.profile_warmup:output_length.append(output_ids.size(0) * output_ids.size(1))@@ -289,11 +282,11 @@ def main(args):encode_time.append(encode_timer.time_taken)generate_time.append(generate_timer.time_taken)- # save results- if has_dp_writter:- result = list(zip(video_files, outputs, video_lengths))- for t in result:- dp_writer.writerow(t)+ for video_file, output_text, video_length in zip(video_files, outputs, video_lengths):+ original_row = next(row for row in original_data if row["path"] == video_file)+ original_row["text"] = output_text+ original_row["num_frames"] = video_length+ results.append(original_row)# display profiling infoif args.profile:@@ -306,15 +299,26 @@ def main(args):print(f"Max GPU allocated / GB: {torch.cuda.max_memory_allocated() / 1024**3}")print(f"Max GPU reserved / GB: {torch.cuda.max_memory_reserved() / 1024**3}")- # ======================================================- # 6. shutdown- # ======================================================- # close file writing- if has_dp_writter:- dp_file.close()- dist.barrier()+ if dist.get_rank() == 0:+ all_results = [None] * dist.get_world_size()+ else:+ all_results = None+ dist.gather_object(results, all_results, dst=0)++ if dist.get_rank() == 0:+ all_results = [item for sublist in all_results if sublist is not None for item in sublist]++ with open(output_file, "w", newline="") as f:+ if "num_frames" not in headers:+ writer = csv.DictWriter(f, fieldnames=headers + ["text", "num_frames"])+ else:+ writer = csv.DictWriter(f, fieldnames=headers + ["text"])+ writer.writeheader()+ writer.writerows(all_results)- # terminate distributed env+ print(f"Results saved to {output_file}")++ dist.barrier()dist.destroy_process_group()@@ -326,7 +330,6 @@ if __name__ == "__main__":parser.add_argument("--resize", type=int, default=336)parser.add_argument("--num-frames", type=int, default=1)parser.add_argument("--max-tokens", type=int, default=300)- # speed relatedparser.add_argument("--bs", type=int, default=16)parser.add_argument("--tp-size", type=int, default=2)parser.add_argument("--dp-size", type=int, default=4)diff --git a/tools/caption/caption_llava_next.py b/tools/caption/caption_llava_next.pynew file mode 100644index 0000000..2d1c727--- /dev/null+++ b/tools/caption/caption_llava_next.py@@ -0,0 +1,379 @@+# code modified based on https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/playground/demo/video_demo.py++import argparse+import base64+import csv+import math+import os+import warnings++import cv2+import numpy as np+import pandas as pd+import torch+import torch.distributed as dist+import torch.multiprocessing as mp+from decord import VideoReader, cpu+from llava.constants import DEFAULT_IM_END_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX+from llava.conversation import conv_templates+from llava.mm_utils import get_model_name_from_path, tokenizer_image_token+from llava.model.builder import load_pretrained_model+from torch.utils.data import DataLoader+from tqdm import tqdm+from transformers import AutoConfig, BitsAndBytesConfig++warnings.filterwarnings("ignore")++PAD_TOKEN_ID = 151643+++# Function to initialize the distributed environment+def setup(rank, world_size):+ print(f"Setting up process {rank} of {world_size}")+ os.environ["MASTER_ADDR"] = "localhost"+ os.environ["MASTER_PORT"] = "12355"+ torch.cuda.set_device(rank)+ # Initialize the process group for communication+ dist.init_process_group(backend="nccl", rank=rank, world_size=world_size)+++# Cleanup after inference is done+def cleanup():+ dist.destroy_process_group()+++class VideoDataset:+ def __init__(self, df, args, rank, world_size, image_processor, model, tokenizer):+ self.df = df+ self.rank = rank+ self.world_size = world_size+ self.args = args+ self.image_processor = image_processor+ self.model = model+ self.tokenizer = tokenizer+ self.length = len([i for i in range(self.rank, len(self.df), self.world_size)])++ def __len__(self):+ return self.length++ def __getitem__(self, idx):+ row = self.df.iloc[self.rank + idx * self.world_size]+ video_path = row["path"]+ info = row+ sample_set = {}+ question = self.args.prompt+ sample_set["video_name"] = video_path++ if os.path.exists(video_path):+ video, frame_time, video_time = load_video(video_path, self.args)+ video = self.image_processor.preprocess(video, return_tensors="pt")["pixel_values"].half()+ sample_set["video"] = video+ sample_set["frame_time"] = frame_time+ sample_set["video_time"] = video_time+ sample_set["info"] = info++ if self.args.add_time_instruction:+ time_instruction = (+ f"The video lasts for {video_time:.2f} seconds, and "+ f"{self.args.for_get_frames_num} frames are uniformly sampled from it. "+ f"These frames are located at {frame_time}. "+ f"Please answer the following questions related to this video."+ )+ qs = f"{time_instruction}\n{question}"+ else:+ qs = question+ if self.model.config.mm_use_im_start_end:+ qs = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + "\n" + qs+ else:+ qs = DEFAULT_IMAGE_TOKEN + "\n" + qs++ conv = conv_templates[self.args.conv_mode].copy()+ conv.append_message(conv.roles[0], qs)+ conv.append_message(conv.roles[1], None)+ prompt = conv.get_prompt()++ input_ids = tokenizer_image_token(prompt, self.tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(+ 0+ )+ # print("input_ids", input_ids)+ sample_set["input_ids"] = input_ids+ attention_masks = input_ids.ne(self.tokenizer.pad_token_id).long()+ sample_set["attention_masks"] = attention_masks++ return sample_set+++def collate_fn(batch):+ # Collate function to handle dynamic padding or combining elements of the batch+ videos = [item["video"] for item in batch if "video" in item]+ input_ids = [item["input_ids"] for item in batch]+ max_len = max([item.shape[1] for item in input_ids])+ # pad token id PAD_TOKEN_ID+ input_ids = [torch.nn.functional.pad(item, (max_len - item.shape[1], 0), value=PAD_TOKEN_ID) for item in input_ids]+ input_ids = torch.cat(input_ids, dim=0)+ attention_masks = [item["attention_masks"] for item in batch]+ attention_masks = [torch.nn.functional.pad(item, (max_len - item.shape[1], 0), value=0) for item in attention_masks]+ attention_masks = torch.cat(attention_masks, dim=0)++ video_names = [item["video_name"] for item in batch]+ frame_times = [item["frame_time"] for item in batch]+ video_times = [item["video_time"] for item in batch]+ infos = [item["info"] for item in batch]++ return {+ "input_ids": input_ids,+ "attention_masks": attention_masks,+ "videos": videos,+ "video_names": video_names,+ "frame_times": frame_times,+ "video_times": video_times,+ "infos": infos,+ }+++def create_dataloader(df, args, rank, world_size, image_processor, model, tokenizer):+ dataset = VideoDataset(df, args, rank, world_size, image_processor, model, tokenizer)+ return DataLoader(+ dataset, batch_size=args.batch_size, shuffle=False, collate_fn=collate_fn, num_workers=args.num_workers+ )+++def split_list(lst, n):+ """Split a list into n (roughly) equal-sized chunks"""+ chunk_size = math.ceil(len(lst) / n) # integer division+ return [lst[i : i + chunk_size] for i in range(0, len(lst), chunk_size)]+++def get_chunk(lst, n, k):+ chunks = split_list(lst, n)+ return chunks[k]+++def parse_args():+ """+ Parse command-line arguments.+ """+ parser = argparse.ArgumentParser()++ # Define the command-line arguments+ parser.add_argument("--data_file", help="Path to the video dataset file.", required=True)+ parser.add_argument("--output_folder", help="Path to the output file.", required=True)+ parser.add_argument("--model-path", type=str, default="facebook/opt-350m")+ parser.add_argument("--model-base", type=str, default=None)+ parser.add_argument("--conv-mode", type=str, default=None)+ parser.add_argument("--chunk-idx", type=int, default=0)+ parser.add_argument("--mm_resampler_type", type=str, default="spatial_pool")+ parser.add_argument("--mm_spatial_pool_stride", type=int, default=4)+ parser.add_argument("--mm_spatial_pool_out_channels", type=int, default=1024)+ parser.add_argument("--mm_spatial_pool_mode", type=str, default="average")+ parser.add_argument("--image_aspect_ratio", type=str, default="anyres")+ parser.add_argument(+ "--image_grid_pinpoints",+ type=str,+ default="[(224, 448), (224, 672), (224, 896), (448, 448), (448, 224), (672, 224), (896, 224)]",+ )+ parser.add_argument("--mm_patch_merge_type", type=str, default="spatial_unpad")+ parser.add_argument("--overwrite", type=lambda x: (str(x).lower() == "true"), default=True)+ parser.add_argument("--for_get_frames_num", type=int, default=4)+ parser.add_argument("--batch_size", type=int, default=1)+ parser.add_argument("--num_workers", type=int, default=10)+ parser.add_argument("--load_8bit", type=lambda x: (str(x).lower() == "true"), default=False)+ parser.add_argument("--prompt", type=str, default=None)+ parser.add_argument("--api_key", type=str, help="OpenAI API key")+ parser.add_argument("--mm_newline_position", type=str, default="no_token")+ parser.add_argument("--force_sample", type=lambda x: (str(x).lower() == "true"), default=False)+ parser.add_argument("--add_time_instruction", type=str, default=False)+ return parser.parse_args()+++def load_video(video_path, args):+ if args.for_get_frames_num == 0:+ return np.zeros((1, 336, 336, 3))+ vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)+ total_frame_num = len(vr)+ video_time = total_frame_num / vr.get_avg_fps()+ fps = round(vr.get_avg_fps())+ frame_idx = [i for i in range(0, len(vr), fps)]+ frame_time = [i / fps for i in frame_idx]+ if len(frame_idx) > args.for_get_frames_num or args.force_sample:+ sample_fps = args.for_get_frames_num+ uniform_sampled_frames = np.linspace(0, total_frame_num - 1, sample_fps, dtype=int)+ frame_idx = uniform_sampled_frames.tolist()+ frame_time = [i / vr.get_avg_fps() for i in frame_idx]+ frame_time = ",".join([f"{i:.2f}s" for i in frame_time])+ spare_frames = vr.get_batch(frame_idx).asnumpy()+ # import pdb;pdb.set_trace()++ return spare_frames, frame_time, video_time+++def load_video_base64(path):+ video = cv2.VideoCapture(path)++ base64Frames = []+ while video.isOpened():+ success, frame = video.read()+ if not success:+ break+ _, buffer = cv2.imencode(".jpg", frame)+ base64Frames.append(base64.b64encode(buffer).decode("utf-8"))++ video.release()+ # print(len(base64Frames), "frames read.")+ return base64Frames+++def run_inference(rank, world_size, args):+ """+ Run inference on ActivityNet QA DataSet using the Video-ChatGPT model.++ Args:+ args: Command-line arguments.+ """+ setup(rank, world_size)++ device = torch.device(f"cuda:{rank}")+ # Initialize the model+ model_name = get_model_name_from_path(args.model_path)+ # Set model configuration parameters if they exist+ if args.overwrite == True:+ overwrite_config = {}+ overwrite_config["mm_spatial_pool_mode"] = args.mm_spatial_pool_mode+ overwrite_config["mm_spatial_pool_stride"] = args.mm_spatial_pool_stride+ overwrite_config["mm_newline_position"] = args.mm_newline_position++ cfg_pretrained = AutoConfig.from_pretrained(args.model_path)++ # import pdb;pdb.set_trace()+ if "qwen" not in args.model_path.lower():+ if "224" in cfg_pretrained.mm_vision_tower:+ # suppose the length of text tokens is around 1000, from bo's report+ least_token_number = args.for_get_frames_num * (16 // args.mm_spatial_pool_stride) ** 2 + 1000+ else:+ least_token_number = args.for_get_frames_num * (24 // args.mm_spatial_pool_stride) ** 2 + 1000++ scaling_factor = math.ceil(least_token_number / 4096)+ if scaling_factor >= 2:+ if "vicuna" in cfg_pretrained._name_or_path.lower():+ print(float(scaling_factor))+ overwrite_config["rope_scaling"] = {"factor": float(scaling_factor), "type": "linear"}+ overwrite_config["max_sequence_length"] = 4096 * scaling_factor+ overwrite_config["tokenizer_model_max_length"] = 4096 * scaling_factor+ if args.load_8bit:+ quantization_config = BitsAndBytesConfig(load_in_8bit=True, bnb_8bit_compute_dtype=torch.bfloat16)+ tokenizer, model, image_processor, context_len = load_pretrained_model(+ args.model_path,+ args.model_base,+ model_name,+ device_map=device,+ quantization_config=quantization_config,+ overwrite_config=overwrite_config,+ )+ else:+ tokenizer, model, image_processor, context_len = load_pretrained_model(+ args.model_path, args.model_base, model_name, device_map=device, overwrite_config=overwrite_config+ )+ else:+ tokenizer, model, image_processor, context_len = load_pretrained_model(+ args.model_path, args.model_base, model_name, device_map=device+ )++ if tokenizer.pad_token_id is None:+ if "qwen" in tokenizer.name_or_path.lower():+ # print("Setting pad token to bos token for qwen model.")+ tokenizer.pad_token_id = PAD_TOKEN_ID++ if args.batch_size > 1:+ tokenizer.padding_side = "left"+ model.config.tokenizer_padding_side = "left"++ # model = DDP(model, device_ids=[rank])++ # import pdb;pdb.set_trace()+ if getattr(model.config, "force_sample", None) is not None:+ args.force_sample = model.config.force_sample+ else:+ args.force_sample = False++ if getattr(model.config, "add_time_instruction", None) is not None:+ args.add_time_instruction = model.config.add_time_instruction+ else:+ args.add_time_instruction = False++ df = pd.read_csv(args.data_file)+ data_name = os.path.basename(args.data_file).split(".csv")[0]+ column_names = df.columns.to_list()+ if "text" not in column_names:+ column_names.append("text")+ text_column_index = column_names.index("text")++ output_file = os.path.join(args.output_folder, f"{data_name}_{rank}.csv")+ with open(output_file, "w", newline="") as csvfile:+ csvwriter = csv.writer(csvfile)+ csvwriter.writerow(column_names)++ dataloader = create_dataloader(df, args, rank, world_size, image_processor, model, tokenizer)++ for batch in tqdm(dataloader):+ videos = [item.to(device) for item in batch["videos"]]+ input_ids = batch["input_ids"].to(device)+ attention_masks = batch["attention_masks"].to(device)+ infos = batch["infos"]+ stop_str = "###"+ with torch.inference_mode():+ modalities = ["video"] * len(videos)+ if "mistral" not in cfg_pretrained._name_or_path.lower():+ output_ids = model.generate(+ inputs=input_ids,+ images=videos,+ attention_mask=attention_masks,+ modalities=modalities,+ do_sample=False,+ temperature=0.0,+ max_new_tokens=1024,+ top_p=0.1,+ num_beams=1,+ use_cache=True,+ )+ # output_ids = model.generate(inputs=input_ids, images=video, attention_mask=attention_masks, modalities="video", do_sample=True, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=[stopping_criteria])+ else:+ output_ids = model.generate(+ inputs=input_ids,+ images=videos,+ attention_mask=attention_masks,+ modalities=modalities,+ do_sample=False,+ temperature=0.0,+ max_new_tokens=1024,+ top_p=0.1,+ num_beams=1,+ use_cache=True,+ )+ # output_ids = model.generate(inputs=input_ids, images=video, attention_mask=attention_masks, modalities="video", do_sample=True, temperature=0.2, max_new_tokens=1024, use_cache=True)+ # print("output_ids", output_ids)++ outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)++ # print("outputs", outputs)+ outputs = [output.split(stop_str)[0].strip() for output in outputs]+ if len(infos[0]) < len(column_names):+ for i in range(len(infos)):+ infos[i].append(outputs[i])+ else:+ for i in range(len(infos)):+ infos[i][text_column_index] = outputs[i]++ # write to csv+ for row in infos:+ csvwriter.writerow(row)++ cleanup()+++if __name__ == "__main__":+ world_size = torch.cuda.device_count()+ print(f"Running inference on {world_size} GPUs.")+ args = parse_args()++ # Spawn one process per GPU+ mp.spawn(run_inference, args=(world_size, args), nprocs=world_size, join=True)diff --git a/tools/caption/caption_trans.py b/tools/caption/caption_trans.pynew file mode 100644index 0000000..9126eb3--- /dev/null+++ b/tools/caption/caption_trans.py@@ -0,0 +1,79 @@+import argparse+import base64+import csv+import os+from io import BytesIO++import tqdm+from openai import OpenAI++from .utils import IMG_EXTENSIONS, PROMPTS, VID_EXTENSIONS, VideoTextDataset++client = OpenAI()+++def to_base64(image):+ buffer = BytesIO()+ image.save(buffer, format="JPEG")+ return base64.b64encode(buffer.getvalue()).decode("utf-8")+++def get_caption(text, prompt):+ response = client.chat.completions.create(+ model="gpt-4o-2024-08-06",+ messages=[+ {"role": "system", "content": prompt},+ {"role": "user", "content": text},+ ],+ max_tokens=300,+ top_p=0.1,+ )+ caption = response.choices[0].message.content+ caption = caption.replace("\n", " ")++ return caption+++def main(args):+ # ======================================================+ # 1. read video list+ # ======================================================+ dataset = VideoTextDataset(args.input, text_only=True)+ output_file = os.path.splitext(args.input)[0] + "_captrans.csv"+ f = open(output_file, "w")+ writer = csv.writer(f)+ writer.writerow(["video", "text"])++ # make sure that the prompt type matches the data type+ data_extension = "." + dataset.data["path"].iloc[0].split(".")[-1]+ prompt_type = PROMPTS[args.prompt]["type"]+ if prompt_type == "image":+ assert (+ data_extension.lower() in IMG_EXTENSIONS+ ), "The prompt is suitable for an image dataset but the data is not image."+ elif prompt_type == "video":+ assert (+ data_extension.lower() in VID_EXTENSIONS+ ), "The prompt is suitable for a video dataset but the data is not video."+ else:+ raise ValueError(f"Found invalid prompt type {prompt_type}")++ # ======================================================+ # 2. generate captions+ # ======================================================+ for sample in tqdm.tqdm(dataset):+ prompt = PROMPTS[args.prompt]["text"]+ text = sample["text"]+ caption = get_caption(text, prompt)++ writer.writerow((sample["path"], caption))+ f.close()+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("input", type=str, help="Path to the input CSV file")+ parser.add_argument("--prompt", type=str, default="video-captrans-template") # 1k/20min+ args = parser.parse_args()++ main(args)diff --git a/tools/caption/pllava_dir/caption_pllava.py b/tools/caption/pllava_dir/caption_pllava.pyindex ceb0721..044ac80 100644--- a/tools/caption/pllava_dir/caption_pllava.py+++ b/tools/caption/pllava_dir/caption_pllava.py@@ -1,6 +1,5 @@-import sys-import osimport os+import sysfrom pathlib import Pathcurrent_file = Path(__file__) # Gets the path of the current file@@ -8,188 +7,43 @@ fourth_level_parent = current_file.parents[3]datasets_dir = os.path.join(fourth_level_parent, "opensora/datasets")import sys+sys.path.append(datasets_dir)from read_video import read_video_av+sys.path.remove(datasets_dir)import itertoolsimport logging-import multiprocessing as mp+import random+import tracebackfrom argparse import ArgumentParserfrom multiprocessing import Process, Queue+import colossalaiimport numpy as npimport pandas as pdimport torch+import torch.distributed as distimport torchvisionimport transformers-from decord import VideoReader, cpu+from colossalai.utils import get_current_devicefrom PIL import Imagefrom tasks.eval.eval_utils import Conversationfrom tasks.eval.model_utils import load_pllava-from torch.utils.data import Dataset+from torch.utils.data import DataLoader, Dataset, DistributedSampler+from torch.utils.data._utils.collate import default_collatefrom tqdm import tqdm-from transformers.feature_extraction_utils import BatchFeature--conv_template = Conversation(- system="Describe this video. Pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. Here are some examples of good descriptions: 1. A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about. 2. Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field. 3. Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff's edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.",- roles=("USER:", "ASSISTANT:"),- messages=[],- sep=(" ", "</s>"),- mm_token="<image>",-)logging.basicConfig()logger = logging.getLogger(__name__)logger.setLevel(logging.INFO)-RESOLUTION = 672 #---def pllava_answer(- conv: Conversation,- model,- processor,- video_list,- do_sample=True,- max_new_tokens=200,- num_beams=1,- min_length=1,- top_p=0.9,- repetition_penalty=1.0,- length_penalty=1,- temperature=1.0,- stop_criteria_keywords=None,- print_res=False,-):- # torch.cuda.empty_cache()- prompt = conv.get_prompt()- inputs_list = [processor(text=prompt, images=video, return_tensors="pt") for video in video_list]- inputs_batched = dict() # add batch dimension by cat- for input_type in list(inputs_list[0].keys()):- inputs_batched[input_type] = torch.cat([inputs[input_type] for inputs in inputs_list])- inputs_batched = BatchFeature(inputs_batched, tensor_type="pt").to(model.device)-- with torch.no_grad():- output_texts = model.generate(- **inputs_batched,- media_type="video",- do_sample=do_sample,- max_new_tokens=max_new_tokens,- num_beams=num_beams,- min_length=min_length,- top_p=top_p,- repetition_penalty=repetition_penalty,- length_penalty=length_penalty,- temperature=temperature,- )- output_texts = processor.batch_decode(- output_texts, skip_special_tokens=True, clean_up_tokenization_spaces=False- )- for i in range(len(output_texts)):- if print_res: # debug usage- print("### PROMPTING LM WITH: ", prompt)- print("### LM OUTPUT TEXT: ", output_texts[i])- if conv.roles[-1] == "<|im_start|>assistant\n":- split_tag = "<|im_start|> assistant\n"- else:- split_tag = conv.roles[-1]- output_texts[i] = output_texts[i].split(split_tag)[-1]- ending = conv.sep if isinstance(conv.sep, str) else conv.sep[1]- output_texts[i] = output_texts[i].removesuffix(ending).strip()- output_texts[i] = output_texts[i].replace("\n", " ")- conv.messages[-1][1] = output_texts[i]- return output_texts, conv---def get_index(num_frames, num_segments):- seg_size = float(num_frames - 1) / num_segments- start = int(seg_size / 2)- offsets = np.array([start + int(np.round(seg_size * idx)) for idx in range(num_segments)])- return offsets---# def load_video(video_path, num_frames, return_msg=False, resolution=336):-# transforms = torchvision.transforms.Resize(size=resolution)-# vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)-# total_num_frames = len(vr)-# frame_indices = get_index(total_num_frames, num_frames)-# images_group = list()-# for frame_index in frame_indices:-# img = Image.fromarray(vr[frame_index].asnumpy())-# images_group.append(transforms(img))-# if return_msg:-# fps = float(vr.get_avg_fps())-# sec = ", ".join([str(round(f / fps, 1)) for f in frame_indices])-# # " " should be added in the start and end-# msg = f"The video contains {len(frame_indices)} frames sampled at {sec} seconds."-# return images_group, msg-# else:-# return images_group---def load_video(video_path, num_frames, return_msg=False, resolution=336):- transforms = torchvision.transforms.Resize(size=resolution)- # vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)- vframes, aframes, info = read_video_av(- video_path,- pts_unit="sec",- output_format="THWC"- )- print(vframes.shape)- total_num_frames = len(vframes)- # print("Video path: ", video_path)- # print("Total number of frames: ", total_num_frames)- frame_indices = get_index(total_num_frames, num_frames)- images_group = list()- for frame_index in frame_indices:- img = Image.fromarray(vframes[frame_index].numpy())- images_group.append(transforms(img))- if return_msg:- # fps = float(vframes.get_avg_fps())- # sec = ", ".join([str(round(f / fps, 1)) for f in frame_indices])- # # " " should be added in the start and end- # msg = f"The video contains {len(frame_indices)} frames sampled at {sec} seconds."- # return images_group, msg- exit('return_msg not implemented yet')- else:- return images_group---def collate_fn(batch):- return batch---class CSVDataset(Dataset):- def __init__(self, csv_path, num_frames):- self.df = pd.read_csv(csv_path)- self.data_list = self.df.path.tolist()- self.num_frames = num_frames-- def __len__(self):- return len(self.data_list)-- def __getitem__(self, idx):- if idx < 0 or idx >= len(self.data_list):- raise IndexError- try:- video = load_video(self.data_list[idx], self.num_frames, resolution=RESOLUTION)- except:- return None- return video-- def set_rank_and_world_size(self, rank, world_size):- self.rank = rank- self.world_size = world_size- self.data_per_gpu = len(self) // world_size- start_index = rank * self.data_per_gpu- end_index = (rank + 1) * self.data_per_gpu if rank != world_size - 1 else len(self)- self.data_list = self.data_list[start_index:end_index]-def parse_args():parser = ArgumentParser()parser.add_argument("--pretrained_model_name_or_path", type=str, required=True, default="llava-hf/llava-1.5-7b-hf")+ parser.add_argument("--prompt_template", type=str, default="general", choices=["general", "person"])parser.add_argument("--batch_size",type=int,@@ -236,15 +90,115 @@ def parse_args():"--error_message",type=str,required=False,- default='error occured during captioning',+ default="error occured during captioning",)+ parser.add_argument("--keep_failed", action="store_true", default=False)+ parser.add_argument(+ "--short_caption_ratio",+ type=float,+ required=False,+ default=0,+ )+args = parser.parse_args()return args+###############+# data processing+###############+++def get_index(num_frames, num_segments):+ seg_size = float(num_frames - 1) / num_segments+ start = int(seg_size / 2)+ offsets = np.array([start + int(np.round(seg_size * idx)) for idx in range(num_segments)])+ return offsets+++def load_video(video_path, num_frames, return_msg=False, resolution=336):+ transforms = torchvision.transforms.Resize(size=resolution)+ # vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)+ vframes, aframes, info = read_video_av(video_path, pts_unit="sec", output_format="THWC")+ # print(vframes.shape)+ total_num_frames = len(vframes)+ # print("Video path: ", video_path)+ # print("Total number of frames: ", total_num_frames)+ frame_indices = get_index(total_num_frames, num_frames)+ images_group = list()+ for frame_index in frame_indices:+ img = Image.fromarray(vframes[frame_index].numpy())+ images_group.append(transforms(img))+ if return_msg:+ # fps = float(vframes.get_avg_fps())+ # sec = ", ".join([str(round(f / fps, 1)) for f in frame_indices])+ # # " " should be added in the start and end+ # msg = f"The video contains {len(frame_indices)} frames sampled at {sec} seconds."+ # return images_group, msg+ exit("return_msg not implemented yet")+ else:+ return images_group+++class CSVDataset(Dataset):+ def __init__(self, csv_path, num_frames):+ self.df = pd.read_csv(csv_path)+ self.data_list = self.df.path.tolist()+ self.num_frames = num_frames++ def __len__(self):+ return len(self.data_list)++ def __getitem__(self, idx):+ try:+ video = load_video(self.data_list[idx], self.num_frames, resolution=RESOLUTION)+ return video+ except:+ return None++ @staticmethod+ def collate_fn(batch):+ batch = [item for item in batch if item is not None]+ if len(batch) == 0:+ return None, None++ if random.random() <= SHORT_CAPTION_RATIO:+ prompt = SHORT_PROMPT+ max_tokens = MAX_SHORT_TOKENS+ else:+ prompt = LONG_PROMPT+ max_tokens = MAX_LONG_TOKENS++ processed_batch = [processor(text=prompt, images=video, return_tensors="pt") for video in batch]+ batch = default_collate(processed_batch)++ for k, v in batch.items():+ if k in ("input_ids", "attention_mask"):+ batch[k] = v.squeeze(1)+ elif k == "pixel_values":+ b, t, c, h, w = v.shape+ batch[k] = v.reshape(b * t, c, h, w)+ return batch, max_tokens++ @staticmethod+ def post_process(output_texts, processor):+ output_texts = processor.batch_decode(+ output_texts, skip_special_tokens=True, clean_up_tokenization_spaces=False+ )+ if LONG_CONV_TEMPLATE.roles[-1] == "<|im_start|>assistant\n":+ split_tag = "<|im_start|> assistant\n"+ else:+ split_tag = LONG_CONV_TEMPLATE.roles[-1]+ ending = LONG_CONV_TEMPLATE.sep if isinstance(LONG_CONV_TEMPLATE.sep, str) else LONG_CONV_TEMPLATE.sep[1]+ for i, output_text in enumerate(output_texts):+ output_text = output_text.split(split_tag)[-1]+ output_text = output_text.removesuffix(ending).strip()+ output_text = output_text.replace("\n", " ")+ output_texts[i] = output_text+ return output_texts++def load_model_and_dataset(- rank,- world_size,pretrained_model_name_or_path,num_frames,use_lora,@@ -262,44 +216,132 @@ def load_model_and_dataset(lora_alpha=lora_alpha,pooling_shape=pooling_shape,)- logger.info("done loading llava")# position embedding- model = model.to(torch.device(rank))+ model = model.to(device=get_current_device())model = model.eval()dataset = CSVDataset(csv_path, num_frames)- dataset.set_rank_and_world_size(rank, world_size)return model, processor, datasetdef infer(model,- processor,- video_list,- conv_mode,- print_res=False,+ batch,+ max_tokens,):- # check if any video in video_list is None, if so, raise an exception- if any([video is None for video in video_list]):- raise Exception("Video not loaded properly")- conv = conv_template.copy()- conv.user_query("Describe the video in details.", is_mm=True)-- llm_responses, conv = pllava_answer(- conv=conv,- model=model,- processor=processor,- video_list=video_list,- max_new_tokens=256,- do_sample=False,- print_res=print_res,+ batch = batch.to(get_current_device())+ with torch.no_grad():+ output_texts = model.generate(+ **batch,+ media_type="video",+ do_sample=False,+ max_new_tokens=max_tokens,+ num_beams=1,+ min_length=1,+ repetition_penalty=1.0,+ length_penalty=1,+ temperature=1.0,+ )+ output_texts = [x.cpu() for x in output_texts]+ return output_texts+++def inference_loop(args, model, dataset, q: Queue):+ dataloader = DataLoader(+ dataset,+ num_workers=2,+ batch_size=args.batch_size,+ collate_fn=CSVDataset.collate_fn,+ pin_memory=True,+ sampler=DistributedSampler(dataset, num_replicas=dist.get_world_size(), rank=dist.get_rank(), shuffle=False),)- return llm_responses+ for i, (batch, max_tokens) in enumerate(tqdm(dataloader, disable=dist.get_rank() != 0)):+ try:+ if batch is None:+ raise Exception("Video not loaded properly")+ preds = infer(+ model,+ batch,+ max_tokens=max_tokens,+ )+ except Exception as e:+ logger.error(f"error at rank {dist.get_rank()} sample {i}: {str(e)}")+ traceback.print_exception(e)+ # preds = args.error_message duplicated for each video in the batch+ preds = [args.error_message] * len(batch)+ q.put(preds)+ # finish the queue+ q.put(None)+++def post_process_loop(processor, q: Queue, result_q: Queue):+ results = []+ while True:+ preds = q.get()+ if preds is not None:+ preds = CSVDataset.post_process(preds, processor)+ results.extend(preds)+ else:+ break+ result_q.put(results)+++def main():+ args = parse_args()+ if args.prompt_template == "general":+ long_pt = "Describe this video. Pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. Here are some examples of good descriptions: 1. A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about. 2. Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field. 3. Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff's edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway."+ short_pt = "Describe the video focusing on key objects and actions. The description should be brief yet detailed enough for AI to recreate the video. Keep the description to no more than three sentences. Here are some examples of good descriptions: 1. A stylish woman walks confidently down a neon-lit Tokyo street, wearing a black leather jacket and a long red dress, with pedestrians and reflective wet pavement around her. 2. Giant wooly mammoths tread through a snowy meadow, their fur blowing lightly in the wind, with snowy trees and mountains in the background. 3. A drone captures waves crashing against rugged cliffs along Big Sur, with golden sunset light illuminating the rocky shore and a lighthouse in the distance."+ elif args.prompt_template == "person":+ # pt = "Describe this video in detail. Pay special attention to all details of the person, including the face, the body, the pose, the action, and the outfit. Also pay attention to the camera angle. The description should be useful for AI to re-generate the video. The description should contain no more than six sentences."+ long_pt = "Describe this video in detail. Pay special attention to all details of the person, including 1. apperance, such as hair, face, body, and outfit; 2. expression and emotion; 3. action and pose. Also pay attention to the background and the surrounding environment. Also pay attention to the camera angle. The description should be useful for AI to re-generate the video. The description should contain no more than six sentences."+ short_pt = "Describe this video in detail. Pay special attention to key details of the person, including 1. apperance, such as hair, face, body, and outfit; 2. expression and emotion; 3. action and pose. Also pay attention to the background and the surrounding environment. Also pay attention to the camera angle. The description should be useful for AI to re-generate the video. The description should contain no more than three sentences."+ else:+ raise ValueError++ assert (+ args.short_caption_ratio >= 0 and args.short_caption_ratio <= 1+ ), "`short_caption_ratio` should be in range [0, 1]"++ global LONG_CONV_TEMPLATE+ global SHORT_CONV_TEMPLATE+ global LONG_PROMPT+ global SHORT_PROMPT+ global RESOLUTION+ global SHORT_CAPTION_RATIO+ global MAX_LONG_TOKENS+ global MAX_SHORT_TOKENS++ LONG_CONV_TEMPLATE = Conversation(+ system=long_pt,+ roles=("USER:", "ASSISTANT:"),+ messages=[],+ sep=(" ", "</s>"),+ mm_token="<image>",+ )+ LONG_CONV_TEMPLATE.user_query("Describe the video in detail.", is_mm=True)+ LONG_PROMPT = LONG_CONV_TEMPLATE.get_prompt()++ SHORT_CONV_TEMPLATE = Conversation(+ system=short_pt,+ roles=("USER:", "ASSISTANT:"),+ messages=[],+ sep=(" ", "</s>"),+ mm_token="<image>",+ )+ SHORT_CONV_TEMPLATE.user_query("Describe the video in detail.", is_mm=True)+ SHORT_PROMPT = SHORT_CONV_TEMPLATE.get_prompt()+ RESOLUTION = 672+ SHORT_CAPTION_RATIO = args.short_caption_ratio+ MAX_LONG_TOKENS = 256+ MAX_SHORT_TOKENS = 128-def run(rank, args, world_size, output_queue):+ colossalai.launch_from_torch()+ rank = dist.get_rank()++ # setup debugif rank == 0:import os@@ -309,19 +351,16 @@ def run(rank, args, world_size, output_queue):ptvsd.enable_attach(address=("localhost", int(os.getenv("DEBUG_ADDRESS"))), redirect_output=True)ptvsd.wait_for_attach()print("waiting for debugger attachment")- if rank != 0:+ else:transformers.utils.logging.set_verbosity_error()logger.setLevel(transformers.logging.ERROR)- print_res = False- conv_mode = args.conv_mode+ # setup model and datasetif args.pooling_shape is not None:pooling_shape = tuple([int(x) for x in args.pooling_shape.split("-")])- logger.info(f"loading model and constructing dataset to gpu {rank}...")+ global processormodel, processor, dataset = load_model_and_dataset(- rank,- world_size,pretrained_model_name_or_path=args.pretrained_model_name_or_path,num_frames=args.num_frames,use_lora=args.use_lora,@@ -330,95 +369,39 @@ def run(rank, args, world_size, output_queue):pooling_shape=pooling_shape,csv_path=args.csv_path,)- logger.info(f"done model and dataset...")- logger.info("constructing dataset...")- logger.info("single test...")- dataloader = torch.utils.data.DataLoader(- dataset,- num_workers=2,- batch_size=args.batch_size,- collate_fn=collate_fn,- shuffle=False,- )-- total = 0- result_list = []- print(len(dataset))- for batch in tqdm(dataloader):- total += 1- try:- preds = infer(- model,- processor,- batch,- conv_mode=conv_mode,- print_res=print_res,- )- except Exception as e:- logger.error(f"error in {batch}: {str(e)}")- # preds = args.error_message duplicated for each video in the batch- preds = [args.error_message] * len(batch)- result_list.extend(preds)- output_queue.put((rank, result_list))- return result_list---def main():- multiprocess = True- mp.set_start_method("spawn")- args = parse_args()- # csv_path = '/home/tom/PLLaVA/test_short_caption_part2.csv'- if multiprocess:- n_gpus = torch.cuda.device_count()- world_size = n_gpus- print(f"world_size: {world_size}")- # Create a queue to collect results from each process- output_queue = Queue()-- # with Pool(world_size) as pool:- # func = functools.partial(run, args=args, world_size=world_size)- # result_lists = pool.map(func, range(world_size))- processes = []- for i in range(world_size):- # Each process will now also take the output queue as an argument- p = Process(target=run, args=(i, args, world_size, output_queue))- p.daemon = False- processes.append(p)- p.start()-- results_by_rank = {}- for _ in range(world_size):- rank, results = output_queue.get() # Retrieve results as they finish- results_by_rank[rank] = results- print(f"Results received from rank {rank}")- # ORDER THE RESULTS BY RANK- logger.info("finished running")- for p in processes:- p.join()-- results_list = list(itertools.chain.from_iterable(results_by_rank[i] for i in range(world_size)))- # results_list = list(itertools.chain([results_by_rank[i] for i in range(world_size)]))- # (data[key] for key in sorted_keys)- # results_list = [item for sublist in results_by_rank.values() for item in sublist]+ logger.info(f"Dataset loaded with {len(dataset)} samples.")+ q = Queue()+ result_q = Queue()+ p = Process(target=post_process_loop, args=(processor, q, result_q))+ p.start()++ inference_loop(args, model, dataset, q)+ results = result_q.get()+ p.join()++ # gather results+ results_list = [None for _ in range(dist.get_world_size())] if rank == 0 else None+ dist.gather_object(results, results_list, dst=0)+ if rank == 0:+ # reorder and merge+ final_results = list(itertools.chain.from_iterable(zip(*results_list)))+ assert len(final_results) >= len(dataset)+ # remove padding+ final_results = final_results[: len(dataset)]++ # write the results to the csv file+ df = pd.read_csv(args.csv_path)+ # add a new column to the dataframe+ df["text"] = final_results+ drop_failed = not args.keep_failed+ if drop_failed:+ df = df[df["text"] != args.error_message]+ print(f"Dropped {len(dataset) - len(df)} samples")+ # write the dataframe to a new csv file called '*_pllava_13b_caption.csv'+ new_csv_path = args.csv_path.replace(".csv", "_text.csv")+ df.to_csv(new_csv_path, index=False)+ print(f"Results saved to {new_csv_path}")- else:- results_list = run(0, world_size=1, args=args) # debug-- print(results_list)-- df = pd.read_csv(args.csv_path)- # add a new column to the dataframe- df["text"] = results_list- drop_failed = True- if drop_failed:- # iterate through the dataframe and delete the entire row if captioning failed- for i in tqdm(range(len(df))):- if df["text"][i] == args.error_message:- df = df.drop(i)- # write the dataframe to a new csv file called '*_pllava_13b_caption.csv'- new_csv_path = args.csv_path.replace(".csv", "_text.csv")- df.to_csv(new_csv_path, index=False)- print(f"Results saved to {new_csv_path}")if __name__ == "__main__":main()diff --git a/tools/caption/utils.py b/tools/caption/utils.pyindex b1137de..882c259 100644--- a/tools/caption/utils.py+++ b/tools/caption/utils.py@@ -26,6 +26,14 @@ PROMPTS = {"text": "Describe this video and its style in a very detailed manner. Pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences.","type": "video",},+ "video-template": { # crafted by gpt-4o+ "text": "Frames of a video clip are given. Write a concise yet detailed paragraph, following this sequence: 1. Subject: Describe the main subject in detail, including their appearance and what they are doing. 2. Subject Movement: Provide a detailed account of the subject’s movements or actions. 3. Scene Description: Describe the environment, setting, or location in as much detail as possible. 4. Atmosphere: Briefly describe the overall mood (e.g., lively, calm, cinematic). 5. Light and Shadow: Provide a brief description of the lighting (e.g., natural, artificial, morning light, sunset). 6. Camera Technique: Mention the camera angles and shots (e.g., close-up, wide shot) briefly.\nStart directly with descriptions, avoiding phrases like ‘the video’ or ‘the subject.’ Focus on providing detailed information for the first three points, while keeping the last three points more concise. Avoid any guesses or assumptions (e.g., avoid words like ‘possibly’ or ‘suggest’). Ensure the description is clear and based on observable details for a video-generation model to recreate the scene.",+ "type": "video",+ },+ "video-captrans-template": { # crafted by gpt-4o+ "text": "A paragraph description of a video clip is given. Rewrite the description in a concise yet detailed manner, following this sequence: 1. Subject: Clearly describe the main subject, including their appearance and what they are doing. 2. Subject Movement: Provide a detailed account of the subject’s movements or actions. 3. Scene Description: Describe the environment or setting as thoroughly as possible. 4. Atmosphere: Briefly summarize the overall mood (e.g., lively, calm, cinematic). 5. Light and Shadow: Provide a brief note on the lighting (e.g., natural, artificial, morning light, sunset). 6. Camera Technique: Mention the camera angles and shots (e.g., close-up, wide shot) briefly.\nEnsure that the rewritten description only includes visible elements from the original paragraph and does not add any new details that were not mentioned. Keep the descriptions of the last three points simple. Avoid any guesses or assumptions (e.g., words like ‘possibly’ or ‘suggest’).",+ "type": "video",+ },"video-text": {"text": "Describe this video and its style in a very detailed manner. Some information about the image is '{}'. Pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences.","type": "video",@@ -66,7 +74,7 @@ def read_file(input_path):class VideoTextDataset(torch.utils.data.Dataset):- def __init__(self, csv_path, transform=None, num_frames=3, get_text_input_ids=None, resize=None):+ def __init__(self, csv_path, transform=None, num_frames=3, get_text_input_ids=None, resize=None, text_only=False):self.csv_path = csv_pathself.transform = transformself.data = read_file(csv_path)@@ -77,28 +85,35 @@ class VideoTextDataset(torch.utils.data.Dataset):self.resize = transforms.Resize(resize, transforms.InterpolationMode.BICUBIC) if resize is not None else Noneif "text" in self.data.columns:self.use_text = True+ self.text_only = text_onlydef getitem(self, index):sample = self.data.iloc[index]path = sample["path"]- if not is_video(path):- images = [pil_loader(path)]- length = 1++ if not self.text_only:+ if not is_video(path):+ images = [pil_loader(path)]+ length = 1+ else:+ images, length = extract_frames(+ sample["path"], points=self.points, backend="opencv", return_length=True+ )+ if self.resize_size is not None:+ images_r = []+ for img in images:+ if img.size[0] > self.resize_size or img.size[1] > self.resize_size:+ img = self.resize(img)+ images_r.append(img)+ images = images_r+ imgs_size = [img.size for img in images]+ if self.transform is not None:+ images = self.transform(images)++ # we put images into a list as pytorch dataloader does not accept Pill+ out = dict(path=path, image=images, length=length, img_size=imgs_size)else:- images, length = extract_frames(sample["path"], points=self.points, backend="opencv", return_length=True)- if self.resize_size is not None:- images_r = []- for img in images:- if img.size[0] > self.resize_size or img.size[1] > self.resize_size:- img = self.resize(img)- images_r.append(img)- images = images_r- imgs_size = [img.size for img in images]- if self.transform is not None:- images = self.transform(images)-- # we put images into a list as pytorch dataloader does not accept Pill- out = dict(path=path, image=images, length=length, img_size=imgs_size)+ out = dict(path=path)if self.get_text_input_ids is not None:if self.use_text:out["text"] = self.get_text_input_ids(sample["text"])diff --git a/tools/datasets/check_integrity.py b/tools/datasets/check_integrity.pynew file mode 100644index 0000000..4735fcc--- /dev/null+++ b/tools/datasets/check_integrity.py@@ -0,0 +1,79 @@+import argparse+import subprocess++import pandas as pd+from tqdm import tqdm++tqdm.pandas()++try:+ from pandarallel import pandarallel++ PANDA_USE_PARALLEL = True+except ImportError:+ PANDA_USE_PARALLEL = False++import shutil++if not shutil.which("ffmpeg"):+ raise ImportError("FFmpeg is not installed")+++def apply(df, func, **kwargs):+ if PANDA_USE_PARALLEL:+ return df.parallel_apply(func, **kwargs)+ return df.progress_apply(func, **kwargs)+++def check_video_integrity(video_path):+ # try:+ can_open_result = subprocess.run(+ ["ffmpeg", "-v", "error", "-i", video_path, "-t", "0", "-f", "null", "-"], # open video and capture 0 seconds+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ text=True,+ )+ fast_scan_result = subprocess.run(+ ["ffmpeg", "-v", "error", "-analyzeduration", "10M", "-probesize", "10M", "-i", video_path, "-f", "null", "-"],+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ text=True,+ )+ if can_open_result.stderr == "" and fast_scan_result.stderr == "":+ return True+ else:+ return False+ # except Exception as e:+ # return False+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("input", type=str, help="path to the input dataset")+ parser.add_argument("--disable-parallel", action="store_true", help="disable parallel processing")+ parser.add_argument("--num-workers", type=int, default=None, help="number of workers")+ args = parser.parse_args()++ if args.disable_parallel:+ PANDA_USE_PARALLEL = False+ if PANDA_USE_PARALLEL:+ if args.num_workers is not None:+ pandarallel.initialize(nb_workers=args.num_workers, progress_bar=True)+ else:+ pandarallel.initialize(progress_bar=True)++ data = pd.read_csv(args.input)+ assert "path" in data.columns+ data["integrity"] = apply(data["path"], check_video_integrity)++ integrity_file_path = args.input.replace(".csv", "_intact.csv")+ broken_file_path = args.input.replace(".csv", "_broken.csv")++ intact_data = data[data["integrity"] == True].drop(columns=["integrity"])+ intact_data.to_csv(integrity_file_path, index=False)+ broken_data = data[data["integrity"] == False].drop(columns=["integrity"])+ broken_data.to_csv(broken_file_path, index=False)++ print(+ f"Integrity check completed. Intact videos saved to: {integrity_file_path}, broken videos saved to {broken_file_path}."+ )diff --git a/tools/datasets/convert.py b/tools/datasets/convert.pyindex fad128f..76f81d1 100644--- a/tools/datasets/convert.py+++ b/tools/datasets/convert.py@@ -98,7 +98,8 @@ def process_general_images(root, output):returnpath_list = get_filelist(root, IMG_EXTENSIONS)fname_list = [os.path.splitext(os.path.basename(x))[0] for x in path_list]- df = pd.DataFrame(dict(id=fname_list, path=path_list))+ relpath_list = [os.path.relpath(x, root) for x in path_list]+ df = pd.DataFrame(dict(path=path_list, id=fname_list, relpath=relpath_list))os.makedirs(os.path.dirname(output), exist_ok=True)df.to_csv(output, index=False)diff --git a/tools/datasets/csv2txt.py b/tools/datasets/csv2txt.pynew file mode 100644index 0000000..116e7e0--- /dev/null+++ b/tools/datasets/csv2txt.py@@ -0,0 +1,14 @@+import argparse++import pandas as pd++parser = argparse.ArgumentParser(description="Convert CSV file to txt file")+parser.add_argument("csv_file", type=str, help="CSV file to convert")+parser.add_argument("txt_file", type=str, help="TXT file to save")+args = parser.parse_args()++data = pd.read_csv(args.csv_file)+text = data["text"].to_list()+text = "\n".join(text)+with open(args.txt_file, "w") as f:+ f.write(text)diff --git a/tools/datasets/datautil.py b/tools/datasets/datautil.pyindex f972dfd..3d78f30 100644--- a/tools/datasets/datautil.py+++ b/tools/datasets/datautil.py@@ -1,6 +1,7 @@import argparseimport htmlimport json+import mathimport osimport randomimport re@@ -34,6 +35,20 @@ def apply(df, func, **kwargs):TRAIN_COLUMNS = ["path", "text", "num_frames", "fps", "height", "width", "aspect_ratio", "resolution", "text_len"]+PRE_TRAIN_COLUMNS = [+ "path",+ "text",+ "num_frames",+ "fps",+ "height",+ "width",+ "aspect_ratio",+ "resolution",+ "text_len",+ "aes",+ "flow",+ "pred_score",+]# ======================================================# --info@@ -50,30 +65,6 @@ def get_video_length(cap, method="header"):return length-def get_info_old(path):- try:- ext = os.path.splitext(path)[1].lower()- if ext in IMG_EXTENSIONS:- im = cv2.imread(path)- if im is None:- return 0, 0, 0, np.nan, np.nan, np.nan- height, width = im.shape[:2]- num_frames, fps = 1, np.nan- else:- cap = cv2.VideoCapture(path)- num_frames, height, width, fps = (- get_video_length(cap, method="header"),- int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)),- int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),- float(cap.get(cv2.CAP_PROP_FPS)),- )- hw = height * width- aspect_ratio = height / width if width > 0 else np.nan- return num_frames, height, width, aspect_ratio, fps, hw- except:- return 0, 0, 0, np.nan, np.nan, np.nan--def get_info(path):try:ext = os.path.splitext(path)[1].lower()@@ -151,23 +142,23 @@ def get_video_info(path, backend="torchvision"):# ======================================================LLAVA_PREFIX = [- "The video shows",- "The video captures",- "The video features",- "The video depicts",- "The video presents",- "The video features",+ "The video shows ",+ "The video captures ",+ "The video features ",+ "The video depicts ",+ "The video presents ",+ "The video features ","The video is ",- "In the video,",- "The image shows",- "The image captures",- "The image features",- "The image depicts",- "The image presents",- "The image features",+ "In the video, ",+ "The image shows ",+ "The image captures ",+ "The image features ",+ "The image depicts ",+ "The image presents ",+ "The image features ","The image is ",- "The image portrays",- "In the image,",+ "The image portrays ",+ "In the image, ",]@@ -261,7 +252,7 @@ BAD_PUNCT_REGEX = re.compile() # noqa-def clean_caption(caption):+def text_refine_t5(caption):import urllib.parse as ulfrom bs4 import BeautifulSoup@@ -382,13 +373,22 @@ def clean_caption(caption):def text_preprocessing(text, use_text_preprocessing: bool = True):if use_text_preprocessing:# The exact text cleaning as was in the training stage:- text = clean_caption(text)- text = clean_caption(text)+ text = text_refine_t5(text)+ text = text_refine_t5(text)return textelse:return text.lower().strip()+def has_human(text):+ first_sentence = text.split(".")[0]+ human_words = ["man", "woman", "child", "girl", "boy"]+ for word in human_words:+ if word in first_sentence:+ return True+ return False++# ======================================================# load caption# ======================================================@@ -413,25 +413,81 @@ def load_caption(path, ext):DROP_SCORE_PROB = 0.2-def score_to_text(data):+def transform_aes(aes):+ # < 4 filter out+ if aes < 4:+ return "terrible"+ elif aes < 4.5:+ return "very poor"+ elif aes < 5:+ return "poor"+ elif aes < 5.5:+ return "fair"+ elif aes < 6:+ return "good"+ elif aes < 6.5:+ return "very good"+ else:+ return "excellent"+++def transform_motion(motion):+ # < 0.3 filter out+ if motion < 0.5:+ return "very low"+ elif motion < 2:+ return "low"+ elif motion < 5:+ return "fair"+ elif motion < 10:+ return "high"+ elif motion < 20:+ return "very high"+ else:+ return "extremely high"+++def score2text(data):text = data["text"]- scores = []+ if not text.endswith("."):+ text += "."# aestheticif "aes" in data:- aes = data["aes"]+ aes = transform_aes(data["aes"])if random.random() > DROP_SCORE_PROB:- score_text = f"aesthetic score: {aes:.1f}"- scores.append(score_text)+ score_text = f" the aesthetic score is {aes}."+ text += score_text+ # flowif "flow" in data:- flow = data["flow"]+ flow = transform_motion(data["flow"])if random.random() > DROP_SCORE_PROB:- score_text = f"motion score: {flow:.1f}"- scores.append(score_text)- if len(scores) > 0:- text = f"{text} [{', '.join(scores)}]"+ score_text = f" the motion strength is {flow}."+ text += score_textreturn text+def undo_score2text(data):+ text = data["text"]+ sentences = text.strip().split(".")[:-1]++ keywords = ["aesthetic score", "motion strength"]+ num_scores = len(keywords)+ num_texts_from_score = 0+ for idx in range(1, num_scores + 1):+ s = sentences[-idx]++ for key in keywords:+ if key in s:+ num_texts_from_score += 1+ break++ new_sentences = sentences[:-num_texts_from_score] if num_texts_from_score > 0 else sentences+ new_text = ".".join(new_sentences)+ if not new_text.endswith("."):+ new_text += "."+ return new_text++# ======================================================# read & write# ======================================================@@ -459,28 +515,82 @@ def save_file(data, output_path):def read_data(input_paths):+ if len(input_paths) == 0:+ print(f"No meta file to process. Exit.")+ exit()+data = []input_name = ""input_list = []for input_path in input_paths:input_list.extend(glob(input_path))- print("Input files:", input_list)+ cnt = len(input_list)+ print(f"==> Total {cnt} input files:")+ for x in input_list:+ print(x)+for i, input_path in enumerate(input_list):if not os.path.exists(input_path):- continue+ raise FileNotFoundErrordata.append(read_file(input_path))- input_name += os.path.basename(input_path).split(".")[0]+ basename = os.path.basename(input_path)+ input_name += os.path.splitext(basename)[0]if i != len(input_list) - 1:input_name += "+"- print(f"Loaded {len(data[-1])} samples from '{input_path}'.")- if len(data) == 0:- print(f"No samples to process. Exit.")- exit()+ print(f"==> Loaded meta (shape={data[-1].shape}) from '{input_path}'")+data = pd.concat(data, ignore_index=True, sort=False)- print(f"Total number of samples: {len(data)}")+ print(f"==> Merged {cnt} files. shape={data.shape}")return data, input_name+def is_verbose_sentence(s):+ if ("is no " in s) or ("are no " in s):+ return True+ if "does not " in s:+ return True+ if "solely " in s:+ return True+ if "only " in s:+ return True+ if "not visible" in s:+ return True+ if "no " in s and "visible" in s:+ return True+ return False+++def is_verbose_caption(caption):+ caption = caption.strip()+ sentences = caption.split(".")+ if not caption.endswith("."):+ sentences = sentences[:-1]++ cnt = 0+ for sentence in sentences:+ if is_verbose_sentence(sentence):+ cnt += 1+ if cnt >= 2:+ return True+ return False+++def refine_sentences(caption):+ caption = caption.strip()+ sentences = caption.split(".")+ if not caption.endswith("."):+ sentences = sentences[:-1]++ new_caption = ""+ for i, sentence in enumerate(sentences):+ if sentence.strip() == "":+ continue+ if is_verbose_sentence(sentence):+ continue+ new_caption += f"{sentence}."+ return new_caption++# ======================================================# main# ======================================================@@ -491,29 +601,24 @@ def main(args):# reading datadata, input_name = read_data(args.input)- # make difference- if args.difference is not None:- data_diff = pd.read_csv(args.difference)- print(f"Difference csv contains {len(data_diff)} samples.")+ # get output path+ output_path = get_output_path(args, input_name)++ # path subtract (difference set)+ if args.path_subtract is not None:+ data_diff = pd.read_csv(args.path_subtract)+ print(f"Meta to subtract: shape={data_diff.shape}.")data = data[~data["path"].isin(data_diff["path"])]- input_name += f"-{os.path.basename(args.difference).split('.')[0]}"- print(f"Filtered number of samples: {len(data)}.")- # make intersection- if args.intersection is not None:- data_new = pd.read_csv(args.intersection)- print(f"Intersection csv contains {len(data_new)} samples.")- cols_to_use = data_new.columns.difference(data.columns)+ # path intersect+ if args.path_intersect is not None:+ data_new = pd.read_csv(args.path_intersect)+ print(f"Meta to intersect: shape={data_new.shape}.")+ new_cols = data_new.columns.difference(data.columns)col_on = "path"- # if 'id' in data.columns and 'id' in data_new.columns:- # col_on = 'id'- cols_to_use = cols_to_use.insert(0, col_on)- data = pd.merge(data, data_new[cols_to_use], on=col_on, how="inner")- print(f"Intersection number of samples: {len(data)}.")-- # get output path- output_path = get_output_path(args, input_name)+ new_cols = new_cols.insert(0, col_on)+ data = pd.merge(data, data_new[new_cols], on=col_on, how="inner")# preparationif args.lang is not None:@@ -547,30 +652,34 @@ def main(args):data["fps"],data["resolution"],) = zip(*info)- if args.ext:++ # filtering path+ if args.path_filter_empty:assert "path" in data.columns- data = data[apply(data["path"], os.path.exists)]+ data = data[data["path"].str.len() > 0]+ data = data[~data["path"].isna()]+ if args.path_filter_substr:+ data = data[~data["path"].str.contains(args.path_filter_substr)]+ if args.path_keep_substr:+ data = data[data["path"].str.contains(args.path_keep_substr)]+ if args.path_dedup:+ assert "path" in data.columns+ data = data.drop_duplicates(subset=["path"])- # filtering- if args.remove_url:+ # filtering text+ if args.text_filter_url:assert "text" in data.columnsdata = data[~data["text"].str.contains(r"(?P<url>https?://[^\s]+)", regex=True)]if args.lang is not None:assert "text" in data.columnsdata = data[data["text"].progress_apply(detect_lang)] # cannot parallelize- if args.remove_empty_path:- assert "path" in data.columns- data = data[data["path"].str.len() > 0]- data = data[~data["path"].isna()]- if args.remove_empty_caption:+ if args.text_filter_empty:assert "text" in data.columnsdata = data[data["text"].str.len() > 0]data = data[~data["text"].isna()]- if args.remove_path_duplication:- assert "path" in data.columns- data = data.drop_duplicates(subset=["path"])- if args.path_subset:- data = data[data["path"].str.contains(args.path_subset)]+ if args.text_filter_substr:+ assert "text" in data.columns+ data = data[~data["text"].str.contains(args.text_filter_substr)]# processingif args.relpath is not None:@@ -581,20 +690,21 @@ def main(args):data["id"] = apply(data["path"], lambda x: os.path.splitext(os.path.basename(x))[0])if args.merge_cmotion:data["text"] = apply(data, lambda x: merge_cmotion(x["text"], x["cmotion"]), axis=1)- if args.refine_llm_caption:+ if args.text_remove_prefix:assert "text" in data.columnsdata["text"] = apply(data["text"], remove_caption_prefix)- if args.append_text is not None:+ if args.text_append is not None:assert "text" in data.columns- data["text"] = data["text"] + args.append_text- if args.score_to_text:- data["text"] = apply(data, score_to_text, axis=1)- if args.clean_caption:+ data["text"] = data["text"] + args.text_append+ if args.text_refine_t5:assert "text" in data.columnsdata["text"] = apply(data["text"],partial(text_preprocessing, use_text_preprocessing=True),)+ if args.text_image2video:+ assert "text" in data.columns+ data["text"] = apply(data["text"], lambda x: x.replace("still image", "video").replace("image", "video"))if args.count_num_token is not None:assert "text" in data.columnsdata["text_len"] = apply(data["text"], lambda x: len(tokenizer(x)["input_ids"]))@@ -606,6 +716,12 @@ def main(args):data_new = data_new[["path", "text"]].set_index("path")data.update(data_new)data = data.reset_index()+ if args.text_refine_sentences:+ data["text"] = apply(data["text"], refine_sentences)+ if args.text_score2text:+ data["text"] = apply(data, score2text, axis=1)+ if args.text_undo_score2text:+ data["text"] = apply(data, undo_score2text, axis=1)# sortif args.sort is not None:@@ -620,7 +736,10 @@ def main(args):if args.fsmax is not None:assert "filesize" in data.columnsdata = data[data["filesize"] <= args.fsmax]- if args.remove_empty_caption:+ if args.fsmin is not None:+ assert "filesize" in data.columns+ data = data[data["filesize"] >= args.fsmin]+ if args.text_filter_empty:assert "text" in data.columnsdata = data[data["text"].str.len() > 0]data = data[~data["text"].isna()]@@ -630,6 +749,19 @@ def main(args):if args.fmax is not None:assert "num_frames" in data.columnsdata = data[data["num_frames"] <= args.fmax]+ if args.filter_dyn_fps is not False:+ assert "fps" in data.columns and "num_frames" in data.columns++ def dyn_fps_filter(row, max_fps=args.dyn_fps_max_fps, keep_frames=args.dyn_fps_keep_frames):+ # get scale factor+ if math.isnan(row["fps"]): # image+ return True+ scale_factor = math.ceil(row["fps"] / max_fps)+ min_frames = keep_frames * scale_factor+ return row["num_frames"] >= min_frames++ dyn_fps_mask = data.apply(dyn_fps_filter, axis=1)+ data = data[dyn_fps_mask]if args.fpsmax is not None:assert "fps" in data.columnsdata = data[(data["fps"] <= args.fpsmax) | np.isnan(data["fps"])]@@ -642,44 +774,75 @@ def main(args):if args.aesmin is not None:assert "aes" in data.columnsdata = data[data["aes"] >= args.aesmin]+ if args.prefmin is not None:+ assert "pred_score" in data.columns+ data = data[data["pred_score"] >= args.prefmin]if args.matchmin is not None:assert "match" in data.columnsdata = data[data["match"] >= args.matchmin]if args.flowmin is not None:assert "flow" in data.columnsdata = data[data["flow"] >= args.flowmin]- if args.remove_text_duplication:+ if args.facemin is not None:+ assert "face_area_ratio" in data.columns+ data = data[data["face_area_ratio"] >= args.facemin]+ if args.text_dedup:data = data.drop_duplicates(subset=["text"], keep="first")if args.img_only:data = data[data["path"].str.lower().str.endswith(IMG_EXTENSIONS)]if args.vid_only:data = data[~data["path"].str.lower().str.endswith(IMG_EXTENSIONS)]+ if args.filter_too_verbose:+ data = data[data["text"].apply(is_verbose_caption)]+ if args.h_le_w:+ data = data[data["height"] <= data["width"]]+ if args.filter_human:+ data = data[data["text"].apply(has_human)]++ if args.ext:+ assert "path" in data.columns+ data = data[apply(data["path"], os.path.exists)]# process dataif args.shuffle:data = data.sample(frac=1).reset_index(drop=True) # shuffleif args.head is not None:data = data.head(args.head)+ if args.sample is not None:+ data = data.sample(args.sample).reset_index(drop=True)# train columnsif args.train_column:+ assert args.pre_train_column is Falseall_columns = data.columnscolumns_to_drop = all_columns.difference(TRAIN_COLUMNS)data = data.drop(columns=columns_to_drop)+ elif args.pre_train_column:+ assert args.train_column is False+ all_columns = data.columns+ columns_to_drop = all_columns.difference(PRE_TRAIN_COLUMNS)+ data = data.drop(columns=columns_to_drop)- print(f"Filtered number of samples: {len(data)}.")-- # shard data- if args.shard is not None:- sharded_data = np.array_split(data, args.shard)- for i in range(args.shard):- output_path_part = output_path.split(".")- output_path_s = ".".join(output_path_part[:-1]) + f"_{i}." + output_path_part[-1]- save_file(sharded_data[i], output_path_s)- print(f"Saved {len(sharded_data[i])} samples to {output_path_s}.")+ if args.chunk is not None:+ assert len(args.input) == 1+ input_path = args.input[0]+ res = np.array_split(data, args.chunk)+ for idx in range(args.chunk):+ out_path = f"_chunk-{idx}-{args.chunk}".join(os.path.splitext(input_path))+ shape = res[idx].shape+ print(f"==> Saving meta file (shape={shape}) to '{out_path}'")+ if args.format == "csv":+ res[idx].to_csv(out_path, index=False)+ elif args.format == "parquet":+ res[idx].to_parquet(out_path, index=False)+ else:+ raise NotImplementedError+ print(f"New meta (shape={shape}) saved to '{out_path}'")else:+ shape = data.shape+ print(f"==> Saving meta file (shape={shape}) to '{output_path}'")save_file(data, output_path)- print(f"Saved {len(data)} samples to {output_path}.")+ print(f"==> New meta (shape={shape}) saved to '{output_path}'")def parse_args():@@ -687,7 +850,7 @@ def parse_args():parser.add_argument("input", type=str, nargs="+", help="path to the input dataset")parser.add_argument("--output", type=str, default=None, help="output path")parser.add_argument("--format", type=str, default="csv", help="output format", choices=["csv", "parquet"])- parser.add_argument("--disable-parallel", action="store_true", help="disable parallel processing")+ parser.add_argument("--disable_parallel", action="store_true", help="disable parallel processing")parser.add_argument("--num-workers", type=int, default=None, help="number of workers")parser.add_argument("--seed", type=int, default=42, help="random seed")@@ -695,11 +858,10 @@ def parse_args():parser.add_argument("--shard", type=int, default=None, help="shard the dataset")parser.add_argument("--sort", type=str, default=None, help="sort by column")parser.add_argument("--sort-ascending", type=str, default=None, help="sort by column (ascending order)")- parser.add_argument("--difference", type=str, default=None, help="get difference from the dataset")- parser.add_argument(- "--intersection", type=str, default=None, help="keep the paths in csv from the dataset and merge columns"- )- parser.add_argument("--train-column", action="store_true", help="only keep the train column")+ parser.add_argument("--path_subtract", type=str, default=None, help="substract path (difference set)")+ parser.add_argument("--path_intersect", type=str, default=None, help="intersect path and merge columns")+ parser.add_argument("--train_column", action="store_true", help="only keep the train column")+ parser.add_argument("--pre_train_column", action="store_true", help="only keep the pre-train column")# IO-relatedparser.add_argument("--info", action="store_true", help="get the basic information of each video and image")@@ -714,54 +876,70 @@ def parse_args():parser.add_argument("--abspath", type=str, default=None, help="modify the path to absolute path by root given")parser.add_argument("--path-to-id", action="store_true", help="add id based on path")parser.add_argument(- "--path-subset", type=str, default=None, help="extract a subset data containing the given `path-subset` value"- )+ "--path_filter_empty", action="store_true", help="remove rows with empty path"+ ) # caused by transform, cannot read pathparser.add_argument(- "--remove-empty-path",- action="store_true",- help="remove rows with empty path", # caused by transform, cannot read path+ "--path_filter_substr", type=str, default=None, help="remove rows whose path contains a substring")+ parser.add_argument("--path_keep_substr", type=str, default=None, help="keep rows whose path contains a substring")+ parser.add_argument("--path_dedup", action="store_true", help="remove rows with duplicated path")# caption filtering- parser.add_argument(- "--remove-empty-caption",- action="store_true",- help="remove rows with empty caption",- )- parser.add_argument("--remove-url", action="store_true", help="remove rows with url in caption")+ parser.add_argument("--text_filter_empty", action="store_true", help="remove rows with empty caption")+ parser.add_argument("--text_filter_url", action="store_true", help="remove rows with url in caption")+ parser.add_argument("--text_filter_substr", type=str, default=None, help="remove text with a substring")+ parser.add_argument("--text_dedup", action="store_true", help="remove rows with duplicated caption")parser.add_argument("--lang", type=str, default=None, help="remove rows with other language")- parser.add_argument("--remove-path-duplication", action="store_true", help="remove rows with duplicated path")- parser.add_argument("--remove-text-duplication", action="store_true", help="remove rows with duplicated caption")+ parser.add_argument("--filter-too-verbose", action="store_true", help="filter samples with too verbose caption")+ parser.add_argument("--filter-human", action="store_true", help="filter samples with human preference score")# caption processing- parser.add_argument("--refine-llm-caption", action="store_true", help="modify the caption generated by LLM")+ parser.add_argument("--text_remove_prefix", action="store_true", help="remove prefix like 'The video shows '")parser.add_argument(- "--clean-caption", action="store_true", help="modify the caption according to T5 pipeline to suit training"+ "--text_refine_t5", action="store_true", help="refine the caption output by T5 with regular expression")+ parser.add_argument("--text_image2video", action="store_true", help="text.replace('image', 'video'")+ parser.add_argument("--text_refine_sentences", action="store_true", help="refine every sentence in the caption")+ parser.add_argument("--text_score2text", action="store_true", help="convert score to text and append to caption")+ parser.add_argument("--text_undo_score2text", action="store_true", help="undo score2text")parser.add_argument("--merge-cmotion", action="store_true", help="merge the camera motion to the caption")parser.add_argument("--count-num-token", type=str, choices=["t5"], default=None, help="Count the number of tokens in the caption")- parser.add_argument("--append-text", type=str, default=None, help="append text to the caption")- parser.add_argument("--score-to-text", action="store_true", help="convert score to text")+ parser.add_argument("--text_append", type=str, default=None, help="append text to the caption")parser.add_argument("--update-text", type=str, default=None, help="update the text with the given text")+ # filter for dynamic fps+ parser.add_argument(+ "--filter_dyn_fps", action="store_true", help="filter data to contain enough frames for dynamic fps"+ )+ parser.add_argument("--dyn_fps_max_fps", type=int, default=16, help="max fps for dynamic fps")+ parser.add_argument("--dyn_fps_keep_frames", type=int, default=32, help="num frames to keep for dynamic fps")+# score filteringparser.add_argument("--filesize", action="store_true", help="get the filesize of each video and image in MB")- parser.add_argument("--fsmax", type=int, default=None, help="filter the dataset by maximum filesize")+ parser.add_argument("--fsmax", type=float, default=None, help="filter the dataset by maximum filesize")+ parser.add_argument("--fsmin", type=float, default=None, help="filter the dataset by minimum filesize")parser.add_argument("--fmin", type=int, default=None, help="filter the dataset by minimum number of frames")parser.add_argument("--fmax", type=int, default=None, help="filter the dataset by maximum number of frames")parser.add_argument("--hwmax", type=int, default=None, help="filter the dataset by maximum resolution")parser.add_argument("--aesmin", type=float, default=None, help="filter the dataset by minimum aes score")+ parser.add_argument(+ "--prefmin", type=float, default=None, help="filter the dataset by minimum human preference score"+ )parser.add_argument("--matchmin", type=float, default=None, help="filter the dataset by minimum match score")parser.add_argument("--flowmin", type=float, default=None, help="filter the dataset by minimum flow score")+ parser.add_argument("--facemin", type=float, default=None, help="filter the dataset by minimum face area ratio")parser.add_argument("--fpsmax", type=float, default=None, help="filter the dataset by maximum fps")parser.add_argument("--img-only", action="store_true", help="only keep the image data")parser.add_argument("--vid-only", action="store_true", help="only keep the video data")+ parser.add_argument("--h-le-w", action="store_true", help="only keep samples with h <= w")# data processingparser.add_argument("--shuffle", default=False, action="store_true", help="shuffle the dataset")parser.add_argument("--head", type=int, default=None, help="return the first n rows of data")+ parser.add_argument("--sample", type=int, default=None, help="randomly sample n rows; using args.seed")+ parser.add_argument("--chunk", type=int, default=None, help="evenly split rows into chunks")return parser.parse_args()@@ -772,6 +950,11 @@ def get_output_path(args, input_name):name = input_namedir_path = os.path.dirname(args.input[0])+ if args.path_subtract is not None:+ name += "_subtract"+ if args.path_intersect is not None:+ name += "_intersect"+# sortif args.sort is not None:assert args.sort_ascending is None@@ -796,44 +979,66 @@ def get_output_path(args, input_name):name += "_relpath"if args.abspath is not None:name += "_abspath"- if args.remove_empty_path:- name += "_noemptypath"+ if args.path_filter_empty:+ name += "_path-filter-empty"+ if args.path_filter_substr is not None:+ name += "_path-filter-substr"+ if args.path_keep_substr is not None:+ name += "_path-keep-substr"+ if args.path_dedup:+ name += "_path-dedup"# caption filtering- if args.remove_empty_caption:- name += "_noempty"- if args.remove_url:- name += "_nourl"+ if args.text_filter_empty:+ name += "_text-filter-empty"+ if args.text_filter_url:+ name += "_text-filter-url"if args.lang is not None:name += f"_{args.lang}"- if args.remove_path_duplication:- name += "_noduppath"- if args.remove_text_duplication:- name += "_noduptext"- if args.path_subset:- name += "_subset"+ if args.text_dedup:+ name += "_text-dedup"+ if args.text_filter_substr is not None:+ name += f"_text-filter-substr"+ if args.filter_too_verbose:+ name += "_noverbose"+ if args.h_le_w:+ name += "_h-le-w"+ if args.filter_human:+ name += "_human"# caption processing- if args.refine_llm_caption:- name += "_llm"- if args.clean_caption:- name += "_clean"+ if args.text_remove_prefix:+ name += "_text-remove-prefix"+ if args.text_refine_t5:+ name += "_text-refine-t5"+ if args.text_image2video:+ name += "_text-image2video"+ if args.text_refine_sentences:+ name += "_text-refine-sentences"+ if args.text_score2text:+ name += "_text-score2text"+ if args.text_undo_score2text:+ name += "_text-undo-score2text"if args.merge_cmotion:name += "_cmcaption"if args.count_num_token:name += "_ntoken"- if args.append_text is not None:- name += "_appendtext"- if args.score_to_text:- name += "_score2text"+ if args.text_append is not None:+ name += "_text-append"if args.update_text is not None:name += "_update"+ # dynmaic fps+ if args.filter_dyn_fps is not False:+ name += "_dynfps"+# score filteringif args.filesize:name += "_filesize"if args.fsmax is not None:name += f"_fsmax{args.fsmax}"+ if args.fsmin is not None:+ name += f"_fsmin{args.fsmin}"if args.fmin is not None:name += f"_fmin{args.fmin}"if args.fmax is not None:@@ -844,10 +1049,14 @@ def get_output_path(args, input_name):name += f"_hwmax{args.hwmax}"if args.aesmin is not None:name += f"_aesmin{args.aesmin}"+ if args.prefmin is not None:+ name += f"_prefmin{args.prefmin}"if args.matchmin is not None:name += f"_matchmin{args.matchmin}"if args.flowmin is not None:name += f"_flowmin{args.flowmin}"+ if args.facemin is not None:+ name += f"_facemin{args.facewmin}"if args.img_only:name += "_img"if args.vid_only:@@ -858,6 +1067,8 @@ def get_output_path(args, input_name):name += f"_shuffled_seed{args.seed}"if args.head is not None:name += f"_first_{args.head}_data"+ if args.sample is not None:+ name += f"_sample-{args.sample}"output_path = os.path.join(dir_path, f"{name}.{args.format}")return output_pathdiff --git a/tools/datasets/save_first_frame.py b/tools/datasets/save_first_frame.pynew file mode 100644index 0000000..bcb63dc--- /dev/null+++ b/tools/datasets/save_first_frame.py@@ -0,0 +1,66 @@+import argparse+import os++import cv2+import pandas as pd+from tqdm import tqdm++tqdm.pandas()++try:+ from pandarallel import pandarallel++ PANDA_USE_PARALLEL = True+except ImportError:+ PANDA_USE_PARALLEL = False+++def save_first_frame(video_path, img_dir):+ if not os.path.exists(video_path):+ print(f"Video not found: {video_path}")+ return ""++ try:+ cap = cv2.VideoCapture(video_path)+ success, frame = cap.read()+ if success:+ video_name = os.path.basename(video_path)+ image_name = os.path.splitext(video_name)[0] + "_first_frame.jpg"+ image_path = os.path.join(img_dir, image_name)++ cv2.imwrite(image_path, frame)+ else:+ raise ValueError("Video broken.")+ cap.release()+ return image_path+ except Exception as e:+ print(f"Save first frame of `{video_path}` failed. {e}")+ return ""+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("input", type=str, help="path to the input csv dataset")+ parser.add_argument("--img-dir", type=str, help="path to save first frame image")+ parser.add_argument("--disable-parallel", action="store_true", help="disable parallel processing")+ parser.add_argument("--num-workers", type=int, default=None, help="number of workers")+ args = parser.parse_args()++ if args.disable_parallel:+ PANDA_USE_PARALLEL = False+ if PANDA_USE_PARALLEL:+ if args.num_workers is not None:+ pandarallel.initialize(nb_workers=args.num_workers, progress_bar=True)+ else:+ pandarallel.initialize(progress_bar=True)++ if not os.path.exists(args.img_dir):+ os.makedirs(args.img_dir)++ data = pd.read_csv(args.input)++ data["first_frame_path"] = data["path"].parallel_apply(save_first_frame, img_dir=args.img_dir)+ data_filtered = data.loc[data["first_frame_path"] != ""]+ output_csv_path = args.input.replace(".csv", "_first-frame.csv")+ data_filtered.to_csv(output_csv_path, index=False)+ print(f"First frame csv saved to: {output_csv_path}, first frame images saved to {args.img_dir}.")diff --git a/tools/datasets/transform.py b/tools/datasets/transform.pyindex bffb191..53dbaf3 100644--- a/tools/datasets/transform.py+++ b/tools/datasets/transform.py@@ -1,27 +1,24 @@import argparseimport osimport random+import shutil+import subprocessimport cv2+import ffmpegimport numpy as npimport pandas as pd+from pandarallel import pandarallelfrom tqdm import tqdmfrom .utils import IMG_EXTENSIONS, extract_framestqdm.pandas()--try:- from pandarallel import pandarallel-- pandarallel.initialize(progress_bar=True)- pandas_has_parallel = True-except ImportError:- pandas_has_parallel = False+USE_PANDARALLEL = Truedef apply(df, func, **kwargs):- if pandas_has_parallel:+ if USE_PANDARALLEL:return df.parallel_apply(func, **kwargs)return df.progress_apply(func, **kwargs)@@ -32,8 +29,8 @@ def get_new_path(path, input_dir, output):return path_new-def resize(path, length, input_dir, output):- path_new = get_new_path(path, input_dir, output)+def resize_longer(path, length, input_dir, output_dir):+ path_new = get_new_path(path, input_dir, output_dir)ext = os.path.splitext(path)[1].lower()assert ext in IMG_EXTENSIONSimg = cv2.imread(path)@@ -42,10 +39,34 @@ def resize(path, length, input_dir, output):if min(h, w) > length:if h > w:new_h = length- new_w = int(w * new_h / h)+ new_w = int(w / h * length)else:new_w = length- new_h = int(h * new_w / w)+ new_h = int(h / w * length)+ img = cv2.resize(img, (new_w, new_h))+ cv2.imwrite(path_new, img)+ else:+ path_new = ""+ return path_new+++def resize_shorter(path, length, input_dir, output_dir):+ path_new = get_new_path(path, input_dir, output_dir)+ if os.path.exists(path_new):+ return path_new++ ext = os.path.splitext(path)[1].lower()+ assert ext in IMG_EXTENSIONS+ img = cv2.imread(path)+ if img is not None:+ h, w = img.shape[:2]+ if min(h, w) > length:+ if h > w:+ new_w = length+ new_h = int(h / w * length)+ else:+ new_h = length+ new_w = int(w / h * length)img = cv2.resize(img, (new_w, new_h))cv2.imwrite(path_new, img)else:@@ -76,15 +97,135 @@ def rand_crop(path, input_dir, output):return path_new-def main(args):- data = pd.read_csv(args.input)- if args.method == "img_rand_crop":- data["path"] = apply(data["path"], lambda x: rand_crop(x, args.input_dir, args.output))- output_csv = args.input.replace(".csv", f"_rand_crop.csv")- elif args.method == "img_resize":- data["path"] = apply(data["path"], lambda x: resize(x, args.length, args.input_dir, args.output))- output_csv = args.input.replace(".csv", f"_resized{args.length}.csv")- elif args.method == "vid_frame_extract":+def m2ts_to_mp4(row, output_dir):+ input_path = row["path"]+ output_name = os.path.basename(input_path).replace(".m2ts", ".mp4")+ output_path = os.path.join(output_dir, output_name)+ # create directory if it doesn't exist+ os.makedirs(os.path.dirname(output_path), exist_ok=True)+ try:+ ffmpeg.input(input_path).output(output_path).overwrite_output().global_args("-loglevel", "quiet").run(+ capture_stdout=True+ )+ row["path"] = output_path+ row["relpath"] = os.path.splitext(row["relpath"])[0] + ".mp4"+ except Exception as e:+ print(f"Error converting {input_path} to mp4: {e}")+ row["path"] = ""+ row["relpath"] = ""+ return row+ return row+++def mkv_to_mp4(row, output_dir):+ # str_to_replace and str_to_replace_with account for the different directory structure+ input_path = row["path"]+ output_name = os.path.basename(input_path).replace(".mkv", ".mp4")+ output_path = os.path.join(output_dir, output_name)++ # create directory if it doesn't exist+ os.makedirs(os.path.dirname(output_path), exist_ok=True)++ try:+ ffmpeg.input(input_path).output(output_path).overwrite_output().global_args("-loglevel", "quiet").run(+ capture_stdout=True+ )+ row["path"] = output_path+ row["relpath"] = os.path.splitext(row["relpath"])[0] + ".mp4"+ except Exception as e:+ print(f"Error converting {input_path} to mp4: {e}")+ row["path"] = ""+ row["relpath"] = ""+ return row+ return row+++def mp4_to_mp4(row, output_dir):+ # str_to_replace and str_to_replace_with account for the different directory structure+ input_path = row["path"]++ # 检查输入文件是否为.mp4文件+ if not input_path.lower().endswith(".mp4"):+ print(f"Error: {input_path} is not an .mp4 file.")+ row["path"] = ""+ row["relpath"] = ""+ return row+ output_name = os.path.basename(input_path)+ output_path = os.path.join(output_dir, output_name)++ # create directory if it doesn't exist+ os.makedirs(os.path.dirname(output_path), exist_ok=True)++ try:+ shutil.copy2(input_path, output_path) # 使用shutil复制文件+ row["path"] = output_path+ row["relpath"] = os.path.splitext(row["relpath"])[0] + ".mp4"+ except Exception as e:+ print(f"Error coy {input_path} to mp4: {e}")+ row["path"] = ""+ row["relpath"] = ""+ return row+ return row+++def crop_to_square(input_path, output_path):+ cmd = (+ f"ffmpeg -i {input_path} "+ f"-vf \"crop='min(in_w,in_h)':'min(in_w,in_h)':'(in_w-min(in_w,in_h))/2':'(in_h-min(in_w,in_h))/2'\" "+ f"-c:v libx264 -an "+ f"-map 0:v {output_path}"+ )+ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)+ stdout, stderr = proc.communicate()+++def vid_crop_center(row, input_dir, output_dir):+ input_path = row["path"]+ relpath = os.path.relpath(input_path, input_dir)+ assert not relpath.startswith("..")+ output_path = os.path.join(output_dir, relpath)++ # create directory if it doesn't exist+ os.makedirs(os.path.dirname(output_path), exist_ok=True)++ try:+ crop_to_square(input_path, output_path)+ size = min(row["height"], row["width"])+ row["path"] = output_path+ row["height"] = size+ row["width"] = size+ row["aspect_ratio"] = 1.0+ row["resolution"] = size**2+ except Exception as e:+ print(f"Error cropping {input_path} to center: {e}")+ row["path"] = ""+ return row+++def main():+ args = parse_args()+ global USE_PANDARALLEL++ assert args.num_workers is None or not args.disable_parallel+ if args.disable_parallel:+ USE_PANDARALLEL = False+ if args.num_workers is not None:+ pandarallel.initialize(progress_bar=True, nb_workers=args.num_workers)+ else:+ pandarallel.initialize(progress_bar=True)++ random.seed(args.seed)+ data = pd.read_csv(args.meta_path)+ if args.task == "img_rand_crop":+ data["path"] = apply(data["path"], lambda x: rand_crop(x, args.input_dir, args.output_dir))+ output_csv = args.meta_path.replace(".csv", "_rand_crop.csv")+ elif args.task == "img_resize_longer":+ data["path"] = apply(data["path"], lambda x: resize_longer(x, args.length, args.input_dir, args.output_dir))+ output_csv = args.meta_path.replace(".csv", f"_resize-longer-{args.length}.csv")+ elif args.task == "img_resize_shorter":+ data["path"] = apply(data["path"], lambda x: resize_shorter(x, args.length, args.input_dir, args.output_dir))+ output_csv = args.meta_path.replace(".csv", f"_resize-shorter-{args.length}.csv")+ elif args.task == "vid_frame_extract":points = args.points if args.points is not None else args.points_indexdata = pd.DataFrame(np.repeat(data.values, 3, axis=0), columns=data.columns)num_points = len(points)@@ -94,21 +235,66 @@ def main(args):data.loc[i::num_points, "point"] = pointelse:data.loc[i::num_points, "point"] = data.loc[i::num_points, "num_frames"] * point- data["path"] = apply(data, lambda x: extract_frames(x["path"], args.input_dir, args.output, x["point"]), axis=1)- output_csv = args.input.replace(".csv", f"_vid_frame_extract.csv")-+ data["path"] = apply(+ data, lambda x: extract_frames(x["path"], args.input_dir, args.output_dir, x["point"]), axis=1+ )+ output_csv = args.meta_path.replace(".csv", "_vid_frame_extract.csv")+ elif args.task == "m2ts_to_mp4":+ print(f"m2ts_to_mp4作业开始:{args.output_dir}")+ assert args.meta_path.endswith("_m2ts.csv"), "Input file must end with '_m2ts.csv'"+ m2ts_to_mp4_partial = lambda x: m2ts_to_mp4(x, args.output_dir)+ data = apply(data, m2ts_to_mp4_partial, axis=1)+ data = data[data["path"] != ""]+ output_csv = args.meta_path.replace("_m2ts.csv", ".csv")+ elif args.task == "mkv_to_mp4":+ print(f"mkv_to_mp4作业开始:{args.output_dir}")+ assert args.meta_path.endswith("_mkv.csv"), "Input file must end with '_mkv.csv'"+ mkv_to_mp4_partial = lambda x: mkv_to_mp4(x, args.output_dir)+ data = apply(data, mkv_to_mp4_partial, axis=1)+ data = data[data["path"] != ""]+ output_csv = args.meta_path.replace("_mkv.csv", ".csv")+ elif args.task == "mp4_to_mp4":+ # assert args.meta_path.endswith("meta.csv"), "Input file must end with '_mkv.csv'"+ print(f"MP4复制作业开始:{args.output_dir}")+ mkv_to_mp4_partial = lambda x: mp4_to_mp4(x, args.output_dir)+ data = apply(data, mkv_to_mp4_partial, axis=1)+ data = data[data["path"] != ""]+ output_csv = args.meta_path+ elif args.task == "vid_crop_center":+ vid_crop_center_partial = lambda x: vid_crop_center(x, args.input_dir, args.output_dir)+ data = apply(data, vid_crop_center_partial, axis=1)+ data = data[data["path"] != ""]+ output_csv = args.meta_path.replace(".csv", "_center-crop.csv")+ else:+ raise ValueErrordata.to_csv(output_csv, index=False)print(f"Saved to {output_csv}")+ raise SystemExit(0)def parse_args():parser = argparse.ArgumentParser()- parser.add_argument("method", type=str, choices=["img_resize", "img_rand_crop", "vid_frame_extract"])- parser.add_argument("input", type=str)- parser.add_argument("input_dir", type=str)- parser.add_argument("output", type=str)+ parser.add_argument(+ "--task",+ type=str,+ required=True,+ choices=[+ "img_resize_longer",+ "img_resize_shorter",+ "img_rand_crop",+ "vid_frame_extract",+ "m2ts_to_mp4",+ "mkv_to_mp4",+ "mp4_to_mp4",+ "vid_crop_center",+ ],+ )+ parser.add_argument("--meta_path", type=str, required=True)+ parser.add_argument("--input_dir", type=str)+ parser.add_argument("--output_dir", type=str)+ parser.add_argument("--length", type=int, default=1080)parser.add_argument("--disable-parallel", action="store_true")- parser.add_argument("--length", type=int, default=2160)+ parser.add_argument("--num_workers", type=int, default=None)parser.add_argument("--seed", type=int, default=42, help="seed for random")parser.add_argument("--points", nargs="+", type=float, default=None)parser.add_argument("--points_index", nargs="+", type=int, default=None)@@ -117,8 +303,4 @@ def parse_args():if __name__ == "__main__":- args = parse_args()- random.seed(args.seed)- if args.disable_parallel:- pandas_has_parallel = False- main(args)+ main()diff --git a/tools/datasets/utils.py b/tools/datasets/utils.pyindex ec7e1b9..4fbe3f6 100644--- a/tools/datasets/utils.py+++ b/tools/datasets/utils.py@@ -49,7 +49,7 @@ def extract_frames(if idx >= total_frames:idx = total_frames - 1target_timestamp = int(idx * av.time_base / container.streams.video[0].average_rate)- container.seek(target_timestamp)+ container.seek(target_timestamp) # return the nearest key frame, not the precise timestamp!!!frame = next(container.decode(video=0)).to_image()frames.append(frame)diff --git a/tools/scene_cut/cut.py b/tools/scene_cut/cut.pyindex 9fe4880..e19c012 100644--- a/tools/scene_cut/cut.py+++ b/tools/scene_cut/cut.py@@ -35,12 +35,14 @@ def process_single_row(row, args):if not (timestamp.startswith("[") and timestamp.endswith("]")):return Falsescene_list = eval(timestamp)- scene_list = [(FrameTimecode(s, fps=100), FrameTimecode(t, fps=100)) for s, t in scene_list]+ # Jul 19 quick fix Tom: remove 0.1s from the end of each scene to avoid the last frame issue+ # scene_list = [(FrameTimecode(s, fps=1000), FrameTimecode(t, fps=1000) - FrameTimecode('00:00:00.100', fps=1000)) for s, t in scene_list]+ scene_list = [(FrameTimecode(s, fps=1000), FrameTimecode(t, fps=1000)) for s, t in scene_list]else:scene_list = [None]if args.drop_invalid_timestamps:return True- except Exception as e:+ except Exception:if args.drop_invalid_timestamps:return False@@ -52,6 +54,7 @@ def process_single_row(row, args):shorter_size = args.shorter_sizeif (shorter_size is not None) and ("height" in row) and ("width" in row):+ # Jul 13 Tom: this is skipped cause we are not doing info before cut atmmin_size = min(row["height"], row["width"])if min_size <= shorter_size:shorter_size = None@@ -68,6 +71,7 @@ def process_single_row(row, args):)return True+def split_video(video_path,scene_list,@@ -115,7 +119,7 @@ def split_video(if os.path.exists(save_path):# print_log(f"File '{save_path}' already exists. Skip.", logger=logger)continue-+# ffmpeg cmdcmd = [FFMPEG_PATH]@@ -137,11 +141,10 @@ def split_video(# aspect ratioif shorter_size is not None:- cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]+ cmd += ["-vf", f"scale='if(gt(iw,ih),-2,min({shorter_size},iw))':'if(gt(iw,ih),min({shorter_size},ih),-2)'"]# cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]cmd += ["-map", "0:v", save_path]- # print(cmd)proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)stdout, stderr = proc.communicate()# stdout = stdout.decode("utf-8")@@ -204,5 +207,8 @@ def main():assert args.meta_path.endswith("timestamp.csv"), "Only support *timestamp.csv"meta.to_csv(args.meta_path.replace("timestamp.csv", "correct_timestamp.csv"), index=False)print(f"Corrected timestamp file saved to '{args.meta_path.replace('timestamp.csv', 'correct_timestamp.csv')}'")++if __name__ == "__main__":main()+ print("cut.py finished successfully.")diff --git a/tools/scene_cut/cut_to_short.py b/tools/scene_cut/cut_to_short.pynew file mode 100644index 0000000..ea6aa32--- /dev/null+++ b/tools/scene_cut/cut_to_short.py@@ -0,0 +1,69 @@+import argparse+import os+import subprocess+from functools import partial++import pandas as pd+from pandarallel import pandarallel+from tqdm import tqdm+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("meta_path", type=str)+ parser.add_argument("--length", type=int, required=True, help="segment length in seconds; example: 1800")+ parser.add_argument("--src_dir", type=str, required=True, help="exmaple: /path/to/dataset/raw/")+ parser.add_argument("--dst_dir", type=str, required=True, help="exmaple: /path/to/dataset/cut_to_30min/")++ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")+ parser.add_argument("--disable_parallel", action="store_true", help="disable parallel processing")++ args = parser.parse_args()+ return args+++def process_single_row(row, args):+ path = row["path"]+ assert path.startswith(args.src_dir), f"\npath: {path}\nsrc_dir:{args.src_dir}"++ out_path = os.path.join(args.dst_dir, os.path.relpath(path, args.src_dir))+ os.makedirs(os.path.dirname(out_path), exist_ok=True)+ wo_ext, ext = os.path.splitext(out_path)+ cmd = (+ f"ffmpeg -i {path} "+ f"-c copy -an " # -an: no audio+ f"-f segment -segment_time 60 -reset_timestamps 1 -map 0 -segment_start_number 0 "+ f"{wo_ext}_%03d{ext}"+ )+ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)+ stdout, stderr = proc.communicate()+++def main():+ args = parse_args()+ meta_path = args.meta_path+ if not os.path.exists(meta_path):+ print(f"Meta file '{meta_path}' not found. Exit.")+ exit()++ # initialize pandarallel+ tqdm.pandas()+ if not args.disable_parallel:+ if args.num_workers is not None:+ pandarallel.initialize(progress_bar=True, nb_workers=args.num_workers)+ else:+ pandarallel.initialize(progress_bar=True)+ process_single_row_partial = partial(process_single_row, args=args)++ # process+ meta = pd.read_csv(meta_path)+ if not args.disable_parallel:+ meta.parallel_apply(process_single_row_partial, axis=1)+ else:+ meta.apply(process_single_row_partial, axis=1)++ print("cut_to_short.py finished successfully.")+++if __name__ == "__main__":+ main()diff --git a/tools/scene_cut/scene_detect.py b/tools/scene_cut/scene_detect.pyindex c564d73..eb9ffd9 100644--- a/tools/scene_cut/scene_detect.py+++ b/tools/scene_cut/scene_detect.py@@ -1,40 +1,223 @@import argparseimport os+from functools import partialimport numpy as npimport pandas as pdfrom pandarallel import pandarallel-from scenedetect import AdaptiveDetector, detect+from scenedetect import AdaptiveDetector, SceneManager, StatsManager, open_videofrom tqdm import tqdmtqdm.pandas()-def process_single_row(row):- # windows- # from scenedetect import detect, ContentDetector, AdaptiveDetector+def detect(+ video_path,+ detector,+ backend="pyav",+ stats_file_path=None,+ show_progress=False,+ start_time=None,+ end_time=None,+ start_in_scene=False,+):+ """+ Adapted from scenedetect.detect()+ Modifications:+ - allow passing backend to open_video()+ """+ video = open_video(video_path, backend=backend)+ if start_time is not None:+ start_time = video.base_timecode + start_time+ video.seek(start_time)+ if end_time is not None:+ end_time = video.base_timecode + end_time+ # To reduce memory consumption when not required, we only add a StatsManager if we+ # need to save frame metrics to disk.+ scene_manager = SceneManager(StatsManager() if stats_file_path else None)+ scene_manager.add_detector(detector)+ scene_manager.detect_scenes(+ video=video,+ show_progress=show_progress,+ end_time=end_time,+ )+ if not scene_manager.stats_manager is None:+ scene_manager.stats_manager.save_to_csv(csv_file=stats_file_path)+ return scene_manager.get_scene_list(start_in_scene=start_in_scene)- video_path = row["path"]+def detect_transition(+ video_path,+ backend="pyav",+ transition_seconds=1.0,+ start_time=None,+ end_time=None,+ start_in_scene=False,+ stats_file_path=None,+ show_progress=False,+):+ video = open_video(video_path, backend=backend)+ fps = video.frame_rate++ frame_skip = int(transition_seconds * fps)+ t_ada, t_con = 2.0, 10.0+ window = 1+ if end_time is not None:+ end_time = video.base_timecode + end_time++ # 1. detect from the very beginning+ detector = AdaptiveDetector(+ adaptive_threshold=t_ada, # default 3.0+ min_content_val=t_con, # default 15.0+ min_scene_len=1,+ window_width=window,+ )+ scene_manager = SceneManager(StatsManager() if stats_file_path else None)+ scene_manager.add_detector(detector)++ if start_time is not None:+ tmp = video.base_timecode + start_time+ video.seek(tmp)++ scene_manager.detect_scenes(+ video=video,+ show_progress=show_progress,+ frame_skip=frame_skip,+ end_time=end_time,+ )+ if scene_manager.stats_manager is not None:+ scene_manager.stats_manager.save_to_csv(csv_file=stats_file_path)+ s0 = scene_manager.get_scene_list(start_in_scene=start_in_scene)++ # 2. detect from half transition_secondsdetector = AdaptiveDetector(- adaptive_threshold=3.0,- # luma_only=True,+ adaptive_threshold=t_ada, # default 3.0+ min_content_val=t_con, # default 15.0+ min_scene_len=1,+ window_width=window,)- # detector = ContentDetector()- # TODO: catch error here+ scene_manager = SceneManager(StatsManager() if stats_file_path else None)+ scene_manager.add_detector(detector)++ tmp = video.base_timecode + transition_seconds / 2+ if start_time is not None:+ tmp = tmp + start_time+ video.seek(tmp)++ scene_manager.detect_scenes(+ video=video,+ show_progress=show_progress,+ frame_skip=frame_skip,+ end_time=end_time,+ )+ if scene_manager.stats_manager is not None:+ scene_manager.stats_manager.save_to_csv(csv_file=f"{stats_file_path}.csv")+ s1 = scene_manager.get_scene_list(start_in_scene=start_in_scene)++ # merge s0, s1+ s0_s = [x[0] for x in s0]+ s1_t = [x[1] for x in s1]+ i, j = 0, 0+ merged = []+ while i < len(s0_s) and j < len(s1_t):+ l, r = s0_s[i], s1_t[j]+ if l < r:+ merged.append(l)+ i += 1+ elif l == r:+ merged.append(l)+ i += 1+ j += 1+ else:+ merged.append(r)+ j += 1+ merged.extend(s0_s[i:])+ merged.extend(s1_t[j:])++ # remove transitions+ scene_list = []+ m = transition_seconds+ for idx in range(len(merged) - 1):+ cur = merged[idx]+ next = merged[idx + 1]+ if next - cur < m:+ continue++ if idx + 2 < len(merged) and (merged[idx + 2] - next) < m:+ # detected by both s0 & s1+ next = next - m+ elif idx + 2 < len(merged):+ next = next - 1.5 * m+ if idx - 1 >= 0 and (cur - merged[idx - 1]) < m:+ # detected by both s0 & s1+ cur = cur + 0.5 * m+ elif idx > 0:+ cur = cur + 0.5 * m++ if cur < next:+ scene_list.append((cur, next))++ return scene_list+++def process_single_row(row, args):+ video_path = row["path"]+try:- scene_list = detect(video_path, detector, start_in_scene=True)- timestamp = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]- return True, str(timestamp)+ if args.transition_seconds is not None:+ scene_list = detect_transition(+ video_path,+ transition_seconds=args.transition_seconds,+ start_time=args.start_time,+ end_time=args.end_time,+ start_in_scene=True,+ )+ timestamp_intact = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]+ else:+ detector = AdaptiveDetector(+ adaptive_threshold=2.0,+ min_content_val=10.0,+ min_scene_len=1,+ )+ scene_list = detect(+ video_path,+ detector,+ start_time=args.start_time,+ end_time=args.end_time,+ start_in_scene=True,+ )+ margin = 1.0 / scene_list[0][0].framerate+ scene_list = [(s, t - margin) for s, t in scene_list if s < t - margin]+ timestamp_intact = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]++ if args.max_seconds is not None:+ timestamp_cut = []+ for s, t in scene_list:+ while (t - s).get_seconds() > args.max_seconds:+ tmp = s + args.max_seconds+ timestamp_cut.append((s.get_timecode(), tmp.get_timecode()))+ s = tmp+ timestamp_cut.append((s.get_timecode(), t.get_timecode()))+ else:+ timestamp_cut = timestamp_intact+ return True, str(timestamp_cut), str(timestamp_intact)+except Exception as e:print(f"Video '{video_path}' with error {e}")- return False, ""+ return False, "", ""def parse_args():parser = argparse.ArgumentParser()parser.add_argument("meta_path", type=str)+ parser.add_argument(+ "--max_seconds", type=float, default=None, help="if not None, any single timestamp will not exceed max_seconds"+ )+ parser.add_argument("--transition_seconds", type=float, default=None, help="if not None, use detect_transition()")+ parser.add_argument("--start_time", type=float, default=None, help="if not None, start detection from start_time")+ parser.add_argument("--end_time", type=float, default=None, help="if not None, end detection at end_time")parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")+ parser.add_argument("--disable_parallel", action="store_true", help="disable parallel processing")args = parser.parse_args()return args@@ -42,21 +225,31 @@ def parse_args():def main():args = parse_args()+ if args.transition_seconds is not None:+ assert args.transition_seconds > 0+meta_path = args.meta_pathif not os.path.exists(meta_path):print(f"Meta file '{meta_path}' not found. Exit.")exit()- if args.num_workers is not None:- pandarallel.initialize(progress_bar=True, nb_workers=args.num_workers)- else:- pandarallel.initialize(progress_bar=True)+ # initialize pandarallel+ if not args.disable_parallel:+ if args.num_workers is not None:+ pandarallel.initialize(progress_bar=True, nb_workers=args.num_workers)+ else:+ pandarallel.initialize(progress_bar=True)+ process_single_row_partial = partial(process_single_row, args=args)meta = pd.read_csv(meta_path)- ret = meta.parallel_apply(process_single_row, axis=1)+ if not args.disable_parallel:+ ret = meta.parallel_apply(process_single_row_partial, axis=1)+ else:+ ret = meta.apply(process_single_row_partial, axis=1)- succ, timestamps = list(zip(*ret))- meta["timestamp"] = timestamps+ succ, timestamp_cut, timestamp_intact = list(zip(*ret))+ meta["timestamp"] = timestamp_cut+ meta["timestamp_intact"] = timestamp_intactmeta = meta[np.array(succ)]wo_ext, ext = os.path.splitext(meta_path)diff --git a/tools/scoring/aesthetic/inference.py b/tools/scoring/aesthetic/inference.pyindex be6618f..d730e93 100644--- a/tools/scoring/aesthetic/inference.py+++ b/tools/scoring/aesthetic/inference.py@@ -65,8 +65,7 @@ class VideoTextDataset(torch.utils.data.Dataset):if not is_video(path):images = [pil_loader(path)]else:- num_frames = sample["num_frames"] if "num_frames" in sample else None- images = extract_frames(sample["path"], points=self.points, backend="opencv", num_frames=num_frames)+ images = extract_frames(sample["path"], points=self.points, backend="opencv")# transformimages = [self.transform(img) for img in images]diff --git a/tools/scoring/deduplicate/__init__.py b/tools/scoring/deduplicate/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/tools/scoring/deduplicate/inference.py b/tools/scoring/deduplicate/inference.pynew file mode 100644index 0000000..84ed1ad--- /dev/null+++ b/tools/scoring/deduplicate/inference.py@@ -0,0 +1,176 @@+import argparse+import os+from datetime import timedelta++import clip+import numpy as np+import pandas as pd+import torch+import torch.distributed as dist+import torch.nn.functional as F+from torch.utils.data import DataLoader, DistributedSampler+from torchvision.datasets.folder import pil_loader+from tqdm import tqdm++from tools.datasets.utils import extract_frames, is_video+++class VideoDataset(torch.utils.data.Dataset):+ def __init__(self, meta_path, transform):+ self.meta_path = meta_path+ self.meta = pd.read_csv(meta_path)+ self.transform = transform++ def __getitem__(self, index):+ row = self.meta.iloc[index]+ path = row["path"]++ if is_video(path):+ img = extract_frames(path, points=[0.5], backend="opencv")[0]+ else:+ img = pil_loader(path)+ img_size = img.size # W, H++ img = self.transform(img)++ ret = dict(index=index, images=img, img_size=str(img_size))+ return ret++ def __len__(self):+ return len(self.meta)+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("meta_path", type=str, help="Path to the input CSV file")+ parser.add_argument("--bs", type=int, default=128, help="Batch size")+ parser.add_argument("--thresh", type=float, default=0.98, help="similarity thresh")+ parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")+ args = parser.parse_args()+ return args+++def main():+ args = parse_args()++ meta_path = args.meta_path+ if not os.path.exists(meta_path):+ print(f"Meta file '{meta_path}' not found. Exit.")+ exit()++ wo_ext, ext = os.path.splitext(meta_path)+ out_path_dedup = f"{wo_ext}_dedup{ext}"+ out_path_dup = f"{wo_ext}_dup{ext}"++ dist.init_process_group(backend="nccl", timeout=timedelta(hours=24))+ torch.cuda.set_device(dist.get_rank() % torch.cuda.device_count())++ # build model+ device = "cuda" if torch.cuda.is_available() else "cpu" # Note: do not use torch.device('cuda')!!!+ model, preprocess = clip.load("ViT-L/14", device=device)+ # preprocess: resize shorter size to 224 by keeping ar, then center crop++ # build dataset+ dataset = VideoDataset(meta_path=meta_path, transform=preprocess)+ dataloader = DataLoader(+ dataset,+ batch_size=args.bs,+ num_workers=args.num_workers,+ sampler=DistributedSampler(+ dataset,+ num_replicas=dist.get_world_size(),+ rank=dist.get_rank(),+ shuffle=False,+ drop_last=False,+ ),+ )++ # encode images and store the feature+ print("Begin to generate features")+ model.eval()+ feat_list = [] # store the feature+ indices_list = [] # store the indices+ img_size_list = []+ for batch in tqdm(dataloader, disable=dist.get_rank() != 0):+ indices = batch["index"]+ imgs = batch["images"].to(device, non_blocking=True)+ img_size = batch["img_size"]++ with torch.no_grad():+ feat_img = model.encode_image(imgs)+ feat_img = F.normalize(feat_img, dim=1)++ feat_list.append(feat_img)+ indices_list.extend(indices.tolist())+ img_size_list.extend(img_size)++ feats = torch.cat(feat_list, dim=0)++ # all_gather+ feats_gathered = [torch.zeros_like(feats, device=device) for _ in range(dist.get_world_size())]+ dist.all_gather(feats_gathered, feats)+ feats_all = torch.cat(feats_gathered)++ indices_gathered = [None for _ in range(dist.get_world_size())]+ dist.all_gather_object(indices_gathered, indices_list)+ indices_all = np.array([x for sub in indices_gathered for x in sub])++ img_size_gathered = [None for _ in range(dist.get_world_size())]+ dist.all_gather_object(img_size_gathered, img_size_list)+ img_size_all = [x for sub in img_size_gathered for x in sub]++ indices_unique, indices_unique_idx = np.unique(indices_all, return_index=True)+ feats_unique = feats_all[torch.from_numpy(indices_unique_idx)]+ img_size_unique = [img_size_all[x] for x in indices_unique_idx]++ if dist.get_rank() == 0:+ # get similarity scores+ non_zero_list = []+ sim_scores_list = []+ chunk = 1000+ for idx in tqdm(range(0, feats_unique.shape[0], chunk)):+ sim_mat = torch.matmul(feats_unique[idx : idx + chunk], feats_unique[idx:].T).cpu().numpy()+ sim_mat_upper = np.triu(sim_mat, k=1)+ non_zero_i = np.nonzero(sim_mat_upper >= args.thresh)+ sim_scores_i = sim_mat[non_zero_i]++ non_zero_np = np.stack(non_zero_i) + idx # [2, N]+ non_zero_list.append(non_zero_np)+ sim_scores_list.append(sim_scores_i)++ non_zero_indices = np.concatenate(non_zero_list, axis=1)+ sim_scores = np.concatenate(sim_scores_list)+ dup_dict = {}+ for x, y, s in zip(non_zero_indices[0].tolist(), non_zero_indices[1].tolist(), sim_scores.tolist()):+ # only count pairs with same the same size+ if img_size_unique[x] != img_size_unique[y]:+ continue++ if y not in dup_dict:+ dup_dict[y] = (x, s)+ elif dup_dict[y][1] < s:+ dup_dict[y] = (x, s)++ dup_list = [(k, v) for k, v in dup_dict.items()]+ dup_list = sorted(dup_list, key=lambda x: x[1][1], reverse=True)+ dup_inds = [x[0] for x in dup_list]+ sim_inds = [x[1][0] for x in dup_list]+ sim_scores_dup = [x[1][1] for x in dup_list]++ remain_inds = sorted(list(set(indices_unique.tolist()) - set(dup_inds)))++ # save+ meta_unique = dataset.meta.iloc[remain_inds]+ meta_unique.to_csv(out_path_dedup, index=False)+ print(f"New meta without duplication saved to '{out_path_dedup}'.")++ meta_dup = dataset.meta.iloc[dup_inds].copy().reset_index(drop=True)+ path_dup = dataset.meta.iloc[sim_inds]["path"].copy().reset_index(drop=True)+ meta_dup["path_dup"] = path_dup+ meta_dup["sim"] = sim_scores_dup+ meta_dup.to_csv(out_path_dup, index=False)+ print(f"New meta with duplicated samples saved to '{out_path_dup}'.")+++if __name__ == "__main__":+ main()diff --git a/tools/scoring/motion/README.md b/tools/scoring/motion/README.mdnew file mode 100644index 0000000..e8c28b7--- /dev/null+++ b/tools/scoring/motion/README.md@@ -0,0 +1,12 @@+# Installation++Please install the VMAF and FFMPEG package to base environment.++## VMAF++For calculating motion score, vmaf score, we use VMAF with FFMPEG, please follow the installation guides [VMAF](https://github.com/Netflix/vmaf/blob/master/libvmaf/README.md#install) and [here](https://github.com/Netflix/vmaf/blob/master/resource/doc/ffmpeg.md) and install the required FFMPEG software with VMAF support. Not that you need to export the path to VMAF before installing FFMPEG.+```+export LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH++# change to the directory on your machine that contains libvmaf.so.3+```diff --git a/tools/scoring/motion/motion.py b/tools/scoring/motion/motion.pynew file mode 100644index 0000000..d197571--- /dev/null+++ b/tools/scoring/motion/motion.py@@ -0,0 +1,148 @@+# Please follow the instructions in the README.md file to install dependency.++import argparse+import json+import math+import multiprocessing+import os+import shutil+import subprocess++import pandas as pd+import tqdm++# Constants+TEMP_FOLDER = "/mnt/ddn/yeanbang/experiments/data_pipeline/temp_videos"+DOCKER_IMAGE = "vmaf_im"+EXTRACT_MVS_RUNNABLE_PATH = "~/extract_mvs"+GET_VMAF_MOTION_SCORE = True+GET_MOTION_VECTOR_AVERAGE = True++VMAF_CMD = """ffmpeg \+ -nostats -loglevel 0 \+ -r 24 -i "$SRC_VIDEO$" \+ -r 24 -i "$SRC_VIDEO$" \+ -lavfi "[0:v]setpts=PTS-STARTPTS[reference]; \+ [1:v]setpts=PTS-STARTPTS[distorted]; \+ [distorted][reference]libvmaf=log_fmt=json:log_path=$OUTPUT_PATH$:n_threads=4" \+ -f null -"""+++def create_temp_folder():+ """Creates a temp folder if it doesn't exist."""+ if not os.path.exists(TEMP_FOLDER):+ os.makedirs(TEMP_FOLDER)+++def calculate_magnitude(mvx, mvy):+ return math.sqrt(mvx**2 + mvy**2)+++def get_average_motion_vector_strength(motion_vector_log_file):+ """Calculates the average strength of motion vectors in a log file."""++ # Initialize variables to calculate the average+ total_magnitude = 0+ vector_count = 0+ # Open and parse the log file+ df = pd.read_csv(motion_vector_log_file)+ motion_x = df["motion_x"]+ motion_y = df["motion_y"]+ # Calculate the magnitude of each motion vector+ for i in range(len(motion_x)):+ magnitude = calculate_magnitude(motion_x[i], motion_y[i])+ total_magnitude += magnitude+ vector_count += 1+ if vector_count == 0:+ return 0+ return total_magnitude / vector_count+++def process_video(video_info):+ """Processes a video using the Docker command."""++ index, video_path = video_info+ average_motion_vector_strength = 0.0+ motion_score = 0.0+ vmaf_score = 0.0++ if GET_VMAF_MOTION_SCORE:+ try:+ vmaf_cmd = VMAF_CMD.replace("$SRC_VIDEO$", video_path).replace(+ "$OUTPUT_PATH$", f"{TEMP_FOLDER}/output_{index}.json"+ )+ subprocess.run(vmaf_cmd, shell=True, check=True)+ except Exception as e:+ print(f"Error processing video {index}: {e}")+ return index, motion_score, vmaf_score, average_motion_vector_strength+ finally:+ # Read the output JSON file+ output_file = f"{TEMP_FOLDER}/output_{index}.json"+ if os.path.exists(output_file):+ with open(output_file) as f:+ data = json.load(f)+ # ref: https://video.stackexchange.com/questions/24210/how-should-i-interpret-the-results-from-netflix-vmaf+ motion_score = (+ data["pooled_metrics"]["integer_motion"]["mean"]+ + data["pooled_metrics"]["integer_motion2"]["mean"]+ ) / 2.0+ vmaf_score = data["pooled_metrics"]["vmaf"]["mean"]++ os.remove(output_file)++ if GET_MOTION_VECTOR_AVERAGE:+ try:+ command = f'{EXTRACT_MVS_RUNNABLE_PATH} "{video_path}" > {TEMP_FOLDER}/{index}_motion_vectors.csv'+ subprocess.run(command, shell=True, check=True)+ average_motion_vector_strength = get_average_motion_vector_strength(+ f"{TEMP_FOLDER}/{index}_motion_vectors.csv"+ )+ except Exception as e:+ print(f"Error processing video {index}: {e}")+ finally:+ if os.path.exists(f"{TEMP_FOLDER}/{index}_motion_vectors.csv"):+ os.remove(f"{TEMP_FOLDER}/{index}_motion_vectors.csv")++ return index, motion_score, vmaf_score, average_motion_vector_strength+++def process_videos_parallel(csv_file, num_processes=100):+ """Processes videos in parallel."""+ # Read CSV file+ df = pd.read_csv(csv_file)++ if "motion_score" not in df.columns and GET_VMAF_MOTION_SCORE:+ df["motion_score"] = 0.0+ df["vmaf_score"] = 0.0+ if "average_motion_vector_strength" not in df.columns and GET_MOTION_VECTOR_AVERAGE:+ df["average_motion_vector_strength"] = 0.0++ # Create temp folder if it doesn't exist+ create_temp_folder()++ # Create a pool of workers to process videos in parallel++ with multiprocessing.Pool(num_processes) as pool:+ # Each worker processes one video+ for result in list(tqdm.tqdm(pool.imap(process_video, enumerate(df["path"])), total=len(df))):+ index, motion_score, vmaf_score, average_motion_vector_strength = result+ # Update the dataframe with the motion score+ if GET_VMAF_MOTION_SCORE:+ df.at[index, "motion_score"] = motion_score+ df.at[index, "vmaf_score"] = vmaf_score+ if GET_MOTION_VECTOR_AVERAGE:+ df.at[index, "average_motion_vector_strength"] = average_motion_vector_strength++ # Remove temp folder after processing+ if os.path.exists(TEMP_FOLDER):+ shutil.rmtree(TEMP_FOLDER)++ # Save the updated dataframe to a new CSV file+ df.to_csv(csv_file, index=False)+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--csv_file", type=str, required=True)+ args = parser.parse_args()+ process_videos_parallel(args.csv_file)diff --git a/tools/scoring/optical_flow/detect_zoom.py b/tools/scoring/optical_flow/detect_zoom.pynew file mode 100644index 0000000..7dafb08--- /dev/null+++ b/tools/scoring/optical_flow/detect_zoom.py@@ -0,0 +1,235 @@+"""+detect zoom in/zoom out videos+"""++import argparse++# cv2.setNumThreads(1)+from functools import partial++import cv2+import pandas as pd+import scipy.stats as stats+from pandarallel import pandarallel+from tqdm import tqdm++# thresholds+max_frames = 5 # max number of times to run correlation analysis+p_threshold = 0.1+tau_threshold = 0.50 # correlation above which to determine as zoom+crop_ratio = 2 / 3+zoom_frame_count_threshold = 0.4 # % of frames above which has zoom+image_size = (256, 256)+min_corr_counts = 10 # minimum number of points required to calc correlation+sample_interval = 10 # need to be larger than 10 to be accurate+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("meta_path", type=str, help="Path to the input CSV file")+ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")+ parser.add_argument("--disable_parallel", action="store_true", help="disable parallel processing")+ parser.add_argument("--num_samples", type=int, default=-1, help="number of samples to process, for quick tests")++ args = parser.parse_args()+ return args+++def crop_resize(image, image_size):+ # crop the edge to remove watermarks+ h, w = image.shape+ crop_h, crop_w = int(h * crop_ratio), int(w * crop_ratio)+ start_h = (h - crop_h) // 2+ start_w = (w - crop_w) // 2+ image = image[start_h : start_h + crop_h, start_w : start_w + crop_w]++ # resize+ h, w = image.shape+ h_t, w_t = image_size+ scale = max(h_t / h, w_t / w)+ resized_image = cv2.resize(image, (int(w * scale), int(h * scale))) # NOTE: strage but cv2 need to put w h++ # center crop+ h, w = resized_image.shape[:2]+ # Calculate the center crop dimensions+ crop_h, crop_w = image_size+ start_h = (h - crop_h) // 2+ start_w = (w - crop_w) // 2+ center_cropped_image = resized_image[start_h : start_h + crop_h, start_w : start_w + crop_w]++ return center_cropped_image+++def is_dynamic_video(+ video_path,+):+ cap = cv2.VideoCapture(video_path)++ # Check if the video opened successfully+ if not cap.isOpened():+ print(f"Error: Could not open video '{video_path}'")+ return False++ frame_index = -1++ p0 = None+ while p0 is None:+ frame_index += 1+ cap.set(cv2.CAP_PROP_POS_FRAMES, frame_index)+ success, frame = cap.read()+ if not success:+ print(f"Warning: Failed to read frame {frame_index} of video '{video_path}'")+ break++ prev_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)+ prev_gray = crop_resize(prev_gray, image_size)++ # Feature parameters+ feature_params = dict(maxCorners=100, qualityLevel=0.3, minDistance=7, blockSize=7)+ p0 = cv2.goodFeaturesToTrack(prev_gray, mask=None, **feature_params)+ if p0 is None:+ return True++ # Calculate flow+ # Parameters for optical flow+ lk_params = dict(winSize=(15, 15), maxLevel=2, criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))++ zoom_count = 0+ analyzed_count = 0+ invalid_count = 0++ while analyzed_count < max_frames:+ if invalid_count >= 3:+ break++ frame_index += sample_interval++ cap.set(cv2.CAP_PROP_POS_FRAMES, frame_index)+ success, frame = cap.read()+ if not success:+ break++ # Convert to grayscale+ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)+ gray = crop_resize(gray, image_size)++ # Calculate optical flow+ p1, status, err = cv2.calcOpticalFlowPyrLK(prev_gray, gray, p0, None, **lk_params)++ if p1 is None:+ invalid_count += 1+ continue++ # Select good points+ good_new = p1[status == 1] # status == 1: found flow; status == 0: didn't+ good_old = p0[status == 1]++ # Calculate motion vectors+ motion_vectors = good_new - good_old++ motion_x_neg = []+ x_neg = []+ motion_x_pos = []+ x_pos = []+ motion_y_neg = []+ y_neg = []+ motion_y_pos = []+ y_pos = []++ for i, (motion_x, motion_y) in enumerate(motion_vectors):+ if motion_x < 0:+ motion_x_neg.append(motion_x)+ x_neg.append(good_old[i][0])+ else:+ motion_x_pos.append(motion_x)+ x_pos.append(good_old[i][0])++ if motion_y < 0:+ motion_y_neg.append(motion_y)+ y_neg.append(good_old[i][1])+ else:+ motion_y_pos.append(motion_y)+ y_pos.append(good_old[i][1])++ is_zoom = True+ corr_checks = 0++ if len(motion_x_neg) > min_corr_counts:+ corr_checks += 1+ tau_x_neg, p_x_neg = stats.kendalltau(motion_x_neg, x_neg)+ if p_x_neg > p_threshold or abs(tau_x_neg) < tau_threshold:+ is_zoom = False+ if len(motion_x_pos) > min_corr_counts:+ corr_checks += 1+ tau_x_pos, p_x_pos = stats.kendalltau(motion_x_pos, x_pos)+ if p_x_pos > p_threshold or abs(tau_x_pos) < tau_threshold:+ is_zoom = False+ if len(motion_y_neg) > min_corr_counts:+ corr_checks += 1+ tau_y_neg, p_y_neg = stats.kendalltau(motion_y_neg, y_neg)+ if p_y_neg > p_threshold or abs(tau_y_neg) < tau_threshold:+ is_zoom = False+ if len(motion_y_pos) > min_corr_counts:+ corr_checks += 1+ tau_y_pos, p_y_pos = stats.kendalltau(motion_y_pos, y_pos)+ if p_y_pos > p_threshold or abs(tau_y_pos) < tau_threshold:+ is_zoom = False++ if corr_checks > 0:+ if is_zoom:+ zoom_count += 1+ analyzed_count += 1+ else:+ invalid_count += 1++ # Update previous frame and points+ prev_gray = gray.copy()+ p0 = good_new.reshape(-1, 1, 2)++ cap.release()++ if analyzed_count > 2 and zoom_count / analyzed_count >= zoom_frame_count_threshold:+ return False+ else:+ return True+++def process_single_row(row, args):+ path = row["path"]+ return is_dynamic_video(path)+++def main():+ args = parse_args()+ meta_path = args.meta_path++ tqdm.pandas()+ if not args.disable_parallel:+ if args.num_workers is not None:+ pandarallel.initialize(progress_bar=True, nb_workers=args.num_workers)+ else:+ pandarallel.initialize(progress_bar=True)+ process_single_row_partial = partial(process_single_row, args=args)++ meta = pd.read_csv(meta_path)+ if args.num_samples > -1:+ meta = meta[: args.num_samples]++ if not args.disable_parallel:+ ret = meta.parallel_apply(process_single_row_partial, axis=1)+ else:+ ret = meta.progress_apply(process_single_row_partial, axis=1)+ meta0 = meta[ret]+ meta1 = meta[~ret]++ out_path = meta_path.replace(".csv", "_filter-zoom.csv")+ meta0.to_csv(out_path, index=False)+ print(f"New meta (shape={meta0.shape}) saved to '{out_path}'")++ out_path = meta_path.replace(".csv", "_zoom.csv")+ meta1.to_csv(out_path, index=False)+ print(f"New meta (shape={meta1.shape}) saved to '{out_path}'")+++if __name__ == "__main__":+ main()diff --git a/tools/scoring/optical_flow/detect_zoom_av.py b/tools/scoring/optical_flow/detect_zoom_av.pynew file mode 100644index 0000000..4edc445--- /dev/null+++ b/tools/scoring/optical_flow/detect_zoom_av.py@@ -0,0 +1,246 @@+"""+detect zoom in/zoom out videos+This file is deprecated: open video with av, inaccurate frames+"""++import argparse++# cv2.setNumThreads(1)+from functools import partial++import cv2+import numpy as np+import pandas as pd+import scipy.stats as stats+from pandarallel import pandarallel+from tqdm import tqdm++from tools.datasets.utils import extract_frames++# hyper-parameters+max_frames = 5 # max number of times to run correlation analysis+p_threshold = 0.1+tau_threshold = 0.60 # correlation above which to determine as zoom+crop_ratio = 2 / 3+zoom_frame_count_threshold = 0.4 # % of frames above which has zoom+image_size = (256, 256)+min_corr_counts = 10 # minimum number of points required to calc correlation+frame_interval = 10 # need to be larger than 10 to be accurate+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("meta_path", type=str, help="Path to the input CSV file")+ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")+ parser.add_argument("--disable_parallel", action="store_true", help="disable parallel processing")+ parser.add_argument("--num_samples", type=int, default=-1, help="number of samples to process, for quick tests")++ args = parser.parse_args()+ return args+++def crop_resize(image, image_size):+ # crop the edge to remove watermarks+ h, w = image.shape+ crop_h, crop_w = int(h * crop_ratio), int(w * crop_ratio)+ start_h = (h - crop_h) // 2+ start_w = (w - crop_w) // 2+ image = image[start_h : start_h + crop_h, start_w : start_w + crop_w]++ # resize+ h, w = image.shape+ h_t, w_t = image_size+ scale = max(h_t / h, w_t / w)+ resized_image = cv2.resize(image, (int(w * scale), int(h * scale))) # NOTE: strage but cv2 need to put w h++ # center crop+ h, w = resized_image.shape[:2]+ # Calculate the center crop dimensions+ crop_h, crop_w = image_size+ start_h = (h - crop_h) // 2+ start_w = (w - crop_w) // 2+ center_cropped_image = resized_image[start_h : start_h + crop_h, start_w : start_w + crop_w]++ return center_cropped_image+++def is_not_zoom_video(+ video_path,+):+ interval_first = 5+ try:+ first_frames = extract_frames(+ video_path,+ frame_inds=list(range(0, 5 * interval_first + 1, interval_first)),+ backend="av",+ )+ except:+ return False++ # fine feature to track+ p0 = None+ feature_params = dict(maxCorners=100, qualityLevel=0.3, minDistance=7, blockSize=7)+ for idx, frame in enumerate(first_frames):+ frame_np = np.array(frame)++ prev_gray = cv2.cvtColor(frame_np, cv2.COLOR_RGB2GRAY)+ prev_gray = crop_resize(prev_gray, image_size)++ # Feature parameters+ p0 = cv2.goodFeaturesToTrack(prev_gray, mask=None, **feature_params) # [N, 1, 2]+ if p0 is not None:+ break+ if p0 is None:+ return False++ # calculate flow at a regular interval+ zoom_count = 0+ analyzed_count = 0+ invalid_counts = 0+ # Parameters for optical flow+ lk_params = dict(winSize=(15, 15), maxLevel=2, criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))++ frame_inds = list(+ range(+ idx * interval_first + frame_interval,+ idx * interval_first + max_frames * frame_interval + 1,+ frame_interval,+ )+ )+ frames = extract_frames(+ video_path,+ frame_inds=frame_inds,+ backend="av",+ ) # TODO: non-precise frames+ for idx, frame in enumerate(frames):+ # Convert to grayscale+ frame_np = np.array(frame)+ gray = cv2.cvtColor(frame_np, cv2.COLOR_RGB2GRAY)+ gray = crop_resize(gray, image_size)++ # Calculate optical flow+ p1, st, err = cv2.calcOpticalFlowPyrLK(prev_gray, gray, p0, None, **lk_params)++ if invalid_counts >= 3:+ break+ if p1 is None:+ invalid_counts += 1+ continue++ # Select good points+ good_new = p1[st == 1]+ good_old = p0[st == 1]++ # Calculate motion vectors+ motion_vectors = good_new - good_old++ motion_x_neg = []+ x_neg = []+ motion_x_pos = []+ x_pos = []+ motion_y_neg = []+ y_neg = []+ motion_y_pos = []+ y_pos = []++ for i, (motion_x, motion_y) in enumerate(motion_vectors):+ if motion_x < 0:+ motion_x_neg.append(motion_x)+ x_neg.append(good_old[i][0])+ else:+ motion_x_pos.append(motion_x)+ x_pos.append(good_old[i][0])++ if motion_y < 0:+ motion_y_neg.append(motion_y)+ y_neg.append(good_old[i][1])+ else:+ motion_y_pos.append(motion_y)+ y_pos.append(good_old[i][1])++ is_zoom = True+ corr_checks = 0++ if len(motion_x_neg) > min_corr_counts:+ corr_checks += 1+ tau_x_neg, p_x_neg = stats.kendalltau(motion_x_neg, x_neg)+ if p_x_neg > p_threshold or abs(tau_x_neg) < tau_threshold:+ is_zoom = False+ if len(motion_x_pos) > min_corr_counts:+ corr_checks += 1+ tau_x_pos, p_x_pos = stats.kendalltau(motion_x_pos, x_pos)+ if p_x_pos > p_threshold or abs(tau_x_pos) < tau_threshold:+ is_zoom = False+ if len(motion_y_neg) > min_corr_counts:+ corr_checks += 1+ tau_y_neg, p_y_neg = stats.kendalltau(motion_y_neg, y_neg)+ if p_y_neg > p_threshold or abs(tau_y_neg) < tau_threshold:+ is_zoom = False+ if len(motion_y_pos) > min_corr_counts:+ corr_checks += 1+ tau_y_pos, p_y_pos = stats.kendalltau(motion_y_pos, y_pos)+ if p_y_pos > p_threshold or abs(tau_y_pos) < tau_threshold:+ is_zoom = False++ if corr_checks > 0:+ if is_zoom:+ zoom_count += 1+ analyzed_count += 1+ else:+ invalid_counts += 1++ # Update previous frame and points+ prev_gray = gray.copy()+ p0 = good_new.reshape(-1, 1, 2)++ if analyzed_count > 2 and zoom_count / analyzed_count >= zoom_frame_count_threshold:+ return False+ else:+ return True+++def process_single_row(row, args):+ path = row["path"]+ return is_not_zoom_video(path)+++def main():+ args = parse_args()+ meta_path = args.meta_path++ # # time test+ # start_time = time.time()++ tqdm.pandas()+ if not args.disable_parallel:+ if args.num_workers is not None:+ pandarallel.initialize(progress_bar=True, nb_workers=args.num_workers)+ else:+ pandarallel.initialize(progress_bar=True)+ process_single_row_partial = partial(process_single_row, args=args)++ meta = pd.read_csv(meta_path)+ if args.num_samples > -1:+ meta = meta[: args.num_samples]++ if not args.disable_parallel:+ ret = meta.parallel_apply(process_single_row_partial, axis=1)+ else:+ ret = meta.progress_apply(process_single_row_partial, axis=1)+ meta0 = meta[ret]+ meta1 = meta[~ret]++ out_path = meta_path.replace(".csv", "_filter-zoom.csv")+ meta0.to_csv(out_path, index=False)+ print(f"New meta (shape={meta0.shape}) saved to '{out_path}'")++ out_path = meta_path.replace(".csv", "_zoom.csv")+ meta1.to_csv(out_path, index=False)+ print(f"New meta (shape={meta1.shape}) saved to '{out_path}'")++ # # time test+ # print("execution time:", time.time() - start_time)+++if __name__ == "__main__":+ main()diff --git a/tools/scoring/optical_flow/inference.py b/tools/scoring/optical_flow/inference.pyindex 09f1b41..057e1f3 100644--- a/tools/scoring/optical_flow/inference.py+++ b/tools/scoring/optical_flow/inference.py@@ -11,16 +11,68 @@ import torchimport torch.distributed as distimport torch.nn.functional as Ffrom einops import rearrange+from PIL import Imagefrom torch.utils.data import DataLoader, DistributedSamplerfrom torchvision.transforms.functional import pil_to_tensorfrom tqdm import tqdm-from tools.datasets.utils import extract_frames+# from tools.datasets.utils import extract_framesfrom tools.scoring.optical_flow.unimatch import UniMatch# torch.backends.cudnn.enabled = False # This line enables large batch, but the speed is similar+def extract_frames(+ video_path,+ frame_inds=None,+ points=None,+ backend="opencv",+ return_length=False,+ num_frames=None,+):+ """+ Args:+ video_path (str): path to video+ frame_inds (List[int]): indices of frames to extract+ points (List[float]): values within [0, 1); multiply #frames to get frame indices+ Return:+ List[PIL.Image]+ """+ assert backend in ["av", "opencv", "decord"]+ assert (frame_inds is None) or (points is None)+ assert backend == "opencv"++ cap = cv2.VideoCapture(video_path)+ if num_frames is not None:+ total_frames = num_frames+ else:+ total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))++ if points is not None:+ frame_inds = [int(p * total_frames) for p in points]++ frames = []+ for idx in frame_inds:+ if idx >= total_frames:+ idx = total_frames - 1++ success = cap.set(cv2.CAP_PROP_POS_FRAMES, idx)+ if not success:+ break++ try:+ ret, frame = cap.read()+ frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)+ frame = Image.fromarray(frame)+ frames.append(frame)+ except Exception:+ continue++ if return_length:+ return frames, total_frames+ return frames++def merge_scores(gathered_list: list, meta: pd.DataFrame, column):# reorderindices_list = list(map(lambda x: x[0], gathered_list))@@ -45,7 +97,7 @@ def merge_scores(gathered_list: list, meta: pd.DataFrame, column):class VideoTextDataset(torch.utils.data.Dataset):- def __init__(self, meta_path, frame_inds=[0, 10, 20, 30]):+ def __init__(self, meta_path, frame_inds=None):self.meta_path = meta_pathself.meta = pd.read_csv(meta_path)self.frame_inds = frame_inds@@ -78,13 +130,14 @@ class VideoTextDataset(torch.utils.data.Dataset):def parse_args():parser = argparse.ArgumentParser()parser.add_argument("meta_path", type=str, help="Path to the input CSV file")- parser.add_argument("--bs", type=int, default=4, help="Batch size") # don't use too large bs for unimatch+ parser.add_argument("--bs", type=int, default=1, help="Batch size") # don't use too large bs for unimatchparser.add_argument("--num_workers", type=int, default=16, help="Number of workers")parser.add_argument("--skip_if_existing", action="store_true")args = parser.parse_args()return args[email protected]_grad()def main():args = parse_args()@@ -121,7 +174,9 @@ def main():model = model.to(device)# build dataset- dataset = VideoTextDataset(meta_path=meta_path, frame_inds=[0, 10, 20, 30])+ NUM_FRAMES = 10+ frames_inds = [15 * i for i in range(0, NUM_FRAMES)]+ dataset = VideoTextDataset(meta_path=meta_path, frame_inds=frames_inds)dataloader = DataLoader(dataset,batch_size=args.bs,@@ -141,31 +196,29 @@ def main():model.eval()for batch in tqdm(dataloader, disable=dist.get_rank() != 0):indices = batch["index"]- images = batch["images"].to(device, non_blocking=True)+ images = batch["images"].to(device)B = images.shape[0]batch_0 = rearrange(images[:, :-1], "B N C H W -> (B N) C H W").contiguous()batch_1 = rearrange(images[:, 1:], "B N C H W -> (B N) C H W").contiguous()- with torch.no_grad():- res = model(- batch_0,- batch_1,- attn_type="swin",- attn_splits_list=[2, 8],- corr_radius_list=[-1, 4],- prop_radius_list=[-1, 1],- num_reg_refine=6,- task="flow",- pred_bidir_flow=False,- )- flow_maps = res["flow_preds"][-1].cpu() # [B * (N-1), 2, H, W]- flow_maps = rearrange(flow_maps, "(B N) C H W -> B N H W C", B=B)- flow_scores = flow_maps.abs().mean(dim=[1, 2, 3, 4])- flow_scores = flow_scores.tolist()+ res = model(+ batch_0,+ batch_1,+ attn_type="swin",+ attn_splits_list=[2, 8],+ corr_radius_list=[-1, 4],+ prop_radius_list=[-1, 1],+ num_reg_refine=6,+ task="flow",+ pred_bidir_flow=False,+ )+ flow_maps = res["flow_preds"][-1] # [B * (N-1), 2, H, W]+ flow_maps = rearrange(flow_maps, "(B N) C H W -> B N H W C", B=B)+ flow_scores = flow_maps.norm(dim=-1).mean(dim=[1, 2, 3]).cpu()indices_list.extend(indices.tolist())- scores_list.extend(flow_scores)+ scores_list.extend(flow_scores.tolist())# save local resultsmeta_local = merge_scores([(indices_list, scores_list)], dataset.meta, column="flow")