Open-Sora: Democratizing Efficient Video Production for All
Python
reformat and update docs
zhengzangw525e29abbecc
diff --git a/README.md b/README.mdindex 4512769..ae74a43 100644--- a/README.md+++ b/README.md@@ -137,9 +137,9 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.* [Gradio Demo](#gradio-demo)* [Inference](#inference)* [Data Processing](#data-processing)-* [VAE](#vae)* [Training](#training)* [Evaluation](#evaluation)+* [VAE Training & Evaluation](#vae-training--evaluation)* [Contribution](#contribution)* [Citation](#citation)* [Acknowledgement](#acknowledgement)@@ -152,14 +152,14 @@ Other useful documents and links are listed below.* Useful commands: [commands.md](docs/commands.md)* Data processing pipeline and dataset: [datasets.md](docs/datasets.md)* Each data processing tool's README: [dataset conventions and management](/tools/datasets/README.md), [scene cutting](/tools/scene_cut/README.md), [scoring](/tools/scoring/README.md), [caption](/tools/caption/README.md)-* Evaluation: [eval](/eval/README.md)+* Evaluation: [eval/README.md](/eval/README.md)* Gallery: [gallery](https://hpcaitech.github.io/Open-Sora/)## Installation### Install from Source-For CUDA 12.1, you can install the dependencies with the following commands. Otherwise, please refer to [Installation](docs/installation.md) for more instructions on different cuda version, and additional dependency for data preprocessing, VAE, and model evaluation.+For CUDA 12.1, you can install the dependencies with the following commands. Otherwise, please refer to [Installation Documentation](docs/installation.md) for more instructions on different cuda version, and additional dependency for data preprocessing, VAE, and model evaluation.```bash# create a virtual env and activate (conda as an example)@@ -175,7 +175,7 @@ cd Open-Sora# the default installation is for inference onlypip install -v . # for development mode, `pip install -v -e .`-+```(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.@@ -211,9 +211,9 @@ docker run -ti --gpus all -v {MOUNT_DIR}:/data opensora| Model | Model Size | Data | #iterations | Batch Size | URL || --------- | ---------- | ---- | ----------- | ---------- | ------------------------------------------------------------- || Diffusion | 1.1B | 30M | 70k | Dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3) |-| VAE | 384M | 3M | 1.18M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2) |+| VAE | 384M | 3M | 1M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2) |-See our **[report 1.2](docs/report_03.md)** for more infomation.+See our **[report 1.2](docs/report_03.md)** for more infomation. Weight will be automatically downloaded when you run the inference script.### Open-Sora 1.1 Model Weights@@ -409,35 +409,6 @@ Also check out the [datasets](docs/datasets.md) we use.-## VAE-We train a VAE pipeline that consists of a spatial VAE followed by a temporal VAE.-For more details, refer to [VAE](docs/vae.md).-Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.--If you want to train your own VAE, we need to prepare data in the csv following the [data processing](#data-processing) pipeline, then run the following commands.-Note that you need to adjust the number of trained epochs (`epochs`) in the config file accordingly with respect to your own csv data size.--```bash-# stage 1 training, 380k steps, 8 GPUs-torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH-# stage 2 training, 260k steps, 8 GPUs-torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH-# stage 3 training, 540k steps, 24 GPUs-torchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH-```-To evaluate the VAE performance, you need to run VAE inference first to generate the videos, then calculate scores on the generated videos:--```bash-# video generation-torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR-# the original videos will be saved to `YOUR_VIDEO_DIR_ori`-# the reconstructed videos through the pipeline will be saved to `YOUR_VIDEO_DIR_rec`-# the reconstructed videos through the spatial VAE only will be saved to `YOUR_VIDEO_DIR_spatial`--# score calculation-python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips-```-## Training### Open-Sora 1.2 Training@@ -499,11 +470,50 @@ For training other models and advanced usage, see [here](docs/commands.md) for m## Evaluation-### VBench+We support evaluation based on:++* Validation loss+* VBench score+* VBench-i2v score+* Batch generation for human evaluation-### VBench-i2v+All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details. Our [report](/docs/report_03.md#evaluation) also provides more information about the evaluation during training. The following table shows Open-Sora 1.2 greatly improves Open-Sora 1.0.-See [here](eval/README.md) for more instructions.+| Model | Total Score | Quality Score | Semantic Score |+| -------------- | ----------- | ------------- | -------------- |+| Open-Sora V1.0 | 75.91% | 78.81% | 64.28% |+| Open-Sora V1.2 | 79.23% | 80.71% | 73.30% |++## VAE Training & Evaluation++We train a VAE pipeline that consists of a spatial VAE followed by a temporal VAE.+For more details, refer to [VAE Documentation](docs/vae.md).+Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.++If you want to train your own VAE, we need to prepare data in the csv following the [data processing](#data-processing) pipeline, then run the following commands.+Note that you need to adjust the number of trained epochs (`epochs`) in the config file accordingly with respect to your own csv data size.++```bash+# stage 1 training, 380k steps, 8 GPUs+torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH+# stage 2 training, 260k steps, 8 GPUs+torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH+# stage 3 training, 540k steps, 24 GPUs+torchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH+```++To evaluate the VAE performance, you need to run VAE inference first to generate the videos, then calculate scores on the generated videos:++```bash+# video generation+torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR+# the original videos will be saved to `YOUR_VIDEO_DIR_ori`+# the reconstructed videos through the pipeline will be saved to `YOUR_VIDEO_DIR_rec`+# the reconstructed videos through the spatial VAE only will be saved to `YOUR_VIDEO_DIR_spatial`++# score calculation+python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips+```## Contributiondiff --git a/configs/opensora-v1-2/train/stage2.py b/configs/opensora-v1-2/train/stage2.pyindex 30dce99..033ae6d 100644--- a/configs/opensora-v1-2/train/stage2.py+++ b/configs/opensora-v1-2/train/stage2.py@@ -88,4 +88,3 @@ grad_clip = 1.0lr = 1e-4ema_decay = 0.99adam_eps = 1e-15-diff --git a/docker/Dockerfile b/docker/Dockerfileindex a361bfb..de97f11 100644--- a/docker/Dockerfile+++ b/docker/Dockerfile@@ -22,4 +22,4 @@ RUN pip install xformers --index-url https://download.pytorch.org/whl/cu121# install this projectRUN git clone https://github.com/hpcaitech/Open-Sora && \cd Open-Sora && \- pip install -v .\ No newline at end of file+ pip install -v .diff --git a/docs/data_processing.md b/docs/data_processing.mdindex b4b677e..129c812 100644--- a/docs/data_processing.md+++ b/docs/data_processing.md@@ -1,81 +1,81 @@-# Data Processing->Open-Sora v1.2 uses Data Propcessing Pipeline v1.1.--We establish a complete pipeline for video/image data processing. The pipeline is shown below.----First, raw videos,-either from the Internet or public datasets, are split into shorter clips based on scene detection.-Then, we evaluate these videos by predicting multiple scores using existing models. We first predict the aesthetic score-and the optical flow score for a video. We also conduct OCR to detect texts in the video. Only videos with satisfactory-evaluation results are sent to the next step for captioning. After captioning, the matching score is also calculated as-an assessment of video-text alignment. Finally, we filter samples based on the matching score and-conduct camera motion detection for the remaining samples.-In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong-semantic consistency.--Below is an example workflow to process videos.--```bash-ROOT_VIDEO="/path/to/video/folder"-ROOT_CLIPS="/path/to/video/clips/folder"-ROOT_META="/path/to/meta/folder"--# 1.1 Create a meta file from a video folder. This should output ${ROOT_META}/meta.csv-python -m tools.datasets.convert video ${ROOT_VIDEO} --output ${ROOT_META}/meta.csv--# 1.2 Get video information and remove broken videos. This should output ${ROOT_META}/meta_info_fmin1.csv-python -m tools.datasets.datautil ${ROOT_META}/meta.csv --info --fmin 1--# 2.1 Detect scenes. This should output ${ROOT_META}/meta_info_fmin1_timestamp.csv-python -m tools.scene_cut.scene_detect ${ROOT_META}/meta_info_fmin1.csv--# 2.2 Cut video into clips based on scenes. This should produce video clips under ${ROOT_CLIPS}-python -m tools.scene_cut.cut ${ROOT_META}/meta_info_fmin1_timestamp.csv --save_dir ${ROOT_CLIPS}--# 2.3 Create a meta file for video clips. This should output ${ROOT_META}/meta_clips.csv-python -m tools.datasets.convert video ${ROOT_CLIPS} --output ${ROOT_META}/meta_clips.csv--# 2.4 Get clips information and remove broken ones. This should output ${ROOT_META}/meta_clips_info_fmin1.csv-python -m tools.datasets.datautil ${ROOT_META}/meta_clips.csv --info --fmin 1--# 3.1 Predict aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes.csv-torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference \- ${ROOT_META}/meta_clips_info_fmin1.csv \- --bs 1024 \- --num_workers 16--# 3.2 Filter by aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv-python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes.csv --aesmin 5--# 4.1 Generate caption. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv-torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava \- ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv \- --dp-size 8 \- --tp-size 1 \- --model-path /path/to/llava-v1.6-mistral-7b \- --prompt video--# 4.2 Merge caption results. This should output ${ROOT_META}/meta_clips_caption.csv-python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv --output ${ROOT_META}/meta_clips_caption.csv--# 4.3 Clean caption. This should output ${ROOT_META}/meta_clips_caption_cleaned.csv-python -m tools.datasets.datautil \- ${ROOT_META}/meta_clips_caption.csv \- --clean-caption \- --refine-llm-caption \- --remove-empty-caption \- --output ${ROOT_META}/meta_clips_caption_cleaned.csv--# 4.4 Optionally generate tags (e.g., objects) based on the captions. This should output your_output_prefix_{key}.csv-torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 ${ROOT_META}/meta_clips_caption_cleaned.csv --key objects --output_prefix your_output_prefix--```---For more information, please refer to:-- [Dataset Management](../tools/datasets/README.md)-- [Scene Detection and Video Splitting](../tools/scene_cut/README.md)-- [Scoring and Filtering](../tools/scoring/README.md)-- [Captioning](../tools/caption/README.md)\ No newline at end of file+# Data Processing+>Open-Sora v1.2 uses Data Propcessing Pipeline v1.1.++We establish a complete pipeline for video/image data processing. The pipeline is shown below.++++First, raw videos,+either from the Internet or public datasets, are split into shorter clips based on scene detection.+Then, we evaluate these videos by predicting multiple scores using existing models. We first predict the aesthetic score+and the optical flow score for a video. We also conduct OCR to detect texts in the video. Only videos with satisfactory+evaluation results are sent to the next step for captioning. After captioning, the matching score is also calculated as+an assessment of video-text alignment. Finally, we filter samples based on the matching score and+conduct camera motion detection for the remaining samples.+In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong+semantic consistency.++Below is an example workflow to process videos.++```bash+ROOT_VIDEO="/path/to/video/folder"+ROOT_CLIPS="/path/to/video/clips/folder"+ROOT_META="/path/to/meta/folder"++# 1.1 Create a meta file from a video folder. This should output ${ROOT_META}/meta.csv+python -m tools.datasets.convert video ${ROOT_VIDEO} --output ${ROOT_META}/meta.csv++# 1.2 Get video information and remove broken videos. This should output ${ROOT_META}/meta_info_fmin1.csv+python -m tools.datasets.datautil ${ROOT_META}/meta.csv --info --fmin 1++# 2.1 Detect scenes. This should output ${ROOT_META}/meta_info_fmin1_timestamp.csv+python -m tools.scene_cut.scene_detect ${ROOT_META}/meta_info_fmin1.csv++# 2.2 Cut video into clips based on scenes. This should produce video clips under ${ROOT_CLIPS}+python -m tools.scene_cut.cut ${ROOT_META}/meta_info_fmin1_timestamp.csv --save_dir ${ROOT_CLIPS}++# 2.3 Create a meta file for video clips. This should output ${ROOT_META}/meta_clips.csv+python -m tools.datasets.convert video ${ROOT_CLIPS} --output ${ROOT_META}/meta_clips.csv++# 2.4 Get clips information and remove broken ones. This should output ${ROOT_META}/meta_clips_info_fmin1.csv+python -m tools.datasets.datautil ${ROOT_META}/meta_clips.csv --info --fmin 1++# 3.1 Predict aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes.csv+torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference \+ ${ROOT_META}/meta_clips_info_fmin1.csv \+ --bs 1024 \+ --num_workers 16++# 3.2 Filter by aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv+python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes.csv --aesmin 5++# 4.1 Generate caption. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv+torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava \+ ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv \+ --dp-size 8 \+ --tp-size 1 \+ --model-path /path/to/llava-v1.6-mistral-7b \+ --prompt video++# 4.2 Merge caption results. This should output ${ROOT_META}/meta_clips_caption.csv+python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv --output ${ROOT_META}/meta_clips_caption.csv++# 4.3 Clean caption. This should output ${ROOT_META}/meta_clips_caption_cleaned.csv+python -m tools.datasets.datautil \+ ${ROOT_META}/meta_clips_caption.csv \+ --clean-caption \+ --refine-llm-caption \+ --remove-empty-caption \+ --output ${ROOT_META}/meta_clips_caption_cleaned.csv++# 4.4 Optionally generate tags (e.g., objects) based on the captions. This should output your_output_prefix_{key}.csv+torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 ${ROOT_META}/meta_clips_caption_cleaned.csv --key objects --output_prefix your_output_prefix++```+++For more information, please refer to:+- [Dataset Management](../tools/datasets/README.md)+- [Scene Detection and Video Splitting](../tools/scene_cut/README.md)+- [Scoring and Filtering](../tools/scoring/README.md)+- [Captioning](../tools/caption/README.md)diff --git a/docs/report_03.md b/docs/report_03.mdindex e9bb1cb..fba2f93 100644--- a/docs/report_03.md+++ b/docs/report_03.md@@ -130,7 +130,7 @@ During inference, we can also use the scores to condition the model. For cameraPreviously, we monitor the training process only by human evaluation, as DDPM traning loss is not well correlated with the quality of generated videos. However, for rectified flow, we find the training loss is well correlated with the quality of generated videos as stated in SD3. Thus, we keep track of rectified flow evaluation loss on 100 images and 1k videos.-We sampled 1k videos from pixabay as validation dataset. We calculate the evaluation loss for image and different lengths of videos (2s, 4s, 8s, 16s) for different resolution (144p, 240p, 360p, 480p, 720p). For each setting, we equidistantly sample 10 timesteps. Then all the losses are averaged.+We sampled 1k videos from pixabay as validation dataset. We calculate the evaluation loss for image and different lengths of videos (2s, 4s, 8s, 16s) for different resolution (144p, 240p, 360p, 480p, 720p). For each setting, we equidistantly sample 10 timesteps. Then all the losses are averaged. We also provide a [video](https://streamable.com/oqkkf1) showing the sampled videos with a fixed prompt for different steps.@@ -141,13 +141,11 @@ In addition, we also keep track of [VBench](https://vchitect.github.io/VBench-prAll the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details.-| Model | Total Score | Quality Score | Semantic Score || -------------- | ----------- | ------------- | -------------- || Open-Sora V1.0 | 75.91% | 78.81% | 64.28% || Open-Sora V1.2 | 79.23% | 80.71% | 73.30% |-## Sequence parallelismWe use sequence parallelism to support long-sequence training and inference. Our implementation is based on Ulysses and the workflow is shown below. When sequence parallelism is enabled, we only need to apply the `all-to-all` communication to the spatial block in STDiT as only spatial computation is dependent on the sequence dimension.@@ -156,7 +154,7 @@ We use sequence parallelism to support long-sequence training and inference. OurCurrently, we have not used sequence parallelism for training as data resolution is small and we plan to do so in the next release. As for inference, we can use sequence parallelism in case your GPU goes out of memory. A simple benchmark shows that sequence parallelism can achieve speedup-| Resolution | Seconds | Number of GPUs | Enable SP | Time taken/s | Speedup per GPU |-| - | - | - | - | - | - |+| Resolution | Seconds | Number of GPUs | Enable SP | Time taken/s | Speedup per GPU |+| ---------- | ------- | -------------- | --------- | ------------ | --------------- || 720p | 16s | 1 | No | 547.97 | - |-| 720p | 16s | 2 | Yes | 244.38 | 12% |\ No newline at end of file+| 720p | 16s | 2 | Yes | 244.38 | 12% |diff --git a/docs/vae.md b/docs/vae.mdindex 1b3f05f..75e3ccd 100644--- a/docs/vae.md+++ b/docs/vae.md@@ -3,6 +3,7 @@As [Pixart-Sigma](https://arxiv.org/abs/2403.04692) finds that adapting to a new VAE is simple, we develop an additional temporal VAE.Specifically, our VAE consists of a pipeline of a [spatial VAE](https://huggingface.co/PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers) followed by a temporal VAE.For the temporal VAE, we follow the implementation of [MAGVIT-v2](https://arxiv.org/abs/2310.05737), with the following modifications:+* We remove the architecture specific to the codebook.* We do not use the discriminator, and use the VAE reconstruction loss, kl loss, and perceptual loss for training.* In the last linear layer of the encoder, we scale down to a diagonal Gaussian Distribution of 4 channels, following our previously trained STDiT that takes in 4 channels input.@@ -15,32 +16,37 @@ We train the model in different stages.We first train the temporal VAE only by freezing the spatial VAE for 380k steps on a single machine (8 GPUs).We use an additional identity loss to make features from the 3D VAE similar to the features from the 2D VAE.We train the VAE using 20% images and 80% videos with 17 frames.+```bashtorchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH```+Next, we remove the indentity loss and train the 3D VAE pipeline to reconstructe the 2D-compressed videos for 260k steps.+```bashtorchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH```+Finally, we remove the reconstruction loss for the 2D-compressed videos and train the VAE pipeline to construct the 3D videos for 540k steps.We train our VAE with a random number within 34 frames to make it more robust to different video lengths.This stage is trained on 24 GPUs.+```bashtorchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH```Note that you need to adjust the `epochs` in the config file accordingly with respect to your own csv data size.-## InferenceTo visually check the performance of the VAE, you may run the following inference.It saves the original video to your specified video directory with `_ori` postfix (i.e. `"YOUR_VIDEO_DIR"_ori`), the reconstructed video from the full pipeline with the `_rec` postfix (i.e. `"YOUR_VIDEO_DIR"_rec`), and the reconstructed video from the 2D compression and decompression with the `_spatial` postfix (i.e. `"YOUR_VIDEO_DIR"_spatial`).+```bashtorchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR```-## Evaluation+We can then calculate the scores of the VAE performances on metrics of SSIM, PSNR, LPIPS, and FLOLPIPS.* SSIM: structural similarity index measure, the higher the better@@ -52,7 +58,6 @@ We can then calculate the scores of the VAE performances on metrics of SSIM, PSNpython eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips```-## AcknowledgementWe are grateful for the following work:* [MAGVIT-v2](https://arxiv.org/abs/2310.05737): Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationdiff --git a/docs/zh_CN/README.md b/docs/zh_CN/README.mdindex f1cbf49..05f26de 100644--- a/docs/zh_CN/README.md+++ b/docs/zh_CN/README.md@@ -70,7 +70,7 @@## 🔆 新功能/更新-* 📍 **Open-Sora 1.2** 发布。模型权重可在[此处](#model-weights)查看。有关更多详细信息,请参阅我们的**[技术报告 v1.2](docs/report_03.md)** 。+* 📍 **Open-Sora 1.2** 发布。模型权重可在[此处](#model-weights)查看。有关更多详细信息,请参阅我们的**[技术报告 v1.2](docs/report_03.md)** 。* ✅ 支持整流流调度。* ✅ 训练我们的 3D-VAE 进行时间维度压缩。* 📍 **Open-Sora 1.1**发布。模型权重可在[此处](#model-weights)获得。它针对**0s~15s、144p 到 720p、各种宽高比**的视频进行训练。有关更多讨论,请参阅我们的**[技术报告 v1.1](/docs/report_02.md)** 。@@ -213,7 +213,7 @@ docker run -ti --gpus all -v {MOUNT_DIR}:/data opensora<details><summary>查看更多</summary>-| 分辨率 | 模型大小 | 数据 | 迭代次数 | 批量大小 | GPU 天数 (H800) | 网址+| 分辨率 | 模型大小 | 数据 | 迭代次数 | 批量大小 | GPU 天数 (H800) | 网址| ---------- | ---------- | ------ | ----------- | ---------- | --------------- || 16×512×512 | 700M | 20K HQ | 20k | 2×64 | 35 | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-HQ-16x512x512.pth) || 16×256×256 | 700M | 20K HQ | 24k | 8×64 | 45 | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-HQ-16x256x256.pth) |diff --git a/docs/zh_CN/READMEv1.1.md b/docs/zh_CN/READMEv1.1.mdindex 545608b..8973798 100644--- a/docs/zh_CN/READMEv1.1.md+++ b/docs/zh_CN/READMEv1.1.md@@ -21,7 +21,7 @@<h4>Open-Sora 项目目前处在早期阶段,并将持续更新。</h4>-## 📰 资讯+## 📰 资讯> 由于文档需要进行翻译,最新资讯请看[英文文档](/README.md#-news)* **[2024.04.25]** 🤗 我们在Hugging Face Spaces上发布了Open-Sora的[Gradio demo](https://huggingface.co/spaces/hpcai-tech/open-sora)。* **[2024.04.25]** 🔥 我们发布了支持**2秒至15秒、144p至720p、任意宽高比**的文本到图像、文本到视频、图像到视频、视频到视频、无限时间生成的**Open-Sora 1.1**版本。此外,还发布了一个完整的视频处理流程。 [[checkpoints]]() [[report]](/docs/report_02.md)diff --git a/eval/vae/cal_lpips.py b/eval/vae/cal_lpips.pyindex 10edc61..8f0b7a6 100644--- a/eval/vae/cal_lpips.py+++ b/eval/vae/cal_lpips.py@@ -1,17 +1,15 @@+import lpipsimport numpy as npimport torchfrom tqdm import tqdm-import math--import torch-import lpips-spatial = True # Return a spatial map of perceptual distance.+spatial = True # Return a spatial map of perceptual distance.# Linearly calibrated models (LPIPS)-loss_fn = lpips.LPIPS(net='alex', spatial=spatial) # Can also set net = 'squeeze' or 'vgg'+loss_fn = lpips.LPIPS(net="alex", spatial=spatial) # Can also set net = 'squeeze' or 'vgg'# loss_fn = lpips.LPIPS(net='alex', spatial=spatial, lpips=False) # Can also set net = 'squeeze' or 'vgg'+def trans(x):# if greyscale images add channelif x.shape[-3] == 1:@@ -22,6 +20,7 @@ def trans(x):return x+def calculate_lpips(videos1, videos2, device):# image should be RGB, IMPORTANT: normalized to [-1,1]print("calculate_lpips...")@@ -51,22 +50,21 @@ def calculate_lpips(videos1, videos2, device):img1 = video1[clip_timestamp].unsqueeze(0).to(device)img2 = video2[clip_timestamp].unsqueeze(0).to(device)-+loss_fn.to(device)# calculate lpips of a videolpips_results_of_a_video.append(loss_fn.forward(img1, img2).mean().detach().cpu().tolist())lpips_results.append(lpips_results_of_a_video)-+lpips_results = np.array(lpips_results)-+lpips = {}lpips_std = {}for clip_timestamp in range(len(video1)):- lpips[clip_timestamp] = np.mean(lpips_results[:,clip_timestamp])- lpips_std[clip_timestamp] = np.std(lpips_results[:,clip_timestamp])-+ lpips[clip_timestamp] = np.mean(lpips_results[:, clip_timestamp])+ lpips_std[clip_timestamp] = np.std(lpips_results[:, clip_timestamp])result = {"value": lpips,@@ -77,8 +75,10 @@ def calculate_lpips(videos1, videos2, device):return result+# test code / using example+def main():NUMBER_OF_VIDEOS = 8VIDEO_LENGTH = 50@@ -90,8 +90,10 @@ def main():# device = torch.device("cpu")import json+result = calculate_lpips(videos1, videos2, device)print(json.dumps(result, indent=4))+if __name__ == "__main__":- main()\ No newline at end of file+ main()diff --git a/eval/vae/cal_psnr.py b/eval/vae/cal_psnr.pyindex b325106..13c30e5 100644--- a/eval/vae/cal_psnr.py+++ b/eval/vae/cal_psnr.py@@ -1,7 +1,9 @@+import math+import numpy as npimport torchfrom tqdm import tqdm-import math+def img_psnr(img1, img2):# [0,1]@@ -14,21 +16,23 @@ def img_psnr(img1, img2):psnr = 20 * math.log10(1 / math.sqrt(mse))return psnr+def trans(x):return x+def calculate_psnr(videos1, videos2):print("calculate_psnr...")# videos [batch_size, timestamps, channel, h, w]-+assert videos1.shape == videos2.shapevideos1 = trans(videos1)videos2 = trans(videos2)psnr_results = []-+for video_num in tqdm(range(videos1.shape[0])):# get a video# video [timestamps, channel, h, w]@@ -43,19 +47,19 @@ def calculate_psnr(videos1, videos2):img1 = video1[clip_timestamp].numpy()img2 = video2[clip_timestamp].numpy()-+# calculate psnr of a videopsnr_results_of_a_video.append(img_psnr(img1, img2))psnr_results.append(psnr_results_of_a_video)-- psnr_results = np.array(psnr_results) # [batch_size, num_frames]++ psnr_results = np.array(psnr_results) # [batch_size, num_frames]psnr = {}psnr_std = {}for clip_timestamp in range(len(video1)):- psnr[clip_timestamp] = np.mean(psnr_results[:,clip_timestamp])- psnr_std[clip_timestamp] = np.std(psnr_results[:,clip_timestamp])+ psnr[clip_timestamp] = np.mean(psnr_results[:, clip_timestamp])+ psnr_std[clip_timestamp] = np.std(psnr_results[:, clip_timestamp])result = {"value": psnr,@@ -66,8 +70,10 @@ def calculate_psnr(videos1, videos2):return result+# test code / using example+def main():NUMBER_OF_VIDEOS = 8VIDEO_LENGTH = 50@@ -77,8 +83,10 @@ def main():videos2 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)import json+result = calculate_psnr(videos1, videos2)print(json.dumps(result, indent=4))+if __name__ == "__main__":- main()\ No newline at end of file+ main()diff --git a/eval/vae/cal_ssim.py b/eval/vae/cal_ssim.pyindex b2de811..65338c8 100644--- a/eval/vae/cal_ssim.py+++ b/eval/vae/cal_ssim.py@@ -1,33 +1,33 @@+import cv2import numpy as npimport torchfrom tqdm import tqdm-import cv2-++def ssim(img1, img2):- C1 = 0.01 ** 2- C2 = 0.03 ** 2+ C1 = 0.01**2+ C2 = 0.03**2img1 = img1.astype(np.float64)img2 = img2.astype(np.float64)kernel = cv2.getGaussianKernel(11, 1.5)window = np.outer(kernel, kernel.transpose())mu1 = cv2.filter2D(img1, -1, window)[5:-5, 5:-5] # validmu2 = cv2.filter2D(img2, -1, window)[5:-5, 5:-5]- mu1_sq = mu1 ** 2- mu2_sq = mu2 ** 2+ mu1_sq = mu1**2+ mu2_sq = mu2**2mu1_mu2 = mu1 * mu2- sigma1_sq = cv2.filter2D(img1 ** 2, -1, window)[5:-5, 5:-5] - mu1_sq- sigma2_sq = cv2.filter2D(img2 ** 2, -1, window)[5:-5, 5:-5] - mu2_sq+ sigma1_sq = cv2.filter2D(img1**2, -1, window)[5:-5, 5:-5] - mu1_sq+ sigma2_sq = cv2.filter2D(img2**2, -1, window)[5:-5, 5:-5] - mu2_sqsigma12 = cv2.filter2D(img1 * img2, -1, window)[5:-5, 5:-5] - mu1_mu2- ssim_map = ((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) *- (sigma1_sq + sigma2_sq + C2))+ ssim_map = ((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) * (sigma1_sq + sigma2_sq + C2))return ssim_map.mean()--++def calculate_ssim_function(img1, img2):# [0,1]- # ssim is the only metric extremely sensitive to gray being compared to b/w+ # ssim is the only metric extremely sensitive to gray being compared to b/wif not img1.shape == img2.shape:- raise ValueError('Input images must have the same dimensions.')+ raise ValueError("Input images must have the same dimensions.")if img1.ndim == 2:return ssim(img1, img2)elif img1.ndim == 3:@@ -35,27 +35,29 @@ def calculate_ssim_function(img1, img2):ssims = []for i in range(3):ssims.append(ssim(img1[i], img2[i]))- return np.array(ssims).mean()+ return np.array(ssims).mean()elif img1.shape[0] == 1:return ssim(np.squeeze(img1), np.squeeze(img2))else:- raise ValueError('Wrong input image dimensions.')+ raise ValueError("Wrong input image dimensions.")+def trans(x):return x+def calculate_ssim(videos1, videos2):print("calculate_ssim...")# videos [batch_size, timestamps, channel, h, w]-+assert videos1.shape == videos2.shapevideos1 = trans(videos1)videos2 = trans(videos2)ssim_results = []-+for video_num in tqdm(range(videos1.shape[0])):# get a video# video [timestamps, channel, h, w]@@ -70,7 +72,7 @@ def calculate_ssim(videos1, videos2):img1 = video1[clip_timestamp].numpy()img2 = video2[clip_timestamp].numpy()-+# calculate ssim of a videossim_results_of_a_video.append(calculate_ssim_function(img1, img2))@@ -82,8 +84,8 @@ def calculate_ssim(videos1, videos2):ssim_std = {}for clip_timestamp in range(len(video1)):- ssim[clip_timestamp] = np.mean(ssim_results[:,clip_timestamp])- ssim_std[clip_timestamp] = np.std(ssim_results[:,clip_timestamp])+ ssim[clip_timestamp] = np.mean(ssim_results[:, clip_timestamp])+ ssim_std[clip_timestamp] = np.std(ssim_results[:, clip_timestamp])result = {"value": ssim,@@ -94,8 +96,10 @@ def calculate_ssim(videos1, videos2):return result+# test code / using example+def main():NUMBER_OF_VIDEOS = 8VIDEO_LENGTH = 50@@ -103,11 +107,13 @@ def main():SIZE = 64videos1 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)videos2 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)- device = torch.device("cuda")+ torch.device("cuda")import json+result = calculate_ssim(videos1, videos2)print(json.dumps(result, indent=4))+if __name__ == "__main__":- main()\ No newline at end of file+ main()diff --git a/eval/vae/eval_common_metric.py b/eval/vae/eval_common_metric.pyindex 4570738..e2fdc00 100644--- a/eval/vae/eval_common_metric.py+++ b/eval/vae/eval_common_metric.py@@ -1,13 +1,13 @@"""Calculates the CLIP ScoresThe CLIP model is a contrasitively learned language-image model. There is-an image encoder and a text encoder. It is believed that the CLIP model could-measure the similarity of cross modalities. Please find more information from+an image encoder and a text encoder. It is believed that the CLIP model could+measure the similarity of cross modalities. Please find more information fromhttps://github.com/openai/CLIP.The CLIP Score measures the Cosine Similarity between two embedded features.-This repository utilizes the pretrained CLIP Model to calculate-the mean average of cosine similarities.+This repository utilizes the pretrained CLIP Model to calculate+the mean average of cosine similarities.See --help to see further details.@@ -30,21 +30,21 @@ limitations under the License.import osimport os.path as osp+import sysfrom argparse import ArgumentDefaultsHelpFormatter, ArgumentParser+import numpy as npimport torch-from torch.utils.data import Dataset, DataLoader, Subsetfrom decord import VideoReader, cpu-import randomfrom pytorchvideo.transforms import ShortSideScale-from torchvision.io import read_video-from torchvision.transforms import Lambda, Compose+from torch.utils.data import DataLoader, Dataset, Subset+from torchvision.transforms import Compose, Lambdafrom torchvision.transforms._transforms_video import CenterCropVideo-import sys+sys.path.append(".")+from cal_flolpips import calculate_flolpipsfrom cal_lpips import calculate_lpipsfrom cal_psnr import calculate_psnr-from cal_flolpips import calculate_flolpipsfrom cal_ssim import calculate_ssimtry:@@ -54,15 +54,17 @@ except ImportError:def tqdm(x):return x+class VideoDataset(Dataset):- def __init__(self,- real_video_dir,- generated_video_dir,- num_frames,- sample_rate = 1,- crop_size=None,- resolution=128,- ) -> None:+ def __init__(+ self,+ real_video_dir,+ generated_video_dir,+ num_frames,+ sample_rate=1,+ crop_size=None,+ resolution=128,+ ) -> None:super().__init__()self.real_video_files = self._combine_without_prefix(real_video_dir)self.generated_video_files = self._combine_without_prefix(generated_video_dir)@@ -71,7 +73,6 @@ class VideoDataset(Dataset):self.crop_size = crop_sizeself.short_size = resolution-def __len__(self):return len(self.real_video_files)@@ -81,10 +82,9 @@ class VideoDataset(Dataset):real_video_file = self.real_video_files[index]generated_video_file = self.generated_video_files[index]print(real_video_file, generated_video_file)- real_video_tensor = self._load_video(real_video_file)- generated_video_tensor = self._load_video(generated_video_file)- return {'real': real_video_tensor, 'generated':generated_video_tensor }-+ real_video_tensor = self._load_video(real_video_file)+ generated_video_tensor = self._load_video(generated_video_file)+ return {"real": real_video_tensor, "generated": generated_video_tensor}def _load_video(self, video_path):num_frames = self.num_frames@@ -101,18 +101,19 @@ class VideoDataset(Dataset):s = 0e = total_framesnum_frames = int(total_frames / sample_frames_len * num_frames)- print(f'sample_frames_len {sample_frames_len}, only can sample {num_frames * sample_rate}', video_path,- total_frames)-+ print(+ f"sample_frames_len {sample_frames_len}, only can sample {num_frames * sample_rate}",+ video_path,+ total_frames,+ )frame_id_list = np.linspace(s, e - 1, num_frames, dtype=int)video_data = decord_vr.get_batch(frame_id_list).asnumpy()video_data = torch.from_numpy(video_data)- video_data = video_data.permute(0, 3, 1, 2) # (T, H, W, C) -> (C, T, H, W)- return _preprocess(video_data, short_size=self.short_size, crop_size = self.crop_size)+ video_data = video_data.permute(0, 3, 1, 2) # (T, H, W, C) -> (C, T, H, W)+ return _preprocess(video_data, short_size=self.short_size, crop_size=self.crop_size)-- def _combine_without_prefix(self, folder_path, prefix='.'):+ def _combine_without_prefix(self, folder_path, prefix="."):folder = []os.makedirs(folder_path, exist_ok=True)for name in os.listdir(folder_path):@@ -123,6 +124,7 @@ class VideoDataset(Dataset):folder.sort()return folder+def _preprocess(video_data, short_size=128, crop_size=None):transform = Compose([@@ -137,26 +139,25 @@ def _preprocess(video_data, short_size=128, crop_size=None):def calculate_common_metric(args, dataloader, device):-metric_dict = {}if type(args.metric) is str:args.metric = [m.strip() for m in args.metric.split(",")]print(args.metric)for metric in args.metric:score_list = []- for batch_data in tqdm(dataloader): # {'real': real_video_tensor, 'generated':generated_video_tensor }- real_videos = batch_data['real']- generated_videos = batch_data['generated']+ for batch_data in tqdm(dataloader): # {'real': real_video_tensor, 'generated':generated_video_tensor }+ real_videos = batch_data["real"]+ generated_videos = batch_data["generated"]assert real_videos.shape[2] == generated_videos.shape[2]- if metric == 'ssim':- tmp_list = list(calculate_ssim(real_videos, generated_videos)['value'].values())- elif metric == 'psnr':- tmp_list = list(calculate_psnr(real_videos, generated_videos)['value'].values())- elif metric == 'flolpips':+ if metric == "ssim":+ tmp_list = list(calculate_ssim(real_videos, generated_videos)["value"].values())+ elif metric == "psnr":+ tmp_list = list(calculate_psnr(real_videos, generated_videos)["value"].values())+ elif metric == "flolpips":result = calculate_flolpips(real_videos, generated_videos, args.device)- tmp_list = list(result['value'].values())+ tmp_list = list(result["value"].values())elif metric == "lpips":- tmp_list = list(calculate_lpips(real_videos, generated_videos, args.device)['value'].values())+ tmp_list = list(calculate_lpips(real_videos, generated_videos, args.device)["value"].values())else:print(f"metric {metric} is not in acceped list, not calculated")continue@@ -164,35 +165,34 @@ def calculate_common_metric(args, dataloader, device):metric_dict[metric] = np.mean(score_list)return metric_dict-++def main():parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)- parser.add_argument('--batch_size', type=int, default=2,- help='Batch size to use')- parser.add_argument('--real_video_dir', type=str,- help=('the path of real videos`'))- parser.add_argument('--generated_video_dir', type=str,- help=('the path of generated videos`'))- parser.add_argument('--device', type=str, default=None,- help='Device to use. Like cuda, cuda:0 or cpu')- parser.add_argument('--num_workers', type=int, default=8,- help=('Number of processes to use for data loading. '- 'Defaults to `min(8, num_cpus)`'))- parser.add_argument('--sample_fps', type=int, default=30)- parser.add_argument('--resolution', type=int, default=336)- parser.add_argument('--crop_size', type=int, default=None)- parser.add_argument('--num_frames', type=int, default=100)- parser.add_argument('--sample_rate', type=int, default=1)- parser.add_argument('--subset_size', type=int, default=None)+ parser.add_argument("--batch_size", type=int, default=2, help="Batch size to use")+ parser.add_argument("--real_video_dir", type=str, help=("the path of real videos`"))+ parser.add_argument("--generated_video_dir", type=str, help=("the path of generated videos`"))+ parser.add_argument("--device", type=str, default=None, help="Device to use. Like cuda, cuda:0 or cpu")+ parser.add_argument(+ "--num_workers",+ type=int,+ default=8,+ help=("Number of processes to use for data loading. " "Defaults to `min(8, num_cpus)`"),+ )+ parser.add_argument("--sample_fps", type=int, default=30)+ parser.add_argument("--resolution", type=int, default=336)+ parser.add_argument("--crop_size", type=int, default=None)+ parser.add_argument("--num_frames", type=int, default=100)+ parser.add_argument("--sample_rate", type=int, default=1)+ parser.add_argument("--subset_size", type=int, default=None)# parser.add_argument("--metric", type=str, default="fvd",choices=['fvd','psnr','ssim','lpips', 'flolpips'])- parser.add_argument("--metric", nargs='+', default=[])- parser.add_argument("--fvd_method", type=str, default='styleganv',choices=['styleganv','videogpt'])-+ parser.add_argument("--metric", nargs="+", default=[])+ parser.add_argument("--fvd_method", type=str, default="styleganv", choices=["styleganv", "videogpt"])args = parser.parse_args()if args.device is None:- device = torch.device('cuda' if (torch.cuda.is_available()) else 'cpu')+ device = torch.device("cuda" if (torch.cuda.is_available()) else "cpu")else:device = torch.device(args.device)@@ -209,24 +209,24 @@ def main():else:num_workers = args.num_workers-- dataset = VideoDataset(args.real_video_dir,- args.generated_video_dir,- num_frames = args.num_frames,- sample_rate = args.sample_rate,- crop_size=args.crop_size,- resolution=args.resolution)+ dataset = VideoDataset(+ args.real_video_dir,+ args.generated_video_dir,+ num_frames=args.num_frames,+ sample_rate=args.sample_rate,+ crop_size=args.crop_size,+ resolution=args.resolution,+ )if args.subset_size:indices = range(args.subset_size)dataset = Subset(dataset, indices=indices)-- dataloader = DataLoader(dataset, args.batch_size,- num_workers=num_workers, pin_memory=True)-- metric_score = calculate_common_metric(args, dataloader,device)- print('metric: ', args.metric, " ",metric_score)+ dataloader = DataLoader(dataset, args.batch_size, num_workers=num_workers, pin_memory=True)++ metric_score = calculate_common_metric(args, dataloader, device)+ print("metric: ", args.metric, " ", metric_score)+-if __name__ == '__main__':+if __name__ == "__main__":main()diff --git a/eval/vae/flolpips/correlation/correlation.py b/eval/vae/flolpips/correlation/correlation.pyindex 7c91055..6f6d6bd 100644--- a/eval/vae/flolpips/correlation/correlation.py+++ b/eval/vae/flolpips/correlation/correlation.py@@ -1,397 +1,461 @@-#!/usr/bin/env python--import torch--import cupy-import re--kernel_Correlation_rearrange = '''- extern "C" __global__ void kernel_Correlation_rearrange(- const int n,- const float* input,- float* output- ) {- int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x;-- if (intIndex >= n) {- return;- }-- int intSample = blockIdx.z;- int intChannel = blockIdx.y;-- float fltValue = input[(((intSample * SIZE_1(input)) + intChannel) * SIZE_2(input) * SIZE_3(input)) + intIndex];-- __syncthreads();-- int intPaddedY = (intIndex / SIZE_3(input)) + 4;- int intPaddedX = (intIndex % SIZE_3(input)) + 4;- int intRearrange = ((SIZE_3(input) + 8) * intPaddedY) + intPaddedX;-- output[(((intSample * SIZE_1(output) * SIZE_2(output)) + intRearrange) * SIZE_1(input)) + intChannel] = fltValue;- }-'''--kernel_Correlation_updateOutput = '''- extern "C" __global__ void kernel_Correlation_updateOutput(- const int n,- const float* rbot0,- const float* rbot1,- float* top- ) {- extern __shared__ char patch_data_char[];-- float *patch_data = (float *)patch_data_char;-- // First (upper left) position of kernel upper-left corner in current center position of neighborhood in image 1- int x1 = blockIdx.x + 4;- int y1 = blockIdx.y + 4;- int item = blockIdx.z;- int ch_off = threadIdx.x;-- // Load 3D patch into shared shared memory- for (int j = 0; j < 1; j++) { // HEIGHT- for (int i = 0; i < 1; i++) { // WIDTH- int ji_off = (j + i) * SIZE_3(rbot0);- for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS- int idx1 = ((item * SIZE_1(rbot0) + y1+j) * SIZE_2(rbot0) + x1+i) * SIZE_3(rbot0) + ch;- int idxPatchData = ji_off + ch;- patch_data[idxPatchData] = rbot0[idx1];- }- }- }-- __syncthreads();-- __shared__ float sum[32];-- // Compute correlation- for (int top_channel = 0; top_channel < SIZE_1(top); top_channel++) {- sum[ch_off] = 0;-- int s2o = top_channel % 9 - 4;- int s2p = top_channel / 9 - 4;-- for (int j = 0; j < 1; j++) { // HEIGHT- for (int i = 0; i < 1; i++) { // WIDTH- int ji_off = (j + i) * SIZE_3(rbot0);- for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS- int x2 = x1 + s2o;- int y2 = y1 + s2p;-- int idxPatchData = ji_off + ch;- int idx2 = ((item * SIZE_1(rbot0) + y2+j) * SIZE_2(rbot0) + x2+i) * SIZE_3(rbot0) + ch;-- sum[ch_off] += patch_data[idxPatchData] * rbot1[idx2];- }- }- }-- __syncthreads();-- if (ch_off == 0) {- float total_sum = 0;- for (int idx = 0; idx < 32; idx++) {- total_sum += sum[idx];- }- const int sumelems = SIZE_3(rbot0);- const int index = ((top_channel*SIZE_2(top) + blockIdx.y)*SIZE_3(top))+blockIdx.x;- top[index + item*SIZE_1(top)*SIZE_2(top)*SIZE_3(top)] = total_sum / (float)sumelems;- }- }- }-'''--kernel_Correlation_updateGradFirst = '''- #define ROUND_OFF 50000-- extern "C" __global__ void kernel_Correlation_updateGradFirst(- const int n,- const int intSample,- const float* rbot0,- const float* rbot1,- const float* gradOutput,- float* gradFirst,- float* gradSecond- ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {- int n = intIndex % SIZE_1(gradFirst); // channels- int l = (intIndex / SIZE_1(gradFirst)) % SIZE_3(gradFirst) + 4; // w-pos- int m = (intIndex / SIZE_1(gradFirst) / SIZE_3(gradFirst)) % SIZE_2(gradFirst) + 4; // h-pos-- // round_off is a trick to enable integer division with ceil, even for negative numbers- // We use a large offset, for the inner part not to become negative.- const int round_off = ROUND_OFF;- const int round_off_s1 = round_off;-- // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:- int xmin = (l - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)- int ymin = (m - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)-- // Same here:- int xmax = (l - 4 + round_off_s1) - round_off; // floor (l - 4)- int ymax = (m - 4 + round_off_s1) - round_off; // floor (m - 4)-- float sum = 0;- if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {- xmin = max(0,xmin);- xmax = min(SIZE_3(gradOutput)-1,xmax);-- ymin = max(0,ymin);- ymax = min(SIZE_2(gradOutput)-1,ymax);-- for (int p = -4; p <= 4; p++) {- for (int o = -4; o <= 4; o++) {- // Get rbot1 data:- int s2o = o;- int s2p = p;- int idxbot1 = ((intSample * SIZE_1(rbot0) + (m+s2p)) * SIZE_2(rbot0) + (l+s2o)) * SIZE_3(rbot0) + n;- float bot1tmp = rbot1[idxbot1]; // rbot1[l+s2o,m+s2p,n]-- // Index offset for gradOutput in following loops:- int op = (p+4) * 9 + (o+4); // index[o,p]- int idxopoffset = (intSample * SIZE_1(gradOutput) + op);-- for (int y = ymin; y <= ymax; y++) {- for (int x = xmin; x <= xmax; x++) {- int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]- sum += gradOutput[idxgradOutput] * bot1tmp;- }- }- }- }- }- const int sumelems = SIZE_1(gradFirst);- const int bot0index = ((n * SIZE_2(gradFirst)) + (m-4)) * SIZE_3(gradFirst) + (l-4);- gradFirst[bot0index + intSample*SIZE_1(gradFirst)*SIZE_2(gradFirst)*SIZE_3(gradFirst)] = sum / (float)sumelems;- } }-'''--kernel_Correlation_updateGradSecond = '''- #define ROUND_OFF 50000-- extern "C" __global__ void kernel_Correlation_updateGradSecond(- const int n,- const int intSample,- const float* rbot0,- const float* rbot1,- const float* gradOutput,- float* gradFirst,- float* gradSecond- ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {- int n = intIndex % SIZE_1(gradSecond); // channels- int l = (intIndex / SIZE_1(gradSecond)) % SIZE_3(gradSecond) + 4; // w-pos- int m = (intIndex / SIZE_1(gradSecond) / SIZE_3(gradSecond)) % SIZE_2(gradSecond) + 4; // h-pos-- // round_off is a trick to enable integer division with ceil, even for negative numbers- // We use a large offset, for the inner part not to become negative.- const int round_off = ROUND_OFF;- const int round_off_s1 = round_off;-- float sum = 0;- for (int p = -4; p <= 4; p++) {- for (int o = -4; o <= 4; o++) {- int s2o = o;- int s2p = p;-- //Get X,Y ranges and clamp- // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:- int xmin = (l - 4 - s2o + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)- int ymin = (m - 4 - s2p + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)-- // Same here:- int xmax = (l - 4 - s2o + round_off_s1) - round_off; // floor (l - 4 - s2o)- int ymax = (m - 4 - s2p + round_off_s1) - round_off; // floor (m - 4 - s2p)-- if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {- xmin = max(0,xmin);- xmax = min(SIZE_3(gradOutput)-1,xmax);-- ymin = max(0,ymin);- ymax = min(SIZE_2(gradOutput)-1,ymax);-- // Get rbot0 data:- int idxbot0 = ((intSample * SIZE_1(rbot0) + (m-s2p)) * SIZE_2(rbot0) + (l-s2o)) * SIZE_3(rbot0) + n;- float bot0tmp = rbot0[idxbot0]; // rbot1[l+s2o,m+s2p,n]-- // Index offset for gradOutput in following loops:- int op = (p+4) * 9 + (o+4); // index[o,p]- int idxopoffset = (intSample * SIZE_1(gradOutput) + op);-- for (int y = ymin; y <= ymax; y++) {- for (int x = xmin; x <= xmax; x++) {- int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]- sum += gradOutput[idxgradOutput] * bot0tmp;- }- }- }- }- }- const int sumelems = SIZE_1(gradSecond);- const int bot1index = ((n * SIZE_2(gradSecond)) + (m-4)) * SIZE_3(gradSecond) + (l-4);- gradSecond[bot1index + intSample*SIZE_1(gradSecond)*SIZE_2(gradSecond)*SIZE_3(gradSecond)] = sum / (float)sumelems;- } }-'''--def cupy_kernel(strFunction, objVariables):- strKernel = globals()[strFunction]-- while True:- objMatch = re.search('(SIZE_)([0-4])(\()([^\)]*)(\))', strKernel)-- if objMatch is None:- break- # end-- intArg = int(objMatch.group(2))-- strTensor = objMatch.group(4)- intSizes = objVariables[strTensor].size()-- strKernel = strKernel.replace(objMatch.group(), str(intSizes[intArg]))- # end-- while True:- objMatch = re.search('(VALUE_)([0-4])(\()([^\)]+)(\))', strKernel)-- if objMatch is None:- break- # end-- intArgs = int(objMatch.group(2))- strArgs = objMatch.group(4).split(',')-- strTensor = strArgs[0]- intStrides = objVariables[strTensor].stride()- strIndex = [ '((' + strArgs[intArg + 1].replace('{', '(').replace('}', ')').strip() + ')*' + str(intStrides[intArg]) + ')' for intArg in range(intArgs) ]-- strKernel = strKernel.replace(objMatch.group(0), strTensor + '[' + str.join('+', strIndex) + ']')- # end-- return strKernel-# end-[email protected](for_each_device=True)-def cupy_launch(strFunction, strKernel):- return cupy.RawKernel(strKernel, strFunction)-# end--class _FunctionCorrelation(torch.autograd.Function):- @staticmethod- def forward(self, first, second):- rbot0 = first.new_zeros([ first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1] ])- rbot1 = first.new_zeros([ first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1] ])-- self.save_for_backward(first, second, rbot0, rbot1)-- first = first.contiguous(); assert(first.is_cuda == True)- second = second.contiguous(); assert(second.is_cuda == True)-- output = first.new_zeros([ first.shape[0], 81, first.shape[2], first.shape[3] ])-- if first.is_cuda == True:- n = first.shape[2] * first.shape[3]- cupy_launch('kernel_Correlation_rearrange', cupy_kernel('kernel_Correlation_rearrange', {- 'input': first,- 'output': rbot0- }))(- grid=tuple([ int((n + 16 - 1) / 16), first.shape[1], first.shape[0] ]),- block=tuple([ 16, 1, 1 ]),- args=[ n, first.data_ptr(), rbot0.data_ptr() ]- )-- n = second.shape[2] * second.shape[3]- cupy_launch('kernel_Correlation_rearrange', cupy_kernel('kernel_Correlation_rearrange', {- 'input': second,- 'output': rbot1- }))(- grid=tuple([ int((n + 16 - 1) / 16), second.shape[1], second.shape[0] ]),- block=tuple([ 16, 1, 1 ]),- args=[ n, second.data_ptr(), rbot1.data_ptr() ]- )-- n = output.shape[1] * output.shape[2] * output.shape[3]- cupy_launch('kernel_Correlation_updateOutput', cupy_kernel('kernel_Correlation_updateOutput', {- 'rbot0': rbot0,- 'rbot1': rbot1,- 'top': output- }))(- grid=tuple([ output.shape[3], output.shape[2], output.shape[0] ]),- block=tuple([ 32, 1, 1 ]),- shared_mem=first.shape[1] * 4,- args=[ n, rbot0.data_ptr(), rbot1.data_ptr(), output.data_ptr() ]- )-- elif first.is_cuda == False:- raise NotImplementedError()-- # end-- return output- # end-- @staticmethod- def backward(self, gradOutput):- first, second, rbot0, rbot1 = self.saved_tensors-- gradOutput = gradOutput.contiguous(); assert(gradOutput.is_cuda == True)-- gradFirst = first.new_zeros([ first.shape[0], first.shape[1], first.shape[2], first.shape[3] ]) if self.needs_input_grad[0] == True else None- gradSecond = first.new_zeros([ first.shape[0], first.shape[1], first.shape[2], first.shape[3] ]) if self.needs_input_grad[1] == True else None-- if first.is_cuda == True:- if gradFirst is not None:- for intSample in range(first.shape[0]):- n = first.shape[1] * first.shape[2] * first.shape[3]- cupy_launch('kernel_Correlation_updateGradFirst', cupy_kernel('kernel_Correlation_updateGradFirst', {- 'rbot0': rbot0,- 'rbot1': rbot1,- 'gradOutput': gradOutput,- 'gradFirst': gradFirst,- 'gradSecond': None- }))(- grid=tuple([ int((n + 512 - 1) / 512), 1, 1 ]),- block=tuple([ 512, 1, 1 ]),- args=[ n, intSample, rbot0.data_ptr(), rbot1.data_ptr(), gradOutput.data_ptr(), gradFirst.data_ptr(), None ]- )- # end- # end-- if gradSecond is not None:- for intSample in range(first.shape[0]):- n = first.shape[1] * first.shape[2] * first.shape[3]- cupy_launch('kernel_Correlation_updateGradSecond', cupy_kernel('kernel_Correlation_updateGradSecond', {- 'rbot0': rbot0,- 'rbot1': rbot1,- 'gradOutput': gradOutput,- 'gradFirst': None,- 'gradSecond': gradSecond- }))(- grid=tuple([ int((n + 512 - 1) / 512), 1, 1 ]),- block=tuple([ 512, 1, 1 ]),- args=[ n, intSample, rbot0.data_ptr(), rbot1.data_ptr(), gradOutput.data_ptr(), None, gradSecond.data_ptr() ]- )- # end- # end-- elif first.is_cuda == False:- raise NotImplementedError()-- # end-- return gradFirst, gradSecond- # end-# end--def FunctionCorrelation(tenFirst, tenSecond):- return _FunctionCorrelation.apply(tenFirst, tenSecond)-# end--class ModuleCorrelation(torch.nn.Module):- def __init__(self):- super(ModuleCorrelation, self).__init__()- # end-- def forward(self, tenFirst, tenSecond):- return _FunctionCorrelation.apply(tenFirst, tenSecond)- # end-# end\ No newline at end of file+#!/usr/bin/env python++import re++import cupy+import torch++kernel_Correlation_rearrange = """+ extern "C" __global__ void kernel_Correlation_rearrange(+ const int n,+ const float* input,+ float* output+ ) {+ int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x;++ if (intIndex >= n) {+ return;+ }++ int intSample = blockIdx.z;+ int intChannel = blockIdx.y;++ float fltValue = input[(((intSample * SIZE_1(input)) + intChannel) * SIZE_2(input) * SIZE_3(input)) + intIndex];++ __syncthreads();++ int intPaddedY = (intIndex / SIZE_3(input)) + 4;+ int intPaddedX = (intIndex % SIZE_3(input)) + 4;+ int intRearrange = ((SIZE_3(input) + 8) * intPaddedY) + intPaddedX;++ output[(((intSample * SIZE_1(output) * SIZE_2(output)) + intRearrange) * SIZE_1(input)) + intChannel] = fltValue;+ }+"""++kernel_Correlation_updateOutput = """+ extern "C" __global__ void kernel_Correlation_updateOutput(+ const int n,+ const float* rbot0,+ const float* rbot1,+ float* top+ ) {+ extern __shared__ char patch_data_char[];++ float *patch_data = (float *)patch_data_char;++ // First (upper left) position of kernel upper-left corner in current center position of neighborhood in image 1+ int x1 = blockIdx.x + 4;+ int y1 = blockIdx.y + 4;+ int item = blockIdx.z;+ int ch_off = threadIdx.x;++ // Load 3D patch into shared shared memory+ for (int j = 0; j < 1; j++) { // HEIGHT+ for (int i = 0; i < 1; i++) { // WIDTH+ int ji_off = (j + i) * SIZE_3(rbot0);+ for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS+ int idx1 = ((item * SIZE_1(rbot0) + y1+j) * SIZE_2(rbot0) + x1+i) * SIZE_3(rbot0) + ch;+ int idxPatchData = ji_off + ch;+ patch_data[idxPatchData] = rbot0[idx1];+ }+ }+ }++ __syncthreads();++ __shared__ float sum[32];++ // Compute correlation+ for (int top_channel = 0; top_channel < SIZE_1(top); top_channel++) {+ sum[ch_off] = 0;++ int s2o = top_channel % 9 - 4;+ int s2p = top_channel / 9 - 4;++ for (int j = 0; j < 1; j++) { // HEIGHT+ for (int i = 0; i < 1; i++) { // WIDTH+ int ji_off = (j + i) * SIZE_3(rbot0);+ for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS+ int x2 = x1 + s2o;+ int y2 = y1 + s2p;++ int idxPatchData = ji_off + ch;+ int idx2 = ((item * SIZE_1(rbot0) + y2+j) * SIZE_2(rbot0) + x2+i) * SIZE_3(rbot0) + ch;++ sum[ch_off] += patch_data[idxPatchData] * rbot1[idx2];+ }+ }+ }++ __syncthreads();++ if (ch_off == 0) {+ float total_sum = 0;+ for (int idx = 0; idx < 32; idx++) {+ total_sum += sum[idx];+ }+ const int sumelems = SIZE_3(rbot0);+ const int index = ((top_channel*SIZE_2(top) + blockIdx.y)*SIZE_3(top))+blockIdx.x;+ top[index + item*SIZE_1(top)*SIZE_2(top)*SIZE_3(top)] = total_sum / (float)sumelems;+ }+ }+ }+"""++kernel_Correlation_updateGradFirst = """+ #define ROUND_OFF 50000++ extern "C" __global__ void kernel_Correlation_updateGradFirst(+ const int n,+ const int intSample,+ const float* rbot0,+ const float* rbot1,+ const float* gradOutput,+ float* gradFirst,+ float* gradSecond+ ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {+ int n = intIndex % SIZE_1(gradFirst); // channels+ int l = (intIndex / SIZE_1(gradFirst)) % SIZE_3(gradFirst) + 4; // w-pos+ int m = (intIndex / SIZE_1(gradFirst) / SIZE_3(gradFirst)) % SIZE_2(gradFirst) + 4; // h-pos++ // round_off is a trick to enable integer division with ceil, even for negative numbers+ // We use a large offset, for the inner part not to become negative.+ const int round_off = ROUND_OFF;+ const int round_off_s1 = round_off;++ // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:+ int xmin = (l - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)+ int ymin = (m - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)++ // Same here:+ int xmax = (l - 4 + round_off_s1) - round_off; // floor (l - 4)+ int ymax = (m - 4 + round_off_s1) - round_off; // floor (m - 4)++ float sum = 0;+ if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {+ xmin = max(0,xmin);+ xmax = min(SIZE_3(gradOutput)-1,xmax);++ ymin = max(0,ymin);+ ymax = min(SIZE_2(gradOutput)-1,ymax);++ for (int p = -4; p <= 4; p++) {+ for (int o = -4; o <= 4; o++) {+ // Get rbot1 data:+ int s2o = o;+ int s2p = p;+ int idxbot1 = ((intSample * SIZE_1(rbot0) + (m+s2p)) * SIZE_2(rbot0) + (l+s2o)) * SIZE_3(rbot0) + n;+ float bot1tmp = rbot1[idxbot1]; // rbot1[l+s2o,m+s2p,n]++ // Index offset for gradOutput in following loops:+ int op = (p+4) * 9 + (o+4); // index[o,p]+ int idxopoffset = (intSample * SIZE_1(gradOutput) + op);++ for (int y = ymin; y <= ymax; y++) {+ for (int x = xmin; x <= xmax; x++) {+ int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]+ sum += gradOutput[idxgradOutput] * bot1tmp;+ }+ }+ }+ }+ }+ const int sumelems = SIZE_1(gradFirst);+ const int bot0index = ((n * SIZE_2(gradFirst)) + (m-4)) * SIZE_3(gradFirst) + (l-4);+ gradFirst[bot0index + intSample*SIZE_1(gradFirst)*SIZE_2(gradFirst)*SIZE_3(gradFirst)] = sum / (float)sumelems;+ } }+"""++kernel_Correlation_updateGradSecond = """+ #define ROUND_OFF 50000++ extern "C" __global__ void kernel_Correlation_updateGradSecond(+ const int n,+ const int intSample,+ const float* rbot0,+ const float* rbot1,+ const float* gradOutput,+ float* gradFirst,+ float* gradSecond+ ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {+ int n = intIndex % SIZE_1(gradSecond); // channels+ int l = (intIndex / SIZE_1(gradSecond)) % SIZE_3(gradSecond) + 4; // w-pos+ int m = (intIndex / SIZE_1(gradSecond) / SIZE_3(gradSecond)) % SIZE_2(gradSecond) + 4; // h-pos++ // round_off is a trick to enable integer division with ceil, even for negative numbers+ // We use a large offset, for the inner part not to become negative.+ const int round_off = ROUND_OFF;+ const int round_off_s1 = round_off;++ float sum = 0;+ for (int p = -4; p <= 4; p++) {+ for (int o = -4; o <= 4; o++) {+ int s2o = o;+ int s2p = p;++ //Get X,Y ranges and clamp+ // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:+ int xmin = (l - 4 - s2o + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)+ int ymin = (m - 4 - s2p + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)++ // Same here:+ int xmax = (l - 4 - s2o + round_off_s1) - round_off; // floor (l - 4 - s2o)+ int ymax = (m - 4 - s2p + round_off_s1) - round_off; // floor (m - 4 - s2p)++ if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {+ xmin = max(0,xmin);+ xmax = min(SIZE_3(gradOutput)-1,xmax);++ ymin = max(0,ymin);+ ymax = min(SIZE_2(gradOutput)-1,ymax);++ // Get rbot0 data:+ int idxbot0 = ((intSample * SIZE_1(rbot0) + (m-s2p)) * SIZE_2(rbot0) + (l-s2o)) * SIZE_3(rbot0) + n;+ float bot0tmp = rbot0[idxbot0]; // rbot1[l+s2o,m+s2p,n]++ // Index offset for gradOutput in following loops:+ int op = (p+4) * 9 + (o+4); // index[o,p]+ int idxopoffset = (intSample * SIZE_1(gradOutput) + op);++ for (int y = ymin; y <= ymax; y++) {+ for (int x = xmin; x <= xmax; x++) {+ int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]+ sum += gradOutput[idxgradOutput] * bot0tmp;+ }+ }+ }+ }+ }+ const int sumelems = SIZE_1(gradSecond);+ const int bot1index = ((n * SIZE_2(gradSecond)) + (m-4)) * SIZE_3(gradSecond) + (l-4);+ gradSecond[bot1index + intSample*SIZE_1(gradSecond)*SIZE_2(gradSecond)*SIZE_3(gradSecond)] = sum / (float)sumelems;+ } }+"""+++def cupy_kernel(strFunction, objVariables):+ strKernel = globals()[strFunction]++ while True:+ objMatch = re.search("(SIZE_)([0-4])(\()([^\)]*)(\))", strKernel)++ if objMatch is None:+ break+ # end++ intArg = int(objMatch.group(2))++ strTensor = objMatch.group(4)+ intSizes = objVariables[strTensor].size()++ strKernel = strKernel.replace(objMatch.group(), str(intSizes[intArg]))+ # end++ while True:+ objMatch = re.search("(VALUE_)([0-4])(\()([^\)]+)(\))", strKernel)++ if objMatch is None:+ break+ # end++ intArgs = int(objMatch.group(2))+ strArgs = objMatch.group(4).split(",")++ strTensor = strArgs[0]+ intStrides = objVariables[strTensor].stride()+ strIndex = [+ "(("+ + strArgs[intArg + 1].replace("{", "(").replace("}", ")").strip()+ + ")*"+ + str(intStrides[intArg])+ + ")"+ for intArg in range(intArgs)+ ]++ strKernel = strKernel.replace(objMatch.group(0), strTensor + "[" + str.join("+", strIndex) + "]")+ # end++ return strKernel+++# end++[email protected](for_each_device=True)+def cupy_launch(strFunction, strKernel):+ return cupy.RawKernel(strKernel, strFunction)+++# end+++class _FunctionCorrelation(torch.autograd.Function):+ @staticmethod+ def forward(self, first, second):+ rbot0 = first.new_zeros([first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1]])+ rbot1 = first.new_zeros([first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1]])++ self.save_for_backward(first, second, rbot0, rbot1)++ first = first.contiguous()+ assert first.is_cuda == True+ second = second.contiguous()+ assert second.is_cuda == True++ output = first.new_zeros([first.shape[0], 81, first.shape[2], first.shape[3]])++ if first.is_cuda == True:+ n = first.shape[2] * first.shape[3]+ cupy_launch(+ "kernel_Correlation_rearrange",+ cupy_kernel("kernel_Correlation_rearrange", {"input": first, "output": rbot0}),+ )(+ grid=tuple([int((n + 16 - 1) / 16), first.shape[1], first.shape[0]]),+ block=tuple([16, 1, 1]),+ args=[n, first.data_ptr(), rbot0.data_ptr()],+ )++ n = second.shape[2] * second.shape[3]+ cupy_launch(+ "kernel_Correlation_rearrange",+ cupy_kernel("kernel_Correlation_rearrange", {"input": second, "output": rbot1}),+ )(+ grid=tuple([int((n + 16 - 1) / 16), second.shape[1], second.shape[0]]),+ block=tuple([16, 1, 1]),+ args=[n, second.data_ptr(), rbot1.data_ptr()],+ )++ n = output.shape[1] * output.shape[2] * output.shape[3]+ cupy_launch(+ "kernel_Correlation_updateOutput",+ cupy_kernel("kernel_Correlation_updateOutput", {"rbot0": rbot0, "rbot1": rbot1, "top": output}),+ )(+ grid=tuple([output.shape[3], output.shape[2], output.shape[0]]),+ block=tuple([32, 1, 1]),+ shared_mem=first.shape[1] * 4,+ args=[n, rbot0.data_ptr(), rbot1.data_ptr(), output.data_ptr()],+ )++ elif first.is_cuda == False:+ raise NotImplementedError()++ # end++ return output++ # end++ @staticmethod+ def backward(self, gradOutput):+ first, second, rbot0, rbot1 = self.saved_tensors++ gradOutput = gradOutput.contiguous()+ assert gradOutput.is_cuda == True++ gradFirst = (+ first.new_zeros([first.shape[0], first.shape[1], first.shape[2], first.shape[3]])+ if self.needs_input_grad[0] == True+ else None+ )+ gradSecond = (+ first.new_zeros([first.shape[0], first.shape[1], first.shape[2], first.shape[3]])+ if self.needs_input_grad[1] == True+ else None+ )++ if first.is_cuda == True:+ if gradFirst is not None:+ for intSample in range(first.shape[0]):+ n = first.shape[1] * first.shape[2] * first.shape[3]+ cupy_launch(+ "kernel_Correlation_updateGradFirst",+ cupy_kernel(+ "kernel_Correlation_updateGradFirst",+ {+ "rbot0": rbot0,+ "rbot1": rbot1,+ "gradOutput": gradOutput,+ "gradFirst": gradFirst,+ "gradSecond": None,+ },+ ),+ )(+ grid=tuple([int((n + 512 - 1) / 512), 1, 1]),+ block=tuple([512, 1, 1]),+ args=[+ n,+ intSample,+ rbot0.data_ptr(),+ rbot1.data_ptr(),+ gradOutput.data_ptr(),+ gradFirst.data_ptr(),+ None,+ ],+ )+ # end+ # end++ if gradSecond is not None:+ for intSample in range(first.shape[0]):+ n = first.shape[1] * first.shape[2] * first.shape[3]+ cupy_launch(+ "kernel_Correlation_updateGradSecond",+ cupy_kernel(+ "kernel_Correlation_updateGradSecond",+ {+ "rbot0": rbot0,+ "rbot1": rbot1,+ "gradOutput": gradOutput,+ "gradFirst": None,+ "gradSecond": gradSecond,+ },+ ),+ )(+ grid=tuple([int((n + 512 - 1) / 512), 1, 1]),+ block=tuple([512, 1, 1]),+ args=[+ n,+ intSample,+ rbot0.data_ptr(),+ rbot1.data_ptr(),+ gradOutput.data_ptr(),+ None,+ gradSecond.data_ptr(),+ ],+ )+ # end+ # end++ elif first.is_cuda == False:+ raise NotImplementedError()++ # end++ return gradFirst, gradSecond++ # end+++# end+++def FunctionCorrelation(tenFirst, tenSecond):+ return _FunctionCorrelation.apply(tenFirst, tenSecond)+++# end+++class ModuleCorrelation(torch.nn.Module):+ def __init__(self):+ super(ModuleCorrelation, self).__init__()++ # end++ def forward(self, tenFirst, tenSecond):+ return _FunctionCorrelation.apply(tenFirst, tenSecond)++ # end+++# enddiff --git a/eval/vae/flolpips/flolpips.py b/eval/vae/flolpips/flolpips.pyindex 6bd2bac..ca75f1d 100644--- a/eval/vae/flolpips/flolpips.py+++ b/eval/vae/flolpips/flolpips.py@@ -1,346 +1,412 @@--from __future__ import absolute_import-import os-import numpy as np-import torch-import torch.nn as nn-from torch.autograd import Variable-from .pretrained_networks import vgg16, alexnet, squeezenet-import torch.nn-import torch.nn.functional as F-import torchvision.transforms.functional as TF-import cv2--import hashlib-import requests-from tqdm import tqdm--from .pwcnet import Network as PWCNet-from .utils import *--URL_MAP = {"alex": "https://raw.githubusercontent.com/danier97/flolpips/main/weights/v0.1/alex.pth"}--CKPT_MAP = {"alex": "alex.pth"}--MD5_MAP = {"alex": "9642209e2b57a85d20f86d812320f9e6"}----def spatial_average(in_tens, keepdim=True):- return in_tens.mean([2,3],keepdim=keepdim)--def mw_spatial_average(in_tens, flow, keepdim=True):- _,_,h,w = in_tens.shape- flow = F.interpolate(flow, (h,w), align_corners=False, mode='bilinear')- flow_mag = torch.sqrt(flow[:,0:1]**2 + flow[:,1:2]**2)- flow_mag = flow_mag / torch.sum(flow_mag, dim=[1,2,3], keepdim=True)- return torch.sum(in_tens*flow_mag, dim=[2,3],keepdim=keepdim)---def mtw_spatial_average(in_tens, flow, texture, keepdim=True):- _,_,h,w = in_tens.shape- flow = F.interpolate(flow, (h,w), align_corners=False, mode='bilinear')- texture = F.interpolate(texture, (h,w), align_corners=False, mode='bilinear')- flow_mag = torch.sqrt(flow[:,0:1]**2 + flow[:,1:2]**2)- flow_mag = (flow_mag - flow_mag.min()) / (flow_mag.max() - flow_mag.min()) + 1e-6- texture = (texture - texture.min()) / (texture.max() - texture.min()) + 1e-6- weight = flow_mag / texture- weight /= torch.sum(weight)- return torch.sum(in_tens*weight, dim=[2,3],keepdim=keepdim)----def m2w_spatial_average(in_tens, flow, keepdim=True):- _,_,h,w = in_tens.shape- flow = F.interpolate(flow, (h,w), align_corners=False, mode='bilinear')- flow_mag = flow[:,0:1]**2 + flow[:,1:2]**2 # B,1,H,W- flow_mag = flow_mag / torch.sum(flow_mag)- return torch.sum(in_tens*flow_mag, dim=[2,3],keepdim=keepdim)--def upsample(in_tens, out_HW=(64,64)): # assumes scale factor is same for H and W- in_H, in_W = in_tens.shape[2], in_tens.shape[3]- return nn.Upsample(size=out_HW, mode='bilinear', align_corners=False)(in_tens)---def md5_hash(path):- with open(path, "rb") as f:- content = f.read()- return hashlib.md5(content).hexdigest()--def download(url, local_path, chunk_size=1024):- os.makedirs(os.path.split(local_path)[0], exist_ok=True)- with requests.get(url, stream=True) as r:- total_size = int(r.headers.get("content-length", 0))- with tqdm(total=total_size, unit="B", unit_scale=True) as pbar:- with open(local_path, "wb") as f:- for data in r.iter_content(chunk_size=chunk_size):- if data:- f.write(data)- pbar.update(chunk_size)--def get_ckpt_path(name, root, check=False):- assert name in URL_MAP- path = os.path.join(root, CKPT_MAP[name])- if not os.path.exists(path) or (check and not md5_hash(path) == MD5_MAP[name]):- print("Downloading {} model from {} to {}".format(name, URL_MAP[name], path))- download(URL_MAP[name], path)- md5 = md5_hash(path)- assert md5 == MD5_MAP[name], md5- return path--# Learned perceptual metric-class LPIPS(nn.Module):- def __init__(self, pretrained=True, net='alex', version='0.1', lpips=True, spatial=False,- pnet_rand=False, pnet_tune=False, use_dropout=True, model_path=None, eval_mode=True, verbose=False):- # lpips - [True] means with linear calibration on top of base network- # pretrained - [True] means load linear weights-- super(LPIPS, self).__init__()- if(verbose):- print('Setting up [%s] perceptual loss: trunk [%s], v[%s], spatial [%s]'%- ('LPIPS' if lpips else 'baseline', net, version, 'on' if spatial else 'off'))-- self.pnet_type = net- self.pnet_tune = pnet_tune- self.pnet_rand = pnet_rand- self.spatial = spatial- self.lpips = lpips # false means baseline of just averaging all layers- self.version = version- self.scaling_layer = ScalingLayer()-- if(self.pnet_type in ['vgg','vgg16']):- net_type = vgg16- self.chns = [64,128,256,512,512]- elif(self.pnet_type=='alex'):- net_type = alexnet- self.chns = [64,192,384,256,256]- elif(self.pnet_type=='squeeze'):- net_type = squeezenet- self.chns = [64,128,256,384,384,512,512]- self.L = len(self.chns)-- self.net = net_type(pretrained=not self.pnet_rand, requires_grad=self.pnet_tune)-- if(lpips):- self.lin0 = NetLinLayer(self.chns[0], use_dropout=use_dropout)- self.lin1 = NetLinLayer(self.chns[1], use_dropout=use_dropout)- self.lin2 = NetLinLayer(self.chns[2], use_dropout=use_dropout)- self.lin3 = NetLinLayer(self.chns[3], use_dropout=use_dropout)- self.lin4 = NetLinLayer(self.chns[4], use_dropout=use_dropout)- self.lins = [self.lin0,self.lin1,self.lin2,self.lin3,self.lin4]- if(self.pnet_type=='squeeze'): # 7 layers for squeezenet- self.lin5 = NetLinLayer(self.chns[5], use_dropout=use_dropout)- self.lin6 = NetLinLayer(self.chns[6], use_dropout=use_dropout)- self.lins+=[self.lin5,self.lin6]- self.lins = nn.ModuleList(self.lins)-- if(pretrained):- self.load_from_pretrained(version, net)- if(verbose):- print('Loaded model from: %s'%model_path)-- if(eval_mode):- self.eval()-- def load_from_pretrained(self, version, net):- ckpt = get_ckpt_path(net, "pretrained_models/flolpips/weights/v%s"%(version))- self.load_state_dict(torch.load(ckpt, map_location='cpu'), strict=False)-- def forward(self, in0, in1, retPerLayer=False, normalize=False):- if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]- in0 = 2 * in0 - 1- in1 = 2 * in1 - 1-- # v0.0 - original release had a bug, where input was not scaled- in0_input, in1_input = (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version=='0.1' else (in0, in1)- outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)- feats0, feats1, diffs = {}, {}, {}-- for kk in range(self.L):- feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])- diffs[kk] = (feats0[kk]-feats1[kk])**2-- if(self.lpips):- if(self.spatial):- res = [upsample(self.lins[kk](diffs[kk]), out_HW=in0.shape[2:]) for kk in range(self.L)]- else:- res = [spatial_average(self.lins[kk](diffs[kk]), keepdim=True) for kk in range(self.L)]- else:- if(self.spatial):- res = [upsample(diffs[kk].sum(dim=1,keepdim=True), out_HW=in0.shape[2:]) for kk in range(self.L)]- else:- res = [spatial_average(diffs[kk].sum(dim=1,keepdim=True), keepdim=True) for kk in range(self.L)]-- # val = res[0]- # for l in range(1,self.L):- # val += res[l]- # print(val)-- # a = spatial_average(self.lins[kk](diffs[kk]), keepdim=True)- # b = torch.max(self.lins[kk](feats0[kk]**2))- # for kk in range(self.L):- # a += spatial_average(self.lins[kk](diffs[kk]), keepdim=True)- # b = torch.max(b,torch.max(self.lins[kk](feats0[kk]**2)))- # a = a/self.L- # from IPython import embed- # embed()- # return 10*torch.log10(b/a)-- # if(retPerLayer):- # return (val, res)- # else:- return torch.sum(torch.cat(res, 1), dim=(1,2,3), keepdims=False)---class ScalingLayer(nn.Module):- def __init__(self):- super(ScalingLayer, self).__init__()- self.register_buffer('shift', torch.Tensor([-.030,-.088,-.188])[None,:,None,None])- self.register_buffer('scale', torch.Tensor([.458,.448,.450])[None,:,None,None])-- def forward(self, inp):- return (inp - self.shift) / self.scale---class NetLinLayer(nn.Module):- ''' A single linear layer which does a 1x1 conv '''- def __init__(self, chn_in, chn_out=1, use_dropout=False):- super(NetLinLayer, self).__init__()-- layers = [nn.Dropout(),] if(use_dropout) else []- layers += [nn.Conv2d(chn_in, chn_out, 1, stride=1, padding=0, bias=False),]- self.model = nn.Sequential(*layers)-- def forward(self, x):- return self.model(x)--class Dist2LogitLayer(nn.Module):- ''' takes 2 distances, puts through fc layers, spits out value between [0,1] (if use_sigmoid is True) '''- def __init__(self, chn_mid=32, use_sigmoid=True):- super(Dist2LogitLayer, self).__init__()-- layers = [nn.Conv2d(5, chn_mid, 1, stride=1, padding=0, bias=True),]- layers += [nn.LeakyReLU(0.2,True),]- layers += [nn.Conv2d(chn_mid, chn_mid, 1, stride=1, padding=0, bias=True),]- layers += [nn.LeakyReLU(0.2,True),]- layers += [nn.Conv2d(chn_mid, 1, 1, stride=1, padding=0, bias=True),]- if(use_sigmoid):- layers += [nn.Sigmoid(),]- self.model = nn.Sequential(*layers)-- def forward(self,d0,d1,eps=0.1):- return self.model.forward(torch.cat((d0,d1,d0-d1,d0/(d1+eps),d1/(d0+eps)),dim=1))--class BCERankingLoss(nn.Module):- def __init__(self, chn_mid=32):- super(BCERankingLoss, self).__init__()- self.net = Dist2LogitLayer(chn_mid=chn_mid)- # self.parameters = list(self.net.parameters())- self.loss = torch.nn.BCELoss()-- def forward(self, d0, d1, judge):- per = (judge+1.)/2.- self.logit = self.net.forward(d0,d1)- return self.loss(self.logit, per)--# L2, DSSIM metrics-class FakeNet(nn.Module):- def __init__(self, use_gpu=True, colorspace='Lab'):- super(FakeNet, self).__init__()- self.use_gpu = use_gpu- self.colorspace = colorspace--class L2(FakeNet):- def forward(self, in0, in1, retPerLayer=None):- assert(in0.size()[0]==1) # currently only supports batchSize 1-- if(self.colorspace=='RGB'):- (N,C,X,Y) = in0.size()- value = torch.mean(torch.mean(torch.mean((in0-in1)**2,dim=1).view(N,1,X,Y),dim=2).view(N,1,1,Y),dim=3).view(N)- return value- elif(self.colorspace=='Lab'):- value = l2(tensor2np(tensor2tensorlab(in0.data,to_norm=False)),- tensor2np(tensor2tensorlab(in1.data,to_norm=False)), range=100.).astype('float')- ret_var = Variable( torch.Tensor((value,) ) )- if(self.use_gpu):- ret_var = ret_var.cuda()- return ret_var--class DSSIM(FakeNet):-- def forward(self, in0, in1, retPerLayer=None):- assert(in0.size()[0]==1) # currently only supports batchSize 1-- if(self.colorspace=='RGB'):- value = dssim(1.*tensor2im(in0.data), 1.*tensor2im(in1.data), range=255.).astype('float')- elif(self.colorspace=='Lab'):- value = dssim(tensor2np(tensor2tensorlab(in0.data,to_norm=False)),- tensor2np(tensor2tensorlab(in1.data,to_norm=False)), range=100.).astype('float')- ret_var = Variable( torch.Tensor((value,) ) )- if(self.use_gpu):- ret_var = ret_var.cuda()- return ret_var--def print_network(net):- num_params = 0- for param in net.parameters():- num_params += param.numel()- print('Network',net)- print('Total number of parameters: %d' % num_params)---class FloLPIPS(LPIPS):- def __init__(self, pretrained=True, net='alex', version='0.1', lpips=True, spatial=False, pnet_rand=False, pnet_tune=False, use_dropout=True, model_path=None, eval_mode=True, verbose=False):- super(FloLPIPS, self).__init__(pretrained, net, version, lpips, spatial, pnet_rand, pnet_tune, use_dropout, model_path, eval_mode, verbose)-- def forward(self, in0, in1, flow, retPerLayer=False, normalize=False):- if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]- in0 = 2 * in0 - 1- in1 = 2 * in1 - 1-- in0_input, in1_input = (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version=='0.1' else (in0, in1)- outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)- feats0, feats1, diffs = {}, {}, {}-- for kk in range(self.L):- feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])- diffs[kk] = (feats0[kk]-feats1[kk])**2-- res = [mw_spatial_average(self.lins[kk](diffs[kk]), flow, keepdim=True) for kk in range(self.L)]-- return torch.sum(torch.cat(res, 1), dim=(1,2,3), keepdims=False)------class Flolpips(nn.Module):- def __init__(self):- super(Flolpips, self).__init__()- self.loss_fn = FloLPIPS(net='alex',version='0.1')- self.flownet = PWCNet()-- @torch.no_grad()- def forward(self, I0, I1, frame_dis, frame_ref):- """- args:- I0: first frame of the triplet, shape: [B, C, H, W]- I1: third frame of the triplet, shape: [B, C, H, W]- frame_dis: prediction of the intermediate frame, shape: [B, C, H, W]- frame_ref: ground-truth of the intermediate frame, shape: [B, C, H, W]- """- assert I0.size() == I1.size() == frame_dis.size() == frame_ref.size(), \- "the 4 input tensors should have same size"-- flow_ref = self.flownet(frame_ref, I0)- flow_dis = self.flownet(frame_dis, I0)- flow_diff = flow_ref - flow_dis- flolpips_wrt_I0 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)-- flow_ref = self.flownet(frame_ref, I1)- flow_dis = self.flownet(frame_dis, I1)- flow_diff = flow_ref - flow_dis- flolpips_wrt_I1 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)-- flolpips = (flolpips_wrt_I0 + flolpips_wrt_I1) / 2- return flolpips\ No newline at end of file+from __future__ import absolute_import++import hashlib+import os++import requests+import torch+import torch.nn+import torch.nn as nn+import torch.nn.functional as F+from torch.autograd import Variable+from tqdm import tqdm++from .pretrained_networks import alexnet, squeezenet, vgg16+from .pwcnet import Network as PWCNet+from .utils import *++URL_MAP = {"alex": "https://raw.githubusercontent.com/danier97/flolpips/main/weights/v0.1/alex.pth"}++CKPT_MAP = {"alex": "alex.pth"}++MD5_MAP = {"alex": "9642209e2b57a85d20f86d812320f9e6"}+++def spatial_average(in_tens, keepdim=True):+ return in_tens.mean([2, 3], keepdim=keepdim)+++def mw_spatial_average(in_tens, flow, keepdim=True):+ _, _, h, w = in_tens.shape+ flow = F.interpolate(flow, (h, w), align_corners=False, mode="bilinear")+ flow_mag = torch.sqrt(flow[:, 0:1] ** 2 + flow[:, 1:2] ** 2)+ flow_mag = flow_mag / torch.sum(flow_mag, dim=[1, 2, 3], keepdim=True)+ return torch.sum(in_tens * flow_mag, dim=[2, 3], keepdim=keepdim)+++def mtw_spatial_average(in_tens, flow, texture, keepdim=True):+ _, _, h, w = in_tens.shape+ flow = F.interpolate(flow, (h, w), align_corners=False, mode="bilinear")+ texture = F.interpolate(texture, (h, w), align_corners=False, mode="bilinear")+ flow_mag = torch.sqrt(flow[:, 0:1] ** 2 + flow[:, 1:2] ** 2)+ flow_mag = (flow_mag - flow_mag.min()) / (flow_mag.max() - flow_mag.min()) + 1e-6+ texture = (texture - texture.min()) / (texture.max() - texture.min()) + 1e-6+ weight = flow_mag / texture+ weight /= torch.sum(weight)+ return torch.sum(in_tens * weight, dim=[2, 3], keepdim=keepdim)+++def m2w_spatial_average(in_tens, flow, keepdim=True):+ _, _, h, w = in_tens.shape+ flow = F.interpolate(flow, (h, w), align_corners=False, mode="bilinear")+ flow_mag = flow[:, 0:1] ** 2 + flow[:, 1:2] ** 2 # B,1,H,W+ flow_mag = flow_mag / torch.sum(flow_mag)+ return torch.sum(in_tens * flow_mag, dim=[2, 3], keepdim=keepdim)+++def upsample(in_tens, out_HW=(64, 64)): # assumes scale factor is same for H and W+ in_H, in_W = in_tens.shape[2], in_tens.shape[3]+ return nn.Upsample(size=out_HW, mode="bilinear", align_corners=False)(in_tens)+++def md5_hash(path):+ with open(path, "rb") as f:+ content = f.read()+ return hashlib.md5(content).hexdigest()+++def download(url, local_path, chunk_size=1024):+ os.makedirs(os.path.split(local_path)[0], exist_ok=True)+ with requests.get(url, stream=True) as r:+ total_size = int(r.headers.get("content-length", 0))+ with tqdm(total=total_size, unit="B", unit_scale=True) as pbar:+ with open(local_path, "wb") as f:+ for data in r.iter_content(chunk_size=chunk_size):+ if data:+ f.write(data)+ pbar.update(chunk_size)+++def get_ckpt_path(name, root, check=False):+ assert name in URL_MAP+ path = os.path.join(root, CKPT_MAP[name])+ if not os.path.exists(path) or (check and not md5_hash(path) == MD5_MAP[name]):+ print("Downloading {} model from {} to {}".format(name, URL_MAP[name], path))+ download(URL_MAP[name], path)+ md5 = md5_hash(path)+ assert md5 == MD5_MAP[name], md5+ return path+++# Learned perceptual metric+class LPIPS(nn.Module):+ def __init__(+ self,+ pretrained=True,+ net="alex",+ version="0.1",+ lpips=True,+ spatial=False,+ pnet_rand=False,+ pnet_tune=False,+ use_dropout=True,+ model_path=None,+ eval_mode=True,+ verbose=False,+ ):+ # lpips - [True] means with linear calibration on top of base network+ # pretrained - [True] means load linear weights++ super(LPIPS, self).__init__()+ if verbose:+ print(+ "Setting up [%s] perceptual loss: trunk [%s], v[%s], spatial [%s]"+ % ("LPIPS" if lpips else "baseline", net, version, "on" if spatial else "off")+ )++ self.pnet_type = net+ self.pnet_tune = pnet_tune+ self.pnet_rand = pnet_rand+ self.spatial = spatial+ self.lpips = lpips # false means baseline of just averaging all layers+ self.version = version+ self.scaling_layer = ScalingLayer()++ if self.pnet_type in ["vgg", "vgg16"]:+ net_type = vgg16+ self.chns = [64, 128, 256, 512, 512]+ elif self.pnet_type == "alex":+ net_type = alexnet+ self.chns = [64, 192, 384, 256, 256]+ elif self.pnet_type == "squeeze":+ net_type = squeezenet+ self.chns = [64, 128, 256, 384, 384, 512, 512]+ self.L = len(self.chns)++ self.net = net_type(pretrained=not self.pnet_rand, requires_grad=self.pnet_tune)++ if lpips:+ self.lin0 = NetLinLayer(self.chns[0], use_dropout=use_dropout)+ self.lin1 = NetLinLayer(self.chns[1], use_dropout=use_dropout)+ self.lin2 = NetLinLayer(self.chns[2], use_dropout=use_dropout)+ self.lin3 = NetLinLayer(self.chns[3], use_dropout=use_dropout)+ self.lin4 = NetLinLayer(self.chns[4], use_dropout=use_dropout)+ self.lins = [self.lin0, self.lin1, self.lin2, self.lin3, self.lin4]+ if self.pnet_type == "squeeze": # 7 layers for squeezenet+ self.lin5 = NetLinLayer(self.chns[5], use_dropout=use_dropout)+ self.lin6 = NetLinLayer(self.chns[6], use_dropout=use_dropout)+ self.lins += [self.lin5, self.lin6]+ self.lins = nn.ModuleList(self.lins)++ if pretrained:+ self.load_from_pretrained(version, net)+ if verbose:+ print("Loaded model from: %s" % model_path)++ if eval_mode:+ self.eval()++ def load_from_pretrained(self, version, net):+ ckpt = get_ckpt_path(net, "pretrained_models/flolpips/weights/v%s" % (version))+ self.load_state_dict(torch.load(ckpt, map_location="cpu"), strict=False)++ def forward(self, in0, in1, retPerLayer=False, normalize=False):+ if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]+ in0 = 2 * in0 - 1+ in1 = 2 * in1 - 1++ # v0.0 - original release had a bug, where input was not scaled+ in0_input, in1_input = (+ (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version == "0.1" else (in0, in1)+ )+ outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)+ feats0, feats1, diffs = {}, {}, {}++ for kk in range(self.L):+ feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])+ diffs[kk] = (feats0[kk] - feats1[kk]) ** 2++ if self.lpips:+ if self.spatial:+ res = [upsample(self.lins[kk](diffs[kk]), out_HW=in0.shape[2:]) for kk in range(self.L)]+ else:+ res = [spatial_average(self.lins[kk](diffs[kk]), keepdim=True) for kk in range(self.L)]+ else:+ if self.spatial:+ res = [upsample(diffs[kk].sum(dim=1, keepdim=True), out_HW=in0.shape[2:]) for kk in range(self.L)]+ else:+ res = [spatial_average(diffs[kk].sum(dim=1, keepdim=True), keepdim=True) for kk in range(self.L)]++ # val = res[0]+ # for l in range(1,self.L):+ # val += res[l]+ # print(val)++ # a = spatial_average(self.lins[kk](diffs[kk]), keepdim=True)+ # b = torch.max(self.lins[kk](feats0[kk]**2))+ # for kk in range(self.L):+ # a += spatial_average(self.lins[kk](diffs[kk]), keepdim=True)+ # b = torch.max(b,torch.max(self.lins[kk](feats0[kk]**2)))+ # a = a/self.L+ # from IPython import embed+ # embed()+ # return 10*torch.log10(b/a)++ # if(retPerLayer):+ # return (val, res)+ # else:+ return torch.sum(torch.cat(res, 1), dim=(1, 2, 3), keepdims=False)+++class ScalingLayer(nn.Module):+ def __init__(self):+ super(ScalingLayer, self).__init__()+ self.register_buffer("shift", torch.Tensor([-0.030, -0.088, -0.188])[None, :, None, None])+ self.register_buffer("scale", torch.Tensor([0.458, 0.448, 0.450])[None, :, None, None])++ def forward(self, inp):+ return (inp - self.shift) / self.scale+++class NetLinLayer(nn.Module):+ """A single linear layer which does a 1x1 conv"""++ def __init__(self, chn_in, chn_out=1, use_dropout=False):+ super(NetLinLayer, self).__init__()++ layers = (+ [+ nn.Dropout(),+ ]+ if (use_dropout)+ else []+ )+ layers += [+ nn.Conv2d(chn_in, chn_out, 1, stride=1, padding=0, bias=False),+ ]+ self.model = nn.Sequential(*layers)++ def forward(self, x):+ return self.model(x)+++class Dist2LogitLayer(nn.Module):+ """takes 2 distances, puts through fc layers, spits out value between [0,1] (if use_sigmoid is True)"""++ def __init__(self, chn_mid=32, use_sigmoid=True):+ super(Dist2LogitLayer, self).__init__()++ layers = [+ nn.Conv2d(5, chn_mid, 1, stride=1, padding=0, bias=True),+ ]+ layers += [+ nn.LeakyReLU(0.2, True),+ ]+ layers += [+ nn.Conv2d(chn_mid, chn_mid, 1, stride=1, padding=0, bias=True),+ ]+ layers += [+ nn.LeakyReLU(0.2, True),+ ]+ layers += [+ nn.Conv2d(chn_mid, 1, 1, stride=1, padding=0, bias=True),+ ]+ if use_sigmoid:+ layers += [+ nn.Sigmoid(),+ ]+ self.model = nn.Sequential(*layers)++ def forward(self, d0, d1, eps=0.1):+ return self.model.forward(torch.cat((d0, d1, d0 - d1, d0 / (d1 + eps), d1 / (d0 + eps)), dim=1))+++class BCERankingLoss(nn.Module):+ def __init__(self, chn_mid=32):+ super(BCERankingLoss, self).__init__()+ self.net = Dist2LogitLayer(chn_mid=chn_mid)+ # self.parameters = list(self.net.parameters())+ self.loss = torch.nn.BCELoss()++ def forward(self, d0, d1, judge):+ per = (judge + 1.0) / 2.0+ self.logit = self.net.forward(d0, d1)+ return self.loss(self.logit, per)+++# L2, DSSIM metrics+class FakeNet(nn.Module):+ def __init__(self, use_gpu=True, colorspace="Lab"):+ super(FakeNet, self).__init__()+ self.use_gpu = use_gpu+ self.colorspace = colorspace+++class L2(FakeNet):+ def forward(self, in0, in1, retPerLayer=None):+ assert in0.size()[0] == 1 # currently only supports batchSize 1++ if self.colorspace == "RGB":+ (N, C, X, Y) = in0.size()+ value = torch.mean(+ torch.mean(torch.mean((in0 - in1) ** 2, dim=1).view(N, 1, X, Y), dim=2).view(N, 1, 1, Y), dim=3+ ).view(N)+ return value+ elif self.colorspace == "Lab":+ value = l2(+ tensor2np(tensor2tensorlab(in0.data, to_norm=False)),+ tensor2np(tensor2tensorlab(in1.data, to_norm=False)),+ range=100.0,+ ).astype("float")+ ret_var = Variable(torch.Tensor((value,)))+ if self.use_gpu:+ ret_var = ret_var.cuda()+ return ret_var+++class DSSIM(FakeNet):+ def forward(self, in0, in1, retPerLayer=None):+ assert in0.size()[0] == 1 # currently only supports batchSize 1++ if self.colorspace == "RGB":+ value = dssim(1.0 * tensor2im(in0.data), 1.0 * tensor2im(in1.data), range=255.0).astype("float")+ elif self.colorspace == "Lab":+ value = dssim(+ tensor2np(tensor2tensorlab(in0.data, to_norm=False)),+ tensor2np(tensor2tensorlab(in1.data, to_norm=False)),+ range=100.0,+ ).astype("float")+ ret_var = Variable(torch.Tensor((value,)))+ if self.use_gpu:+ ret_var = ret_var.cuda()+ return ret_var+++def print_network(net):+ num_params = 0+ for param in net.parameters():+ num_params += param.numel()+ print("Network", net)+ print("Total number of parameters: %d" % num_params)+++class FloLPIPS(LPIPS):+ def __init__(+ self,+ pretrained=True,+ net="alex",+ version="0.1",+ lpips=True,+ spatial=False,+ pnet_rand=False,+ pnet_tune=False,+ use_dropout=True,+ model_path=None,+ eval_mode=True,+ verbose=False,+ ):+ super(FloLPIPS, self).__init__(+ pretrained, net, version, lpips, spatial, pnet_rand, pnet_tune, use_dropout, model_path, eval_mode, verbose+ )++ def forward(self, in0, in1, flow, retPerLayer=False, normalize=False):+ if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]+ in0 = 2 * in0 - 1+ in1 = 2 * in1 - 1++ in0_input, in1_input = (+ (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version == "0.1" else (in0, in1)+ )+ outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)+ feats0, feats1, diffs = {}, {}, {}++ for kk in range(self.L):+ feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])+ diffs[kk] = (feats0[kk] - feats1[kk]) ** 2++ res = [mw_spatial_average(self.lins[kk](diffs[kk]), flow, keepdim=True) for kk in range(self.L)]++ return torch.sum(torch.cat(res, 1), dim=(1, 2, 3), keepdims=False)+++class Flolpips(nn.Module):+ def __init__(self):+ super(Flolpips, self).__init__()+ self.loss_fn = FloLPIPS(net="alex", version="0.1")+ self.flownet = PWCNet()++ @torch.no_grad()+ def forward(self, I0, I1, frame_dis, frame_ref):+ """+ args:+ I0: first frame of the triplet, shape: [B, C, H, W]+ I1: third frame of the triplet, shape: [B, C, H, W]+ frame_dis: prediction of the intermediate frame, shape: [B, C, H, W]+ frame_ref: ground-truth of the intermediate frame, shape: [B, C, H, W]+ """+ assert (+ I0.size() == I1.size() == frame_dis.size() == frame_ref.size()+ ), "the 4 input tensors should have same size"++ flow_ref = self.flownet(frame_ref, I0)+ flow_dis = self.flownet(frame_dis, I0)+ flow_diff = flow_ref - flow_dis+ flolpips_wrt_I0 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)++ flow_ref = self.flownet(frame_ref, I1)+ flow_dis = self.flownet(frame_dis, I1)+ flow_diff = flow_ref - flow_dis+ flolpips_wrt_I1 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)++ flolpips = (flolpips_wrt_I0 + flolpips_wrt_I1) / 2+ return flolpipsdiff --git a/eval/vae/flolpips/pretrained_networks.py b/eval/vae/flolpips/pretrained_networks.pyindex 82f14e2..5915b05 100644--- a/eval/vae/flolpips/pretrained_networks.py+++ b/eval/vae/flolpips/pretrained_networks.py@@ -1,180 +1,182 @@-from collections import namedtuple-import torch-from torchvision import models as tv--class squeezenet(torch.nn.Module):- def __init__(self, requires_grad=False, pretrained=True):- super(squeezenet, self).__init__()- pretrained_features = tv.squeezenet1_1(pretrained=pretrained).features- self.slice1 = torch.nn.Sequential()- self.slice2 = torch.nn.Sequential()- self.slice3 = torch.nn.Sequential()- self.slice4 = torch.nn.Sequential()- self.slice5 = torch.nn.Sequential()- self.slice6 = torch.nn.Sequential()- self.slice7 = torch.nn.Sequential()- self.N_slices = 7- for x in range(2):- self.slice1.add_module(str(x), pretrained_features[x])- for x in range(2,5):- self.slice2.add_module(str(x), pretrained_features[x])- for x in range(5, 8):- self.slice3.add_module(str(x), pretrained_features[x])- for x in range(8, 10):- self.slice4.add_module(str(x), pretrained_features[x])- for x in range(10, 11):- self.slice5.add_module(str(x), pretrained_features[x])- for x in range(11, 12):- self.slice6.add_module(str(x), pretrained_features[x])- for x in range(12, 13):- self.slice7.add_module(str(x), pretrained_features[x])- if not requires_grad:- for param in self.parameters():- param.requires_grad = False-- def forward(self, X):- h = self.slice1(X)- h_relu1 = h- h = self.slice2(h)- h_relu2 = h- h = self.slice3(h)- h_relu3 = h- h = self.slice4(h)- h_relu4 = h- h = self.slice5(h)- h_relu5 = h- h = self.slice6(h)- h_relu6 = h- h = self.slice7(h)- h_relu7 = h- vgg_outputs = namedtuple("SqueezeOutputs", ['relu1','relu2','relu3','relu4','relu5','relu6','relu7'])- out = vgg_outputs(h_relu1,h_relu2,h_relu3,h_relu4,h_relu5,h_relu6,h_relu7)-- return out---class alexnet(torch.nn.Module):- def __init__(self, requires_grad=False, pretrained=True):- super(alexnet, self).__init__()- alexnet_pretrained_features = tv.alexnet(pretrained=pretrained).features- self.slice1 = torch.nn.Sequential()- self.slice2 = torch.nn.Sequential()- self.slice3 = torch.nn.Sequential()- self.slice4 = torch.nn.Sequential()- self.slice5 = torch.nn.Sequential()- self.N_slices = 5- for x in range(2):- self.slice1.add_module(str(x), alexnet_pretrained_features[x])- for x in range(2, 5):- self.slice2.add_module(str(x), alexnet_pretrained_features[x])- for x in range(5, 8):- self.slice3.add_module(str(x), alexnet_pretrained_features[x])- for x in range(8, 10):- self.slice4.add_module(str(x), alexnet_pretrained_features[x])- for x in range(10, 12):- self.slice5.add_module(str(x), alexnet_pretrained_features[x])- if not requires_grad:- for param in self.parameters():- param.requires_grad = False-- def forward(self, X):- h = self.slice1(X)- h_relu1 = h- h = self.slice2(h)- h_relu2 = h- h = self.slice3(h)- h_relu3 = h- h = self.slice4(h)- h_relu4 = h- h = self.slice5(h)- h_relu5 = h- alexnet_outputs = namedtuple("AlexnetOutputs", ['relu1', 'relu2', 'relu3', 'relu4', 'relu5'])- out = alexnet_outputs(h_relu1, h_relu2, h_relu3, h_relu4, h_relu5)-- return out--class vgg16(torch.nn.Module):- def __init__(self, requires_grad=False, pretrained=True):- super(vgg16, self).__init__()- vgg_pretrained_features = tv.vgg16(pretrained=pretrained).features- self.slice1 = torch.nn.Sequential()- self.slice2 = torch.nn.Sequential()- self.slice3 = torch.nn.Sequential()- self.slice4 = torch.nn.Sequential()- self.slice5 = torch.nn.Sequential()- self.N_slices = 5- for x in range(4):- self.slice1.add_module(str(x), vgg_pretrained_features[x])- for x in range(4, 9):- self.slice2.add_module(str(x), vgg_pretrained_features[x])- for x in range(9, 16):- self.slice3.add_module(str(x), vgg_pretrained_features[x])- for x in range(16, 23):- self.slice4.add_module(str(x), vgg_pretrained_features[x])- for x in range(23, 30):- self.slice5.add_module(str(x), vgg_pretrained_features[x])- if not requires_grad:- for param in self.parameters():- param.requires_grad = False-- def forward(self, X):- h = self.slice1(X)- h_relu1_2 = h- h = self.slice2(h)- h_relu2_2 = h- h = self.slice3(h)- h_relu3_3 = h- h = self.slice4(h)- h_relu4_3 = h- h = self.slice5(h)- h_relu5_3 = h- vgg_outputs = namedtuple("VggOutputs", ['relu1_2', 'relu2_2', 'relu3_3', 'relu4_3', 'relu5_3'])- out = vgg_outputs(h_relu1_2, h_relu2_2, h_relu3_3, h_relu4_3, h_relu5_3)-- return out----class resnet(torch.nn.Module):- def __init__(self, requires_grad=False, pretrained=True, num=18):- super(resnet, self).__init__()- if(num==18):- self.net = tv.resnet18(pretrained=pretrained)- elif(num==34):- self.net = tv.resnet34(pretrained=pretrained)- elif(num==50):- self.net = tv.resnet50(pretrained=pretrained)- elif(num==101):- self.net = tv.resnet101(pretrained=pretrained)- elif(num==152):- self.net = tv.resnet152(pretrained=pretrained)- self.N_slices = 5-- self.conv1 = self.net.conv1- self.bn1 = self.net.bn1- self.relu = self.net.relu- self.maxpool = self.net.maxpool- self.layer1 = self.net.layer1- self.layer2 = self.net.layer2- self.layer3 = self.net.layer3- self.layer4 = self.net.layer4-- def forward(self, X):- h = self.conv1(X)- h = self.bn1(h)- h = self.relu(h)- h_relu1 = h- h = self.maxpool(h)- h = self.layer1(h)- h_conv2 = h- h = self.layer2(h)- h_conv3 = h- h = self.layer3(h)- h_conv4 = h- h = self.layer4(h)- h_conv5 = h-- outputs = namedtuple("Outputs", ['relu1','conv2','conv3','conv4','conv5'])- out = outputs(h_relu1, h_conv2, h_conv3, h_conv4, h_conv5)-- return out+from collections import namedtuple++import torch+from torchvision import models as tv+++class squeezenet(torch.nn.Module):+ def __init__(self, requires_grad=False, pretrained=True):+ super(squeezenet, self).__init__()+ pretrained_features = tv.squeezenet1_1(pretrained=pretrained).features+ self.slice1 = torch.nn.Sequential()+ self.slice2 = torch.nn.Sequential()+ self.slice3 = torch.nn.Sequential()+ self.slice4 = torch.nn.Sequential()+ self.slice5 = torch.nn.Sequential()+ self.slice6 = torch.nn.Sequential()+ self.slice7 = torch.nn.Sequential()+ self.N_slices = 7+ for x in range(2):+ self.slice1.add_module(str(x), pretrained_features[x])+ for x in range(2, 5):+ self.slice2.add_module(str(x), pretrained_features[x])+ for x in range(5, 8):+ self.slice3.add_module(str(x), pretrained_features[x])+ for x in range(8, 10):+ self.slice4.add_module(str(x), pretrained_features[x])+ for x in range(10, 11):+ self.slice5.add_module(str(x), pretrained_features[x])+ for x in range(11, 12):+ self.slice6.add_module(str(x), pretrained_features[x])+ for x in range(12, 13):+ self.slice7.add_module(str(x), pretrained_features[x])+ if not requires_grad:+ for param in self.parameters():+ param.requires_grad = False++ def forward(self, X):+ h = self.slice1(X)+ h_relu1 = h+ h = self.slice2(h)+ h_relu2 = h+ h = self.slice3(h)+ h_relu3 = h+ h = self.slice4(h)+ h_relu4 = h+ h = self.slice5(h)+ h_relu5 = h+ h = self.slice6(h)+ h_relu6 = h+ h = self.slice7(h)+ h_relu7 = h+ vgg_outputs = namedtuple("SqueezeOutputs", ["relu1", "relu2", "relu3", "relu4", "relu5", "relu6", "relu7"])+ out = vgg_outputs(h_relu1, h_relu2, h_relu3, h_relu4, h_relu5, h_relu6, h_relu7)++ return out+++class alexnet(torch.nn.Module):+ def __init__(self, requires_grad=False, pretrained=True):+ super(alexnet, self).__init__()+ alexnet_pretrained_features = tv.alexnet(pretrained=pretrained).features+ self.slice1 = torch.nn.Sequential()+ self.slice2 = torch.nn.Sequential()+ self.slice3 = torch.nn.Sequential()+ self.slice4 = torch.nn.Sequential()+ self.slice5 = torch.nn.Sequential()+ self.N_slices = 5+ for x in range(2):+ self.slice1.add_module(str(x), alexnet_pretrained_features[x])+ for x in range(2, 5):+ self.slice2.add_module(str(x), alexnet_pretrained_features[x])+ for x in range(5, 8):+ self.slice3.add_module(str(x), alexnet_pretrained_features[x])+ for x in range(8, 10):+ self.slice4.add_module(str(x), alexnet_pretrained_features[x])+ for x in range(10, 12):+ self.slice5.add_module(str(x), alexnet_pretrained_features[x])+ if not requires_grad:+ for param in self.parameters():+ param.requires_grad = False++ def forward(self, X):+ h = self.slice1(X)+ h_relu1 = h+ h = self.slice2(h)+ h_relu2 = h+ h = self.slice3(h)+ h_relu3 = h+ h = self.slice4(h)+ h_relu4 = h+ h = self.slice5(h)+ h_relu5 = h+ alexnet_outputs = namedtuple("AlexnetOutputs", ["relu1", "relu2", "relu3", "relu4", "relu5"])+ out = alexnet_outputs(h_relu1, h_relu2, h_relu3, h_relu4, h_relu5)++ return out+++class vgg16(torch.nn.Module):+ def __init__(self, requires_grad=False, pretrained=True):+ super(vgg16, self).__init__()+ vgg_pretrained_features = tv.vgg16(pretrained=pretrained).features+ self.slice1 = torch.nn.Sequential()+ self.slice2 = torch.nn.Sequential()+ self.slice3 = torch.nn.Sequential()+ self.slice4 = torch.nn.Sequential()+ self.slice5 = torch.nn.Sequential()+ self.N_slices = 5+ for x in range(4):+ self.slice1.add_module(str(x), vgg_pretrained_features[x])+ for x in range(4, 9):+ self.slice2.add_module(str(x), vgg_pretrained_features[x])+ for x in range(9, 16):+ self.slice3.add_module(str(x), vgg_pretrained_features[x])+ for x in range(16, 23):+ self.slice4.add_module(str(x), vgg_pretrained_features[x])+ for x in range(23, 30):+ self.slice5.add_module(str(x), vgg_pretrained_features[x])+ if not requires_grad:+ for param in self.parameters():+ param.requires_grad = False++ def forward(self, X):+ h = self.slice1(X)+ h_relu1_2 = h+ h = self.slice2(h)+ h_relu2_2 = h+ h = self.slice3(h)+ h_relu3_3 = h+ h = self.slice4(h)+ h_relu4_3 = h+ h = self.slice5(h)+ h_relu5_3 = h+ vgg_outputs = namedtuple("VggOutputs", ["relu1_2", "relu2_2", "relu3_3", "relu4_3", "relu5_3"])+ out = vgg_outputs(h_relu1_2, h_relu2_2, h_relu3_3, h_relu4_3, h_relu5_3)++ return out+++class resnet(torch.nn.Module):+ def __init__(self, requires_grad=False, pretrained=True, num=18):+ super(resnet, self).__init__()+ if num == 18:+ self.net = tv.resnet18(pretrained=pretrained)+ elif num == 34:+ self.net = tv.resnet34(pretrained=pretrained)+ elif num == 50:+ self.net = tv.resnet50(pretrained=pretrained)+ elif num == 101:+ self.net = tv.resnet101(pretrained=pretrained)+ elif num == 152:+ self.net = tv.resnet152(pretrained=pretrained)+ self.N_slices = 5++ self.conv1 = self.net.conv1+ self.bn1 = self.net.bn1+ self.relu = self.net.relu+ self.maxpool = self.net.maxpool+ self.layer1 = self.net.layer1+ self.layer2 = self.net.layer2+ self.layer3 = self.net.layer3+ self.layer4 = self.net.layer4++ def forward(self, X):+ h = self.conv1(X)+ h = self.bn1(h)+ h = self.relu(h)+ h_relu1 = h+ h = self.maxpool(h)+ h = self.layer1(h)+ h_conv2 = h+ h = self.layer2(h)+ h_conv3 = h+ h = self.layer3(h)+ h_conv4 = h+ h = self.layer4(h)+ h_conv5 = h++ outputs = namedtuple("Outputs", ["relu1", "conv2", "conv3", "conv4", "conv5"])+ out = outputs(h_relu1, h_conv2, h_conv3, h_conv4, h_conv5)++ return outdiff --git a/eval/vae/flolpips/pwcnet.py b/eval/vae/flolpips/pwcnet.pyindex 374f18e..3abc0ed 100644--- a/eval/vae/flolpips/pwcnet.py+++ b/eval/vae/flolpips/pwcnet.py@@ -1,344 +1,472 @@-#!/usr/bin/env python--import torch--import getopt-import math-import numpy-import os-import PIL-import PIL.Image-import sys--# try:-from .correlation import correlation # the custom cost volume layer-# except:-# sys.path.insert(0, './correlation'); import correlation # you should consider upgrading python-# end--##########################################################--# assert(int(str('').join(torch.__version__.split('.')[0:2])) >= 13) # requires at least pytorch version 1.3.0--# torch.set_grad_enabled(False) # make sure to not compute gradients for computational performance--# torch.backends.cudnn.enabled = True # make sure to use cudnn for computational performance--# ##########################################################--# arguments_strModel = 'default' # 'default', or 'chairs-things'-# arguments_strFirst = './images/first.png'-# arguments_strSecond = './images/second.png'-# arguments_strOut = './out.flo'--# for strOption, strArgument in getopt.getopt(sys.argv[1:], '', [ strParameter[2:] + '=' for strParameter in sys.argv[1::2] ])[0]:-# if strOption == '--model' and strArgument != '': arguments_strModel = strArgument # which model to use-# if strOption == '--first' and strArgument != '': arguments_strFirst = strArgument # path to the first frame-# if strOption == '--second' and strArgument != '': arguments_strSecond = strArgument # path to the second frame-# if strOption == '--out' and strArgument != '': arguments_strOut = strArgument # path to where the output should be stored-# end--##########################################################----def backwarp(tenInput, tenFlow):- backwarp_tenGrid = {}- backwarp_tenPartial = {}- if str(tenFlow.shape) not in backwarp_tenGrid:- tenHor = torch.linspace(-1.0 + (1.0 / tenFlow.shape[3]), 1.0 - (1.0 / tenFlow.shape[3]), tenFlow.shape[3]).view(1, 1, 1, -1).expand(-1, -1, tenFlow.shape[2], -1)- tenVer = torch.linspace(-1.0 + (1.0 / tenFlow.shape[2]), 1.0 - (1.0 / tenFlow.shape[2]), tenFlow.shape[2]).view(1, 1, -1, 1).expand(-1, -1, -1, tenFlow.shape[3])-- backwarp_tenGrid[str(tenFlow.shape)] = torch.cat([ tenHor, tenVer ], 1).cuda()- # end-- if str(tenFlow.shape) not in backwarp_tenPartial:- backwarp_tenPartial[str(tenFlow.shape)] = tenFlow.new_ones([ tenFlow.shape[0], 1, tenFlow.shape[2], tenFlow.shape[3] ])- # end-- tenFlow = torch.cat([ tenFlow[:, 0:1, :, :] / ((tenInput.shape[3] - 1.0) / 2.0), tenFlow[:, 1:2, :, :] / ((tenInput.shape[2] - 1.0) / 2.0) ], 1)- tenInput = torch.cat([ tenInput, backwarp_tenPartial[str(tenFlow.shape)] ], 1)-- tenOutput = torch.nn.functional.grid_sample(input=tenInput, grid=(backwarp_tenGrid[str(tenFlow.shape)] + tenFlow).permute(0, 2, 3, 1), mode='bilinear', padding_mode='zeros', align_corners=False)-- tenMask = tenOutput[:, -1:, :, :]; tenMask[tenMask > 0.999] = 1.0; tenMask[tenMask < 1.0] = 0.0-- return tenOutput[:, :-1, :, :] * tenMask-# end--##########################################################--class Network(torch.nn.Module):- def __init__(self):- super(Network, self).__init__()-- class Extractor(torch.nn.Module):- def __init__(self):- super(Extractor, self).__init__()-- self.netOne = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=2, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netTwo = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=2, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netThr = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netFou = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=64, out_channels=96, kernel_size=3, stride=2, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netFiv = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=96, out_channels=128, kernel_size=3, stride=2, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netSix = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=128, out_channels=196, kernel_size=3, stride=2, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )- # end-- def forward(self, tenInput):- tenOne = self.netOne(tenInput)- tenTwo = self.netTwo(tenOne)- tenThr = self.netThr(tenTwo)- tenFou = self.netFou(tenThr)- tenFiv = self.netFiv(tenFou)- tenSix = self.netSix(tenFiv)-- return [ tenOne, tenTwo, tenThr, tenFou, tenFiv, tenSix ]- # end- # end-- class Decoder(torch.nn.Module):- def __init__(self, intLevel):- super(Decoder, self).__init__()-- intPrevious = [ None, None, 81 + 32 + 2 + 2, 81 + 64 + 2 + 2, 81 + 96 + 2 + 2, 81 + 128 + 2 + 2, 81, None ][intLevel + 1]- intCurrent = [ None, None, 81 + 32 + 2 + 2, 81 + 64 + 2 + 2, 81 + 96 + 2 + 2, 81 + 128 + 2 + 2, 81, None ][intLevel + 0]-- if intLevel < 6: self.netUpflow = torch.nn.ConvTranspose2d(in_channels=2, out_channels=2, kernel_size=4, stride=2, padding=1)- if intLevel < 6: self.netUpfeat = torch.nn.ConvTranspose2d(in_channels=intPrevious + 128 + 128 + 96 + 64 + 32, out_channels=2, kernel_size=4, stride=2, padding=1)- if intLevel < 6: self.fltBackwarp = [ None, None, None, 5.0, 2.5, 1.25, 0.625, None ][intLevel + 1]-- self.netOne = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=intCurrent, out_channels=128, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netTwo = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=intCurrent + 128, out_channels=128, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netThr = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128, out_channels=96, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netFou = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128 + 96, out_channels=64, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netFiv = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128 + 96 + 64, out_channels=32, kernel_size=3, stride=1, padding=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)- )-- self.netSix = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128 + 96 + 64 + 32, out_channels=2, kernel_size=3, stride=1, padding=1)- )- # end-- def forward(self, tenFirst, tenSecond, objPrevious):- tenFlow = None- tenFeat = None-- if objPrevious is None:- tenFlow = None- tenFeat = None-- tenVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond), negative_slope=0.1, inplace=False)-- tenFeat = torch.cat([ tenVolume ], 1)-- elif objPrevious is not None:- tenFlow = self.netUpflow(objPrevious['tenFlow'])- tenFeat = self.netUpfeat(objPrevious['tenFeat'])-- tenVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=backwarp(tenInput=tenSecond, tenFlow=tenFlow * self.fltBackwarp)), negative_slope=0.1, inplace=False)-- tenFeat = torch.cat([ tenVolume, tenFirst, tenFlow, tenFeat ], 1)-- # end-- tenFeat = torch.cat([ self.netOne(tenFeat), tenFeat ], 1)- tenFeat = torch.cat([ self.netTwo(tenFeat), tenFeat ], 1)- tenFeat = torch.cat([ self.netThr(tenFeat), tenFeat ], 1)- tenFeat = torch.cat([ self.netFou(tenFeat), tenFeat ], 1)- tenFeat = torch.cat([ self.netFiv(tenFeat), tenFeat ], 1)-- tenFlow = self.netSix(tenFeat)-- return {- 'tenFlow': tenFlow,- 'tenFeat': tenFeat- }- # end- # end-- class Refiner(torch.nn.Module):- def __init__(self):- super(Refiner, self).__init__()-- self.netMain = torch.nn.Sequential(- torch.nn.Conv2d(in_channels=81 + 32 + 2 + 2 + 128 + 128 + 96 + 64 + 32, out_channels=128, kernel_size=3, stride=1, padding=1, dilation=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=2, dilation=2),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=4, dilation=4),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=128, out_channels=96, kernel_size=3, stride=1, padding=8, dilation=8),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=96, out_channels=64, kernel_size=3, stride=1, padding=16, dilation=16),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=64, out_channels=32, kernel_size=3, stride=1, padding=1, dilation=1),- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),- torch.nn.Conv2d(in_channels=32, out_channels=2, kernel_size=3, stride=1, padding=1, dilation=1)- )- # end-- def forward(self, tenInput):- return self.netMain(tenInput)- # end- # end-- self.netExtractor = Extractor()-- self.netTwo = Decoder(2)- self.netThr = Decoder(3)- self.netFou = Decoder(4)- self.netFiv = Decoder(5)- self.netSix = Decoder(6)-- self.netRefiner = Refiner()-- self.load_state_dict({ strKey.replace('module', 'net'): tenWeight for strKey, tenWeight in torch.hub.load_state_dict_from_url(url='http://content.sniklaus.com/github/pytorch-pwc/network-' + 'default' + '.pytorch').items() })- # end-- def forward(self, tenFirst, tenSecond):- intWidth = tenFirst.shape[3]- intHeight = tenFirst.shape[2]-- intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))- intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))-- tenPreprocessedFirst = torch.nn.functional.interpolate(input=tenFirst, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)- tenPreprocessedSecond = torch.nn.functional.interpolate(input=tenSecond, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)-- tenFirst = self.netExtractor(tenPreprocessedFirst)- tenSecond = self.netExtractor(tenPreprocessedSecond)--- objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)- objEstimate = self.netFiv(tenFirst[-2], tenSecond[-2], objEstimate)- objEstimate = self.netFou(tenFirst[-3], tenSecond[-3], objEstimate)- objEstimate = self.netThr(tenFirst[-4], tenSecond[-4], objEstimate)- objEstimate = self.netTwo(tenFirst[-5], tenSecond[-5], objEstimate)-- tenFlow = objEstimate['tenFlow'] + self.netRefiner(objEstimate['tenFeat'])- tenFlow = 20.0 * torch.nn.functional.interpolate(input=tenFlow, size=(intHeight, intWidth), mode='bilinear', align_corners=False)- tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)- tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)-- return tenFlow- # end-# end--netNetwork = None--##########################################################--def estimate(tenFirst, tenSecond):- global netNetwork-- if netNetwork is None:- netNetwork = Network().cuda().eval()- # end-- assert(tenFirst.shape[1] == tenSecond.shape[1])- assert(tenFirst.shape[2] == tenSecond.shape[2])-- intWidth = tenFirst.shape[2]- intHeight = tenFirst.shape[1]-- assert(intWidth == 1024) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue- assert(intHeight == 436) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue-- tenPreprocessedFirst = tenFirst.cuda().view(1, 3, intHeight, intWidth)- tenPreprocessedSecond = tenSecond.cuda().view(1, 3, intHeight, intWidth)-- intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))- intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))-- tenPreprocessedFirst = torch.nn.functional.interpolate(input=tenPreprocessedFirst, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)- tenPreprocessedSecond = torch.nn.functional.interpolate(input=tenPreprocessedSecond, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)-- tenFlow = 20.0 * torch.nn.functional.interpolate(input=netNetwork(tenPreprocessedFirst, tenPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)-- tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)- tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)-- return tenFlow[0, :, :, :].cpu()-# end--##########################################################--# if __name__ == '__main__':-# tenFirst = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strFirst))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))-# tenSecond = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strSecond))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))--# tenOutput = estimate(tenFirst, tenSecond)--# objOutput = open(arguments_strOut, 'wb')--# numpy.array([ 80, 73, 69, 72 ], numpy.uint8).tofile(objOutput)-# numpy.array([ tenOutput.shape[2], tenOutput.shape[1] ], numpy.int32).tofile(objOutput)-# numpy.array(tenOutput.numpy().transpose(1, 2, 0), numpy.float32).tofile(objOutput)--# objOutput.close()-# end\ No newline at end of file+#!/usr/bin/env python++import math++import torch++# try:+from .correlation import correlation # the custom cost volume layer++# except:+# sys.path.insert(0, './correlation'); import correlation # you should consider upgrading python+# end++##########################################################++# assert(int(str('').join(torch.__version__.split('.')[0:2])) >= 13) # requires at least pytorch version 1.3.0++# torch.set_grad_enabled(False) # make sure to not compute gradients for computational performance++# torch.backends.cudnn.enabled = True # make sure to use cudnn for computational performance++# ##########################################################++# arguments_strModel = 'default' # 'default', or 'chairs-things'+# arguments_strFirst = './images/first.png'+# arguments_strSecond = './images/second.png'+# arguments_strOut = './out.flo'++# for strOption, strArgument in getopt.getopt(sys.argv[1:], '', [ strParameter[2:] + '=' for strParameter in sys.argv[1::2] ])[0]:+# if strOption == '--model' and strArgument != '': arguments_strModel = strArgument # which model to use+# if strOption == '--first' and strArgument != '': arguments_strFirst = strArgument # path to the first frame+# if strOption == '--second' and strArgument != '': arguments_strSecond = strArgument # path to the second frame+# if strOption == '--out' and strArgument != '': arguments_strOut = strArgument # path to where the output should be stored+# end++##########################################################+++def backwarp(tenInput, tenFlow):+ backwarp_tenGrid = {}+ backwarp_tenPartial = {}+ if str(tenFlow.shape) not in backwarp_tenGrid:+ tenHor = (+ torch.linspace(-1.0 + (1.0 / tenFlow.shape[3]), 1.0 - (1.0 / tenFlow.shape[3]), tenFlow.shape[3])+ .view(1, 1, 1, -1)+ .expand(-1, -1, tenFlow.shape[2], -1)+ )+ tenVer = (+ torch.linspace(-1.0 + (1.0 / tenFlow.shape[2]), 1.0 - (1.0 / tenFlow.shape[2]), tenFlow.shape[2])+ .view(1, 1, -1, 1)+ .expand(-1, -1, -1, tenFlow.shape[3])+ )++ backwarp_tenGrid[str(tenFlow.shape)] = torch.cat([tenHor, tenVer], 1).cuda()+ # end++ if str(tenFlow.shape) not in backwarp_tenPartial:+ backwarp_tenPartial[str(tenFlow.shape)] = tenFlow.new_ones(+ [tenFlow.shape[0], 1, tenFlow.shape[2], tenFlow.shape[3]]+ )+ # end++ tenFlow = torch.cat(+ [+ tenFlow[:, 0:1, :, :] / ((tenInput.shape[3] - 1.0) / 2.0),+ tenFlow[:, 1:2, :, :] / ((tenInput.shape[2] - 1.0) / 2.0),+ ],+ 1,+ )+ tenInput = torch.cat([tenInput, backwarp_tenPartial[str(tenFlow.shape)]], 1)++ tenOutput = torch.nn.functional.grid_sample(+ input=tenInput,+ grid=(backwarp_tenGrid[str(tenFlow.shape)] + tenFlow).permute(0, 2, 3, 1),+ mode="bilinear",+ padding_mode="zeros",+ align_corners=False,+ )++ tenMask = tenOutput[:, -1:, :, :]+ tenMask[tenMask > 0.999] = 1.0+ tenMask[tenMask < 1.0] = 0.0++ return tenOutput[:, :-1, :, :] * tenMask+++# end++##########################################################+++class Network(torch.nn.Module):+ def __init__(self):+ super(Network, self).__init__()++ class Extractor(torch.nn.Module):+ def __init__(self):+ super(Extractor, self).__init__()++ self.netOne = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=2, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netTwo = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=2, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netThr = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netFou = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=64, out_channels=96, kernel_size=3, stride=2, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netFiv = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=96, out_channels=128, kernel_size=3, stride=2, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netSix = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=128, out_channels=196, kernel_size=3, stride=2, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ # end++ def forward(self, tenInput):+ tenOne = self.netOne(tenInput)+ tenTwo = self.netTwo(tenOne)+ tenThr = self.netThr(tenTwo)+ tenFou = self.netFou(tenThr)+ tenFiv = self.netFiv(tenFou)+ tenSix = self.netSix(tenFiv)++ return [tenOne, tenTwo, tenThr, tenFou, tenFiv, tenSix]++ # end++ # end++ class Decoder(torch.nn.Module):+ def __init__(self, intLevel):+ super(Decoder, self).__init__()++ intPrevious = [+ None,+ None,+ 81 + 32 + 2 + 2,+ 81 + 64 + 2 + 2,+ 81 + 96 + 2 + 2,+ 81 + 128 + 2 + 2,+ 81,+ None,+ ][intLevel + 1]+ intCurrent = [+ None,+ None,+ 81 + 32 + 2 + 2,+ 81 + 64 + 2 + 2,+ 81 + 96 + 2 + 2,+ 81 + 128 + 2 + 2,+ 81,+ None,+ ][intLevel + 0]++ if intLevel < 6:+ self.netUpflow = torch.nn.ConvTranspose2d(+ in_channels=2, out_channels=2, kernel_size=4, stride=2, padding=1+ )+ if intLevel < 6:+ self.netUpfeat = torch.nn.ConvTranspose2d(+ in_channels=intPrevious + 128 + 128 + 96 + 64 + 32,+ out_channels=2,+ kernel_size=4,+ stride=2,+ padding=1,+ )+ if intLevel < 6:+ self.fltBackwarp = [None, None, None, 5.0, 2.5, 1.25, 0.625, None][intLevel + 1]++ self.netOne = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=intCurrent, out_channels=128, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netTwo = torch.nn.Sequential(+ torch.nn.Conv2d(in_channels=intCurrent + 128, out_channels=128, kernel_size=3, stride=1, padding=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netThr = torch.nn.Sequential(+ torch.nn.Conv2d(+ in_channels=intCurrent + 128 + 128, out_channels=96, kernel_size=3, stride=1, padding=1+ ),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netFou = torch.nn.Sequential(+ torch.nn.Conv2d(+ in_channels=intCurrent + 128 + 128 + 96, out_channels=64, kernel_size=3, stride=1, padding=1+ ),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netFiv = torch.nn.Sequential(+ torch.nn.Conv2d(+ in_channels=intCurrent + 128 + 128 + 96 + 64,+ out_channels=32,+ kernel_size=3,+ stride=1,+ padding=1,+ ),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ )++ self.netSix = torch.nn.Sequential(+ torch.nn.Conv2d(+ in_channels=intCurrent + 128 + 128 + 96 + 64 + 32,+ out_channels=2,+ kernel_size=3,+ stride=1,+ padding=1,+ )+ )++ # end++ def forward(self, tenFirst, tenSecond, objPrevious):+ tenFlow = None+ tenFeat = None++ if objPrevious is None:+ tenFlow = None+ tenFeat = None++ tenVolume = torch.nn.functional.leaky_relu(+ input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond),+ negative_slope=0.1,+ inplace=False,+ )++ tenFeat = torch.cat([tenVolume], 1)++ elif objPrevious is not None:+ tenFlow = self.netUpflow(objPrevious["tenFlow"])+ tenFeat = self.netUpfeat(objPrevious["tenFeat"])++ tenVolume = torch.nn.functional.leaky_relu(+ input=correlation.FunctionCorrelation(+ tenFirst=tenFirst,+ tenSecond=backwarp(tenInput=tenSecond, tenFlow=tenFlow * self.fltBackwarp),+ ),+ negative_slope=0.1,+ inplace=False,+ )++ tenFeat = torch.cat([tenVolume, tenFirst, tenFlow, tenFeat], 1)++ # end++ tenFeat = torch.cat([self.netOne(tenFeat), tenFeat], 1)+ tenFeat = torch.cat([self.netTwo(tenFeat), tenFeat], 1)+ tenFeat = torch.cat([self.netThr(tenFeat), tenFeat], 1)+ tenFeat = torch.cat([self.netFou(tenFeat), tenFeat], 1)+ tenFeat = torch.cat([self.netFiv(tenFeat), tenFeat], 1)++ tenFlow = self.netSix(tenFeat)++ return {"tenFlow": tenFlow, "tenFeat": tenFeat}++ # end++ # end++ class Refiner(torch.nn.Module):+ def __init__(self):+ super(Refiner, self).__init__()++ self.netMain = torch.nn.Sequential(+ torch.nn.Conv2d(+ in_channels=81 + 32 + 2 + 2 + 128 + 128 + 96 + 64 + 32,+ out_channels=128,+ kernel_size=3,+ stride=1,+ padding=1,+ dilation=1,+ ),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=2, dilation=2),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=4, dilation=4),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=128, out_channels=96, kernel_size=3, stride=1, padding=8, dilation=8),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=96, out_channels=64, kernel_size=3, stride=1, padding=16, dilation=16),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=64, out_channels=32, kernel_size=3, stride=1, padding=1, dilation=1),+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),+ torch.nn.Conv2d(in_channels=32, out_channels=2, kernel_size=3, stride=1, padding=1, dilation=1),+ )++ # end++ def forward(self, tenInput):+ return self.netMain(tenInput)++ # end++ # end++ self.netExtractor = Extractor()++ self.netTwo = Decoder(2)+ self.netThr = Decoder(3)+ self.netFou = Decoder(4)+ self.netFiv = Decoder(5)+ self.netSix = Decoder(6)++ self.netRefiner = Refiner()++ self.load_state_dict(+ {+ strKey.replace("module", "net"): tenWeight+ for strKey, tenWeight in torch.hub.load_state_dict_from_url(+ url="http://content.sniklaus.com/github/pytorch-pwc/network-" + "default" + ".pytorch"+ ).items()+ }+ )++ # end++ def forward(self, tenFirst, tenSecond):+ intWidth = tenFirst.shape[3]+ intHeight = tenFirst.shape[2]++ intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))+ intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))++ tenPreprocessedFirst = torch.nn.functional.interpolate(+ input=tenFirst, size=(intPreprocessedHeight, intPreprocessedWidth), mode="bilinear", align_corners=False+ )+ tenPreprocessedSecond = torch.nn.functional.interpolate(+ input=tenSecond, size=(intPreprocessedHeight, intPreprocessedWidth), mode="bilinear", align_corners=False+ )++ tenFirst = self.netExtractor(tenPreprocessedFirst)+ tenSecond = self.netExtractor(tenPreprocessedSecond)++ objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)+ objEstimate = self.netFiv(tenFirst[-2], tenSecond[-2], objEstimate)+ objEstimate = self.netFou(tenFirst[-3], tenSecond[-3], objEstimate)+ objEstimate = self.netThr(tenFirst[-4], tenSecond[-4], objEstimate)+ objEstimate = self.netTwo(tenFirst[-5], tenSecond[-5], objEstimate)++ tenFlow = objEstimate["tenFlow"] + self.netRefiner(objEstimate["tenFeat"])+ tenFlow = 20.0 * torch.nn.functional.interpolate(+ input=tenFlow, size=(intHeight, intWidth), mode="bilinear", align_corners=False+ )+ tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)+ tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)++ return tenFlow++ # end+++# end++netNetwork = None++##########################################################+++def estimate(tenFirst, tenSecond):+ global netNetwork++ if netNetwork is None:+ netNetwork = Network().cuda().eval()+ # end++ assert tenFirst.shape[1] == tenSecond.shape[1]+ assert tenFirst.shape[2] == tenSecond.shape[2]++ intWidth = tenFirst.shape[2]+ intHeight = tenFirst.shape[1]++ assert (+ intWidth == 1024+ ) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue+ assert (+ intHeight == 436+ ) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue++ tenPreprocessedFirst = tenFirst.cuda().view(1, 3, intHeight, intWidth)+ tenPreprocessedSecond = tenSecond.cuda().view(1, 3, intHeight, intWidth)++ intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))+ intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))++ tenPreprocessedFirst = torch.nn.functional.interpolate(+ input=tenPreprocessedFirst,+ size=(intPreprocessedHeight, intPreprocessedWidth),+ mode="bilinear",+ align_corners=False,+ )+ tenPreprocessedSecond = torch.nn.functional.interpolate(+ input=tenPreprocessedSecond,+ size=(intPreprocessedHeight, intPreprocessedWidth),+ mode="bilinear",+ align_corners=False,+ )++ tenFlow = 20.0 * torch.nn.functional.interpolate(+ input=netNetwork(tenPreprocessedFirst, tenPreprocessedSecond),+ size=(intHeight, intWidth),+ mode="bilinear",+ align_corners=False,+ )++ tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)+ tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)++ return tenFlow[0, :, :, :].cpu()+++# end++##########################################################++# if __name__ == '__main__':+# tenFirst = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strFirst))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))+# tenSecond = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strSecond))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))++# tenOutput = estimate(tenFirst, tenSecond)++# objOutput = open(arguments_strOut, 'wb')++# numpy.array([ 80, 73, 69, 72 ], numpy.uint8).tofile(objOutput)+# numpy.array([ tenOutput.shape[2], tenOutput.shape[1] ], numpy.int32).tofile(objOutput)+# numpy.array(tenOutput.numpy().transpose(1, 2, 0), numpy.float32).tofile(objOutput)++# objOutput.close()+# enddiff --git a/eval/vae/flolpips/utils.py b/eval/vae/flolpips/utils.pyindex 98d4502..21a3504 100644--- a/eval/vae/flolpips/utils.py+++ b/eval/vae/flolpips/utils.py@@ -1,95 +1,107 @@-import numpy as np-import cv2-import torch---def normalize_tensor(in_feat,eps=1e-10):- norm_factor = torch.sqrt(torch.sum(in_feat**2,dim=1,keepdim=True))- return in_feat/(norm_factor+eps)--def l2(p0, p1, range=255.):- return .5*np.mean((p0 / range - p1 / range)**2)--def dssim(p0, p1, range=255.):- from skimage.measure import compare_ssim- return (1 - compare_ssim(p0, p1, data_range=range, multichannel=True)) / 2.--def tensor2im(image_tensor, imtype=np.uint8, cent=1., factor=255./2.):- image_numpy = image_tensor[0].cpu().float().numpy()- image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + cent) * factor- return image_numpy.astype(imtype)--def tensor2np(tensor_obj):- # change dimension of a tensor object into a numpy array- return tensor_obj[0].cpu().float().numpy().transpose((1,2,0))--def np2tensor(np_obj):- # change dimenion of np array into tensor array- return torch.Tensor(np_obj[:, :, :, np.newaxis].transpose((3, 2, 0, 1)))--def tensor2tensorlab(image_tensor,to_norm=True,mc_only=False):- # image tensor to lab tensor- from skimage import color-- img = tensor2im(image_tensor)- img_lab = color.rgb2lab(img)- if(mc_only):- img_lab[:,:,0] = img_lab[:,:,0]-50- if(to_norm and not mc_only):- img_lab[:,:,0] = img_lab[:,:,0]-50- img_lab = img_lab/100.-- return np2tensor(img_lab)--def read_frame_yuv2rgb(stream, width, height, iFrame, bit_depth, pix_fmt='420'):- if pix_fmt == '420':- multiplier = 1- uv_factor = 2- elif pix_fmt == '444':- multiplier = 2- uv_factor = 1- else:- print('Pixel format {} is not supported'.format(pix_fmt))- return-- if bit_depth == 8:- datatype = np.uint8- stream.seek(iFrame*1.5*width*height*multiplier)- Y = np.fromfile(stream, dtype=datatype, count=width*height).reshape((height, width))-- # read chroma samples and upsample since original is 4:2:0 sampling- U = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\- reshape((height//uv_factor, width//uv_factor))- V = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\- reshape((height//uv_factor, width//uv_factor))-- else:- datatype = np.uint16- stream.seek(iFrame*3*width*height*multiplier)- Y = np.fromfile(stream, dtype=datatype, count=width*height).reshape((height, width))-- U = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\- reshape((height//uv_factor, width//uv_factor))- V = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\- reshape((height//uv_factor, width//uv_factor))-- if pix_fmt == '420':- yuv = np.empty((height*3//2, width), dtype=datatype)- yuv[0:height,:] = Y-- yuv[height:height+height//4,:] = U.reshape(-1, width)- yuv[height+height//4:,:] = V.reshape(-1, width)-- if bit_depth != 8:- yuv = (yuv/(2**bit_depth-1)*255).astype(np.uint8)-- #convert to rgb- rgb = cv2.cvtColor(yuv, cv2.COLOR_YUV2RGB_I420)-- else:- yvu = np.stack([Y,V,U],axis=2)- if bit_depth != 8:- yvu = (yvu/(2**bit_depth-1)*255).astype(np.uint8)- rgb = cv2.cvtColor(yvu, cv2.COLOR_YCrCb2RGB)-- return rgb+import cv2+import numpy as np+import torch+++def normalize_tensor(in_feat, eps=1e-10):+ norm_factor = torch.sqrt(torch.sum(in_feat**2, dim=1, keepdim=True))+ return in_feat / (norm_factor + eps)+++def l2(p0, p1, range=255.0):+ return 0.5 * np.mean((p0 / range - p1 / range) ** 2)+++def dssim(p0, p1, range=255.0):+ from skimage.measure import compare_ssim++ return (1 - compare_ssim(p0, p1, data_range=range, multichannel=True)) / 2.0+++def tensor2im(image_tensor, imtype=np.uint8, cent=1.0, factor=255.0 / 2.0):+ image_numpy = image_tensor[0].cpu().float().numpy()+ image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + cent) * factor+ return image_numpy.astype(imtype)+++def tensor2np(tensor_obj):+ # change dimension of a tensor object into a numpy array+ return tensor_obj[0].cpu().float().numpy().transpose((1, 2, 0))+++def np2tensor(np_obj):+ # change dimenion of np array into tensor array+ return torch.Tensor(np_obj[:, :, :, np.newaxis].transpose((3, 2, 0, 1)))+++def tensor2tensorlab(image_tensor, to_norm=True, mc_only=False):+ # image tensor to lab tensor+ from skimage import color++ img = tensor2im(image_tensor)+ img_lab = color.rgb2lab(img)+ if mc_only:+ img_lab[:, :, 0] = img_lab[:, :, 0] - 50+ if to_norm and not mc_only:+ img_lab[:, :, 0] = img_lab[:, :, 0] - 50+ img_lab = img_lab / 100.0++ return np2tensor(img_lab)+++def read_frame_yuv2rgb(stream, width, height, iFrame, bit_depth, pix_fmt="420"):+ if pix_fmt == "420":+ multiplier = 1+ uv_factor = 2+ elif pix_fmt == "444":+ multiplier = 2+ uv_factor = 1+ else:+ print("Pixel format {} is not supported".format(pix_fmt))+ return++ if bit_depth == 8:+ datatype = np.uint8+ stream.seek(iFrame * 1.5 * width * height * multiplier)+ Y = np.fromfile(stream, dtype=datatype, count=width * height).reshape((height, width))++ # read chroma samples and upsample since original is 4:2:0 sampling+ U = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(+ (height // uv_factor, width // uv_factor)+ )+ V = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(+ (height // uv_factor, width // uv_factor)+ )++ else:+ datatype = np.uint16+ stream.seek(iFrame * 3 * width * height * multiplier)+ Y = np.fromfile(stream, dtype=datatype, count=width * height).reshape((height, width))++ U = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(+ (height // uv_factor, width // uv_factor)+ )+ V = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(+ (height // uv_factor, width // uv_factor)+ )++ if pix_fmt == "420":+ yuv = np.empty((height * 3 // 2, width), dtype=datatype)+ yuv[0:height, :] = Y++ yuv[height : height + height // 4, :] = U.reshape(-1, width)+ yuv[height + height // 4 :, :] = V.reshape(-1, width)++ if bit_depth != 8:+ yuv = (yuv / (2**bit_depth - 1) * 255).astype(np.uint8)++ # convert to rgb+ rgb = cv2.cvtColor(yuv, cv2.COLOR_YUV2RGB_I420)++ else:+ yvu = np.stack([Y, V, U], axis=2)+ if bit_depth != 8:+ yvu = (yvu / (2**bit_depth - 1) * 255).astype(np.uint8)+ rgb = cv2.cvtColor(yvu, cv2.COLOR_YCrCb2RGB)++ return rgbdiff --git a/eval/vae/script/eval.sh b/eval/vae/script/eval.shindex 897166e..3716719 100644--- a/eval/vae/script/eval.sh+++ b/eval/vae/script/eval.sh@@ -9,4 +9,4 @@ python eval/eval_common_metric.py \--num_frames 17 \--sample_rate 1 \--subset_size 100 \- --metric ssim psnr lpips flolpips\ No newline at end of file+ --metric ssim psnr lpips flolpipsdiff --git a/eval/vbench/calc_vbench.py b/eval/vbench/calc_vbench.pyindex b2505bf..b456f28 100644--- a/eval/vbench/calc_vbench.py+++ b/eval/vbench/calc_vbench.py@@ -1,9 +1,11 @@import argparseimport os-from vbench import VBench-import torchimport time+import torch++from vbench import VBench+full_info_path = "eval/vbench/VBench_full_info.json"dimensions = [# Quality Score@@ -26,12 +28,13 @@ dimensions = ["appearance_style",]+def parse_args():parser = argparse.ArgumentParser()parser.add_argument("video_folder", type=str) # samples/samples..._vbench/evalparser.add_argument("model_ckpt", type=str)- parser.add_argument("--start", type=int, default=0) # start index of dimension to be evaluated- parser.add_argument("--end", type=int, default=-1) # start index of dimension to be evaluated+ parser.add_argument("--start", type=int, default=0) # start index of dimension to be evaluated+ parser.add_argument("--end", type=int, default=-1) # start index of dimension to be evaluatedargs = parser.parse_args()return args@@ -44,23 +47,23 @@ if __name__ == "__main__":video_path = args.video_folderkwargs = {}- kwargs['imaging_quality_preprocessing_mode'] = 'longer' # use VBench/evaluate.py default+ kwargs["imaging_quality_preprocessing_mode"] = "longer" # use VBench/evaluate.py defaultstart_time = time.time()# NOTE: important to use torch.device("cuda"), else will have issue with object_class third_party modulemy_VBench = VBench(torch.device("cuda"), full_info_path, output_dir)- if args.end == -1: # adjust end accordingly+ if args.end == -1: # adjust end accordinglyargs.end = len(dimensions)- for dim in dimensions[args.start:args.end]:+ for dim in dimensions[args.start : args.end]:my_VBench.evaluate(videos_path=video_path,name=dim,local=False,read_frame=False,dimension_list=[dim],- mode='vbench_standard',- **kwargs+ mode="vbench_standard",+ **kwargs,)- print("Runtime: %s seconds " % (time.time() - start_time))\ No newline at end of file+ print("Runtime: %s seconds " % (time.time() - start_time))diff --git a/gradio/README.md b/gradio/README.mdindex 609bd7c..5dddd47 100644--- a/gradio/README.md+++ b/gradio/README.md@@ -11,7 +11,7 @@ license: apache-2.0preload_from_hub:- hpcai-tech/OpenSora-STDiT-v3- hpcai-tech/OpenSora-VAE-v1.2- - DeepFloyd/t5-v1_1-xxl+ - DeepFloyd/t5-v1_1-xxl---diff --git a/gradio/app.py b/gradio/app.pyindex 887e1e2..d488c2a 100644--- a/gradio/app.py+++ b/gradio/app.py@@ -7,27 +7,25 @@ Usage:"""import argparse+import datetimeimport importlibimport osimport subprocessimport sys+from tempfile import NamedTemporaryFile+import spacesimport torchimport gradio as gr-from tempfile import NamedTemporaryFile-import datetime--MODEL_TYPES = ["v1.2-stage3"]WATERMARK_PATH = "./assets/images/watermark/watermark.png"CONFIG_MAP = {"v1.2-stage3": "configs/opensora-v1-2/inference/sample.py",}-HF_STDIT_MAP = {- "v1.2-stage3": "hpcai-tech/OpenSora-STDiT-v3"-}+HF_STDIT_MAP = {"v1.2-stage3": "hpcai-tech/OpenSora-STDiT-v3"}+# ============================# Prepare Runtime Environment@@ -103,6 +101,7 @@ def build_models(model_type, config, enable_optimization=False):# we load model from HuggingFace directly so that we don't need to# handle model download logic in HuggingFace Spacefrom opensora.models.stdit.stdit3 import STDiT3+stdit = STDiT3.from_pretrained(HF_STDIT_MAP[model_type])stdit = stdit.cuda()@@ -169,39 +168,59 @@ install_dependencies(enable_optimization=args.enable_optimization)# import after installationfrom opensora.datasets import IMG_FPS, save_sample-from opensora.utils.misc import to_torch_dtype+from opensora.datasets.aspect import get_image_size, get_num_frames+from opensora.models.text_encoder.t5 import text_preprocessingfrom opensora.utils.inference_utils import (+ add_watermark,append_generated,+ append_score_to_prompts,apply_mask_strategy,collect_references_batch,+ dframe_to_frame,extract_json_from_prompts,extract_prompts_loop,- prepare_multi_resolution_info,- dframe_to_frame,- append_score_to_prompts,+ get_random_prompt_by_openai,has_openai_key,+ merge_prompt,+ prepare_multi_resolution_info,refine_prompts_by_openai,- add_watermark,- get_random_prompt_by_openai,split_prompt,- merge_prompt)-from opensora.models.text_encoder.t5 import text_preprocessing-from opensora.datasets.aspect import get_image_size, get_num_frames+from opensora.utils.misc import to_torch_dtype# some global variablesdtype = to_torch_dtype(config.dtype)device = torch.device("cuda")# build model-vae, text_encoder, stdit, scheduler = build_models(args.model_type, config, enable_optimization=args.enable_optimization)+vae, text_encoder, stdit, scheduler = build_models(+ args.model_type, config, enable_optimization=args.enable_optimization+)-def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_strength, aesthetic_score, use_motion_strength, use_aesthetic_score, camera_motion, reference_image, refine_prompt, fps, num_loop, seed, sampling_steps, cfg_scale):+def run_inference(+ mode,+ prompt_text,+ resolution,+ aspect_ratio,+ length,+ motion_strength,+ aesthetic_score,+ use_motion_strength,+ use_aesthetic_score,+ camera_motion,+ reference_image,+ refine_prompt,+ fps,+ num_loop,+ seed,+ sampling_steps,+ cfg_scale,+):if prompt_text is None or prompt_text == "":gr.Warning("Your prompt is empty, please enter a valid prompt")return None-+torch.manual_seed(seed)with torch.inference_mode():# ======================@@ -210,7 +229,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st# parse the inputs# frame_interval must be 1 so we ignore it hereimage_size = get_image_size(resolution, aspect_ratio)-+# compute generation parametersif mode == "Text2Image":num_frames = 1@@ -218,26 +237,26 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_stelse:num_frames = config.num_framesnum_frames = get_num_frames(length)-+condition_frame_length = int(num_frames / 17 * 5 / 3)condition_frame_edit = 0.0-+input_size = (num_frames, *image_size)latent_size = vae.get_latent_size(input_size)multi_resolution = "OpenSora"align = 5-+# == prepare mask strategy ==if mode == "Text2Image":mask_strategy = [None]elif mode == "Text2Video":if reference_image is not None:- mask_strategy = ['0']+ mask_strategy = ["0"]else:mask_strategy = [None]else:raise ValueError(f"Invalid mode: {mode}")-+# == prepare reference ==if mode == "Text2Image":refs = [""]@@ -245,6 +264,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_stif reference_image is not None:# save image to diskfrom PIL import Image+im = Image.fromarray(reference_image)temp_file = NamedTemporaryFile(suffix=".png")im.save(temp_file.name)@@ -253,7 +273,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_strefs = [""]else:raise ValueError(f"Invalid mode: {mode}")-+# == get json from prompts ==batch_prompts = [prompt_text]batch_prompts, refs, mask_strategy = extract_json_from_prompts(batch_prompts, refs, mask_strategy)@@ -265,17 +285,17 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_stmodel_args = prepare_multi_resolution_info(multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype)-- # == process prompts step by step ==++ # == process prompts step by step ==# 0. split prompt# each element in the list is [prompt_segment_list, loop_idx_list]batched_prompt_segment_list = []batched_loop_idx_list = []for prompt in batch_prompts:- prompt_segment_list, loop_idx_list = split_prompt(prompt)+ prompt_segment_list, loop_idx_list = split_prompt(prompt)batched_prompt_segment_list.append(prompt_segment_list)batched_loop_idx_list.append(loop_idx_list)-+# 1. refine prompt by openaiif refine_prompt:# check if openai key is provided@@ -284,7 +304,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_stelse:for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):batched_prompt_segment_list[idx] = refine_prompts_by_openai(prompt_segment_list)-+# process scoresaesthetic_score = aesthetic_score if use_aesthetic_score else Nonemotion_strength = motion_strength if use_motion_strength and mode != "Text2Image" else None@@ -301,48 +321,39 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st# 3. clean prompt with T5for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):batched_prompt_segment_list[idx] = [text_preprocessing(prompt) for prompt in prompt_segment_list]-+# 4. merge to obtain the final promptbatch_prompts = []for prompt_segment_list, loop_idx_list in zip(batched_prompt_segment_list, batched_loop_idx_list):batch_prompts.append(merge_prompt(prompt_segment_list, loop_idx_list))-# =========================# Generate image/video# =========================video_clips = []-+for loop_i in range(num_loop):# 4.4 sample in hidden spacebatch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)-+# == loop ==if loop_i > 0:refs, mask_strategy = append_generated(- vae,- video_clips[-1],- refs,- mask_strategy,- loop_i,- condition_frame_length,- condition_frame_edit- )-+ vae, video_clips[-1], refs, mask_strategy, loop_i, condition_frame_length, condition_frame_edit+ )+# == sampling ==z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)masks = apply_mask_strategy(z, refs, mask_strategy, loop_i, align=align)-+# 4.6. diffusion sampling# hack to update num_sampling_steps and cfg_scalescheduler_kwargs = config.scheduler.copy()- scheduler_kwargs.pop('type')- scheduler_kwargs['num_sampling_steps'] = sampling_steps- scheduler_kwargs['cfg_scale'] = cfg_scale+ scheduler_kwargs.pop("type")+ scheduler_kwargs["num_sampling_steps"] = sampling_steps+ scheduler_kwargs["cfg_scale"] = cfg_scale- scheduler.__init__(- **scheduler_kwargs- )+ scheduler.__init__(**scheduler_kwargs)samples = scheduler.sample(stdit,text_encoder,@@ -355,7 +366,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st)samples = vae.decode(samples.to(dtype), num_frames=num_frames)video_clips.append(samples)-+# =========================# Save output# =========================@@ -368,7 +379,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_stsave_path = os.path.join(args.output, f"output_{timestamp}")saved_path = save_sample(video, save_path=save_path, fps=24)torch.cuda.empty_cache()-+# add watermark# all watermarked videos should have a _watermarked suffixif mode != "Text2Image" and os.path.exists(WATERMARK_PATH):@@ -380,29 +391,30 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_streturn saved_pathelse:return saved_path--++@spaces.GPU(duration=200)def run_image_inference(- prompt_text,- resolution,- aspect_ratio,- length,- motion_strength,- aesthetic_score,- use_motion_strength,+ prompt_text,+ resolution,+ aspect_ratio,+ length,+ motion_strength,+ aesthetic_score,+ use_motion_strength,use_aesthetic_score,camera_motion,reference_image,refine_prompt,fps,- num_loop,+ num_loop,seed,sampling_steps,- cfg_scale):+ cfg_scale,+):return run_inference(- "Text2Image",- prompt_text,+ "Text2Image",+ prompt_text,resolution,aspect_ratio,length,@@ -414,10 +426,12 @@ def run_image_inference(reference_image,refine_prompt,fps,- num_loop,+ num_loop,seed,sampling_steps,- cfg_scale)+ cfg_scale,+ )+@spaces.GPU(duration=200)def run_video_inference(@@ -428,38 +442,39 @@ def run_video_inference(motion_strength,aesthetic_score,use_motion_strength,- use_aesthetic_score,+ use_aesthetic_score,camera_motion,- reference_image,+ reference_image,refine_prompt,fps,- num_loop,+ num_loop,seed,sampling_steps,- cfg_scale):+ cfg_scale,+):# if (resolution == "480p" and length == "16s") or \# (resolution == "720p" and length in ["8s", "16s"]):# gr.Warning("Generation is interrupted as the combination of 480p and 16s will lead to CUDA out of memory")# else:return run_inference("Text2Video",- prompt_text,+ prompt_text,resolution,- aspect_ratio,- length,- motion_strength,- aesthetic_score,+ aspect_ratio,+ length,+ motion_strength,+ aesthetic_score,use_motion_strength,- use_aesthetic_score,+ use_aesthetic_score,camera_motion,- reference_image,+ reference_image,refine_prompt,fps,- num_loop,+ num_loop,seed,- sampling_steps,- cfg_scale- )+ sampling_steps,+ cfg_scale,+ )def generate_random_prompt():@@ -498,56 +513,34 @@ def main():with gr.Row():with gr.Column():- prompt_text = gr.Textbox(- label="Prompt",- placeholder="Describe your video here",- lines=4- )+ prompt_text = gr.Textbox(label="Prompt", placeholder="Describe your video here", lines=4)refine_prompt = gr.Checkbox(value=True, label="Refine prompt with GPT4o")random_prompt_btn = gr.Button("Random Prompt By GPT4o")-+gr.Markdown("## Basic Settings")resolution = gr.Radio(- choices=["144p", "240p", "360p", "480p", "720p"],- value="480p",- label="Resolution",+ choices=["144p", "240p", "360p", "480p", "720p"],+ value="480p",+ label="Resolution",)aspect_ratio = gr.Radio(- choices=["9:16", "16:9", "3:4", "4:3", "1:1"],- value="9:16",- label="Aspect Ratio (H:W)",+ choices=["9:16", "16:9", "3:4", "4:3", "1:1"],+ value="9:16",+ label="Aspect Ratio (H:W)",)length = gr.Radio(- choices=["2s", "4s", "8s", "16s"],+ choices=["2s", "4s", "8s", "16s"],value="2s",- label="Video Length",- info="only effective for video generation, 8s may fail as Hugging Face ZeroGPU has the limitation of max 200 seconds inference time."+ label="Video Length",+ info="only effective for video generation, 8s may fail as Hugging Face ZeroGPU has the limitation of max 200 seconds inference time.",)with gr.Row():- seed = gr.Slider(- value=1024,- minimum=1,- maximum=2048,- step=1,- label="Seed"- )+ seed = gr.Slider(value=1024, minimum=1, maximum=2048, step=1, label="Seed")++ sampling_steps = gr.Slider(value=30, minimum=1, maximum=200, step=1, label="Sampling steps")+ cfg_scale = gr.Slider(value=7.0, minimum=0.0, maximum=10.0, step=0.1, label="CFG Scale")- sampling_steps = gr.Slider(- value=30,- minimum=1,- maximum=200,- step=1,- label="Sampling steps"- )- cfg_scale = gr.Slider(- value=7.0,- minimum=0.0,- maximum=10.0,- step=0.1,- label="CFG Scale"- )-with gr.Row():with gr.Column():motion_strength = gr.Slider(@@ -556,10 +549,10 @@ def main():maximum=100,step=1,label="Motion Strength",- info="only effective for video generation"+ info="only effective for video generation",)use_motion_strength = gr.Checkbox(value=False, label="Enable")-+with gr.Column():aesthetic_score = gr.Slider(value=6.5,@@ -567,26 +560,17 @@ def main():maximum=7,step=0.1,label="Aesthetic",- info="effective for text & video generation"+ info="effective for text & video generation",)use_aesthetic_score = gr.Checkbox(value=True, label="Enable")-+camera_motion = gr.Radio(value="none",label="Camera Motion",- choices=[- "none",- "pan right",- "pan left",- "tilt up",- "tilt down",- "zoom in",- "zoom out",- "static"- ],- interactive=True+ choices=["none", "pan right", "pan left", "tilt up", "tilt down", "zoom in", "zoom out", "static"],+ interactive=True,)-+gr.Markdown("## Advanced Settings")with gr.Row():fps = gr.Slider(@@ -595,7 +579,7 @@ def main():maximum=60,step=1,label="FPS",- info="This is the frames per seconds for video generation, keep it to 24 if you are not sure"+ info="This is the frames per seconds for video generation, keep it to 24 if you are not sure",)num_loop = gr.Slider(value=1,@@ -603,41 +587,64 @@ def main():maximum=20,step=1,label="Number of Loops",- info="This will change the length of the generated video, keep it to 1 if you are not sure"+ info="This will change the length of the generated video, keep it to 1 if you are not sure",)--+gr.Markdown("## Reference Image")- reference_image = gr.Image(- label="Image (optional)",- show_download_button=True- )-+ reference_image = gr.Image(label="Image (optional)", show_download_button=True)+with gr.Column():- output_video = gr.Video(- label="Output Video",- height="100%"- )+ output_video = gr.Video(label="Output Video", height="100%")with gr.Row():- image_gen_button = gr.Button("Generate image")- video_gen_button = gr.Button("Generate video")-+ image_gen_button = gr.Button("Generate image")+ video_gen_button = gr.Button("Generate video")image_gen_button.click(- fn=run_image_inference,- inputs=[prompt_text, resolution, aspect_ratio, length, motion_strength, aesthetic_score, use_motion_strength, use_aesthetic_score, camera_motion, reference_image, refine_prompt, fps, num_loop, seed, sampling_steps, cfg_scale],- outputs=reference_image- )+ fn=run_image_inference,+ inputs=[+ prompt_text,+ resolution,+ aspect_ratio,+ length,+ motion_strength,+ aesthetic_score,+ use_motion_strength,+ use_aesthetic_score,+ camera_motion,+ reference_image,+ refine_prompt,+ fps,+ num_loop,+ seed,+ sampling_steps,+ cfg_scale,+ ],+ outputs=reference_image,+ )video_gen_button.click(- fn=run_video_inference,- inputs=[prompt_text, resolution, aspect_ratio, length, motion_strength, aesthetic_score, use_motion_strength, use_aesthetic_score, camera_motion, reference_image, refine_prompt, fps, num_loop, seed, sampling_steps, cfg_scale],- outputs=output_video- )- random_prompt_btn.click(- fn=generate_random_prompt,- outputs=prompt_text+ fn=run_video_inference,+ inputs=[+ prompt_text,+ resolution,+ aspect_ratio,+ length,+ motion_strength,+ aesthetic_score,+ use_motion_strength,+ use_aesthetic_score,+ camera_motion,+ reference_image,+ refine_prompt,+ fps,+ num_loop,+ seed,+ sampling_steps,+ cfg_scale,+ ],+ outputs=output_video,)+ random_prompt_btn.click(fn=generate_random_prompt, outputs=prompt_text)# launchdemo.launch(server_port=args.port, server_name=args.host, share=args.share)diff --git a/opensora/datasets/datasets.py b/opensora/datasets/datasets.pyindex a8e48f8..7229bb5 100644--- a/opensora/datasets/datasets.py+++ b/opensora/datasets/datasets.py@@ -3,7 +3,6 @@ from glob import globimport numpy as npimport torch-import torchvisionfrom PIL import ImageFilefrom torchvision.datasets.folder import IMG_EXTENSIONS, pil_loaderdiff --git a/opensora/datasets/read_video.py b/opensora/datasets/read_video.pyindex 8cc4bca..f988c30 100644--- a/opensora/datasets/read_video.py+++ b/opensora/datasets/read_video.py@@ -1,11 +1,11 @@import gcimport math-import cv2import osfrom fractions import Fraction-from typing import Any, Dict, List, Optional, Tuple, Union+from typing import Any, Dict, Optional, Tuple, Unionimport av+import cv2import numpy as npimport torchfrom torchvision.io.video import (@@ -144,7 +144,7 @@ def read_video_cv2(video_path):else:fps = cap.get(cv2.CAP_PROP_FPS)vinfo = {- 'video_fps': fps,+ "video_fps": fps,}frames = []@@ -159,7 +159,7 @@ def read_video_cv2(video_path):frames.append(frame[:, :, ::-1]) # BGR to RGB# Exit if 'q' is pressed- if cv2.waitKey(25) & 0xFF == ord('q'):+ if cv2.waitKey(25) & 0xFF == ord("q"):break# Release the video capture object and close all windows@@ -172,10 +172,10 @@ def read_video_cv2(video_path):return frames, vinfo-def read_video(video_path, backend='av'):- if backend == 'cv2':+def read_video(video_path, backend="av"):+ if backend == "cv2":vframes, vinfo = read_video_cv2(video_path)- elif backend == 'av':+ elif backend == "av":vframes, _, vinfo = read_video_av(filename=video_path, pts_unit="sec", output_format="TCHW")else:raise ValueError@@ -183,6 +183,6 @@ def read_video(video_path, backend='av'):return vframes, vinfo-if __name__ == '__main__':- vframes, vinfo = read_video('./data/colors/9.mp4', backend='cv2')+if __name__ == "__main__":+ vframes, vinfo = read_video("./data/colors/9.mp4", backend="cv2")x = 0diff --git a/opensora/datasets/utils.py b/opensora/datasets/utils.pyindex 194ea24..6ce155e 100644--- a/opensora/datasets/utils.py+++ b/opensora/datasets/utils.py@@ -67,7 +67,9 @@ def temporal_random_crop(vframes, num_frames, frame_interval):temporal_sample = video_transforms.TemporalRandomCrop(num_frames * frame_interval)total_frames = len(vframes)start_frame_ind, end_frame_ind = temporal_sample(total_frames)- assert end_frame_ind - start_frame_ind >= num_frames, f"Not enough frames to sample, {end_frame_ind} - {start_frame_ind} < {num_frames}"+ assert (+ end_frame_ind - start_frame_ind >= num_frames+ ), f"Not enough frames to sample, {end_frame_ind} - {start_frame_ind} < {num_frames}"frame_indice = np.linspace(start_frame_ind, end_frame_ind - 1, num_frames, dtype=int)video = vframes[frame_indice]return videodiff --git a/opensora/models/__init__.py b/opensora/models/__init__.pyindex 56db0a5..6025349 100644--- a/opensora/models/__init__.py+++ b/opensora/models/__init__.py@@ -4,4 +4,3 @@ from .pixart import *from .stdit import *from .text_encoder import *from .vae import *-diff --git a/opensora/models/layers/blocks.py b/opensora/models/layers/blocks.pyindex 6a2d793..8bc7e72 100644--- a/opensora/models/layers/blocks.py+++ b/opensora/models/layers/blocks.py@@ -494,7 +494,7 @@ class SeqParallelMultiHeadCrossAttention(MultiHeadCrossAttention):# query/value: img tokens; key: condition; mask: if padding tokenssp_group = get_sequence_parallel_group()sp_size = dist.get_world_size(sp_group)- B, SUB_N, C = x.shape # [B, TS/p, C]+ B, SUB_N, C = x.shape # [B, TS/p, C]N = SUB_N * sp_size# shape:diff --git a/opensora/models/pixart/__init__.py b/opensora/models/pixart/__init__.pyindex 4c74300..8f359d2 100644--- a/opensora/models/pixart/__init__.py+++ b/opensora/models/pixart/__init__.py@@ -1,2 +1,2 @@-from .pixart import PixArt, PixArt_XL_2, PixArt_1B_2+from .pixart import PixArt, PixArt_1B_2, PixArt_XL_2from .pixart_sigma import PixArt_Sigma_XL_2diff --git a/opensora/models/stdit/stdit2.py b/opensora/models/stdit/stdit2.pyindex 56fd361..35f6fc3 100644--- a/opensora/models/stdit/stdit2.py+++ b/opensora/models/stdit/stdit2.py@@ -1,11 +1,13 @@+import os+import numpy as npimport torchimport torch.nn as nn-import osfrom einops import rearrangefrom rotary_embedding_torch import RotaryEmbeddingfrom timm.models.layers import DropPathfrom timm.models.vision_transformer import Mlp+from transformers import PretrainedConfig, PreTrainedModelfrom opensora.acceleration.checkpoint import auto_grad_checkpointfrom opensora.models.layers.blocks import (@@ -23,7 +25,6 @@ from opensora.models.layers.blocks import (t2i_modulate,)from opensora.registry import MODELS-from transformers import PretrainedConfig, PreTrainedModelfrom opensora.utils.ckpt_utils import load_checkpoint@@ -167,7 +168,6 @@ class STDiT2Block(nn.Module):class STDiT2Config(PretrainedConfig):-model_type = "STDiT2"def __init__(@@ -217,13 +217,9 @@ class STDiT2Config(PretrainedConfig):@MODELS.register_module()class STDiT2(PreTrainedModel):-config_class = STDiT2Config- def __init__(- self,- config- ):+ def __init__(self, config):super().__init__(config)self.pred_sigma = config.pred_sigmaself.in_channels = config.in_channels@@ -245,7 +241,9 @@ class STDiT2(PreTrainedModel):self.x_embedder = PatchEmbed3D(config.patch_size, config.in_channels, config.hidden_size)self.t_embedder = TimestepEmbedder(config.hidden_size)self.t_block = nn.Sequential(nn.SiLU(), nn.Linear(config.hidden_size, 6 * config.hidden_size, bias=True))- self.t_block_temp = nn.Sequential(nn.SiLU(), nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=True)) # new+ self.t_block_temp = nn.Sequential(+ nn.SiLU(), nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=True)+ ) # newself.y_embedder = CaptionEmbedder(in_channels=config.caption_channels,hidden_size=config.hidden_size,@@ -512,12 +510,7 @@ def STDiT2_XL_2(from_pretrained=None, **kwargs):if from_pretrained is not None:if os.path.isdir(from_pretrained) or os.path.isfile(from_pretrained):# if it is a directory or a file, we load the checkpoint manually- config = STDiT2Config(- depth=28,- hidden_size=1152,- patch_size=(1, 2, 2),- num_heads=16, **kwargs- )+ config = STDiT2Config(depth=28, hidden_size=1152, patch_size=(1, 2, 2), num_heads=16, **kwargs)model = STDiT2(config)load_checkpoint(model, from_pretrained)return model@@ -526,11 +519,6 @@ def STDiT2_XL_2(from_pretrained=None, **kwargs):return STDiT2.from_pretrained(from_pretrained)else:# create a new model- config = STDiT2Config(- depth=28,- hidden_size=1152,- patch_size=(1, 2, 2),- num_heads=16, **kwargs- )+ config = STDiT2Config(depth=28, hidden_size=1152, patch_size=(1, 2, 2), num_heads=16, **kwargs)model = STDiT2(config)return modeldiff --git a/opensora/models/stdit/stdit3.py b/opensora/models/stdit/stdit3.pyindex dd9fe86..e857a37 100644--- a/opensora/models/stdit/stdit3.py+++ b/opensora/models/stdit/stdit3.py@@ -1,24 +1,26 @@import os+import numpy as npimport torch-import torch.nn as nnimport torch.distributed as dist+import torch.nn as nnfrom einops import rearrangefrom rotary_embedding_torch import RotaryEmbeddingfrom timm.models.layers import DropPathfrom timm.models.vision_transformer import Mlpfrom transformers import PretrainedConfig, PreTrainedModel++from opensora.acceleration.checkpoint import auto_grad_checkpointfrom opensora.acceleration.communications import gather_forward_split_backward, split_forward_gather_backwardfrom opensora.acceleration.parallel_states import get_sequence_parallel_group-from opensora.acceleration.checkpoint import auto_grad_checkpointfrom opensora.models.layers.blocks import (- SeqParallelMultiHeadCrossAttention,- SeqParallelAttention,Attention,CaptionEmbedder,MultiHeadCrossAttention,PatchEmbed3D,PositionEmbedding2D,+ SeqParallelAttention,+ SeqParallelMultiHeadCrossAttention,SizeEmbedder,T2IFinalLayer,TimestepEmbedder,@@ -49,14 +51,14 @@ class STDiT3Block(nn.Module):self.hidden_size = hidden_sizeself.enable_flash_attn = enable_flash_attnself.enable_sequence_parallelism = enable_sequence_parallelism-+if self.enable_sequence_parallelism and not temporal:attn_cls = SeqParallelAttentionmha_cls = SeqParallelMultiHeadCrossAttentionelse:attn_cls = Attentionmha_cls = MultiHeadCrossAttention-+self.norm1 = get_layernorm(hidden_size, eps=1e-6, affine=False, use_kernel=enable_layernorm_kernel)self.attn = attn_cls(hidden_size,@@ -389,19 +391,19 @@ class STDiT3(PreTrainedModel):x = self.x_embedder(x) # [B, N, C]x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)x = x + pos_emb-+# shard over the sequence dim if sp is enabledif self.enable_sequence_parallelism:x = split_forward_gather_backward(x, get_sequence_parallel_group(), dim=2, grad_scale="down")S = S // dist.get_world_size(get_sequence_parallel_group())-+x = rearrange(x, "B T S C -> B (T S) C", T=T, S=S)-+# === blocks ===for spatial_block, temporal_block in zip(self.spatial_blocks, self.temporal_blocks):x = auto_grad_checkpoint(spatial_block, x, y, t_mlp, y_lens, x_mask, t0_mlp, T, S)x = auto_grad_checkpoint(temporal_block, x, y, t_mlp, y_lens, x_mask, t0_mlp, T, S)-+if self.enable_sequence_parallelism:x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)x = gather_forward_split_backward(x, get_sequence_parallel_group(), dim=2, grad_scale="up")@@ -411,7 +413,7 @@ class STDiT3(PreTrainedModel):# === final layer ===x = self.final_layer(x, t, x_mask, t0, T, S)x = self.unpatchify(x, T, H, W, Tx, Hx, Wx)-+# cast to float32 for better accuracyx = x.to(torch.float32)return xdiff --git a/opensora/models/vae/discriminator.py b/opensora/models/vae/discriminator.pyindex ab00b93..2a0c367 100644--- a/opensora/models/vae/discriminator.py+++ b/opensora/models/vae/discriminator.py@@ -148,11 +148,13 @@ class ResBlockDown(nn.Module):out = (residual + x) / math.sqrt(2)return out+@MODELS.register_module()class NLayerDiscriminator(nn.Module):"""Defines a PatchGAN discriminator as in Pix2Pix- --> see https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py+ --> see https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py"""+def __init__(self, input_nc=3, ndf=64, n_layers=3, use_actnorm=False, from_pretrained=None):"""Construct a PatchGAN discriminatorParameters:@@ -162,7 +164,7 @@ class NLayerDiscriminator(nn.Module):norm_layer -- normalization layer"""super(NLayerDiscriminator, self).__init__()-+norm_layer = nn.BatchNorm2dif type(norm_layer) == functools.partial: # no need to use bias as BatchNorm2d has affine parameters@@ -177,23 +179,24 @@ class NLayerDiscriminator(nn.Module):nf_mult_prev = 1for n in range(1, n_layers): # gradually increase the number of filtersnf_mult_prev = nf_mult- nf_mult = min(2 ** n, 8)+ nf_mult = min(2**n, 8)sequence += [nn.Conv2d(ndf * nf_mult_prev, ndf * nf_mult, kernel_size=kw, stride=2, padding=padw, bias=use_bias),norm_layer(ndf * nf_mult),- nn.LeakyReLU(0.2, True)+ nn.LeakyReLU(0.2, True),]nf_mult_prev = nf_mult- nf_mult = min(2 ** n_layers, 8)+ nf_mult = min(2**n_layers, 8)sequence += [nn.Conv2d(ndf * nf_mult_prev, ndf * nf_mult, kernel_size=kw, stride=1, padding=padw, bias=use_bias),norm_layer(ndf * nf_mult),- nn.LeakyReLU(0.2, True)+ nn.LeakyReLU(0.2, True),]sequence += [- nn.Conv2d(ndf * nf_mult, 1, kernel_size=kw, stride=1, padding=padw)] # output 1 channel prediction map+ nn.Conv2d(ndf * nf_mult, 1, kernel_size=kw, stride=1, padding=padw)+ ] # output 1 channel prediction mapself.main = nn.Sequential(*sequence)if from_pretrained is not None:@@ -204,7 +207,6 @@ class NLayerDiscriminator(nn.Module):return self.main(input)-class NLayerDiscriminator3D(nn.Module):"""Defines a 3D PatchGAN discriminator as in Pix2Pix but for 3D inputs."""diff --git a/opensora/models/vae/vae.py b/opensora/models/vae/vae.pyindex f9823d9..4a9a278 100644--- a/opensora/models/vae/vae.py+++ b/opensora/models/vae/vae.py@@ -1,12 +1,13 @@import os+import torchimport torch.nn as nnfrom diffusers.models import AutoencoderKL, AutoencoderKLTemporalDecoderfrom einops import rearrange+from transformers import PretrainedConfig, PreTrainedModelfrom opensora.registry import MODELS, build_modulefrom opensora.utils.ckpt_utils import load_checkpoint-from transformers import PretrainedConfig, PreTrainedModel@MODELS.register_module()@@ -117,9 +118,10 @@ class VideoAutoencoderKLTemporalDecoder(nn.Module):def dtype(self):return next(self.parameters()).dtype+class VideoAutoencoderPipelineConfig(PretrainedConfig):model_type = "VideoAutoencoderPipeline"-+def __init__(self,vae_2d=None,@@ -130,7 +132,7 @@ class VideoAutoencoderPipelineConfig(PretrainedConfig):micro_frame_size=None,shift=0.0,scale=1.0,- **kwargs+ **kwargs,):self.vae_2d = vae_2dself.vae_temporal = vae_temporal@@ -146,11 +148,8 @@ class VideoAutoencoderPipelineConfig(PretrainedConfig):@MODELS.register_module()class VideoAutoencoderPipeline(PreTrainedModel):config_class = VideoAutoencoderPipelineConfig-- def __init__(- self,- config: VideoAutoencoderPipelineConfig- ):++ def __init__(self, config: VideoAutoencoderPipelineConfig):super().__init__(config=config)self.spatial_vae = build_module(config.vae_2d, MODELS)self.temporal_vae = build_module(config.vae_temporal, MODELS)@@ -245,6 +244,7 @@ class VideoAutoencoderPipeline(PreTrainedModel):def dtype(self):return next(self.parameters()).dtype+@MODELS.register_module()def OpenSoraVAE_V1_2(micro_batch_size=4,@@ -255,12 +255,12 @@ def OpenSoraVAE_V1_2(cal_loss=False,):vae_2d = dict(- type="VideoAutoencoderKL",- from_pretrained="PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers",- subfolder="vae",- micro_batch_size=micro_batch_size,- local_files_only=local_files_only,- )+ type="VideoAutoencoderKL",+ from_pretrained="PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers",+ subfolder="vae",+ micro_batch_size=micro_batch_size,+ local_files_only=local_files_only,+ )vae_temporal = dict(type="VAE_Temporal_SD",from_pretrained=None,@@ -274,15 +274,15 @@ def OpenSoraVAE_V1_2(cal_loss=cal_loss,micro_frame_size=micro_frame_size,shift=shift,- scale=scale+ scale=scale,)-+if from_pretrained is not None and not os.path.isdir(from_pretrained):model = VideoAutoencoderPipeline.from_pretrained(from_pretrained, **kwargs)else:config = VideoAutoencoderPipelineConfig(**kwargs)model = VideoAutoencoderPipeline(config)-+if from_pretrained:load_checkpoint(model, from_pretrained)return modeldiff --git a/opensora/schedulers/rf/__init__.py b/opensora/schedulers/rf/__init__.pyindex 4bc4ca6..e5931ec 100644--- a/opensora/schedulers/rf/__init__.py+++ b/opensora/schedulers/rf/__init__.py@@ -100,4 +100,4 @@ class RFLOW:return zdef training_losses(self, model, x_start, model_kwargs=None, noise=None, mask=None, weights=None, t=None):- return self.scheduler.training_losses(model, x_start, model_kwargs, noise, mask, weights,t)+ return self.scheduler.training_losses(model, x_start, model_kwargs, noise, mask, weights, t)diff --git a/opensora/utils/misc.py b/opensora/utils/misc.pyindex f413b6f..b2f6730 100644--- a/opensora/utils/misc.py+++ b/opensora/utils/misc.py@@ -409,4 +409,3 @@ class FeatureSaver:get_logger().info("Saved to %s", save_path)self.data_list = []self.bin_cnt += 1-diff --git a/scripts/inference.py b/scripts/inference.pyindex 2a0f4a0..5095fd2 100644--- a/scripts/inference.py+++ b/scripts/inference.py@@ -26,10 +26,10 @@ from opensora.utils.inference_utils import (extract_prompts_loop,get_save_path_name,load_prompts,+ merge_prompt,prepare_multi_resolution_info,refine_prompts_by_openai,split_prompt,- merge_prompt,)from opensora.utils.misc import all_exists, create_logger, is_distributed, is_main_process, to_torch_dtype@@ -224,7 +224,9 @@ def main():segment_start_idx = 0all_prompts = broadcast_obj_list[0]for num_segment in prompt_segment_length:- batched_prompt_segment_list.append(all_prompts[segment_start_idx : segment_start_idx + num_segment])+ batched_prompt_segment_list.append(+ all_prompts[segment_start_idx : segment_start_idx + num_segment]+ )segment_start_idx += num_segment# 2. append scorediff --git a/scripts/train.py b/scripts/train.pyindex 1dd774f..2ebdd41 100644--- a/scripts/train.py+++ b/scripts/train.py@@ -16,9 +16,9 @@ from opensora.acceleration.checkpoint import set_grad_checkpointfrom opensora.acceleration.parallel_states import get_data_parallel_groupfrom opensora.datasets.dataloader import prepare_dataloaderfrom opensora.registry import DATASETS, MODELS, SCHEDULERS, build_module-from opensora.utils.lr_scheduler import LinearWarmupLRfrom opensora.utils.ckpt_utils import load, model_gathering, model_sharding, record_model_param_shape, savefrom opensora.utils.config_utils import define_experiment_workspace, parse_configs, save_training_config+from opensora.utils.lr_scheduler import LinearWarmupLRfrom opensora.utils.misc import (Timer,all_reduce_mean,@@ -140,7 +140,7 @@ def main():in_channels=vae_out_channels,caption_channels=text_encoder_output_dim,model_max_length=text_encoder_model_max_length,- enable_sequence_parallelism=cfg.get("sp_size", 1) > 1+ enable_sequence_parallelism=cfg.get("sp_size", 1) > 1,).to(device, dtype).train()@@ -298,7 +298,7 @@ def main():# update learning rateif lr_scheduler is not None:- lr_scheduler.step()+ lr_scheduler.step()coordinator.block_all()timer_list.append(backward_t)diff --git a/scripts/train_vae.py b/scripts/train_vae.pyindex 8913d07..5376fee 100644--- a/scripts/train_vae.py+++ b/scripts/train_vae.py@@ -5,6 +5,7 @@ from pprint import pformatimport torchimport torch.distributed as dist+import wandbfrom colossalai.booster import Boosterfrom colossalai.cluster import DistCoordinatorfrom colossalai.nn.optimizer import HybridAdam@@ -12,7 +13,6 @@ from colossalai.utils import get_current_device, set_seedfrom einops import rearrangefrom tqdm import tqdm-import wandbfrom opensora.acceleration.checkpoint import set_grad_checkpointfrom opensora.acceleration.parallel_states import get_data_parallel_groupfrom opensora.datasets.dataloader import prepare_dataloaderdiff --git a/tests/test_lr_scheduler.py b/tests/test_lr_scheduler.pyindex c37b31a..8a3fd31 100644--- a/tests/test_lr_scheduler.py+++ b/tests/test_lr_scheduler.py@@ -2,6 +2,7 @@ import torchfrom torch.optim import Adamfrom torchvision.models import resnet50from tqdm import tqdm+from opensora.utils.lr_scheduler import LinearWarmupLR@@ -12,13 +13,13 @@ def test_lr_scheduler():scheduler = LinearWarmupLR(optimizer, warmup_steps=warmup_steps)current_lr = scheduler.get_lr()[0]data = torch.rand(1, 3, 224, 224).cuda()-- for i in tqdm(range(warmup_steps*2)):++ for i in tqdm(range(warmup_steps * 2)):out = model(data)out.mean().backward()optimizer.step()scheduler.step()-+if i >= warmup_steps:assert scheduler.get_lr()[0] == 0.01else:diff --git a/tests/test_stdit3_sequence_parallelism.py b/tests/test_stdit3_sequence_parallelism.pyindex 3e06028..ba715b7 100644--- a/tests/test_stdit3_sequence_parallelism.py+++ b/tests/test_stdit3_sequence_parallelism.py@@ -1,75 +1,66 @@import colossalaiimport torchimport torch.distributed as dist-from opensora.models.stdit.stdit3 import STDiT3Config, STDiT3-from colossalai.testing import spawn, free_port-from opensora.acceleration.parallel_states import set_data_parallel_group, set_sequence_parallel_group+from colossalai.testing import spawnfrom colossalai.utils.common import set_seed+from opensora.acceleration.parallel_states import set_sequence_parallel_group+from opensora.models.stdit.stdit3 import STDiT3, STDiT3Config+def get_sample_data():- x = torch.rand([1, 4, 15, 20, 27], dtype=torch.bfloat16) # (B, C, T, H, W)- timestep = torch.Tensor([924.]).to(torch.bfloat16)+ x = torch.rand([1, 4, 15, 20, 27], dtype=torch.bfloat16) # (B, C, T, H, W)+ timestep = torch.Tensor([924.0]).to(torch.bfloat16)y = torch.rand(1, 1, 300, 4096, dtype=torch.bfloat16)mask = torch.ones([1, 300], dtype=torch.int32)x_mask = torch.ones([1, 15]).bool()- fps = torch.Tensor([25.]).to(torch.bfloat16)- height = torch.Tensor([166.]).to(torch.bfloat16)- width = torch.Tensor([221.]).to(torch.bfloat16)+ fps = torch.Tensor([25.0]).to(torch.bfloat16)+ height = torch.Tensor([166.0]).to(torch.bfloat16)+ width = torch.Tensor([221.0]).to(torch.bfloat16)return dict(x=x, timestep=timestep, y=y, mask=mask, x_mask=x_mask, fps=fps, height=height, width=width)-def get_stdit3_config(enable_sequence_parallelism = False):++def get_stdit3_config(enable_sequence_parallelism=False):config = {- "caption_channels": 4096,- "class_dropout_prob": 0.0,- "depth": 1,- "drop_path": 0.0,- "enable_flash_attn": True,- "enable_layernorm_kernel": True,- "enable_sequence_parallelism": enable_sequence_parallelism,- "freeze_y_embedder": True,- "hidden_size": 1152,- "in_channels": 4,- "input_size": [- None,- None,- None- ],- "input_sq_size": 512,- "mlp_ratio": 4.0,- "model_max_length": 300,- "model_type": "STDiT3",- "num_heads": 16,- "only_train_temporal": False,- "patch_size": [- 1,- 2,- 2- ],- "pred_sigma": True,- "qk_norm": True,- "skip_y_embedder": False,- }+ "caption_channels": 4096,+ "class_dropout_prob": 0.0,+ "depth": 1,+ "drop_path": 0.0,+ "enable_flash_attn": True,+ "enable_layernorm_kernel": True,+ "enable_sequence_parallelism": enable_sequence_parallelism,+ "freeze_y_embedder": True,+ "hidden_size": 1152,+ "in_channels": 4,+ "input_size": [None, None, None],+ "input_sq_size": 512,+ "mlp_ratio": 4.0,+ "model_max_length": 300,+ "model_type": "STDiT3",+ "num_heads": 16,+ "only_train_temporal": False,+ "patch_size": [1, 2, 2],+ "pred_sigma": True,+ "qk_norm": True,+ "skip_y_embedder": False,+ }return STDiT3Config(**config)def run_model(rank, world_size, port):colossalai.launch({}, rank=rank, world_size=world_size, port=port, host="localhost")-+# prepare datadata = get_sample_data()- data = {- k: v.cuda()- for k, v in data.items()- }-+ data = {k: v.cuda() for k, v in data.items()}+# test single-gpu outptuset_seed(1024)non_dist_model_cfg = get_stdit3_config(enable_sequence_parallelism=False)non_dist_model = STDiT3(non_dist_model_cfg).cuda().to(torch.bfloat16)non_dist_out = non_dist_model(**data)non_dist_out.mean().backward()-+# run seq parallelismset_sequence_parallel_group(dist.group.WORLD)set_seed(1024)@@ -77,32 +68,32 @@ def run_model(rank, world_size, port):dist_model = STDiT3(dist_model_cfg).cuda().to(torch.bfloat16)dist_out = dist_model(**data)dist_out.mean().backward()-+# run all reduce for gradientsfor param in dist_model.parameters():if param.grad is not None:dist.all_reduce(param.grad, group=dist.group.WORLD)param.grad /= world_size-+# ensure model weights are equal- for (p1, p2) in zip(non_dist_model.parameters(), dist_model.parameters()):+ for p1, p2 in zip(non_dist_model.parameters(), dist_model.parameters()):assert torch.equal(p1, p2)-+# checktorch.testing.assert_close(non_dist_out, dist_out)- for ((n1, p1), (n2, p2)) in zip(non_dist_model.named_parameters(), dist_model.named_parameters()):+ for (n1, p1), (n2, p2) in zip(non_dist_model.named_parameters(), dist_model.named_parameters()):assert n1 == n2if p1.grad is not None and p2.grad is not None:- if not torch.allclose(p1.grad, p2.grad,rtol=1e-2, atol=1e-4):+ if not torch.allclose(p1.grad, p2.grad, rtol=1e-2, atol=1e-4):if dist.get_rank() == 0:print(f"gradient of {n1} is not equal, {p1.grad} vs {p2.grad}")else:assert p1.grad is None and p2.grad is None-+def test_stdit3_sp():spawn(run_model, 2)--if __name__ == '__main__':++if __name__ == "__main__":test_stdit3_sp()diff --git a/tools/caption/README.md b/tools/caption/README.mdindex a252363..f6fe0c8 100644--- a/tools/caption/README.md+++ b/tools/caption/README.md@@ -61,7 +61,7 @@ nohup python caption_pllava.py \```### PLLaVA vs. LLaVA-In our previous releases, we used [LLaVA](#llava-captioning) for video captioning.+In our previous releases, we used [LLaVA](#llava-captioning) for video captioning.Qualitatively speaking, we observe that PLLaVA has a somewhat higher chance of accurately capture the details in the video than LLaVA. See below for their comparison on a video sample.<!-- <img src="../../assets/readme/llava_vs_pllava_sample.gif" width="300" height="200" alt="LLaVA vs PLLaVA"> -->@@ -211,4 +211,3 @@ torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 meta.cs```This will generate tags based on the `text` column of `meta.csv` and put the results to `output_prefix + key.csv`. Currently the prompts for `objects` and `actions` are supported.-diff --git a/tools/caption/camera_motion/camera_motion.py b/tools/caption/camera_motion/camera_motion.pyindex 21ac818..9f283cb 100644--- a/tools/caption/camera_motion/camera_motion.py+++ b/tools/caption/camera_motion/camera_motion.py@@ -1,10 +1,10 @@-import torchimport os+import numpy as np-from tqdm import tqdm+import torch-from .visualizer import Visualizerfrom .utils import load_video+from .visualizer import Visualizerdef transform(vector):@@ -23,6 +23,7 @@ class CameraPredict:except:# workaround for CERTIFICATE_VERIFY_FAILED (see: https://github.com/pytorch/pytorch/issues/33288#issuecomment-954160699)import ssl+ssl._create_default_https_context = ssl._create_unverified_contextself.model = torch.hub.load(submodules_list["repo"], submodules_list["model"]).to(self.device)@@ -32,17 +33,17 @@ class CameraPredict:# set scaleheight, width = video.shape[1], video.shape[2]self.scale = min(height, width)- video = torch.from_numpy(video).permute(0, 3, 1, 2)[None].float().to(self.device) # B T C H W- pred_tracks, pred_visibility = self.model(video, grid_size=self.grid_size) # B T N 2, B T N 1-+ video = torch.from_numpy(video).permute(0, 3, 1, 2)[None].float().to(self.device) # B T C H W+ pred_tracks, pred_visibility = self.model(video, grid_size=self.grid_size) # B T N 2, B T N 1+if save_video:video_name = os.path.basename(video_path)[:-4]vis = Visualizer(save_dir=save_dir, pad_value=120, linewidth=3)vis.visualize(video, pred_tracks, pred_visibility, filename=video_name)return pred_tracks[0].long().detach().cpu().numpy()-- def transform_class(self, vector, min_reso): # 768*0.05++ def transform_class(self, vector, min_reso): # 768*0.05scale = min_reso * self.factorx, y = vectordirection = []@@ -50,7 +51,7 @@ class CameraPredict:direction.append("right")elif x < -scale:direction.append("left")-+if y > scale:direction.append("down")elif y < -scale:@@ -60,13 +61,12 @@ class CameraPredict:def get_edge_point(self, track):middle = self.grid_size // 2- top = [list(track[0, i, :]) for i in range(middle-2, middle+2)]- down = [list(track[self.grid_size-1, i, :]) for i in range(middle-2, middle+2)]- left = [list(track[i, 0, :]) for i in range(middle-2, middle+2)]- right = [list(track[i, self.grid_size-1, :]) for i in range(middle-2, middle+2)]-+ top = [list(track[0, i, :]) for i in range(middle - 2, middle + 2)]+ down = [list(track[self.grid_size - 1, i, :]) for i in range(middle - 2, middle + 2)]+ left = [list(track[i, 0, :]) for i in range(middle - 2, middle + 2)]+ right = [list(track[i, self.grid_size - 1, :]) for i in range(middle - 2, middle + 2)]+return top, down, left, right-def get_edge_direction(self, track1, track2):edge_points1 = self.get_edge_point(track1)@@ -74,14 +74,13 @@ class CameraPredict:vector_results = []for points1, points2 in zip(edge_points1, edge_points2):- vectors = [[end[0]-start[0], end[1]-start[1]] for start, end in zip(points1, points2)]+ vectors = [[end[0] - start[0], end[1] - start[1]] for start, end in zip(points1, points2)]vector_results.append(vectors)- vector_results = list(map(transform, vector_results))+ vector_results = list(map(transform, vector_results))class_results = [self.transform_class(vector, min_reso=self.scale) for vector in vector_results]return class_results-def classify_top_down(self, top, down):results = []classes = [f"{item_t}_{item_d}" for item_t in top for item_d in down]@@ -93,12 +92,11 @@ class CameraPredict:"up_up": "tilt_down","up_down": "zoom_in","down_up": "zoom_out",- "static_static": "static"+ "static_static": "static",}results = [results_mapping.get(cls) for cls in classes if cls in results_mapping]return results if results else ["None"]-def classify_left_right(self, left, right):results = []classes = [f"{item_l}_{item_r}" for item_l in left for item_r in right]@@ -109,28 +107,26 @@ class CameraPredict:"up_up": "tilt_down","left_right": "zoom_in","right_left": "zoom_out",- "static_static": "static"+ "static_static": "static",}results = [results_mapping.get(cls) for cls in classes if cls in results_mapping]return results if results else ["None"]-def camera_classify(self, track1, track2):top, down, left, right = self.get_edge_direction(track1, track2)top_results = self.classify_top_down(top, down)left_results = self.classify_left_right(left, right)- results = list(set(top_results+left_results))- if "None" in results and len(results)>1:- results.remove("None")- if "static" in results and len(results)>1:+ results = list(set(top_results + left_results))+ if "None" in results and len(results) > 1:+ results.remove("None")+ if "static" in results and len(results) > 1:results.remove("static")- if len(results) == 1 and results[0] == "None": # Tom added this to deal with edge cases+ if len(results) == 1 and results[0] == "None": # Tom added this to deal with edge casesresults = ["Undetermined"]return results-def predict(self, video_path):pred_track = self.infer(video_path)track1 = pred_track[0].reshape((self.grid_size, self.grid_size, 2))@@ -146,5 +142,5 @@ def compute_camera_motion(device, submodules_dict, video_paths, factor):all_predictions = []for video_path in video_paths:camera_motion_types = camera.predict(video_path)- all_predictions.append('+'.join(camera_motion_types))+ all_predictions.append("+".join(camera_motion_types))return all_predictionsdiff --git a/tools/caption/camera_motion/detect.py b/tools/caption/camera_motion/detect.pyindex 7f41b11..7e7eff8 100644--- a/tools/caption/camera_motion/detect.py+++ b/tools/caption/camera_motion/detect.py@@ -1,15 +1,16 @@# Originally developed by https://github.com/Vchitect/VBench based on https://github.com/facebookresearch/co-tracker.import argparse-from .camera_motion import compute_camera_motion-import pandas as pdfrom typing import List-import ptvsd++import pandas as pd++from .camera_motion import compute_camera_motiondef process(paths: List[str], threshold: float) -> List[str]:device = "cuda"- submodules = {'repo': 'facebookresearch/co-tracker', 'model': 'cotracker2'}+ submodules = {"repo": "facebookresearch/co-tracker", "model": "cotracker2"}camera_motion_types = compute_camera_motion(device, submodules, paths, factor=threshold)return camera_motion_typesdiff --git a/tools/caption/camera_motion/requirements.txt b/tools/caption/camera_motion/requirements.txtindex ce5c4bd..b699a95 100644--- a/tools/caption/camera_motion/requirements.txt+++ b/tools/caption/camera_motion/requirements.txt@@ -1,3 +1,3 @@decordptvsd-imageio-ffmpeg\ No newline at end of file+imageio-ffmpegdiff --git a/tools/caption/camera_motion/utils.py b/tools/caption/camera_motion/utils.pyindex e05fadd..15426ea 100755--- a/tools/caption/camera_motion/utils.py+++ b/tools/caption/camera_motion/utils.py@@ -1,18 +1,18 @@import numpy as npimport torch-from PIL import Image, ImageSequencefrom decord import VideoReader+from PIL import Image, ImageSequence-def get_frame_indices(num_frames, vlen, sample='rand', fix_start=None, input_fps=1, max_num_frames=-1):- if sample in ["rand", "middle"]: # uniform sampling+def get_frame_indices(num_frames, vlen, sample="rand", fix_start=None, input_fps=1, max_num_frames=-1):+ if sample in ["rand", "middle"]: # uniform samplingacc_samples = min(num_frames, vlen)# split the video into `acc_samples` intervals, and sample from each interval.intervals = np.linspace(start=0, stop=vlen, num=acc_samples + 1).astype(int)ranges = []for idx, interv in enumerate(intervals[:-1]):ranges.append((interv, intervals[idx + 1] - 1))- if sample == 'rand':+ if sample == "rand":try:frame_indices = [random.choice(range(x[0], x[1])) for x in ranges]except:@@ -21,14 +21,14 @@ def get_frame_indices(num_frames, vlen, sample='rand', fix_start=None, input_fpsframe_indices = list(frame_indices)elif fix_start is not None:frame_indices = [x[0] + fix_start for x in ranges]- elif sample == 'middle':+ elif sample == "middle":frame_indices = [(x[0] + x[1]) // 2 for x in ranges]else:raise NotImplementedErrorif len(frame_indices) < num_frames: # padded with last framepadded_frame_indices = [frame_indices[-1]] * num_frames- padded_frame_indices[:len(frame_indices)] = frame_indices+ padded_frame_indices[: len(frame_indices)] = frame_indicesframe_indices = padded_frame_indiceselif "fps" in sample: # fps0.5, sequentially sample frames at 0.5 fpsoutput_fps = float(sample[3:])@@ -44,24 +44,25 @@ def get_frame_indices(num_frames, vlen, sample='rand', fix_start=None, input_fpsraise ValueErrorreturn frame_indices+def load_video(video_path, data_transform=None, num_frames=None, return_tensor=True, width=None, height=None):"""Load a video from a given path and apply optional data transformations.The function supports loading video in GIF (.gif), PNG (.png), and MP4 (.mp4) formats.Depending on the format, it processes and extracts frames accordingly.-+Parameters:- video_path (str): The file path to the video or image to be loaded.- data_transform (callable, optional): A function that applies transformations to the video data.-+Returns:- frames (torch.Tensor): A tensor containing the video frames with shape (T, C, H, W),where T is the number of frames, C is the number of channels, H is the height, and W is the width.-+Raises:- NotImplementedError: If the video format is not supported.-+The function first determines the format of the video file by its extension.For GIFs, it iterates over each frame and converts them to RGB.For PNGs, it reads the single frame, converts it to RGB.@@ -69,23 +70,24 @@ def load_video(video_path, data_transform=None, num_frames=None, return_tensor=TIf a data_transform is provided, it is applied to the buffer before converting it to a tensor.Finally, the tensor is permuted to match the expected (T, C, H, W) format."""- if video_path.endswith('.gif'):+ if video_path.endswith(".gif"):frame_ls = []img = Image.open(video_path)for frame in ImageSequence.Iterator(img):- frame = frame.convert('RGB')+ frame = frame.convert("RGB")frame = np.array(frame).astype(np.uint8)frame_ls.append(frame)buffer = np.array(frame_ls).astype(np.uint8)- elif video_path.endswith('.png'):+ elif video_path.endswith(".png"):frame = Image.open(video_path)- frame = frame.convert('RGB')+ frame = frame.convert("RGB")frame = np.array(frame).astype(np.uint8)frame_ls = [frame]buffer = np.array(frame_ls)- elif video_path.endswith('.mp4'):+ elif video_path.endswith(".mp4"):import decord- decord.bridge.set_bridge('native')++ decord.bridge.set_bridge("native")if width:video_reader = VideoReader(video_path, width=width, height=height, num_threads=1)else:@@ -95,14 +97,12 @@ def load_video(video_path, data_transform=None, num_frames=None, return_tensor=Tbuffer = frames.asnumpy().astype(np.uint8)else:raise NotImplementedError-+frames = bufferif num_frames:- frame_indices = get_frame_indices(- num_frames, len(frames), sample="middle"- )+ frame_indices = get_frame_indices(num_frames, len(frames), sample="middle")frames = frames[frame_indices]-+if data_transform:frames = data_transform(frames)elif return_tensor:diff --git a/tools/caption/camera_motion/visualizer.py b/tools/caption/camera_motion/visualizer.pyindex edb14f8..f4f2dc6 100644--- a/tools/caption/camera_motion/visualizer.py+++ b/tools/caption/camera_motion/visualizer.py@@ -3,14 +3,14 @@# This source code is licensed under the license found in the cotracker github repo. https://github.com/facebookresearch/co-tracker.import os-import numpy as np+import imageio+import matplotlib.pyplot as plt+import numpy as npimport torch--from matplotlib import cmimport torch.nn.functional as Fimport torchvision.transforms as transforms-import matplotlib.pyplot as plt+from matplotlib import cmfrom PIL import Image, ImageDraw@@ -108,7 +108,7 @@ class Visualizer:)print("video shape after pad is: ", video.shape)tracks = tracks + self.pad_value-+print(tracks)print("tracks shape after pad is: ", tracks.shape)@@ -230,16 +230,13 @@ class Visualizer:# draw tracksif self.tracks_leave_trace != 0:for t in range(query_frame + 1, T):- first_ind = (- max(0, t - self.tracks_leave_trace) if self.tracks_leave_trace >= 0 else 0- )+ first_ind = max(0, t - self.tracks_leave_trace) if self.tracks_leave_trace >= 0 else 0curr_tracks = tracks[first_ind : t + 1]curr_colors = vector_colors[first_ind : t + 1]if compensate_for_camera_motion:- diff = (- tracks[first_ind : t + 1, segm_mask <= 0]- - tracks[t : t + 1, segm_mask <= 0]- ).mean(1)[:, None]+ diff = (tracks[first_ind : t + 1, segm_mask <= 0] - tracks[t : t + 1, segm_mask <= 0]).mean(1)[+ :, None+ ]curr_tracks = curr_tracks - diffcurr_tracks = curr_tracks[:, segm_mask > 0]@@ -262,9 +259,7 @@ class Visualizer:if visibility is not None:visibile = visibility[0, t, i]if coord[0] != 0 and coord[1] != 0:- if not compensate_for_camera_motion or (- compensate_for_camera_motion and segm_mask[i] > 0- ):+ if not compensate_for_camera_motion or (compensate_for_camera_motion and segm_mask[i] > 0):img = draw_circle(img,coord=coord,@@ -304,9 +299,7 @@ class Visualizer:self.linewidth,)if self.tracks_leave_trace > 0:- rgb = Image.fromarray(- np.uint8(add_weighted(np.array(rgb), alpha, np.array(original), 1 - alpha, 0))- )+ rgb = Image.fromarray(np.uint8(add_weighted(np.array(rgb), alpha, np.array(original), 1 - alpha, 0)))rgb = np.array(rgb)return rgbdiff --git a/tools/caption/caption_llama3.py b/tools/caption/caption_llama3.pyindex e7ff5ef..754f5a6 100644--- a/tools/caption/caption_llama3.py+++ b/tools/caption/caption_llama3.py@@ -1,30 +1,30 @@-from datetime import timedeltaimport argparseimport csvimport os+import warnings+from datetime import timedelta++import pandas as pd+import torchimport torch.distributed as dist-import argparse-import csv-import osfrom torch.utils.data import Dataset-from tqdm import tqdm-from transformers import AutoTokenizer, AutoModelForCausalLM-import torch+from transformers import AutoModelForCausalLM, AutoTokenizer+from .utils import read_file-import warnings-import pandas as pd-os.system(f'cp {__file__} ~/backup/') # optionally backup the script++os.system(f"cp {__file__} ~/backup/") # optionally backup the scriptwarnings.filterwarnings("ignore")os.environ["TOKENIZERS_PARALLELISM"] = "false"from torch.distributed.elastic.multiprocessing.errors import record+class CSVTextDataset(Dataset):def __init__(self, csv_path):self.df = pd.read_csv(csv_path)# assert text is in the columns- assert 'text' in self.df.columns, "text column not found in the csv file"-+ assert "text" in self.df.columns, "text column not found in the csv file"+def __len__(self):return len(self.df)@@ -39,10 +39,10 @@ class CSVTextDataset(Dataset):self.data_per_gpu = len(self) // world_sizeself.start_index = rank * self.data_per_gpuself.end_index = (rank + 1) * self.data_per_gpu if rank != world_size - 1 else len(self)- self.df = self.df.iloc[self.start_index:self.end_index]-+ self.df = self.df.iloc[self.start_index : self.end_index]+def write_to_csv(self, output_file, data, new_key):- ''' write the part of the df to a csv file corresponding to the rank and write self.data_list as a new column '''+ """write the part of the df to a csv file corresponding to the rank and write self.data_list as a new column"""writer = csv.writer(open(output_file, "w"))columns = self.df.columns + [new_key]writer.writerow(columns)@@ -50,8 +50,9 @@ class CSVTextDataset(Dataset):if index < self.start_index or index >= self.end_index:continuewriter.writerow([*row, data[index - self.start_index]])- writer.close()-+ writer.close()++def pad_left(sequences, padding_value=0):# Determine the maximum length of the sequencesmax_len = max([s.size(0) for s in sequences])@@ -69,6 +70,7 @@ def pad_left(sequences, padding_value=0):batch = torch.stack(padded_sequences)return batch+@recorddef main(args):# ======================================================@@ -76,30 +78,29 @@ def main(args):# ======================================================dist.init_process_group(backend="nccl", timeout=timedelta(hours=24))torch.cuda.set_device(dist.get_rank() % torch.cuda.device_count())-+# ======================================================# 2. Prep rank-wise dataloader# ======================================================dataframe = read_file(args.input)print("read data from {}".format(args.input))dataset = CSVTextDataset(args.input)- dataset.set_rank_and_world_size(- dist.get_rank(),- dist.get_world_size()- )-+ dataset.set_rank_and_world_size(dist.get_rank(), dist.get_world_size())+import os- if os.getenv('DEBUG_ADDRESS') != None and dist.get_rank() == 2:- import ptvsd;- print('waiting for debugger attachment')- ptvsd.enable_attach(address=('localhost', int(os.getenv('DEBUG_ADDRESS'))), redirect_output=True)++ if os.getenv("DEBUG_ADDRESS") != None and dist.get_rank() == 2:+ import ptvsd++ print("waiting for debugger attachment")+ ptvsd.enable_attach(address=("localhost", int(os.getenv("DEBUG_ADDRESS"))), redirect_output=True)ptvsd.wait_for_attach()-+output_file = args.output_prefix + f"_rank{dist.get_rank()}" + f"_{args.key}.csv"output_file_handle = open(output_file, "w")writer = csv.writer(output_file_handle)columns = list(dataframe.columns) + [args.key]-+writer.writerow(columns)# add a new key named summary, write in csv file@@ -115,7 +116,7 @@ def main(args):collate_fn=collate_fn,shuffle=False,)-+# ======================================================# 2. process using llama3 and prompt# ======================================================@@ -135,40 +136,35 @@ def main(args):def extract_batch(texts, prompt):input_ids_list = [tokenizer.apply_chat_template(- [- {"role": "system", "content": prompt},- {"role": "user", "content": text}- ],+ [{"role": "system", "content": prompt}, {"role": "user", "content": text}],add_generation_prompt=True,- return_tensors="pt"+ return_tensors="pt",).to(model.device)[0]for text in texts]-+attention_mask_list = [- torch.ones(input_ids.shape, dtype=torch.long, device=model.device)- for input_ids in input_ids_list+ torch.ones(input_ids.shape, dtype=torch.long, device=model.device) for input_ids in input_ids_list]-+# input_ids_batch = pad_left(# input_ids_list, padding_value=tokenizer.eos_token_id# )-+input_ids_batch = torch.nn.utils.rnn.pad_sequence(input_ids_list, batch_first=True, padding_value=tokenizer.eos_token_id)-- attention_mask_batch = torch.nn.utils.rnn.pad_sequence(- attention_mask_list, batch_first=True, padding_value=0- )-++ attention_mask_batch = torch.nn.utils.rnn.pad_sequence(attention_mask_list, batch_first=True, padding_value=0)+# attention_mask_batch = pad_left(# attention_mask_list, padding_value=0# )- terminators = [tokenizer.eos_token_id,- tokenizer.convert_tokens_to_ids("<|eot_id|>"),- ]+ terminators = [+ tokenizer.eos_token_id,+ tokenizer.convert_tokens_to_ids("<|eot_id|>"),+ ]outputs = model.generate(input_ids_batch,max_new_tokens=512,@@ -182,7 +178,7 @@ def main(args):responses = []for i in range(len(texts)):- response = outputs[i][input_ids_list[i].shape[-1]:]+ response = outputs[i][input_ids_list[i].shape[-1] :]response = tokenizer.decode(response, skip_special_tokens=True)responses.append(response)@@ -190,42 +186,49 @@ def main(args):print("Processing starting...")if args.prompt == "" and args.key == "objects":- prompt = ("You are a AI assistant to extract objects from user's text. "- "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of objects separated by ',' and wrapped by '[' and ']': '[dog, person]' ")+ prompt = (+ "You are a AI assistant to extract objects from user's text. "+ "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of objects separated by ',' and wrapped by '[' and ']': '[dog, person]' "+ )elif args.prompt == "" and args.key == "actions":- prompt = ("You are a AI assistant to extract actions from user's text. "- "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of actions separated by ',' and wrapped by '[' and ']': '[run, laugh]' ")+ prompt = (+ "You are a AI assistant to extract actions from user's text. "+ "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of actions separated by ',' and wrapped by '[' and ']': '[run, laugh]' "+ )else:prompt = args.promptprint("Prompt: {}".format(prompt))- batch_size = args.batch_size+ args.batch_size# for i in tqdm(range(0, len(dataframe), batch_size)):for _, batch in enumerate(tqdm(dataloader)):# get the text column from the batch- texts = [batch[i]['text'] for i in range(len(batch))]+ texts = [batch[i]["text"] for i in range(len(batch))]list_keywords = extract_batch(texts, prompt)-+for idx, keywords in enumerate(list_keywords):try:keywords_start = keywords.find("[")keywords_end = keywords.find("]")- keywords = keywords[keywords_start+1:keywords_end]- if "\n" in keywords or len(keywords.strip()) == 0: # we empirically observe that it produces newlines when no keywords are found+ keywords = keywords[keywords_start + 1 : keywords_end]+ if (+ "\n" in keywords or len(keywords.strip()) == 0+ ): # we empirically observe that it produces newlines when no keywords are foundkeywords = "NONE_FOUND"except:keywords = "NONE_FOUND"row = batch[idx]writer.writerow([*row, keywords])-+output_file_handle.close()dist.barrier()-+if dist.get_rank() == 0:collated_file = args.output_prefix + f"_{args.key}.csv"print("All ranks are finished. Collating the processed data to {}".format(collated_file))import pandas as pd+csv_files = [args.output_prefix + f"_rank{i}" + f"_{args.key}.csv" for i in range(dist.get_world_size())]# List to hold DataFramesdataframes = []@@ -252,7 +255,7 @@ if __name__ == "__main__":parser.add_argument("--output_prefix", type=str, help="Path to the output CSV file")parser.add_argument("--prompt", type=str, default="")parser.add_argument("--batch_size", type=int, default=32)- parser.add_argument('--key', type=str)+ parser.add_argument("--key", type=str)args = parser.parse_args()- main(args)\ No newline at end of file+ main(args)diff --git a/tools/caption/caption_llava.py b/tools/caption/caption_llava.pyindex 7946128..ab69145 100644--- a/tools/caption/caption_llava.py+++ b/tools/caption/caption_llava.py@@ -17,9 +17,9 @@ from llava.utils import disable_torch_initfrom torch.utils.data.distributed import DistributedSamplerfrom tqdm import tqdm+from ..datasets.utils import IMG_EXTENSIONS, VID_EXTENSIONSfrom .acceleration.llava.policies import LlavaLlamaForCausalLMPolicy, LlavaMistralForCausalLMPolicyfrom .utils import PROMPTS, Timer, VideoTextDataset, collate_fn-from ..datasets.utils import IMG_EXTENSIONS, VID_EXTENSIONSdisable_torch_init()diff --git a/tools/caption/pllava_dir/caption_pllava.py b/tools/caption/pllava_dir/caption_pllava.pyindex 6f65152..2213367 100644--- a/tools/caption/pllava_dir/caption_pllava.py+++ b/tools/caption/pllava_dir/caption_pllava.py@@ -1,58 +1,81 @@-from multiprocessing import Process, Queue-import os-import pandas as pd-import functoolsimport itertoolsimport logging-from tqdm import tqdm-from PIL import Image-from multiprocessing import Poolimport multiprocessing as mpfrom argparse import ArgumentParser+from multiprocessing import Process, Queue+import numpy as np-from torch.utils.data import Dataset+import pandas as pdimport torchimport torchvision-from decord import VideoReader, cpuimport transformers-import pandas as pd-import itertools-from transformers.feature_extraction_utils import BatchFeature-from tasks.eval.model_utils import load_pllava+from decord import VideoReader, cpu+from PIL import Imagefrom tasks.eval.eval_utils import Conversation+from tasks.eval.model_utils import load_pllava+from torch.utils.data import Dataset+from tqdm import tqdm+from transformers.feature_extraction_utils import BatchFeatureconv_template = Conversation(system="Describe this video. Pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. Here are some examples of good descriptions: 1. A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about. 2. Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field. 3. Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff's edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.",roles=("USER:", "ASSISTANT:"),messages=[],sep=(" ", "</s>"),- mm_token='<image>'+ mm_token="<image>",)logging.basicConfig()logger = logging.getLogger(__name__)logger.setLevel(logging.INFO)-RESOLUTION = 672 #+RESOLUTION = 672 #+-def pllava_answer(conv: Conversation, model, processor, video_list, do_sample=True, max_new_tokens=200, num_beams=1, min_length=1, top_p=0.9, repetition_penalty=1.0, length_penalty=1, temperature=1.0, stop_criteria_keywords=None, print_res=False):+def pllava_answer(+ conv: Conversation,+ model,+ processor,+ video_list,+ do_sample=True,+ max_new_tokens=200,+ num_beams=1,+ min_length=1,+ top_p=0.9,+ repetition_penalty=1.0,+ length_penalty=1,+ temperature=1.0,+ stop_criteria_keywords=None,+ print_res=False,+):# torch.cuda.empty_cache()prompt = conv.get_prompt()inputs_list = [processor(text=prompt, images=video, return_tensors="pt") for video in video_list]- inputs_batched = dict() # add batch dimension by cat+ inputs_batched = dict() # add batch dimension by catfor input_type in list(inputs_list[0].keys()):inputs_batched[input_type] = torch.cat([inputs[input_type] for inputs in inputs_list])inputs_batched = BatchFeature(inputs_batched, tensor_type="pt").to(model.device)with torch.no_grad():- output_texts = model.generate(**inputs_batched, media_type='video',- do_sample=do_sample, max_new_tokens=max_new_tokens, num_beams=num_beams, min_length=min_length,- top_p=top_p, repetition_penalty=repetition_penalty, length_penalty=length_penalty, temperature=temperature)- output_texts = processor.batch_decode(output_texts, skip_special_tokens=True, clean_up_tokenization_spaces=False)+ output_texts = model.generate(+ **inputs_batched,+ media_type="video",+ do_sample=do_sample,+ max_new_tokens=max_new_tokens,+ num_beams=num_beams,+ min_length=min_length,+ top_p=top_p,+ repetition_penalty=repetition_penalty,+ length_penalty=length_penalty,+ temperature=temperature,+ )+ output_texts = processor.batch_decode(+ output_texts, skip_special_tokens=True, clean_up_tokenization_spaces=False+ )for i in range(len(output_texts)):- if print_res: # debug usage- print('### PROMPTING LM WITH: ', prompt)- print('### LM OUTPUT TEXT: ', output_texts[i])+ if print_res: # debug usage+ print("### PROMPTING LM WITH: ", prompt)+ print("### LM OUTPUT TEXT: ", output_texts[i])if conv.roles[-1] == "<|im_start|>assistant\n":split_tag = "<|im_start|> assistant\n"else:@@ -60,18 +83,18 @@ def pllava_answer(conv: Conversation, model, processor, video_list, do_sample=Troutput_texts[i] = output_texts[i].split(split_tag)[-1]ending = conv.sep if isinstance(conv.sep, str) else conv.sep[1]output_texts[i] = output_texts[i].removesuffix(ending).strip()- output_texts[i] = output_texts[i].replace('\n', ' ')+ output_texts[i] = output_texts[i].replace("\n", " ")conv.messages[-1][1] = output_texts[i]return output_texts, conv+def get_index(num_frames, num_segments):seg_size = float(num_frames - 1) / num_segmentsstart = int(seg_size / 2)- offsets = np.array([- start + int(np.round(seg_size * idx)) for idx in range(num_segments)- ])+ offsets = np.array([start + int(np.round(seg_size * idx)) for idx in range(num_segments)])return offsets+def load_video(video_path, num_frames, return_msg=False, resolution=336):transforms = torchvision.transforms.Resize(size=resolution)vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)@@ -90,15 +113,17 @@ def load_video(video_path, num_frames, return_msg=False, resolution=336):else:return images_group+def collate_fn(batch):return batch+class CSVDataset(Dataset):def __init__(self, csv_path, num_frames):self.df = pd.read_csv(csv_path)self.data_list = self.df.path.tolist()self.num_frames = num_frames-+def __len__(self):return len(self.data_list)@@ -116,19 +141,15 @@ class CSVDataset(Dataset):end_index = (rank + 1) * self.data_per_gpu if rank != world_size - 1 else len(self)self.data_list = self.data_list[start_index:end_index]+def parse_args():parser = ArgumentParser()- parser.add_argument(- "--pretrained_model_name_or_path",- type=str,- required=True,- default='llava-hf/llava-1.5-7b-hf'- )+ parser.add_argument("--pretrained_model_name_or_path", type=str, required=True, default="llava-hf/llava-1.5-7b-hf")parser.add_argument("--batch_size",type=int,required=False,- default=1,+ default=1,)parser.add_argument("--csv_path",@@ -141,10 +162,7 @@ def parse_args():required=True,default=4,)- parser.add_argument(- "--use_lora",- action='store_true'- )+ parser.add_argument("--use_lora", action="store_true")parser.add_argument("--lora_alpha",type=int,@@ -158,13 +176,13 @@ def parse_args():default=None,)parser.add_argument(- "--conv_mode",+ "--conv_mode",type=str,required=False,- default='eval_mvbench',+ default="eval_mvbench",)parser.add_argument(- "--pooling_shape",+ "--pooling_shape",type=str,required=False,default=None,@@ -178,10 +196,28 @@ def parse_args():args = parser.parse_args()return args-def load_model_and_dataset(rank, world_size, pretrained_model_name_or_path, num_frames, use_lora, lora_alpha, weight_dir, csv_path, pooling_shape=(16,12,12)):++def load_model_and_dataset(+ rank,+ world_size,+ pretrained_model_name_or_path,+ num_frames,+ use_lora,+ lora_alpha,+ weight_dir,+ csv_path,+ pooling_shape=(16, 12, 12),+):# remind that, once the model goes larger (30B+) may cause the memory to be heavily used up. Even Tearing Nodes.- model, processor = load_pllava(pretrained_model_name_or_path, num_frames=num_frames, use_lora=use_lora, weight_dir=weight_dir, lora_alpha=lora_alpha, pooling_shape=pooling_shape)- logger.info('done loading llava')+ model, processor = load_pllava(+ pretrained_model_name_or_path,+ num_frames=num_frames,+ use_lora=use_lora,+ weight_dir=weight_dir,+ lora_alpha=lora_alpha,+ pooling_shape=pooling_shape,+ )+ logger.info("done loading llava")# position embeddingmodel = model.to(torch.device(rank))@@ -191,13 +227,14 @@ def load_model_and_dataset(rank, world_size, pretrained_model_name_or_path, num_dataset.set_rank_and_world_size(rank, world_size)return model, processor, dataset-def infer(\- model,- processor,- video_list,- conv_mode,- print_res=True,- ):++def infer(+ model,+ processor,+ video_list,+ conv_mode,+ print_res=True,+):conv = conv_template.copy()conv.user_query("Describe the video in details.", is_mm=True)@@ -208,39 +245,46 @@ def infer(\video_list=video_list,max_new_tokens=256,do_sample=False,- print_res=print_res+ print_res=print_res,)-+return llm_responses+def run(rank, args, world_size, output_queue):if rank == 0:- import os;- if os.getenv('DEBUG_ADDRESS') != None:- import ptvsd; ptvsd.enable_attach(address=('localhost', int(os.getenv('DEBUG_ADDRESS'))), redirect_output=True); ptvsd.wait_for_attach(); print('waiting for debugger attachment')- pass+ import os++ if os.getenv("DEBUG_ADDRESS") != None:+ import ptvsd++ ptvsd.enable_attach(address=("localhost", int(os.getenv("DEBUG_ADDRESS"))), redirect_output=True)+ ptvsd.wait_for_attach()+ print("waiting for debugger attachment")if rank != 0:transformers.utils.logging.set_verbosity_error()logger.setLevel(transformers.logging.ERROR)print_res = False- conv_mode= args.conv_mode+ conv_mode = args.conv_modeif args.pooling_shape is not None:- pooling_shape=tuple([int(x) for x in args.pooling_shape.split("-")])-- logger.info(f'loading model and constructing dataset to gpu {rank}...')- model, processor, dataset = load_model_and_dataset(rank,- world_size,- pretrained_model_name_or_path=args.pretrained_model_name_or_path,- num_frames=args.num_frames,- use_lora=args.use_lora,- lora_alpha=args.lora_alpha,- weight_dir=args.weight_dir,- pooling_shape=pooling_shape,- csv_path=args.csv_path)- logger.info(f'done model and dataset...')- logger.info('constructing dataset...')- logger.info('single test...')+ pooling_shape = tuple([int(x) for x in args.pooling_shape.split("-")])++ logger.info(f"loading model and constructing dataset to gpu {rank}...")+ model, processor, dataset = load_model_and_dataset(+ rank,+ world_size,+ pretrained_model_name_or_path=args.pretrained_model_name_or_path,+ num_frames=args.num_frames,+ use_lora=args.use_lora,+ lora_alpha=args.lora_alpha,+ weight_dir=args.weight_dir,+ pooling_shape=pooling_shape,+ csv_path=args.csv_path,+ )+ logger.info(f"done model and dataset...")+ logger.info("constructing dataset...")+ logger.info("single test...")dataloader = torch.utils.data.DataLoader(dataset,num_workers=2,@@ -248,7 +292,7 @@ def run(rank, args, world_size, output_queue):collate_fn=collate_fn,shuffle=False,)-+total = 0result_list = []print(len(dataset))@@ -263,24 +307,25 @@ def run(rank, args, world_size, output_queue):print_res=print_res,)except Exception as e:- logger.error(f'error in {batch}: {str(e)}')+ logger.error(f"error in {batch}: {str(e)}")preds = args.error_messageresult_list.extend(preds)output_queue.put((rank, result_list))return result_list+def main():multiprocess = True- mp.set_start_method('spawn')+ mp.set_start_method("spawn")args = parse_args()# csv_path = '/home/tom/PLLaVA/test_short_caption_part2.csv'if multiprocess:n_gpus = torch.cuda.device_count()world_size = n_gpus- print(f'world_size: {world_size}')+ print(f"world_size: {world_size}")# Create a queue to collect results from each processoutput_queue = Queue()-+# with Pool(world_size) as pool:# func = functools.partial(run, args=args, world_size=world_size)# result_lists = pool.map(func, range(world_size))@@ -291,39 +336,40 @@ def main():p.daemon = Falseprocesses.append(p)p.start()-+results_by_rank = {}for _ in range(world_size):rank, results = output_queue.get() # Retrieve results as they finishresults_by_rank[rank] = results- print(f'Results received from rank {rank}')+ print(f"Results received from rank {rank}")# ORDER THE RESULTS BY RANK- logger.info('finished running')+ logger.info("finished running")for p in processes:p.join()-+results_list = list(itertools.chain.from_iterable(results_by_rank[i] for i in range(world_size)))# results_list = list(itertools.chain([results_by_rank[i] for i in range(world_size)]))# (data[key] for key in sorted_keys)# results_list = [item for sublist in results_by_rank.values() for item in sublist]else:- results_list = run(0, world_size=1, args=args) # debug-+ results_list = run(0, world_size=1, args=args) # debug+print(results_list)df = pd.read_csv(args.csv_path)# add a new column to the dataframe- df['text'] = results_list+ df["text"] = results_listdrop_failed = Trueif drop_failed:# iterate through the dataframe and delete the entire row if captioning failedfor i in tqdm(range(len(df))):- if df['text'][i] == args.error_message:+ if df["text"][i] == args.error_message:df = df.drop(i)# write the dataframe to a new csv file called '*_pllava_13b_caption.csv'- new_csv_path = args.csv_path.replace('.csv', '_text.csv')+ new_csv_path = args.csv_path.replace(".csv", "_text.csv")df.to_csv(new_csv_path, index=False)+if __name__ == "__main__":- main()\ No newline at end of file+ main()diff --git a/tools/caption/utils.py b/tools/caption/utils.pyindex 42c3d56..b1137de 100644--- a/tools/caption/utils.py+++ b/tools/caption/utils.py@@ -6,6 +6,7 @@ import torchvision.transforms as transformsfrom torchvision.datasets.folder import pil_loaderfrom tools.datasets.utils import extract_frames, is_video+IMG_EXTENSIONS = (".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp")VID_EXTENSIONS = (".mp4", ".avi", ".mov", ".mkv")PROMPTS = {diff --git a/tools/datasets/convert.py b/tools/datasets/convert.pyindex 2d49c4d..6253e8e 100644--- a/tools/datasets/convert.py+++ b/tools/datasets/convert.py@@ -110,7 +110,7 @@ def process_general_videos(root, output):if not os.path.exists(root):returnpath_list = get_filelist(root, VID_EXTENSIONS)- path_list = list(set(path_list)) # remove duplicates+ path_list = list(set(path_list)) # remove duplicatesfname_list = [os.path.splitext(os.path.basename(x))[0] for x in path_list]relpath_list = [os.path.relpath(x, root) for x in path_list]df = pd.DataFrame(dict(path=path_list, id=fname_list, relpath=relpath_list))@@ -126,7 +126,7 @@ if __name__ == "__main__":parser.add_argument("root", type=str)parser.add_argument("--split", type=str, default="train")parser.add_argument("--info", type=str, default=None)- parser.add_argument("--output", type=str, default=None, required=True, help='Output path')+ parser.add_argument("--output", type=str, default=None, required=True, help="Output path")args = parser.parse_args()if args.dataset == "imagenet":diff --git a/tools/datasets/filter_panda10m.py b/tools/datasets/filter_panda10m.pyindex c6c3ef3..86a9f19 100644--- a/tools/datasets/filter_panda10m.py+++ b/tools/datasets/filter_panda10m.py@@ -1,262 +1,262 @@-# TODO: remove this file before releasing--import argparse-import os-import pandas as pd-import json-import html-from tqdm import tqdm-import re--tqdm.pandas()--try:- from pandarallel import pandarallel-- pandarallel.initialize(progress_bar=True)- pandas_has_parallel = True-except ImportError:- pandas_has_parallel = False---def apply(df, func, **kwargs):- if pandas_has_parallel:- return df.parallel_apply(func, **kwargs)- return df.progress_apply(func, **kwargs)---def basic_clean(text):- import ftfy-- text = ftfy.fix_text(text)- text = html.unescape(html.unescape(text))- return text.strip()---BAD_PUNCT_REGEX = re.compile(- r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"-) # noqa---def clean_caption(caption):- import urllib.parse as ul-- from bs4 import BeautifulSoup-- caption = str(caption)- caption = ul.unquote_plus(caption)- caption = caption.strip().lower()- caption = re.sub("<person>", "person", caption)- # urls:- caption = re.sub(- r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa- "",- caption,- ) # regex for urls- caption = re.sub(- r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa- "",- caption,- ) # regex for urls- # html:- caption = BeautifulSoup(caption, features="html.parser").text-- # @<nickname>- caption = re.sub(r"@[\w\d]+\b", "", caption)-- # 31C0—31EF CJK Strokes- # 31F0—31FF Katakana Phonetic Extensions- # 3200—32FF Enclosed CJK Letters and Months- # 3300—33FF CJK Compatibility- # 3400—4DBF CJK Unified Ideographs Extension A- # 4DC0—4DFF Yijing Hexagram Symbols- # 4E00—9FFF CJK Unified Ideographs- caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)- caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)- caption = re.sub(r"[\u3200-\u32ff]+", "", caption)- caption = re.sub(r"[\u3300-\u33ff]+", "", caption)- caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)- caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)- caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)- #######################################################-- # все виды тире / all types of dash --> "-"- caption = re.sub(- r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa- "-",- caption,- )-- # кавычки к одному стандарту- caption = re.sub(r"[`´«»“”¨]", '"', caption)- caption = re.sub(r"[‘’]", "'", caption)-- # "- caption = re.sub(r""?", "", caption)- # &- caption = re.sub(r"&", "", caption)-- # ip adresses:- caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)-- # article ids:- caption = re.sub(r"\d:\d\d\s+$", "", caption)-- # \n- caption = re.sub(r"\\n", " ", caption)-- # "#123"- caption = re.sub(r"#\d{1,3}\b", "", caption)- # "#12345.."- caption = re.sub(r"#\d{5,}\b", "", caption)- # "123456.."- caption = re.sub(r"\b\d{6,}\b", "", caption)- # filenames:- caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)-- #- caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""- caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""-- caption = re.sub(BAD_PUNCT_REGEX, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT- caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "-- # this-is-my-cute-cat / this_is_my_cute_cat- regex2 = re.compile(r"(?:\-|\_)")- if len(re.findall(regex2, caption)) > 3:- caption = re.sub(regex2, " ", caption)-- caption = basic_clean(caption)-- caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640- caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc- caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231-- caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)- caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)- caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)- caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)- caption = re.sub(r"\bpage\s+\d+\b", "", caption)-- caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...-- caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)-- caption = re.sub(r"\b\s+\:\s+", r": ", caption)- caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)- caption = re.sub(r"\s+", " ", caption)-- caption.strip()-- caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)- caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)- caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)- caption = re.sub(r"^\.\S+$", "", caption)-- return caption.strip()---def get_10m_set():- meta_path_10m = '/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv'- meta_10m = pd.read_csv(meta_path_10m)-- def process_single_caption(row):- text_list = eval(row['caption'])- clean_list = [clean_caption(x) for x in text_list]- return str(clean_list)-- ret = apply(meta_10m, process_single_caption, axis=1)- # ret = meta_10m.progress_apply(process_single_caption, axis=1)- print('==> text processed.')-- text_list = []- for x in ret:- text_list += eval(x)- # text_set = text_set.union(set(eval(x)))- text_set = set(text_list)- # meta_10m['caption_new'] = ret- # meta_10m.to_csv('/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m_new-cap.csv')-- # video_id_set = set(meta_10m['videoID'])- # id2t = {}- # for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):- # video_id = row['videoID']- # text_list = eval(row['caption'])- # id2t[video_id] = set(text_list)-- print(f"==> Loaded meta_10m from '{meta_path_10m}'")- return text_set---def filter_panda10m_text(meta_path, text_set):- def process_single_row(row):- # path = row['path']- t = row['text']- # fname = os.path.basename(path)- # video_id = fname[:fname.rindex('_')]- if t not in text_set:- return False- return True-- meta = pd.read_csv(meta_path)- ret = apply(meta, process_single_row, axis=1)- # ret = meta.progress_apply(process_single_row, axis=1)-- meta = meta[ret]- wo_ext, ext = os.path.splitext(meta_path)- out_path = f"{wo_ext}_filter-10m{ext}"- meta.to_csv(out_path, index=False)- print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")---def filter_panda10m_timestamp(meta_path):- meta_path_10m = '/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv'- meta_10m = pd.read_csv(meta_path_10m)-- id2t = {}- for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):- video_id = row['videoID']- timestamp = eval(row['timestamp'])- timestamp = [str(tuple(x)) for x in timestamp]- id2t[video_id] = timestamp-- # video_id_set_10m = set(meta_10m['videoID'])- print(f"==> Loaded meta_10m from '{meta_path_10m}'")-- def process_single_row(row):- path = row['path']- t = row['timestamp']- fname = os.path.basename(path)- video_id = fname[:fname.rindex('_')]- if video_id not in id2t:- return False- if t not in id2t[video_id]:- return False- return True- # return video_id in video_id_set_10m-- meta = pd.read_csv(meta_path)- ret = apply(meta, process_single_row, axis=1)-- meta = meta[ret]- wo_ext, ext = os.path.splitext(meta_path)- out_path = f"{wo_ext}_filter-10m{ext}"- meta.to_csv(out_path, index=False)- print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument('--meta_path', type=str, nargs='+')- parser.add_argument('--num_workers', default=5, type=int)-- args = parser.parse_args()- return args---if __name__ == '__main__':- args = parse_args()-- text_set = get_10m_set()- for x in args.meta_path:- filter_panda10m_text(x, text_set)+# TODO: remove this file before releasing++import argparse+import html+import os+import re++import pandas as pd+from tqdm import tqdm++tqdm.pandas()++try:+ from pandarallel import pandarallel++ pandarallel.initialize(progress_bar=True)+ pandas_has_parallel = True+except ImportError:+ pandas_has_parallel = False+++def apply(df, func, **kwargs):+ if pandas_has_parallel:+ return df.parallel_apply(func, **kwargs)+ return df.progress_apply(func, **kwargs)+++def basic_clean(text):+ import ftfy++ text = ftfy.fix_text(text)+ text = html.unescape(html.unescape(text))+ return text.strip()+++BAD_PUNCT_REGEX = re.compile(+ r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"+) # noqa+++def clean_caption(caption):+ import urllib.parse as ul++ from bs4 import BeautifulSoup++ caption = str(caption)+ caption = ul.unquote_plus(caption)+ caption = caption.strip().lower()+ caption = re.sub("<person>", "person", caption)+ # urls:+ caption = re.sub(+ r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa+ "",+ caption,+ ) # regex for urls+ caption = re.sub(+ r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa+ "",+ caption,+ ) # regex for urls+ # html:+ caption = BeautifulSoup(caption, features="html.parser").text++ # @<nickname>+ caption = re.sub(r"@[\w\d]+\b", "", caption)++ # 31C0—31EF CJK Strokes+ # 31F0—31FF Katakana Phonetic Extensions+ # 3200—32FF Enclosed CJK Letters and Months+ # 3300—33FF CJK Compatibility+ # 3400—4DBF CJK Unified Ideographs Extension A+ # 4DC0—4DFF Yijing Hexagram Symbols+ # 4E00—9FFF CJK Unified Ideographs+ caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)+ caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)+ caption = re.sub(r"[\u3200-\u32ff]+", "", caption)+ caption = re.sub(r"[\u3300-\u33ff]+", "", caption)+ caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)+ caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)+ caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)+ #######################################################++ # все виды тире / all types of dash --> "-"+ caption = re.sub(+ r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa+ "-",+ caption,+ )++ # кавычки к одному стандарту+ caption = re.sub(r"[`´«»“”¨]", '"', caption)+ caption = re.sub(r"[‘’]", "'", caption)++ # "+ caption = re.sub(r""?", "", caption)+ # &+ caption = re.sub(r"&", "", caption)++ # ip adresses:+ caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)++ # article ids:+ caption = re.sub(r"\d:\d\d\s+$", "", caption)++ # \n+ caption = re.sub(r"\\n", " ", caption)++ # "#123"+ caption = re.sub(r"#\d{1,3}\b", "", caption)+ # "#12345.."+ caption = re.sub(r"#\d{5,}\b", "", caption)+ # "123456.."+ caption = re.sub(r"\b\d{6,}\b", "", caption)+ # filenames:+ caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)++ #+ caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""+ caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""++ caption = re.sub(BAD_PUNCT_REGEX, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT+ caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "++ # this-is-my-cute-cat / this_is_my_cute_cat+ regex2 = re.compile(r"(?:\-|\_)")+ if len(re.findall(regex2, caption)) > 3:+ caption = re.sub(regex2, " ", caption)++ caption = basic_clean(caption)++ caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640+ caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc+ caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231++ caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)+ caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)+ caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)+ caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)+ caption = re.sub(r"\bpage\s+\d+\b", "", caption)++ caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...++ caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)++ caption = re.sub(r"\b\s+\:\s+", r": ", caption)+ caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)+ caption = re.sub(r"\s+", " ", caption)++ caption.strip()++ caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)+ caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)+ caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)+ caption = re.sub(r"^\.\S+$", "", caption)++ return caption.strip()+++def get_10m_set():+ meta_path_10m = "/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv"+ meta_10m = pd.read_csv(meta_path_10m)++ def process_single_caption(row):+ text_list = eval(row["caption"])+ clean_list = [clean_caption(x) for x in text_list]+ return str(clean_list)++ ret = apply(meta_10m, process_single_caption, axis=1)+ # ret = meta_10m.progress_apply(process_single_caption, axis=1)+ print("==> text processed.")++ text_list = []+ for x in ret:+ text_list += eval(x)+ # text_set = text_set.union(set(eval(x)))+ text_set = set(text_list)+ # meta_10m['caption_new'] = ret+ # meta_10m.to_csv('/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m_new-cap.csv')++ # video_id_set = set(meta_10m['videoID'])+ # id2t = {}+ # for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):+ # video_id = row['videoID']+ # text_list = eval(row['caption'])+ # id2t[video_id] = set(text_list)++ print(f"==> Loaded meta_10m from '{meta_path_10m}'")+ return text_set+++def filter_panda10m_text(meta_path, text_set):+ def process_single_row(row):+ # path = row['path']+ t = row["text"]+ # fname = os.path.basename(path)+ # video_id = fname[:fname.rindex('_')]+ if t not in text_set:+ return False+ return True++ meta = pd.read_csv(meta_path)+ ret = apply(meta, process_single_row, axis=1)+ # ret = meta.progress_apply(process_single_row, axis=1)++ meta = meta[ret]+ wo_ext, ext = os.path.splitext(meta_path)+ out_path = f"{wo_ext}_filter-10m{ext}"+ meta.to_csv(out_path, index=False)+ print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")+++def filter_panda10m_timestamp(meta_path):+ meta_path_10m = "/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv"+ meta_10m = pd.read_csv(meta_path_10m)++ id2t = {}+ for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):+ video_id = row["videoID"]+ timestamp = eval(row["timestamp"])+ timestamp = [str(tuple(x)) for x in timestamp]+ id2t[video_id] = timestamp++ # video_id_set_10m = set(meta_10m['videoID'])+ print(f"==> Loaded meta_10m from '{meta_path_10m}'")++ def process_single_row(row):+ path = row["path"]+ t = row["timestamp"]+ fname = os.path.basename(path)+ video_id = fname[: fname.rindex("_")]+ if video_id not in id2t:+ return False+ if t not in id2t[video_id]:+ return False+ return True+ # return video_id in video_id_set_10m++ meta = pd.read_csv(meta_path)+ ret = apply(meta, process_single_row, axis=1)++ meta = meta[ret]+ wo_ext, ext = os.path.splitext(meta_path)+ out_path = f"{wo_ext}_filter-10m{ext}"+ meta.to_csv(out_path, index=False)+ print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("--meta_path", type=str, nargs="+")+ parser.add_argument("--num_workers", default=5, type=int)++ args = parser.parse_args()+ return args+++if __name__ == "__main__":+ args = parse_args()++ text_set = get_10m_set()+ for x in args.meta_path:+ filter_panda10m_text(x, text_set)diff --git a/tools/scene_cut/convert_id_to_path.py b/tools/scene_cut/convert_id_to_path.pyindex d217a78..b8122d3 100644--- a/tools/scene_cut/convert_id_to_path.py+++ b/tools/scene_cut/convert_id_to_path.py@@ -1,15 +1,14 @@-import os-import argparseimport json+import osfrom functools import partial+import cv2import numpy as npimport pandas as pd-from pandarallel import pandarallel-import cv2from mmengine.logging import print_logfrom moviepy.editor import VideoFileClip+from pandarallel import pandarallelfrom tqdm import tqdmtqdm.pandas()@@ -68,7 +67,7 @@ def parse_args():parser.add_argument("meta_path", type=str)parser.add_argument("--folder_path", type=str, required=True)parser.add_argument("--mode", type=str, default=None)- parser.add_argument("--num_workers", type=int, default=None, help='#workers for pandarallel')+ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")args = parser.parse_args()return argsdiff --git a/tools/scene_cut/scene_detect.py b/tools/scene_cut/scene_detect.pyindex 0df68d7..c564d73 100644--- a/tools/scene_cut/scene_detect.py+++ b/tools/scene_cut/scene_detect.py@@ -34,7 +34,7 @@ def process_single_row(row):def parse_args():parser = argparse.ArgumentParser()parser.add_argument("meta_path", type=str)- parser.add_argument("--num_workers", type=int, default=None, help='#workers for pandarallel')+ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")args = parser.parse_args()return args@@ -44,7 +44,7 @@ def main():args = parse_args()meta_path = args.meta_pathif not os.path.exists(meta_path):- print(f"Meta file \'{meta_path}\' not found. Exit.")+ print(f"Meta file '{meta_path}' not found. Exit.")exit()if args.num_workers is not None:diff --git a/tools/scoring/aesthetic/inference.py b/tools/scoring/aesthetic/inference.pyindex 06a8e92..be6618f 100644--- a/tools/scoring/aesthetic/inference.py+++ b/tools/scoring/aesthetic/inference.py@@ -122,7 +122,7 @@ def parse_args():parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")parser.add_argument("--prefetch_factor", type=int, default=3, help="Prefetch factor")parser.add_argument("--num_frames", type=int, default=3, help="Number of frames to extract")- parser.add_argument("--skip_if_existing", action='store_true')+ parser.add_argument("--skip_if_existing", action="store_true")args = parser.parse_args()return args@@ -133,13 +133,13 @@ def main():meta_path = args.meta_pathif not os.path.exists(meta_path):- print(f"Meta file \'{meta_path}\' not found. Exit.")+ print(f"Meta file '{meta_path}' not found. Exit.")exit()wo_ext, ext = os.path.splitext(meta_path)out_path = f"{wo_ext}_aes{ext}"if args.skip_if_existing and os.path.exists(out_path):- print(f"Output meta file \'{out_path}\' already exists. Exit.")+ print(f"Output meta file '{out_path}' already exists. Exit.")exit()dist.init_process_group(backend="nccl", timeout=timedelta(hours=24))@@ -190,9 +190,11 @@ def main():# save local resultsmeta_local = merge_scores([(indices_list, scores_list)], dataset.meta, column="aes")- save_dir_local = os.path.join(os.path.dirname(out_path), 'parts')+ save_dir_local = os.path.join(os.path.dirname(out_path), "parts")os.makedirs(save_dir_local, exist_ok=True)- out_path_local = os.path.join(save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv"))+ out_path_local = os.path.join(+ save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv")+ )meta_local.to_csv(out_path_local, index=False)# wait for all ranks to finish data processingdiff --git a/tools/scoring/matching/inference.py b/tools/scoring/matching/inference.pyindex e5cd95b..70209eb 100644--- a/tools/scoring/matching/inference.py+++ b/tools/scoring/matching/inference.py@@ -66,7 +66,7 @@ def parse_args():parser.add_argument("meta_path", type=str, help="Path to the input CSV file")parser.add_argument("--bs", type=int, default=16, help="Batch size")parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")- parser.add_argument("--skip_if_existing", action='store_true')+ parser.add_argument("--skip_if_existing", action="store_true")args = parser.parse_args()return args@@ -76,16 +76,15 @@ def main():meta_path = args.meta_pathif not os.path.exists(meta_path):- print(f"Meta file \'{meta_path}\' not found. Exit.")+ print(f"Meta file '{meta_path}' not found. Exit.")exit()wo_ext, ext = os.path.splitext(meta_path)out_path = f"{wo_ext}_match{ext}"if args.skip_if_existing and os.path.exists(out_path):- print(f"Output meta file \'{out_path}\' already exists. Exit.")+ print(f"Output meta file '{out_path}' already exists. Exit.")exit()-colossalai.launch_from_torch({})# build model@@ -130,7 +129,7 @@ def main():gathered_list = [None] * dist.get_world_size()dist.all_gather_object(gathered_list, (indices_list, scores_list))if dist.get_rank() == 0:- meta_new = merge_scores(gathered_list, dataset.meta, column='match')+ meta_new = merge_scores(gathered_list, dataset.meta, column="match")meta_new.to_csv(out_path, index=False)print(f"New meta with matching scores saved to '{out_path}'.")diff --git a/tools/scoring/ocr/dbnetpp.py b/tools/scoring/ocr/dbnetpp.pyindex e313fd4..7c64615 100644--- a/tools/scoring/ocr/dbnetpp.py+++ b/tools/scoring/ocr/dbnetpp.py@@ -1,64 +1,65 @@model = dict(- type='DBNet',+ type="DBNet",backbone=dict(- type='CLIPResNet',+ type="CLIPResNet",depth=50,num_stages=4,out_indices=(0, 1, 2, 3),frozen_stages=-1,- norm_cfg=dict(type='BN', requires_grad=True),+ norm_cfg=dict(type="BN", requires_grad=True),norm_eval=False,- style='pytorch',- dcn=dict(type='DCNv2', deform_groups=1, fallback_on_stride=False),+ style="pytorch",+ dcn=dict(type="DCNv2", deform_groups=1, fallback_on_stride=False),# init_cfg=dict(# type='Pretrained',# checkpoint='https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth'),stage_with_dcn=(False, True, True, True),),neck=dict(- type='FPNC',+ type="FPNC",in_channels=[256, 512, 1024, 2048],lateral_channels=256,- asf_cfg=dict(attention_type='ScaleChannelSpatial'),+ asf_cfg=dict(attention_type="ScaleChannelSpatial"),),det_head=dict(- type='DBHead',+ type="DBHead",in_channels=256,- module_loss=dict(type='DBModuleLoss'),+ module_loss=dict(type="DBModuleLoss"),postprocessor=dict(- type='DBPostprocessor', text_repr_type='quad',+ type="DBPostprocessor",+ text_repr_type="quad",epsilon_ratio=0.002,),),data_preprocessor=dict(- type='TextDetDataPreprocessor',+ type="TextDetDataPreprocessor",mean=[123.675, 116.28, 103.53],std=[58.395, 57.12, 57.375],bgr_to_rgb=True,pad_size_divisor=32,),init_cfg=dict(- type='Pretrained',- checkpoint='https://download.openmmlab.com/mmocr/textdet/dbnetpp/'- 'dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/'- 'dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth',- )+ type="Pretrained",+ checkpoint="https://download.openmmlab.com/mmocr/textdet/dbnetpp/"+ "dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/"+ "dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth",+ ),)test_pipeline = [# dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),- dict(type='Resize', scale=(4068, 1024), keep_ratio=True),+ dict(type="Resize", scale=(4068, 1024), keep_ratio=True),dict(- type='PackTextDetInputs',+ type="PackTextDetInputs",# meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'),- meta_keys=('img_shape', 'scale_factor'),- )+ meta_keys=("img_shape", "scale_factor"),+ ),]# Visualization-vis_backends = [dict(type='LocalVisBackend')]+vis_backends = [dict(type="LocalVisBackend")]visualizer = dict(- type='TextDetLocalVisualizer',- name='visualizer',+ type="TextDetLocalVisualizer",+ name="visualizer",vis_backends=vis_backends,)diff --git a/tools/scoring/ocr/inference.py b/tools/scoring/ocr/inference.pyindex 5045fed..95a1737 100644--- a/tools/scoring/ocr/inference.py+++ b/tools/scoring/ocr/inference.py@@ -1,158 +1,158 @@-import argparse-import os--import numpy as np-import pandas as pd-import torch-import torch.distributed as dist-import torch.nn.functional as F-from torchvision.transforms import Resize, CenterCrop, Compose-from torch.utils.data import DataLoader, DistributedSampler-from torchvision.datasets.folder import pil_loader-from tqdm import tqdm--import colossalai-from mmengine import Config-from mmengine.registry import DefaultScope-from mmengine.dataset import Compose, default_collate-from mmocr.registry import MODELS-from mmocr.datasets import PackTextDetInputs--from tools.datasets.utils import extract_frames, is_video---def merge_scores(gathered_list: list, meta: pd.DataFrame):- # reorder- indices_list = list(map(lambda x: x[0], gathered_list))- scores_list = list(map(lambda x: x[1], gathered_list))- flat_indices = []- for x in zip(*indices_list):- flat_indices.extend(x)- flat_scores = []- for x in zip(*scores_list):- flat_scores.extend(x)- flat_indices = np.array(flat_indices)- flat_scores = np.array(flat_scores)- # filter duplicates- unique_indices, unique_indices_idx = np.unique(flat_indices, return_index=True)- meta.loc[unique_indices, "ocr"] = flat_scores[unique_indices_idx]---class VideoTextDataset(torch.utils.data.Dataset):- def __init__(self, meta_path, transform):- self.meta_path = meta_path- self.meta = pd.read_csv(meta_path)- self.transform = transform- self.transform = Compose([- Resize(1024),- CenterCrop(1024),- ])- self.formatting = PackTextDetInputs(meta_keys=['scale_factor'])-- def __getitem__(self, index):- row = self.meta.iloc[index]- path = row["path"]-- if is_video(path):- img = extract_frames(path, frame_inds=[10], backend="opencv")[0]- else:- img = pil_loader(path)-- img = self.transform(img)- img_array = np.array(img)[:, :, ::-1].copy() # bgr- results = {- 'img': img_array,- 'scale_factor': 1.0,- # 'img_shape': img_array.shape[-2],- # 'ori_shape': img_array.shape[-2],- }- results = self.formatting(results)- results['index'] = index-- return results-- def __len__(self):- return len(self.meta)---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument("meta_path", type=str, help="Path to the input CSV file")- parser.add_argument("--bs", type=int, default=16, help="Batch size")- parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")- parser.add_argument("--skip_if_existing", action='store_true')- args = parser.parse_args()-- return args---def main():- args = parse_args()-- meta_path = args.meta_path- if not os.path.exists(meta_path):- print(f"Meta file \'{meta_path}\' not found. Exit.")- exit()-- wo_ext, ext = os.path.splitext(meta_path)- out_path = f"{wo_ext}_ocr{ext}"- if args.skip_if_existing and os.path.exists(out_path):- print(f"Output meta file \'{out_path}\' already exists. Exit.")- exit()-- cfg = Config.fromfile('./tools/scoring/ocr/dbnetpp.py')- colossalai.launch_from_torch({})-- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")- DefaultScope.get_instance('ocr', scope_name='mmocr') # use mmocr Registry as default-- # build model- model = MODELS.build(cfg.model)- model.init_weights()- model.to(device) # set data_preprocessor._device- print('==> Model built.')-- # build dataset- transform = Compose(cfg.test_pipeline)- dataset = VideoTextDataset(meta_path=meta_path, transform=transform)- dataloader = DataLoader(- dataset,- batch_size=args.bs,- num_workers=args.num_workers,- sampler=DistributedSampler(- dataset,- num_replicas=dist.get_world_size(),- rank=dist.get_rank(),- shuffle=False,- drop_last=False,- ),- collate_fn=default_collate,- )- print('==> Dataloader built.')-- # compute scores- dataset.meta["ocr"] = np.nan- indices_list = []- scores_list = []- model.eval()- for data in tqdm(dataloader, disable=dist.get_rank() != 0):- indices_i = data['index']- indices_list.extend(indices_i.tolist())- del data['index']-- pred = model.test_step(data) # this line will cast data to device-- num_texts_i = [(x.pred_instances.scores > 0.3).sum().item() for x in pred]- scores_list.extend(num_texts_i)-- gathered_list = [None] * dist.get_world_size()- dist.all_gather_object(gathered_list, (indices_list, scores_list))-- if dist.get_rank() == 0:- merge_scores(gathered_list, dataset.meta)- dataset.meta.to_csv(out_path, index=False)- print(f"New meta (shape={dataset.meta.shape}) with ocr results saved to '{out_path}'.")---if __name__ == '__main__':- main()+import argparse+import os++import colossalai+import numpy as np+import pandas as pd+import torch+import torch.distributed as dist+from mmengine import Config+from mmengine.dataset import Compose, default_collate+from mmengine.registry import DefaultScope+from mmocr.datasets import PackTextDetInputs+from mmocr.registry import MODELS+from torch.utils.data import DataLoader, DistributedSampler+from torchvision.datasets.folder import pil_loader+from torchvision.transforms import CenterCrop, Compose, Resize+from tqdm import tqdm++from tools.datasets.utils import extract_frames, is_video+++def merge_scores(gathered_list: list, meta: pd.DataFrame):+ # reorder+ indices_list = list(map(lambda x: x[0], gathered_list))+ scores_list = list(map(lambda x: x[1], gathered_list))+ flat_indices = []+ for x in zip(*indices_list):+ flat_indices.extend(x)+ flat_scores = []+ for x in zip(*scores_list):+ flat_scores.extend(x)+ flat_indices = np.array(flat_indices)+ flat_scores = np.array(flat_scores)+ # filter duplicates+ unique_indices, unique_indices_idx = np.unique(flat_indices, return_index=True)+ meta.loc[unique_indices, "ocr"] = flat_scores[unique_indices_idx]+++class VideoTextDataset(torch.utils.data.Dataset):+ def __init__(self, meta_path, transform):+ self.meta_path = meta_path+ self.meta = pd.read_csv(meta_path)+ self.transform = transform+ self.transform = Compose(+ [+ Resize(1024),+ CenterCrop(1024),+ ]+ )+ self.formatting = PackTextDetInputs(meta_keys=["scale_factor"])++ def __getitem__(self, index):+ row = self.meta.iloc[index]+ path = row["path"]++ if is_video(path):+ img = extract_frames(path, frame_inds=[10], backend="opencv")[0]+ else:+ img = pil_loader(path)++ img = self.transform(img)+ img_array = np.array(img)[:, :, ::-1].copy() # bgr+ results = {+ "img": img_array,+ "scale_factor": 1.0,+ # 'img_shape': img_array.shape[-2],+ # 'ori_shape': img_array.shape[-2],+ }+ results = self.formatting(results)+ results["index"] = index++ return results++ def __len__(self):+ return len(self.meta)+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("meta_path", type=str, help="Path to the input CSV file")+ parser.add_argument("--bs", type=int, default=16, help="Batch size")+ parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")+ parser.add_argument("--skip_if_existing", action="store_true")+ args = parser.parse_args()++ return args+++def main():+ args = parse_args()++ meta_path = args.meta_path+ if not os.path.exists(meta_path):+ print(f"Meta file '{meta_path}' not found. Exit.")+ exit()++ wo_ext, ext = os.path.splitext(meta_path)+ out_path = f"{wo_ext}_ocr{ext}"+ if args.skip_if_existing and os.path.exists(out_path):+ print(f"Output meta file '{out_path}' already exists. Exit.")+ exit()++ cfg = Config.fromfile("./tools/scoring/ocr/dbnetpp.py")+ colossalai.launch_from_torch({})++ device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")+ DefaultScope.get_instance("ocr", scope_name="mmocr") # use mmocr Registry as default++ # build model+ model = MODELS.build(cfg.model)+ model.init_weights()+ model.to(device) # set data_preprocessor._device+ print("==> Model built.")++ # build dataset+ transform = Compose(cfg.test_pipeline)+ dataset = VideoTextDataset(meta_path=meta_path, transform=transform)+ dataloader = DataLoader(+ dataset,+ batch_size=args.bs,+ num_workers=args.num_workers,+ sampler=DistributedSampler(+ dataset,+ num_replicas=dist.get_world_size(),+ rank=dist.get_rank(),+ shuffle=False,+ drop_last=False,+ ),+ collate_fn=default_collate,+ )+ print("==> Dataloader built.")++ # compute scores+ dataset.meta["ocr"] = np.nan+ indices_list = []+ scores_list = []+ model.eval()+ for data in tqdm(dataloader, disable=dist.get_rank() != 0):+ indices_i = data["index"]+ indices_list.extend(indices_i.tolist())+ del data["index"]++ pred = model.test_step(data) # this line will cast data to device++ num_texts_i = [(x.pred_instances.scores > 0.3).sum().item() for x in pred]+ scores_list.extend(num_texts_i)++ gathered_list = [None] * dist.get_world_size()+ dist.all_gather_object(gathered_list, (indices_list, scores_list))++ if dist.get_rank() == 0:+ merge_scores(gathered_list, dataset.meta)+ dataset.meta.to_csv(out_path, index=False)+ print(f"New meta (shape={dataset.meta.shape}) with ocr results saved to '{out_path}'.")+++if __name__ == "__main__":+ main()diff --git a/tools/scoring/optical_flow/inference.py b/tools/scoring/optical_flow/inference.pyindex 31a6011..09f1b41 100644--- a/tools/scoring/optical_flow/inference.py+++ b/tools/scoring/optical_flow/inference.py@@ -80,7 +80,7 @@ def parse_args():parser.add_argument("meta_path", type=str, help="Path to the input CSV file")parser.add_argument("--bs", type=int, default=4, help="Batch size") # don't use too large bs for unimatchparser.add_argument("--num_workers", type=int, default=16, help="Number of workers")- parser.add_argument("--skip_if_existing", action='store_true')+ parser.add_argument("--skip_if_existing", action="store_true")args = parser.parse_args()return args@@ -90,13 +90,13 @@ def main():meta_path = args.meta_pathif not os.path.exists(meta_path):- print(f"Meta file \'{meta_path}\' not found. Exit.")+ print(f"Meta file '{meta_path}' not found. Exit.")exit()wo_ext, ext = os.path.splitext(meta_path)out_path = f"{wo_ext}_flow{ext}"if args.skip_if_existing and os.path.exists(out_path):- print(f"Output meta file \'{out_path}\' already exists. Exit.")+ print(f"Output meta file '{out_path}' already exists. Exit.")exit()torch.backends.cudnn.deterministic = True@@ -169,9 +169,11 @@ def main():# save local resultsmeta_local = merge_scores([(indices_list, scores_list)], dataset.meta, column="flow")- save_dir_local = os.path.join(os.path.dirname(out_path), 'parts')+ save_dir_local = os.path.join(os.path.dirname(out_path), "parts")os.makedirs(save_dir_local, exist_ok=True)- out_path_local = os.path.join(save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv"))+ out_path_local = os.path.join(+ save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv")+ )meta_local.to_csv(out_path_local, index=False)# wait for all ranks to finish data processing