Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

reformat and update docs

zhengzangw525e29abbecc
diff --git a/README.md b/README.md
index 4512769..ae74a43 100644
--- a/README.md
+++ b/README.md
@@ -137,9 +137,9 @@ see [here](/assets/texts/t2v_samples.txt) for full prompts.
* [Gradio Demo](#gradio-demo)
* [Inference](#inference)
* [Data Processing](#data-processing)
-* [VAE](#vae)
* [Training](#training)
* [Evaluation](#evaluation)
+* [VAE Training & Evaluation](#vae-training--evaluation)
* [Contribution](#contribution)
* [Citation](#citation)
* [Acknowledgement](#acknowledgement)
@@ -152,14 +152,14 @@ Other useful documents and links are listed below.
* Useful commands: [commands.md](docs/commands.md)
* Data processing pipeline and dataset: [datasets.md](docs/datasets.md)
* Each data processing tool's README: [dataset conventions and management](/tools/datasets/README.md), [scene cutting](/tools/scene_cut/README.md), [scoring](/tools/scoring/README.md), [caption](/tools/caption/README.md)
-* Evaluation: [eval](/eval/README.md)
+* Evaluation: [eval/README.md](/eval/README.md)
* Gallery: [gallery](https://hpcaitech.github.io/Open-Sora/)
## Installation
### Install from Source
-For CUDA 12.1, you can install the dependencies with the following commands. Otherwise, please refer to [Installation](docs/installation.md) for more instructions on different cuda version, and additional dependency for data preprocessing, VAE, and model evaluation.
+For CUDA 12.1, you can install the dependencies with the following commands. Otherwise, please refer to [Installation Documentation](docs/installation.md) for more instructions on different cuda version, and additional dependency for data preprocessing, VAE, and model evaluation.
```bash
# create a virtual env and activate (conda as an example)
@@ -175,7 +175,7 @@ cd Open-Sora
# the default installation is for inference only
pip install -v . # for development mode, `pip install -v -e .`
-
+```
(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.
@@ -211,9 +211,9 @@ docker run -ti --gpus all -v {MOUNT_DIR}:/data opensora
| Model | Model Size | Data | #iterations | Batch Size | URL |
| --------- | ---------- | ---- | ----------- | ---------- | ------------------------------------------------------------- |
| Diffusion | 1.1B | 30M | 70k | Dynamic | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3) |
-| VAE | 384M | 3M | 1.18M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2) |
+| VAE | 384M | 3M | 1M | 8 | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2) |
-See our **[report 1.2](docs/report_03.md)** for more infomation.
+See our **[report 1.2](docs/report_03.md)** for more infomation. Weight will be automatically downloaded when you run the inference script.
### Open-Sora 1.1 Model Weights
@@ -409,35 +409,6 @@ Also check out the [datasets](docs/datasets.md) we use.
![Data Processing Pipeline](assets/readme/report_data_pipeline.png)
-## VAE
-We train a VAE pipeline that consists of a spatial VAE followed by a temporal VAE.
-For more details, refer to [VAE](docs/vae.md).
-Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.
-
-If you want to train your own VAE, we need to prepare data in the csv following the [data processing](#data-processing) pipeline, then run the following commands.
-Note that you need to adjust the number of trained epochs (`epochs`) in the config file accordingly with respect to your own csv data size.
-
-```bash
-# stage 1 training, 380k steps, 8 GPUs
-torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH
-# stage 2 training, 260k steps, 8 GPUs
-torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH
-# stage 3 training, 540k steps, 24 GPUs
-torchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH
-```
-To evaluate the VAE performance, you need to run VAE inference first to generate the videos, then calculate scores on the generated videos:
-
-```bash
-# video generation
-torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR
-# the original videos will be saved to `YOUR_VIDEO_DIR_ori`
-# the reconstructed videos through the pipeline will be saved to `YOUR_VIDEO_DIR_rec`
-# the reconstructed videos through the spatial VAE only will be saved to `YOUR_VIDEO_DIR_spatial`
-
-# score calculation
-python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips
-```
-
## Training
### Open-Sora 1.2 Training
@@ -499,11 +470,50 @@ For training other models and advanced usage, see [here](docs/commands.md) for m
## Evaluation
-### VBench
+We support evaluation based on:
+
+* Validation loss
+* VBench score
+* VBench-i2v score
+* Batch generation for human evaluation
-### VBench-i2v
+All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details. Our [report](/docs/report_03.md#evaluation) also provides more information about the evaluation during training. The following table shows Open-Sora 1.2 greatly improves Open-Sora 1.0.
-See [here](eval/README.md) for more instructions.
+| Model | Total Score | Quality Score | Semantic Score |
+| -------------- | ----------- | ------------- | -------------- |
+| Open-Sora V1.0 | 75.91% | 78.81% | 64.28% |
+| Open-Sora V1.2 | 79.23% | 80.71% | 73.30% |
+
+## VAE Training & Evaluation
+
+We train a VAE pipeline that consists of a spatial VAE followed by a temporal VAE.
+For more details, refer to [VAE Documentation](docs/vae.md).
+Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.
+
+If you want to train your own VAE, we need to prepare data in the csv following the [data processing](#data-processing) pipeline, then run the following commands.
+Note that you need to adjust the number of trained epochs (`epochs`) in the config file accordingly with respect to your own csv data size.
+
+```bash
+# stage 1 training, 380k steps, 8 GPUs
+torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH
+# stage 2 training, 260k steps, 8 GPUs
+torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH
+# stage 3 training, 540k steps, 24 GPUs
+torchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH
+```
+
+To evaluate the VAE performance, you need to run VAE inference first to generate the videos, then calculate scores on the generated videos:
+
+```bash
+# video generation
+torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR
+# the original videos will be saved to `YOUR_VIDEO_DIR_ori`
+# the reconstructed videos through the pipeline will be saved to `YOUR_VIDEO_DIR_rec`
+# the reconstructed videos through the spatial VAE only will be saved to `YOUR_VIDEO_DIR_spatial`
+
+# score calculation
+python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips
+```
## Contribution
diff --git a/configs/opensora-v1-2/train/stage2.py b/configs/opensora-v1-2/train/stage2.py
index 30dce99..033ae6d 100644
--- a/configs/opensora-v1-2/train/stage2.py
+++ b/configs/opensora-v1-2/train/stage2.py
@@ -88,4 +88,3 @@ grad_clip = 1.0
lr = 1e-4
ema_decay = 0.99
adam_eps = 1e-15
-
diff --git a/docker/Dockerfile b/docker/Dockerfile
index a361bfb..de97f11 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -22,4 +22,4 @@ RUN pip install xformers --index-url https://download.pytorch.org/whl/cu121
# install this project
RUN git clone https://github.com/hpcaitech/Open-Sora && \
cd Open-Sora && \
- pip install -v .
\ No newline at end of file
+ pip install -v .
diff --git a/docs/data_processing.md b/docs/data_processing.md
index b4b677e..129c812 100644
--- a/docs/data_processing.md
+++ b/docs/data_processing.md
@@ -1,81 +1,81 @@
-# Data Processing
->Open-Sora v1.2 uses Data Propcessing Pipeline v1.1.
-
-We establish a complete pipeline for video/image data processing. The pipeline is shown below.
-
-![pipeline](/assets/readme/report_data_pipeline.png)
-
-First, raw videos,
-either from the Internet or public datasets, are split into shorter clips based on scene detection.
-Then, we evaluate these videos by predicting multiple scores using existing models. We first predict the aesthetic score
-and the optical flow score for a video. We also conduct OCR to detect texts in the video. Only videos with satisfactory
-evaluation results are sent to the next step for captioning. After captioning, the matching score is also calculated as
-an assessment of video-text alignment. Finally, we filter samples based on the matching score and
-conduct camera motion detection for the remaining samples.
-In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong
-semantic consistency.
-
-Below is an example workflow to process videos.
-
-```bash
-ROOT_VIDEO="/path/to/video/folder"
-ROOT_CLIPS="/path/to/video/clips/folder"
-ROOT_META="/path/to/meta/folder"
-
-# 1.1 Create a meta file from a video folder. This should output ${ROOT_META}/meta.csv
-python -m tools.datasets.convert video ${ROOT_VIDEO} --output ${ROOT_META}/meta.csv
-
-# 1.2 Get video information and remove broken videos. This should output ${ROOT_META}/meta_info_fmin1.csv
-python -m tools.datasets.datautil ${ROOT_META}/meta.csv --info --fmin 1
-
-# 2.1 Detect scenes. This should output ${ROOT_META}/meta_info_fmin1_timestamp.csv
-python -m tools.scene_cut.scene_detect ${ROOT_META}/meta_info_fmin1.csv
-
-# 2.2 Cut video into clips based on scenes. This should produce video clips under ${ROOT_CLIPS}
-python -m tools.scene_cut.cut ${ROOT_META}/meta_info_fmin1_timestamp.csv --save_dir ${ROOT_CLIPS}
-
-# 2.3 Create a meta file for video clips. This should output ${ROOT_META}/meta_clips.csv
-python -m tools.datasets.convert video ${ROOT_CLIPS} --output ${ROOT_META}/meta_clips.csv
-
-# 2.4 Get clips information and remove broken ones. This should output ${ROOT_META}/meta_clips_info_fmin1.csv
-python -m tools.datasets.datautil ${ROOT_META}/meta_clips.csv --info --fmin 1
-
-# 3.1 Predict aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes.csv
-torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference \
- ${ROOT_META}/meta_clips_info_fmin1.csv \
- --bs 1024 \
- --num_workers 16
-
-# 3.2 Filter by aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv
-python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes.csv --aesmin 5
-
-# 4.1 Generate caption. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv
-torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava \
- ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv \
- --dp-size 8 \
- --tp-size 1 \
- --model-path /path/to/llava-v1.6-mistral-7b \
- --prompt video
-
-# 4.2 Merge caption results. This should output ${ROOT_META}/meta_clips_caption.csv
-python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv --output ${ROOT_META}/meta_clips_caption.csv
-
-# 4.3 Clean caption. This should output ${ROOT_META}/meta_clips_caption_cleaned.csv
-python -m tools.datasets.datautil \
- ${ROOT_META}/meta_clips_caption.csv \
- --clean-caption \
- --refine-llm-caption \
- --remove-empty-caption \
- --output ${ROOT_META}/meta_clips_caption_cleaned.csv
-
-# 4.4 Optionally generate tags (e.g., objects) based on the captions. This should output your_output_prefix_{key}.csv
-torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 ${ROOT_META}/meta_clips_caption_cleaned.csv --key objects --output_prefix your_output_prefix
-
-```
-
-
-For more information, please refer to:
-- [Dataset Management](../tools/datasets/README.md)
-- [Scene Detection and Video Splitting](../tools/scene_cut/README.md)
-- [Scoring and Filtering](../tools/scoring/README.md)
-- [Captioning](../tools/caption/README.md)
\ No newline at end of file
+# Data Processing
+>Open-Sora v1.2 uses Data Propcessing Pipeline v1.1.
+
+We establish a complete pipeline for video/image data processing. The pipeline is shown below.
+
+![pipeline](/assets/readme/report_data_pipeline.png)
+
+First, raw videos,
+either from the Internet or public datasets, are split into shorter clips based on scene detection.
+Then, we evaluate these videos by predicting multiple scores using existing models. We first predict the aesthetic score
+and the optical flow score for a video. We also conduct OCR to detect texts in the video. Only videos with satisfactory
+evaluation results are sent to the next step for captioning. After captioning, the matching score is also calculated as
+an assessment of video-text alignment. Finally, we filter samples based on the matching score and
+conduct camera motion detection for the remaining samples.
+In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong
+semantic consistency.
+
+Below is an example workflow to process videos.
+
+```bash
+ROOT_VIDEO="/path/to/video/folder"
+ROOT_CLIPS="/path/to/video/clips/folder"
+ROOT_META="/path/to/meta/folder"
+
+# 1.1 Create a meta file from a video folder. This should output ${ROOT_META}/meta.csv
+python -m tools.datasets.convert video ${ROOT_VIDEO} --output ${ROOT_META}/meta.csv
+
+# 1.2 Get video information and remove broken videos. This should output ${ROOT_META}/meta_info_fmin1.csv
+python -m tools.datasets.datautil ${ROOT_META}/meta.csv --info --fmin 1
+
+# 2.1 Detect scenes. This should output ${ROOT_META}/meta_info_fmin1_timestamp.csv
+python -m tools.scene_cut.scene_detect ${ROOT_META}/meta_info_fmin1.csv
+
+# 2.2 Cut video into clips based on scenes. This should produce video clips under ${ROOT_CLIPS}
+python -m tools.scene_cut.cut ${ROOT_META}/meta_info_fmin1_timestamp.csv --save_dir ${ROOT_CLIPS}
+
+# 2.3 Create a meta file for video clips. This should output ${ROOT_META}/meta_clips.csv
+python -m tools.datasets.convert video ${ROOT_CLIPS} --output ${ROOT_META}/meta_clips.csv
+
+# 2.4 Get clips information and remove broken ones. This should output ${ROOT_META}/meta_clips_info_fmin1.csv
+python -m tools.datasets.datautil ${ROOT_META}/meta_clips.csv --info --fmin 1
+
+# 3.1 Predict aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes.csv
+torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference \
+ ${ROOT_META}/meta_clips_info_fmin1.csv \
+ --bs 1024 \
+ --num_workers 16
+
+# 3.2 Filter by aesthetic scores. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv
+python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes.csv --aesmin 5
+
+# 4.1 Generate caption. This should output ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv
+torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava \
+ ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5.csv \
+ --dp-size 8 \
+ --tp-size 1 \
+ --model-path /path/to/llava-v1.6-mistral-7b \
+ --prompt video
+
+# 4.2 Merge caption results. This should output ${ROOT_META}/meta_clips_caption.csv
+python -m tools.datasets.datautil ${ROOT_META}/meta_clips_info_fmin1_aes_aesmin5_caption_part*.csv --output ${ROOT_META}/meta_clips_caption.csv
+
+# 4.3 Clean caption. This should output ${ROOT_META}/meta_clips_caption_cleaned.csv
+python -m tools.datasets.datautil \
+ ${ROOT_META}/meta_clips_caption.csv \
+ --clean-caption \
+ --refine-llm-caption \
+ --remove-empty-caption \
+ --output ${ROOT_META}/meta_clips_caption_cleaned.csv
+
+# 4.4 Optionally generate tags (e.g., objects) based on the captions. This should output your_output_prefix_{key}.csv
+torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 ${ROOT_META}/meta_clips_caption_cleaned.csv --key objects --output_prefix your_output_prefix
+
+```
+
+
+For more information, please refer to:
+- [Dataset Management](../tools/datasets/README.md)
+- [Scene Detection and Video Splitting](../tools/scene_cut/README.md)
+- [Scoring and Filtering](../tools/scoring/README.md)
+- [Captioning](../tools/caption/README.md)
diff --git a/docs/report_03.md b/docs/report_03.md
index e9bb1cb..fba2f93 100644
--- a/docs/report_03.md
+++ b/docs/report_03.md
@@ -130,7 +130,7 @@ During inference, we can also use the scores to condition the model. For camera
Previously, we monitor the training process only by human evaluation, as DDPM traning loss is not well correlated with the quality of generated videos. However, for rectified flow, we find the training loss is well correlated with the quality of generated videos as stated in SD3. Thus, we keep track of rectified flow evaluation loss on 100 images and 1k videos.
-We sampled 1k videos from pixabay as validation dataset. We calculate the evaluation loss for image and different lengths of videos (2s, 4s, 8s, 16s) for different resolution (144p, 240p, 360p, 480p, 720p). For each setting, we equidistantly sample 10 timesteps. Then all the losses are averaged.
+We sampled 1k videos from pixabay as validation dataset. We calculate the evaluation loss for image and different lengths of videos (2s, 4s, 8s, 16s) for different resolution (144p, 240p, 360p, 480p, 720p). For each setting, we equidistantly sample 10 timesteps. Then all the losses are averaged. We also provide a [video](https://streamable.com/oqkkf1) showing the sampled videos with a fixed prompt for different steps.
![Evaluation Loss](/assets/readme/report_val_loss.png)
![Video Evaluation Loss](/assets/readme/report_vid_val_loss.png)
@@ -141,13 +141,11 @@ In addition, we also keep track of [VBench](https://vchitect.github.io/VBench-pr
All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details.
-
| Model | Total Score | Quality Score | Semantic Score |
| -------------- | ----------- | ------------- | -------------- |
| Open-Sora V1.0 | 75.91% | 78.81% | 64.28% |
| Open-Sora V1.2 | 79.23% | 80.71% | 73.30% |
-
## Sequence parallelism
We use sequence parallelism to support long-sequence training and inference. Our implementation is based on Ulysses and the workflow is shown below. When sequence parallelism is enabled, we only need to apply the `all-to-all` communication to the spatial block in STDiT as only spatial computation is dependent on the sequence dimension.
@@ -156,7 +154,7 @@ We use sequence parallelism to support long-sequence training and inference. Our
Currently, we have not used sequence parallelism for training as data resolution is small and we plan to do so in the next release. As for inference, we can use sequence parallelism in case your GPU goes out of memory. A simple benchmark shows that sequence parallelism can achieve speedup
-| Resolution | Seconds | Number of GPUs | Enable SP | Time taken/s | Speedup per GPU |
-| - | - | - | - | - | - |
+| Resolution | Seconds | Number of GPUs | Enable SP | Time taken/s | Speedup per GPU |
+| ---------- | ------- | -------------- | --------- | ------------ | --------------- |
| 720p | 16s | 1 | No | 547.97 | - |
-| 720p | 16s | 2 | Yes | 244.38 | 12% |
\ No newline at end of file
+| 720p | 16s | 2 | Yes | 244.38 | 12% |
diff --git a/docs/vae.md b/docs/vae.md
index 1b3f05f..75e3ccd 100644
--- a/docs/vae.md
+++ b/docs/vae.md
@@ -3,6 +3,7 @@
As [Pixart-Sigma](https://arxiv.org/abs/2403.04692) finds that adapting to a new VAE is simple, we develop an additional temporal VAE.
Specifically, our VAE consists of a pipeline of a [spatial VAE](https://huggingface.co/PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers) followed by a temporal VAE.
For the temporal VAE, we follow the implementation of [MAGVIT-v2](https://arxiv.org/abs/2310.05737), with the following modifications:
+
* We remove the architecture specific to the codebook.
* We do not use the discriminator, and use the VAE reconstruction loss, kl loss, and perceptual loss for training.
* In the last linear layer of the encoder, we scale down to a diagonal Gaussian Distribution of 4 channels, following our previously trained STDiT that takes in 4 channels input.
@@ -15,32 +16,37 @@ We train the model in different stages.
We first train the temporal VAE only by freezing the spatial VAE for 380k steps on a single machine (8 GPUs).
We use an additional identity loss to make features from the 3D VAE similar to the features from the 2D VAE.
We train the VAE using 20% images and 80% videos with 17 frames.
+
```bash
torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH
```
+
Next, we remove the indentity loss and train the 3D VAE pipeline to reconstructe the 2D-compressed videos for 260k steps.
+
```bash
torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH
```
+
Finally, we remove the reconstruction loss for the 2D-compressed videos and train the VAE pipeline to construct the 3D videos for 540k steps.
We train our VAE with a random number within 34 frames to make it more robust to different video lengths.
This stage is trained on 24 GPUs.
+
```bash
torchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH
```
Note that you need to adjust the `epochs` in the config file accordingly with respect to your own csv data size.
-
## Inference
To visually check the performance of the VAE, you may run the following inference.
It saves the original video to your specified video directory with `_ori` postfix (i.e. `"YOUR_VIDEO_DIR"_ori`), the reconstructed video from the full pipeline with the `_rec` postfix (i.e. `"YOUR_VIDEO_DIR"_rec`), and the reconstructed video from the 2D compression and decompression with the `_spatial` postfix (i.e. `"YOUR_VIDEO_DIR"_spatial`).
+
```bash
torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR
```
-
## Evaluation
+
We can then calculate the scores of the VAE performances on metrics of SSIM, PSNR, LPIPS, and FLOLPIPS.
* SSIM: structural similarity index measure, the higher the better
@@ -52,7 +58,6 @@ We can then calculate the scores of the VAE performances on metrics of SSIM, PSN
python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips
```
-
## Acknowledgement
We are grateful for the following work:
* [MAGVIT-v2](https://arxiv.org/abs/2310.05737): Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
diff --git a/docs/zh_CN/README.md b/docs/zh_CN/README.md
index f1cbf49..05f26de 100644
--- a/docs/zh_CN/README.md
+++ b/docs/zh_CN/README.md
@@ -70,7 +70,7 @@
## 🔆 新功能/更新
-* 📍 **Open-Sora 1.2** 发布。模型权重可在[此处](#model-weights)查看。有关更多详细信息,请参阅我们的**[技术报告 v1.2](docs/report_03.md)** 。
+* 📍 **Open-Sora 1.2** 发布。模型权重可在[此处](#model-weights)查看。有关更多详细信息,请参阅我们的**[技术报告 v1.2](docs/report_03.md)** 。
* ✅ 支持整流流调度。
* ✅ 训练我们的 3D-VAE 进行时间维度压缩。
* 📍 **Open-Sora 1.1**发布。模型权重可在[此处](#model-weights)获得。它针对**0s~15s、144p 到 720p、各种宽高比**的视频进行训练。有关更多讨论,请参阅我们的**[技术报告 v1.1](/docs/report_02.md)** 。
@@ -213,7 +213,7 @@ docker run -ti --gpus all -v {MOUNT_DIR}:/data opensora
<details>
<summary>查看更多</summary>
-| 分辨率 | 模型大小 | 数据 | 迭代次数 | 批量大小 | GPU 天数 (H800) | 网址
+| 分辨率 | 模型大小 | 数据 | 迭代次数 | 批量大小 | GPU 天数 (H800) | 网址
| ---------- | ---------- | ------ | ----------- | ---------- | --------------- |
| 16×512×512 | 700M | 20K HQ | 20k | 2×64 | 35 | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-HQ-16x512x512.pth) |
| 16×256×256 | 700M | 20K HQ | 24k | 8×64 | 45 | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-HQ-16x256x256.pth) |
diff --git a/docs/zh_CN/READMEv1.1.md b/docs/zh_CN/READMEv1.1.md
index 545608b..8973798 100644
--- a/docs/zh_CN/READMEv1.1.md
+++ b/docs/zh_CN/READMEv1.1.md
@@ -21,7 +21,7 @@
<h4>Open-Sora 项目目前处在早期阶段,并将持续更新。</h4>
-## 📰 资讯
+## 📰 资讯
> 由于文档需要进行翻译,最新资讯请看[英文文档](/README.md#-news)
* **[2024.04.25]** 🤗 我们在Hugging Face Spaces上发布了Open-Sora的[Gradio demo](https://huggingface.co/spaces/hpcai-tech/open-sora)。
* **[2024.04.25]** 🔥 我们发布了支持**2秒至15秒、144p至720p、任意宽高比**的文本到图像、文本到视频、图像到视频、视频到视频、无限时间生成的**Open-Sora 1.1**版本。此外,还发布了一个完整的视频处理流程。 [[checkpoints]]() [[report]](/docs/report_02.md)
diff --git a/eval/vae/cal_lpips.py b/eval/vae/cal_lpips.py
index 10edc61..8f0b7a6 100644
--- a/eval/vae/cal_lpips.py
+++ b/eval/vae/cal_lpips.py
@@ -1,17 +1,15 @@
+import lpips
import numpy as np
import torch
from tqdm import tqdm
-import math
-
-import torch
-import lpips
-spatial = True # Return a spatial map of perceptual distance.
+spatial = True # Return a spatial map of perceptual distance.
# Linearly calibrated models (LPIPS)
-loss_fn = lpips.LPIPS(net='alex', spatial=spatial) # Can also set net = 'squeeze' or 'vgg'
+loss_fn = lpips.LPIPS(net="alex", spatial=spatial) # Can also set net = 'squeeze' or 'vgg'
# loss_fn = lpips.LPIPS(net='alex', spatial=spatial, lpips=False) # Can also set net = 'squeeze' or 'vgg'
+
def trans(x):
# if greyscale images add channel
if x.shape[-3] == 1:
@@ -22,6 +20,7 @@ def trans(x):
return x
+
def calculate_lpips(videos1, videos2, device):
# image should be RGB, IMPORTANT: normalized to [-1,1]
print("calculate_lpips...")
@@ -51,22 +50,21 @@ def calculate_lpips(videos1, videos2, device):
img1 = video1[clip_timestamp].unsqueeze(0).to(device)
img2 = video2[clip_timestamp].unsqueeze(0).to(device)
-
+
loss_fn.to(device)
# calculate lpips of a video
lpips_results_of_a_video.append(loss_fn.forward(img1, img2).mean().detach().cpu().tolist())
lpips_results.append(lpips_results_of_a_video)
-
+
lpips_results = np.array(lpips_results)
-
+
lpips = {}
lpips_std = {}
for clip_timestamp in range(len(video1)):
- lpips[clip_timestamp] = np.mean(lpips_results[:,clip_timestamp])
- lpips_std[clip_timestamp] = np.std(lpips_results[:,clip_timestamp])
-
+ lpips[clip_timestamp] = np.mean(lpips_results[:, clip_timestamp])
+ lpips_std[clip_timestamp] = np.std(lpips_results[:, clip_timestamp])
result = {
"value": lpips,
@@ -77,8 +75,10 @@ def calculate_lpips(videos1, videos2, device):
return result
+
# test code / using example
+
def main():
NUMBER_OF_VIDEOS = 8
VIDEO_LENGTH = 50
@@ -90,8 +90,10 @@ def main():
# device = torch.device("cpu")
import json
+
result = calculate_lpips(videos1, videos2, device)
print(json.dumps(result, indent=4))
+
if __name__ == "__main__":
- main()
\ No newline at end of file
+ main()
diff --git a/eval/vae/cal_psnr.py b/eval/vae/cal_psnr.py
index b325106..13c30e5 100644
--- a/eval/vae/cal_psnr.py
+++ b/eval/vae/cal_psnr.py
@@ -1,7 +1,9 @@
+import math
+
import numpy as np
import torch
from tqdm import tqdm
-import math
+
def img_psnr(img1, img2):
# [0,1]
@@ -14,21 +16,23 @@ def img_psnr(img1, img2):
psnr = 20 * math.log10(1 / math.sqrt(mse))
return psnr
+
def trans(x):
return x
+
def calculate_psnr(videos1, videos2):
print("calculate_psnr...")
# videos [batch_size, timestamps, channel, h, w]
-
+
assert videos1.shape == videos2.shape
videos1 = trans(videos1)
videos2 = trans(videos2)
psnr_results = []
-
+
for video_num in tqdm(range(videos1.shape[0])):
# get a video
# video [timestamps, channel, h, w]
@@ -43,19 +47,19 @@ def calculate_psnr(videos1, videos2):
img1 = video1[clip_timestamp].numpy()
img2 = video2[clip_timestamp].numpy()
-
+
# calculate psnr of a video
psnr_results_of_a_video.append(img_psnr(img1, img2))
psnr_results.append(psnr_results_of_a_video)
-
- psnr_results = np.array(psnr_results) # [batch_size, num_frames]
+
+ psnr_results = np.array(psnr_results) # [batch_size, num_frames]
psnr = {}
psnr_std = {}
for clip_timestamp in range(len(video1)):
- psnr[clip_timestamp] = np.mean(psnr_results[:,clip_timestamp])
- psnr_std[clip_timestamp] = np.std(psnr_results[:,clip_timestamp])
+ psnr[clip_timestamp] = np.mean(psnr_results[:, clip_timestamp])
+ psnr_std[clip_timestamp] = np.std(psnr_results[:, clip_timestamp])
result = {
"value": psnr,
@@ -66,8 +70,10 @@ def calculate_psnr(videos1, videos2):
return result
+
# test code / using example
+
def main():
NUMBER_OF_VIDEOS = 8
VIDEO_LENGTH = 50
@@ -77,8 +83,10 @@ def main():
videos2 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)
import json
+
result = calculate_psnr(videos1, videos2)
print(json.dumps(result, indent=4))
+
if __name__ == "__main__":
- main()
\ No newline at end of file
+ main()
diff --git a/eval/vae/cal_ssim.py b/eval/vae/cal_ssim.py
index b2de811..65338c8 100644
--- a/eval/vae/cal_ssim.py
+++ b/eval/vae/cal_ssim.py
@@ -1,33 +1,33 @@
+import cv2
import numpy as np
import torch
from tqdm import tqdm
-import cv2
-
+
+
def ssim(img1, img2):
- C1 = 0.01 ** 2
- C2 = 0.03 ** 2
+ C1 = 0.01**2
+ C2 = 0.03**2
img1 = img1.astype(np.float64)
img2 = img2.astype(np.float64)
kernel = cv2.getGaussianKernel(11, 1.5)
window = np.outer(kernel, kernel.transpose())
mu1 = cv2.filter2D(img1, -1, window)[5:-5, 5:-5] # valid
mu2 = cv2.filter2D(img2, -1, window)[5:-5, 5:-5]
- mu1_sq = mu1 ** 2
- mu2_sq = mu2 ** 2
+ mu1_sq = mu1**2
+ mu2_sq = mu2**2
mu1_mu2 = mu1 * mu2
- sigma1_sq = cv2.filter2D(img1 ** 2, -1, window)[5:-5, 5:-5] - mu1_sq
- sigma2_sq = cv2.filter2D(img2 ** 2, -1, window)[5:-5, 5:-5] - mu2_sq
+ sigma1_sq = cv2.filter2D(img1**2, -1, window)[5:-5, 5:-5] - mu1_sq
+ sigma2_sq = cv2.filter2D(img2**2, -1, window)[5:-5, 5:-5] - mu2_sq
sigma12 = cv2.filter2D(img1 * img2, -1, window)[5:-5, 5:-5] - mu1_mu2
- ssim_map = ((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) *
- (sigma1_sq + sigma2_sq + C2))
+ ssim_map = ((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) * (sigma1_sq + sigma2_sq + C2))
return ssim_map.mean()
-
-
+
+
def calculate_ssim_function(img1, img2):
# [0,1]
- # ssim is the only metric extremely sensitive to gray being compared to b/w
+ # ssim is the only metric extremely sensitive to gray being compared to b/w
if not img1.shape == img2.shape:
- raise ValueError('Input images must have the same dimensions.')
+ raise ValueError("Input images must have the same dimensions.")
if img1.ndim == 2:
return ssim(img1, img2)
elif img1.ndim == 3:
@@ -35,27 +35,29 @@ def calculate_ssim_function(img1, img2):
ssims = []
for i in range(3):
ssims.append(ssim(img1[i], img2[i]))
- return np.array(ssims).mean()
+ return np.array(ssims).mean()
elif img1.shape[0] == 1:
return ssim(np.squeeze(img1), np.squeeze(img2))
else:
- raise ValueError('Wrong input image dimensions.')
+ raise ValueError("Wrong input image dimensions.")
+
def trans(x):
return x
+
def calculate_ssim(videos1, videos2):
print("calculate_ssim...")
# videos [batch_size, timestamps, channel, h, w]
-
+
assert videos1.shape == videos2.shape
videos1 = trans(videos1)
videos2 = trans(videos2)
ssim_results = []
-
+
for video_num in tqdm(range(videos1.shape[0])):
# get a video
# video [timestamps, channel, h, w]
@@ -70,7 +72,7 @@ def calculate_ssim(videos1, videos2):
img1 = video1[clip_timestamp].numpy()
img2 = video2[clip_timestamp].numpy()
-
+
# calculate ssim of a video
ssim_results_of_a_video.append(calculate_ssim_function(img1, img2))
@@ -82,8 +84,8 @@ def calculate_ssim(videos1, videos2):
ssim_std = {}
for clip_timestamp in range(len(video1)):
- ssim[clip_timestamp] = np.mean(ssim_results[:,clip_timestamp])
- ssim_std[clip_timestamp] = np.std(ssim_results[:,clip_timestamp])
+ ssim[clip_timestamp] = np.mean(ssim_results[:, clip_timestamp])
+ ssim_std[clip_timestamp] = np.std(ssim_results[:, clip_timestamp])
result = {
"value": ssim,
@@ -94,8 +96,10 @@ def calculate_ssim(videos1, videos2):
return result
+
# test code / using example
+
def main():
NUMBER_OF_VIDEOS = 8
VIDEO_LENGTH = 50
@@ -103,11 +107,13 @@ def main():
SIZE = 64
videos1 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)
videos2 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)
- device = torch.device("cuda")
+ torch.device("cuda")
import json
+
result = calculate_ssim(videos1, videos2)
print(json.dumps(result, indent=4))
+
if __name__ == "__main__":
- main()
\ No newline at end of file
+ main()
diff --git a/eval/vae/eval_common_metric.py b/eval/vae/eval_common_metric.py
index 4570738..e2fdc00 100644
--- a/eval/vae/eval_common_metric.py
+++ b/eval/vae/eval_common_metric.py
@@ -1,13 +1,13 @@
"""Calculates the CLIP Scores
The CLIP model is a contrasitively learned language-image model. There is
-an image encoder and a text encoder. It is believed that the CLIP model could
-measure the similarity of cross modalities. Please find more information from
+an image encoder and a text encoder. It is believed that the CLIP model could
+measure the similarity of cross modalities. Please find more information from
https://github.com/openai/CLIP.
The CLIP Score measures the Cosine Similarity between two embedded features.
-This repository utilizes the pretrained CLIP Model to calculate
-the mean average of cosine similarities.
+This repository utilizes the pretrained CLIP Model to calculate
+the mean average of cosine similarities.
See --help to see further details.
@@ -30,21 +30,21 @@ limitations under the License.
import os
import os.path as osp
+import sys
from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
+
import numpy as np
import torch
-from torch.utils.data import Dataset, DataLoader, Subset
from decord import VideoReader, cpu
-import random
from pytorchvideo.transforms import ShortSideScale
-from torchvision.io import read_video
-from torchvision.transforms import Lambda, Compose
+from torch.utils.data import DataLoader, Dataset, Subset
+from torchvision.transforms import Compose, Lambda
from torchvision.transforms._transforms_video import CenterCropVideo
-import sys
+
sys.path.append(".")
+from cal_flolpips import calculate_flolpips
from cal_lpips import calculate_lpips
from cal_psnr import calculate_psnr
-from cal_flolpips import calculate_flolpips
from cal_ssim import calculate_ssim
try:
@@ -54,15 +54,17 @@ except ImportError:
def tqdm(x):
return x
+
class VideoDataset(Dataset):
- def __init__(self,
- real_video_dir,
- generated_video_dir,
- num_frames,
- sample_rate = 1,
- crop_size=None,
- resolution=128,
- ) -> None:
+ def __init__(
+ self,
+ real_video_dir,
+ generated_video_dir,
+ num_frames,
+ sample_rate=1,
+ crop_size=None,
+ resolution=128,
+ ) -> None:
super().__init__()
self.real_video_files = self._combine_without_prefix(real_video_dir)
self.generated_video_files = self._combine_without_prefix(generated_video_dir)
@@ -71,7 +73,6 @@ class VideoDataset(Dataset):
self.crop_size = crop_size
self.short_size = resolution
-
def __len__(self):
return len(self.real_video_files)
@@ -81,10 +82,9 @@ class VideoDataset(Dataset):
real_video_file = self.real_video_files[index]
generated_video_file = self.generated_video_files[index]
print(real_video_file, generated_video_file)
- real_video_tensor = self._load_video(real_video_file)
- generated_video_tensor = self._load_video(generated_video_file)
- return {'real': real_video_tensor, 'generated':generated_video_tensor }
-
+ real_video_tensor = self._load_video(real_video_file)
+ generated_video_tensor = self._load_video(generated_video_file)
+ return {"real": real_video_tensor, "generated": generated_video_tensor}
def _load_video(self, video_path):
num_frames = self.num_frames
@@ -101,18 +101,19 @@ class VideoDataset(Dataset):
s = 0
e = total_frames
num_frames = int(total_frames / sample_frames_len * num_frames)
- print(f'sample_frames_len {sample_frames_len}, only can sample {num_frames * sample_rate}', video_path,
- total_frames)
-
+ print(
+ f"sample_frames_len {sample_frames_len}, only can sample {num_frames * sample_rate}",
+ video_path,
+ total_frames,
+ )
frame_id_list = np.linspace(s, e - 1, num_frames, dtype=int)
video_data = decord_vr.get_batch(frame_id_list).asnumpy()
video_data = torch.from_numpy(video_data)
- video_data = video_data.permute(0, 3, 1, 2) # (T, H, W, C) -> (C, T, H, W)
- return _preprocess(video_data, short_size=self.short_size, crop_size = self.crop_size)
+ video_data = video_data.permute(0, 3, 1, 2) # (T, H, W, C) -> (C, T, H, W)
+ return _preprocess(video_data, short_size=self.short_size, crop_size=self.crop_size)
-
- def _combine_without_prefix(self, folder_path, prefix='.'):
+ def _combine_without_prefix(self, folder_path, prefix="."):
folder = []
os.makedirs(folder_path, exist_ok=True)
for name in os.listdir(folder_path):
@@ -123,6 +124,7 @@ class VideoDataset(Dataset):
folder.sort()
return folder
+
def _preprocess(video_data, short_size=128, crop_size=None):
transform = Compose(
[
@@ -137,26 +139,25 @@ def _preprocess(video_data, short_size=128, crop_size=None):
def calculate_common_metric(args, dataloader, device):
-
metric_dict = {}
if type(args.metric) is str:
args.metric = [m.strip() for m in args.metric.split(",")]
print(args.metric)
for metric in args.metric:
score_list = []
- for batch_data in tqdm(dataloader): # {'real': real_video_tensor, 'generated':generated_video_tensor }
- real_videos = batch_data['real']
- generated_videos = batch_data['generated']
+ for batch_data in tqdm(dataloader): # {'real': real_video_tensor, 'generated':generated_video_tensor }
+ real_videos = batch_data["real"]
+ generated_videos = batch_data["generated"]
assert real_videos.shape[2] == generated_videos.shape[2]
- if metric == 'ssim':
- tmp_list = list(calculate_ssim(real_videos, generated_videos)['value'].values())
- elif metric == 'psnr':
- tmp_list = list(calculate_psnr(real_videos, generated_videos)['value'].values())
- elif metric == 'flolpips':
+ if metric == "ssim":
+ tmp_list = list(calculate_ssim(real_videos, generated_videos)["value"].values())
+ elif metric == "psnr":
+ tmp_list = list(calculate_psnr(real_videos, generated_videos)["value"].values())
+ elif metric == "flolpips":
result = calculate_flolpips(real_videos, generated_videos, args.device)
- tmp_list = list(result['value'].values())
+ tmp_list = list(result["value"].values())
elif metric == "lpips":
- tmp_list = list(calculate_lpips(real_videos, generated_videos, args.device)['value'].values())
+ tmp_list = list(calculate_lpips(real_videos, generated_videos, args.device)["value"].values())
else:
print(f"metric {metric} is not in acceped list, not calculated")
continue
@@ -164,35 +165,34 @@ def calculate_common_metric(args, dataloader, device):
metric_dict[metric] = np.mean(score_list)
return metric_dict
-
+
+
def main():
parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
- parser.add_argument('--batch_size', type=int, default=2,
- help='Batch size to use')
- parser.add_argument('--real_video_dir', type=str,
- help=('the path of real videos`'))
- parser.add_argument('--generated_video_dir', type=str,
- help=('the path of generated videos`'))
- parser.add_argument('--device', type=str, default=None,
- help='Device to use. Like cuda, cuda:0 or cpu')
- parser.add_argument('--num_workers', type=int, default=8,
- help=('Number of processes to use for data loading. '
- 'Defaults to `min(8, num_cpus)`'))
- parser.add_argument('--sample_fps', type=int, default=30)
- parser.add_argument('--resolution', type=int, default=336)
- parser.add_argument('--crop_size', type=int, default=None)
- parser.add_argument('--num_frames', type=int, default=100)
- parser.add_argument('--sample_rate', type=int, default=1)
- parser.add_argument('--subset_size', type=int, default=None)
+ parser.add_argument("--batch_size", type=int, default=2, help="Batch size to use")
+ parser.add_argument("--real_video_dir", type=str, help=("the path of real videos`"))
+ parser.add_argument("--generated_video_dir", type=str, help=("the path of generated videos`"))
+ parser.add_argument("--device", type=str, default=None, help="Device to use. Like cuda, cuda:0 or cpu")
+ parser.add_argument(
+ "--num_workers",
+ type=int,
+ default=8,
+ help=("Number of processes to use for data loading. " "Defaults to `min(8, num_cpus)`"),
+ )
+ parser.add_argument("--sample_fps", type=int, default=30)
+ parser.add_argument("--resolution", type=int, default=336)
+ parser.add_argument("--crop_size", type=int, default=None)
+ parser.add_argument("--num_frames", type=int, default=100)
+ parser.add_argument("--sample_rate", type=int, default=1)
+ parser.add_argument("--subset_size", type=int, default=None)
# parser.add_argument("--metric", type=str, default="fvd",choices=['fvd','psnr','ssim','lpips', 'flolpips'])
- parser.add_argument("--metric", nargs='+', default=[])
- parser.add_argument("--fvd_method", type=str, default='styleganv',choices=['styleganv','videogpt'])
-
+ parser.add_argument("--metric", nargs="+", default=[])
+ parser.add_argument("--fvd_method", type=str, default="styleganv", choices=["styleganv", "videogpt"])
args = parser.parse_args()
if args.device is None:
- device = torch.device('cuda' if (torch.cuda.is_available()) else 'cpu')
+ device = torch.device("cuda" if (torch.cuda.is_available()) else "cpu")
else:
device = torch.device(args.device)
@@ -209,24 +209,24 @@ def main():
else:
num_workers = args.num_workers
-
- dataset = VideoDataset(args.real_video_dir,
- args.generated_video_dir,
- num_frames = args.num_frames,
- sample_rate = args.sample_rate,
- crop_size=args.crop_size,
- resolution=args.resolution)
+ dataset = VideoDataset(
+ args.real_video_dir,
+ args.generated_video_dir,
+ num_frames=args.num_frames,
+ sample_rate=args.sample_rate,
+ crop_size=args.crop_size,
+ resolution=args.resolution,
+ )
if args.subset_size:
indices = range(args.subset_size)
dataset = Subset(dataset, indices=indices)
-
- dataloader = DataLoader(dataset, args.batch_size,
- num_workers=num_workers, pin_memory=True)
-
- metric_score = calculate_common_metric(args, dataloader,device)
- print('metric: ', args.metric, " ",metric_score)
+ dataloader = DataLoader(dataset, args.batch_size, num_workers=num_workers, pin_memory=True)
+
+ metric_score = calculate_common_metric(args, dataloader, device)
+ print("metric: ", args.metric, " ", metric_score)
+
-if __name__ == '__main__':
+if __name__ == "__main__":
main()
diff --git a/eval/vae/flolpips/correlation/correlation.py b/eval/vae/flolpips/correlation/correlation.py
index 7c91055..6f6d6bd 100644
--- a/eval/vae/flolpips/correlation/correlation.py
+++ b/eval/vae/flolpips/correlation/correlation.py
@@ -1,397 +1,461 @@
-#!/usr/bin/env python
-
-import torch
-
-import cupy
-import re
-
-kernel_Correlation_rearrange = '''
- extern "C" __global__ void kernel_Correlation_rearrange(
- const int n,
- const float* input,
- float* output
- ) {
- int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x;
-
- if (intIndex >= n) {
- return;
- }
-
- int intSample = blockIdx.z;
- int intChannel = blockIdx.y;
-
- float fltValue = input[(((intSample * SIZE_1(input)) + intChannel) * SIZE_2(input) * SIZE_3(input)) + intIndex];
-
- __syncthreads();
-
- int intPaddedY = (intIndex / SIZE_3(input)) + 4;
- int intPaddedX = (intIndex % SIZE_3(input)) + 4;
- int intRearrange = ((SIZE_3(input) + 8) * intPaddedY) + intPaddedX;
-
- output[(((intSample * SIZE_1(output) * SIZE_2(output)) + intRearrange) * SIZE_1(input)) + intChannel] = fltValue;
- }
-'''
-
-kernel_Correlation_updateOutput = '''
- extern "C" __global__ void kernel_Correlation_updateOutput(
- const int n,
- const float* rbot0,
- const float* rbot1,
- float* top
- ) {
- extern __shared__ char patch_data_char[];
-
- float *patch_data = (float *)patch_data_char;
-
- // First (upper left) position of kernel upper-left corner in current center position of neighborhood in image 1
- int x1 = blockIdx.x + 4;
- int y1 = blockIdx.y + 4;
- int item = blockIdx.z;
- int ch_off = threadIdx.x;
-
- // Load 3D patch into shared shared memory
- for (int j = 0; j < 1; j++) { // HEIGHT
- for (int i = 0; i < 1; i++) { // WIDTH
- int ji_off = (j + i) * SIZE_3(rbot0);
- for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS
- int idx1 = ((item * SIZE_1(rbot0) + y1+j) * SIZE_2(rbot0) + x1+i) * SIZE_3(rbot0) + ch;
- int idxPatchData = ji_off + ch;
- patch_data[idxPatchData] = rbot0[idx1];
- }
- }
- }
-
- __syncthreads();
-
- __shared__ float sum[32];
-
- // Compute correlation
- for (int top_channel = 0; top_channel < SIZE_1(top); top_channel++) {
- sum[ch_off] = 0;
-
- int s2o = top_channel % 9 - 4;
- int s2p = top_channel / 9 - 4;
-
- for (int j = 0; j < 1; j++) { // HEIGHT
- for (int i = 0; i < 1; i++) { // WIDTH
- int ji_off = (j + i) * SIZE_3(rbot0);
- for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS
- int x2 = x1 + s2o;
- int y2 = y1 + s2p;
-
- int idxPatchData = ji_off + ch;
- int idx2 = ((item * SIZE_1(rbot0) + y2+j) * SIZE_2(rbot0) + x2+i) * SIZE_3(rbot0) + ch;
-
- sum[ch_off] += patch_data[idxPatchData] * rbot1[idx2];
- }
- }
- }
-
- __syncthreads();
-
- if (ch_off == 0) {
- float total_sum = 0;
- for (int idx = 0; idx < 32; idx++) {
- total_sum += sum[idx];
- }
- const int sumelems = SIZE_3(rbot0);
- const int index = ((top_channel*SIZE_2(top) + blockIdx.y)*SIZE_3(top))+blockIdx.x;
- top[index + item*SIZE_1(top)*SIZE_2(top)*SIZE_3(top)] = total_sum / (float)sumelems;
- }
- }
- }
-'''
-
-kernel_Correlation_updateGradFirst = '''
- #define ROUND_OFF 50000
-
- extern "C" __global__ void kernel_Correlation_updateGradFirst(
- const int n,
- const int intSample,
- const float* rbot0,
- const float* rbot1,
- const float* gradOutput,
- float* gradFirst,
- float* gradSecond
- ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {
- int n = intIndex % SIZE_1(gradFirst); // channels
- int l = (intIndex / SIZE_1(gradFirst)) % SIZE_3(gradFirst) + 4; // w-pos
- int m = (intIndex / SIZE_1(gradFirst) / SIZE_3(gradFirst)) % SIZE_2(gradFirst) + 4; // h-pos
-
- // round_off is a trick to enable integer division with ceil, even for negative numbers
- // We use a large offset, for the inner part not to become negative.
- const int round_off = ROUND_OFF;
- const int round_off_s1 = round_off;
-
- // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:
- int xmin = (l - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)
- int ymin = (m - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)
-
- // Same here:
- int xmax = (l - 4 + round_off_s1) - round_off; // floor (l - 4)
- int ymax = (m - 4 + round_off_s1) - round_off; // floor (m - 4)
-
- float sum = 0;
- if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {
- xmin = max(0,xmin);
- xmax = min(SIZE_3(gradOutput)-1,xmax);
-
- ymin = max(0,ymin);
- ymax = min(SIZE_2(gradOutput)-1,ymax);
-
- for (int p = -4; p <= 4; p++) {
- for (int o = -4; o <= 4; o++) {
- // Get rbot1 data:
- int s2o = o;
- int s2p = p;
- int idxbot1 = ((intSample * SIZE_1(rbot0) + (m+s2p)) * SIZE_2(rbot0) + (l+s2o)) * SIZE_3(rbot0) + n;
- float bot1tmp = rbot1[idxbot1]; // rbot1[l+s2o,m+s2p,n]
-
- // Index offset for gradOutput in following loops:
- int op = (p+4) * 9 + (o+4); // index[o,p]
- int idxopoffset = (intSample * SIZE_1(gradOutput) + op);
-
- for (int y = ymin; y <= ymax; y++) {
- for (int x = xmin; x <= xmax; x++) {
- int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]
- sum += gradOutput[idxgradOutput] * bot1tmp;
- }
- }
- }
- }
- }
- const int sumelems = SIZE_1(gradFirst);
- const int bot0index = ((n * SIZE_2(gradFirst)) + (m-4)) * SIZE_3(gradFirst) + (l-4);
- gradFirst[bot0index + intSample*SIZE_1(gradFirst)*SIZE_2(gradFirst)*SIZE_3(gradFirst)] = sum / (float)sumelems;
- } }
-'''
-
-kernel_Correlation_updateGradSecond = '''
- #define ROUND_OFF 50000
-
- extern "C" __global__ void kernel_Correlation_updateGradSecond(
- const int n,
- const int intSample,
- const float* rbot0,
- const float* rbot1,
- const float* gradOutput,
- float* gradFirst,
- float* gradSecond
- ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {
- int n = intIndex % SIZE_1(gradSecond); // channels
- int l = (intIndex / SIZE_1(gradSecond)) % SIZE_3(gradSecond) + 4; // w-pos
- int m = (intIndex / SIZE_1(gradSecond) / SIZE_3(gradSecond)) % SIZE_2(gradSecond) + 4; // h-pos
-
- // round_off is a trick to enable integer division with ceil, even for negative numbers
- // We use a large offset, for the inner part not to become negative.
- const int round_off = ROUND_OFF;
- const int round_off_s1 = round_off;
-
- float sum = 0;
- for (int p = -4; p <= 4; p++) {
- for (int o = -4; o <= 4; o++) {
- int s2o = o;
- int s2p = p;
-
- //Get X,Y ranges and clamp
- // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:
- int xmin = (l - 4 - s2o + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)
- int ymin = (m - 4 - s2p + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)
-
- // Same here:
- int xmax = (l - 4 - s2o + round_off_s1) - round_off; // floor (l - 4 - s2o)
- int ymax = (m - 4 - s2p + round_off_s1) - round_off; // floor (m - 4 - s2p)
-
- if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {
- xmin = max(0,xmin);
- xmax = min(SIZE_3(gradOutput)-1,xmax);
-
- ymin = max(0,ymin);
- ymax = min(SIZE_2(gradOutput)-1,ymax);
-
- // Get rbot0 data:
- int idxbot0 = ((intSample * SIZE_1(rbot0) + (m-s2p)) * SIZE_2(rbot0) + (l-s2o)) * SIZE_3(rbot0) + n;
- float bot0tmp = rbot0[idxbot0]; // rbot1[l+s2o,m+s2p,n]
-
- // Index offset for gradOutput in following loops:
- int op = (p+4) * 9 + (o+4); // index[o,p]
- int idxopoffset = (intSample * SIZE_1(gradOutput) + op);
-
- for (int y = ymin; y <= ymax; y++) {
- for (int x = xmin; x <= xmax; x++) {
- int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]
- sum += gradOutput[idxgradOutput] * bot0tmp;
- }
- }
- }
- }
- }
- const int sumelems = SIZE_1(gradSecond);
- const int bot1index = ((n * SIZE_2(gradSecond)) + (m-4)) * SIZE_3(gradSecond) + (l-4);
- gradSecond[bot1index + intSample*SIZE_1(gradSecond)*SIZE_2(gradSecond)*SIZE_3(gradSecond)] = sum / (float)sumelems;
- } }
-'''
-
-def cupy_kernel(strFunction, objVariables):
- strKernel = globals()[strFunction]
-
- while True:
- objMatch = re.search('(SIZE_)([0-4])(\()([^\)]*)(\))', strKernel)
-
- if objMatch is None:
- break
- # end
-
- intArg = int(objMatch.group(2))
-
- strTensor = objMatch.group(4)
- intSizes = objVariables[strTensor].size()
-
- strKernel = strKernel.replace(objMatch.group(), str(intSizes[intArg]))
- # end
-
- while True:
- objMatch = re.search('(VALUE_)([0-4])(\()([^\)]+)(\))', strKernel)
-
- if objMatch is None:
- break
- # end
-
- intArgs = int(objMatch.group(2))
- strArgs = objMatch.group(4).split(',')
-
- strTensor = strArgs[0]
- intStrides = objVariables[strTensor].stride()
- strIndex = [ '((' + strArgs[intArg + 1].replace('{', '(').replace('}', ')').strip() + ')*' + str(intStrides[intArg]) + ')' for intArg in range(intArgs) ]
-
- strKernel = strKernel.replace(objMatch.group(0), strTensor + '[' + str.join('+', strIndex) + ']')
- # end
-
- return strKernel
-# end
-
[email protected](for_each_device=True)
-def cupy_launch(strFunction, strKernel):
- return cupy.RawKernel(strKernel, strFunction)
-# end
-
-class _FunctionCorrelation(torch.autograd.Function):
- @staticmethod
- def forward(self, first, second):
- rbot0 = first.new_zeros([ first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1] ])
- rbot1 = first.new_zeros([ first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1] ])
-
- self.save_for_backward(first, second, rbot0, rbot1)
-
- first = first.contiguous(); assert(first.is_cuda == True)
- second = second.contiguous(); assert(second.is_cuda == True)
-
- output = first.new_zeros([ first.shape[0], 81, first.shape[2], first.shape[3] ])
-
- if first.is_cuda == True:
- n = first.shape[2] * first.shape[3]
- cupy_launch('kernel_Correlation_rearrange', cupy_kernel('kernel_Correlation_rearrange', {
- 'input': first,
- 'output': rbot0
- }))(
- grid=tuple([ int((n + 16 - 1) / 16), first.shape[1], first.shape[0] ]),
- block=tuple([ 16, 1, 1 ]),
- args=[ n, first.data_ptr(), rbot0.data_ptr() ]
- )
-
- n = second.shape[2] * second.shape[3]
- cupy_launch('kernel_Correlation_rearrange', cupy_kernel('kernel_Correlation_rearrange', {
- 'input': second,
- 'output': rbot1
- }))(
- grid=tuple([ int((n + 16 - 1) / 16), second.shape[1], second.shape[0] ]),
- block=tuple([ 16, 1, 1 ]),
- args=[ n, second.data_ptr(), rbot1.data_ptr() ]
- )
-
- n = output.shape[1] * output.shape[2] * output.shape[3]
- cupy_launch('kernel_Correlation_updateOutput', cupy_kernel('kernel_Correlation_updateOutput', {
- 'rbot0': rbot0,
- 'rbot1': rbot1,
- 'top': output
- }))(
- grid=tuple([ output.shape[3], output.shape[2], output.shape[0] ]),
- block=tuple([ 32, 1, 1 ]),
- shared_mem=first.shape[1] * 4,
- args=[ n, rbot0.data_ptr(), rbot1.data_ptr(), output.data_ptr() ]
- )
-
- elif first.is_cuda == False:
- raise NotImplementedError()
-
- # end
-
- return output
- # end
-
- @staticmethod
- def backward(self, gradOutput):
- first, second, rbot0, rbot1 = self.saved_tensors
-
- gradOutput = gradOutput.contiguous(); assert(gradOutput.is_cuda == True)
-
- gradFirst = first.new_zeros([ first.shape[0], first.shape[1], first.shape[2], first.shape[3] ]) if self.needs_input_grad[0] == True else None
- gradSecond = first.new_zeros([ first.shape[0], first.shape[1], first.shape[2], first.shape[3] ]) if self.needs_input_grad[1] == True else None
-
- if first.is_cuda == True:
- if gradFirst is not None:
- for intSample in range(first.shape[0]):
- n = first.shape[1] * first.shape[2] * first.shape[3]
- cupy_launch('kernel_Correlation_updateGradFirst', cupy_kernel('kernel_Correlation_updateGradFirst', {
- 'rbot0': rbot0,
- 'rbot1': rbot1,
- 'gradOutput': gradOutput,
- 'gradFirst': gradFirst,
- 'gradSecond': None
- }))(
- grid=tuple([ int((n + 512 - 1) / 512), 1, 1 ]),
- block=tuple([ 512, 1, 1 ]),
- args=[ n, intSample, rbot0.data_ptr(), rbot1.data_ptr(), gradOutput.data_ptr(), gradFirst.data_ptr(), None ]
- )
- # end
- # end
-
- if gradSecond is not None:
- for intSample in range(first.shape[0]):
- n = first.shape[1] * first.shape[2] * first.shape[3]
- cupy_launch('kernel_Correlation_updateGradSecond', cupy_kernel('kernel_Correlation_updateGradSecond', {
- 'rbot0': rbot0,
- 'rbot1': rbot1,
- 'gradOutput': gradOutput,
- 'gradFirst': None,
- 'gradSecond': gradSecond
- }))(
- grid=tuple([ int((n + 512 - 1) / 512), 1, 1 ]),
- block=tuple([ 512, 1, 1 ]),
- args=[ n, intSample, rbot0.data_ptr(), rbot1.data_ptr(), gradOutput.data_ptr(), None, gradSecond.data_ptr() ]
- )
- # end
- # end
-
- elif first.is_cuda == False:
- raise NotImplementedError()
-
- # end
-
- return gradFirst, gradSecond
- # end
-# end
-
-def FunctionCorrelation(tenFirst, tenSecond):
- return _FunctionCorrelation.apply(tenFirst, tenSecond)
-# end
-
-class ModuleCorrelation(torch.nn.Module):
- def __init__(self):
- super(ModuleCorrelation, self).__init__()
- # end
-
- def forward(self, tenFirst, tenSecond):
- return _FunctionCorrelation.apply(tenFirst, tenSecond)
- # end
-# end
\ No newline at end of file
+#!/usr/bin/env python
+
+import re
+
+import cupy
+import torch
+
+kernel_Correlation_rearrange = """
+ extern "C" __global__ void kernel_Correlation_rearrange(
+ const int n,
+ const float* input,
+ float* output
+ ) {
+ int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x;
+
+ if (intIndex >= n) {
+ return;
+ }
+
+ int intSample = blockIdx.z;
+ int intChannel = blockIdx.y;
+
+ float fltValue = input[(((intSample * SIZE_1(input)) + intChannel) * SIZE_2(input) * SIZE_3(input)) + intIndex];
+
+ __syncthreads();
+
+ int intPaddedY = (intIndex / SIZE_3(input)) + 4;
+ int intPaddedX = (intIndex % SIZE_3(input)) + 4;
+ int intRearrange = ((SIZE_3(input) + 8) * intPaddedY) + intPaddedX;
+
+ output[(((intSample * SIZE_1(output) * SIZE_2(output)) + intRearrange) * SIZE_1(input)) + intChannel] = fltValue;
+ }
+"""
+
+kernel_Correlation_updateOutput = """
+ extern "C" __global__ void kernel_Correlation_updateOutput(
+ const int n,
+ const float* rbot0,
+ const float* rbot1,
+ float* top
+ ) {
+ extern __shared__ char patch_data_char[];
+
+ float *patch_data = (float *)patch_data_char;
+
+ // First (upper left) position of kernel upper-left corner in current center position of neighborhood in image 1
+ int x1 = blockIdx.x + 4;
+ int y1 = blockIdx.y + 4;
+ int item = blockIdx.z;
+ int ch_off = threadIdx.x;
+
+ // Load 3D patch into shared shared memory
+ for (int j = 0; j < 1; j++) { // HEIGHT
+ for (int i = 0; i < 1; i++) { // WIDTH
+ int ji_off = (j + i) * SIZE_3(rbot0);
+ for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS
+ int idx1 = ((item * SIZE_1(rbot0) + y1+j) * SIZE_2(rbot0) + x1+i) * SIZE_3(rbot0) + ch;
+ int idxPatchData = ji_off + ch;
+ patch_data[idxPatchData] = rbot0[idx1];
+ }
+ }
+ }
+
+ __syncthreads();
+
+ __shared__ float sum[32];
+
+ // Compute correlation
+ for (int top_channel = 0; top_channel < SIZE_1(top); top_channel++) {
+ sum[ch_off] = 0;
+
+ int s2o = top_channel % 9 - 4;
+ int s2p = top_channel / 9 - 4;
+
+ for (int j = 0; j < 1; j++) { // HEIGHT
+ for (int i = 0; i < 1; i++) { // WIDTH
+ int ji_off = (j + i) * SIZE_3(rbot0);
+ for (int ch = ch_off; ch < SIZE_3(rbot0); ch += 32) { // CHANNELS
+ int x2 = x1 + s2o;
+ int y2 = y1 + s2p;
+
+ int idxPatchData = ji_off + ch;
+ int idx2 = ((item * SIZE_1(rbot0) + y2+j) * SIZE_2(rbot0) + x2+i) * SIZE_3(rbot0) + ch;
+
+ sum[ch_off] += patch_data[idxPatchData] * rbot1[idx2];
+ }
+ }
+ }
+
+ __syncthreads();
+
+ if (ch_off == 0) {
+ float total_sum = 0;
+ for (int idx = 0; idx < 32; idx++) {
+ total_sum += sum[idx];
+ }
+ const int sumelems = SIZE_3(rbot0);
+ const int index = ((top_channel*SIZE_2(top) + blockIdx.y)*SIZE_3(top))+blockIdx.x;
+ top[index + item*SIZE_1(top)*SIZE_2(top)*SIZE_3(top)] = total_sum / (float)sumelems;
+ }
+ }
+ }
+"""
+
+kernel_Correlation_updateGradFirst = """
+ #define ROUND_OFF 50000
+
+ extern "C" __global__ void kernel_Correlation_updateGradFirst(
+ const int n,
+ const int intSample,
+ const float* rbot0,
+ const float* rbot1,
+ const float* gradOutput,
+ float* gradFirst,
+ float* gradSecond
+ ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {
+ int n = intIndex % SIZE_1(gradFirst); // channels
+ int l = (intIndex / SIZE_1(gradFirst)) % SIZE_3(gradFirst) + 4; // w-pos
+ int m = (intIndex / SIZE_1(gradFirst) / SIZE_3(gradFirst)) % SIZE_2(gradFirst) + 4; // h-pos
+
+ // round_off is a trick to enable integer division with ceil, even for negative numbers
+ // We use a large offset, for the inner part not to become negative.
+ const int round_off = ROUND_OFF;
+ const int round_off_s1 = round_off;
+
+ // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:
+ int xmin = (l - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)
+ int ymin = (m - 4 + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4)
+
+ // Same here:
+ int xmax = (l - 4 + round_off_s1) - round_off; // floor (l - 4)
+ int ymax = (m - 4 + round_off_s1) - round_off; // floor (m - 4)
+
+ float sum = 0;
+ if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {
+ xmin = max(0,xmin);
+ xmax = min(SIZE_3(gradOutput)-1,xmax);
+
+ ymin = max(0,ymin);
+ ymax = min(SIZE_2(gradOutput)-1,ymax);
+
+ for (int p = -4; p <= 4; p++) {
+ for (int o = -4; o <= 4; o++) {
+ // Get rbot1 data:
+ int s2o = o;
+ int s2p = p;
+ int idxbot1 = ((intSample * SIZE_1(rbot0) + (m+s2p)) * SIZE_2(rbot0) + (l+s2o)) * SIZE_3(rbot0) + n;
+ float bot1tmp = rbot1[idxbot1]; // rbot1[l+s2o,m+s2p,n]
+
+ // Index offset for gradOutput in following loops:
+ int op = (p+4) * 9 + (o+4); // index[o,p]
+ int idxopoffset = (intSample * SIZE_1(gradOutput) + op);
+
+ for (int y = ymin; y <= ymax; y++) {
+ for (int x = xmin; x <= xmax; x++) {
+ int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]
+ sum += gradOutput[idxgradOutput] * bot1tmp;
+ }
+ }
+ }
+ }
+ }
+ const int sumelems = SIZE_1(gradFirst);
+ const int bot0index = ((n * SIZE_2(gradFirst)) + (m-4)) * SIZE_3(gradFirst) + (l-4);
+ gradFirst[bot0index + intSample*SIZE_1(gradFirst)*SIZE_2(gradFirst)*SIZE_3(gradFirst)] = sum / (float)sumelems;
+ } }
+"""
+
+kernel_Correlation_updateGradSecond = """
+ #define ROUND_OFF 50000
+
+ extern "C" __global__ void kernel_Correlation_updateGradSecond(
+ const int n,
+ const int intSample,
+ const float* rbot0,
+ const float* rbot1,
+ const float* gradOutput,
+ float* gradFirst,
+ float* gradSecond
+ ) { for (int intIndex = (blockIdx.x * blockDim.x) + threadIdx.x; intIndex < n; intIndex += blockDim.x * gridDim.x) {
+ int n = intIndex % SIZE_1(gradSecond); // channels
+ int l = (intIndex / SIZE_1(gradSecond)) % SIZE_3(gradSecond) + 4; // w-pos
+ int m = (intIndex / SIZE_1(gradSecond) / SIZE_3(gradSecond)) % SIZE_2(gradSecond) + 4; // h-pos
+
+ // round_off is a trick to enable integer division with ceil, even for negative numbers
+ // We use a large offset, for the inner part not to become negative.
+ const int round_off = ROUND_OFF;
+ const int round_off_s1 = round_off;
+
+ float sum = 0;
+ for (int p = -4; p <= 4; p++) {
+ for (int o = -4; o <= 4; o++) {
+ int s2o = o;
+ int s2p = p;
+
+ //Get X,Y ranges and clamp
+ // We add round_off before_s1 the int division and subtract round_off after it, to ensure the formula matches ceil behavior:
+ int xmin = (l - 4 - s2o + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)
+ int ymin = (m - 4 - s2p + round_off_s1 - 1) + 1 - round_off; // ceil (l - 4 - s2o)
+
+ // Same here:
+ int xmax = (l - 4 - s2o + round_off_s1) - round_off; // floor (l - 4 - s2o)
+ int ymax = (m - 4 - s2p + round_off_s1) - round_off; // floor (m - 4 - s2p)
+
+ if (xmax>=0 && ymax>=0 && (xmin<=SIZE_3(gradOutput)-1) && (ymin<=SIZE_2(gradOutput)-1)) {
+ xmin = max(0,xmin);
+ xmax = min(SIZE_3(gradOutput)-1,xmax);
+
+ ymin = max(0,ymin);
+ ymax = min(SIZE_2(gradOutput)-1,ymax);
+
+ // Get rbot0 data:
+ int idxbot0 = ((intSample * SIZE_1(rbot0) + (m-s2p)) * SIZE_2(rbot0) + (l-s2o)) * SIZE_3(rbot0) + n;
+ float bot0tmp = rbot0[idxbot0]; // rbot1[l+s2o,m+s2p,n]
+
+ // Index offset for gradOutput in following loops:
+ int op = (p+4) * 9 + (o+4); // index[o,p]
+ int idxopoffset = (intSample * SIZE_1(gradOutput) + op);
+
+ for (int y = ymin; y <= ymax; y++) {
+ for (int x = xmin; x <= xmax; x++) {
+ int idxgradOutput = (idxopoffset * SIZE_2(gradOutput) + y) * SIZE_3(gradOutput) + x; // gradOutput[x,y,o,p]
+ sum += gradOutput[idxgradOutput] * bot0tmp;
+ }
+ }
+ }
+ }
+ }
+ const int sumelems = SIZE_1(gradSecond);
+ const int bot1index = ((n * SIZE_2(gradSecond)) + (m-4)) * SIZE_3(gradSecond) + (l-4);
+ gradSecond[bot1index + intSample*SIZE_1(gradSecond)*SIZE_2(gradSecond)*SIZE_3(gradSecond)] = sum / (float)sumelems;
+ } }
+"""
+
+
+def cupy_kernel(strFunction, objVariables):
+ strKernel = globals()[strFunction]
+
+ while True:
+ objMatch = re.search("(SIZE_)([0-4])(\()([^\)]*)(\))", strKernel)
+
+ if objMatch is None:
+ break
+ # end
+
+ intArg = int(objMatch.group(2))
+
+ strTensor = objMatch.group(4)
+ intSizes = objVariables[strTensor].size()
+
+ strKernel = strKernel.replace(objMatch.group(), str(intSizes[intArg]))
+ # end
+
+ while True:
+ objMatch = re.search("(VALUE_)([0-4])(\()([^\)]+)(\))", strKernel)
+
+ if objMatch is None:
+ break
+ # end
+
+ intArgs = int(objMatch.group(2))
+ strArgs = objMatch.group(4).split(",")
+
+ strTensor = strArgs[0]
+ intStrides = objVariables[strTensor].stride()
+ strIndex = [
+ "(("
+ + strArgs[intArg + 1].replace("{", "(").replace("}", ")").strip()
+ + ")*"
+ + str(intStrides[intArg])
+ + ")"
+ for intArg in range(intArgs)
+ ]
+
+ strKernel = strKernel.replace(objMatch.group(0), strTensor + "[" + str.join("+", strIndex) + "]")
+ # end
+
+ return strKernel
+
+
+# end
+
+
[email protected](for_each_device=True)
+def cupy_launch(strFunction, strKernel):
+ return cupy.RawKernel(strKernel, strFunction)
+
+
+# end
+
+
+class _FunctionCorrelation(torch.autograd.Function):
+ @staticmethod
+ def forward(self, first, second):
+ rbot0 = first.new_zeros([first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1]])
+ rbot1 = first.new_zeros([first.shape[0], first.shape[2] + 8, first.shape[3] + 8, first.shape[1]])
+
+ self.save_for_backward(first, second, rbot0, rbot1)
+
+ first = first.contiguous()
+ assert first.is_cuda == True
+ second = second.contiguous()
+ assert second.is_cuda == True
+
+ output = first.new_zeros([first.shape[0], 81, first.shape[2], first.shape[3]])
+
+ if first.is_cuda == True:
+ n = first.shape[2] * first.shape[3]
+ cupy_launch(
+ "kernel_Correlation_rearrange",
+ cupy_kernel("kernel_Correlation_rearrange", {"input": first, "output": rbot0}),
+ )(
+ grid=tuple([int((n + 16 - 1) / 16), first.shape[1], first.shape[0]]),
+ block=tuple([16, 1, 1]),
+ args=[n, first.data_ptr(), rbot0.data_ptr()],
+ )
+
+ n = second.shape[2] * second.shape[3]
+ cupy_launch(
+ "kernel_Correlation_rearrange",
+ cupy_kernel("kernel_Correlation_rearrange", {"input": second, "output": rbot1}),
+ )(
+ grid=tuple([int((n + 16 - 1) / 16), second.shape[1], second.shape[0]]),
+ block=tuple([16, 1, 1]),
+ args=[n, second.data_ptr(), rbot1.data_ptr()],
+ )
+
+ n = output.shape[1] * output.shape[2] * output.shape[3]
+ cupy_launch(
+ "kernel_Correlation_updateOutput",
+ cupy_kernel("kernel_Correlation_updateOutput", {"rbot0": rbot0, "rbot1": rbot1, "top": output}),
+ )(
+ grid=tuple([output.shape[3], output.shape[2], output.shape[0]]),
+ block=tuple([32, 1, 1]),
+ shared_mem=first.shape[1] * 4,
+ args=[n, rbot0.data_ptr(), rbot1.data_ptr(), output.data_ptr()],
+ )
+
+ elif first.is_cuda == False:
+ raise NotImplementedError()
+
+ # end
+
+ return output
+
+ # end
+
+ @staticmethod
+ def backward(self, gradOutput):
+ first, second, rbot0, rbot1 = self.saved_tensors
+
+ gradOutput = gradOutput.contiguous()
+ assert gradOutput.is_cuda == True
+
+ gradFirst = (
+ first.new_zeros([first.shape[0], first.shape[1], first.shape[2], first.shape[3]])
+ if self.needs_input_grad[0] == True
+ else None
+ )
+ gradSecond = (
+ first.new_zeros([first.shape[0], first.shape[1], first.shape[2], first.shape[3]])
+ if self.needs_input_grad[1] == True
+ else None
+ )
+
+ if first.is_cuda == True:
+ if gradFirst is not None:
+ for intSample in range(first.shape[0]):
+ n = first.shape[1] * first.shape[2] * first.shape[3]
+ cupy_launch(
+ "kernel_Correlation_updateGradFirst",
+ cupy_kernel(
+ "kernel_Correlation_updateGradFirst",
+ {
+ "rbot0": rbot0,
+ "rbot1": rbot1,
+ "gradOutput": gradOutput,
+ "gradFirst": gradFirst,
+ "gradSecond": None,
+ },
+ ),
+ )(
+ grid=tuple([int((n + 512 - 1) / 512), 1, 1]),
+ block=tuple([512, 1, 1]),
+ args=[
+ n,
+ intSample,
+ rbot0.data_ptr(),
+ rbot1.data_ptr(),
+ gradOutput.data_ptr(),
+ gradFirst.data_ptr(),
+ None,
+ ],
+ )
+ # end
+ # end
+
+ if gradSecond is not None:
+ for intSample in range(first.shape[0]):
+ n = first.shape[1] * first.shape[2] * first.shape[3]
+ cupy_launch(
+ "kernel_Correlation_updateGradSecond",
+ cupy_kernel(
+ "kernel_Correlation_updateGradSecond",
+ {
+ "rbot0": rbot0,
+ "rbot1": rbot1,
+ "gradOutput": gradOutput,
+ "gradFirst": None,
+ "gradSecond": gradSecond,
+ },
+ ),
+ )(
+ grid=tuple([int((n + 512 - 1) / 512), 1, 1]),
+ block=tuple([512, 1, 1]),
+ args=[
+ n,
+ intSample,
+ rbot0.data_ptr(),
+ rbot1.data_ptr(),
+ gradOutput.data_ptr(),
+ None,
+ gradSecond.data_ptr(),
+ ],
+ )
+ # end
+ # end
+
+ elif first.is_cuda == False:
+ raise NotImplementedError()
+
+ # end
+
+ return gradFirst, gradSecond
+
+ # end
+
+
+# end
+
+
+def FunctionCorrelation(tenFirst, tenSecond):
+ return _FunctionCorrelation.apply(tenFirst, tenSecond)
+
+
+# end
+
+
+class ModuleCorrelation(torch.nn.Module):
+ def __init__(self):
+ super(ModuleCorrelation, self).__init__()
+
+ # end
+
+ def forward(self, tenFirst, tenSecond):
+ return _FunctionCorrelation.apply(tenFirst, tenSecond)
+
+ # end
+
+
+# end
diff --git a/eval/vae/flolpips/flolpips.py b/eval/vae/flolpips/flolpips.py
index 6bd2bac..ca75f1d 100644
--- a/eval/vae/flolpips/flolpips.py
+++ b/eval/vae/flolpips/flolpips.py
@@ -1,346 +1,412 @@
-
-from __future__ import absolute_import
-import os
-import numpy as np
-import torch
-import torch.nn as nn
-from torch.autograd import Variable
-from .pretrained_networks import vgg16, alexnet, squeezenet
-import torch.nn
-import torch.nn.functional as F
-import torchvision.transforms.functional as TF
-import cv2
-
-import hashlib
-import requests
-from tqdm import tqdm
-
-from .pwcnet import Network as PWCNet
-from .utils import *
-
-URL_MAP = {"alex": "https://raw.githubusercontent.com/danier97/flolpips/main/weights/v0.1/alex.pth"}
-
-CKPT_MAP = {"alex": "alex.pth"}
-
-MD5_MAP = {"alex": "9642209e2b57a85d20f86d812320f9e6"}
-
-
-
-def spatial_average(in_tens, keepdim=True):
- return in_tens.mean([2,3],keepdim=keepdim)
-
-def mw_spatial_average(in_tens, flow, keepdim=True):
- _,_,h,w = in_tens.shape
- flow = F.interpolate(flow, (h,w), align_corners=False, mode='bilinear')
- flow_mag = torch.sqrt(flow[:,0:1]**2 + flow[:,1:2]**2)
- flow_mag = flow_mag / torch.sum(flow_mag, dim=[1,2,3], keepdim=True)
- return torch.sum(in_tens*flow_mag, dim=[2,3],keepdim=keepdim)
-
-
-def mtw_spatial_average(in_tens, flow, texture, keepdim=True):
- _,_,h,w = in_tens.shape
- flow = F.interpolate(flow, (h,w), align_corners=False, mode='bilinear')
- texture = F.interpolate(texture, (h,w), align_corners=False, mode='bilinear')
- flow_mag = torch.sqrt(flow[:,0:1]**2 + flow[:,1:2]**2)
- flow_mag = (flow_mag - flow_mag.min()) / (flow_mag.max() - flow_mag.min()) + 1e-6
- texture = (texture - texture.min()) / (texture.max() - texture.min()) + 1e-6
- weight = flow_mag / texture
- weight /= torch.sum(weight)
- return torch.sum(in_tens*weight, dim=[2,3],keepdim=keepdim)
-
-
-
-def m2w_spatial_average(in_tens, flow, keepdim=True):
- _,_,h,w = in_tens.shape
- flow = F.interpolate(flow, (h,w), align_corners=False, mode='bilinear')
- flow_mag = flow[:,0:1]**2 + flow[:,1:2]**2 # B,1,H,W
- flow_mag = flow_mag / torch.sum(flow_mag)
- return torch.sum(in_tens*flow_mag, dim=[2,3],keepdim=keepdim)
-
-def upsample(in_tens, out_HW=(64,64)): # assumes scale factor is same for H and W
- in_H, in_W = in_tens.shape[2], in_tens.shape[3]
- return nn.Upsample(size=out_HW, mode='bilinear', align_corners=False)(in_tens)
-
-
-def md5_hash(path):
- with open(path, "rb") as f:
- content = f.read()
- return hashlib.md5(content).hexdigest()
-
-def download(url, local_path, chunk_size=1024):
- os.makedirs(os.path.split(local_path)[0], exist_ok=True)
- with requests.get(url, stream=True) as r:
- total_size = int(r.headers.get("content-length", 0))
- with tqdm(total=total_size, unit="B", unit_scale=True) as pbar:
- with open(local_path, "wb") as f:
- for data in r.iter_content(chunk_size=chunk_size):
- if data:
- f.write(data)
- pbar.update(chunk_size)
-
-def get_ckpt_path(name, root, check=False):
- assert name in URL_MAP
- path = os.path.join(root, CKPT_MAP[name])
- if not os.path.exists(path) or (check and not md5_hash(path) == MD5_MAP[name]):
- print("Downloading {} model from {} to {}".format(name, URL_MAP[name], path))
- download(URL_MAP[name], path)
- md5 = md5_hash(path)
- assert md5 == MD5_MAP[name], md5
- return path
-
-# Learned perceptual metric
-class LPIPS(nn.Module):
- def __init__(self, pretrained=True, net='alex', version='0.1', lpips=True, spatial=False,
- pnet_rand=False, pnet_tune=False, use_dropout=True, model_path=None, eval_mode=True, verbose=False):
- # lpips - [True] means with linear calibration on top of base network
- # pretrained - [True] means load linear weights
-
- super(LPIPS, self).__init__()
- if(verbose):
- print('Setting up [%s] perceptual loss: trunk [%s], v[%s], spatial [%s]'%
- ('LPIPS' if lpips else 'baseline', net, version, 'on' if spatial else 'off'))
-
- self.pnet_type = net
- self.pnet_tune = pnet_tune
- self.pnet_rand = pnet_rand
- self.spatial = spatial
- self.lpips = lpips # false means baseline of just averaging all layers
- self.version = version
- self.scaling_layer = ScalingLayer()
-
- if(self.pnet_type in ['vgg','vgg16']):
- net_type = vgg16
- self.chns = [64,128,256,512,512]
- elif(self.pnet_type=='alex'):
- net_type = alexnet
- self.chns = [64,192,384,256,256]
- elif(self.pnet_type=='squeeze'):
- net_type = squeezenet
- self.chns = [64,128,256,384,384,512,512]
- self.L = len(self.chns)
-
- self.net = net_type(pretrained=not self.pnet_rand, requires_grad=self.pnet_tune)
-
- if(lpips):
- self.lin0 = NetLinLayer(self.chns[0], use_dropout=use_dropout)
- self.lin1 = NetLinLayer(self.chns[1], use_dropout=use_dropout)
- self.lin2 = NetLinLayer(self.chns[2], use_dropout=use_dropout)
- self.lin3 = NetLinLayer(self.chns[3], use_dropout=use_dropout)
- self.lin4 = NetLinLayer(self.chns[4], use_dropout=use_dropout)
- self.lins = [self.lin0,self.lin1,self.lin2,self.lin3,self.lin4]
- if(self.pnet_type=='squeeze'): # 7 layers for squeezenet
- self.lin5 = NetLinLayer(self.chns[5], use_dropout=use_dropout)
- self.lin6 = NetLinLayer(self.chns[6], use_dropout=use_dropout)
- self.lins+=[self.lin5,self.lin6]
- self.lins = nn.ModuleList(self.lins)
-
- if(pretrained):
- self.load_from_pretrained(version, net)
- if(verbose):
- print('Loaded model from: %s'%model_path)
-
- if(eval_mode):
- self.eval()
-
- def load_from_pretrained(self, version, net):
- ckpt = get_ckpt_path(net, "pretrained_models/flolpips/weights/v%s"%(version))
- self.load_state_dict(torch.load(ckpt, map_location='cpu'), strict=False)
-
- def forward(self, in0, in1, retPerLayer=False, normalize=False):
- if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]
- in0 = 2 * in0 - 1
- in1 = 2 * in1 - 1
-
- # v0.0 - original release had a bug, where input was not scaled
- in0_input, in1_input = (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version=='0.1' else (in0, in1)
- outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)
- feats0, feats1, diffs = {}, {}, {}
-
- for kk in range(self.L):
- feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])
- diffs[kk] = (feats0[kk]-feats1[kk])**2
-
- if(self.lpips):
- if(self.spatial):
- res = [upsample(self.lins[kk](diffs[kk]), out_HW=in0.shape[2:]) for kk in range(self.L)]
- else:
- res = [spatial_average(self.lins[kk](diffs[kk]), keepdim=True) for kk in range(self.L)]
- else:
- if(self.spatial):
- res = [upsample(diffs[kk].sum(dim=1,keepdim=True), out_HW=in0.shape[2:]) for kk in range(self.L)]
- else:
- res = [spatial_average(diffs[kk].sum(dim=1,keepdim=True), keepdim=True) for kk in range(self.L)]
-
- # val = res[0]
- # for l in range(1,self.L):
- # val += res[l]
- # print(val)
-
- # a = spatial_average(self.lins[kk](diffs[kk]), keepdim=True)
- # b = torch.max(self.lins[kk](feats0[kk]**2))
- # for kk in range(self.L):
- # a += spatial_average(self.lins[kk](diffs[kk]), keepdim=True)
- # b = torch.max(b,torch.max(self.lins[kk](feats0[kk]**2)))
- # a = a/self.L
- # from IPython import embed
- # embed()
- # return 10*torch.log10(b/a)
-
- # if(retPerLayer):
- # return (val, res)
- # else:
- return torch.sum(torch.cat(res, 1), dim=(1,2,3), keepdims=False)
-
-
-class ScalingLayer(nn.Module):
- def __init__(self):
- super(ScalingLayer, self).__init__()
- self.register_buffer('shift', torch.Tensor([-.030,-.088,-.188])[None,:,None,None])
- self.register_buffer('scale', torch.Tensor([.458,.448,.450])[None,:,None,None])
-
- def forward(self, inp):
- return (inp - self.shift) / self.scale
-
-
-class NetLinLayer(nn.Module):
- ''' A single linear layer which does a 1x1 conv '''
- def __init__(self, chn_in, chn_out=1, use_dropout=False):
- super(NetLinLayer, self).__init__()
-
- layers = [nn.Dropout(),] if(use_dropout) else []
- layers += [nn.Conv2d(chn_in, chn_out, 1, stride=1, padding=0, bias=False),]
- self.model = nn.Sequential(*layers)
-
- def forward(self, x):
- return self.model(x)
-
-class Dist2LogitLayer(nn.Module):
- ''' takes 2 distances, puts through fc layers, spits out value between [0,1] (if use_sigmoid is True) '''
- def __init__(self, chn_mid=32, use_sigmoid=True):
- super(Dist2LogitLayer, self).__init__()
-
- layers = [nn.Conv2d(5, chn_mid, 1, stride=1, padding=0, bias=True),]
- layers += [nn.LeakyReLU(0.2,True),]
- layers += [nn.Conv2d(chn_mid, chn_mid, 1, stride=1, padding=0, bias=True),]
- layers += [nn.LeakyReLU(0.2,True),]
- layers += [nn.Conv2d(chn_mid, 1, 1, stride=1, padding=0, bias=True),]
- if(use_sigmoid):
- layers += [nn.Sigmoid(),]
- self.model = nn.Sequential(*layers)
-
- def forward(self,d0,d1,eps=0.1):
- return self.model.forward(torch.cat((d0,d1,d0-d1,d0/(d1+eps),d1/(d0+eps)),dim=1))
-
-class BCERankingLoss(nn.Module):
- def __init__(self, chn_mid=32):
- super(BCERankingLoss, self).__init__()
- self.net = Dist2LogitLayer(chn_mid=chn_mid)
- # self.parameters = list(self.net.parameters())
- self.loss = torch.nn.BCELoss()
-
- def forward(self, d0, d1, judge):
- per = (judge+1.)/2.
- self.logit = self.net.forward(d0,d1)
- return self.loss(self.logit, per)
-
-# L2, DSSIM metrics
-class FakeNet(nn.Module):
- def __init__(self, use_gpu=True, colorspace='Lab'):
- super(FakeNet, self).__init__()
- self.use_gpu = use_gpu
- self.colorspace = colorspace
-
-class L2(FakeNet):
- def forward(self, in0, in1, retPerLayer=None):
- assert(in0.size()[0]==1) # currently only supports batchSize 1
-
- if(self.colorspace=='RGB'):
- (N,C,X,Y) = in0.size()
- value = torch.mean(torch.mean(torch.mean((in0-in1)**2,dim=1).view(N,1,X,Y),dim=2).view(N,1,1,Y),dim=3).view(N)
- return value
- elif(self.colorspace=='Lab'):
- value = l2(tensor2np(tensor2tensorlab(in0.data,to_norm=False)),
- tensor2np(tensor2tensorlab(in1.data,to_norm=False)), range=100.).astype('float')
- ret_var = Variable( torch.Tensor((value,) ) )
- if(self.use_gpu):
- ret_var = ret_var.cuda()
- return ret_var
-
-class DSSIM(FakeNet):
-
- def forward(self, in0, in1, retPerLayer=None):
- assert(in0.size()[0]==1) # currently only supports batchSize 1
-
- if(self.colorspace=='RGB'):
- value = dssim(1.*tensor2im(in0.data), 1.*tensor2im(in1.data), range=255.).astype('float')
- elif(self.colorspace=='Lab'):
- value = dssim(tensor2np(tensor2tensorlab(in0.data,to_norm=False)),
- tensor2np(tensor2tensorlab(in1.data,to_norm=False)), range=100.).astype('float')
- ret_var = Variable( torch.Tensor((value,) ) )
- if(self.use_gpu):
- ret_var = ret_var.cuda()
- return ret_var
-
-def print_network(net):
- num_params = 0
- for param in net.parameters():
- num_params += param.numel()
- print('Network',net)
- print('Total number of parameters: %d' % num_params)
-
-
-class FloLPIPS(LPIPS):
- def __init__(self, pretrained=True, net='alex', version='0.1', lpips=True, spatial=False, pnet_rand=False, pnet_tune=False, use_dropout=True, model_path=None, eval_mode=True, verbose=False):
- super(FloLPIPS, self).__init__(pretrained, net, version, lpips, spatial, pnet_rand, pnet_tune, use_dropout, model_path, eval_mode, verbose)
-
- def forward(self, in0, in1, flow, retPerLayer=False, normalize=False):
- if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]
- in0 = 2 * in0 - 1
- in1 = 2 * in1 - 1
-
- in0_input, in1_input = (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version=='0.1' else (in0, in1)
- outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)
- feats0, feats1, diffs = {}, {}, {}
-
- for kk in range(self.L):
- feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])
- diffs[kk] = (feats0[kk]-feats1[kk])**2
-
- res = [mw_spatial_average(self.lins[kk](diffs[kk]), flow, keepdim=True) for kk in range(self.L)]
-
- return torch.sum(torch.cat(res, 1), dim=(1,2,3), keepdims=False)
-
-
-
-
-
-class Flolpips(nn.Module):
- def __init__(self):
- super(Flolpips, self).__init__()
- self.loss_fn = FloLPIPS(net='alex',version='0.1')
- self.flownet = PWCNet()
-
- @torch.no_grad()
- def forward(self, I0, I1, frame_dis, frame_ref):
- """
- args:
- I0: first frame of the triplet, shape: [B, C, H, W]
- I1: third frame of the triplet, shape: [B, C, H, W]
- frame_dis: prediction of the intermediate frame, shape: [B, C, H, W]
- frame_ref: ground-truth of the intermediate frame, shape: [B, C, H, W]
- """
- assert I0.size() == I1.size() == frame_dis.size() == frame_ref.size(), \
- "the 4 input tensors should have same size"
-
- flow_ref = self.flownet(frame_ref, I0)
- flow_dis = self.flownet(frame_dis, I0)
- flow_diff = flow_ref - flow_dis
- flolpips_wrt_I0 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)
-
- flow_ref = self.flownet(frame_ref, I1)
- flow_dis = self.flownet(frame_dis, I1)
- flow_diff = flow_ref - flow_dis
- flolpips_wrt_I1 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)
-
- flolpips = (flolpips_wrt_I0 + flolpips_wrt_I1) / 2
- return flolpips
\ No newline at end of file
+from __future__ import absolute_import
+
+import hashlib
+import os
+
+import requests
+import torch
+import torch.nn
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Variable
+from tqdm import tqdm
+
+from .pretrained_networks import alexnet, squeezenet, vgg16
+from .pwcnet import Network as PWCNet
+from .utils import *
+
+URL_MAP = {"alex": "https://raw.githubusercontent.com/danier97/flolpips/main/weights/v0.1/alex.pth"}
+
+CKPT_MAP = {"alex": "alex.pth"}
+
+MD5_MAP = {"alex": "9642209e2b57a85d20f86d812320f9e6"}
+
+
+def spatial_average(in_tens, keepdim=True):
+ return in_tens.mean([2, 3], keepdim=keepdim)
+
+
+def mw_spatial_average(in_tens, flow, keepdim=True):
+ _, _, h, w = in_tens.shape
+ flow = F.interpolate(flow, (h, w), align_corners=False, mode="bilinear")
+ flow_mag = torch.sqrt(flow[:, 0:1] ** 2 + flow[:, 1:2] ** 2)
+ flow_mag = flow_mag / torch.sum(flow_mag, dim=[1, 2, 3], keepdim=True)
+ return torch.sum(in_tens * flow_mag, dim=[2, 3], keepdim=keepdim)
+
+
+def mtw_spatial_average(in_tens, flow, texture, keepdim=True):
+ _, _, h, w = in_tens.shape
+ flow = F.interpolate(flow, (h, w), align_corners=False, mode="bilinear")
+ texture = F.interpolate(texture, (h, w), align_corners=False, mode="bilinear")
+ flow_mag = torch.sqrt(flow[:, 0:1] ** 2 + flow[:, 1:2] ** 2)
+ flow_mag = (flow_mag - flow_mag.min()) / (flow_mag.max() - flow_mag.min()) + 1e-6
+ texture = (texture - texture.min()) / (texture.max() - texture.min()) + 1e-6
+ weight = flow_mag / texture
+ weight /= torch.sum(weight)
+ return torch.sum(in_tens * weight, dim=[2, 3], keepdim=keepdim)
+
+
+def m2w_spatial_average(in_tens, flow, keepdim=True):
+ _, _, h, w = in_tens.shape
+ flow = F.interpolate(flow, (h, w), align_corners=False, mode="bilinear")
+ flow_mag = flow[:, 0:1] ** 2 + flow[:, 1:2] ** 2 # B,1,H,W
+ flow_mag = flow_mag / torch.sum(flow_mag)
+ return torch.sum(in_tens * flow_mag, dim=[2, 3], keepdim=keepdim)
+
+
+def upsample(in_tens, out_HW=(64, 64)): # assumes scale factor is same for H and W
+ in_H, in_W = in_tens.shape[2], in_tens.shape[3]
+ return nn.Upsample(size=out_HW, mode="bilinear", align_corners=False)(in_tens)
+
+
+def md5_hash(path):
+ with open(path, "rb") as f:
+ content = f.read()
+ return hashlib.md5(content).hexdigest()
+
+
+def download(url, local_path, chunk_size=1024):
+ os.makedirs(os.path.split(local_path)[0], exist_ok=True)
+ with requests.get(url, stream=True) as r:
+ total_size = int(r.headers.get("content-length", 0))
+ with tqdm(total=total_size, unit="B", unit_scale=True) as pbar:
+ with open(local_path, "wb") as f:
+ for data in r.iter_content(chunk_size=chunk_size):
+ if data:
+ f.write(data)
+ pbar.update(chunk_size)
+
+
+def get_ckpt_path(name, root, check=False):
+ assert name in URL_MAP
+ path = os.path.join(root, CKPT_MAP[name])
+ if not os.path.exists(path) or (check and not md5_hash(path) == MD5_MAP[name]):
+ print("Downloading {} model from {} to {}".format(name, URL_MAP[name], path))
+ download(URL_MAP[name], path)
+ md5 = md5_hash(path)
+ assert md5 == MD5_MAP[name], md5
+ return path
+
+
+# Learned perceptual metric
+class LPIPS(nn.Module):
+ def __init__(
+ self,
+ pretrained=True,
+ net="alex",
+ version="0.1",
+ lpips=True,
+ spatial=False,
+ pnet_rand=False,
+ pnet_tune=False,
+ use_dropout=True,
+ model_path=None,
+ eval_mode=True,
+ verbose=False,
+ ):
+ # lpips - [True] means with linear calibration on top of base network
+ # pretrained - [True] means load linear weights
+
+ super(LPIPS, self).__init__()
+ if verbose:
+ print(
+ "Setting up [%s] perceptual loss: trunk [%s], v[%s], spatial [%s]"
+ % ("LPIPS" if lpips else "baseline", net, version, "on" if spatial else "off")
+ )
+
+ self.pnet_type = net
+ self.pnet_tune = pnet_tune
+ self.pnet_rand = pnet_rand
+ self.spatial = spatial
+ self.lpips = lpips # false means baseline of just averaging all layers
+ self.version = version
+ self.scaling_layer = ScalingLayer()
+
+ if self.pnet_type in ["vgg", "vgg16"]:
+ net_type = vgg16
+ self.chns = [64, 128, 256, 512, 512]
+ elif self.pnet_type == "alex":
+ net_type = alexnet
+ self.chns = [64, 192, 384, 256, 256]
+ elif self.pnet_type == "squeeze":
+ net_type = squeezenet
+ self.chns = [64, 128, 256, 384, 384, 512, 512]
+ self.L = len(self.chns)
+
+ self.net = net_type(pretrained=not self.pnet_rand, requires_grad=self.pnet_tune)
+
+ if lpips:
+ self.lin0 = NetLinLayer(self.chns[0], use_dropout=use_dropout)
+ self.lin1 = NetLinLayer(self.chns[1], use_dropout=use_dropout)
+ self.lin2 = NetLinLayer(self.chns[2], use_dropout=use_dropout)
+ self.lin3 = NetLinLayer(self.chns[3], use_dropout=use_dropout)
+ self.lin4 = NetLinLayer(self.chns[4], use_dropout=use_dropout)
+ self.lins = [self.lin0, self.lin1, self.lin2, self.lin3, self.lin4]
+ if self.pnet_type == "squeeze": # 7 layers for squeezenet
+ self.lin5 = NetLinLayer(self.chns[5], use_dropout=use_dropout)
+ self.lin6 = NetLinLayer(self.chns[6], use_dropout=use_dropout)
+ self.lins += [self.lin5, self.lin6]
+ self.lins = nn.ModuleList(self.lins)
+
+ if pretrained:
+ self.load_from_pretrained(version, net)
+ if verbose:
+ print("Loaded model from: %s" % model_path)
+
+ if eval_mode:
+ self.eval()
+
+ def load_from_pretrained(self, version, net):
+ ckpt = get_ckpt_path(net, "pretrained_models/flolpips/weights/v%s" % (version))
+ self.load_state_dict(torch.load(ckpt, map_location="cpu"), strict=False)
+
+ def forward(self, in0, in1, retPerLayer=False, normalize=False):
+ if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]
+ in0 = 2 * in0 - 1
+ in1 = 2 * in1 - 1
+
+ # v0.0 - original release had a bug, where input was not scaled
+ in0_input, in1_input = (
+ (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version == "0.1" else (in0, in1)
+ )
+ outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)
+ feats0, feats1, diffs = {}, {}, {}
+
+ for kk in range(self.L):
+ feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])
+ diffs[kk] = (feats0[kk] - feats1[kk]) ** 2
+
+ if self.lpips:
+ if self.spatial:
+ res = [upsample(self.lins[kk](diffs[kk]), out_HW=in0.shape[2:]) for kk in range(self.L)]
+ else:
+ res = [spatial_average(self.lins[kk](diffs[kk]), keepdim=True) for kk in range(self.L)]
+ else:
+ if self.spatial:
+ res = [upsample(diffs[kk].sum(dim=1, keepdim=True), out_HW=in0.shape[2:]) for kk in range(self.L)]
+ else:
+ res = [spatial_average(diffs[kk].sum(dim=1, keepdim=True), keepdim=True) for kk in range(self.L)]
+
+ # val = res[0]
+ # for l in range(1,self.L):
+ # val += res[l]
+ # print(val)
+
+ # a = spatial_average(self.lins[kk](diffs[kk]), keepdim=True)
+ # b = torch.max(self.lins[kk](feats0[kk]**2))
+ # for kk in range(self.L):
+ # a += spatial_average(self.lins[kk](diffs[kk]), keepdim=True)
+ # b = torch.max(b,torch.max(self.lins[kk](feats0[kk]**2)))
+ # a = a/self.L
+ # from IPython import embed
+ # embed()
+ # return 10*torch.log10(b/a)
+
+ # if(retPerLayer):
+ # return (val, res)
+ # else:
+ return torch.sum(torch.cat(res, 1), dim=(1, 2, 3), keepdims=False)
+
+
+class ScalingLayer(nn.Module):
+ def __init__(self):
+ super(ScalingLayer, self).__init__()
+ self.register_buffer("shift", torch.Tensor([-0.030, -0.088, -0.188])[None, :, None, None])
+ self.register_buffer("scale", torch.Tensor([0.458, 0.448, 0.450])[None, :, None, None])
+
+ def forward(self, inp):
+ return (inp - self.shift) / self.scale
+
+
+class NetLinLayer(nn.Module):
+ """A single linear layer which does a 1x1 conv"""
+
+ def __init__(self, chn_in, chn_out=1, use_dropout=False):
+ super(NetLinLayer, self).__init__()
+
+ layers = (
+ [
+ nn.Dropout(),
+ ]
+ if (use_dropout)
+ else []
+ )
+ layers += [
+ nn.Conv2d(chn_in, chn_out, 1, stride=1, padding=0, bias=False),
+ ]
+ self.model = nn.Sequential(*layers)
+
+ def forward(self, x):
+ return self.model(x)
+
+
+class Dist2LogitLayer(nn.Module):
+ """takes 2 distances, puts through fc layers, spits out value between [0,1] (if use_sigmoid is True)"""
+
+ def __init__(self, chn_mid=32, use_sigmoid=True):
+ super(Dist2LogitLayer, self).__init__()
+
+ layers = [
+ nn.Conv2d(5, chn_mid, 1, stride=1, padding=0, bias=True),
+ ]
+ layers += [
+ nn.LeakyReLU(0.2, True),
+ ]
+ layers += [
+ nn.Conv2d(chn_mid, chn_mid, 1, stride=1, padding=0, bias=True),
+ ]
+ layers += [
+ nn.LeakyReLU(0.2, True),
+ ]
+ layers += [
+ nn.Conv2d(chn_mid, 1, 1, stride=1, padding=0, bias=True),
+ ]
+ if use_sigmoid:
+ layers += [
+ nn.Sigmoid(),
+ ]
+ self.model = nn.Sequential(*layers)
+
+ def forward(self, d0, d1, eps=0.1):
+ return self.model.forward(torch.cat((d0, d1, d0 - d1, d0 / (d1 + eps), d1 / (d0 + eps)), dim=1))
+
+
+class BCERankingLoss(nn.Module):
+ def __init__(self, chn_mid=32):
+ super(BCERankingLoss, self).__init__()
+ self.net = Dist2LogitLayer(chn_mid=chn_mid)
+ # self.parameters = list(self.net.parameters())
+ self.loss = torch.nn.BCELoss()
+
+ def forward(self, d0, d1, judge):
+ per = (judge + 1.0) / 2.0
+ self.logit = self.net.forward(d0, d1)
+ return self.loss(self.logit, per)
+
+
+# L2, DSSIM metrics
+class FakeNet(nn.Module):
+ def __init__(self, use_gpu=True, colorspace="Lab"):
+ super(FakeNet, self).__init__()
+ self.use_gpu = use_gpu
+ self.colorspace = colorspace
+
+
+class L2(FakeNet):
+ def forward(self, in0, in1, retPerLayer=None):
+ assert in0.size()[0] == 1 # currently only supports batchSize 1
+
+ if self.colorspace == "RGB":
+ (N, C, X, Y) = in0.size()
+ value = torch.mean(
+ torch.mean(torch.mean((in0 - in1) ** 2, dim=1).view(N, 1, X, Y), dim=2).view(N, 1, 1, Y), dim=3
+ ).view(N)
+ return value
+ elif self.colorspace == "Lab":
+ value = l2(
+ tensor2np(tensor2tensorlab(in0.data, to_norm=False)),
+ tensor2np(tensor2tensorlab(in1.data, to_norm=False)),
+ range=100.0,
+ ).astype("float")
+ ret_var = Variable(torch.Tensor((value,)))
+ if self.use_gpu:
+ ret_var = ret_var.cuda()
+ return ret_var
+
+
+class DSSIM(FakeNet):
+ def forward(self, in0, in1, retPerLayer=None):
+ assert in0.size()[0] == 1 # currently only supports batchSize 1
+
+ if self.colorspace == "RGB":
+ value = dssim(1.0 * tensor2im(in0.data), 1.0 * tensor2im(in1.data), range=255.0).astype("float")
+ elif self.colorspace == "Lab":
+ value = dssim(
+ tensor2np(tensor2tensorlab(in0.data, to_norm=False)),
+ tensor2np(tensor2tensorlab(in1.data, to_norm=False)),
+ range=100.0,
+ ).astype("float")
+ ret_var = Variable(torch.Tensor((value,)))
+ if self.use_gpu:
+ ret_var = ret_var.cuda()
+ return ret_var
+
+
+def print_network(net):
+ num_params = 0
+ for param in net.parameters():
+ num_params += param.numel()
+ print("Network", net)
+ print("Total number of parameters: %d" % num_params)
+
+
+class FloLPIPS(LPIPS):
+ def __init__(
+ self,
+ pretrained=True,
+ net="alex",
+ version="0.1",
+ lpips=True,
+ spatial=False,
+ pnet_rand=False,
+ pnet_tune=False,
+ use_dropout=True,
+ model_path=None,
+ eval_mode=True,
+ verbose=False,
+ ):
+ super(FloLPIPS, self).__init__(
+ pretrained, net, version, lpips, spatial, pnet_rand, pnet_tune, use_dropout, model_path, eval_mode, verbose
+ )
+
+ def forward(self, in0, in1, flow, retPerLayer=False, normalize=False):
+ if normalize: # turn on this flag if input is [0,1] so it can be adjusted to [-1, +1]
+ in0 = 2 * in0 - 1
+ in1 = 2 * in1 - 1
+
+ in0_input, in1_input = (
+ (self.scaling_layer(in0), self.scaling_layer(in1)) if self.version == "0.1" else (in0, in1)
+ )
+ outs0, outs1 = self.net.forward(in0_input), self.net.forward(in1_input)
+ feats0, feats1, diffs = {}, {}, {}
+
+ for kk in range(self.L):
+ feats0[kk], feats1[kk] = normalize_tensor(outs0[kk]), normalize_tensor(outs1[kk])
+ diffs[kk] = (feats0[kk] - feats1[kk]) ** 2
+
+ res = [mw_spatial_average(self.lins[kk](diffs[kk]), flow, keepdim=True) for kk in range(self.L)]
+
+ return torch.sum(torch.cat(res, 1), dim=(1, 2, 3), keepdims=False)
+
+
+class Flolpips(nn.Module):
+ def __init__(self):
+ super(Flolpips, self).__init__()
+ self.loss_fn = FloLPIPS(net="alex", version="0.1")
+ self.flownet = PWCNet()
+
+ @torch.no_grad()
+ def forward(self, I0, I1, frame_dis, frame_ref):
+ """
+ args:
+ I0: first frame of the triplet, shape: [B, C, H, W]
+ I1: third frame of the triplet, shape: [B, C, H, W]
+ frame_dis: prediction of the intermediate frame, shape: [B, C, H, W]
+ frame_ref: ground-truth of the intermediate frame, shape: [B, C, H, W]
+ """
+ assert (
+ I0.size() == I1.size() == frame_dis.size() == frame_ref.size()
+ ), "the 4 input tensors should have same size"
+
+ flow_ref = self.flownet(frame_ref, I0)
+ flow_dis = self.flownet(frame_dis, I0)
+ flow_diff = flow_ref - flow_dis
+ flolpips_wrt_I0 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)
+
+ flow_ref = self.flownet(frame_ref, I1)
+ flow_dis = self.flownet(frame_dis, I1)
+ flow_diff = flow_ref - flow_dis
+ flolpips_wrt_I1 = self.loss_fn.forward(frame_ref, frame_dis, flow_diff, normalize=True)
+
+ flolpips = (flolpips_wrt_I0 + flolpips_wrt_I1) / 2
+ return flolpips
diff --git a/eval/vae/flolpips/pretrained_networks.py b/eval/vae/flolpips/pretrained_networks.py
index 82f14e2..5915b05 100644
--- a/eval/vae/flolpips/pretrained_networks.py
+++ b/eval/vae/flolpips/pretrained_networks.py
@@ -1,180 +1,182 @@
-from collections import namedtuple
-import torch
-from torchvision import models as tv
-
-class squeezenet(torch.nn.Module):
- def __init__(self, requires_grad=False, pretrained=True):
- super(squeezenet, self).__init__()
- pretrained_features = tv.squeezenet1_1(pretrained=pretrained).features
- self.slice1 = torch.nn.Sequential()
- self.slice2 = torch.nn.Sequential()
- self.slice3 = torch.nn.Sequential()
- self.slice4 = torch.nn.Sequential()
- self.slice5 = torch.nn.Sequential()
- self.slice6 = torch.nn.Sequential()
- self.slice7 = torch.nn.Sequential()
- self.N_slices = 7
- for x in range(2):
- self.slice1.add_module(str(x), pretrained_features[x])
- for x in range(2,5):
- self.slice2.add_module(str(x), pretrained_features[x])
- for x in range(5, 8):
- self.slice3.add_module(str(x), pretrained_features[x])
- for x in range(8, 10):
- self.slice4.add_module(str(x), pretrained_features[x])
- for x in range(10, 11):
- self.slice5.add_module(str(x), pretrained_features[x])
- for x in range(11, 12):
- self.slice6.add_module(str(x), pretrained_features[x])
- for x in range(12, 13):
- self.slice7.add_module(str(x), pretrained_features[x])
- if not requires_grad:
- for param in self.parameters():
- param.requires_grad = False
-
- def forward(self, X):
- h = self.slice1(X)
- h_relu1 = h
- h = self.slice2(h)
- h_relu2 = h
- h = self.slice3(h)
- h_relu3 = h
- h = self.slice4(h)
- h_relu4 = h
- h = self.slice5(h)
- h_relu5 = h
- h = self.slice6(h)
- h_relu6 = h
- h = self.slice7(h)
- h_relu7 = h
- vgg_outputs = namedtuple("SqueezeOutputs", ['relu1','relu2','relu3','relu4','relu5','relu6','relu7'])
- out = vgg_outputs(h_relu1,h_relu2,h_relu3,h_relu4,h_relu5,h_relu6,h_relu7)
-
- return out
-
-
-class alexnet(torch.nn.Module):
- def __init__(self, requires_grad=False, pretrained=True):
- super(alexnet, self).__init__()
- alexnet_pretrained_features = tv.alexnet(pretrained=pretrained).features
- self.slice1 = torch.nn.Sequential()
- self.slice2 = torch.nn.Sequential()
- self.slice3 = torch.nn.Sequential()
- self.slice4 = torch.nn.Sequential()
- self.slice5 = torch.nn.Sequential()
- self.N_slices = 5
- for x in range(2):
- self.slice1.add_module(str(x), alexnet_pretrained_features[x])
- for x in range(2, 5):
- self.slice2.add_module(str(x), alexnet_pretrained_features[x])
- for x in range(5, 8):
- self.slice3.add_module(str(x), alexnet_pretrained_features[x])
- for x in range(8, 10):
- self.slice4.add_module(str(x), alexnet_pretrained_features[x])
- for x in range(10, 12):
- self.slice5.add_module(str(x), alexnet_pretrained_features[x])
- if not requires_grad:
- for param in self.parameters():
- param.requires_grad = False
-
- def forward(self, X):
- h = self.slice1(X)
- h_relu1 = h
- h = self.slice2(h)
- h_relu2 = h
- h = self.slice3(h)
- h_relu3 = h
- h = self.slice4(h)
- h_relu4 = h
- h = self.slice5(h)
- h_relu5 = h
- alexnet_outputs = namedtuple("AlexnetOutputs", ['relu1', 'relu2', 'relu3', 'relu4', 'relu5'])
- out = alexnet_outputs(h_relu1, h_relu2, h_relu3, h_relu4, h_relu5)
-
- return out
-
-class vgg16(torch.nn.Module):
- def __init__(self, requires_grad=False, pretrained=True):
- super(vgg16, self).__init__()
- vgg_pretrained_features = tv.vgg16(pretrained=pretrained).features
- self.slice1 = torch.nn.Sequential()
- self.slice2 = torch.nn.Sequential()
- self.slice3 = torch.nn.Sequential()
- self.slice4 = torch.nn.Sequential()
- self.slice5 = torch.nn.Sequential()
- self.N_slices = 5
- for x in range(4):
- self.slice1.add_module(str(x), vgg_pretrained_features[x])
- for x in range(4, 9):
- self.slice2.add_module(str(x), vgg_pretrained_features[x])
- for x in range(9, 16):
- self.slice3.add_module(str(x), vgg_pretrained_features[x])
- for x in range(16, 23):
- self.slice4.add_module(str(x), vgg_pretrained_features[x])
- for x in range(23, 30):
- self.slice5.add_module(str(x), vgg_pretrained_features[x])
- if not requires_grad:
- for param in self.parameters():
- param.requires_grad = False
-
- def forward(self, X):
- h = self.slice1(X)
- h_relu1_2 = h
- h = self.slice2(h)
- h_relu2_2 = h
- h = self.slice3(h)
- h_relu3_3 = h
- h = self.slice4(h)
- h_relu4_3 = h
- h = self.slice5(h)
- h_relu5_3 = h
- vgg_outputs = namedtuple("VggOutputs", ['relu1_2', 'relu2_2', 'relu3_3', 'relu4_3', 'relu5_3'])
- out = vgg_outputs(h_relu1_2, h_relu2_2, h_relu3_3, h_relu4_3, h_relu5_3)
-
- return out
-
-
-
-class resnet(torch.nn.Module):
- def __init__(self, requires_grad=False, pretrained=True, num=18):
- super(resnet, self).__init__()
- if(num==18):
- self.net = tv.resnet18(pretrained=pretrained)
- elif(num==34):
- self.net = tv.resnet34(pretrained=pretrained)
- elif(num==50):
- self.net = tv.resnet50(pretrained=pretrained)
- elif(num==101):
- self.net = tv.resnet101(pretrained=pretrained)
- elif(num==152):
- self.net = tv.resnet152(pretrained=pretrained)
- self.N_slices = 5
-
- self.conv1 = self.net.conv1
- self.bn1 = self.net.bn1
- self.relu = self.net.relu
- self.maxpool = self.net.maxpool
- self.layer1 = self.net.layer1
- self.layer2 = self.net.layer2
- self.layer3 = self.net.layer3
- self.layer4 = self.net.layer4
-
- def forward(self, X):
- h = self.conv1(X)
- h = self.bn1(h)
- h = self.relu(h)
- h_relu1 = h
- h = self.maxpool(h)
- h = self.layer1(h)
- h_conv2 = h
- h = self.layer2(h)
- h_conv3 = h
- h = self.layer3(h)
- h_conv4 = h
- h = self.layer4(h)
- h_conv5 = h
-
- outputs = namedtuple("Outputs", ['relu1','conv2','conv3','conv4','conv5'])
- out = outputs(h_relu1, h_conv2, h_conv3, h_conv4, h_conv5)
-
- return out
+from collections import namedtuple
+
+import torch
+from torchvision import models as tv
+
+
+class squeezenet(torch.nn.Module):
+ def __init__(self, requires_grad=False, pretrained=True):
+ super(squeezenet, self).__init__()
+ pretrained_features = tv.squeezenet1_1(pretrained=pretrained).features
+ self.slice1 = torch.nn.Sequential()
+ self.slice2 = torch.nn.Sequential()
+ self.slice3 = torch.nn.Sequential()
+ self.slice4 = torch.nn.Sequential()
+ self.slice5 = torch.nn.Sequential()
+ self.slice6 = torch.nn.Sequential()
+ self.slice7 = torch.nn.Sequential()
+ self.N_slices = 7
+ for x in range(2):
+ self.slice1.add_module(str(x), pretrained_features[x])
+ for x in range(2, 5):
+ self.slice2.add_module(str(x), pretrained_features[x])
+ for x in range(5, 8):
+ self.slice3.add_module(str(x), pretrained_features[x])
+ for x in range(8, 10):
+ self.slice4.add_module(str(x), pretrained_features[x])
+ for x in range(10, 11):
+ self.slice5.add_module(str(x), pretrained_features[x])
+ for x in range(11, 12):
+ self.slice6.add_module(str(x), pretrained_features[x])
+ for x in range(12, 13):
+ self.slice7.add_module(str(x), pretrained_features[x])
+ if not requires_grad:
+ for param in self.parameters():
+ param.requires_grad = False
+
+ def forward(self, X):
+ h = self.slice1(X)
+ h_relu1 = h
+ h = self.slice2(h)
+ h_relu2 = h
+ h = self.slice3(h)
+ h_relu3 = h
+ h = self.slice4(h)
+ h_relu4 = h
+ h = self.slice5(h)
+ h_relu5 = h
+ h = self.slice6(h)
+ h_relu6 = h
+ h = self.slice7(h)
+ h_relu7 = h
+ vgg_outputs = namedtuple("SqueezeOutputs", ["relu1", "relu2", "relu3", "relu4", "relu5", "relu6", "relu7"])
+ out = vgg_outputs(h_relu1, h_relu2, h_relu3, h_relu4, h_relu5, h_relu6, h_relu7)
+
+ return out
+
+
+class alexnet(torch.nn.Module):
+ def __init__(self, requires_grad=False, pretrained=True):
+ super(alexnet, self).__init__()
+ alexnet_pretrained_features = tv.alexnet(pretrained=pretrained).features
+ self.slice1 = torch.nn.Sequential()
+ self.slice2 = torch.nn.Sequential()
+ self.slice3 = torch.nn.Sequential()
+ self.slice4 = torch.nn.Sequential()
+ self.slice5 = torch.nn.Sequential()
+ self.N_slices = 5
+ for x in range(2):
+ self.slice1.add_module(str(x), alexnet_pretrained_features[x])
+ for x in range(2, 5):
+ self.slice2.add_module(str(x), alexnet_pretrained_features[x])
+ for x in range(5, 8):
+ self.slice3.add_module(str(x), alexnet_pretrained_features[x])
+ for x in range(8, 10):
+ self.slice4.add_module(str(x), alexnet_pretrained_features[x])
+ for x in range(10, 12):
+ self.slice5.add_module(str(x), alexnet_pretrained_features[x])
+ if not requires_grad:
+ for param in self.parameters():
+ param.requires_grad = False
+
+ def forward(self, X):
+ h = self.slice1(X)
+ h_relu1 = h
+ h = self.slice2(h)
+ h_relu2 = h
+ h = self.slice3(h)
+ h_relu3 = h
+ h = self.slice4(h)
+ h_relu4 = h
+ h = self.slice5(h)
+ h_relu5 = h
+ alexnet_outputs = namedtuple("AlexnetOutputs", ["relu1", "relu2", "relu3", "relu4", "relu5"])
+ out = alexnet_outputs(h_relu1, h_relu2, h_relu3, h_relu4, h_relu5)
+
+ return out
+
+
+class vgg16(torch.nn.Module):
+ def __init__(self, requires_grad=False, pretrained=True):
+ super(vgg16, self).__init__()
+ vgg_pretrained_features = tv.vgg16(pretrained=pretrained).features
+ self.slice1 = torch.nn.Sequential()
+ self.slice2 = torch.nn.Sequential()
+ self.slice3 = torch.nn.Sequential()
+ self.slice4 = torch.nn.Sequential()
+ self.slice5 = torch.nn.Sequential()
+ self.N_slices = 5
+ for x in range(4):
+ self.slice1.add_module(str(x), vgg_pretrained_features[x])
+ for x in range(4, 9):
+ self.slice2.add_module(str(x), vgg_pretrained_features[x])
+ for x in range(9, 16):
+ self.slice3.add_module(str(x), vgg_pretrained_features[x])
+ for x in range(16, 23):
+ self.slice4.add_module(str(x), vgg_pretrained_features[x])
+ for x in range(23, 30):
+ self.slice5.add_module(str(x), vgg_pretrained_features[x])
+ if not requires_grad:
+ for param in self.parameters():
+ param.requires_grad = False
+
+ def forward(self, X):
+ h = self.slice1(X)
+ h_relu1_2 = h
+ h = self.slice2(h)
+ h_relu2_2 = h
+ h = self.slice3(h)
+ h_relu3_3 = h
+ h = self.slice4(h)
+ h_relu4_3 = h
+ h = self.slice5(h)
+ h_relu5_3 = h
+ vgg_outputs = namedtuple("VggOutputs", ["relu1_2", "relu2_2", "relu3_3", "relu4_3", "relu5_3"])
+ out = vgg_outputs(h_relu1_2, h_relu2_2, h_relu3_3, h_relu4_3, h_relu5_3)
+
+ return out
+
+
+class resnet(torch.nn.Module):
+ def __init__(self, requires_grad=False, pretrained=True, num=18):
+ super(resnet, self).__init__()
+ if num == 18:
+ self.net = tv.resnet18(pretrained=pretrained)
+ elif num == 34:
+ self.net = tv.resnet34(pretrained=pretrained)
+ elif num == 50:
+ self.net = tv.resnet50(pretrained=pretrained)
+ elif num == 101:
+ self.net = tv.resnet101(pretrained=pretrained)
+ elif num == 152:
+ self.net = tv.resnet152(pretrained=pretrained)
+ self.N_slices = 5
+
+ self.conv1 = self.net.conv1
+ self.bn1 = self.net.bn1
+ self.relu = self.net.relu
+ self.maxpool = self.net.maxpool
+ self.layer1 = self.net.layer1
+ self.layer2 = self.net.layer2
+ self.layer3 = self.net.layer3
+ self.layer4 = self.net.layer4
+
+ def forward(self, X):
+ h = self.conv1(X)
+ h = self.bn1(h)
+ h = self.relu(h)
+ h_relu1 = h
+ h = self.maxpool(h)
+ h = self.layer1(h)
+ h_conv2 = h
+ h = self.layer2(h)
+ h_conv3 = h
+ h = self.layer3(h)
+ h_conv4 = h
+ h = self.layer4(h)
+ h_conv5 = h
+
+ outputs = namedtuple("Outputs", ["relu1", "conv2", "conv3", "conv4", "conv5"])
+ out = outputs(h_relu1, h_conv2, h_conv3, h_conv4, h_conv5)
+
+ return out
diff --git a/eval/vae/flolpips/pwcnet.py b/eval/vae/flolpips/pwcnet.py
index 374f18e..3abc0ed 100644
--- a/eval/vae/flolpips/pwcnet.py
+++ b/eval/vae/flolpips/pwcnet.py
@@ -1,344 +1,472 @@
-#!/usr/bin/env python
-
-import torch
-
-import getopt
-import math
-import numpy
-import os
-import PIL
-import PIL.Image
-import sys
-
-# try:
-from .correlation import correlation # the custom cost volume layer
-# except:
-# sys.path.insert(0, './correlation'); import correlation # you should consider upgrading python
-# end
-
-##########################################################
-
-# assert(int(str('').join(torch.__version__.split('.')[0:2])) >= 13) # requires at least pytorch version 1.3.0
-
-# torch.set_grad_enabled(False) # make sure to not compute gradients for computational performance
-
-# torch.backends.cudnn.enabled = True # make sure to use cudnn for computational performance
-
-# ##########################################################
-
-# arguments_strModel = 'default' # 'default', or 'chairs-things'
-# arguments_strFirst = './images/first.png'
-# arguments_strSecond = './images/second.png'
-# arguments_strOut = './out.flo'
-
-# for strOption, strArgument in getopt.getopt(sys.argv[1:], '', [ strParameter[2:] + '=' for strParameter in sys.argv[1::2] ])[0]:
-# if strOption == '--model' and strArgument != '': arguments_strModel = strArgument # which model to use
-# if strOption == '--first' and strArgument != '': arguments_strFirst = strArgument # path to the first frame
-# if strOption == '--second' and strArgument != '': arguments_strSecond = strArgument # path to the second frame
-# if strOption == '--out' and strArgument != '': arguments_strOut = strArgument # path to where the output should be stored
-# end
-
-##########################################################
-
-
-
-def backwarp(tenInput, tenFlow):
- backwarp_tenGrid = {}
- backwarp_tenPartial = {}
- if str(tenFlow.shape) not in backwarp_tenGrid:
- tenHor = torch.linspace(-1.0 + (1.0 / tenFlow.shape[3]), 1.0 - (1.0 / tenFlow.shape[3]), tenFlow.shape[3]).view(1, 1, 1, -1).expand(-1, -1, tenFlow.shape[2], -1)
- tenVer = torch.linspace(-1.0 + (1.0 / tenFlow.shape[2]), 1.0 - (1.0 / tenFlow.shape[2]), tenFlow.shape[2]).view(1, 1, -1, 1).expand(-1, -1, -1, tenFlow.shape[3])
-
- backwarp_tenGrid[str(tenFlow.shape)] = torch.cat([ tenHor, tenVer ], 1).cuda()
- # end
-
- if str(tenFlow.shape) not in backwarp_tenPartial:
- backwarp_tenPartial[str(tenFlow.shape)] = tenFlow.new_ones([ tenFlow.shape[0], 1, tenFlow.shape[2], tenFlow.shape[3] ])
- # end
-
- tenFlow = torch.cat([ tenFlow[:, 0:1, :, :] / ((tenInput.shape[3] - 1.0) / 2.0), tenFlow[:, 1:2, :, :] / ((tenInput.shape[2] - 1.0) / 2.0) ], 1)
- tenInput = torch.cat([ tenInput, backwarp_tenPartial[str(tenFlow.shape)] ], 1)
-
- tenOutput = torch.nn.functional.grid_sample(input=tenInput, grid=(backwarp_tenGrid[str(tenFlow.shape)] + tenFlow).permute(0, 2, 3, 1), mode='bilinear', padding_mode='zeros', align_corners=False)
-
- tenMask = tenOutput[:, -1:, :, :]; tenMask[tenMask > 0.999] = 1.0; tenMask[tenMask < 1.0] = 0.0
-
- return tenOutput[:, :-1, :, :] * tenMask
-# end
-
-##########################################################
-
-class Network(torch.nn.Module):
- def __init__(self):
- super(Network, self).__init__()
-
- class Extractor(torch.nn.Module):
- def __init__(self):
- super(Extractor, self).__init__()
-
- self.netOne = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=2, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netTwo = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=2, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netThr = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netFou = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=64, out_channels=96, kernel_size=3, stride=2, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netFiv = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=96, out_channels=128, kernel_size=3, stride=2, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netSix = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=128, out_channels=196, kernel_size=3, stride=2, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
- # end
-
- def forward(self, tenInput):
- tenOne = self.netOne(tenInput)
- tenTwo = self.netTwo(tenOne)
- tenThr = self.netThr(tenTwo)
- tenFou = self.netFou(tenThr)
- tenFiv = self.netFiv(tenFou)
- tenSix = self.netSix(tenFiv)
-
- return [ tenOne, tenTwo, tenThr, tenFou, tenFiv, tenSix ]
- # end
- # end
-
- class Decoder(torch.nn.Module):
- def __init__(self, intLevel):
- super(Decoder, self).__init__()
-
- intPrevious = [ None, None, 81 + 32 + 2 + 2, 81 + 64 + 2 + 2, 81 + 96 + 2 + 2, 81 + 128 + 2 + 2, 81, None ][intLevel + 1]
- intCurrent = [ None, None, 81 + 32 + 2 + 2, 81 + 64 + 2 + 2, 81 + 96 + 2 + 2, 81 + 128 + 2 + 2, 81, None ][intLevel + 0]
-
- if intLevel < 6: self.netUpflow = torch.nn.ConvTranspose2d(in_channels=2, out_channels=2, kernel_size=4, stride=2, padding=1)
- if intLevel < 6: self.netUpfeat = torch.nn.ConvTranspose2d(in_channels=intPrevious + 128 + 128 + 96 + 64 + 32, out_channels=2, kernel_size=4, stride=2, padding=1)
- if intLevel < 6: self.fltBackwarp = [ None, None, None, 5.0, 2.5, 1.25, 0.625, None ][intLevel + 1]
-
- self.netOne = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=intCurrent, out_channels=128, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netTwo = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=intCurrent + 128, out_channels=128, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netThr = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128, out_channels=96, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netFou = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128 + 96, out_channels=64, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netFiv = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128 + 96 + 64, out_channels=32, kernel_size=3, stride=1, padding=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1)
- )
-
- self.netSix = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=intCurrent + 128 + 128 + 96 + 64 + 32, out_channels=2, kernel_size=3, stride=1, padding=1)
- )
- # end
-
- def forward(self, tenFirst, tenSecond, objPrevious):
- tenFlow = None
- tenFeat = None
-
- if objPrevious is None:
- tenFlow = None
- tenFeat = None
-
- tenVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond), negative_slope=0.1, inplace=False)
-
- tenFeat = torch.cat([ tenVolume ], 1)
-
- elif objPrevious is not None:
- tenFlow = self.netUpflow(objPrevious['tenFlow'])
- tenFeat = self.netUpfeat(objPrevious['tenFeat'])
-
- tenVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=backwarp(tenInput=tenSecond, tenFlow=tenFlow * self.fltBackwarp)), negative_slope=0.1, inplace=False)
-
- tenFeat = torch.cat([ tenVolume, tenFirst, tenFlow, tenFeat ], 1)
-
- # end
-
- tenFeat = torch.cat([ self.netOne(tenFeat), tenFeat ], 1)
- tenFeat = torch.cat([ self.netTwo(tenFeat), tenFeat ], 1)
- tenFeat = torch.cat([ self.netThr(tenFeat), tenFeat ], 1)
- tenFeat = torch.cat([ self.netFou(tenFeat), tenFeat ], 1)
- tenFeat = torch.cat([ self.netFiv(tenFeat), tenFeat ], 1)
-
- tenFlow = self.netSix(tenFeat)
-
- return {
- 'tenFlow': tenFlow,
- 'tenFeat': tenFeat
- }
- # end
- # end
-
- class Refiner(torch.nn.Module):
- def __init__(self):
- super(Refiner, self).__init__()
-
- self.netMain = torch.nn.Sequential(
- torch.nn.Conv2d(in_channels=81 + 32 + 2 + 2 + 128 + 128 + 96 + 64 + 32, out_channels=128, kernel_size=3, stride=1, padding=1, dilation=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=2, dilation=2),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=4, dilation=4),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=128, out_channels=96, kernel_size=3, stride=1, padding=8, dilation=8),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=96, out_channels=64, kernel_size=3, stride=1, padding=16, dilation=16),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=64, out_channels=32, kernel_size=3, stride=1, padding=1, dilation=1),
- torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
- torch.nn.Conv2d(in_channels=32, out_channels=2, kernel_size=3, stride=1, padding=1, dilation=1)
- )
- # end
-
- def forward(self, tenInput):
- return self.netMain(tenInput)
- # end
- # end
-
- self.netExtractor = Extractor()
-
- self.netTwo = Decoder(2)
- self.netThr = Decoder(3)
- self.netFou = Decoder(4)
- self.netFiv = Decoder(5)
- self.netSix = Decoder(6)
-
- self.netRefiner = Refiner()
-
- self.load_state_dict({ strKey.replace('module', 'net'): tenWeight for strKey, tenWeight in torch.hub.load_state_dict_from_url(url='http://content.sniklaus.com/github/pytorch-pwc/network-' + 'default' + '.pytorch').items() })
- # end
-
- def forward(self, tenFirst, tenSecond):
- intWidth = tenFirst.shape[3]
- intHeight = tenFirst.shape[2]
-
- intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))
- intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))
-
- tenPreprocessedFirst = torch.nn.functional.interpolate(input=tenFirst, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)
- tenPreprocessedSecond = torch.nn.functional.interpolate(input=tenSecond, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)
-
- tenFirst = self.netExtractor(tenPreprocessedFirst)
- tenSecond = self.netExtractor(tenPreprocessedSecond)
-
-
- objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
- objEstimate = self.netFiv(tenFirst[-2], tenSecond[-2], objEstimate)
- objEstimate = self.netFou(tenFirst[-3], tenSecond[-3], objEstimate)
- objEstimate = self.netThr(tenFirst[-4], tenSecond[-4], objEstimate)
- objEstimate = self.netTwo(tenFirst[-5], tenSecond[-5], objEstimate)
-
- tenFlow = objEstimate['tenFlow'] + self.netRefiner(objEstimate['tenFeat'])
- tenFlow = 20.0 * torch.nn.functional.interpolate(input=tenFlow, size=(intHeight, intWidth), mode='bilinear', align_corners=False)
- tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)
- tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)
-
- return tenFlow
- # end
-# end
-
-netNetwork = None
-
-##########################################################
-
-def estimate(tenFirst, tenSecond):
- global netNetwork
-
- if netNetwork is None:
- netNetwork = Network().cuda().eval()
- # end
-
- assert(tenFirst.shape[1] == tenSecond.shape[1])
- assert(tenFirst.shape[2] == tenSecond.shape[2])
-
- intWidth = tenFirst.shape[2]
- intHeight = tenFirst.shape[1]
-
- assert(intWidth == 1024) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue
- assert(intHeight == 436) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue
-
- tenPreprocessedFirst = tenFirst.cuda().view(1, 3, intHeight, intWidth)
- tenPreprocessedSecond = tenSecond.cuda().view(1, 3, intHeight, intWidth)
-
- intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))
- intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))
-
- tenPreprocessedFirst = torch.nn.functional.interpolate(input=tenPreprocessedFirst, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)
- tenPreprocessedSecond = torch.nn.functional.interpolate(input=tenPreprocessedSecond, size=(intPreprocessedHeight, intPreprocessedWidth), mode='bilinear', align_corners=False)
-
- tenFlow = 20.0 * torch.nn.functional.interpolate(input=netNetwork(tenPreprocessedFirst, tenPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)
-
- tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)
- tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)
-
- return tenFlow[0, :, :, :].cpu()
-# end
-
-##########################################################
-
-# if __name__ == '__main__':
-# tenFirst = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strFirst))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))
-# tenSecond = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strSecond))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))
-
-# tenOutput = estimate(tenFirst, tenSecond)
-
-# objOutput = open(arguments_strOut, 'wb')
-
-# numpy.array([ 80, 73, 69, 72 ], numpy.uint8).tofile(objOutput)
-# numpy.array([ tenOutput.shape[2], tenOutput.shape[1] ], numpy.int32).tofile(objOutput)
-# numpy.array(tenOutput.numpy().transpose(1, 2, 0), numpy.float32).tofile(objOutput)
-
-# objOutput.close()
-# end
\ No newline at end of file
+#!/usr/bin/env python
+
+import math
+
+import torch
+
+# try:
+from .correlation import correlation # the custom cost volume layer
+
+# except:
+# sys.path.insert(0, './correlation'); import correlation # you should consider upgrading python
+# end
+
+##########################################################
+
+# assert(int(str('').join(torch.__version__.split('.')[0:2])) >= 13) # requires at least pytorch version 1.3.0
+
+# torch.set_grad_enabled(False) # make sure to not compute gradients for computational performance
+
+# torch.backends.cudnn.enabled = True # make sure to use cudnn for computational performance
+
+# ##########################################################
+
+# arguments_strModel = 'default' # 'default', or 'chairs-things'
+# arguments_strFirst = './images/first.png'
+# arguments_strSecond = './images/second.png'
+# arguments_strOut = './out.flo'
+
+# for strOption, strArgument in getopt.getopt(sys.argv[1:], '', [ strParameter[2:] + '=' for strParameter in sys.argv[1::2] ])[0]:
+# if strOption == '--model' and strArgument != '': arguments_strModel = strArgument # which model to use
+# if strOption == '--first' and strArgument != '': arguments_strFirst = strArgument # path to the first frame
+# if strOption == '--second' and strArgument != '': arguments_strSecond = strArgument # path to the second frame
+# if strOption == '--out' and strArgument != '': arguments_strOut = strArgument # path to where the output should be stored
+# end
+
+##########################################################
+
+
+def backwarp(tenInput, tenFlow):
+ backwarp_tenGrid = {}
+ backwarp_tenPartial = {}
+ if str(tenFlow.shape) not in backwarp_tenGrid:
+ tenHor = (
+ torch.linspace(-1.0 + (1.0 / tenFlow.shape[3]), 1.0 - (1.0 / tenFlow.shape[3]), tenFlow.shape[3])
+ .view(1, 1, 1, -1)
+ .expand(-1, -1, tenFlow.shape[2], -1)
+ )
+ tenVer = (
+ torch.linspace(-1.0 + (1.0 / tenFlow.shape[2]), 1.0 - (1.0 / tenFlow.shape[2]), tenFlow.shape[2])
+ .view(1, 1, -1, 1)
+ .expand(-1, -1, -1, tenFlow.shape[3])
+ )
+
+ backwarp_tenGrid[str(tenFlow.shape)] = torch.cat([tenHor, tenVer], 1).cuda()
+ # end
+
+ if str(tenFlow.shape) not in backwarp_tenPartial:
+ backwarp_tenPartial[str(tenFlow.shape)] = tenFlow.new_ones(
+ [tenFlow.shape[0], 1, tenFlow.shape[2], tenFlow.shape[3]]
+ )
+ # end
+
+ tenFlow = torch.cat(
+ [
+ tenFlow[:, 0:1, :, :] / ((tenInput.shape[3] - 1.0) / 2.0),
+ tenFlow[:, 1:2, :, :] / ((tenInput.shape[2] - 1.0) / 2.0),
+ ],
+ 1,
+ )
+ tenInput = torch.cat([tenInput, backwarp_tenPartial[str(tenFlow.shape)]], 1)
+
+ tenOutput = torch.nn.functional.grid_sample(
+ input=tenInput,
+ grid=(backwarp_tenGrid[str(tenFlow.shape)] + tenFlow).permute(0, 2, 3, 1),
+ mode="bilinear",
+ padding_mode="zeros",
+ align_corners=False,
+ )
+
+ tenMask = tenOutput[:, -1:, :, :]
+ tenMask[tenMask > 0.999] = 1.0
+ tenMask[tenMask < 1.0] = 0.0
+
+ return tenOutput[:, :-1, :, :] * tenMask
+
+
+# end
+
+##########################################################
+
+
+class Network(torch.nn.Module):
+ def __init__(self):
+ super(Network, self).__init__()
+
+ class Extractor(torch.nn.Module):
+ def __init__(self):
+ super(Extractor, self).__init__()
+
+ self.netOne = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=2, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netTwo = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=2, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netThr = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netFou = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=64, out_channels=96, kernel_size=3, stride=2, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netFiv = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=96, out_channels=128, kernel_size=3, stride=2, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netSix = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=128, out_channels=196, kernel_size=3, stride=2, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=196, out_channels=196, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ # end
+
+ def forward(self, tenInput):
+ tenOne = self.netOne(tenInput)
+ tenTwo = self.netTwo(tenOne)
+ tenThr = self.netThr(tenTwo)
+ tenFou = self.netFou(tenThr)
+ tenFiv = self.netFiv(tenFou)
+ tenSix = self.netSix(tenFiv)
+
+ return [tenOne, tenTwo, tenThr, tenFou, tenFiv, tenSix]
+
+ # end
+
+ # end
+
+ class Decoder(torch.nn.Module):
+ def __init__(self, intLevel):
+ super(Decoder, self).__init__()
+
+ intPrevious = [
+ None,
+ None,
+ 81 + 32 + 2 + 2,
+ 81 + 64 + 2 + 2,
+ 81 + 96 + 2 + 2,
+ 81 + 128 + 2 + 2,
+ 81,
+ None,
+ ][intLevel + 1]
+ intCurrent = [
+ None,
+ None,
+ 81 + 32 + 2 + 2,
+ 81 + 64 + 2 + 2,
+ 81 + 96 + 2 + 2,
+ 81 + 128 + 2 + 2,
+ 81,
+ None,
+ ][intLevel + 0]
+
+ if intLevel < 6:
+ self.netUpflow = torch.nn.ConvTranspose2d(
+ in_channels=2, out_channels=2, kernel_size=4, stride=2, padding=1
+ )
+ if intLevel < 6:
+ self.netUpfeat = torch.nn.ConvTranspose2d(
+ in_channels=intPrevious + 128 + 128 + 96 + 64 + 32,
+ out_channels=2,
+ kernel_size=4,
+ stride=2,
+ padding=1,
+ )
+ if intLevel < 6:
+ self.fltBackwarp = [None, None, None, 5.0, 2.5, 1.25, 0.625, None][intLevel + 1]
+
+ self.netOne = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=intCurrent, out_channels=128, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netTwo = torch.nn.Sequential(
+ torch.nn.Conv2d(in_channels=intCurrent + 128, out_channels=128, kernel_size=3, stride=1, padding=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netThr = torch.nn.Sequential(
+ torch.nn.Conv2d(
+ in_channels=intCurrent + 128 + 128, out_channels=96, kernel_size=3, stride=1, padding=1
+ ),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netFou = torch.nn.Sequential(
+ torch.nn.Conv2d(
+ in_channels=intCurrent + 128 + 128 + 96, out_channels=64, kernel_size=3, stride=1, padding=1
+ ),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netFiv = torch.nn.Sequential(
+ torch.nn.Conv2d(
+ in_channels=intCurrent + 128 + 128 + 96 + 64,
+ out_channels=32,
+ kernel_size=3,
+ stride=1,
+ padding=1,
+ ),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ )
+
+ self.netSix = torch.nn.Sequential(
+ torch.nn.Conv2d(
+ in_channels=intCurrent + 128 + 128 + 96 + 64 + 32,
+ out_channels=2,
+ kernel_size=3,
+ stride=1,
+ padding=1,
+ )
+ )
+
+ # end
+
+ def forward(self, tenFirst, tenSecond, objPrevious):
+ tenFlow = None
+ tenFeat = None
+
+ if objPrevious is None:
+ tenFlow = None
+ tenFeat = None
+
+ tenVolume = torch.nn.functional.leaky_relu(
+ input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond),
+ negative_slope=0.1,
+ inplace=False,
+ )
+
+ tenFeat = torch.cat([tenVolume], 1)
+
+ elif objPrevious is not None:
+ tenFlow = self.netUpflow(objPrevious["tenFlow"])
+ tenFeat = self.netUpfeat(objPrevious["tenFeat"])
+
+ tenVolume = torch.nn.functional.leaky_relu(
+ input=correlation.FunctionCorrelation(
+ tenFirst=tenFirst,
+ tenSecond=backwarp(tenInput=tenSecond, tenFlow=tenFlow * self.fltBackwarp),
+ ),
+ negative_slope=0.1,
+ inplace=False,
+ )
+
+ tenFeat = torch.cat([tenVolume, tenFirst, tenFlow, tenFeat], 1)
+
+ # end
+
+ tenFeat = torch.cat([self.netOne(tenFeat), tenFeat], 1)
+ tenFeat = torch.cat([self.netTwo(tenFeat), tenFeat], 1)
+ tenFeat = torch.cat([self.netThr(tenFeat), tenFeat], 1)
+ tenFeat = torch.cat([self.netFou(tenFeat), tenFeat], 1)
+ tenFeat = torch.cat([self.netFiv(tenFeat), tenFeat], 1)
+
+ tenFlow = self.netSix(tenFeat)
+
+ return {"tenFlow": tenFlow, "tenFeat": tenFeat}
+
+ # end
+
+ # end
+
+ class Refiner(torch.nn.Module):
+ def __init__(self):
+ super(Refiner, self).__init__()
+
+ self.netMain = torch.nn.Sequential(
+ torch.nn.Conv2d(
+ in_channels=81 + 32 + 2 + 2 + 128 + 128 + 96 + 64 + 32,
+ out_channels=128,
+ kernel_size=3,
+ stride=1,
+ padding=1,
+ dilation=1,
+ ),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=2, dilation=2),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=4, dilation=4),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=128, out_channels=96, kernel_size=3, stride=1, padding=8, dilation=8),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=96, out_channels=64, kernel_size=3, stride=1, padding=16, dilation=16),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=64, out_channels=32, kernel_size=3, stride=1, padding=1, dilation=1),
+ torch.nn.LeakyReLU(inplace=False, negative_slope=0.1),
+ torch.nn.Conv2d(in_channels=32, out_channels=2, kernel_size=3, stride=1, padding=1, dilation=1),
+ )
+
+ # end
+
+ def forward(self, tenInput):
+ return self.netMain(tenInput)
+
+ # end
+
+ # end
+
+ self.netExtractor = Extractor()
+
+ self.netTwo = Decoder(2)
+ self.netThr = Decoder(3)
+ self.netFou = Decoder(4)
+ self.netFiv = Decoder(5)
+ self.netSix = Decoder(6)
+
+ self.netRefiner = Refiner()
+
+ self.load_state_dict(
+ {
+ strKey.replace("module", "net"): tenWeight
+ for strKey, tenWeight in torch.hub.load_state_dict_from_url(
+ url="http://content.sniklaus.com/github/pytorch-pwc/network-" + "default" + ".pytorch"
+ ).items()
+ }
+ )
+
+ # end
+
+ def forward(self, tenFirst, tenSecond):
+ intWidth = tenFirst.shape[3]
+ intHeight = tenFirst.shape[2]
+
+ intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))
+ intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))
+
+ tenPreprocessedFirst = torch.nn.functional.interpolate(
+ input=tenFirst, size=(intPreprocessedHeight, intPreprocessedWidth), mode="bilinear", align_corners=False
+ )
+ tenPreprocessedSecond = torch.nn.functional.interpolate(
+ input=tenSecond, size=(intPreprocessedHeight, intPreprocessedWidth), mode="bilinear", align_corners=False
+ )
+
+ tenFirst = self.netExtractor(tenPreprocessedFirst)
+ tenSecond = self.netExtractor(tenPreprocessedSecond)
+
+ objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
+ objEstimate = self.netFiv(tenFirst[-2], tenSecond[-2], objEstimate)
+ objEstimate = self.netFou(tenFirst[-3], tenSecond[-3], objEstimate)
+ objEstimate = self.netThr(tenFirst[-4], tenSecond[-4], objEstimate)
+ objEstimate = self.netTwo(tenFirst[-5], tenSecond[-5], objEstimate)
+
+ tenFlow = objEstimate["tenFlow"] + self.netRefiner(objEstimate["tenFeat"])
+ tenFlow = 20.0 * torch.nn.functional.interpolate(
+ input=tenFlow, size=(intHeight, intWidth), mode="bilinear", align_corners=False
+ )
+ tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)
+ tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)
+
+ return tenFlow
+
+ # end
+
+
+# end
+
+netNetwork = None
+
+##########################################################
+
+
+def estimate(tenFirst, tenSecond):
+ global netNetwork
+
+ if netNetwork is None:
+ netNetwork = Network().cuda().eval()
+ # end
+
+ assert tenFirst.shape[1] == tenSecond.shape[1]
+ assert tenFirst.shape[2] == tenSecond.shape[2]
+
+ intWidth = tenFirst.shape[2]
+ intHeight = tenFirst.shape[1]
+
+ assert (
+ intWidth == 1024
+ ) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue
+ assert (
+ intHeight == 436
+ ) # remember that there is no guarantee for correctness, comment this line out if you acknowledge this and want to continue
+
+ tenPreprocessedFirst = tenFirst.cuda().view(1, 3, intHeight, intWidth)
+ tenPreprocessedSecond = tenSecond.cuda().view(1, 3, intHeight, intWidth)
+
+ intPreprocessedWidth = int(math.floor(math.ceil(intWidth / 64.0) * 64.0))
+ intPreprocessedHeight = int(math.floor(math.ceil(intHeight / 64.0) * 64.0))
+
+ tenPreprocessedFirst = torch.nn.functional.interpolate(
+ input=tenPreprocessedFirst,
+ size=(intPreprocessedHeight, intPreprocessedWidth),
+ mode="bilinear",
+ align_corners=False,
+ )
+ tenPreprocessedSecond = torch.nn.functional.interpolate(
+ input=tenPreprocessedSecond,
+ size=(intPreprocessedHeight, intPreprocessedWidth),
+ mode="bilinear",
+ align_corners=False,
+ )
+
+ tenFlow = 20.0 * torch.nn.functional.interpolate(
+ input=netNetwork(tenPreprocessedFirst, tenPreprocessedSecond),
+ size=(intHeight, intWidth),
+ mode="bilinear",
+ align_corners=False,
+ )
+
+ tenFlow[:, 0, :, :] *= float(intWidth) / float(intPreprocessedWidth)
+ tenFlow[:, 1, :, :] *= float(intHeight) / float(intPreprocessedHeight)
+
+ return tenFlow[0, :, :, :].cpu()
+
+
+# end
+
+##########################################################
+
+# if __name__ == '__main__':
+# tenFirst = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strFirst))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))
+# tenSecond = torch.FloatTensor(numpy.ascontiguousarray(numpy.array(PIL.Image.open(arguments_strSecond))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))
+
+# tenOutput = estimate(tenFirst, tenSecond)
+
+# objOutput = open(arguments_strOut, 'wb')
+
+# numpy.array([ 80, 73, 69, 72 ], numpy.uint8).tofile(objOutput)
+# numpy.array([ tenOutput.shape[2], tenOutput.shape[1] ], numpy.int32).tofile(objOutput)
+# numpy.array(tenOutput.numpy().transpose(1, 2, 0), numpy.float32).tofile(objOutput)
+
+# objOutput.close()
+# end
diff --git a/eval/vae/flolpips/utils.py b/eval/vae/flolpips/utils.py
index 98d4502..21a3504 100644
--- a/eval/vae/flolpips/utils.py
+++ b/eval/vae/flolpips/utils.py
@@ -1,95 +1,107 @@
-import numpy as np
-import cv2
-import torch
-
-
-def normalize_tensor(in_feat,eps=1e-10):
- norm_factor = torch.sqrt(torch.sum(in_feat**2,dim=1,keepdim=True))
- return in_feat/(norm_factor+eps)
-
-def l2(p0, p1, range=255.):
- return .5*np.mean((p0 / range - p1 / range)**2)
-
-def dssim(p0, p1, range=255.):
- from skimage.measure import compare_ssim
- return (1 - compare_ssim(p0, p1, data_range=range, multichannel=True)) / 2.
-
-def tensor2im(image_tensor, imtype=np.uint8, cent=1., factor=255./2.):
- image_numpy = image_tensor[0].cpu().float().numpy()
- image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + cent) * factor
- return image_numpy.astype(imtype)
-
-def tensor2np(tensor_obj):
- # change dimension of a tensor object into a numpy array
- return tensor_obj[0].cpu().float().numpy().transpose((1,2,0))
-
-def np2tensor(np_obj):
- # change dimenion of np array into tensor array
- return torch.Tensor(np_obj[:, :, :, np.newaxis].transpose((3, 2, 0, 1)))
-
-def tensor2tensorlab(image_tensor,to_norm=True,mc_only=False):
- # image tensor to lab tensor
- from skimage import color
-
- img = tensor2im(image_tensor)
- img_lab = color.rgb2lab(img)
- if(mc_only):
- img_lab[:,:,0] = img_lab[:,:,0]-50
- if(to_norm and not mc_only):
- img_lab[:,:,0] = img_lab[:,:,0]-50
- img_lab = img_lab/100.
-
- return np2tensor(img_lab)
-
-def read_frame_yuv2rgb(stream, width, height, iFrame, bit_depth, pix_fmt='420'):
- if pix_fmt == '420':
- multiplier = 1
- uv_factor = 2
- elif pix_fmt == '444':
- multiplier = 2
- uv_factor = 1
- else:
- print('Pixel format {} is not supported'.format(pix_fmt))
- return
-
- if bit_depth == 8:
- datatype = np.uint8
- stream.seek(iFrame*1.5*width*height*multiplier)
- Y = np.fromfile(stream, dtype=datatype, count=width*height).reshape((height, width))
-
- # read chroma samples and upsample since original is 4:2:0 sampling
- U = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\
- reshape((height//uv_factor, width//uv_factor))
- V = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\
- reshape((height//uv_factor, width//uv_factor))
-
- else:
- datatype = np.uint16
- stream.seek(iFrame*3*width*height*multiplier)
- Y = np.fromfile(stream, dtype=datatype, count=width*height).reshape((height, width))
-
- U = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\
- reshape((height//uv_factor, width//uv_factor))
- V = np.fromfile(stream, dtype=datatype, count=(width//uv_factor)*(height//uv_factor)).\
- reshape((height//uv_factor, width//uv_factor))
-
- if pix_fmt == '420':
- yuv = np.empty((height*3//2, width), dtype=datatype)
- yuv[0:height,:] = Y
-
- yuv[height:height+height//4,:] = U.reshape(-1, width)
- yuv[height+height//4:,:] = V.reshape(-1, width)
-
- if bit_depth != 8:
- yuv = (yuv/(2**bit_depth-1)*255).astype(np.uint8)
-
- #convert to rgb
- rgb = cv2.cvtColor(yuv, cv2.COLOR_YUV2RGB_I420)
-
- else:
- yvu = np.stack([Y,V,U],axis=2)
- if bit_depth != 8:
- yvu = (yvu/(2**bit_depth-1)*255).astype(np.uint8)
- rgb = cv2.cvtColor(yvu, cv2.COLOR_YCrCb2RGB)
-
- return rgb
+import cv2
+import numpy as np
+import torch
+
+
+def normalize_tensor(in_feat, eps=1e-10):
+ norm_factor = torch.sqrt(torch.sum(in_feat**2, dim=1, keepdim=True))
+ return in_feat / (norm_factor + eps)
+
+
+def l2(p0, p1, range=255.0):
+ return 0.5 * np.mean((p0 / range - p1 / range) ** 2)
+
+
+def dssim(p0, p1, range=255.0):
+ from skimage.measure import compare_ssim
+
+ return (1 - compare_ssim(p0, p1, data_range=range, multichannel=True)) / 2.0
+
+
+def tensor2im(image_tensor, imtype=np.uint8, cent=1.0, factor=255.0 / 2.0):
+ image_numpy = image_tensor[0].cpu().float().numpy()
+ image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + cent) * factor
+ return image_numpy.astype(imtype)
+
+
+def tensor2np(tensor_obj):
+ # change dimension of a tensor object into a numpy array
+ return tensor_obj[0].cpu().float().numpy().transpose((1, 2, 0))
+
+
+def np2tensor(np_obj):
+ # change dimenion of np array into tensor array
+ return torch.Tensor(np_obj[:, :, :, np.newaxis].transpose((3, 2, 0, 1)))
+
+
+def tensor2tensorlab(image_tensor, to_norm=True, mc_only=False):
+ # image tensor to lab tensor
+ from skimage import color
+
+ img = tensor2im(image_tensor)
+ img_lab = color.rgb2lab(img)
+ if mc_only:
+ img_lab[:, :, 0] = img_lab[:, :, 0] - 50
+ if to_norm and not mc_only:
+ img_lab[:, :, 0] = img_lab[:, :, 0] - 50
+ img_lab = img_lab / 100.0
+
+ return np2tensor(img_lab)
+
+
+def read_frame_yuv2rgb(stream, width, height, iFrame, bit_depth, pix_fmt="420"):
+ if pix_fmt == "420":
+ multiplier = 1
+ uv_factor = 2
+ elif pix_fmt == "444":
+ multiplier = 2
+ uv_factor = 1
+ else:
+ print("Pixel format {} is not supported".format(pix_fmt))
+ return
+
+ if bit_depth == 8:
+ datatype = np.uint8
+ stream.seek(iFrame * 1.5 * width * height * multiplier)
+ Y = np.fromfile(stream, dtype=datatype, count=width * height).reshape((height, width))
+
+ # read chroma samples and upsample since original is 4:2:0 sampling
+ U = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(
+ (height // uv_factor, width // uv_factor)
+ )
+ V = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(
+ (height // uv_factor, width // uv_factor)
+ )
+
+ else:
+ datatype = np.uint16
+ stream.seek(iFrame * 3 * width * height * multiplier)
+ Y = np.fromfile(stream, dtype=datatype, count=width * height).reshape((height, width))
+
+ U = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(
+ (height // uv_factor, width // uv_factor)
+ )
+ V = np.fromfile(stream, dtype=datatype, count=(width // uv_factor) * (height // uv_factor)).reshape(
+ (height // uv_factor, width // uv_factor)
+ )
+
+ if pix_fmt == "420":
+ yuv = np.empty((height * 3 // 2, width), dtype=datatype)
+ yuv[0:height, :] = Y
+
+ yuv[height : height + height // 4, :] = U.reshape(-1, width)
+ yuv[height + height // 4 :, :] = V.reshape(-1, width)
+
+ if bit_depth != 8:
+ yuv = (yuv / (2**bit_depth - 1) * 255).astype(np.uint8)
+
+ # convert to rgb
+ rgb = cv2.cvtColor(yuv, cv2.COLOR_YUV2RGB_I420)
+
+ else:
+ yvu = np.stack([Y, V, U], axis=2)
+ if bit_depth != 8:
+ yvu = (yvu / (2**bit_depth - 1) * 255).astype(np.uint8)
+ rgb = cv2.cvtColor(yvu, cv2.COLOR_YCrCb2RGB)
+
+ return rgb
diff --git a/eval/vae/script/eval.sh b/eval/vae/script/eval.sh
index 897166e..3716719 100644
--- a/eval/vae/script/eval.sh
+++ b/eval/vae/script/eval.sh
@@ -9,4 +9,4 @@ python eval/eval_common_metric.py \
--num_frames 17 \
--sample_rate 1 \
--subset_size 100 \
- --metric ssim psnr lpips flolpips
\ No newline at end of file
+ --metric ssim psnr lpips flolpips
diff --git a/eval/vbench/calc_vbench.py b/eval/vbench/calc_vbench.py
index b2505bf..b456f28 100644
--- a/eval/vbench/calc_vbench.py
+++ b/eval/vbench/calc_vbench.py
@@ -1,9 +1,11 @@
import argparse
import os
-from vbench import VBench
-import torch
import time
+import torch
+
+from vbench import VBench
+
full_info_path = "eval/vbench/VBench_full_info.json"
dimensions = [
# Quality Score
@@ -26,12 +28,13 @@ dimensions = [
"appearance_style",
]
+
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("video_folder", type=str) # samples/samples..._vbench/eval
parser.add_argument("model_ckpt", type=str)
- parser.add_argument("--start", type=int, default=0) # start index of dimension to be evaluated
- parser.add_argument("--end", type=int, default=-1) # start index of dimension to be evaluated
+ parser.add_argument("--start", type=int, default=0) # start index of dimension to be evaluated
+ parser.add_argument("--end", type=int, default=-1) # start index of dimension to be evaluated
args = parser.parse_args()
return args
@@ -44,23 +47,23 @@ if __name__ == "__main__":
video_path = args.video_folder
kwargs = {}
- kwargs['imaging_quality_preprocessing_mode'] = 'longer' # use VBench/evaluate.py default
+ kwargs["imaging_quality_preprocessing_mode"] = "longer" # use VBench/evaluate.py default
start_time = time.time()
# NOTE: important to use torch.device("cuda"), else will have issue with object_class third_party module
my_VBench = VBench(torch.device("cuda"), full_info_path, output_dir)
- if args.end == -1: # adjust end accordingly
+ if args.end == -1: # adjust end accordingly
args.end = len(dimensions)
- for dim in dimensions[args.start:args.end]:
+ for dim in dimensions[args.start : args.end]:
my_VBench.evaluate(
videos_path=video_path,
name=dim,
local=False,
read_frame=False,
dimension_list=[dim],
- mode='vbench_standard',
- **kwargs
+ mode="vbench_standard",
+ **kwargs,
)
- print("Runtime: %s seconds " % (time.time() - start_time))
\ No newline at end of file
+ print("Runtime: %s seconds " % (time.time() - start_time))
diff --git a/gradio/README.md b/gradio/README.md
index 609bd7c..5dddd47 100644
--- a/gradio/README.md
+++ b/gradio/README.md
@@ -11,7 +11,7 @@ license: apache-2.0
preload_from_hub:
- hpcai-tech/OpenSora-STDiT-v3
- hpcai-tech/OpenSora-VAE-v1.2
- - DeepFloyd/t5-v1_1-xxl
+ - DeepFloyd/t5-v1_1-xxl
---
diff --git a/gradio/app.py b/gradio/app.py
index 887e1e2..d488c2a 100644
--- a/gradio/app.py
+++ b/gradio/app.py
@@ -7,27 +7,25 @@ Usage:
"""
import argparse
+import datetime
import importlib
import os
import subprocess
import sys
+from tempfile import NamedTemporaryFile
+
import spaces
import torch
import gradio as gr
-from tempfile import NamedTemporaryFile
-import datetime
-
-
MODEL_TYPES = ["v1.2-stage3"]
WATERMARK_PATH = "./assets/images/watermark/watermark.png"
CONFIG_MAP = {
"v1.2-stage3": "configs/opensora-v1-2/inference/sample.py",
}
-HF_STDIT_MAP = {
- "v1.2-stage3": "hpcai-tech/OpenSora-STDiT-v3"
-}
+HF_STDIT_MAP = {"v1.2-stage3": "hpcai-tech/OpenSora-STDiT-v3"}
+
# ============================
# Prepare Runtime Environment
@@ -103,6 +101,7 @@ def build_models(model_type, config, enable_optimization=False):
# we load model from HuggingFace directly so that we don't need to
# handle model download logic in HuggingFace Space
from opensora.models.stdit.stdit3 import STDiT3
+
stdit = STDiT3.from_pretrained(HF_STDIT_MAP[model_type])
stdit = stdit.cuda()
@@ -169,39 +168,59 @@ install_dependencies(enable_optimization=args.enable_optimization)
# import after installation
from opensora.datasets import IMG_FPS, save_sample
-from opensora.utils.misc import to_torch_dtype
+from opensora.datasets.aspect import get_image_size, get_num_frames
+from opensora.models.text_encoder.t5 import text_preprocessing
from opensora.utils.inference_utils import (
+ add_watermark,
append_generated,
+ append_score_to_prompts,
apply_mask_strategy,
collect_references_batch,
+ dframe_to_frame,
extract_json_from_prompts,
extract_prompts_loop,
- prepare_multi_resolution_info,
- dframe_to_frame,
- append_score_to_prompts,
+ get_random_prompt_by_openai,
has_openai_key,
+ merge_prompt,
+ prepare_multi_resolution_info,
refine_prompts_by_openai,
- add_watermark,
- get_random_prompt_by_openai,
split_prompt,
- merge_prompt
)
-from opensora.models.text_encoder.t5 import text_preprocessing
-from opensora.datasets.aspect import get_image_size, get_num_frames
+from opensora.utils.misc import to_torch_dtype
# some global variables
dtype = to_torch_dtype(config.dtype)
device = torch.device("cuda")
# build model
-vae, text_encoder, stdit, scheduler = build_models(args.model_type, config, enable_optimization=args.enable_optimization)
+vae, text_encoder, stdit, scheduler = build_models(
+ args.model_type, config, enable_optimization=args.enable_optimization
+)
-def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_strength, aesthetic_score, use_motion_strength, use_aesthetic_score, camera_motion, reference_image, refine_prompt, fps, num_loop, seed, sampling_steps, cfg_scale):
+def run_inference(
+ mode,
+ prompt_text,
+ resolution,
+ aspect_ratio,
+ length,
+ motion_strength,
+ aesthetic_score,
+ use_motion_strength,
+ use_aesthetic_score,
+ camera_motion,
+ reference_image,
+ refine_prompt,
+ fps,
+ num_loop,
+ seed,
+ sampling_steps,
+ cfg_scale,
+):
if prompt_text is None or prompt_text == "":
gr.Warning("Your prompt is empty, please enter a valid prompt")
return None
-
+
torch.manual_seed(seed)
with torch.inference_mode():
# ======================
@@ -210,7 +229,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
# parse the inputs
# frame_interval must be 1 so we ignore it here
image_size = get_image_size(resolution, aspect_ratio)
-
+
# compute generation parameters
if mode == "Text2Image":
num_frames = 1
@@ -218,26 +237,26 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
else:
num_frames = config.num_frames
num_frames = get_num_frames(length)
-
+
condition_frame_length = int(num_frames / 17 * 5 / 3)
condition_frame_edit = 0.0
-
+
input_size = (num_frames, *image_size)
latent_size = vae.get_latent_size(input_size)
multi_resolution = "OpenSora"
align = 5
-
+
# == prepare mask strategy ==
if mode == "Text2Image":
mask_strategy = [None]
elif mode == "Text2Video":
if reference_image is not None:
- mask_strategy = ['0']
+ mask_strategy = ["0"]
else:
mask_strategy = [None]
else:
raise ValueError(f"Invalid mode: {mode}")
-
+
# == prepare reference ==
if mode == "Text2Image":
refs = [""]
@@ -245,6 +264,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
if reference_image is not None:
# save image to disk
from PIL import Image
+
im = Image.fromarray(reference_image)
temp_file = NamedTemporaryFile(suffix=".png")
im.save(temp_file.name)
@@ -253,7 +273,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
refs = [""]
else:
raise ValueError(f"Invalid mode: {mode}")
-
+
# == get json from prompts ==
batch_prompts = [prompt_text]
batch_prompts, refs, mask_strategy = extract_json_from_prompts(batch_prompts, refs, mask_strategy)
@@ -265,17 +285,17 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
model_args = prepare_multi_resolution_info(
multi_resolution, len(batch_prompts), image_size, num_frames, fps, device, dtype
)
-
- # == process prompts step by step ==
+
+ # == process prompts step by step ==
# 0. split prompt
# each element in the list is [prompt_segment_list, loop_idx_list]
batched_prompt_segment_list = []
batched_loop_idx_list = []
for prompt in batch_prompts:
- prompt_segment_list, loop_idx_list = split_prompt(prompt)
+ prompt_segment_list, loop_idx_list = split_prompt(prompt)
batched_prompt_segment_list.append(prompt_segment_list)
batched_loop_idx_list.append(loop_idx_list)
-
+
# 1. refine prompt by openai
if refine_prompt:
# check if openai key is provided
@@ -284,7 +304,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
else:
for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):
batched_prompt_segment_list[idx] = refine_prompts_by_openai(prompt_segment_list)
-
+
# process scores
aesthetic_score = aesthetic_score if use_aesthetic_score else None
motion_strength = motion_strength if use_motion_strength and mode != "Text2Image" else None
@@ -301,48 +321,39 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
# 3. clean prompt with T5
for idx, prompt_segment_list in enumerate(batched_prompt_segment_list):
batched_prompt_segment_list[idx] = [text_preprocessing(prompt) for prompt in prompt_segment_list]
-
+
# 4. merge to obtain the final prompt
batch_prompts = []
for prompt_segment_list, loop_idx_list in zip(batched_prompt_segment_list, batched_loop_idx_list):
batch_prompts.append(merge_prompt(prompt_segment_list, loop_idx_list))
-
# =========================
# Generate image/video
# =========================
video_clips = []
-
+
for loop_i in range(num_loop):
# 4.4 sample in hidden space
batch_prompts_loop = extract_prompts_loop(batch_prompts, loop_i)
-
+
# == loop ==
if loop_i > 0:
refs, mask_strategy = append_generated(
- vae,
- video_clips[-1],
- refs,
- mask_strategy,
- loop_i,
- condition_frame_length,
- condition_frame_edit
- )
-
+ vae, video_clips[-1], refs, mask_strategy, loop_i, condition_frame_length, condition_frame_edit
+ )
+
# == sampling ==
z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)
masks = apply_mask_strategy(z, refs, mask_strategy, loop_i, align=align)
-
+
# 4.6. diffusion sampling
# hack to update num_sampling_steps and cfg_scale
scheduler_kwargs = config.scheduler.copy()
- scheduler_kwargs.pop('type')
- scheduler_kwargs['num_sampling_steps'] = sampling_steps
- scheduler_kwargs['cfg_scale'] = cfg_scale
+ scheduler_kwargs.pop("type")
+ scheduler_kwargs["num_sampling_steps"] = sampling_steps
+ scheduler_kwargs["cfg_scale"] = cfg_scale
- scheduler.__init__(
- **scheduler_kwargs
- )
+ scheduler.__init__(**scheduler_kwargs)
samples = scheduler.sample(
stdit,
text_encoder,
@@ -355,7 +366,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
)
samples = vae.decode(samples.to(dtype), num_frames=num_frames)
video_clips.append(samples)
-
+
# =========================
# Save output
# =========================
@@ -368,7 +379,7 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
save_path = os.path.join(args.output, f"output_{timestamp}")
saved_path = save_sample(video, save_path=save_path, fps=24)
torch.cuda.empty_cache()
-
+
# add watermark
# all watermarked videos should have a _watermarked suffix
if mode != "Text2Image" and os.path.exists(WATERMARK_PATH):
@@ -380,29 +391,30 @@ def run_inference(mode, prompt_text, resolution, aspect_ratio, length, motion_st
return saved_path
else:
return saved_path
-
-
+
+
@spaces.GPU(duration=200)
def run_image_inference(
- prompt_text,
- resolution,
- aspect_ratio,
- length,
- motion_strength,
- aesthetic_score,
- use_motion_strength,
+ prompt_text,
+ resolution,
+ aspect_ratio,
+ length,
+ motion_strength,
+ aesthetic_score,
+ use_motion_strength,
use_aesthetic_score,
camera_motion,
reference_image,
refine_prompt,
fps,
- num_loop,
+ num_loop,
seed,
sampling_steps,
- cfg_scale):
+ cfg_scale,
+):
return run_inference(
- "Text2Image",
- prompt_text,
+ "Text2Image",
+ prompt_text,
resolution,
aspect_ratio,
length,
@@ -414,10 +426,12 @@ def run_image_inference(
reference_image,
refine_prompt,
fps,
- num_loop,
+ num_loop,
seed,
sampling_steps,
- cfg_scale)
+ cfg_scale,
+ )
+
@spaces.GPU(duration=200)
def run_video_inference(
@@ -428,38 +442,39 @@ def run_video_inference(
motion_strength,
aesthetic_score,
use_motion_strength,
- use_aesthetic_score,
+ use_aesthetic_score,
camera_motion,
- reference_image,
+ reference_image,
refine_prompt,
fps,
- num_loop,
+ num_loop,
seed,
sampling_steps,
- cfg_scale):
+ cfg_scale,
+):
# if (resolution == "480p" and length == "16s") or \
# (resolution == "720p" and length in ["8s", "16s"]):
# gr.Warning("Generation is interrupted as the combination of 480p and 16s will lead to CUDA out of memory")
# else:
return run_inference(
"Text2Video",
- prompt_text,
+ prompt_text,
resolution,
- aspect_ratio,
- length,
- motion_strength,
- aesthetic_score,
+ aspect_ratio,
+ length,
+ motion_strength,
+ aesthetic_score,
use_motion_strength,
- use_aesthetic_score,
+ use_aesthetic_score,
camera_motion,
- reference_image,
+ reference_image,
refine_prompt,
fps,
- num_loop,
+ num_loop,
seed,
- sampling_steps,
- cfg_scale
- )
+ sampling_steps,
+ cfg_scale,
+ )
def generate_random_prompt():
@@ -498,56 +513,34 @@ def main():
with gr.Row():
with gr.Column():
- prompt_text = gr.Textbox(
- label="Prompt",
- placeholder="Describe your video here",
- lines=4
- )
+ prompt_text = gr.Textbox(label="Prompt", placeholder="Describe your video here", lines=4)
refine_prompt = gr.Checkbox(value=True, label="Refine prompt with GPT4o")
random_prompt_btn = gr.Button("Random Prompt By GPT4o")
-
+
gr.Markdown("## Basic Settings")
resolution = gr.Radio(
- choices=["144p", "240p", "360p", "480p", "720p"],
- value="480p",
- label="Resolution",
+ choices=["144p", "240p", "360p", "480p", "720p"],
+ value="480p",
+ label="Resolution",
)
aspect_ratio = gr.Radio(
- choices=["9:16", "16:9", "3:4", "4:3", "1:1"],
- value="9:16",
- label="Aspect Ratio (H:W)",
+ choices=["9:16", "16:9", "3:4", "4:3", "1:1"],
+ value="9:16",
+ label="Aspect Ratio (H:W)",
)
length = gr.Radio(
- choices=["2s", "4s", "8s", "16s"],
+ choices=["2s", "4s", "8s", "16s"],
value="2s",
- label="Video Length",
- info="only effective for video generation, 8s may fail as Hugging Face ZeroGPU has the limitation of max 200 seconds inference time."
+ label="Video Length",
+ info="only effective for video generation, 8s may fail as Hugging Face ZeroGPU has the limitation of max 200 seconds inference time.",
)
with gr.Row():
- seed = gr.Slider(
- value=1024,
- minimum=1,
- maximum=2048,
- step=1,
- label="Seed"
- )
+ seed = gr.Slider(value=1024, minimum=1, maximum=2048, step=1, label="Seed")
+
+ sampling_steps = gr.Slider(value=30, minimum=1, maximum=200, step=1, label="Sampling steps")
+ cfg_scale = gr.Slider(value=7.0, minimum=0.0, maximum=10.0, step=0.1, label="CFG Scale")
- sampling_steps = gr.Slider(
- value=30,
- minimum=1,
- maximum=200,
- step=1,
- label="Sampling steps"
- )
- cfg_scale = gr.Slider(
- value=7.0,
- minimum=0.0,
- maximum=10.0,
- step=0.1,
- label="CFG Scale"
- )
-
with gr.Row():
with gr.Column():
motion_strength = gr.Slider(
@@ -556,10 +549,10 @@ def main():
maximum=100,
step=1,
label="Motion Strength",
- info="only effective for video generation"
+ info="only effective for video generation",
)
use_motion_strength = gr.Checkbox(value=False, label="Enable")
-
+
with gr.Column():
aesthetic_score = gr.Slider(
value=6.5,
@@ -567,26 +560,17 @@ def main():
maximum=7,
step=0.1,
label="Aesthetic",
- info="effective for text & video generation"
+ info="effective for text & video generation",
)
use_aesthetic_score = gr.Checkbox(value=True, label="Enable")
-
+
camera_motion = gr.Radio(
value="none",
label="Camera Motion",
- choices=[
- "none",
- "pan right",
- "pan left",
- "tilt up",
- "tilt down",
- "zoom in",
- "zoom out",
- "static"
- ],
- interactive=True
+ choices=["none", "pan right", "pan left", "tilt up", "tilt down", "zoom in", "zoom out", "static"],
+ interactive=True,
)
-
+
gr.Markdown("## Advanced Settings")
with gr.Row():
fps = gr.Slider(
@@ -595,7 +579,7 @@ def main():
maximum=60,
step=1,
label="FPS",
- info="This is the frames per seconds for video generation, keep it to 24 if you are not sure"
+ info="This is the frames per seconds for video generation, keep it to 24 if you are not sure",
)
num_loop = gr.Slider(
value=1,
@@ -603,41 +587,64 @@ def main():
maximum=20,
step=1,
label="Number of Loops",
- info="This will change the length of the generated video, keep it to 1 if you are not sure"
+ info="This will change the length of the generated video, keep it to 1 if you are not sure",
)
-
-
+
gr.Markdown("## Reference Image")
- reference_image = gr.Image(
- label="Image (optional)",
- show_download_button=True
- )
-
+ reference_image = gr.Image(label="Image (optional)", show_download_button=True)
+
with gr.Column():
- output_video = gr.Video(
- label="Output Video",
- height="100%"
- )
+ output_video = gr.Video(label="Output Video", height="100%")
with gr.Row():
- image_gen_button = gr.Button("Generate image")
- video_gen_button = gr.Button("Generate video")
-
+ image_gen_button = gr.Button("Generate image")
+ video_gen_button = gr.Button("Generate video")
image_gen_button.click(
- fn=run_image_inference,
- inputs=[prompt_text, resolution, aspect_ratio, length, motion_strength, aesthetic_score, use_motion_strength, use_aesthetic_score, camera_motion, reference_image, refine_prompt, fps, num_loop, seed, sampling_steps, cfg_scale],
- outputs=reference_image
- )
+ fn=run_image_inference,
+ inputs=[
+ prompt_text,
+ resolution,
+ aspect_ratio,
+ length,
+ motion_strength,
+ aesthetic_score,
+ use_motion_strength,
+ use_aesthetic_score,
+ camera_motion,
+ reference_image,
+ refine_prompt,
+ fps,
+ num_loop,
+ seed,
+ sampling_steps,
+ cfg_scale,
+ ],
+ outputs=reference_image,
+ )
video_gen_button.click(
- fn=run_video_inference,
- inputs=[prompt_text, resolution, aspect_ratio, length, motion_strength, aesthetic_score, use_motion_strength, use_aesthetic_score, camera_motion, reference_image, refine_prompt, fps, num_loop, seed, sampling_steps, cfg_scale],
- outputs=output_video
- )
- random_prompt_btn.click(
- fn=generate_random_prompt,
- outputs=prompt_text
+ fn=run_video_inference,
+ inputs=[
+ prompt_text,
+ resolution,
+ aspect_ratio,
+ length,
+ motion_strength,
+ aesthetic_score,
+ use_motion_strength,
+ use_aesthetic_score,
+ camera_motion,
+ reference_image,
+ refine_prompt,
+ fps,
+ num_loop,
+ seed,
+ sampling_steps,
+ cfg_scale,
+ ],
+ outputs=output_video,
)
+ random_prompt_btn.click(fn=generate_random_prompt, outputs=prompt_text)
# launch
demo.launch(server_port=args.port, server_name=args.host, share=args.share)
diff --git a/opensora/datasets/datasets.py b/opensora/datasets/datasets.py
index a8e48f8..7229bb5 100644
--- a/opensora/datasets/datasets.py
+++ b/opensora/datasets/datasets.py
@@ -3,7 +3,6 @@ from glob import glob
import numpy as np
import torch
-import torchvision
from PIL import ImageFile
from torchvision.datasets.folder import IMG_EXTENSIONS, pil_loader
diff --git a/opensora/datasets/read_video.py b/opensora/datasets/read_video.py
index 8cc4bca..f988c30 100644
--- a/opensora/datasets/read_video.py
+++ b/opensora/datasets/read_video.py
@@ -1,11 +1,11 @@
import gc
import math
-import cv2
import os
from fractions import Fraction
-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import Any, Dict, Optional, Tuple, Union
import av
+import cv2
import numpy as np
import torch
from torchvision.io.video import (
@@ -144,7 +144,7 @@ def read_video_cv2(video_path):
else:
fps = cap.get(cv2.CAP_PROP_FPS)
vinfo = {
- 'video_fps': fps,
+ "video_fps": fps,
}
frames = []
@@ -159,7 +159,7 @@ def read_video_cv2(video_path):
frames.append(frame[:, :, ::-1]) # BGR to RGB
# Exit if 'q' is pressed
- if cv2.waitKey(25) & 0xFF == ord('q'):
+ if cv2.waitKey(25) & 0xFF == ord("q"):
break
# Release the video capture object and close all windows
@@ -172,10 +172,10 @@ def read_video_cv2(video_path):
return frames, vinfo
-def read_video(video_path, backend='av'):
- if backend == 'cv2':
+def read_video(video_path, backend="av"):
+ if backend == "cv2":
vframes, vinfo = read_video_cv2(video_path)
- elif backend == 'av':
+ elif backend == "av":
vframes, _, vinfo = read_video_av(filename=video_path, pts_unit="sec", output_format="TCHW")
else:
raise ValueError
@@ -183,6 +183,6 @@ def read_video(video_path, backend='av'):
return vframes, vinfo
-if __name__ == '__main__':
- vframes, vinfo = read_video('./data/colors/9.mp4', backend='cv2')
+if __name__ == "__main__":
+ vframes, vinfo = read_video("./data/colors/9.mp4", backend="cv2")
x = 0
diff --git a/opensora/datasets/utils.py b/opensora/datasets/utils.py
index 194ea24..6ce155e 100644
--- a/opensora/datasets/utils.py
+++ b/opensora/datasets/utils.py
@@ -67,7 +67,9 @@ def temporal_random_crop(vframes, num_frames, frame_interval):
temporal_sample = video_transforms.TemporalRandomCrop(num_frames * frame_interval)
total_frames = len(vframes)
start_frame_ind, end_frame_ind = temporal_sample(total_frames)
- assert end_frame_ind - start_frame_ind >= num_frames, f"Not enough frames to sample, {end_frame_ind} - {start_frame_ind} < {num_frames}"
+ assert (
+ end_frame_ind - start_frame_ind >= num_frames
+ ), f"Not enough frames to sample, {end_frame_ind} - {start_frame_ind} < {num_frames}"
frame_indice = np.linspace(start_frame_ind, end_frame_ind - 1, num_frames, dtype=int)
video = vframes[frame_indice]
return video
diff --git a/opensora/models/__init__.py b/opensora/models/__init__.py
index 56db0a5..6025349 100644
--- a/opensora/models/__init__.py
+++ b/opensora/models/__init__.py
@@ -4,4 +4,3 @@ from .pixart import *
from .stdit import *
from .text_encoder import *
from .vae import *
-
diff --git a/opensora/models/layers/blocks.py b/opensora/models/layers/blocks.py
index 6a2d793..8bc7e72 100644
--- a/opensora/models/layers/blocks.py
+++ b/opensora/models/layers/blocks.py
@@ -494,7 +494,7 @@ class SeqParallelMultiHeadCrossAttention(MultiHeadCrossAttention):
# query/value: img tokens; key: condition; mask: if padding tokens
sp_group = get_sequence_parallel_group()
sp_size = dist.get_world_size(sp_group)
- B, SUB_N, C = x.shape # [B, TS/p, C]
+ B, SUB_N, C = x.shape # [B, TS/p, C]
N = SUB_N * sp_size
# shape:
diff --git a/opensora/models/pixart/__init__.py b/opensora/models/pixart/__init__.py
index 4c74300..8f359d2 100644
--- a/opensora/models/pixart/__init__.py
+++ b/opensora/models/pixart/__init__.py
@@ -1,2 +1,2 @@
-from .pixart import PixArt, PixArt_XL_2, PixArt_1B_2
+from .pixart import PixArt, PixArt_1B_2, PixArt_XL_2
from .pixart_sigma import PixArt_Sigma_XL_2
diff --git a/opensora/models/stdit/stdit2.py b/opensora/models/stdit/stdit2.py
index 56fd361..35f6fc3 100644
--- a/opensora/models/stdit/stdit2.py
+++ b/opensora/models/stdit/stdit2.py
@@ -1,11 +1,13 @@
+import os
+
import numpy as np
import torch
import torch.nn as nn
-import os
from einops import rearrange
from rotary_embedding_torch import RotaryEmbedding
from timm.models.layers import DropPath
from timm.models.vision_transformer import Mlp
+from transformers import PretrainedConfig, PreTrainedModel
from opensora.acceleration.checkpoint import auto_grad_checkpoint
from opensora.models.layers.blocks import (
@@ -23,7 +25,6 @@ from opensora.models.layers.blocks import (
t2i_modulate,
)
from opensora.registry import MODELS
-from transformers import PretrainedConfig, PreTrainedModel
from opensora.utils.ckpt_utils import load_checkpoint
@@ -167,7 +168,6 @@ class STDiT2Block(nn.Module):
class STDiT2Config(PretrainedConfig):
-
model_type = "STDiT2"
def __init__(
@@ -217,13 +217,9 @@ class STDiT2Config(PretrainedConfig):
@MODELS.register_module()
class STDiT2(PreTrainedModel):
-
config_class = STDiT2Config
- def __init__(
- self,
- config
- ):
+ def __init__(self, config):
super().__init__(config)
self.pred_sigma = config.pred_sigma
self.in_channels = config.in_channels
@@ -245,7 +241,9 @@ class STDiT2(PreTrainedModel):
self.x_embedder = PatchEmbed3D(config.patch_size, config.in_channels, config.hidden_size)
self.t_embedder = TimestepEmbedder(config.hidden_size)
self.t_block = nn.Sequential(nn.SiLU(), nn.Linear(config.hidden_size, 6 * config.hidden_size, bias=True))
- self.t_block_temp = nn.Sequential(nn.SiLU(), nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=True)) # new
+ self.t_block_temp = nn.Sequential(
+ nn.SiLU(), nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=True)
+ ) # new
self.y_embedder = CaptionEmbedder(
in_channels=config.caption_channels,
hidden_size=config.hidden_size,
@@ -512,12 +510,7 @@ def STDiT2_XL_2(from_pretrained=None, **kwargs):
if from_pretrained is not None:
if os.path.isdir(from_pretrained) or os.path.isfile(from_pretrained):
# if it is a directory or a file, we load the checkpoint manually
- config = STDiT2Config(
- depth=28,
- hidden_size=1152,
- patch_size=(1, 2, 2),
- num_heads=16, **kwargs
- )
+ config = STDiT2Config(depth=28, hidden_size=1152, patch_size=(1, 2, 2), num_heads=16, **kwargs)
model = STDiT2(config)
load_checkpoint(model, from_pretrained)
return model
@@ -526,11 +519,6 @@ def STDiT2_XL_2(from_pretrained=None, **kwargs):
return STDiT2.from_pretrained(from_pretrained)
else:
# create a new model
- config = STDiT2Config(
- depth=28,
- hidden_size=1152,
- patch_size=(1, 2, 2),
- num_heads=16, **kwargs
- )
+ config = STDiT2Config(depth=28, hidden_size=1152, patch_size=(1, 2, 2), num_heads=16, **kwargs)
model = STDiT2(config)
return model
diff --git a/opensora/models/stdit/stdit3.py b/opensora/models/stdit/stdit3.py
index dd9fe86..e857a37 100644
--- a/opensora/models/stdit/stdit3.py
+++ b/opensora/models/stdit/stdit3.py
@@ -1,24 +1,26 @@
import os
+
import numpy as np
import torch
-import torch.nn as nn
import torch.distributed as dist
+import torch.nn as nn
from einops import rearrange
from rotary_embedding_torch import RotaryEmbedding
from timm.models.layers import DropPath
from timm.models.vision_transformer import Mlp
from transformers import PretrainedConfig, PreTrainedModel
+
+from opensora.acceleration.checkpoint import auto_grad_checkpoint
from opensora.acceleration.communications import gather_forward_split_backward, split_forward_gather_backward
from opensora.acceleration.parallel_states import get_sequence_parallel_group
-from opensora.acceleration.checkpoint import auto_grad_checkpoint
from opensora.models.layers.blocks import (
- SeqParallelMultiHeadCrossAttention,
- SeqParallelAttention,
Attention,
CaptionEmbedder,
MultiHeadCrossAttention,
PatchEmbed3D,
PositionEmbedding2D,
+ SeqParallelAttention,
+ SeqParallelMultiHeadCrossAttention,
SizeEmbedder,
T2IFinalLayer,
TimestepEmbedder,
@@ -49,14 +51,14 @@ class STDiT3Block(nn.Module):
self.hidden_size = hidden_size
self.enable_flash_attn = enable_flash_attn
self.enable_sequence_parallelism = enable_sequence_parallelism
-
+
if self.enable_sequence_parallelism and not temporal:
attn_cls = SeqParallelAttention
mha_cls = SeqParallelMultiHeadCrossAttention
else:
attn_cls = Attention
mha_cls = MultiHeadCrossAttention
-
+
self.norm1 = get_layernorm(hidden_size, eps=1e-6, affine=False, use_kernel=enable_layernorm_kernel)
self.attn = attn_cls(
hidden_size,
@@ -389,19 +391,19 @@ class STDiT3(PreTrainedModel):
x = self.x_embedder(x) # [B, N, C]
x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)
x = x + pos_emb
-
+
# shard over the sequence dim if sp is enabled
if self.enable_sequence_parallelism:
x = split_forward_gather_backward(x, get_sequence_parallel_group(), dim=2, grad_scale="down")
S = S // dist.get_world_size(get_sequence_parallel_group())
-
+
x = rearrange(x, "B T S C -> B (T S) C", T=T, S=S)
-
+
# === blocks ===
for spatial_block, temporal_block in zip(self.spatial_blocks, self.temporal_blocks):
x = auto_grad_checkpoint(spatial_block, x, y, t_mlp, y_lens, x_mask, t0_mlp, T, S)
x = auto_grad_checkpoint(temporal_block, x, y, t_mlp, y_lens, x_mask, t0_mlp, T, S)
-
+
if self.enable_sequence_parallelism:
x = rearrange(x, "B (T S) C -> B T S C", T=T, S=S)
x = gather_forward_split_backward(x, get_sequence_parallel_group(), dim=2, grad_scale="up")
@@ -411,7 +413,7 @@ class STDiT3(PreTrainedModel):
# === final layer ===
x = self.final_layer(x, t, x_mask, t0, T, S)
x = self.unpatchify(x, T, H, W, Tx, Hx, Wx)
-
+
# cast to float32 for better accuracy
x = x.to(torch.float32)
return x
diff --git a/opensora/models/vae/discriminator.py b/opensora/models/vae/discriminator.py
index ab00b93..2a0c367 100644
--- a/opensora/models/vae/discriminator.py
+++ b/opensora/models/vae/discriminator.py
@@ -148,11 +148,13 @@ class ResBlockDown(nn.Module):
out = (residual + x) / math.sqrt(2)
return out
+
@MODELS.register_module()
class NLayerDiscriminator(nn.Module):
"""Defines a PatchGAN discriminator as in Pix2Pix
- --> see https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py
+ --> see https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py
"""
+
def __init__(self, input_nc=3, ndf=64, n_layers=3, use_actnorm=False, from_pretrained=None):
"""Construct a PatchGAN discriminator
Parameters:
@@ -162,7 +164,7 @@ class NLayerDiscriminator(nn.Module):
norm_layer -- normalization layer
"""
super(NLayerDiscriminator, self).__init__()
-
+
norm_layer = nn.BatchNorm2d
if type(norm_layer) == functools.partial: # no need to use bias as BatchNorm2d has affine parameters
@@ -177,23 +179,24 @@ class NLayerDiscriminator(nn.Module):
nf_mult_prev = 1
for n in range(1, n_layers): # gradually increase the number of filters
nf_mult_prev = nf_mult
- nf_mult = min(2 ** n, 8)
+ nf_mult = min(2**n, 8)
sequence += [
nn.Conv2d(ndf * nf_mult_prev, ndf * nf_mult, kernel_size=kw, stride=2, padding=padw, bias=use_bias),
norm_layer(ndf * nf_mult),
- nn.LeakyReLU(0.2, True)
+ nn.LeakyReLU(0.2, True),
]
nf_mult_prev = nf_mult
- nf_mult = min(2 ** n_layers, 8)
+ nf_mult = min(2**n_layers, 8)
sequence += [
nn.Conv2d(ndf * nf_mult_prev, ndf * nf_mult, kernel_size=kw, stride=1, padding=padw, bias=use_bias),
norm_layer(ndf * nf_mult),
- nn.LeakyReLU(0.2, True)
+ nn.LeakyReLU(0.2, True),
]
sequence += [
- nn.Conv2d(ndf * nf_mult, 1, kernel_size=kw, stride=1, padding=padw)] # output 1 channel prediction map
+ nn.Conv2d(ndf * nf_mult, 1, kernel_size=kw, stride=1, padding=padw)
+ ] # output 1 channel prediction map
self.main = nn.Sequential(*sequence)
if from_pretrained is not None:
@@ -204,7 +207,6 @@ class NLayerDiscriminator(nn.Module):
return self.main(input)
-
class NLayerDiscriminator3D(nn.Module):
"""Defines a 3D PatchGAN discriminator as in Pix2Pix but for 3D inputs."""
diff --git a/opensora/models/vae/vae.py b/opensora/models/vae/vae.py
index f9823d9..4a9a278 100644
--- a/opensora/models/vae/vae.py
+++ b/opensora/models/vae/vae.py
@@ -1,12 +1,13 @@
import os
+
import torch
import torch.nn as nn
from diffusers.models import AutoencoderKL, AutoencoderKLTemporalDecoder
from einops import rearrange
+from transformers import PretrainedConfig, PreTrainedModel
from opensora.registry import MODELS, build_module
from opensora.utils.ckpt_utils import load_checkpoint
-from transformers import PretrainedConfig, PreTrainedModel
@MODELS.register_module()
@@ -117,9 +118,10 @@ class VideoAutoencoderKLTemporalDecoder(nn.Module):
def dtype(self):
return next(self.parameters()).dtype
+
class VideoAutoencoderPipelineConfig(PretrainedConfig):
model_type = "VideoAutoencoderPipeline"
-
+
def __init__(
self,
vae_2d=None,
@@ -130,7 +132,7 @@ class VideoAutoencoderPipelineConfig(PretrainedConfig):
micro_frame_size=None,
shift=0.0,
scale=1.0,
- **kwargs
+ **kwargs,
):
self.vae_2d = vae_2d
self.vae_temporal = vae_temporal
@@ -146,11 +148,8 @@ class VideoAutoencoderPipelineConfig(PretrainedConfig):
@MODELS.register_module()
class VideoAutoencoderPipeline(PreTrainedModel):
config_class = VideoAutoencoderPipelineConfig
-
- def __init__(
- self,
- config: VideoAutoencoderPipelineConfig
- ):
+
+ def __init__(self, config: VideoAutoencoderPipelineConfig):
super().__init__(config=config)
self.spatial_vae = build_module(config.vae_2d, MODELS)
self.temporal_vae = build_module(config.vae_temporal, MODELS)
@@ -245,6 +244,7 @@ class VideoAutoencoderPipeline(PreTrainedModel):
def dtype(self):
return next(self.parameters()).dtype
+
@MODELS.register_module()
def OpenSoraVAE_V1_2(
micro_batch_size=4,
@@ -255,12 +255,12 @@ def OpenSoraVAE_V1_2(
cal_loss=False,
):
vae_2d = dict(
- type="VideoAutoencoderKL",
- from_pretrained="PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers",
- subfolder="vae",
- micro_batch_size=micro_batch_size,
- local_files_only=local_files_only,
- )
+ type="VideoAutoencoderKL",
+ from_pretrained="PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers",
+ subfolder="vae",
+ micro_batch_size=micro_batch_size,
+ local_files_only=local_files_only,
+ )
vae_temporal = dict(
type="VAE_Temporal_SD",
from_pretrained=None,
@@ -274,15 +274,15 @@ def OpenSoraVAE_V1_2(
cal_loss=cal_loss,
micro_frame_size=micro_frame_size,
shift=shift,
- scale=scale
+ scale=scale,
)
-
+
if from_pretrained is not None and not os.path.isdir(from_pretrained):
model = VideoAutoencoderPipeline.from_pretrained(from_pretrained, **kwargs)
else:
config = VideoAutoencoderPipelineConfig(**kwargs)
model = VideoAutoencoderPipeline(config)
-
+
if from_pretrained:
load_checkpoint(model, from_pretrained)
return model
diff --git a/opensora/schedulers/rf/__init__.py b/opensora/schedulers/rf/__init__.py
index 4bc4ca6..e5931ec 100644
--- a/opensora/schedulers/rf/__init__.py
+++ b/opensora/schedulers/rf/__init__.py
@@ -100,4 +100,4 @@ class RFLOW:
return z
def training_losses(self, model, x_start, model_kwargs=None, noise=None, mask=None, weights=None, t=None):
- return self.scheduler.training_losses(model, x_start, model_kwargs, noise, mask, weights,t)
+ return self.scheduler.training_losses(model, x_start, model_kwargs, noise, mask, weights, t)
diff --git a/opensora/utils/misc.py b/opensora/utils/misc.py
index f413b6f..b2f6730 100644
--- a/opensora/utils/misc.py
+++ b/opensora/utils/misc.py
@@ -409,4 +409,3 @@ class FeatureSaver:
get_logger().info("Saved to %s", save_path)
self.data_list = []
self.bin_cnt += 1
-
diff --git a/scripts/inference.py b/scripts/inference.py
index 2a0f4a0..5095fd2 100644
--- a/scripts/inference.py
+++ b/scripts/inference.py
@@ -26,10 +26,10 @@ from opensora.utils.inference_utils import (
extract_prompts_loop,
get_save_path_name,
load_prompts,
+ merge_prompt,
prepare_multi_resolution_info,
refine_prompts_by_openai,
split_prompt,
- merge_prompt,
)
from opensora.utils.misc import all_exists, create_logger, is_distributed, is_main_process, to_torch_dtype
@@ -224,7 +224,9 @@ def main():
segment_start_idx = 0
all_prompts = broadcast_obj_list[0]
for num_segment in prompt_segment_length:
- batched_prompt_segment_list.append(all_prompts[segment_start_idx : segment_start_idx + num_segment])
+ batched_prompt_segment_list.append(
+ all_prompts[segment_start_idx : segment_start_idx + num_segment]
+ )
segment_start_idx += num_segment
# 2. append score
diff --git a/scripts/train.py b/scripts/train.py
index 1dd774f..2ebdd41 100644
--- a/scripts/train.py
+++ b/scripts/train.py
@@ -16,9 +16,9 @@ from opensora.acceleration.checkpoint import set_grad_checkpoint
from opensora.acceleration.parallel_states import get_data_parallel_group
from opensora.datasets.dataloader import prepare_dataloader
from opensora.registry import DATASETS, MODELS, SCHEDULERS, build_module
-from opensora.utils.lr_scheduler import LinearWarmupLR
from opensora.utils.ckpt_utils import load, model_gathering, model_sharding, record_model_param_shape, save
from opensora.utils.config_utils import define_experiment_workspace, parse_configs, save_training_config
+from opensora.utils.lr_scheduler import LinearWarmupLR
from opensora.utils.misc import (
Timer,
all_reduce_mean,
@@ -140,7 +140,7 @@ def main():
in_channels=vae_out_channels,
caption_channels=text_encoder_output_dim,
model_max_length=text_encoder_model_max_length,
- enable_sequence_parallelism=cfg.get("sp_size", 1) > 1
+ enable_sequence_parallelism=cfg.get("sp_size", 1) > 1,
)
.to(device, dtype)
.train()
@@ -298,7 +298,7 @@ def main():
# update learning rate
if lr_scheduler is not None:
- lr_scheduler.step()
+ lr_scheduler.step()
coordinator.block_all()
timer_list.append(backward_t)
diff --git a/scripts/train_vae.py b/scripts/train_vae.py
index 8913d07..5376fee 100644
--- a/scripts/train_vae.py
+++ b/scripts/train_vae.py
@@ -5,6 +5,7 @@ from pprint import pformat
import torch
import torch.distributed as dist
+import wandb
from colossalai.booster import Booster
from colossalai.cluster import DistCoordinator
from colossalai.nn.optimizer import HybridAdam
@@ -12,7 +13,6 @@ from colossalai.utils import get_current_device, set_seed
from einops import rearrange
from tqdm import tqdm
-import wandb
from opensora.acceleration.checkpoint import set_grad_checkpoint
from opensora.acceleration.parallel_states import get_data_parallel_group
from opensora.datasets.dataloader import prepare_dataloader
diff --git a/tests/test_lr_scheduler.py b/tests/test_lr_scheduler.py
index c37b31a..8a3fd31 100644
--- a/tests/test_lr_scheduler.py
+++ b/tests/test_lr_scheduler.py
@@ -2,6 +2,7 @@ import torch
from torch.optim import Adam
from torchvision.models import resnet50
from tqdm import tqdm
+
from opensora.utils.lr_scheduler import LinearWarmupLR
@@ -12,13 +13,13 @@ def test_lr_scheduler():
scheduler = LinearWarmupLR(optimizer, warmup_steps=warmup_steps)
current_lr = scheduler.get_lr()[0]
data = torch.rand(1, 3, 224, 224).cuda()
-
- for i in tqdm(range(warmup_steps*2)):
+
+ for i in tqdm(range(warmup_steps * 2)):
out = model(data)
out.mean().backward()
optimizer.step()
scheduler.step()
-
+
if i >= warmup_steps:
assert scheduler.get_lr()[0] == 0.01
else:
diff --git a/tests/test_stdit3_sequence_parallelism.py b/tests/test_stdit3_sequence_parallelism.py
index 3e06028..ba715b7 100644
--- a/tests/test_stdit3_sequence_parallelism.py
+++ b/tests/test_stdit3_sequence_parallelism.py
@@ -1,75 +1,66 @@
import colossalai
import torch
import torch.distributed as dist
-from opensora.models.stdit.stdit3 import STDiT3Config, STDiT3
-from colossalai.testing import spawn, free_port
-from opensora.acceleration.parallel_states import set_data_parallel_group, set_sequence_parallel_group
+from colossalai.testing import spawn
from colossalai.utils.common import set_seed
+from opensora.acceleration.parallel_states import set_sequence_parallel_group
+from opensora.models.stdit.stdit3 import STDiT3, STDiT3Config
+
def get_sample_data():
- x = torch.rand([1, 4, 15, 20, 27], dtype=torch.bfloat16) # (B, C, T, H, W)
- timestep = torch.Tensor([924.]).to(torch.bfloat16)
+ x = torch.rand([1, 4, 15, 20, 27], dtype=torch.bfloat16) # (B, C, T, H, W)
+ timestep = torch.Tensor([924.0]).to(torch.bfloat16)
y = torch.rand(1, 1, 300, 4096, dtype=torch.bfloat16)
mask = torch.ones([1, 300], dtype=torch.int32)
x_mask = torch.ones([1, 15]).bool()
- fps = torch.Tensor([25.]).to(torch.bfloat16)
- height = torch.Tensor([166.]).to(torch.bfloat16)
- width = torch.Tensor([221.]).to(torch.bfloat16)
+ fps = torch.Tensor([25.0]).to(torch.bfloat16)
+ height = torch.Tensor([166.0]).to(torch.bfloat16)
+ width = torch.Tensor([221.0]).to(torch.bfloat16)
return dict(x=x, timestep=timestep, y=y, mask=mask, x_mask=x_mask, fps=fps, height=height, width=width)
-def get_stdit3_config(enable_sequence_parallelism = False):
+
+def get_stdit3_config(enable_sequence_parallelism=False):
config = {
- "caption_channels": 4096,
- "class_dropout_prob": 0.0,
- "depth": 1,
- "drop_path": 0.0,
- "enable_flash_attn": True,
- "enable_layernorm_kernel": True,
- "enable_sequence_parallelism": enable_sequence_parallelism,
- "freeze_y_embedder": True,
- "hidden_size": 1152,
- "in_channels": 4,
- "input_size": [
- None,
- None,
- None
- ],
- "input_sq_size": 512,
- "mlp_ratio": 4.0,
- "model_max_length": 300,
- "model_type": "STDiT3",
- "num_heads": 16,
- "only_train_temporal": False,
- "patch_size": [
- 1,
- 2,
- 2
- ],
- "pred_sigma": True,
- "qk_norm": True,
- "skip_y_embedder": False,
- }
+ "caption_channels": 4096,
+ "class_dropout_prob": 0.0,
+ "depth": 1,
+ "drop_path": 0.0,
+ "enable_flash_attn": True,
+ "enable_layernorm_kernel": True,
+ "enable_sequence_parallelism": enable_sequence_parallelism,
+ "freeze_y_embedder": True,
+ "hidden_size": 1152,
+ "in_channels": 4,
+ "input_size": [None, None, None],
+ "input_sq_size": 512,
+ "mlp_ratio": 4.0,
+ "model_max_length": 300,
+ "model_type": "STDiT3",
+ "num_heads": 16,
+ "only_train_temporal": False,
+ "patch_size": [1, 2, 2],
+ "pred_sigma": True,
+ "qk_norm": True,
+ "skip_y_embedder": False,
+ }
return STDiT3Config(**config)
def run_model(rank, world_size, port):
colossalai.launch({}, rank=rank, world_size=world_size, port=port, host="localhost")
-
+
# prepare data
data = get_sample_data()
- data = {
- k: v.cuda()
- for k, v in data.items()
- }
-
+ data = {k: v.cuda() for k, v in data.items()}
+
# test single-gpu outptu
set_seed(1024)
non_dist_model_cfg = get_stdit3_config(enable_sequence_parallelism=False)
non_dist_model = STDiT3(non_dist_model_cfg).cuda().to(torch.bfloat16)
non_dist_out = non_dist_model(**data)
non_dist_out.mean().backward()
-
+
# run seq parallelism
set_sequence_parallel_group(dist.group.WORLD)
set_seed(1024)
@@ -77,32 +68,32 @@ def run_model(rank, world_size, port):
dist_model = STDiT3(dist_model_cfg).cuda().to(torch.bfloat16)
dist_out = dist_model(**data)
dist_out.mean().backward()
-
+
# run all reduce for gradients
for param in dist_model.parameters():
if param.grad is not None:
dist.all_reduce(param.grad, group=dist.group.WORLD)
param.grad /= world_size
-
+
# ensure model weights are equal
- for (p1, p2) in zip(non_dist_model.parameters(), dist_model.parameters()):
+ for p1, p2 in zip(non_dist_model.parameters(), dist_model.parameters()):
assert torch.equal(p1, p2)
-
+
# check
torch.testing.assert_close(non_dist_out, dist_out)
- for ((n1, p1), (n2, p2)) in zip(non_dist_model.named_parameters(), dist_model.named_parameters()):
+ for (n1, p1), (n2, p2) in zip(non_dist_model.named_parameters(), dist_model.named_parameters()):
assert n1 == n2
if p1.grad is not None and p2.grad is not None:
- if not torch.allclose(p1.grad, p2.grad,rtol=1e-2, atol=1e-4):
+ if not torch.allclose(p1.grad, p2.grad, rtol=1e-2, atol=1e-4):
if dist.get_rank() == 0:
print(f"gradient of {n1} is not equal, {p1.grad} vs {p2.grad}")
else:
assert p1.grad is None and p2.grad is None
-
+
def test_stdit3_sp():
spawn(run_model, 2)
-
-if __name__ == '__main__':
+
+if __name__ == "__main__":
test_stdit3_sp()
diff --git a/tools/caption/README.md b/tools/caption/README.md
index a252363..f6fe0c8 100644
--- a/tools/caption/README.md
+++ b/tools/caption/README.md
@@ -61,7 +61,7 @@ nohup python caption_pllava.py \
```
### PLLaVA vs. LLaVA
-In our previous releases, we used [LLaVA](#llava-captioning) for video captioning.
+In our previous releases, we used [LLaVA](#llava-captioning) for video captioning.
Qualitatively speaking, we observe that PLLaVA has a somewhat higher chance of accurately capture the details in the video than LLaVA. See below for their comparison on a video sample.
<!-- <img src="../../assets/readme/llava_vs_pllava_sample.gif" width="300" height="200" alt="LLaVA vs PLLaVA"> -->
@@ -211,4 +211,3 @@ torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llama3 meta.cs
```
This will generate tags based on the `text` column of `meta.csv` and put the results to `output_prefix + key.csv`. Currently the prompts for `objects` and `actions` are supported.
-
diff --git a/tools/caption/camera_motion/camera_motion.py b/tools/caption/camera_motion/camera_motion.py
index 21ac818..9f283cb 100644
--- a/tools/caption/camera_motion/camera_motion.py
+++ b/tools/caption/camera_motion/camera_motion.py
@@ -1,10 +1,10 @@
-import torch
import os
+
import numpy as np
-from tqdm import tqdm
+import torch
-from .visualizer import Visualizer
from .utils import load_video
+from .visualizer import Visualizer
def transform(vector):
@@ -23,6 +23,7 @@ class CameraPredict:
except:
# workaround for CERTIFICATE_VERIFY_FAILED (see: https://github.com/pytorch/pytorch/issues/33288#issuecomment-954160699)
import ssl
+
ssl._create_default_https_context = ssl._create_unverified_context
self.model = torch.hub.load(submodules_list["repo"], submodules_list["model"]).to(self.device)
@@ -32,17 +33,17 @@ class CameraPredict:
# set scale
height, width = video.shape[1], video.shape[2]
self.scale = min(height, width)
- video = torch.from_numpy(video).permute(0, 3, 1, 2)[None].float().to(self.device) # B T C H W
- pred_tracks, pred_visibility = self.model(video, grid_size=self.grid_size) # B T N 2, B T N 1
-
+ video = torch.from_numpy(video).permute(0, 3, 1, 2)[None].float().to(self.device) # B T C H W
+ pred_tracks, pred_visibility = self.model(video, grid_size=self.grid_size) # B T N 2, B T N 1
+
if save_video:
video_name = os.path.basename(video_path)[:-4]
vis = Visualizer(save_dir=save_dir, pad_value=120, linewidth=3)
vis.visualize(video, pred_tracks, pred_visibility, filename=video_name)
return pred_tracks[0].long().detach().cpu().numpy()
-
- def transform_class(self, vector, min_reso): # 768*0.05
+
+ def transform_class(self, vector, min_reso): # 768*0.05
scale = min_reso * self.factor
x, y = vector
direction = []
@@ -50,7 +51,7 @@ class CameraPredict:
direction.append("right")
elif x < -scale:
direction.append("left")
-
+
if y > scale:
direction.append("down")
elif y < -scale:
@@ -60,13 +61,12 @@ class CameraPredict:
def get_edge_point(self, track):
middle = self.grid_size // 2
- top = [list(track[0, i, :]) for i in range(middle-2, middle+2)]
- down = [list(track[self.grid_size-1, i, :]) for i in range(middle-2, middle+2)]
- left = [list(track[i, 0, :]) for i in range(middle-2, middle+2)]
- right = [list(track[i, self.grid_size-1, :]) for i in range(middle-2, middle+2)]
-
+ top = [list(track[0, i, :]) for i in range(middle - 2, middle + 2)]
+ down = [list(track[self.grid_size - 1, i, :]) for i in range(middle - 2, middle + 2)]
+ left = [list(track[i, 0, :]) for i in range(middle - 2, middle + 2)]
+ right = [list(track[i, self.grid_size - 1, :]) for i in range(middle - 2, middle + 2)]
+
return top, down, left, right
-
def get_edge_direction(self, track1, track2):
edge_points1 = self.get_edge_point(track1)
@@ -74,14 +74,13 @@ class CameraPredict:
vector_results = []
for points1, points2 in zip(edge_points1, edge_points2):
- vectors = [[end[0]-start[0], end[1]-start[1]] for start, end in zip(points1, points2)]
+ vectors = [[end[0] - start[0], end[1] - start[1]] for start, end in zip(points1, points2)]
vector_results.append(vectors)
- vector_results = list(map(transform, vector_results))
+ vector_results = list(map(transform, vector_results))
class_results = [self.transform_class(vector, min_reso=self.scale) for vector in vector_results]
return class_results
-
def classify_top_down(self, top, down):
results = []
classes = [f"{item_t}_{item_d}" for item_t in top for item_d in down]
@@ -93,12 +92,11 @@ class CameraPredict:
"up_up": "tilt_down",
"up_down": "zoom_in",
"down_up": "zoom_out",
- "static_static": "static"
+ "static_static": "static",
}
results = [results_mapping.get(cls) for cls in classes if cls in results_mapping]
return results if results else ["None"]
-
def classify_left_right(self, left, right):
results = []
classes = [f"{item_l}_{item_r}" for item_l in left for item_r in right]
@@ -109,28 +107,26 @@ class CameraPredict:
"up_up": "tilt_down",
"left_right": "zoom_in",
"right_left": "zoom_out",
- "static_static": "static"
+ "static_static": "static",
}
results = [results_mapping.get(cls) for cls in classes if cls in results_mapping]
return results if results else ["None"]
-
def camera_classify(self, track1, track2):
top, down, left, right = self.get_edge_direction(track1, track2)
top_results = self.classify_top_down(top, down)
left_results = self.classify_left_right(left, right)
- results = list(set(top_results+left_results))
- if "None" in results and len(results)>1:
- results.remove("None")
- if "static" in results and len(results)>1:
+ results = list(set(top_results + left_results))
+ if "None" in results and len(results) > 1:
+ results.remove("None")
+ if "static" in results and len(results) > 1:
results.remove("static")
- if len(results) == 1 and results[0] == "None": # Tom added this to deal with edge cases
+ if len(results) == 1 and results[0] == "None": # Tom added this to deal with edge cases
results = ["Undetermined"]
return results
-
def predict(self, video_path):
pred_track = self.infer(video_path)
track1 = pred_track[0].reshape((self.grid_size, self.grid_size, 2))
@@ -146,5 +142,5 @@ def compute_camera_motion(device, submodules_dict, video_paths, factor):
all_predictions = []
for video_path in video_paths:
camera_motion_types = camera.predict(video_path)
- all_predictions.append('+'.join(camera_motion_types))
+ all_predictions.append("+".join(camera_motion_types))
return all_predictions
diff --git a/tools/caption/camera_motion/detect.py b/tools/caption/camera_motion/detect.py
index 7f41b11..7e7eff8 100644
--- a/tools/caption/camera_motion/detect.py
+++ b/tools/caption/camera_motion/detect.py
@@ -1,15 +1,16 @@
# Originally developed by https://github.com/Vchitect/VBench based on https://github.com/facebookresearch/co-tracker.
import argparse
-from .camera_motion import compute_camera_motion
-import pandas as pd
from typing import List
-import ptvsd
+
+import pandas as pd
+
+from .camera_motion import compute_camera_motion
def process(paths: List[str], threshold: float) -> List[str]:
device = "cuda"
- submodules = {'repo': 'facebookresearch/co-tracker', 'model': 'cotracker2'}
+ submodules = {"repo": "facebookresearch/co-tracker", "model": "cotracker2"}
camera_motion_types = compute_camera_motion(device, submodules, paths, factor=threshold)
return camera_motion_types
diff --git a/tools/caption/camera_motion/requirements.txt b/tools/caption/camera_motion/requirements.txt
index ce5c4bd..b699a95 100644
--- a/tools/caption/camera_motion/requirements.txt
+++ b/tools/caption/camera_motion/requirements.txt
@@ -1,3 +1,3 @@
decord
ptvsd
-imageio-ffmpeg
\ No newline at end of file
+imageio-ffmpeg
diff --git a/tools/caption/camera_motion/utils.py b/tools/caption/camera_motion/utils.py
index e05fadd..15426ea 100755
--- a/tools/caption/camera_motion/utils.py
+++ b/tools/caption/camera_motion/utils.py
@@ -1,18 +1,18 @@
import numpy as np
import torch
-from PIL import Image, ImageSequence
from decord import VideoReader
+from PIL import Image, ImageSequence
-def get_frame_indices(num_frames, vlen, sample='rand', fix_start=None, input_fps=1, max_num_frames=-1):
- if sample in ["rand", "middle"]: # uniform sampling
+def get_frame_indices(num_frames, vlen, sample="rand", fix_start=None, input_fps=1, max_num_frames=-1):
+ if sample in ["rand", "middle"]: # uniform sampling
acc_samples = min(num_frames, vlen)
# split the video into `acc_samples` intervals, and sample from each interval.
intervals = np.linspace(start=0, stop=vlen, num=acc_samples + 1).astype(int)
ranges = []
for idx, interv in enumerate(intervals[:-1]):
ranges.append((interv, intervals[idx + 1] - 1))
- if sample == 'rand':
+ if sample == "rand":
try:
frame_indices = [random.choice(range(x[0], x[1])) for x in ranges]
except:
@@ -21,14 +21,14 @@ def get_frame_indices(num_frames, vlen, sample='rand', fix_start=None, input_fps
frame_indices = list(frame_indices)
elif fix_start is not None:
frame_indices = [x[0] + fix_start for x in ranges]
- elif sample == 'middle':
+ elif sample == "middle":
frame_indices = [(x[0] + x[1]) // 2 for x in ranges]
else:
raise NotImplementedError
if len(frame_indices) < num_frames: # padded with last frame
padded_frame_indices = [frame_indices[-1]] * num_frames
- padded_frame_indices[:len(frame_indices)] = frame_indices
+ padded_frame_indices[: len(frame_indices)] = frame_indices
frame_indices = padded_frame_indices
elif "fps" in sample: # fps0.5, sequentially sample frames at 0.5 fps
output_fps = float(sample[3:])
@@ -44,24 +44,25 @@ def get_frame_indices(num_frames, vlen, sample='rand', fix_start=None, input_fps
raise ValueError
return frame_indices
+
def load_video(video_path, data_transform=None, num_frames=None, return_tensor=True, width=None, height=None):
"""
Load a video from a given path and apply optional data transformations.
The function supports loading video in GIF (.gif), PNG (.png), and MP4 (.mp4) formats.
Depending on the format, it processes and extracts frames accordingly.
-
+
Parameters:
- video_path (str): The file path to the video or image to be loaded.
- data_transform (callable, optional): A function that applies transformations to the video data.
-
+
Returns:
- frames (torch.Tensor): A tensor containing the video frames with shape (T, C, H, W),
where T is the number of frames, C is the number of channels, H is the height, and W is the width.
-
+
Raises:
- NotImplementedError: If the video format is not supported.
-
+
The function first determines the format of the video file by its extension.
For GIFs, it iterates over each frame and converts them to RGB.
For PNGs, it reads the single frame, converts it to RGB.
@@ -69,23 +70,24 @@ def load_video(video_path, data_transform=None, num_frames=None, return_tensor=T
If a data_transform is provided, it is applied to the buffer before converting it to a tensor.
Finally, the tensor is permuted to match the expected (T, C, H, W) format.
"""
- if video_path.endswith('.gif'):
+ if video_path.endswith(".gif"):
frame_ls = []
img = Image.open(video_path)
for frame in ImageSequence.Iterator(img):
- frame = frame.convert('RGB')
+ frame = frame.convert("RGB")
frame = np.array(frame).astype(np.uint8)
frame_ls.append(frame)
buffer = np.array(frame_ls).astype(np.uint8)
- elif video_path.endswith('.png'):
+ elif video_path.endswith(".png"):
frame = Image.open(video_path)
- frame = frame.convert('RGB')
+ frame = frame.convert("RGB")
frame = np.array(frame).astype(np.uint8)
frame_ls = [frame]
buffer = np.array(frame_ls)
- elif video_path.endswith('.mp4'):
+ elif video_path.endswith(".mp4"):
import decord
- decord.bridge.set_bridge('native')
+
+ decord.bridge.set_bridge("native")
if width:
video_reader = VideoReader(video_path, width=width, height=height, num_threads=1)
else:
@@ -95,14 +97,12 @@ def load_video(video_path, data_transform=None, num_frames=None, return_tensor=T
buffer = frames.asnumpy().astype(np.uint8)
else:
raise NotImplementedError
-
+
frames = buffer
if num_frames:
- frame_indices = get_frame_indices(
- num_frames, len(frames), sample="middle"
- )
+ frame_indices = get_frame_indices(num_frames, len(frames), sample="middle")
frames = frames[frame_indices]
-
+
if data_transform:
frames = data_transform(frames)
elif return_tensor:
diff --git a/tools/caption/camera_motion/visualizer.py b/tools/caption/camera_motion/visualizer.py
index edb14f8..f4f2dc6 100644
--- a/tools/caption/camera_motion/visualizer.py
+++ b/tools/caption/camera_motion/visualizer.py
@@ -3,14 +3,14 @@
# This source code is licensed under the license found in the cotracker github repo. https://github.com/facebookresearch/co-tracker.
import os
-import numpy as np
+
import imageio
+import matplotlib.pyplot as plt
+import numpy as np
import torch
-
-from matplotlib import cm
import torch.nn.functional as F
import torchvision.transforms as transforms
-import matplotlib.pyplot as plt
+from matplotlib import cm
from PIL import Image, ImageDraw
@@ -108,7 +108,7 @@ class Visualizer:
)
print("video shape after pad is: ", video.shape)
tracks = tracks + self.pad_value
-
+
print(tracks)
print("tracks shape after pad is: ", tracks.shape)
@@ -230,16 +230,13 @@ class Visualizer:
# draw tracks
if self.tracks_leave_trace != 0:
for t in range(query_frame + 1, T):
- first_ind = (
- max(0, t - self.tracks_leave_trace) if self.tracks_leave_trace >= 0 else 0
- )
+ first_ind = max(0, t - self.tracks_leave_trace) if self.tracks_leave_trace >= 0 else 0
curr_tracks = tracks[first_ind : t + 1]
curr_colors = vector_colors[first_ind : t + 1]
if compensate_for_camera_motion:
- diff = (
- tracks[first_ind : t + 1, segm_mask <= 0]
- - tracks[t : t + 1, segm_mask <= 0]
- ).mean(1)[:, None]
+ diff = (tracks[first_ind : t + 1, segm_mask <= 0] - tracks[t : t + 1, segm_mask <= 0]).mean(1)[
+ :, None
+ ]
curr_tracks = curr_tracks - diff
curr_tracks = curr_tracks[:, segm_mask > 0]
@@ -262,9 +259,7 @@ class Visualizer:
if visibility is not None:
visibile = visibility[0, t, i]
if coord[0] != 0 and coord[1] != 0:
- if not compensate_for_camera_motion or (
- compensate_for_camera_motion and segm_mask[i] > 0
- ):
+ if not compensate_for_camera_motion or (compensate_for_camera_motion and segm_mask[i] > 0):
img = draw_circle(
img,
coord=coord,
@@ -304,9 +299,7 @@ class Visualizer:
self.linewidth,
)
if self.tracks_leave_trace > 0:
- rgb = Image.fromarray(
- np.uint8(add_weighted(np.array(rgb), alpha, np.array(original), 1 - alpha, 0))
- )
+ rgb = Image.fromarray(np.uint8(add_weighted(np.array(rgb), alpha, np.array(original), 1 - alpha, 0)))
rgb = np.array(rgb)
return rgb
diff --git a/tools/caption/caption_llama3.py b/tools/caption/caption_llama3.py
index e7ff5ef..754f5a6 100644
--- a/tools/caption/caption_llama3.py
+++ b/tools/caption/caption_llama3.py
@@ -1,30 +1,30 @@
-from datetime import timedelta
import argparse
import csv
import os
+import warnings
+from datetime import timedelta
+
+import pandas as pd
+import torch
import torch.distributed as dist
-import argparse
-import csv
-import os
from torch.utils.data import Dataset
-
from tqdm import tqdm
-from transformers import AutoTokenizer, AutoModelForCausalLM
-import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
from .utils import read_file
-import warnings
-import pandas as pd
-os.system(f'cp {__file__} ~/backup/') # optionally backup the script
+
+os.system(f"cp {__file__} ~/backup/") # optionally backup the script
warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"
from torch.distributed.elastic.multiprocessing.errors import record
+
class CSVTextDataset(Dataset):
def __init__(self, csv_path):
self.df = pd.read_csv(csv_path)
# assert text is in the columns
- assert 'text' in self.df.columns, "text column not found in the csv file"
-
+ assert "text" in self.df.columns, "text column not found in the csv file"
+
def __len__(self):
return len(self.df)
@@ -39,10 +39,10 @@ class CSVTextDataset(Dataset):
self.data_per_gpu = len(self) // world_size
self.start_index = rank * self.data_per_gpu
self.end_index = (rank + 1) * self.data_per_gpu if rank != world_size - 1 else len(self)
- self.df = self.df.iloc[self.start_index:self.end_index]
-
+ self.df = self.df.iloc[self.start_index : self.end_index]
+
def write_to_csv(self, output_file, data, new_key):
- ''' write the part of the df to a csv file corresponding to the rank and write self.data_list as a new column '''
+ """write the part of the df to a csv file corresponding to the rank and write self.data_list as a new column"""
writer = csv.writer(open(output_file, "w"))
columns = self.df.columns + [new_key]
writer.writerow(columns)
@@ -50,8 +50,9 @@ class CSVTextDataset(Dataset):
if index < self.start_index or index >= self.end_index:
continue
writer.writerow([*row, data[index - self.start_index]])
- writer.close()
-
+ writer.close()
+
+
def pad_left(sequences, padding_value=0):
# Determine the maximum length of the sequences
max_len = max([s.size(0) for s in sequences])
@@ -69,6 +70,7 @@ def pad_left(sequences, padding_value=0):
batch = torch.stack(padded_sequences)
return batch
+
@record
def main(args):
# ======================================================
@@ -76,30 +78,29 @@ def main(args):
# ======================================================
dist.init_process_group(backend="nccl", timeout=timedelta(hours=24))
torch.cuda.set_device(dist.get_rank() % torch.cuda.device_count())
-
+
# ======================================================
# 2. Prep rank-wise dataloader
# ======================================================
dataframe = read_file(args.input)
print("read data from {}".format(args.input))
dataset = CSVTextDataset(args.input)
- dataset.set_rank_and_world_size(
- dist.get_rank(),
- dist.get_world_size()
- )
-
+ dataset.set_rank_and_world_size(dist.get_rank(), dist.get_world_size())
+
import os
- if os.getenv('DEBUG_ADDRESS') != None and dist.get_rank() == 2:
- import ptvsd;
- print('waiting for debugger attachment')
- ptvsd.enable_attach(address=('localhost', int(os.getenv('DEBUG_ADDRESS'))), redirect_output=True)
+
+ if os.getenv("DEBUG_ADDRESS") != None and dist.get_rank() == 2:
+ import ptvsd
+
+ print("waiting for debugger attachment")
+ ptvsd.enable_attach(address=("localhost", int(os.getenv("DEBUG_ADDRESS"))), redirect_output=True)
ptvsd.wait_for_attach()
-
+
output_file = args.output_prefix + f"_rank{dist.get_rank()}" + f"_{args.key}.csv"
output_file_handle = open(output_file, "w")
writer = csv.writer(output_file_handle)
columns = list(dataframe.columns) + [args.key]
-
+
writer.writerow(columns)
# add a new key named summary, write in csv file
@@ -115,7 +116,7 @@ def main(args):
collate_fn=collate_fn,
shuffle=False,
)
-
+
# ======================================================
# 2. process using llama3 and prompt
# ======================================================
@@ -135,40 +136,35 @@ def main(args):
def extract_batch(texts, prompt):
input_ids_list = [
tokenizer.apply_chat_template(
- [
- {"role": "system", "content": prompt},
- {"role": "user", "content": text}
- ],
+ [{"role": "system", "content": prompt}, {"role": "user", "content": text}],
add_generation_prompt=True,
- return_tensors="pt"
+ return_tensors="pt",
).to(model.device)[0]
for text in texts
]
-
+
attention_mask_list = [
- torch.ones(input_ids.shape, dtype=torch.long, device=model.device)
- for input_ids in input_ids_list
+ torch.ones(input_ids.shape, dtype=torch.long, device=model.device) for input_ids in input_ids_list
]
-
+
# input_ids_batch = pad_left(
# input_ids_list, padding_value=tokenizer.eos_token_id
# )
-
+
input_ids_batch = torch.nn.utils.rnn.pad_sequence(
input_ids_list, batch_first=True, padding_value=tokenizer.eos_token_id
)
-
- attention_mask_batch = torch.nn.utils.rnn.pad_sequence(
- attention_mask_list, batch_first=True, padding_value=0
- )
-
+
+ attention_mask_batch = torch.nn.utils.rnn.pad_sequence(attention_mask_list, batch_first=True, padding_value=0)
+
# attention_mask_batch = pad_left(
# attention_mask_list, padding_value=0
# )
- terminators = [tokenizer.eos_token_id,
- tokenizer.convert_tokens_to_ids("<|eot_id|>"),
- ]
+ terminators = [
+ tokenizer.eos_token_id,
+ tokenizer.convert_tokens_to_ids("<|eot_id|>"),
+ ]
outputs = model.generate(
input_ids_batch,
max_new_tokens=512,
@@ -182,7 +178,7 @@ def main(args):
responses = []
for i in range(len(texts)):
- response = outputs[i][input_ids_list[i].shape[-1]:]
+ response = outputs[i][input_ids_list[i].shape[-1] :]
response = tokenizer.decode(response, skip_special_tokens=True)
responses.append(response)
@@ -190,42 +186,49 @@ def main(args):
print("Processing starting...")
if args.prompt == "" and args.key == "objects":
- prompt = ("You are a AI assistant to extract objects from user's text. "
- "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of objects separated by ',' and wrapped by '[' and ']': '[dog, person]' ")
+ prompt = (
+ "You are a AI assistant to extract objects from user's text. "
+ "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of objects separated by ',' and wrapped by '[' and ']': '[dog, person]' "
+ )
elif args.prompt == "" and args.key == "actions":
- prompt = ("You are a AI assistant to extract actions from user's text. "
- "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of actions separated by ',' and wrapped by '[' and ']': '[run, laugh]' ")
+ prompt = (
+ "You are a AI assistant to extract actions from user's text. "
+ "For example: user: 'In this video a dog is running around. In addition, a person is laughing at the dog.', you produce a list of actions separated by ',' and wrapped by '[' and ']': '[run, laugh]' "
+ )
else:
prompt = args.prompt
print("Prompt: {}".format(prompt))
- batch_size = args.batch_size
+ args.batch_size
# for i in tqdm(range(0, len(dataframe), batch_size)):
for _, batch in enumerate(tqdm(dataloader)):
# get the text column from the batch
- texts = [batch[i]['text'] for i in range(len(batch))]
+ texts = [batch[i]["text"] for i in range(len(batch))]
list_keywords = extract_batch(texts, prompt)
-
+
for idx, keywords in enumerate(list_keywords):
try:
keywords_start = keywords.find("[")
keywords_end = keywords.find("]")
- keywords = keywords[keywords_start+1:keywords_end]
- if "\n" in keywords or len(keywords.strip()) == 0: # we empirically observe that it produces newlines when no keywords are found
+ keywords = keywords[keywords_start + 1 : keywords_end]
+ if (
+ "\n" in keywords or len(keywords.strip()) == 0
+ ): # we empirically observe that it produces newlines when no keywords are found
keywords = "NONE_FOUND"
except:
keywords = "NONE_FOUND"
row = batch[idx]
writer.writerow([*row, keywords])
-
+
output_file_handle.close()
dist.barrier()
-
+
if dist.get_rank() == 0:
collated_file = args.output_prefix + f"_{args.key}.csv"
print("All ranks are finished. Collating the processed data to {}".format(collated_file))
import pandas as pd
+
csv_files = [args.output_prefix + f"_rank{i}" + f"_{args.key}.csv" for i in range(dist.get_world_size())]
# List to hold DataFrames
dataframes = []
@@ -252,7 +255,7 @@ if __name__ == "__main__":
parser.add_argument("--output_prefix", type=str, help="Path to the output CSV file")
parser.add_argument("--prompt", type=str, default="")
parser.add_argument("--batch_size", type=int, default=32)
- parser.add_argument('--key', type=str)
+ parser.add_argument("--key", type=str)
args = parser.parse_args()
- main(args)
\ No newline at end of file
+ main(args)
diff --git a/tools/caption/caption_llava.py b/tools/caption/caption_llava.py
index 7946128..ab69145 100644
--- a/tools/caption/caption_llava.py
+++ b/tools/caption/caption_llava.py
@@ -17,9 +17,9 @@ from llava.utils import disable_torch_init
from torch.utils.data.distributed import DistributedSampler
from tqdm import tqdm
+from ..datasets.utils import IMG_EXTENSIONS, VID_EXTENSIONS
from .acceleration.llava.policies import LlavaLlamaForCausalLMPolicy, LlavaMistralForCausalLMPolicy
from .utils import PROMPTS, Timer, VideoTextDataset, collate_fn
-from ..datasets.utils import IMG_EXTENSIONS, VID_EXTENSIONS
disable_torch_init()
diff --git a/tools/caption/pllava_dir/caption_pllava.py b/tools/caption/pllava_dir/caption_pllava.py
index 6f65152..2213367 100644
--- a/tools/caption/pllava_dir/caption_pllava.py
+++ b/tools/caption/pllava_dir/caption_pllava.py
@@ -1,58 +1,81 @@
-from multiprocessing import Process, Queue
-import os
-import pandas as pd
-import functools
import itertools
import logging
-from tqdm import tqdm
-from PIL import Image
-from multiprocessing import Pool
import multiprocessing as mp
from argparse import ArgumentParser
+from multiprocessing import Process, Queue
+
import numpy as np
-from torch.utils.data import Dataset
+import pandas as pd
import torch
import torchvision
-from decord import VideoReader, cpu
import transformers
-import pandas as pd
-import itertools
-from transformers.feature_extraction_utils import BatchFeature
-from tasks.eval.model_utils import load_pllava
+from decord import VideoReader, cpu
+from PIL import Image
from tasks.eval.eval_utils import Conversation
+from tasks.eval.model_utils import load_pllava
+from torch.utils.data import Dataset
+from tqdm import tqdm
+from transformers.feature_extraction_utils import BatchFeature
conv_template = Conversation(
system="Describe this video. Pay attention to all objects in the video. The description should be useful for AI to re-generate the video. The description should be no more than six sentences. Here are some examples of good descriptions: 1. A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about. 2. Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field. 3. Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff's edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.",
roles=("USER:", "ASSISTANT:"),
messages=[],
sep=(" ", "</s>"),
- mm_token='<image>'
+ mm_token="<image>",
)
logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
-RESOLUTION = 672 #
+RESOLUTION = 672 #
+
-def pllava_answer(conv: Conversation, model, processor, video_list, do_sample=True, max_new_tokens=200, num_beams=1, min_length=1, top_p=0.9, repetition_penalty=1.0, length_penalty=1, temperature=1.0, stop_criteria_keywords=None, print_res=False):
+def pllava_answer(
+ conv: Conversation,
+ model,
+ processor,
+ video_list,
+ do_sample=True,
+ max_new_tokens=200,
+ num_beams=1,
+ min_length=1,
+ top_p=0.9,
+ repetition_penalty=1.0,
+ length_penalty=1,
+ temperature=1.0,
+ stop_criteria_keywords=None,
+ print_res=False,
+):
# torch.cuda.empty_cache()
prompt = conv.get_prompt()
inputs_list = [processor(text=prompt, images=video, return_tensors="pt") for video in video_list]
- inputs_batched = dict() # add batch dimension by cat
+ inputs_batched = dict() # add batch dimension by cat
for input_type in list(inputs_list[0].keys()):
inputs_batched[input_type] = torch.cat([inputs[input_type] for inputs in inputs_list])
inputs_batched = BatchFeature(inputs_batched, tensor_type="pt").to(model.device)
with torch.no_grad():
- output_texts = model.generate(**inputs_batched, media_type='video',
- do_sample=do_sample, max_new_tokens=max_new_tokens, num_beams=num_beams, min_length=min_length,
- top_p=top_p, repetition_penalty=repetition_penalty, length_penalty=length_penalty, temperature=temperature)
- output_texts = processor.batch_decode(output_texts, skip_special_tokens=True, clean_up_tokenization_spaces=False)
+ output_texts = model.generate(
+ **inputs_batched,
+ media_type="video",
+ do_sample=do_sample,
+ max_new_tokens=max_new_tokens,
+ num_beams=num_beams,
+ min_length=min_length,
+ top_p=top_p,
+ repetition_penalty=repetition_penalty,
+ length_penalty=length_penalty,
+ temperature=temperature,
+ )
+ output_texts = processor.batch_decode(
+ output_texts, skip_special_tokens=True, clean_up_tokenization_spaces=False
+ )
for i in range(len(output_texts)):
- if print_res: # debug usage
- print('### PROMPTING LM WITH: ', prompt)
- print('### LM OUTPUT TEXT: ', output_texts[i])
+ if print_res: # debug usage
+ print("### PROMPTING LM WITH: ", prompt)
+ print("### LM OUTPUT TEXT: ", output_texts[i])
if conv.roles[-1] == "<|im_start|>assistant\n":
split_tag = "<|im_start|> assistant\n"
else:
@@ -60,18 +83,18 @@ def pllava_answer(conv: Conversation, model, processor, video_list, do_sample=Tr
output_texts[i] = output_texts[i].split(split_tag)[-1]
ending = conv.sep if isinstance(conv.sep, str) else conv.sep[1]
output_texts[i] = output_texts[i].removesuffix(ending).strip()
- output_texts[i] = output_texts[i].replace('\n', ' ')
+ output_texts[i] = output_texts[i].replace("\n", " ")
conv.messages[-1][1] = output_texts[i]
return output_texts, conv
+
def get_index(num_frames, num_segments):
seg_size = float(num_frames - 1) / num_segments
start = int(seg_size / 2)
- offsets = np.array([
- start + int(np.round(seg_size * idx)) for idx in range(num_segments)
- ])
+ offsets = np.array([start + int(np.round(seg_size * idx)) for idx in range(num_segments)])
return offsets
+
def load_video(video_path, num_frames, return_msg=False, resolution=336):
transforms = torchvision.transforms.Resize(size=resolution)
vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
@@ -90,15 +113,17 @@ def load_video(video_path, num_frames, return_msg=False, resolution=336):
else:
return images_group
+
def collate_fn(batch):
return batch
+
class CSVDataset(Dataset):
def __init__(self, csv_path, num_frames):
self.df = pd.read_csv(csv_path)
self.data_list = self.df.path.tolist()
self.num_frames = num_frames
-
+
def __len__(self):
return len(self.data_list)
@@ -116,19 +141,15 @@ class CSVDataset(Dataset):
end_index = (rank + 1) * self.data_per_gpu if rank != world_size - 1 else len(self)
self.data_list = self.data_list[start_index:end_index]
+
def parse_args():
parser = ArgumentParser()
- parser.add_argument(
- "--pretrained_model_name_or_path",
- type=str,
- required=True,
- default='llava-hf/llava-1.5-7b-hf'
- )
+ parser.add_argument("--pretrained_model_name_or_path", type=str, required=True, default="llava-hf/llava-1.5-7b-hf")
parser.add_argument(
"--batch_size",
type=int,
required=False,
- default=1,
+ default=1,
)
parser.add_argument(
"--csv_path",
@@ -141,10 +162,7 @@ def parse_args():
required=True,
default=4,
)
- parser.add_argument(
- "--use_lora",
- action='store_true'
- )
+ parser.add_argument("--use_lora", action="store_true")
parser.add_argument(
"--lora_alpha",
type=int,
@@ -158,13 +176,13 @@ def parse_args():
default=None,
)
parser.add_argument(
- "--conv_mode",
+ "--conv_mode",
type=str,
required=False,
- default='eval_mvbench',
+ default="eval_mvbench",
)
parser.add_argument(
- "--pooling_shape",
+ "--pooling_shape",
type=str,
required=False,
default=None,
@@ -178,10 +196,28 @@ def parse_args():
args = parser.parse_args()
return args
-def load_model_and_dataset(rank, world_size, pretrained_model_name_or_path, num_frames, use_lora, lora_alpha, weight_dir, csv_path, pooling_shape=(16,12,12)):
+
+def load_model_and_dataset(
+ rank,
+ world_size,
+ pretrained_model_name_or_path,
+ num_frames,
+ use_lora,
+ lora_alpha,
+ weight_dir,
+ csv_path,
+ pooling_shape=(16, 12, 12),
+):
# remind that, once the model goes larger (30B+) may cause the memory to be heavily used up. Even Tearing Nodes.
- model, processor = load_pllava(pretrained_model_name_or_path, num_frames=num_frames, use_lora=use_lora, weight_dir=weight_dir, lora_alpha=lora_alpha, pooling_shape=pooling_shape)
- logger.info('done loading llava')
+ model, processor = load_pllava(
+ pretrained_model_name_or_path,
+ num_frames=num_frames,
+ use_lora=use_lora,
+ weight_dir=weight_dir,
+ lora_alpha=lora_alpha,
+ pooling_shape=pooling_shape,
+ )
+ logger.info("done loading llava")
# position embedding
model = model.to(torch.device(rank))
@@ -191,13 +227,14 @@ def load_model_and_dataset(rank, world_size, pretrained_model_name_or_path, num_
dataset.set_rank_and_world_size(rank, world_size)
return model, processor, dataset
-def infer(\
- model,
- processor,
- video_list,
- conv_mode,
- print_res=True,
- ):
+
+def infer(
+ model,
+ processor,
+ video_list,
+ conv_mode,
+ print_res=True,
+):
conv = conv_template.copy()
conv.user_query("Describe the video in details.", is_mm=True)
@@ -208,39 +245,46 @@ def infer(\
video_list=video_list,
max_new_tokens=256,
do_sample=False,
- print_res=print_res
+ print_res=print_res,
)
-
+
return llm_responses
+
def run(rank, args, world_size, output_queue):
if rank == 0:
- import os;
- if os.getenv('DEBUG_ADDRESS') != None:
- import ptvsd; ptvsd.enable_attach(address=('localhost', int(os.getenv('DEBUG_ADDRESS'))), redirect_output=True); ptvsd.wait_for_attach(); print('waiting for debugger attachment')
- pass
+ import os
+
+ if os.getenv("DEBUG_ADDRESS") != None:
+ import ptvsd
+
+ ptvsd.enable_attach(address=("localhost", int(os.getenv("DEBUG_ADDRESS"))), redirect_output=True)
+ ptvsd.wait_for_attach()
+ print("waiting for debugger attachment")
if rank != 0:
transformers.utils.logging.set_verbosity_error()
logger.setLevel(transformers.logging.ERROR)
print_res = False
- conv_mode= args.conv_mode
+ conv_mode = args.conv_mode
if args.pooling_shape is not None:
- pooling_shape=tuple([int(x) for x in args.pooling_shape.split("-")])
-
- logger.info(f'loading model and constructing dataset to gpu {rank}...')
- model, processor, dataset = load_model_and_dataset(rank,
- world_size,
- pretrained_model_name_or_path=args.pretrained_model_name_or_path,
- num_frames=args.num_frames,
- use_lora=args.use_lora,
- lora_alpha=args.lora_alpha,
- weight_dir=args.weight_dir,
- pooling_shape=pooling_shape,
- csv_path=args.csv_path)
- logger.info(f'done model and dataset...')
- logger.info('constructing dataset...')
- logger.info('single test...')
+ pooling_shape = tuple([int(x) for x in args.pooling_shape.split("-")])
+
+ logger.info(f"loading model and constructing dataset to gpu {rank}...")
+ model, processor, dataset = load_model_and_dataset(
+ rank,
+ world_size,
+ pretrained_model_name_or_path=args.pretrained_model_name_or_path,
+ num_frames=args.num_frames,
+ use_lora=args.use_lora,
+ lora_alpha=args.lora_alpha,
+ weight_dir=args.weight_dir,
+ pooling_shape=pooling_shape,
+ csv_path=args.csv_path,
+ )
+ logger.info(f"done model and dataset...")
+ logger.info("constructing dataset...")
+ logger.info("single test...")
dataloader = torch.utils.data.DataLoader(
dataset,
num_workers=2,
@@ -248,7 +292,7 @@ def run(rank, args, world_size, output_queue):
collate_fn=collate_fn,
shuffle=False,
)
-
+
total = 0
result_list = []
print(len(dataset))
@@ -263,24 +307,25 @@ def run(rank, args, world_size, output_queue):
print_res=print_res,
)
except Exception as e:
- logger.error(f'error in {batch}: {str(e)}')
+ logger.error(f"error in {batch}: {str(e)}")
preds = args.error_message
result_list.extend(preds)
output_queue.put((rank, result_list))
return result_list
+
def main():
multiprocess = True
- mp.set_start_method('spawn')
+ mp.set_start_method("spawn")
args = parse_args()
# csv_path = '/home/tom/PLLaVA/test_short_caption_part2.csv'
if multiprocess:
n_gpus = torch.cuda.device_count()
world_size = n_gpus
- print(f'world_size: {world_size}')
+ print(f"world_size: {world_size}")
# Create a queue to collect results from each process
output_queue = Queue()
-
+
# with Pool(world_size) as pool:
# func = functools.partial(run, args=args, world_size=world_size)
# result_lists = pool.map(func, range(world_size))
@@ -291,39 +336,40 @@ def main():
p.daemon = False
processes.append(p)
p.start()
-
+
results_by_rank = {}
for _ in range(world_size):
rank, results = output_queue.get() # Retrieve results as they finish
results_by_rank[rank] = results
- print(f'Results received from rank {rank}')
+ print(f"Results received from rank {rank}")
# ORDER THE RESULTS BY RANK
- logger.info('finished running')
+ logger.info("finished running")
for p in processes:
p.join()
-
+
results_list = list(itertools.chain.from_iterable(results_by_rank[i] for i in range(world_size)))
# results_list = list(itertools.chain([results_by_rank[i] for i in range(world_size)]))
# (data[key] for key in sorted_keys)
# results_list = [item for sublist in results_by_rank.values() for item in sublist]
else:
- results_list = run(0, world_size=1, args=args) # debug
-
+ results_list = run(0, world_size=1, args=args) # debug
+
print(results_list)
df = pd.read_csv(args.csv_path)
# add a new column to the dataframe
- df['text'] = results_list
+ df["text"] = results_list
drop_failed = True
if drop_failed:
# iterate through the dataframe and delete the entire row if captioning failed
for i in tqdm(range(len(df))):
- if df['text'][i] == args.error_message:
+ if df["text"][i] == args.error_message:
df = df.drop(i)
# write the dataframe to a new csv file called '*_pllava_13b_caption.csv'
- new_csv_path = args.csv_path.replace('.csv', '_text.csv')
+ new_csv_path = args.csv_path.replace(".csv", "_text.csv")
df.to_csv(new_csv_path, index=False)
+
if __name__ == "__main__":
- main()
\ No newline at end of file
+ main()
diff --git a/tools/caption/utils.py b/tools/caption/utils.py
index 42c3d56..b1137de 100644
--- a/tools/caption/utils.py
+++ b/tools/caption/utils.py
@@ -6,6 +6,7 @@ import torchvision.transforms as transforms
from torchvision.datasets.folder import pil_loader
from tools.datasets.utils import extract_frames, is_video
+
IMG_EXTENSIONS = (".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp")
VID_EXTENSIONS = (".mp4", ".avi", ".mov", ".mkv")
PROMPTS = {
diff --git a/tools/datasets/convert.py b/tools/datasets/convert.py
index 2d49c4d..6253e8e 100644
--- a/tools/datasets/convert.py
+++ b/tools/datasets/convert.py
@@ -110,7 +110,7 @@ def process_general_videos(root, output):
if not os.path.exists(root):
return
path_list = get_filelist(root, VID_EXTENSIONS)
- path_list = list(set(path_list)) # remove duplicates
+ path_list = list(set(path_list)) # remove duplicates
fname_list = [os.path.splitext(os.path.basename(x))[0] for x in path_list]
relpath_list = [os.path.relpath(x, root) for x in path_list]
df = pd.DataFrame(dict(path=path_list, id=fname_list, relpath=relpath_list))
@@ -126,7 +126,7 @@ if __name__ == "__main__":
parser.add_argument("root", type=str)
parser.add_argument("--split", type=str, default="train")
parser.add_argument("--info", type=str, default=None)
- parser.add_argument("--output", type=str, default=None, required=True, help='Output path')
+ parser.add_argument("--output", type=str, default=None, required=True, help="Output path")
args = parser.parse_args()
if args.dataset == "imagenet":
diff --git a/tools/datasets/filter_panda10m.py b/tools/datasets/filter_panda10m.py
index c6c3ef3..86a9f19 100644
--- a/tools/datasets/filter_panda10m.py
+++ b/tools/datasets/filter_panda10m.py
@@ -1,262 +1,262 @@
-# TODO: remove this file before releasing
-
-import argparse
-import os
-import pandas as pd
-import json
-import html
-from tqdm import tqdm
-import re
-
-tqdm.pandas()
-
-try:
- from pandarallel import pandarallel
-
- pandarallel.initialize(progress_bar=True)
- pandas_has_parallel = True
-except ImportError:
- pandas_has_parallel = False
-
-
-def apply(df, func, **kwargs):
- if pandas_has_parallel:
- return df.parallel_apply(func, **kwargs)
- return df.progress_apply(func, **kwargs)
-
-
-def basic_clean(text):
- import ftfy
-
- text = ftfy.fix_text(text)
- text = html.unescape(html.unescape(text))
- return text.strip()
-
-
-BAD_PUNCT_REGEX = re.compile(
- r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"
-) # noqa
-
-
-def clean_caption(caption):
- import urllib.parse as ul
-
- from bs4 import BeautifulSoup
-
- caption = str(caption)
- caption = ul.unquote_plus(caption)
- caption = caption.strip().lower()
- caption = re.sub("<person>", "person", caption)
- # urls:
- caption = re.sub(
- r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
- "",
- caption,
- ) # regex for urls
- caption = re.sub(
- r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
- "",
- caption,
- ) # regex for urls
- # html:
- caption = BeautifulSoup(caption, features="html.parser").text
-
- # @<nickname>
- caption = re.sub(r"@[\w\d]+\b", "", caption)
-
- # 31C0—31EF CJK Strokes
- # 31F0—31FF Katakana Phonetic Extensions
- # 3200—32FF Enclosed CJK Letters and Months
- # 3300—33FF CJK Compatibility
- # 3400—4DBF CJK Unified Ideographs Extension A
- # 4DC0—4DFF Yijing Hexagram Symbols
- # 4E00—9FFF CJK Unified Ideographs
- caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)
- caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)
- caption = re.sub(r"[\u3200-\u32ff]+", "", caption)
- caption = re.sub(r"[\u3300-\u33ff]+", "", caption)
- caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)
- caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)
- caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)
- #######################################################
-
- # все виды тире / all types of dash --> "-"
- caption = re.sub(
- r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa
- "-",
- caption,
- )
-
- # кавычки к одному стандарту
- caption = re.sub(r"[`´«»“”¨]", '"', caption)
- caption = re.sub(r"[‘’]", "'", caption)
-
- # &quot;
- caption = re.sub(r"&quot;?", "", caption)
- # &amp
- caption = re.sub(r"&amp", "", caption)
-
- # ip adresses:
- caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)
-
- # article ids:
- caption = re.sub(r"\d:\d\d\s+$", "", caption)
-
- # \n
- caption = re.sub(r"\\n", " ", caption)
-
- # "#123"
- caption = re.sub(r"#\d{1,3}\b", "", caption)
- # "#12345.."
- caption = re.sub(r"#\d{5,}\b", "", caption)
- # "123456.."
- caption = re.sub(r"\b\d{6,}\b", "", caption)
- # filenames:
- caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)
-
- #
- caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""
- caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""
-
- caption = re.sub(BAD_PUNCT_REGEX, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT
- caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "
-
- # this-is-my-cute-cat / this_is_my_cute_cat
- regex2 = re.compile(r"(?:\-|\_)")
- if len(re.findall(regex2, caption)) > 3:
- caption = re.sub(regex2, " ", caption)
-
- caption = basic_clean(caption)
-
- caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640
- caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc
- caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231
-
- caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)
- caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)
- caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)
- caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)
- caption = re.sub(r"\bpage\s+\d+\b", "", caption)
-
- caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...
-
- caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)
-
- caption = re.sub(r"\b\s+\:\s+", r": ", caption)
- caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)
- caption = re.sub(r"\s+", " ", caption)
-
- caption.strip()
-
- caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)
- caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)
- caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)
- caption = re.sub(r"^\.\S+$", "", caption)
-
- return caption.strip()
-
-
-def get_10m_set():
- meta_path_10m = '/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv'
- meta_10m = pd.read_csv(meta_path_10m)
-
- def process_single_caption(row):
- text_list = eval(row['caption'])
- clean_list = [clean_caption(x) for x in text_list]
- return str(clean_list)
-
- ret = apply(meta_10m, process_single_caption, axis=1)
- # ret = meta_10m.progress_apply(process_single_caption, axis=1)
- print('==> text processed.')
-
- text_list = []
- for x in ret:
- text_list += eval(x)
- # text_set = text_set.union(set(eval(x)))
- text_set = set(text_list)
- # meta_10m['caption_new'] = ret
- # meta_10m.to_csv('/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m_new-cap.csv')
-
- # video_id_set = set(meta_10m['videoID'])
- # id2t = {}
- # for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):
- # video_id = row['videoID']
- # text_list = eval(row['caption'])
- # id2t[video_id] = set(text_list)
-
- print(f"==> Loaded meta_10m from '{meta_path_10m}'")
- return text_set
-
-
-def filter_panda10m_text(meta_path, text_set):
- def process_single_row(row):
- # path = row['path']
- t = row['text']
- # fname = os.path.basename(path)
- # video_id = fname[:fname.rindex('_')]
- if t not in text_set:
- return False
- return True
-
- meta = pd.read_csv(meta_path)
- ret = apply(meta, process_single_row, axis=1)
- # ret = meta.progress_apply(process_single_row, axis=1)
-
- meta = meta[ret]
- wo_ext, ext = os.path.splitext(meta_path)
- out_path = f"{wo_ext}_filter-10m{ext}"
- meta.to_csv(out_path, index=False)
- print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")
-
-
-def filter_panda10m_timestamp(meta_path):
- meta_path_10m = '/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv'
- meta_10m = pd.read_csv(meta_path_10m)
-
- id2t = {}
- for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):
- video_id = row['videoID']
- timestamp = eval(row['timestamp'])
- timestamp = [str(tuple(x)) for x in timestamp]
- id2t[video_id] = timestamp
-
- # video_id_set_10m = set(meta_10m['videoID'])
- print(f"==> Loaded meta_10m from '{meta_path_10m}'")
-
- def process_single_row(row):
- path = row['path']
- t = row['timestamp']
- fname = os.path.basename(path)
- video_id = fname[:fname.rindex('_')]
- if video_id not in id2t:
- return False
- if t not in id2t[video_id]:
- return False
- return True
- # return video_id in video_id_set_10m
-
- meta = pd.read_csv(meta_path)
- ret = apply(meta, process_single_row, axis=1)
-
- meta = meta[ret]
- wo_ext, ext = os.path.splitext(meta_path)
- out_path = f"{wo_ext}_filter-10m{ext}"
- meta.to_csv(out_path, index=False)
- print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")
-
-
-def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument('--meta_path', type=str, nargs='+')
- parser.add_argument('--num_workers', default=5, type=int)
-
- args = parser.parse_args()
- return args
-
-
-if __name__ == '__main__':
- args = parse_args()
-
- text_set = get_10m_set()
- for x in args.meta_path:
- filter_panda10m_text(x, text_set)
+# TODO: remove this file before releasing
+
+import argparse
+import html
+import os
+import re
+
+import pandas as pd
+from tqdm import tqdm
+
+tqdm.pandas()
+
+try:
+ from pandarallel import pandarallel
+
+ pandarallel.initialize(progress_bar=True)
+ pandas_has_parallel = True
+except ImportError:
+ pandas_has_parallel = False
+
+
+def apply(df, func, **kwargs):
+ if pandas_has_parallel:
+ return df.parallel_apply(func, **kwargs)
+ return df.progress_apply(func, **kwargs)
+
+
+def basic_clean(text):
+ import ftfy
+
+ text = ftfy.fix_text(text)
+ text = html.unescape(html.unescape(text))
+ return text.strip()
+
+
+BAD_PUNCT_REGEX = re.compile(
+ r"[" + "#®•©™&@·º½¾¿¡§~" + "\)" + "\(" + "\]" + "\[" + "\}" + "\{" + "\|" + "\\" + "\/" + "\*" + r"]{1,}"
+) # noqa
+
+
+def clean_caption(caption):
+ import urllib.parse as ul
+
+ from bs4 import BeautifulSoup
+
+ caption = str(caption)
+ caption = ul.unquote_plus(caption)
+ caption = caption.strip().lower()
+ caption = re.sub("<person>", "person", caption)
+ # urls:
+ caption = re.sub(
+ r"\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
+ "",
+ caption,
+ ) # regex for urls
+ caption = re.sub(
+ r"\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))", # noqa
+ "",
+ caption,
+ ) # regex for urls
+ # html:
+ caption = BeautifulSoup(caption, features="html.parser").text
+
+ # @<nickname>
+ caption = re.sub(r"@[\w\d]+\b", "", caption)
+
+ # 31C0—31EF CJK Strokes
+ # 31F0—31FF Katakana Phonetic Extensions
+ # 3200—32FF Enclosed CJK Letters and Months
+ # 3300—33FF CJK Compatibility
+ # 3400—4DBF CJK Unified Ideographs Extension A
+ # 4DC0—4DFF Yijing Hexagram Symbols
+ # 4E00—9FFF CJK Unified Ideographs
+ caption = re.sub(r"[\u31c0-\u31ef]+", "", caption)
+ caption = re.sub(r"[\u31f0-\u31ff]+", "", caption)
+ caption = re.sub(r"[\u3200-\u32ff]+", "", caption)
+ caption = re.sub(r"[\u3300-\u33ff]+", "", caption)
+ caption = re.sub(r"[\u3400-\u4dbf]+", "", caption)
+ caption = re.sub(r"[\u4dc0-\u4dff]+", "", caption)
+ caption = re.sub(r"[\u4e00-\u9fff]+", "", caption)
+ #######################################################
+
+ # все виды тире / all types of dash --> "-"
+ caption = re.sub(
+ r"[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+", # noqa
+ "-",
+ caption,
+ )
+
+ # кавычки к одному стандарту
+ caption = re.sub(r"[`´«»“”¨]", '"', caption)
+ caption = re.sub(r"[‘’]", "'", caption)
+
+ # &quot;
+ caption = re.sub(r"&quot;?", "", caption)
+ # &amp
+ caption = re.sub(r"&amp", "", caption)
+
+ # ip adresses:
+ caption = re.sub(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", " ", caption)
+
+ # article ids:
+ caption = re.sub(r"\d:\d\d\s+$", "", caption)
+
+ # \n
+ caption = re.sub(r"\\n", " ", caption)
+
+ # "#123"
+ caption = re.sub(r"#\d{1,3}\b", "", caption)
+ # "#12345.."
+ caption = re.sub(r"#\d{5,}\b", "", caption)
+ # "123456.."
+ caption = re.sub(r"\b\d{6,}\b", "", caption)
+ # filenames:
+ caption = re.sub(r"[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)", "", caption)
+
+ #
+ caption = re.sub(r"[\"\']{2,}", r'"', caption) # """AUSVERKAUFT"""
+ caption = re.sub(r"[\.]{2,}", r" ", caption) # """AUSVERKAUFT"""
+
+ caption = re.sub(BAD_PUNCT_REGEX, r" ", caption) # ***AUSVERKAUFT***, #AUSVERKAUFT
+ caption = re.sub(r"\s+\.\s+", r" ", caption) # " . "
+
+ # this-is-my-cute-cat / this_is_my_cute_cat
+ regex2 = re.compile(r"(?:\-|\_)")
+ if len(re.findall(regex2, caption)) > 3:
+ caption = re.sub(regex2, " ", caption)
+
+ caption = basic_clean(caption)
+
+ caption = re.sub(r"\b[a-zA-Z]{1,3}\d{3,15}\b", "", caption) # jc6640
+ caption = re.sub(r"\b[a-zA-Z]+\d+[a-zA-Z]+\b", "", caption) # jc6640vc
+ caption = re.sub(r"\b\d+[a-zA-Z]+\d+\b", "", caption) # 6640vc231
+
+ caption = re.sub(r"(worldwide\s+)?(free\s+)?shipping", "", caption)
+ caption = re.sub(r"(free\s)?download(\sfree)?", "", caption)
+ caption = re.sub(r"\bclick\b\s(?:for|on)\s\w+", "", caption)
+ caption = re.sub(r"\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?", "", caption)
+ caption = re.sub(r"\bpage\s+\d+\b", "", caption)
+
+ caption = re.sub(r"\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b", r" ", caption) # j2d1a2a...
+
+ caption = re.sub(r"\b\d+\.?\d*[xх×]\d+\.?\d*\b", "", caption)
+
+ caption = re.sub(r"\b\s+\:\s+", r": ", caption)
+ caption = re.sub(r"(\D[,\./])\b", r"\1 ", caption)
+ caption = re.sub(r"\s+", " ", caption)
+
+ caption.strip()
+
+ caption = re.sub(r"^[\"\']([\w\W]+)[\"\']$", r"\1", caption)
+ caption = re.sub(r"^[\'\_,\-\:;]", r"", caption)
+ caption = re.sub(r"[\'\_,\-\:\-\+]$", r"", caption)
+ caption = re.sub(r"^\.\S+$", "", caption)
+
+ return caption.strip()
+
+
+def get_10m_set():
+ meta_path_10m = "/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv"
+ meta_10m = pd.read_csv(meta_path_10m)
+
+ def process_single_caption(row):
+ text_list = eval(row["caption"])
+ clean_list = [clean_caption(x) for x in text_list]
+ return str(clean_list)
+
+ ret = apply(meta_10m, process_single_caption, axis=1)
+ # ret = meta_10m.progress_apply(process_single_caption, axis=1)
+ print("==> text processed.")
+
+ text_list = []
+ for x in ret:
+ text_list += eval(x)
+ # text_set = text_set.union(set(eval(x)))
+ text_set = set(text_list)
+ # meta_10m['caption_new'] = ret
+ # meta_10m.to_csv('/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m_new-cap.csv')
+
+ # video_id_set = set(meta_10m['videoID'])
+ # id2t = {}
+ # for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):
+ # video_id = row['videoID']
+ # text_list = eval(row['caption'])
+ # id2t[video_id] = set(text_list)
+
+ print(f"==> Loaded meta_10m from '{meta_path_10m}'")
+ return text_set
+
+
+def filter_panda10m_text(meta_path, text_set):
+ def process_single_row(row):
+ # path = row['path']
+ t = row["text"]
+ # fname = os.path.basename(path)
+ # video_id = fname[:fname.rindex('_')]
+ if t not in text_set:
+ return False
+ return True
+
+ meta = pd.read_csv(meta_path)
+ ret = apply(meta, process_single_row, axis=1)
+ # ret = meta.progress_apply(process_single_row, axis=1)
+
+ meta = meta[ret]
+ wo_ext, ext = os.path.splitext(meta_path)
+ out_path = f"{wo_ext}_filter-10m{ext}"
+ meta.to_csv(out_path, index=False)
+ print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")
+
+
+def filter_panda10m_timestamp(meta_path):
+ meta_path_10m = "/mnt/hdd/data/Panda-70M/raw/meta/train/panda70m_training_10m.csv"
+ meta_10m = pd.read_csv(meta_path_10m)
+
+ id2t = {}
+ for idx, row in tqdm(meta_10m.iterrows(), total=len(meta_10m)):
+ video_id = row["videoID"]
+ timestamp = eval(row["timestamp"])
+ timestamp = [str(tuple(x)) for x in timestamp]
+ id2t[video_id] = timestamp
+
+ # video_id_set_10m = set(meta_10m['videoID'])
+ print(f"==> Loaded meta_10m from '{meta_path_10m}'")
+
+ def process_single_row(row):
+ path = row["path"]
+ t = row["timestamp"]
+ fname = os.path.basename(path)
+ video_id = fname[: fname.rindex("_")]
+ if video_id not in id2t:
+ return False
+ if t not in id2t[video_id]:
+ return False
+ return True
+ # return video_id in video_id_set_10m
+
+ meta = pd.read_csv(meta_path)
+ ret = apply(meta, process_single_row, axis=1)
+
+ meta = meta[ret]
+ wo_ext, ext = os.path.splitext(meta_path)
+ out_path = f"{wo_ext}_filter-10m{ext}"
+ meta.to_csv(out_path, index=False)
+ print(f"New meta (shape={meta.shape}) saved to '{out_path}'.")
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--meta_path", type=str, nargs="+")
+ parser.add_argument("--num_workers", default=5, type=int)
+
+ args = parser.parse_args()
+ return args
+
+
+if __name__ == "__main__":
+ args = parse_args()
+
+ text_set = get_10m_set()
+ for x in args.meta_path:
+ filter_panda10m_text(x, text_set)
diff --git a/tools/scene_cut/convert_id_to_path.py b/tools/scene_cut/convert_id_to_path.py
index d217a78..b8122d3 100644
--- a/tools/scene_cut/convert_id_to_path.py
+++ b/tools/scene_cut/convert_id_to_path.py
@@ -1,15 +1,14 @@
-import os
-
import argparse
import json
+import os
from functools import partial
+import cv2
import numpy as np
import pandas as pd
-from pandarallel import pandarallel
-import cv2
from mmengine.logging import print_log
from moviepy.editor import VideoFileClip
+from pandarallel import pandarallel
from tqdm import tqdm
tqdm.pandas()
@@ -68,7 +67,7 @@ def parse_args():
parser.add_argument("meta_path", type=str)
parser.add_argument("--folder_path", type=str, required=True)
parser.add_argument("--mode", type=str, default=None)
- parser.add_argument("--num_workers", type=int, default=None, help='#workers for pandarallel')
+ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")
args = parser.parse_args()
return args
diff --git a/tools/scene_cut/scene_detect.py b/tools/scene_cut/scene_detect.py
index 0df68d7..c564d73 100644
--- a/tools/scene_cut/scene_detect.py
+++ b/tools/scene_cut/scene_detect.py
@@ -34,7 +34,7 @@ def process_single_row(row):
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("meta_path", type=str)
- parser.add_argument("--num_workers", type=int, default=None, help='#workers for pandarallel')
+ parser.add_argument("--num_workers", type=int, default=None, help="#workers for pandarallel")
args = parser.parse_args()
return args
@@ -44,7 +44,7 @@ def main():
args = parse_args()
meta_path = args.meta_path
if not os.path.exists(meta_path):
- print(f"Meta file \'{meta_path}\' not found. Exit.")
+ print(f"Meta file '{meta_path}' not found. Exit.")
exit()
if args.num_workers is not None:
diff --git a/tools/scoring/aesthetic/inference.py b/tools/scoring/aesthetic/inference.py
index 06a8e92..be6618f 100644
--- a/tools/scoring/aesthetic/inference.py
+++ b/tools/scoring/aesthetic/inference.py
@@ -122,7 +122,7 @@ def parse_args():
parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")
parser.add_argument("--prefetch_factor", type=int, default=3, help="Prefetch factor")
parser.add_argument("--num_frames", type=int, default=3, help="Number of frames to extract")
- parser.add_argument("--skip_if_existing", action='store_true')
+ parser.add_argument("--skip_if_existing", action="store_true")
args = parser.parse_args()
return args
@@ -133,13 +133,13 @@ def main():
meta_path = args.meta_path
if not os.path.exists(meta_path):
- print(f"Meta file \'{meta_path}\' not found. Exit.")
+ print(f"Meta file '{meta_path}' not found. Exit.")
exit()
wo_ext, ext = os.path.splitext(meta_path)
out_path = f"{wo_ext}_aes{ext}"
if args.skip_if_existing and os.path.exists(out_path):
- print(f"Output meta file \'{out_path}\' already exists. Exit.")
+ print(f"Output meta file '{out_path}' already exists. Exit.")
exit()
dist.init_process_group(backend="nccl", timeout=timedelta(hours=24))
@@ -190,9 +190,11 @@ def main():
# save local results
meta_local = merge_scores([(indices_list, scores_list)], dataset.meta, column="aes")
- save_dir_local = os.path.join(os.path.dirname(out_path), 'parts')
+ save_dir_local = os.path.join(os.path.dirname(out_path), "parts")
os.makedirs(save_dir_local, exist_ok=True)
- out_path_local = os.path.join(save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv"))
+ out_path_local = os.path.join(
+ save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv")
+ )
meta_local.to_csv(out_path_local, index=False)
# wait for all ranks to finish data processing
diff --git a/tools/scoring/matching/inference.py b/tools/scoring/matching/inference.py
index e5cd95b..70209eb 100644
--- a/tools/scoring/matching/inference.py
+++ b/tools/scoring/matching/inference.py
@@ -66,7 +66,7 @@ def parse_args():
parser.add_argument("meta_path", type=str, help="Path to the input CSV file")
parser.add_argument("--bs", type=int, default=16, help="Batch size")
parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")
- parser.add_argument("--skip_if_existing", action='store_true')
+ parser.add_argument("--skip_if_existing", action="store_true")
args = parser.parse_args()
return args
@@ -76,16 +76,15 @@ def main():
meta_path = args.meta_path
if not os.path.exists(meta_path):
- print(f"Meta file \'{meta_path}\' not found. Exit.")
+ print(f"Meta file '{meta_path}' not found. Exit.")
exit()
wo_ext, ext = os.path.splitext(meta_path)
out_path = f"{wo_ext}_match{ext}"
if args.skip_if_existing and os.path.exists(out_path):
- print(f"Output meta file \'{out_path}\' already exists. Exit.")
+ print(f"Output meta file '{out_path}' already exists. Exit.")
exit()
-
colossalai.launch_from_torch({})
# build model
@@ -130,7 +129,7 @@ def main():
gathered_list = [None] * dist.get_world_size()
dist.all_gather_object(gathered_list, (indices_list, scores_list))
if dist.get_rank() == 0:
- meta_new = merge_scores(gathered_list, dataset.meta, column='match')
+ meta_new = merge_scores(gathered_list, dataset.meta, column="match")
meta_new.to_csv(out_path, index=False)
print(f"New meta with matching scores saved to '{out_path}'.")
diff --git a/tools/scoring/ocr/dbnetpp.py b/tools/scoring/ocr/dbnetpp.py
index e313fd4..7c64615 100644
--- a/tools/scoring/ocr/dbnetpp.py
+++ b/tools/scoring/ocr/dbnetpp.py
@@ -1,64 +1,65 @@
model = dict(
- type='DBNet',
+ type="DBNet",
backbone=dict(
- type='CLIPResNet',
+ type="CLIPResNet",
depth=50,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=-1,
- norm_cfg=dict(type='BN', requires_grad=True),
+ norm_cfg=dict(type="BN", requires_grad=True),
norm_eval=False,
- style='pytorch',
- dcn=dict(type='DCNv2', deform_groups=1, fallback_on_stride=False),
+ style="pytorch",
+ dcn=dict(type="DCNv2", deform_groups=1, fallback_on_stride=False),
# init_cfg=dict(
# type='Pretrained',
# checkpoint='https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth'),
stage_with_dcn=(False, True, True, True),
),
neck=dict(
- type='FPNC',
+ type="FPNC",
in_channels=[256, 512, 1024, 2048],
lateral_channels=256,
- asf_cfg=dict(attention_type='ScaleChannelSpatial'),
+ asf_cfg=dict(attention_type="ScaleChannelSpatial"),
),
det_head=dict(
- type='DBHead',
+ type="DBHead",
in_channels=256,
- module_loss=dict(type='DBModuleLoss'),
+ module_loss=dict(type="DBModuleLoss"),
postprocessor=dict(
- type='DBPostprocessor', text_repr_type='quad',
+ type="DBPostprocessor",
+ text_repr_type="quad",
epsilon_ratio=0.002,
),
),
data_preprocessor=dict(
- type='TextDetDataPreprocessor',
+ type="TextDetDataPreprocessor",
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True,
pad_size_divisor=32,
),
init_cfg=dict(
- type='Pretrained',
- checkpoint='https://download.openmmlab.com/mmocr/textdet/dbnetpp/'
- 'dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/'
- 'dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth',
- )
+ type="Pretrained",
+ checkpoint="https://download.openmmlab.com/mmocr/textdet/dbnetpp/"
+ "dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/"
+ "dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth",
+ ),
)
test_pipeline = [
# dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
- dict(type='Resize', scale=(4068, 1024), keep_ratio=True),
+ dict(type="Resize", scale=(4068, 1024), keep_ratio=True),
dict(
- type='PackTextDetInputs',
+ type="PackTextDetInputs",
# meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor'),
- meta_keys=('img_shape', 'scale_factor'),
- )
+ meta_keys=("img_shape", "scale_factor"),
+ ),
]
# Visualization
-vis_backends = [dict(type='LocalVisBackend')]
+vis_backends = [dict(type="LocalVisBackend")]
visualizer = dict(
- type='TextDetLocalVisualizer',
- name='visualizer',
+ type="TextDetLocalVisualizer",
+ name="visualizer",
vis_backends=vis_backends,
)
diff --git a/tools/scoring/ocr/inference.py b/tools/scoring/ocr/inference.py
index 5045fed..95a1737 100644
--- a/tools/scoring/ocr/inference.py
+++ b/tools/scoring/ocr/inference.py
@@ -1,158 +1,158 @@
-import argparse
-import os
-
-import numpy as np
-import pandas as pd
-import torch
-import torch.distributed as dist
-import torch.nn.functional as F
-from torchvision.transforms import Resize, CenterCrop, Compose
-from torch.utils.data import DataLoader, DistributedSampler
-from torchvision.datasets.folder import pil_loader
-from tqdm import tqdm
-
-import colossalai
-from mmengine import Config
-from mmengine.registry import DefaultScope
-from mmengine.dataset import Compose, default_collate
-from mmocr.registry import MODELS
-from mmocr.datasets import PackTextDetInputs
-
-from tools.datasets.utils import extract_frames, is_video
-
-
-def merge_scores(gathered_list: list, meta: pd.DataFrame):
- # reorder
- indices_list = list(map(lambda x: x[0], gathered_list))
- scores_list = list(map(lambda x: x[1], gathered_list))
- flat_indices = []
- for x in zip(*indices_list):
- flat_indices.extend(x)
- flat_scores = []
- for x in zip(*scores_list):
- flat_scores.extend(x)
- flat_indices = np.array(flat_indices)
- flat_scores = np.array(flat_scores)
- # filter duplicates
- unique_indices, unique_indices_idx = np.unique(flat_indices, return_index=True)
- meta.loc[unique_indices, "ocr"] = flat_scores[unique_indices_idx]
-
-
-class VideoTextDataset(torch.utils.data.Dataset):
- def __init__(self, meta_path, transform):
- self.meta_path = meta_path
- self.meta = pd.read_csv(meta_path)
- self.transform = transform
- self.transform = Compose([
- Resize(1024),
- CenterCrop(1024),
- ])
- self.formatting = PackTextDetInputs(meta_keys=['scale_factor'])
-
- def __getitem__(self, index):
- row = self.meta.iloc[index]
- path = row["path"]
-
- if is_video(path):
- img = extract_frames(path, frame_inds=[10], backend="opencv")[0]
- else:
- img = pil_loader(path)
-
- img = self.transform(img)
- img_array = np.array(img)[:, :, ::-1].copy() # bgr
- results = {
- 'img': img_array,
- 'scale_factor': 1.0,
- # 'img_shape': img_array.shape[-2],
- # 'ori_shape': img_array.shape[-2],
- }
- results = self.formatting(results)
- results['index'] = index
-
- return results
-
- def __len__(self):
- return len(self.meta)
-
-
-def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument("meta_path", type=str, help="Path to the input CSV file")
- parser.add_argument("--bs", type=int, default=16, help="Batch size")
- parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")
- parser.add_argument("--skip_if_existing", action='store_true')
- args = parser.parse_args()
-
- return args
-
-
-def main():
- args = parse_args()
-
- meta_path = args.meta_path
- if not os.path.exists(meta_path):
- print(f"Meta file \'{meta_path}\' not found. Exit.")
- exit()
-
- wo_ext, ext = os.path.splitext(meta_path)
- out_path = f"{wo_ext}_ocr{ext}"
- if args.skip_if_existing and os.path.exists(out_path):
- print(f"Output meta file \'{out_path}\' already exists. Exit.")
- exit()
-
- cfg = Config.fromfile('./tools/scoring/ocr/dbnetpp.py')
- colossalai.launch_from_torch({})
-
- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- DefaultScope.get_instance('ocr', scope_name='mmocr') # use mmocr Registry as default
-
- # build model
- model = MODELS.build(cfg.model)
- model.init_weights()
- model.to(device) # set data_preprocessor._device
- print('==> Model built.')
-
- # build dataset
- transform = Compose(cfg.test_pipeline)
- dataset = VideoTextDataset(meta_path=meta_path, transform=transform)
- dataloader = DataLoader(
- dataset,
- batch_size=args.bs,
- num_workers=args.num_workers,
- sampler=DistributedSampler(
- dataset,
- num_replicas=dist.get_world_size(),
- rank=dist.get_rank(),
- shuffle=False,
- drop_last=False,
- ),
- collate_fn=default_collate,
- )
- print('==> Dataloader built.')
-
- # compute scores
- dataset.meta["ocr"] = np.nan
- indices_list = []
- scores_list = []
- model.eval()
- for data in tqdm(dataloader, disable=dist.get_rank() != 0):
- indices_i = data['index']
- indices_list.extend(indices_i.tolist())
- del data['index']
-
- pred = model.test_step(data) # this line will cast data to device
-
- num_texts_i = [(x.pred_instances.scores > 0.3).sum().item() for x in pred]
- scores_list.extend(num_texts_i)
-
- gathered_list = [None] * dist.get_world_size()
- dist.all_gather_object(gathered_list, (indices_list, scores_list))
-
- if dist.get_rank() == 0:
- merge_scores(gathered_list, dataset.meta)
- dataset.meta.to_csv(out_path, index=False)
- print(f"New meta (shape={dataset.meta.shape}) with ocr results saved to '{out_path}'.")
-
-
-if __name__ == '__main__':
- main()
+import argparse
+import os
+
+import colossalai
+import numpy as np
+import pandas as pd
+import torch
+import torch.distributed as dist
+from mmengine import Config
+from mmengine.dataset import Compose, default_collate
+from mmengine.registry import DefaultScope
+from mmocr.datasets import PackTextDetInputs
+from mmocr.registry import MODELS
+from torch.utils.data import DataLoader, DistributedSampler
+from torchvision.datasets.folder import pil_loader
+from torchvision.transforms import CenterCrop, Compose, Resize
+from tqdm import tqdm
+
+from tools.datasets.utils import extract_frames, is_video
+
+
+def merge_scores(gathered_list: list, meta: pd.DataFrame):
+ # reorder
+ indices_list = list(map(lambda x: x[0], gathered_list))
+ scores_list = list(map(lambda x: x[1], gathered_list))
+ flat_indices = []
+ for x in zip(*indices_list):
+ flat_indices.extend(x)
+ flat_scores = []
+ for x in zip(*scores_list):
+ flat_scores.extend(x)
+ flat_indices = np.array(flat_indices)
+ flat_scores = np.array(flat_scores)
+ # filter duplicates
+ unique_indices, unique_indices_idx = np.unique(flat_indices, return_index=True)
+ meta.loc[unique_indices, "ocr"] = flat_scores[unique_indices_idx]
+
+
+class VideoTextDataset(torch.utils.data.Dataset):
+ def __init__(self, meta_path, transform):
+ self.meta_path = meta_path
+ self.meta = pd.read_csv(meta_path)
+ self.transform = transform
+ self.transform = Compose(
+ [
+ Resize(1024),
+ CenterCrop(1024),
+ ]
+ )
+ self.formatting = PackTextDetInputs(meta_keys=["scale_factor"])
+
+ def __getitem__(self, index):
+ row = self.meta.iloc[index]
+ path = row["path"]
+
+ if is_video(path):
+ img = extract_frames(path, frame_inds=[10], backend="opencv")[0]
+ else:
+ img = pil_loader(path)
+
+ img = self.transform(img)
+ img_array = np.array(img)[:, :, ::-1].copy() # bgr
+ results = {
+ "img": img_array,
+ "scale_factor": 1.0,
+ # 'img_shape': img_array.shape[-2],
+ # 'ori_shape': img_array.shape[-2],
+ }
+ results = self.formatting(results)
+ results["index"] = index
+
+ return results
+
+ def __len__(self):
+ return len(self.meta)
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("meta_path", type=str, help="Path to the input CSV file")
+ parser.add_argument("--bs", type=int, default=16, help="Batch size")
+ parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")
+ parser.add_argument("--skip_if_existing", action="store_true")
+ args = parser.parse_args()
+
+ return args
+
+
+def main():
+ args = parse_args()
+
+ meta_path = args.meta_path
+ if not os.path.exists(meta_path):
+ print(f"Meta file '{meta_path}' not found. Exit.")
+ exit()
+
+ wo_ext, ext = os.path.splitext(meta_path)
+ out_path = f"{wo_ext}_ocr{ext}"
+ if args.skip_if_existing and os.path.exists(out_path):
+ print(f"Output meta file '{out_path}' already exists. Exit.")
+ exit()
+
+ cfg = Config.fromfile("./tools/scoring/ocr/dbnetpp.py")
+ colossalai.launch_from_torch({})
+
+ device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
+ DefaultScope.get_instance("ocr", scope_name="mmocr") # use mmocr Registry as default
+
+ # build model
+ model = MODELS.build(cfg.model)
+ model.init_weights()
+ model.to(device) # set data_preprocessor._device
+ print("==> Model built.")
+
+ # build dataset
+ transform = Compose(cfg.test_pipeline)
+ dataset = VideoTextDataset(meta_path=meta_path, transform=transform)
+ dataloader = DataLoader(
+ dataset,
+ batch_size=args.bs,
+ num_workers=args.num_workers,
+ sampler=DistributedSampler(
+ dataset,
+ num_replicas=dist.get_world_size(),
+ rank=dist.get_rank(),
+ shuffle=False,
+ drop_last=False,
+ ),
+ collate_fn=default_collate,
+ )
+ print("==> Dataloader built.")
+
+ # compute scores
+ dataset.meta["ocr"] = np.nan
+ indices_list = []
+ scores_list = []
+ model.eval()
+ for data in tqdm(dataloader, disable=dist.get_rank() != 0):
+ indices_i = data["index"]
+ indices_list.extend(indices_i.tolist())
+ del data["index"]
+
+ pred = model.test_step(data) # this line will cast data to device
+
+ num_texts_i = [(x.pred_instances.scores > 0.3).sum().item() for x in pred]
+ scores_list.extend(num_texts_i)
+
+ gathered_list = [None] * dist.get_world_size()
+ dist.all_gather_object(gathered_list, (indices_list, scores_list))
+
+ if dist.get_rank() == 0:
+ merge_scores(gathered_list, dataset.meta)
+ dataset.meta.to_csv(out_path, index=False)
+ print(f"New meta (shape={dataset.meta.shape}) with ocr results saved to '{out_path}'.")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/scoring/optical_flow/inference.py b/tools/scoring/optical_flow/inference.py
index 31a6011..09f1b41 100644
--- a/tools/scoring/optical_flow/inference.py
+++ b/tools/scoring/optical_flow/inference.py
@@ -80,7 +80,7 @@ def parse_args():
parser.add_argument("meta_path", type=str, help="Path to the input CSV file")
parser.add_argument("--bs", type=int, default=4, help="Batch size") # don't use too large bs for unimatch
parser.add_argument("--num_workers", type=int, default=16, help="Number of workers")
- parser.add_argument("--skip_if_existing", action='store_true')
+ parser.add_argument("--skip_if_existing", action="store_true")
args = parser.parse_args()
return args
@@ -90,13 +90,13 @@ def main():
meta_path = args.meta_path
if not os.path.exists(meta_path):
- print(f"Meta file \'{meta_path}\' not found. Exit.")
+ print(f"Meta file '{meta_path}' not found. Exit.")
exit()
wo_ext, ext = os.path.splitext(meta_path)
out_path = f"{wo_ext}_flow{ext}"
if args.skip_if_existing and os.path.exists(out_path):
- print(f"Output meta file \'{out_path}\' already exists. Exit.")
+ print(f"Output meta file '{out_path}' already exists. Exit.")
exit()
torch.backends.cudnn.deterministic = True
@@ -169,9 +169,11 @@ def main():
# save local results
meta_local = merge_scores([(indices_list, scores_list)], dataset.meta, column="flow")
- save_dir_local = os.path.join(os.path.dirname(out_path), 'parts')
+ save_dir_local = os.path.join(os.path.dirname(out_path), "parts")
os.makedirs(save_dir_local, exist_ok=True)
- out_path_local = os.path.join(save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv"))
+ out_path_local = os.path.join(
+ save_dir_local, os.path.basename(out_path).replace(".csv", f"_part_{dist.get_rank()}.csv")
+ )
meta_local.to_csv(out_path_local, index=False)
# wait for all ranks to finish data processing