Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

Feature/installation (#109)

Shen Chenhuia57c6c71bbfd
diff --git a/.gitignore b/.gitignore
index f0a41f1..14dc0e3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -128,6 +128,7 @@ ENV/
env.bak/
venv.bak/
+
# Spyder project settings
.spyderproject
.spyproject
@@ -192,5 +193,5 @@ node_modules/
package-lock.json
package.json
-
+# PLLaVA
tools/caption/pllava_dir/PLLaVA/
diff --git a/README.md b/README.md
index 2cdb604..4ef88ea 100644
--- a/README.md
+++ b/README.md
@@ -154,15 +154,15 @@ conda create -n opensora python=3.10
conda activate opensora
# install torch, torchvision and xformers
-pip install -r requirements/requirements_cu121.txt
+pip install -r requirements/requirements-cu121.txt
# download the repo
git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora
-# install Open-Sora
-pip install -v .
-```
+# the default installation is for inference only
+pip install -v . # for development mode, `pip install -v -e .`
+
(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.
diff --git a/docs/installation.md b/docs/installation.md
index 9c41d0d..71a36cd 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -1,28 +1,185 @@
# Installation
Requirements are listed in `requirements` folder.
+Note that besides these packages, some packages needs to be mannually installed, and are detailed in the following sections.
-## Different CUDA versions
+# Different CUDA versions
-You need to mannually install `torch`, `torchvision` and `xformers` for different CUDA versions.
+You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions.
+For CUDA 12.1,
```bash
-# install torch (>=2.1 is recommended)
-# the command below is for CUDA 12.1, choose install commands from
-# https://pytorch.org/get-started/locally/ based on your own CUDA version
-pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121
+# need to update first, else may run into weird issues with apex
+pip install -U pip
+pip install -U setuptools
+pip install -U wheel
-# install xformers
-# the command below is for CUDA 12.1, choose install commands from
-# https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers based on your own CUDA version
-pip install xformers --index-url https://download.pytorch.org/whl/cu121
+# install pytorch, torchvision, and xformers
+pip install -r requirements/requirements-cu121.txt
+
+# install this project
+git clone https://github.com/hpcaitech/Open-Sora
+cd Open-Sora
+
+# the default installation is for inference only
+pip install -v . # NOTE: for development mode, run `pip install -v -e .`
+
+(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.
+
+# install flash attention
+# set enable_flash_attn=False in config to disable flash attention
+pip install packaging ninja
+pip install flash-attn --no-build-isolation
+
+# install apex, the compilation will take a long time
+# set enable_layernorm_kernel=False in config to disable apex
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
+```
+
+# Optional Dependencies
+
+The default installation is for inference only. Other optional dependencies are detailed in sections below, namel "Data Dependencies" and "Evaluation Dependencies".
+
+<!-- ```bash
+pip install -v .[data] # for data preprocessing, still need to manually install some packages detailed in sections below
+pip install -v .[eval] # for evaluation, still need to manually install some packages detailed in sections below
+
+# Note: if in development mode, use the following commands instead:
+pip install -v -e .[data]
+pip install -v -e .[eval]
+``` -->
+
+
+## Data Dependencies
+
+First, run the following command to install requirements:
+```bash
+pip install -v .[data] # For development: `pip install -v -e .[eval]`
```
+Next, you need to manually install the packages listed in the following sections specific to your data processing needs.
-## Different Dependencies
+### Datasets
+To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)
-The default installation is for inference only. Other optional dependencies are listed below.
+However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:
+```bash
+conda install -c conda-forge opencv
+```
+
+### LLaVA Captioning
+You need to manually install LLaVA with the following command:
+```bash
+pip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]
+```
+
+### PLLaVA Captioning
+
+You need to manually install PLLaVa with the following commands:
+```bash
+cd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directory
+git clone https://github.com/magic-research/PLLaVA.git
+cd PLLaVA
+git checkout fd9194a # since there is no version tag, we use this commit
+python python_scripts/hf.py # download the PLLaVA weights
+
+# IMPORTANT: create new environment for reliable pllava performances:
+conda create -n pllava python=3.10
+# You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions, the following works for CUDA 12.1:
+conda activate pllava
+pip install -r ../../../requirements/requirements-cu121.txt
+pip install packaging ninja
+pip install flash-attn --no-build-isolation
+# You may manually remove any lines in requirements.txt that contains `cu11`, then run `pip install -r requirements.txt`
+# Alternatively, use our prepared pllava environment:
+pip install -r ../../../../requirements/requirements-pllava.txt
+```
+
+
+### Frame Interpolation
+```bash
+conda install -c conda-forge opencv
+```
+### Scene Detection
+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:
```bash
-pip install -v .[data] # for data preprocessing
-pip install -v .[eval] # for evaluation
+pip install scenedetect[opencv] --upgrade
+```
+
+### OCR
+
+You need to go into `path_to_your_env/lib/python3.10/site-packages/mmdet/__init__.py`
+and change the assert of `mmcv_version < digit_version(mmcv_maximum_version)` to `mmcv_version <= digit_version(mmcv_maximum_version)`.
+
+If you are unsure of your path to the mmdet init file, simply run our [OCR command](../tools/scoring/README.md), wait for the mmdeet assertion error on mmcv versions.
+The error will contain the exact path to the mmdet init file.
+
+
+<!-- We need to manualy create new environment for torch==2.0.1 in order to install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
+This is because the current latest MMOCR version (1.0.1) is not compatible with higher versions of mmcv that work with higher torch versions.
+
+```bash
+conda create -n ocr python=3.8
+conda activate ocr
+nvcc --version # Check that you have CUDA 11.8 or 11.7, if not install CUDA first
+pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118
+pip install packaging==24.0
+pip install openmim==0.3.9
+mim install mmengine==0.10.4
+mim install mmcv==2.0.1
+mim install mmdet==3.1.0
+mim install mmocr==1.0.1
+pip install colossalai==0.3.6
+
+# install apex
+pip install -U pip
+pip install -U wheel
+pip install setuptools==60.2.0
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
+``` -->
+
+
+
+## Evaluation Dependencies
+
+First, run the following command to install requirements:
+```bash
+pip install -v .[eval] # For development:`pip install -v -e .[eval]`
+```
+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.
+
+### Human Eval
+
+You need to manually install apex from source by:
+```bash
+# use latest pip, setuptools, and wheel; else may run into weird issues with apex
+pip install -U pip
+pip install -U setuptools
+pip install -U wheel
+# set enable_layernorm_kernel=False in config to disable apex
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
+
+```
+
+### VBench
+You need to manually install [VBench](https://github.com/Vchitect/VBench):
+```bash
+pip install --no-deps vbench==0.1.1
+# If the installation shows a warning about the intalled vbench not in PATH, you need to add it by:
+export PATH="/path/to/vbench:$PATH"
+```
+
+
+### VAE
+
+You need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).
+* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`
+* For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`
+
+Note that for VAE evaluation, you may run into error with `ModuleNotFoundError: No module named 'torchvision.transforms.functional_tensor'`, in this case, you need to go to the corresponding file (`.../pytorchvideo/transforms/augmentations.py`) reporting this error, then change as following:
+```yaml
+# find the original line:
+import torchvision.transforms.functional_tensor as F_t
+# change to:
+import torchvision.transforms._functional_tensor as F_t
```
diff --git a/eval/README.md b/eval/README.md
index 891fd87..b80ab0c 100644
--- a/eval/README.md
+++ b/eval/README.md
@@ -6,11 +6,11 @@ To conduct human evaluation, we need to generate various samples. We provide man
```bash
# image (1)
-bash eval/sample.sh /path/to/ckpt -1
+bash eval/sample.sh /path/to/ckpt num_frames model_name_for_log -1
# video (2a 2b 2c ...)
-bash eval/sample.sh /path/to/ckpt -2a
+bash eval/sample.sh /path/to/ckpt num_frames model_name_for_log -2a
# launch 8 jobs at once (you must read the script to understand the details)
-bash eval/launch.sh /path/to/ckpt
+bash eval/human_eval/launch.sh /path/to/ckpt num_frames model_name_for_log
```
## Rectified Flow Loss
@@ -31,21 +31,23 @@ torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora
To launch multiple jobs at once, use the following script.
```bash
-bash eval/loss/launch.sh /path/to/ckpt
+bash eval/loss/launch.sh /path/to/ckpt model_name
```
## VBench
[VBench](https://github.com/Vchitect/VBench) is a benchmark for short text to video generation. We provide a script for easily generating samples required by VBench.
+First, generate the relevant videos with the following commands:
+
```bash
# vbench tasks (4a 4b 4c ...)
-bash eval/sample.sh /path/to/ckpt -4a
+bash eval/sample.sh /path/to/ckpt -4a
# launch 8 jobs at once (you must read the script to understand the details)
-bash eval/launch.sh /path/to/ckpt
+bash eval/vbench/launch.sh /path/to/ckpt
```
-After generation, install the VBench package according to their [instructions](https://github.com/Vchitect/VBench?tab=readme-ov-file#hammer-installation). Then, run the following commands to evaluate the generated samples.
+After generation, install the VBench package following our [installation](../docs/installation.md)'s sections of "Evaluation Dependencies". Then, run the following commands to evaluate the generated samples.
```bash
bash eval/vbench/vbench.sh /path/to/video_folder
@@ -59,24 +61,9 @@ TBD
## VAE
-### Dependencies
-
-- Install cupy: follow https://docs.cupy.dev/en/stable/install.html
-
-- To use flolpips model, download from https://github.com/danier97/flolpips/blob/main/weights/v0.1/alex.pth and place it under: `eval/vae/flolpips/weights/v0.1/alex.pth`
-
-``` bash
-pip install decord
-pip install pytorchvideo
-pip install lpips
-pip install scipy
-# Also, if torchvision.transforms.augentation still use `functional_tensor` and cause error,change to use `_functional_tensor`, follow https://blog.csdn.net/lanxing147/article/details/136625264
-```
-
-
-### Commands: carefule to change the setting to training setting
+Install the dependencies package following our [installation](../docs/installation.md)'s s sections of "Evaluation Dependencies". Then, run the following evaluation command:
```bash
# metric can any one or list of: ssim, psnr, lpips, flolpips
-python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir <path/to/original/videos> --generated_video_dir <path/to/generated/videos> --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips
+python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir path/to/original/videos --generated_video_dir path/to/generated/videos --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips
```
diff --git a/eval/human_eval/launch.sh b/eval/human_eval/launch.sh
index 5b25abd..c5ec0a2 100644
--- a/eval/human_eval/launch.sh
+++ b/eval/human_eval/launch.sh
@@ -16,16 +16,12 @@ fi
LOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}
echo "Logging to $LOG_BASE"
-# == sample & human evaluation ==
-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT 1 $MODEL_NAME -1 >${LOG_BASE}_1.log 2>&1 &
-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2a >${LOG_BASE}_2a.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2b >${LOG_BASE}_2b.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2c >${LOG_BASE}_2c.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2d >${LOG_BASE}_2d.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2e >${LOG_BASE}_2e.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2f >${LOG_BASE}_2f.log 2>&1 &
-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2g >${LOG_BASE}_2g.log 2>&1 &
+GPUS=(0 1 2 3 4 5 6 7)
+TASK_ID_LIST=(1 2a 2b 2c 2d 2e 2f 2g) # 2h not supported yet
+FRAME_LIST=(1 $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES)
-# CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2h >${LOG_BASE}_2h.log 2>&1 &
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT ${FRAME_LIST[i]} $MODEL_NAME -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &
+done
# kill all by: pkill -f "inference"
diff --git a/eval/loss/launch.sh b/eval/loss/launch.sh
index ac55f1d..f29e832 100644
--- a/eval/loss/launch.sh
+++ b/eval/loss/launch.sh
@@ -5,6 +5,7 @@ set -e
CMD="torchrun --standalone --nproc_per_node 1 eval/loss/eval_loss.py configs/opensora-v1-2/misc/eval_loss.py"
CKPT_PATH=$1
+MODEL_NAME=$2
IMG_PATH="/mnt/nfs-207/sora_data/meta/img_1k.csv"
VID_PATH="/mnt/nfs-207/sora_data/meta/vid_100.csv"
@@ -14,13 +15,16 @@ if [[ $CKPT_PATH == *"ema"* ]]; then
else
CKPT_BASE=$(basename $CKPT_PATH)
fi
-LOG_BASE=logs/loss/${CKPT_BASE}
+LOG_BASE=logs/loss/${MODEL_NAME}_${CKPT_BASE}
mkdir -p logs/loss
echo "Logging to $LOG_BASE"
+
+GPUS=(1 2 3 4 5)
+RESOLUTION=(144p 240p 360p 480p 720p)
+
CUDA_VISIBLE_DEVICES=0 $CMD --data-path $IMG_PATH --ckpt-path $CKPT_PATH >${LOG_BASE}_img.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 144p >${LOG_BASE}_144p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 240p >${LOG_BASE}_240p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 360p >${LOG_BASE}_360p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 480p >${LOG_BASE}_480p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 720p >${LOG_BASE}_720p_vid.log 2>&1 &
+
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution ${RESOLUTION[i]} >${LOG_BASE}_${RESOLUTION[i]}_vid.log 2>&1 &
+done
diff --git a/eval/sample.sh b/eval/sample.sh
index dc29e4b..5d94d23 100644
--- a/eval/sample.sh
+++ b/eval/sample.sh
@@ -7,9 +7,21 @@ CKPT=$1
NUM_FRAMES=$2
MODEL_NAME=$3
-let DOUBLE_FRAMES=$2*2
-let QUAD_FRAMES=$2*4
-let OCT_FRAMES=$2*8
+echo "NUM_FRAMES=${NUM_FRAMES}"
+
+if [ -z "${NUM_FRAMES}" ]; then
+ echo "you need to pass NUM_FRAMES"
+else
+ let DOUBLE_FRAMES=$2*2
+ let QUAD_FRAMES=$2*4
+ let OCT_FRAMES=$2*8
+fi
+
+echo "DOUBLE_FRAMES=${DOUBLE_FRAMES}"
+echo "QUAD_FRAMES=${QUAD_FRAMES}"
+echo "OCT_FRAMES=${OCT_FRAMES}"
+
+
CMD="python scripts/inference.py configs/opensora-v1-2/inference/sample.py"
CMD_REF="python scripts/inference-long.py configs/opensora-v1-2/inference/sample.py"
diff --git a/eval/vbench/launch.sh b/eval/vbench/launch.sh
index ac25c01..acb99aa 100644
--- a/eval/vbench/launch.sh
+++ b/eval/vbench/launch.sh
@@ -4,8 +4,7 @@ set -x
set -e
CKPT=$1
-NUM_FRAMES=$2
-MODEL_NAME=$3
+MODEL_NAME=$2
if [[ $CKPT == *"ema"* ]]; then
parentdir=$(dirname $CKPT)
@@ -16,11 +15,9 @@ fi
LOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}
echo "Logging to $LOG_BASE"
-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT -4a >${LOG_BASE}_4a.log 2>&1 &
-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT -4b >${LOG_BASE}_4b.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT -4c >${LOG_BASE}_4c.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT -4d >${LOG_BASE}_4d.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT -4e >${LOG_BASE}_4e.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT -4f >${LOG_BASE}_4f.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT -4g >${LOG_BASE}_4g.log 2>&1 &
-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT -4h >${LOG_BASE}_4h.log 2>&1 &
+GPUS=(0 1 2 3 4 5 6 7)
+TASK_ID_LIST=(4a 4b 4c 4d 4e 4f 4g 4h)
+
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &
+done
diff --git a/eval/vbench_i2v/launch.sh b/eval/vbench_i2v/launch.sh
index 82703d6..1c07a71 100644
--- a/eval/vbench_i2v/launch.sh
+++ b/eval/vbench_i2v/launch.sh
@@ -4,8 +4,7 @@ set -x
set -e
CKPT=$1
-NUM_FRAMES=$2
-MODEL_NAME=$3
+MODEL_NAME=$2
if [[ $CKPT == *"ema"* ]]; then
parentdir=$(dirname $CKPT)
@@ -16,11 +15,9 @@ fi
LOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}
echo "Logging to $LOG_BASE"
-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT -5a >${LOG_BASE}_5a.log 2>&1 &
-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT -5b >${LOG_BASE}_5b.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT -5c >${LOG_BASE}_5c.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT -5d >${LOG_BASE}_5d.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT -5e >${LOG_BASE}_5e.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT -5f >${LOG_BASE}_5f.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT -5g >${LOG_BASE}_5g.log 2>&1 &
-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT -5h >${LOG_BASE}_5h.log 2>&1 &
+GPUS=(0 1 2 3 4 5 6 7)
+TASK_ID_LIST=(5a 5b 5c 5d 5e 5f 5g 5h)
+
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &
+done
diff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txt
index d427bea..8fabae1 100644
--- a/requirements/requirements-data.txt
+++ b/requirements/requirements-data.txt
@@ -1,4 +1,53 @@
-gdown
+gdown==5.2.0
+
+# [caption llava]
+ninja==1.11.1.1
+flash-attn==2.5.7
+transformers==4.36.2
+tokenizers==0.15.2
+sentencepiece==0.2.0
+shortuuid==1.0.13
+accelerate==0.29.2
+peft==0.4.0
+bitsandbytes==0.41.0
+pydantic==2.7.1
+markdown2[all]
+numpy==1.26.3
+scikit-learn==1.4.2
+gradio==4.31.3
+gradio_client==0.16.3
+requests==2.31.0
+httpx==0.27.0
+uvicorn==0.29.0
+fastapi==0.110.0
+einops==0.8.0
+einops-exts==0.0.4
+timm==0.9.16
+
+# [camera_motion]
+decord==0.6.0
+ptvsd==4.3.2
+imageio-ffmpeg==0.4.9
+
+# [datasets]
+pandarallel==1.6.5
+opencv-python==4.9.0.80
+ffmpeg-python==0.2.0
+lingua-language-detector==2.0.2
+imageio-ffmpeg==0.4.9
+
+# [frame interpolation]
+imageio==2.34.1
+
+# [scene cut]
+imageio-ffmpeg==0.4.9
# [aesthetic]
clip @ git+https://github.com/openai/CLIP.git
+decord==0.6.0
+
+# [ocr]
+mmcv==2.1.0
+mmdet==3.1.0
+mmocr==1.0.1
+detectron2 @ git+https://github.com/facebookresearch/detectron2.git@ff53992
diff --git a/requirements/requirements-eval.txt b/requirements/requirements-eval.txt
index e69de29..4a1293c 100644
--- a/requirements/requirements-eval.txt
+++ b/requirements/requirements-eval.txt
@@ -0,0 +1,14 @@
+# [vbench]
+detectron2 @ git+https://github.com/facebookresearch/detectron2.git@ff53992
+
+
+
+# [vae]
+decord==0.6.0
+pytorchvideo==0.1.5
+lpips==0.1.4
+scipy==1.13.0
+
+# [human eval]
+requests==2.31.0
+urllib3==2.2.1
diff --git a/requirements/requirements-pllava.txt b/requirements/requirements-pllava.txt
new file mode 100644
index 0000000..238dcc4
--- /dev/null
+++ b/requirements/requirements-pllava.txt
@@ -0,0 +1,248 @@
+absl-py==2.1.0
+accelerate==0.26.1
+addict==2.4.0
+aiofiles==23.2.1
+aliyun-python-sdk-core==2.15.0
+aliyun-python-sdk-kms==2.16.2
+altair==5.2.0
+annotated-types==0.6.0
+antlr4-python3-runtime==4.9.3
+anyio==4.3.0
+anykeystore==0.2
+apex==0.9.10.dev0
+appdirs==1.4.4
+argcomplete==3.2.3
+attrs==23.2.0
+av==10.0.0
+beautifulsoup4==4.12.3
+blessed==1.20.0
+blessings==1.7
+boto3==1.34.63
+botocore==1.34.63
+Brotli==1.1.0
+cachetools==5.3.3
+certifi==2024.2.2
+cffi==1.16.0
+charset-normalizer==3.3.2
+click==8.1.7
+colorama==0.4.6
+contourpy==1.2.0
+crcmod==1.7
+cryptacular==1.6.2
+cryptography==42.0.5
+cycler==0.12.1
+dacite==1.7.0
+decorator==4.4.2
+decord==0.6.0
+deepspeed==0.14.0
+defusedxml==0.7.1
+Deprecated==1.2.14
+dill==0.3.8
+distro==1.9.0
+dnspython==2.6.1
+docker-pycreds==0.4.0
+einops==0.6.1
+exceptiongroup==1.2.0
+fastapi==0.110.0
+ffmpeg==1.4
+ffmpy==0.3.2
+fiftyone==0.23.6
+fiftyone-brain==0.16.1
+fiftyone_db==1.1.2
+filelock==3.9.0
+fonttools==4.49.0
+fsspec==2024.2.0
+ftfy==6.1.3
+future==1.0.0
+fvcore==0.1.5.post20221221
+gdown==5.1.0
+gitdb==4.0.11
+GitPython==3.1.42
+glob2==0.7
+google-auth==2.28.2
+google-auth-oauthlib==1.2.0
+gpustat==1.1.1
+gradio==4.21.0
+gradio_client==0.12.0
+graphql-core==3.2.3
+greenlet==3.0.3
+grpcio==1.62.1
+h11==0.14.0
+h2==4.1.0
+hjson==3.1.0
+hpack==4.0.0
+httpcore==1.0.4
+httpx==0.27.0
+huggingface-hub==0.21.4
+humanize==4.9.0
+hupper==1.12.1
+Hypercorn==0.16.0
+hyperframe==6.0.1
+idna==3.6
+idscheck==2.3.0
+imageio==2.27.0
+imageio-ffmpeg==0.4.9
+importlib_metadata==7.0.2
+importlib_resources==6.3.0
+inflate64==1.0.0
+iopath==0.1.10
+Jinja2==3.1.2
+jmespath==0.10.0
+joblib==1.3.2
+jsonlines==4.0.0
+jsonschema==4.21.1
+jsonschema-specifications==2023.12.1
+kaleido==0.2.1
+kiwisolver==1.4.5
+lazy_loader==0.3
+Markdown==3.6
+markdown-it-py==3.0.0
+MarkupSafe==2.1.3
+matplotlib==3.8.3
+mdurl==0.1.2
+mmcv-full==1.7.2
+model-index==0.1.11
+mongoengine==0.24.2
+motor==3.3.2
+moviepy==1.0.3
+mpmath==1.3.0
+multivolumefile==0.2.3
+networkx==3.2.1
+ninja==1.11.1.1
+numpy==1.23.5
+nvidia-ml-py==12.535.133
+nvidia-ml-py3==7.352.0
+oauthlib==3.2.2
+omegaconf==2.3.0
+openai==1.14.0
+opencv-python==4.9.0.80
+opencv-python-headless==4.9.0.80
+opendatalab==0.0.10
+openmim==0.3.9
+openxlab==0.0.36
+ordered-set==4.1.0
+orjson==3.9.15
+oss2==2.17.0
+packaging==24.0
+pandas==1.5.3
+PasteDeploy==3.1.0
+pathtools==0.1.2
+pbkdf2==1.3
+peft==0.10.0
+pillow==10.2.0
+plaster==1.1.2
+plaster-pastedeploy==1.0.1
+platformdirs==4.2.0
+plotly==5.20.0
+portalocker==2.8.2
+pprintpp==0.4.0
+priority==2.0.0
+proglog==0.1.10
+protobuf==4.23.4
+psutil==5.9.4
+py-cpuinfo==9.0.0
+py7zr==0.21.0
+pyasn1==0.5.1
+pyasn1-modules==0.3.0
+pybcj==1.0.2
+pycparser==2.21
+pycryptodome==3.20.0
+pycryptodomex==3.20.0
+pydantic==2.6.4
+pydantic_core==2.16.3
+pydub==0.25.1
+Pygments==2.17.2
+pymongo==4.6.2
+pynvml==11.5.0
+pyparsing==3.1.2
+pyppmd==1.1.0
+pyramid==2.0.2
+pyramid-mailer==0.15.1
+PySocks==1.7.1
+python-dateutil==2.9.0.post0
+python-multipart==0.0.9
+python3-openid==3.2.0
+pytz==2023.4
+PyYAML==6.0
+pyzstd==0.15.9
+rarfile==4.1
+referencing==0.33.0
+regex==2023.12.25
+repoze.sendmail==4.4.1
+requests==2.28.2
+requests-oauthlib==1.4.0
+retrying==1.3.4
+rich==13.4.2
+rpds-py==0.18.0
+rsa==4.9
+ruff==0.3.2
+s3transfer==0.10.1
+safetensors==0.4.2
+scikit-image==0.22.0
+scikit-learn==1.4.1.post1
+scipy==1.10.1
+semantic-version==2.10.0
+sentencepiece==0.2.0
+sentry-sdk==1.42.0
+setproctitle==1.3.3
+shellingham==1.5.4
+six==1.16.0
+smmap==5.0.1
+sniffio==1.3.1
+sortedcontainers==2.4.0
+soupsieve==2.5
+SQLAlchemy==2.0.28
+sse-starlette==0.10.3
+sseclient-py==1.8.0
+starlette==0.36.3
+strawberry-graphql==0.138.1
+sympy==1.12
+tabulate==0.9.0
+taskgroup==0.0.0a4
+tenacity==8.2.3
+tensorboard==2.15.1
+tensorboard-data-server==0.7.2
+tensorboardX==2.6.2.2
+termcolor==2.3.0
+texttable==1.7.0
+threadpoolctl==3.3.0
+tifffile==2024.2.12
+timm==0.6.12
+tokenizers==0.15.2
+tomli==2.0.1
+tomlkit==0.12.0
+toolz==0.12.1
+torch==2.2.2
+torchaudio
+torchvision==0.17.2
+tqdm==4.65.2
+transaction==4.0
+transformers==4.37.1
+translationstring==1.4
+triton==2.2.0
+typer==0.9.0
+typing_extensions==4.8.0
+tzdata==2024.1
+tzlocal==5.2
+universal-analytics-python3==1.1.1
+urllib3==1.26.18
+uvicorn==0.28.0
+velruse==1.1.1
+venusian==3.1.0
+voxel51-eta==0.12.6
+wandb==0.14.0
+wcwidth==0.2.13
+WebOb==1.8.7
+websockets==11.0.3
+Werkzeug==3.0.1
+wrapt==1.16.0
+wsproto==1.2.0
+WTForms==3.1.2
+wtforms-recaptcha==0.3.2
+xmltodict==0.13.0
+yacs==0.1.8
+yapf==0.40.2
+zipp==3.18.1
+zope.deprecation==5.0
+zope.interface==6.2
+zope.sqlalchemy==3.1
diff --git a/requirements/requirements.txt b/requirements/requirements.txt
index 8a7b440..65fdfa0 100644
--- a/requirements/requirements.txt
+++ b/requirements/requirements.txt
@@ -9,18 +9,18 @@ accelerate==0.29.2 # for t5
av>=12.0.0
# [gradio]
-gradio
-spaces
+gradio>=4.31.3
+spaces>=0.28.3
# [notebook]
-ipykernel
-ipywidgets
+ipykernel>=6.29.4
+ipywidgets>=8.1.2
# [training]
-wandb
-tensorboard
-pandarallel
-pyarrow # for parquet
+wandb>=0.17.0
+tensorboard>=2.16.2
+pandarallel>=1.6.5
+pyarrow>=16.1.0 # for parquet
# [dev]
-pre-commit
+pre-commit>=3.7.1
diff --git a/tools/caption/README.md b/tools/caption/README.md
index 9c78518..fbaf3c9 100644
--- a/tools/caption/README.md
+++ b/tools/caption/README.md
@@ -8,7 +8,11 @@ Human labeling of videos is expensive and time-consuming. We adopt powerful imag
We extract three frames from the video for captioning. With batch inference, we can achieve 10 times speedup. With approximately 720p resolution and 1 frames, the speed is 2~3 videos/s on 8 GPUs. If we resize the smaller side to 336, the speed can be 8 videos/s. In Open-Sora v1.1, to lower the cost, we use the 7B model.
-### Requirement
+### Installation
+
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "LLaVA Captioning" sections.
+
+<!-- ### Requirement
```bash
# create conda env
@@ -34,7 +38,7 @@ pip install -e .
pip install flash-attn --no-build-isolation
# install colossalai and decord
pip install colossalai decord
-```
+``` -->
### Usage
@@ -82,14 +86,21 @@ Then use the output csv file to resume the process.
## PLLaVA Captioning
-### Download the PLLaVA repo
+
+### Installation
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "PLLaVA Captioning" sections.
+
+
+<!-- ### Download the PLLaVA repo
First, make sure you are under the directory of tools/caption/pllava_dir. Then,
```bash
git clone https://github.com/magic-research/PLLaVA.git
-
cd PLLaVA
+git checkout fd9194a
+
+
```
### Environment
@@ -107,8 +118,8 @@ pip install -r requirements.txt # change to your own torch version if neccessary
### Download weights
```bash
-python python_scripts/hf.py # download the weights
-```
+python python_scripts/hf.py # download the weights
+``` -->
### Usage
Since PLLaVA is not fashioned as a package, we will use PYTHONPATH to use it as a package.
@@ -144,8 +155,8 @@ The cost is approximately $0.01 per video (3 frames per video).
## Camera Motion Detection
-Install additional required packages: `tools/caption/camera_motion/requirements.txt`.
-
+<!-- Install additional required packages: `tools/caption/camera_motion/requirements.txt`. -->
+Install required packages with `pip install -v .[data]` (See [installation.md](../../docs/installation.md)).
Run the following command to classify camera motion:
```bash
@@ -159,7 +170,6 @@ You may additionally specify `threshold` to indicate how "sensitive" the detecti
python -m tools.caption.camera_motion.detect tools/caption/camera_motion/meta.csv --threshold 0.2
```
-
Each video is classified according to 8 categories:
`pan_right,
pan_left,
@@ -168,8 +178,5 @@ Each video is classified according to 8 categories:
zoom_in,
zoom_out,
static,
- unclassified`.
+ unclassified`.
Categories of `tilt`, `pan` and `zoom` can overlap with each other.
-
-
-
diff --git a/tools/datasets/README.md b/tools/datasets/README.md
index ac14f33..73f25b7 100644
--- a/tools/datasets/README.md
+++ b/tools/datasets/README.md
@@ -79,33 +79,34 @@ Use `datautil` to manage the dataset.
### Requirement
-To accelerate processing speed, you can install [pandarallel](https://github.com/nalepae/pandarallel):
+Follow our [installation guide](../../docs/installation.md)'s "Data Dependencies" and "Datasets" section to install the required packages.
+<!-- To accelerate processing speed, you can install [pandarallel](https://github.com/nalepae/pandarallel):
```bash
pip install pandarallel
-```
+``` -->
-To get image and video information, you need to install [opencv-python](https://github.com/opencv/opencv-python):
+<!-- To get image and video information, you need to install [opencv-python](https://github.com/opencv/opencv-python): -->
-```bash
+<!-- ```bash
pip install opencv-python
# If your videos are in av1 codec instead of h264, you need to
# - install ffmpeg first
# - install via conda to support av1 codec
conda install -c conda-forge opencv
-```
+``` -->
-Or to get video information, you can install ffmpeg and ffmpeg-python:
+<!-- Or to get video information, you can install ffmpeg and ffmpeg-python:
```bash
pip install ffmpeg-python
-```
+``` -->
-To filter a specific language, you need to install [lingua](https://github.com/pemistahl/lingua-py):
+<!-- To filter a specific language, you need to install [lingua](https://github.com/pemistahl/lingua-py):
```bash
pip install lingua-language-detector
-```
+``` -->
### Basic Usage
diff --git a/tools/frame_interpolation/README.md b/tools/frame_interpolation/README.md
index 8418e66..3886018 100644
--- a/tools/frame_interpolation/README.md
+++ b/tools/frame_interpolation/README.md
@@ -6,10 +6,12 @@ Interpolation can be useful for scenery videos, but it may not be suitable for v
## Requirement
-```bash
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "Frame Interpolation" sections.
+
+<!-- ```bash
conda install -c conda-forge opencv
pip install imageio
-```
+``` -->
## Model
diff --git a/tools/frame_interpolation/interpolation.py b/tools/frame_interpolation/interpolation.py
index 0cd822b..c9d8d1d 100644
--- a/tools/frame_interpolation/interpolation.py
+++ b/tools/frame_interpolation/interpolation.py
@@ -147,7 +147,7 @@ def write(outputs, input_path, output_path, fps=30):
file_name, _ = os.path.splitext(file_name_with_extension)
save_video_path = f"{output_path}/fps{fps}_{file_name}.mp4"
- fourcc = cv2.VideoWriter_fourcc(*"avc1")
+ fourcc = cv2.VideoWriter_fourcc(*"mp4v")
writer = cv2.VideoWriter(save_video_path, fourcc, fps, size)
for i, imgt_pred in enumerate(outputs):
diff --git a/tools/scene_cut/README.md b/tools/scene_cut/README.md
index 454bdac..1c1bae5 100644
--- a/tools/scene_cut/README.md
+++ b/tools/scene_cut/README.md
@@ -5,7 +5,7 @@
- [Scene Detection](#scene-detection)
- [Video Splitting](#video-splitting)
-In many cases, raw videos contain several scenes and are too long for training. Thus, it is essential to split them into shorter
+In many cases, raw videos contain several scenes and are too long for training. Thus, it is essential to split them into shorter
clips based on scenes. Here, we provide code for scene detection and video splitting.
## Prepare Meta Files
@@ -24,24 +24,27 @@ The following command will add a new column `path` to the meta file.
python tools/scene_cut/convert_id_to_path.py /path/to/meta.csv --folder_path /path/to/video/folder
```
This should output
-- `{prefix}_path-filtered.csv` with column `path` (broken videos filtered)
+- `{prefix}_path-filtered.csv` with column `path` (broken videos filtered)
- `{prefix}_path_intact.csv` with column `path` and `intact` (`intact` indicating a video is intact or not)
## Scene Detection
-The next step is to detect scenes in a video.
-We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job.
+
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "Scene Detection" sections.
+
+<!-- The next step is to detect scenes in a video.
+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job.
```bash
pip install scenedetect[opencv] --upgrade
-```
+``` -->
**Make sure** the input meta file has column `path`, which is the path of a video.
```bash
python tools/scene_cut/scene_detect.py /path/to/meta.csv
```
-The output is `{prefix}_timestamp.csv` with column `timestamp`. Each cell in column `timestamp` is a list of tuples,
-with each tuple indicating the start and end timestamp of a scene
+The output is `{prefix}_timestamp.csv` with column `timestamp`. Each cell in column `timestamp` is a list of tuples,
+with each tuple indicating the start and end timestamp of a scene
(e.g., `[('00:00:01.234', '00:00:02.345'), ('00:00:03.456', '00:00:04.567')]`).
## Video Splitting
diff --git a/tools/scoring/README.md b/tools/scoring/README.md
index a944d5c..40949c7 100644
--- a/tools/scoring/README.md
+++ b/tools/scoring/README.md
@@ -13,10 +13,18 @@ To evaluate the aesthetic quality of videos, we use the scoring model from [CLIP
The aesthetic score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for high aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.
-For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.
+For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.
The throughput of our code is ~1K videos/s on a single H800 GPU. It also supports running on multiple GPUs for further acceleration.
-First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.
+First, install the required packages following our [installation instructions](../../docs/installation.md)'s "Data Dependencies".
+
+Next, download the scoring model to `./pretrained_models/aesthetic.pth`.
+
+```bash
+wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth
+```
+
+<!-- First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.
```bash
# pip install
pip install git+https://github.com/openai/CLIP.git
@@ -24,7 +32,7 @@ pip install decord
# get pretrained model
wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth
-```
+``` -->
Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).
```bash
@@ -37,7 +45,9 @@ This will generate multiple part files, each corresponding to a node . Run `pyth
Optical flow scores are used to assess the motion of a video. Higher optical flow scores indicate larger movement.
We use the [UniMatch](https://github.com/autonomousvision/unimatch) model for this task.
-First, download the pretrained model to `./pretrained_model/unimatch/`
+First, install the required packages following our [installation instructions](../../docs/installation.md)'s "Data Dependencies".
+
+Next, download the pretrained model to `./pretrained_model/unimatch/`
```bash
wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P ./pretrained_models/unimatch/
```
@@ -54,18 +64,23 @@ Some videos are of dense text scenes like news broadcast and advertisement, whic
We apply Optical Character Recognition (OCR) to detect texts and drop samples with dense texts. Here, we use
the [DBNet++](https://arxiv.org/abs/2202.10304) model implemented by [MMOCR](https://github.com/open-mmlab/mmocr/).
-First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
+First, install the required packages following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "OCR" section.
+
+<!-- First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
For reference, we install packages of these versions.
```
torch==2.0.1
mmcv==2.0.1
mmdet==3.1.0
mmocr==1.0.1
-```
+``` -->
Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).
-```bash
+<!-- ```bash
torchrun --standalone --nproc_per_node 8 tools/scoring/ocr/inference.py /path/to/meta.csv
+``` -->
+```bash
+torchrun --standalone --nproc_per_node 8 -m tools.scoring.ocr.inference /path/to/meta.csv
```
This should output `/path/to/meta_ocr.csv` with column `ocr`, indicating the number of text regions with detection confidence > 0.3.
@@ -73,7 +88,7 @@ This should output `/path/to/meta_ocr.csv` with column `ocr`, indicating the num
## Matching Score
Matching scores are calculated to evaluate the alignment between an image/video and its caption.
-Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.
+Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.
We simply use the cosine similarity as the matching score.
For videos, we extract the middle frame and compare it with the caption.
@@ -97,4 +112,4 @@ samples of aesthetic score < 5.0.
```
python -m tools.datasets.datautil /path/to/meta.csv --aesmin 5.0
```
-This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0
\ No newline at end of file
+This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0
diff --git a/tools/scoring/optical_flow/inference.py b/tools/scoring/optical_flow/inference.py
index 499ff06..ff5dd21 100644
--- a/tools/scoring/optical_flow/inference.py
+++ b/tools/scoring/optical_flow/inference.py
@@ -13,7 +13,7 @@ from torchvision.transforms.functional import pil_to_tensor
from tqdm import tqdm
from tools.datasets.utils import extract_frames
-from .unimatch import UniMatch
+from tools.scoring.optical_flow.unimatch import UniMatch
def merge_scores(gathered_list: list, meta: pd.DataFrame, column):
@@ -146,7 +146,7 @@ def main():
gathered_list = [None] * dist.get_world_size()
dist.all_gather_object(gathered_list, (indices_list, scores_list))
if dist.get_rank() == 0:
- meta_new = merge_scores(gathered_list, dataset.meta, column='flow')
+ meta_new = merge_scores(gathered_list, dataset.meta, column="flow")
meta_new.to_csv(out_path, index=False)
print(f"New meta with optical flow scores saved to '{out_path}'.")