Open-Sora: Democratizing Efficient Video Production for All
Python
format
Shen-Chenhuiebc637a5d884
diff --git a/README.md b/README.mdindex 43a683f..2cdeb45 100644--- a/README.md+++ b/README.md@@ -161,7 +161,7 @@ git clone https://github.com/hpcaitech/Open-Soracd Open-Sora# the default installation is for inference only-pip install -v .+pip install -v . # for development mode, `pip install -v -e .````(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.diff --git a/docs/installation.md b/docs/installation.mdindex 0ba5d9b..05330fb 100644--- a/docs/installation.md+++ b/docs/installation.md@@ -6,16 +6,27 @@ Requirements are listed in `requirements` folder.You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions.+For CUDA 12.1,```bash-# install torch (>=2.1 is recommended)-# the command below is for CUDA 12.1, choose install commands from-# https://pytorch.org/get-started/locally/ based on your own CUDA version-pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121--# install xformers-# the command below is for CUDA 12.1, choose install commands from-# https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers based on your own CUDA version-pip install xformers==0.0.25.post1 --index-url https://download.pytorch.org/whl/cu121+pip install -r requirements/requirements-cu121.txt++# install this project+git clone https://github.com/hpcaitech/Open-Sora+cd Open-Sora++# the default installation is for inference only+pip install -v . # NOTE: for development mode, run `pip install -v -e .`++(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.++# install flash attention+# set enable_flash_attn=False in config to disable flash attention+pip install packaging ninja+pip install flash-attn --no-build-isolation++# install apex+# set enable_layernorm_kernel=False in config to disable apex+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git```## Different Dependencies@@ -40,6 +51,14 @@ pip install -v .[data] # For development: `pip install -v -e .[eval]````Next, you need to manually install the packages listed in the following sections specific to your data processing needs.+### Datasets+To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)++However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:+```bash+conda install -c conda-forge opencv+```+### LLaVA CaptioningYou need to manually install LLaVA with the following command:```bash@@ -56,6 +75,19 @@ git checkout fd9194a # since there is no version tag, we use this commitpython python_scripts/hf.py # download the PLLaVA weights```+### Frame Interpolation+```bash+conda install -c conda-forge opencv+```++### Scene Detection+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:+```bash+pip install scenedetect[opencv] --upgrade+```+++## Evaluation DependenciesFirst, run the following command to install requirements:@@ -64,6 +96,15 @@ pip install -v .[eval] # For development:`pip install -v -e .[eval]````Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.+### Human Eval++You need to manually install apex from source by:+```bash+# set enable_layernorm_kernel=False in config to disable apex+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git++```+### VBenchYou need to manually install [VBench](https://github.com/Vchitect/VBench):```bashdiff --git a/eval/human_eval/launch.sh b/eval/human_eval/launch.shindex 5b25abd..c5ec0a2 100644--- a/eval/human_eval/launch.sh+++ b/eval/human_eval/launch.sh@@ -16,16 +16,12 @@ fiLOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}echo "Logging to $LOG_BASE"-# == sample & human evaluation ==-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT 1 $MODEL_NAME -1 >${LOG_BASE}_1.log 2>&1 &-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2a >${LOG_BASE}_2a.log 2>&1 &-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2b >${LOG_BASE}_2b.log 2>&1 &-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2c >${LOG_BASE}_2c.log 2>&1 &-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2d >${LOG_BASE}_2d.log 2>&1 &-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2e >${LOG_BASE}_2e.log 2>&1 &-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2f >${LOG_BASE}_2f.log 2>&1 &-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2g >${LOG_BASE}_2g.log 2>&1 &+GPUS=(0 1 2 3 4 5 6 7)+TASK_ID_LIST=(1 2a 2b 2c 2d 2e 2f 2g) # 2h not supported yet+FRAME_LIST=(1 $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES)-# CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2h >${LOG_BASE}_2h.log 2>&1 &+for i in "${!GPUS[@]}"; do+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT ${FRAME_LIST[i]} $MODEL_NAME -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &+done# kill all by: pkill -f "inference"diff --git a/eval/vbench/launch.sh b/eval/vbench/launch.shindex ac25c01..acb99aa 100644--- a/eval/vbench/launch.sh+++ b/eval/vbench/launch.sh@@ -4,8 +4,7 @@ set -xset -eCKPT=$1-NUM_FRAMES=$2-MODEL_NAME=$3+MODEL_NAME=$2if [[ $CKPT == *"ema"* ]]; thenparentdir=$(dirname $CKPT)@@ -16,11 +15,9 @@ fiLOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}echo "Logging to $LOG_BASE"-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT -4a >${LOG_BASE}_4a.log 2>&1 &-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT -4b >${LOG_BASE}_4b.log 2>&1 &-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT -4c >${LOG_BASE}_4c.log 2>&1 &-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT -4d >${LOG_BASE}_4d.log 2>&1 &-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT -4e >${LOG_BASE}_4e.log 2>&1 &-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT -4f >${LOG_BASE}_4f.log 2>&1 &-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT -4g >${LOG_BASE}_4g.log 2>&1 &-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT -4h >${LOG_BASE}_4h.log 2>&1 &+GPUS=(0 1 2 3 4 5 6 7)+TASK_ID_LIST=(4a 4b 4c 4d 4e 4f 4g 4h)++for i in "${!GPUS[@]}"; do+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &+donediff --git a/eval/vbench_i2v/launch.sh b/eval/vbench_i2v/launch.shindex 82703d6..1c07a71 100644--- a/eval/vbench_i2v/launch.sh+++ b/eval/vbench_i2v/launch.sh@@ -4,8 +4,7 @@ set -xset -eCKPT=$1-NUM_FRAMES=$2-MODEL_NAME=$3+MODEL_NAME=$2if [[ $CKPT == *"ema"* ]]; thenparentdir=$(dirname $CKPT)@@ -16,11 +15,9 @@ fiLOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}echo "Logging to $LOG_BASE"-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT -5a >${LOG_BASE}_5a.log 2>&1 &-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT -5b >${LOG_BASE}_5b.log 2>&1 &-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT -5c >${LOG_BASE}_5c.log 2>&1 &-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT -5d >${LOG_BASE}_5d.log 2>&1 &-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT -5e >${LOG_BASE}_5e.log 2>&1 &-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT -5f >${LOG_BASE}_5f.log 2>&1 &-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT -5g >${LOG_BASE}_5g.log 2>&1 &-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT -5h >${LOG_BASE}_5h.log 2>&1 &+GPUS=(0 1 2 3 4 5 6 7)+TASK_ID_LIST=(5a 5b 5c 5d 5e 5f 5g 5h)++for i in "${!GPUS[@]}"; do+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &+donediff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txtindex 8ebf2ed..8d060d7 100644--- a/requirements/requirements-data.txt+++ b/requirements/requirements-data.txt@@ -1,8 +1,5 @@gdown==5.2.0-# [aesthetic]-clip @ git+https://github.com/openai/CLIP.git-# [caption llava]ninja==1.11.1.1flash-attn==2.5.7@@ -32,5 +29,21 @@ decord==0.6.0ptvsd==4.3.2imageio-ffmpeg==0.4.9+# [datasets]+pandarallel==1.6.5+opencv-python==4.9.0.80+ffmpeg-python==0.2.0+lingua-language-detector==2.0.2+imageio-ffmpeg==0.4.9++# [frame interpolation]+imageio==2.34.1++# [scene detection]+# [scene cut]imageio-ffmpeg==0.4.9++# [aesthetic]+clip @ git+https://github.com/openai/CLIP.git+decord==0.6.0diff --git a/requirements/requirements-eval.txt b/requirements/requirements-eval.txtindex 33c693f..be9bff5 100644--- a/requirements/requirements-eval.txt+++ b/requirements/requirements-eval.txt@@ -7,3 +7,7 @@ decord==0.6.0pytorchvideo==0.1.5lpips==0.1.4scipy==1.13.0++# [human eval]+requests==2.31.0+urllib3==2.2.1diff --git a/tools/caption/README.md b/tools/caption/README.mdindex cbd4ca8..fbaf3c9 100644--- a/tools/caption/README.md+++ b/tools/caption/README.md@@ -170,7 +170,6 @@ You may additionally specify `threshold` to indicate how "sensitive" the detectipython -m tools.caption.camera_motion.detect tools/caption/camera_motion/meta.csv --threshold 0.2```-Each video is classified according to 8 categories:`pan_right,pan_left,diff --git a/tools/datasets/README.md b/tools/datasets/README.mdindex ac14f33..73f25b7 100644--- a/tools/datasets/README.md+++ b/tools/datasets/README.md@@ -79,33 +79,34 @@ Use `datautil` to manage the dataset.### Requirement-To accelerate processing speed, you can install [pandarallel](https://github.com/nalepae/pandarallel):+Follow our [installation guide](../../docs/installation.md)'s "Data Dependencies" and "Datasets" section to install the required packages.+<!-- To accelerate processing speed, you can install [pandarallel](https://github.com/nalepae/pandarallel):```bashpip install pandarallel-```+``` -->-To get image and video information, you need to install [opencv-python](https://github.com/opencv/opencv-python):+<!-- To get image and video information, you need to install [opencv-python](https://github.com/opencv/opencv-python): -->-```bash+<!-- ```bashpip install opencv-python# If your videos are in av1 codec instead of h264, you need to# - install ffmpeg first# - install via conda to support av1 codecconda install -c conda-forge opencv-```+``` -->-Or to get video information, you can install ffmpeg and ffmpeg-python:+<!-- Or to get video information, you can install ffmpeg and ffmpeg-python:```bashpip install ffmpeg-python-```+``` -->-To filter a specific language, you need to install [lingua](https://github.com/pemistahl/lingua-py):+<!-- To filter a specific language, you need to install [lingua](https://github.com/pemistahl/lingua-py):```bashpip install lingua-language-detector-```+``` -->### Basic Usagediff --git a/tools/frame_interpolation/README.md b/tools/frame_interpolation/README.mdindex 8418e66..3886018 100644--- a/tools/frame_interpolation/README.md+++ b/tools/frame_interpolation/README.md@@ -6,10 +6,12 @@ Interpolation can be useful for scenery videos, but it may not be suitable for v## Requirement-```bash+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "Frame Interpolation" sections.++<!-- ```bashconda install -c conda-forge opencvpip install imageio-```+``` -->## Modeldiff --git a/tools/scene_cut/README.md b/tools/scene_cut/README.mdindex 454bdac..1c1bae5 100644--- a/tools/scene_cut/README.md+++ b/tools/scene_cut/README.md@@ -5,7 +5,7 @@- [Scene Detection](#scene-detection)- [Video Splitting](#video-splitting)-In many cases, raw videos contain several scenes and are too long for training. Thus, it is essential to split them into shorter+In many cases, raw videos contain several scenes and are too long for training. Thus, it is essential to split them into shorterclips based on scenes. Here, we provide code for scene detection and video splitting.## Prepare Meta Files@@ -24,24 +24,27 @@ The following command will add a new column `path` to the meta file.python tools/scene_cut/convert_id_to_path.py /path/to/meta.csv --folder_path /path/to/video/folder```This should output-- `{prefix}_path-filtered.csv` with column `path` (broken videos filtered)+- `{prefix}_path-filtered.csv` with column `path` (broken videos filtered)- `{prefix}_path_intact.csv` with column `path` and `intact` (`intact` indicating a video is intact or not)## Scene Detection-The next step is to detect scenes in a video.-We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job.++Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "Scene Detection" sections.++<!-- The next step is to detect scenes in a video.+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job.```bashpip install scenedetect[opencv] --upgrade-```+``` -->**Make sure** the input meta file has column `path`, which is the path of a video.```bashpython tools/scene_cut/scene_detect.py /path/to/meta.csv```-The output is `{prefix}_timestamp.csv` with column `timestamp`. Each cell in column `timestamp` is a list of tuples,-with each tuple indicating the start and end timestamp of a scene+The output is `{prefix}_timestamp.csv` with column `timestamp`. Each cell in column `timestamp` is a list of tuples,+with each tuple indicating the start and end timestamp of a scene(e.g., `[('00:00:01.234', '00:00:02.345'), ('00:00:03.456', '00:00:04.567')]`).## Video Splittingdiff --git a/tools/scoring/README.md b/tools/scoring/README.mdindex a944d5c..7c5ea2e 100644--- a/tools/scoring/README.md+++ b/tools/scoring/README.md@@ -13,10 +13,18 @@ To evaluate the aesthetic quality of videos, we use the scoring model from [CLIPThe aesthetic score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for high aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.-For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.+For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.The throughput of our code is ~1K videos/s on a single H800 GPU. It also supports running on multiple GPUs for further acceleration.-First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.+First, install the required packages following our [installation instructions](../../docs/installation.md)'s "Data Dependencies".++Next, download the scoring model to `./pretrained_models/aesthetic.pth`.++```bash+wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth+```++<!-- First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.```bash# pip installpip install git+https://github.com/openai/CLIP.git@@ -24,7 +32,7 @@ pip install decord# get pretrained modelwget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth-```+``` -->Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).```bash@@ -54,7 +62,7 @@ Some videos are of dense text scenes like news broadcast and advertisement, whicWe apply Optical Character Recognition (OCR) to detect texts and drop samples with dense texts. Here, we usethe [DBNet++](https://arxiv.org/abs/2202.10304) model implemented by [MMOCR](https://github.com/open-mmlab/mmocr/).-First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).+First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).For reference, we install packages of these versions.```torch==2.0.1@@ -73,7 +81,7 @@ This should output `/path/to/meta_ocr.csv` with column `ocr`, indicating the num## Matching ScoreMatching scores are calculated to evaluate the alignment between an image/video and its caption.-Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.+Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.We simply use the cosine similarity as the matching score.For videos, we extract the middle frame and compare it with the caption.@@ -97,4 +105,4 @@ samples of aesthetic score < 5.0.```python -m tools.datasets.datautil /path/to/meta.csv --aesmin 5.0```-This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0\ No newline at end of file+This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0