Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

format

Shen-Chenhuiebc637a5d884
diff --git a/README.md b/README.md
index 43a683f..2cdeb45 100644
--- a/README.md
+++ b/README.md
@@ -161,7 +161,7 @@ git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora
# the default installation is for inference only
-pip install -v .
+pip install -v . # for development mode, `pip install -v -e .`
```
(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.
diff --git a/docs/installation.md b/docs/installation.md
index 0ba5d9b..05330fb 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -6,16 +6,27 @@ Requirements are listed in `requirements` folder.
You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions.
+For CUDA 12.1,
```bash
-# install torch (>=2.1 is recommended)
-# the command below is for CUDA 12.1, choose install commands from
-# https://pytorch.org/get-started/locally/ based on your own CUDA version
-pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121
-
-# install xformers
-# the command below is for CUDA 12.1, choose install commands from
-# https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers based on your own CUDA version
-pip install xformers==0.0.25.post1 --index-url https://download.pytorch.org/whl/cu121
+pip install -r requirements/requirements-cu121.txt
+
+# install this project
+git clone https://github.com/hpcaitech/Open-Sora
+cd Open-Sora
+
+# the default installation is for inference only
+pip install -v . # NOTE: for development mode, run `pip install -v -e .`
+
+(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.
+
+# install flash attention
+# set enable_flash_attn=False in config to disable flash attention
+pip install packaging ninja
+pip install flash-attn --no-build-isolation
+
+# install apex
+# set enable_layernorm_kernel=False in config to disable apex
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
```
## Different Dependencies
@@ -40,6 +51,14 @@ pip install -v .[data] # For development: `pip install -v -e .[eval]`
```
Next, you need to manually install the packages listed in the following sections specific to your data processing needs.
+### Datasets
+To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)
+
+However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:
+```bash
+conda install -c conda-forge opencv
+```
+
### LLaVA Captioning
You need to manually install LLaVA with the following command:
```bash
@@ -56,6 +75,19 @@ git checkout fd9194a # since there is no version tag, we use this commit
python python_scripts/hf.py # download the PLLaVA weights
```
+### Frame Interpolation
+```bash
+conda install -c conda-forge opencv
+```
+
+### Scene Detection
+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:
+```bash
+pip install scenedetect[opencv] --upgrade
+```
+
+
+
## Evaluation Dependencies
First, run the following command to install requirements:
@@ -64,6 +96,15 @@ pip install -v .[eval] # For development:`pip install -v -e .[eval]`
```
Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.
+### Human Eval
+
+You need to manually install apex from source by:
+```bash
+# set enable_layernorm_kernel=False in config to disable apex
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
+
+```
+
### VBench
You need to manually install [VBench](https://github.com/Vchitect/VBench):
```bash
diff --git a/eval/human_eval/launch.sh b/eval/human_eval/launch.sh
index 5b25abd..c5ec0a2 100644
--- a/eval/human_eval/launch.sh
+++ b/eval/human_eval/launch.sh
@@ -16,16 +16,12 @@ fi
LOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}
echo "Logging to $LOG_BASE"
-# == sample & human evaluation ==
-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT 1 $MODEL_NAME -1 >${LOG_BASE}_1.log 2>&1 &
-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2a >${LOG_BASE}_2a.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2b >${LOG_BASE}_2b.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2c >${LOG_BASE}_2c.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2d >${LOG_BASE}_2d.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2e >${LOG_BASE}_2e.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2f >${LOG_BASE}_2f.log 2>&1 &
-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2g >${LOG_BASE}_2g.log 2>&1 &
+GPUS=(0 1 2 3 4 5 6 7)
+TASK_ID_LIST=(1 2a 2b 2c 2d 2e 2f 2g) # 2h not supported yet
+FRAME_LIST=(1 $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES $NUM_FRAMES)
-# CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT $NUM_FRAMES $MODEL_NAME -2h >${LOG_BASE}_2h.log 2>&1 &
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT ${FRAME_LIST[i]} $MODEL_NAME -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &
+done
# kill all by: pkill -f "inference"
diff --git a/eval/vbench/launch.sh b/eval/vbench/launch.sh
index ac25c01..acb99aa 100644
--- a/eval/vbench/launch.sh
+++ b/eval/vbench/launch.sh
@@ -4,8 +4,7 @@ set -x
set -e
CKPT=$1
-NUM_FRAMES=$2
-MODEL_NAME=$3
+MODEL_NAME=$2
if [[ $CKPT == *"ema"* ]]; then
parentdir=$(dirname $CKPT)
@@ -16,11 +15,9 @@ fi
LOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}
echo "Logging to $LOG_BASE"
-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT -4a >${LOG_BASE}_4a.log 2>&1 &
-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT -4b >${LOG_BASE}_4b.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT -4c >${LOG_BASE}_4c.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT -4d >${LOG_BASE}_4d.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT -4e >${LOG_BASE}_4e.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT -4f >${LOG_BASE}_4f.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT -4g >${LOG_BASE}_4g.log 2>&1 &
-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT -4h >${LOG_BASE}_4h.log 2>&1 &
+GPUS=(0 1 2 3 4 5 6 7)
+TASK_ID_LIST=(4a 4b 4c 4d 4e 4f 4g 4h)
+
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &
+done
diff --git a/eval/vbench_i2v/launch.sh b/eval/vbench_i2v/launch.sh
index 82703d6..1c07a71 100644
--- a/eval/vbench_i2v/launch.sh
+++ b/eval/vbench_i2v/launch.sh
@@ -4,8 +4,7 @@ set -x
set -e
CKPT=$1
-NUM_FRAMES=$2
-MODEL_NAME=$3
+MODEL_NAME=$2
if [[ $CKPT == *"ema"* ]]; then
parentdir=$(dirname $CKPT)
@@ -16,11 +15,9 @@ fi
LOG_BASE=logs/sample/${MODEL_NAME}_${CKPT_BASE}
echo "Logging to $LOG_BASE"
-CUDA_VISIBLE_DEVICES=0 bash eval/sample.sh $CKPT -5a >${LOG_BASE}_5a.log 2>&1 &
-CUDA_VISIBLE_DEVICES=1 bash eval/sample.sh $CKPT -5b >${LOG_BASE}_5b.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 bash eval/sample.sh $CKPT -5c >${LOG_BASE}_5c.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 bash eval/sample.sh $CKPT -5d >${LOG_BASE}_5d.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 bash eval/sample.sh $CKPT -5e >${LOG_BASE}_5e.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 bash eval/sample.sh $CKPT -5f >${LOG_BASE}_5f.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 bash eval/sample.sh $CKPT -5g >${LOG_BASE}_5g.log 2>&1 &
-CUDA_VISIBLE_DEVICES=7 bash eval/sample.sh $CKPT -5h >${LOG_BASE}_5h.log 2>&1 &
+GPUS=(0 1 2 3 4 5 6 7)
+TASK_ID_LIST=(5a 5b 5c 5d 5e 5f 5g 5h)
+
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} bash eval/sample.sh $CKPT -${TASK_ID_LIST[i]} >${LOG_BASE}_${TASK_ID_LIST[i]}.log 2>&1 &
+done
diff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txt
index 8ebf2ed..8d060d7 100644
--- a/requirements/requirements-data.txt
+++ b/requirements/requirements-data.txt
@@ -1,8 +1,5 @@
gdown==5.2.0
-# [aesthetic]
-clip @ git+https://github.com/openai/CLIP.git
-
# [caption llava]
ninja==1.11.1.1
flash-attn==2.5.7
@@ -32,5 +29,21 @@ decord==0.6.0
ptvsd==4.3.2
imageio-ffmpeg==0.4.9
+# [datasets]
+pandarallel==1.6.5
+opencv-python==4.9.0.80
+ffmpeg-python==0.2.0
+lingua-language-detector==2.0.2
+imageio-ffmpeg==0.4.9
+
+# [frame interpolation]
+imageio==2.34.1
+
+# [scene detection]
+
# [scene cut]
imageio-ffmpeg==0.4.9
+
+# [aesthetic]
+clip @ git+https://github.com/openai/CLIP.git
+decord==0.6.0
diff --git a/requirements/requirements-eval.txt b/requirements/requirements-eval.txt
index 33c693f..be9bff5 100644
--- a/requirements/requirements-eval.txt
+++ b/requirements/requirements-eval.txt
@@ -7,3 +7,7 @@ decord==0.6.0
pytorchvideo==0.1.5
lpips==0.1.4
scipy==1.13.0
+
+# [human eval]
+requests==2.31.0
+urllib3==2.2.1
diff --git a/tools/caption/README.md b/tools/caption/README.md
index cbd4ca8..fbaf3c9 100644
--- a/tools/caption/README.md
+++ b/tools/caption/README.md
@@ -170,7 +170,6 @@ You may additionally specify `threshold` to indicate how "sensitive" the detecti
python -m tools.caption.camera_motion.detect tools/caption/camera_motion/meta.csv --threshold 0.2
```
-
Each video is classified according to 8 categories:
`pan_right,
pan_left,
diff --git a/tools/datasets/README.md b/tools/datasets/README.md
index ac14f33..73f25b7 100644
--- a/tools/datasets/README.md
+++ b/tools/datasets/README.md
@@ -79,33 +79,34 @@ Use `datautil` to manage the dataset.
### Requirement
-To accelerate processing speed, you can install [pandarallel](https://github.com/nalepae/pandarallel):
+Follow our [installation guide](../../docs/installation.md)'s "Data Dependencies" and "Datasets" section to install the required packages.
+<!-- To accelerate processing speed, you can install [pandarallel](https://github.com/nalepae/pandarallel):
```bash
pip install pandarallel
-```
+``` -->
-To get image and video information, you need to install [opencv-python](https://github.com/opencv/opencv-python):
+<!-- To get image and video information, you need to install [opencv-python](https://github.com/opencv/opencv-python): -->
-```bash
+<!-- ```bash
pip install opencv-python
# If your videos are in av1 codec instead of h264, you need to
# - install ffmpeg first
# - install via conda to support av1 codec
conda install -c conda-forge opencv
-```
+``` -->
-Or to get video information, you can install ffmpeg and ffmpeg-python:
+<!-- Or to get video information, you can install ffmpeg and ffmpeg-python:
```bash
pip install ffmpeg-python
-```
+``` -->
-To filter a specific language, you need to install [lingua](https://github.com/pemistahl/lingua-py):
+<!-- To filter a specific language, you need to install [lingua](https://github.com/pemistahl/lingua-py):
```bash
pip install lingua-language-detector
-```
+``` -->
### Basic Usage
diff --git a/tools/frame_interpolation/README.md b/tools/frame_interpolation/README.md
index 8418e66..3886018 100644
--- a/tools/frame_interpolation/README.md
+++ b/tools/frame_interpolation/README.md
@@ -6,10 +6,12 @@ Interpolation can be useful for scenery videos, but it may not be suitable for v
## Requirement
-```bash
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "Frame Interpolation" sections.
+
+<!-- ```bash
conda install -c conda-forge opencv
pip install imageio
-```
+``` -->
## Model
diff --git a/tools/scene_cut/README.md b/tools/scene_cut/README.md
index 454bdac..1c1bae5 100644
--- a/tools/scene_cut/README.md
+++ b/tools/scene_cut/README.md
@@ -5,7 +5,7 @@
- [Scene Detection](#scene-detection)
- [Video Splitting](#video-splitting)
-In many cases, raw videos contain several scenes and are too long for training. Thus, it is essential to split them into shorter
+In many cases, raw videos contain several scenes and are too long for training. Thus, it is essential to split them into shorter
clips based on scenes. Here, we provide code for scene detection and video splitting.
## Prepare Meta Files
@@ -24,24 +24,27 @@ The following command will add a new column `path` to the meta file.
python tools/scene_cut/convert_id_to_path.py /path/to/meta.csv --folder_path /path/to/video/folder
```
This should output
-- `{prefix}_path-filtered.csv` with column `path` (broken videos filtered)
+- `{prefix}_path-filtered.csv` with column `path` (broken videos filtered)
- `{prefix}_path_intact.csv` with column `path` and `intact` (`intact` indicating a video is intact or not)
## Scene Detection
-The next step is to detect scenes in a video.
-We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job.
+
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "Scene Detection" sections.
+
+<!-- The next step is to detect scenes in a video.
+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job.
```bash
pip install scenedetect[opencv] --upgrade
-```
+``` -->
**Make sure** the input meta file has column `path`, which is the path of a video.
```bash
python tools/scene_cut/scene_detect.py /path/to/meta.csv
```
-The output is `{prefix}_timestamp.csv` with column `timestamp`. Each cell in column `timestamp` is a list of tuples,
-with each tuple indicating the start and end timestamp of a scene
+The output is `{prefix}_timestamp.csv` with column `timestamp`. Each cell in column `timestamp` is a list of tuples,
+with each tuple indicating the start and end timestamp of a scene
(e.g., `[('00:00:01.234', '00:00:02.345'), ('00:00:03.456', '00:00:04.567')]`).
## Video Splitting
diff --git a/tools/scoring/README.md b/tools/scoring/README.md
index a944d5c..7c5ea2e 100644
--- a/tools/scoring/README.md
+++ b/tools/scoring/README.md
@@ -13,10 +13,18 @@ To evaluate the aesthetic quality of videos, we use the scoring model from [CLIP
The aesthetic score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for high aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.
-For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.
+For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.
The throughput of our code is ~1K videos/s on a single H800 GPU. It also supports running on multiple GPUs for further acceleration.
-First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.
+First, install the required packages following our [installation instructions](../../docs/installation.md)'s "Data Dependencies".
+
+Next, download the scoring model to `./pretrained_models/aesthetic.pth`.
+
+```bash
+wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth
+```
+
+<!-- First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.
```bash
# pip install
pip install git+https://github.com/openai/CLIP.git
@@ -24,7 +32,7 @@ pip install decord
# get pretrained model
wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth
-```
+``` -->
Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).
```bash
@@ -54,7 +62,7 @@ Some videos are of dense text scenes like news broadcast and advertisement, whic
We apply Optical Character Recognition (OCR) to detect texts and drop samples with dense texts. Here, we use
the [DBNet++](https://arxiv.org/abs/2202.10304) model implemented by [MMOCR](https://github.com/open-mmlab/mmocr/).
-First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
+First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
For reference, we install packages of these versions.
```
torch==2.0.1
@@ -73,7 +81,7 @@ This should output `/path/to/meta_ocr.csv` with column `ocr`, indicating the num
## Matching Score
Matching scores are calculated to evaluate the alignment between an image/video and its caption.
-Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.
+Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.
We simply use the cosine similarity as the matching score.
For videos, we extract the middle frame and compare it with the caption.
@@ -97,4 +105,4 @@ samples of aesthetic score < 5.0.
```
python -m tools.datasets.datautil /path/to/meta.csv --aesmin 5.0
```
-This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0
\ No newline at end of file
+This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0