Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

format

Shen-Chenhuif96035d4808e
diff --git a/.gitignore b/.gitignore
index cd88597..a534719 100644
--- a/.gitignore
+++ b/.gitignore
@@ -128,6 +128,7 @@ ENV/
env.bak/
venv.bak/
+
# Spyder project settings
.spyderproject
.spyproject
@@ -191,3 +192,6 @@ eval/vae/flolpips/weights/
node_modules/
package-lock.json
package.json
+
+# PLLaVA
+tools/caption/pllava_dir/PLLaVA
diff --git a/README.md b/README.md
index 3ca35ef..43a683f 100644
--- a/README.md
+++ b/README.md
@@ -154,7 +154,7 @@ conda create -n opensora python=3.10
conda activate opensora
# install torch, torchvision and xformers
-pip install -r requirements/requirements_cu121.txt
+pip install -r requirements/requirements-cu121.txt
# install this project
git clone https://github.com/hpcaitech/Open-Sora
diff --git a/docs/installation.md b/docs/installation.md
index 6229105..0ba5d9b 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -15,7 +15,7 @@ pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorc
# install xformers
# the command below is for CUDA 12.1, choose install commands from
# https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers based on your own CUDA version
-pip install xformers --index-url https://download.pytorch.org/whl/cu121
+pip install xformers==0.0.25.post1 --index-url https://download.pytorch.org/whl/cu121
```
## Different Dependencies
@@ -25,16 +25,46 @@ The default installation is for inference only. Other optional dependencies are
```bash
pip install -v .[data] # for data preprocessing
pip install -v .[eval] # for evaluation, need to manually install some packages detailed below
+
+# Note: if in development mode, use the following commands instead:
+pip install -v -e .[data]
+pip install -v -e .[eval]
+```
+
+
+## Data Dependencies
+
+First, run the following command to install requirements:
+```bash
+pip install -v .[data] # For development: `pip install -v -e .[eval]`
```
+Next, you need to manually install the packages listed in the following sections specific to your data processing needs.
+### LLaVA Captioning
+You need to manually install LLaVA with the following command:
+```bash
+pip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]
+```
+### PLLaVA Captioning
+
+You need to manually install PLLaVa with the following commands:
+```bash
+cd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directory
+git clone https://github.com/magic-research/PLLaVA.git
+cd PLLaVA
+git checkout fd9194a # since there is no version tag, we use this commit
+python python_scripts/hf.py # download the PLLaVA weights
+```
## Evaluation Dependencies
+First, run the following command to install requirements:
```bash
-pip install -v .[eval]
+pip install -v .[eval] # For development:`pip install -v -e .[eval]`
```
+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.
-#### VBench
+### VBench
You need to manually install [VBench](https://github.com/Vchitect/VBench):
```bash
pip install --no-deps vbench==0.1.1
@@ -43,7 +73,7 @@ export PATH="/path/to/vbench:$PATH"
```
-#### VAE
+### VAE
You need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).
* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`
diff --git a/eval/loss/launch.sh b/eval/loss/launch.sh
index ac55f1d..de603b2 100644
--- a/eval/loss/launch.sh
+++ b/eval/loss/launch.sh
@@ -18,9 +18,12 @@ LOG_BASE=logs/loss/${CKPT_BASE}
mkdir -p logs/loss
echo "Logging to $LOG_BASE"
+
+GPUS=(1 2 3 4 5)
+RESOLUTION=(144p 240p 360p 480p 720p)
+
CUDA_VISIBLE_DEVICES=0 $CMD --data-path $IMG_PATH --ckpt-path $CKPT_PATH >${LOG_BASE}_img.log 2>&1 &
-CUDA_VISIBLE_DEVICES=2 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 144p >${LOG_BASE}_144p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=3 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 240p >${LOG_BASE}_240p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=4 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 360p >${LOG_BASE}_360p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=5 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 480p >${LOG_BASE}_480p_vid.log 2>&1 &
-CUDA_VISIBLE_DEVICES=6 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 720p >${LOG_BASE}_720p_vid.log 2>&1 &
+
+for i in "${!GPUS[@]}"; do
+ CUDA_VISIBLE_DEVICES=${GPUS[i]} $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution ${RESOLUTION[i]} >${LOG_BASE}_${RESOLUTION[i]}_vid.log 2>&1 &
+done
diff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txt
index d427bea..4463f9e 100644
--- a/requirements/requirements-data.txt
+++ b/requirements/requirements-data.txt
@@ -1,4 +1,31 @@
-gdown
+gdown==5.2.0
# [aesthetic]
clip @ git+https://github.com/openai/CLIP.git
+
+# [caption llava]
+ninja==1.11.1.1
+flash-attn==2.5.7
+transformers==4.36.2
+tokenizers==0.15.2
+sentencepiece==0.2.0
+shortuuid==1.0.13
+accelerate==0.29.2
+peft==0.4.0
+bitsandbytes==0.41.0
+pydantic==2.7.1
+markdown2[all]
+numpy==1.26.3
+scikit-learn==1.4.2
+gradio==4.31.3
+gradio_client==0.16.3
+requests==2.31.0
+httpx==0.27.0
+uvicorn==0.29.0
+fastapi==0.110.0
+einops==0.8.0
+einops-exts==0.0.4
+timm==0.9.16
+
+# [scene cut]
+imageio-ffmpeg==0.4.9
diff --git a/tools/caption/README.md b/tools/caption/README.md
index 9c78518..8d4a567 100644
--- a/tools/caption/README.md
+++ b/tools/caption/README.md
@@ -8,7 +8,11 @@ Human labeling of videos is expensive and time-consuming. We adopt powerful imag
We extract three frames from the video for captioning. With batch inference, we can achieve 10 times speedup. With approximately 720p resolution and 1 frames, the speed is 2~3 videos/s on 8 GPUs. If we resize the smaller side to 336, the speed can be 8 videos/s. In Open-Sora v1.1, to lower the cost, we use the 7B model.
-### Requirement
+### Installation
+
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "LLaVA Captioning" sections.
+
+<!-- ### Requirement
```bash
# create conda env
@@ -34,7 +38,7 @@ pip install -e .
pip install flash-attn --no-build-isolation
# install colossalai and decord
pip install colossalai decord
-```
+``` -->
### Usage
@@ -82,14 +86,21 @@ Then use the output csv file to resume the process.
## PLLaVA Captioning
-### Download the PLLaVA repo
+
+### Installation
+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "PLLaVA Captioning" sections.
+
+
+<!-- ### Download the PLLaVA repo
First, make sure you are under the directory of tools/caption/pllava_dir. Then,
```bash
git clone https://github.com/magic-research/PLLaVA.git
-
cd PLLaVA
+git checkout fd9194a
+
+
```
### Environment
@@ -107,8 +118,8 @@ pip install -r requirements.txt # change to your own torch version if neccessary
### Download weights
```bash
-python python_scripts/hf.py # download the weights
-```
+python python_scripts/hf.py # download the weights
+``` -->
### Usage
Since PLLaVA is not fashioned as a package, we will use PYTHONPATH to use it as a package.
@@ -168,8 +179,5 @@ Each video is classified according to 8 categories:
zoom_in,
zoom_out,
static,
- unclassified`.
+ unclassified`.
Categories of `tilt`, `pan` and `zoom` can overlap with each other.
-
-
-