Open-Sora: Democratizing Efficient Video Production for All
Python
format
Shen-Chenhuif96035d4808e
diff --git a/.gitignore b/.gitignoreindex cd88597..a534719 100644--- a/.gitignore+++ b/.gitignore@@ -128,6 +128,7 @@ ENV/env.bak/venv.bak/+# Spyder project settings.spyderproject.spyproject@@ -191,3 +192,6 @@ eval/vae/flolpips/weights/node_modules/package-lock.jsonpackage.json++# PLLaVA+tools/caption/pllava_dir/PLLaVAdiff --git a/README.md b/README.mdindex 3ca35ef..43a683f 100644--- a/README.md+++ b/README.md@@ -154,7 +154,7 @@ conda create -n opensora python=3.10conda activate opensora# install torch, torchvision and xformers-pip install -r requirements/requirements_cu121.txt+pip install -r requirements/requirements-cu121.txt# install this projectgit clone https://github.com/hpcaitech/Open-Soradiff --git a/docs/installation.md b/docs/installation.mdindex 6229105..0ba5d9b 100644--- a/docs/installation.md+++ b/docs/installation.md@@ -15,7 +15,7 @@ pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorc# install xformers# the command below is for CUDA 12.1, choose install commands from# https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers based on your own CUDA version-pip install xformers --index-url https://download.pytorch.org/whl/cu121+pip install xformers==0.0.25.post1 --index-url https://download.pytorch.org/whl/cu121```## Different Dependencies@@ -25,16 +25,46 @@ The default installation is for inference only. Other optional dependencies are```bashpip install -v .[data] # for data preprocessingpip install -v .[eval] # for evaluation, need to manually install some packages detailed below++# Note: if in development mode, use the following commands instead:+pip install -v -e .[data]+pip install -v -e .[eval]+```+++## Data Dependencies++First, run the following command to install requirements:+```bash+pip install -v .[data] # For development: `pip install -v -e .[eval]````+Next, you need to manually install the packages listed in the following sections specific to your data processing needs.+### LLaVA Captioning+You need to manually install LLaVA with the following command:+```bash+pip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]+```+### PLLaVA Captioning++You need to manually install PLLaVa with the following commands:+```bash+cd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directory+git clone https://github.com/magic-research/PLLaVA.git+cd PLLaVA+git checkout fd9194a # since there is no version tag, we use this commit+python python_scripts/hf.py # download the PLLaVA weights+```## Evaluation Dependencies+First, run the following command to install requirements:```bash-pip install -v .[eval]+pip install -v .[eval] # For development:`pip install -v -e .[eval]````+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.-#### VBench+### VBenchYou need to manually install [VBench](https://github.com/Vchitect/VBench):```bashpip install --no-deps vbench==0.1.1@@ -43,7 +73,7 @@ export PATH="/path/to/vbench:$PATH"```-#### VAE+### VAEYou need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`diff --git a/eval/loss/launch.sh b/eval/loss/launch.shindex ac55f1d..de603b2 100644--- a/eval/loss/launch.sh+++ b/eval/loss/launch.sh@@ -18,9 +18,12 @@ LOG_BASE=logs/loss/${CKPT_BASE}mkdir -p logs/lossecho "Logging to $LOG_BASE"++GPUS=(1 2 3 4 5)+RESOLUTION=(144p 240p 360p 480p 720p)+CUDA_VISIBLE_DEVICES=0 $CMD --data-path $IMG_PATH --ckpt-path $CKPT_PATH >${LOG_BASE}_img.log 2>&1 &-CUDA_VISIBLE_DEVICES=2 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 144p >${LOG_BASE}_144p_vid.log 2>&1 &-CUDA_VISIBLE_DEVICES=3 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 240p >${LOG_BASE}_240p_vid.log 2>&1 &-CUDA_VISIBLE_DEVICES=4 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 360p >${LOG_BASE}_360p_vid.log 2>&1 &-CUDA_VISIBLE_DEVICES=5 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 480p >${LOG_BASE}_480p_vid.log 2>&1 &-CUDA_VISIBLE_DEVICES=6 $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution 720p >${LOG_BASE}_720p_vid.log 2>&1 &++for i in "${!GPUS[@]}"; do+ CUDA_VISIBLE_DEVICES=${GPUS[i]} $CMD --data-path $VID_PATH --ckpt-path $CKPT_PATH --resolution ${RESOLUTION[i]} >${LOG_BASE}_${RESOLUTION[i]}_vid.log 2>&1 &+donediff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txtindex d427bea..4463f9e 100644--- a/requirements/requirements-data.txt+++ b/requirements/requirements-data.txt@@ -1,4 +1,31 @@-gdown+gdown==5.2.0# [aesthetic]clip @ git+https://github.com/openai/CLIP.git++# [caption llava]+ninja==1.11.1.1+flash-attn==2.5.7+transformers==4.36.2+tokenizers==0.15.2+sentencepiece==0.2.0+shortuuid==1.0.13+accelerate==0.29.2+peft==0.4.0+bitsandbytes==0.41.0+pydantic==2.7.1+markdown2[all]+numpy==1.26.3+scikit-learn==1.4.2+gradio==4.31.3+gradio_client==0.16.3+requests==2.31.0+httpx==0.27.0+uvicorn==0.29.0+fastapi==0.110.0+einops==0.8.0+einops-exts==0.0.4+timm==0.9.16++# [scene cut]+imageio-ffmpeg==0.4.9diff --git a/tools/caption/README.md b/tools/caption/README.mdindex 9c78518..8d4a567 100644--- a/tools/caption/README.md+++ b/tools/caption/README.md@@ -8,7 +8,11 @@ Human labeling of videos is expensive and time-consuming. We adopt powerful imagWe extract three frames from the video for captioning. With batch inference, we can achieve 10 times speedup. With approximately 720p resolution and 1 frames, the speed is 2~3 videos/s on 8 GPUs. If we resize the smaller side to 336, the speed can be 8 videos/s. In Open-Sora v1.1, to lower the cost, we use the 7B model.-### Requirement+### Installation++Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "LLaVA Captioning" sections.++<!-- ### Requirement```bash# create conda env@@ -34,7 +38,7 @@ pip install -e .pip install flash-attn --no-build-isolation# install colossalai and decordpip install colossalai decord-```+``` -->### Usage@@ -82,14 +86,21 @@ Then use the output csv file to resume the process.## PLLaVA Captioning-### Download the PLLaVA repo++### Installation+Install the required dependancies by following our [installation instructions](../../docs/installation.md)'s "Data Dependencies" and "PLLaVA Captioning" sections.+++<!-- ### Download the PLLaVA repoFirst, make sure you are under the directory of tools/caption/pllava_dir. Then,```bashgit clone https://github.com/magic-research/PLLaVA.git-cd PLLaVA+git checkout fd9194a++```### Environment@@ -107,8 +118,8 @@ pip install -r requirements.txt # change to your own torch version if neccessary### Download weights```bash-python python_scripts/hf.py # download the weights-```+python python_scripts/hf.py # download the weights+``` -->### UsageSince PLLaVA is not fashioned as a package, we will use PYTHONPATH to use it as a package.@@ -168,8 +179,5 @@ Each video is classified according to 8 categories:zoom_in,zoom_out,static,- unclassified`.+ unclassified`.Categories of `tilt`, `pan` and `zoom` can overlap with each other.---