Open-Sora: Democratizing Efficient Video Production for All
Python
update
Shen-Chenhuib80308ac83b9
diff --cc .gitignoreindex a534719,f0a41f1..14dc0e3--- a/.gitignore+++ b/.gitignore@@@ -193,5 -192,5 +193,5 @@@ node_modulespackage-lock.jsonpackage.json-+# PLLaVA- tools/caption/pllava_dir/PLLaVA+ tools/caption/pllava_dir/PLLaVA/diff --cc README.mdindex 2cdeb45,2cdb604..4ef88ea--- a/README.md+++ b/README.md@@@ -154,15 -154,15 +154,15 @@@ conda create -n opensora python=3.1conda activate opensora# install torch, torchvision and xformers-pip install -r requirements/requirements_cu121.txt+pip install -r requirements/requirements-cu121.txt- # install this project+ # download the repogit clone https://github.com/hpcaitech/Open-Soracd Open-Sora-# install Open-Sora-pip install -v .-```+# the default installation is for inference only+pip install -v . # for development mode, `pip install -v -e .`- ```++(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.diff --cc docs/installation.mdindex 95cdecf,9c41d0d..96fc8bf--- a/docs/installation.md+++ b/docs/installation.md@@@ -41,95 -24,5 +41,105 @@@ The default installation is for inferen```bashpip install -v .[data] # for data preprocessing-pip install -v .[eval] # for evaluation+pip install -v .[eval] # for evaluation, need to manually install some packages detailed below++# Note: if in development mode, use the following commands instead:+pip install -v -e .[data]+pip install -v -e .[eval]+```+++## Data Dependencies++First, run the following command to install requirements:+```bash+pip install -v .[data] # For development: `pip install -v -e .[eval]`+```+Next, you need to manually install the packages listed in the following sections specific to your data processing needs.++### Datasets+To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)++However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:+```bash+conda install -c conda-forge opencv+```++### LLaVA Captioning+You need to manually install LLaVA with the following command:+```bash+pip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]+```+### PLLaVA Captioning++You need to manually install PLLaVa with the following commands:+```bash+cd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directory+git clone https://github.com/magic-research/PLLaVA.git+cd PLLaVA+git checkout fd9194a # since there is no version tag, we use this commit+python python_scripts/hf.py # download the PLLaVA weights++++# create new environment because pllava only works well in its own environments:++conda create -n pllava python=3.10++# You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions, the following works for CUDA 12.1:++conda activate pllava++pip install -r ../../../requirements/requirements-cu121.txt++pip install packaging ninja++pip install flash-attn --no-build-isolation++# then manually remove any lines in requirements.txt that contains `cu11`, before installing:++pip install -r requirements.txt+```++### Frame Interpolation+```bash+conda install -c conda-forge opencv+```++### Scene Detection+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:+```bash+pip install scenedetect[opencv] --upgrade+```++++## Evaluation Dependencies++First, run the following command to install requirements:+```bash+pip install -v .[eval] # For development:`pip install -v -e .[eval]`+```+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.++### Human Eval++You need to manually install apex from source by:+```bash+# set enable_layernorm_kernel=False in config to disable apex+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git++```++### VBench+You need to manually install [VBench](https://github.com/Vchitect/VBench):+```bash+pip install --no-deps vbench==0.1.1+# If the installation shows a warning about the intalled vbench not in PATH, you need to add it by:+export PATH="/path/to/vbench:$PATH"+```+++### VAE++You need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).+* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`+* For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`++Note that for VAE evaluation, you may run into error with `ModuleNotFoundError: No module named 'torchvision.transforms.functional_tensor'`, in this case, you need to go to the corresponding file (`.../pytorchvideo/transforms/augmentations.py`) reporting this error, then change as following:+```yaml+# find the original line:+import torchvision.transforms.functional_tensor as F_t+# change to:+import torchvision.transforms._functional_tensor as F_t```diff --cc requirements/requirements-data.txtindex 8d060d7,d427bea..01c6736--- a/requirements/requirements-data.txt+++ b/requirements/requirements-data.txt@@@ -1,48 -1,4 +1,46 @@@-gdown+gdown==5.2.0++# [caption llava]+ninja==1.11.1.1+flash-attn==2.5.7+transformers==4.36.2+tokenizers==0.15.2+sentencepiece==0.2.0+shortuuid==1.0.13+accelerate==0.29.2+peft==0.4.0+bitsandbytes==0.41.0+pydantic==2.7.1+markdown2[all]+numpy==1.26.3+scikit-learn==1.4.2+gradio==4.31.3+gradio_client==0.16.3+requests==2.31.0+httpx==0.27.0+uvicorn==0.29.0+fastapi==0.110.0+einops==0.8.0+einops-exts==0.0.4+timm==0.9.16++# [camera_motion]+decord==0.6.0+ptvsd==4.3.2+imageio-ffmpeg==0.4.9++# [datasets]+pandarallel==1.6.5+opencv-python==4.9.0.80+ffmpeg-python==0.2.0+lingua-language-detector==2.0.2+imageio-ffmpeg==0.4.9++# [frame interpolation]+imageio==2.34.1+- # [scene detection]-+# [scene cut]+imageio-ffmpeg==0.4.9# [aesthetic]clip @ git+https://github.com/openai/CLIP.git