Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

update

Shen-Chenhuib80308ac83b9
diff --cc .gitignore
index a534719,f0a41f1..14dc0e3
--- a/.gitignore
+++ b/.gitignore
@@@ -193,5 -192,5 +193,5 @@@ node_modules
package-lock.json
package.json
-
+# PLLaVA
- tools/caption/pllava_dir/PLLaVA
+ tools/caption/pllava_dir/PLLaVA/
diff --cc README.md
index 2cdeb45,2cdb604..4ef88ea
--- a/README.md
+++ b/README.md
@@@ -154,15 -154,15 +154,15 @@@ conda create -n opensora python=3.1
conda activate opensora
# install torch, torchvision and xformers
-pip install -r requirements/requirements_cu121.txt
+pip install -r requirements/requirements-cu121.txt
- # install this project
+ # download the repo
git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora
-# install Open-Sora
-pip install -v .
-```
+# the default installation is for inference only
+pip install -v . # for development mode, `pip install -v -e .`
- ```
++
(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.
diff --cc docs/installation.md
index 95cdecf,9c41d0d..96fc8bf
--- a/docs/installation.md
+++ b/docs/installation.md
@@@ -41,95 -24,5 +41,105 @@@ The default installation is for inferen
```bash
pip install -v .[data] # for data preprocessing
-pip install -v .[eval] # for evaluation
+pip install -v .[eval] # for evaluation, need to manually install some packages detailed below
+
+# Note: if in development mode, use the following commands instead:
+pip install -v -e .[data]
+pip install -v -e .[eval]
+```
+
+
+## Data Dependencies
+
+First, run the following command to install requirements:
+```bash
+pip install -v .[data] # For development: `pip install -v -e .[eval]`
+```
+Next, you need to manually install the packages listed in the following sections specific to your data processing needs.
+
+### Datasets
+To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)
+
+However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:
+```bash
+conda install -c conda-forge opencv
+```
+
+### LLaVA Captioning
+You need to manually install LLaVA with the following command:
+```bash
+pip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]
+```
+### PLLaVA Captioning
+
+You need to manually install PLLaVa with the following commands:
+```bash
+cd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directory
+git clone https://github.com/magic-research/PLLaVA.git
+cd PLLaVA
+git checkout fd9194a # since there is no version tag, we use this commit
+python python_scripts/hf.py # download the PLLaVA weights
++
++# create new environment because pllava only works well in its own environments:
++conda create -n pllava python=3.10
++# You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions, the following works for CUDA 12.1:
++conda activate pllava
++pip install -r ../../../requirements/requirements-cu121.txt
++pip install packaging ninja
++pip install flash-attn --no-build-isolation
++# then manually remove any lines in requirements.txt that contains `cu11`, before installing:
++pip install -r requirements.txt
+```
+
+### Frame Interpolation
+```bash
+conda install -c conda-forge opencv
+```
+
+### Scene Detection
+We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:
+```bash
+pip install scenedetect[opencv] --upgrade
+```
+
+
+
+## Evaluation Dependencies
+
+First, run the following command to install requirements:
+```bash
+pip install -v .[eval] # For development:`pip install -v -e .[eval]`
+```
+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.
+
+### Human Eval
+
+You need to manually install apex from source by:
+```bash
+# set enable_layernorm_kernel=False in config to disable apex
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
+
+```
+
+### VBench
+You need to manually install [VBench](https://github.com/Vchitect/VBench):
+```bash
+pip install --no-deps vbench==0.1.1
+# If the installation shows a warning about the intalled vbench not in PATH, you need to add it by:
+export PATH="/path/to/vbench:$PATH"
+```
+
+
+### VAE
+
+You need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).
+* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`
+* For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`
+
+Note that for VAE evaluation, you may run into error with `ModuleNotFoundError: No module named 'torchvision.transforms.functional_tensor'`, in this case, you need to go to the corresponding file (`.../pytorchvideo/transforms/augmentations.py`) reporting this error, then change as following:
+```yaml
+# find the original line:
+import torchvision.transforms.functional_tensor as F_t
+# change to:
+import torchvision.transforms._functional_tensor as F_t
```
diff --cc requirements/requirements-data.txt
index 8d060d7,d427bea..01c6736
--- a/requirements/requirements-data.txt
+++ b/requirements/requirements-data.txt
@@@ -1,48 -1,4 +1,46 @@@
-gdown
+gdown==5.2.0
+
+# [caption llava]
+ninja==1.11.1.1
+flash-attn==2.5.7
+transformers==4.36.2
+tokenizers==0.15.2
+sentencepiece==0.2.0
+shortuuid==1.0.13
+accelerate==0.29.2
+peft==0.4.0
+bitsandbytes==0.41.0
+pydantic==2.7.1
+markdown2[all]
+numpy==1.26.3
+scikit-learn==1.4.2
+gradio==4.31.3
+gradio_client==0.16.3
+requests==2.31.0
+httpx==0.27.0
+uvicorn==0.29.0
+fastapi==0.110.0
+einops==0.8.0
+einops-exts==0.0.4
+timm==0.9.16
+
+# [camera_motion]
+decord==0.6.0
+ptvsd==4.3.2
+imageio-ffmpeg==0.4.9
+
+# [datasets]
+pandarallel==1.6.5
+opencv-python==4.9.0.80
+ffmpeg-python==0.2.0
+lingua-language-detector==2.0.2
+imageio-ffmpeg==0.4.9
+
+# [frame interpolation]
+imageio==2.34.1
+
- # [scene detection]
-
+# [scene cut]
+imageio-ffmpeg==0.4.9
# [aesthetic]
clip @ git+https://github.com/openai/CLIP.git