Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

[fix] installation

zhengzangwd8fc60862989
diff --git a/docs/installation.md b/docs/installation.md
index 1acd4f1..c827fef 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -3,20 +3,30 @@
Requirements are listed in `requirements` folder.
Note that besides these packages, some packages needs to be mannually installed, and are detailed in the following sections.
-# Different CUDA versions
+## Different CUDA versions
You need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions.
For CUDA 12.1,
+
```bash
-# need to update first, else may run into weird issues with apex
pip install -U pip
+# update, otherwise may run into weird issues with apex
pip install -U setuptools
pip install -U wheel
# install pytorch, torchvision, and xformers
pip install -r requirements/requirements-cu121.txt
+```
+
+For other CUDA versions, you need to manually install the corresponding packages.
+
+- PyTorch: choose install commands from [PyTorch installation page](https://pytorch.org/get-started/locally/) based on your own CUDA version.
+- xformers: choose install commands from [xformers repo](https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers) based on your own CUDA version.
+Then, you can install the project for training and inference with the following commands:
+
+```bash
# install this project
git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora
@@ -31,50 +41,53 @@ pip install -v . # NOTE: for development mode, run `pip install -v -e .`
pip install packaging ninja
pip install flash-attn --no-build-isolation
+
# install apex, the compilation will take a long time
# set enable_layernorm_kernel=False in config to disable apex
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
```
-# Optional Dependencies
+## Optional Dependencies
The default installation is for inference only. Other optional dependencies are detailed in sections below, namel "Data Dependencies" and "Evaluation Dependencies".
-<!-- ```bash
-pip install -v .[data] # for data preprocessing, still need to manually install some packages detailed in sections below
-pip install -v .[eval] # for evaluation, still need to manually install some packages detailed in sections below
-
-# Note: if in development mode, use the following commands instead:
-pip install -v -e .[data]
-pip install -v -e .[eval]
-``` -->
-
-
-## Data Dependencies
+### Data Dependencies
First, run the following command to install requirements:
+
```bash
-pip install -v .[data] # For development: `pip install -v -e .[eval]`
+pip install -v .[data]
+# For development: `pip install -v -e .[eval]`
```
+
Next, you need to manually install the packages listed in the following sections specific to your data processing needs.
-### Datasets
-To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)
+#### OpenCV
+
+To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python). You can install it with pip:
+
+```bash
+pip install opencv-python
+```
However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:
+
```bash
conda install -c conda-forge opencv
```
-### LLaVA Captioning
+#### LLaVA Captioning
+
You need to manually install LLaVA with the following command:
+
```bash
pip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]
```
-### PLLaVA Captioning
+#### PLLaVA Captioning
You need to manually install PLLaVa with the following commands:
+
```bash
cd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directory
git clone https://github.com/magic-research/PLLaVA.git
@@ -94,19 +107,15 @@ pip install flash-attn --no-build-isolation
pip install -r ../../../../requirements/requirements-pllava.txt
```
+#### Scene Detection
-### Frame Interpolation
-```bash
-conda install -c conda-forge opencv
-```
-
-### Scene Detection
We use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:
+
```bash
pip install scenedetect[opencv] --upgrade
```
-### OCR
+#### OCR
You need to go into `path_to_your_env/lib/python3.10/site-packages/mmdet/__init__.py`
and change the assert of `mmcv_version < digit_version(mmcv_maximum_version)` to `mmcv_version <= digit_version(mmcv_maximum_version)`.
@@ -114,70 +123,37 @@ and change the assert of `mmcv_version < digit_version(mmcv_maximum_version)` to
If you are unsure of your path to the mmdet init file, simply run our [OCR command](../tools/scoring/README.md), wait for the mmdeet assertion error on mmcv versions.
The error will contain the exact path to the mmdet init file.
-
-<!-- We need to manualy create new environment for torch==2.0.1 in order to install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
-This is because the current latest MMOCR version (1.0.1) is not compatible with higher versions of mmcv that work with higher torch versions.
-
-```bash
-conda create -n ocr python=3.8
-conda activate ocr
-nvcc --version # Check that you have CUDA 11.8 or 11.7, if not install CUDA first
-pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118
-pip install packaging==24.0
-pip install openmim==0.3.9
-mim install mmengine==0.10.4
-mim install mmcv==2.0.1
-mim install mmdet==3.1.0
-mim install mmocr==1.0.1
-pip install colossalai==0.3.6
-
-# install apex
-pip install -U pip
-pip install -U wheel
-pip install setuptools==60.2.0
-pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
-``` -->
-
-
-
-## Evaluation Dependencies
+### Evaluation Dependencies
First, run the following command to install requirements:
+
```bash
-pip install -v .[eval] # For development:`pip install -v -e .[eval]`
+pip install -v .[eval]
+# For development:`pip install -v -e .[eval]`
```
-Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.
-
-### Human Eval
-You need to manually install apex from source by:
-```bash
-# use latest pip, setuptools, and wheel; else may run into weird issues with apex
-pip install -U pip
-pip install -U setuptools
-pip install -U wheel
-# set enable_layernorm_kernel=False in config to disable apex
-pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.
-```
+#### VBench
-### VBench
You need to manually install [VBench](https://github.com/Vchitect/VBench):
+
```bash
pip install --no-deps vbench==0.1.1
# If the installation shows a warning about the intalled vbench not in PATH, you need to add it by:
export PATH="/path/to/vbench:$PATH"
```
-
-### VAE
+#### VAE
You need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).
-* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`
-* For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`
+
+- For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`
+- For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`
Note that for VAE evaluation, you may run into error with `ModuleNotFoundError: No module named 'torchvision.transforms.functional_tensor'`, in this case, you need to go to the corresponding file (`.../pytorchvideo/transforms/augmentations.py`) reporting this error, then change as following:
-```yaml
+
+```python
# find the original line:
import torchvision.transforms.functional_tensor as F_t
# change to:
diff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txt
index 8fabae1..2933e69 100644
--- a/requirements/requirements-data.txt
+++ b/requirements/requirements-data.txt
@@ -1,27 +1,26 @@
-gdown==5.2.0
+gdown>=5.2.0
# [caption llava]
-ninja==1.11.1.1
-flash-attn==2.5.7
-transformers==4.36.2
-tokenizers==0.15.2
-sentencepiece==0.2.0
-shortuuid==1.0.13
-accelerate==0.29.2
-peft==0.4.0
-bitsandbytes==0.41.0
-pydantic==2.7.1
+ninja>=1.11.1.1
+transformers>=4.36.2
+tokenizers>=0.15.2
+sentencepiece>=0.2.0
+shortuuid>=1.0.13
+accelerate>=0.29.2
+peft>=0.4.0
+bitsandbytes>=0.41.0
+pydantic>=2.7.1
markdown2[all]
-numpy==1.26.3
-scikit-learn==1.4.2
-gradio==4.31.3
-gradio_client==0.16.3
-requests==2.31.0
-httpx==0.27.0
-uvicorn==0.29.0
-fastapi==0.110.0
-einops==0.8.0
-einops-exts==0.0.4
+numpy>=1.26.3
+scikit-learn>=1.4.2
+gradio>=4.31.3
+gradio_client>=0.16.3
+requests>=2.31.0
+httpx>=0.27.0
+uvicorn>=0.29.0
+fastapi>=0.110.0
+einops>=0.8.0
+einops-exts>=0.0.4
timm==0.9.16
# [camera_motion]
@@ -31,20 +30,14 @@ imageio-ffmpeg==0.4.9
# [datasets]
pandarallel==1.6.5
-opencv-python==4.9.0.80
ffmpeg-python==0.2.0
lingua-language-detector==2.0.2
-imageio-ffmpeg==0.4.9
# [frame interpolation]
imageio==2.34.1
-# [scene cut]
-imageio-ffmpeg==0.4.9
-
# [aesthetic]
clip @ git+https://github.com/openai/CLIP.git
-decord==0.6.0
# [ocr]
mmcv==2.1.0