Open-Sora: Democratizing Efficient Video Production for All
Python
[fix] installation
zhengzangwd8fc60862989
diff --git a/docs/installation.md b/docs/installation.mdindex 1acd4f1..c827fef 100644--- a/docs/installation.md+++ b/docs/installation.md@@ -3,20 +3,30 @@Requirements are listed in `requirements` folder.Note that besides these packages, some packages needs to be mannually installed, and are detailed in the following sections.-# Different CUDA versions+## Different CUDA versionsYou need to manually install `torch`, `torchvision` and `xformers` for different CUDA versions.For CUDA 12.1,+```bash-# need to update first, else may run into weird issues with apexpip install -U pip+# update, otherwise may run into weird issues with apexpip install -U setuptoolspip install -U wheel# install pytorch, torchvision, and xformerspip install -r requirements/requirements-cu121.txt+```++For other CUDA versions, you need to manually install the corresponding packages.++- PyTorch: choose install commands from [PyTorch installation page](https://pytorch.org/get-started/locally/) based on your own CUDA version.+- xformers: choose install commands from [xformers repo](https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers) based on your own CUDA version.+Then, you can install the project for training and inference with the following commands:++```bash# install this projectgit clone https://github.com/hpcaitech/Open-Soracd Open-Sora@@ -31,50 +41,53 @@ pip install -v . # NOTE: for development mode, run `pip install -v -e .`pip install packaging ninjapip install flash-attn --no-build-isolation+# install apex, the compilation will take a long time# set enable_layernorm_kernel=False in config to disable apexpip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git```-# Optional Dependencies+## Optional DependenciesThe default installation is for inference only. Other optional dependencies are detailed in sections below, namel "Data Dependencies" and "Evaluation Dependencies".-<!-- ```bash-pip install -v .[data] # for data preprocessing, still need to manually install some packages detailed in sections below-pip install -v .[eval] # for evaluation, still need to manually install some packages detailed in sections below--# Note: if in development mode, use the following commands instead:-pip install -v -e .[data]-pip install -v -e .[eval]-``` -->---## Data Dependencies+### Data DependenciesFirst, run the following command to install requirements:+```bash-pip install -v .[data] # For development: `pip install -v -e .[eval]`+pip install -v .[data]+# For development: `pip install -v -e .[eval]````+Next, you need to manually install the packages listed in the following sections specific to your data processing needs.-### Datasets-To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python) in our [requirement script](../requirements/requirements-data.txt)+#### OpenCV++To get image and video information, we use [opencv-python](https://github.com/opencv/opencv-python). You can install it with pip:++```bash+pip install opencv-python+```However, if your videos are in av1 codec instead of h264, you need to install ffmpeg (already in our [requirement script](../requirements/requirements-data.txt)), then run the following to make conda support av1 codec:+```bashconda install -c conda-forge opencv```-### LLaVA Captioning+#### LLaVA Captioning+You need to manually install LLaVA with the following command:+```bashpip install --no-deps llava@git+https://github.com/haotian-liu/[email protected]```-### PLLaVA Captioning+#### PLLaVA CaptioningYou need to manually install PLLaVa with the following commands:+```bashcd tools/caption/pllava_dir # Assume you are in Open-Sora-dev root directorygit clone https://github.com/magic-research/PLLaVA.git@@ -94,19 +107,15 @@ pip install flash-attn --no-build-isolationpip install -r ../../../../requirements/requirements-pllava.txt```+#### Scene Detection-### Frame Interpolation-```bash-conda install -c conda-forge opencv-```--### Scene DetectionWe use [`PySceneDetect`](https://github.com/Breakthrough/PySceneDetect) for this job. You need to manually run the following:+```bashpip install scenedetect[opencv] --upgrade```-### OCR+#### OCRYou need to go into `path_to_your_env/lib/python3.10/site-packages/mmdet/__init__.py`and change the assert of `mmcv_version < digit_version(mmcv_maximum_version)` to `mmcv_version <= digit_version(mmcv_maximum_version)`.@@ -114,70 +123,37 @@ and change the assert of `mmcv_version < digit_version(mmcv_maximum_version)` toIf you are unsure of your path to the mmdet init file, simply run our [OCR command](../tools/scoring/README.md), wait for the mmdeet assertion error on mmcv versions.The error will contain the exact path to the mmdet init file.--<!-- We need to manualy create new environment for torch==2.0.1 in order to install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).-This is because the current latest MMOCR version (1.0.1) is not compatible with higher versions of mmcv that work with higher torch versions.--```bash-conda create -n ocr python=3.8-conda activate ocr-nvcc --version # Check that you have CUDA 11.8 or 11.7, if not install CUDA first-pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118-pip install packaging==24.0-pip install openmim==0.3.9-mim install mmengine==0.10.4-mim install mmcv==2.0.1-mim install mmdet==3.1.0-mim install mmocr==1.0.1-pip install colossalai==0.3.6--# install apex-pip install -U pip-pip install -U wheel-pip install setuptools==60.2.0-pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git-``` -->----## Evaluation Dependencies+### Evaluation DependenciesFirst, run the following command to install requirements:+```bash-pip install -v .[eval] # For development:`pip install -v -e .[eval]`+pip install -v .[eval]+# For development:`pip install -v -e .[eval]````-Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.--### Human Eval-You need to manually install apex from source by:-```bash-# use latest pip, setuptools, and wheel; else may run into weird issues with apex-pip install -U pip-pip install -U setuptools-pip install -U wheel-# set enable_layernorm_kernel=False in config to disable apex-pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git+Next, you need to manually install the packages listed in the following sections specific to different evaluation methods.-```+#### VBench-### VBenchYou need to manually install [VBench](https://github.com/Vchitect/VBench):+```bashpip install --no-deps vbench==0.1.1# If the installation shows a warning about the intalled vbench not in PATH, you need to add it by:export PATH="/path/to/vbench:$PATH"```--### VAE+#### VAEYou need to mannually install [cupy](https://docs.cupy.dev/en/stable/install.html).-* For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`-* For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`++- For CUDA v11.2~11.8 (x86_64 / aarch64), `pip install cupy-cuda11x`+- For CUDA v12.x (x86_64 / aarch64), `pip install cupy-cuda12x`Note that for VAE evaluation, you may run into error with `ModuleNotFoundError: No module named 'torchvision.transforms.functional_tensor'`, in this case, you need to go to the corresponding file (`.../pytorchvideo/transforms/augmentations.py`) reporting this error, then change as following:-```yaml++```python# find the original line:import torchvision.transforms.functional_tensor as F_t# change to:diff --git a/requirements/requirements-data.txt b/requirements/requirements-data.txtindex 8fabae1..2933e69 100644--- a/requirements/requirements-data.txt+++ b/requirements/requirements-data.txt@@ -1,27 +1,26 @@-gdown==5.2.0+gdown>=5.2.0# [caption llava]-ninja==1.11.1.1-flash-attn==2.5.7-transformers==4.36.2-tokenizers==0.15.2-sentencepiece==0.2.0-shortuuid==1.0.13-accelerate==0.29.2-peft==0.4.0-bitsandbytes==0.41.0-pydantic==2.7.1+ninja>=1.11.1.1+transformers>=4.36.2+tokenizers>=0.15.2+sentencepiece>=0.2.0+shortuuid>=1.0.13+accelerate>=0.29.2+peft>=0.4.0+bitsandbytes>=0.41.0+pydantic>=2.7.1markdown2[all]-numpy==1.26.3-scikit-learn==1.4.2-gradio==4.31.3-gradio_client==0.16.3-requests==2.31.0-httpx==0.27.0-uvicorn==0.29.0-fastapi==0.110.0-einops==0.8.0-einops-exts==0.0.4+numpy>=1.26.3+scikit-learn>=1.4.2+gradio>=4.31.3+gradio_client>=0.16.3+requests>=2.31.0+httpx>=0.27.0+uvicorn>=0.29.0+fastapi>=0.110.0+einops>=0.8.0+einops-exts>=0.0.4timm==0.9.16# [camera_motion]@@ -31,20 +30,14 @@ imageio-ffmpeg==0.4.9# [datasets]pandarallel==1.6.5-opencv-python==4.9.0.80ffmpeg-python==0.2.0lingua-language-detector==2.0.2-imageio-ffmpeg==0.4.9# [frame interpolation]imageio==2.34.1-# [scene cut]-imageio-ffmpeg==0.4.9-# [aesthetic]clip @ git+https://github.com/openai/CLIP.git-decord==0.6.0# [ocr]mmcv==2.1.0