Open-Sora: Democratizing Efficient Video Production for All
Python
Dev/pxy (#66)
xyupengd79c8c6b47cd
diff --git a/docs/data_processing.md b/docs/data_processing.mdnew file mode 100644index 0000000..47532e8--- /dev/null+++ b/docs/data_processing.md@@ -0,0 +1,9 @@+# Data Processing++## Dataset Management++## Scene Detection and Video Splitting++## Scoring and Filtering++## Captioning\ No newline at end of filediff --git a/tools/scoring/README.md b/tools/scoring/README.mdindex e4ccae8..06ddd17 100644--- a/tools/scoring/README.md+++ b/tools/scoring/README.md@@ -1,24 +1,24 @@-# Data Scoring and Filtering+# Scoring and Filtering-- [Data Scoring and Filtering](#data-scoring-and-filtering)- - [Aesthetic Scoring](#aesthetic-scoring)- - [Requirement](#requirement)- - [Usage](#usage)+- [Scoring and Filtering](#scoring-and-filtering)+ - [Aesthetic Score](#aesthetic-score)- [Optical Flow Score](#optical-flow-score)+ - [OCR](#ocr)- [Matching Score](#matching-score)+ - [Filtering](#filtering)-## Aesthetic Scoring+## Aesthetic Score-To evaluate the aesthetic quality of videos, we use a pretrained model from [CLIP+MLP Aesthetic Score Predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor). This model is trained on 176K SAC (Simulacra Aesthetic Captions) pairs, 15K LAION-Logos (Logos) pairs, and 250K AVA (The Aesthetic Visual Analysis) image-text pairs.+To evaluate the aesthetic quality of videos, we use the scoring model from [CLIP+MLP Aesthetic Score Predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor). This model is trained on 176K SAC (Simulacra Aesthetic Captions) pairs, 15K LAION-Logos (Logos) pairs, and 250K AVA (The Aesthetic Visual Analysis) image-text pairs.-The score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for good aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.+The aesthetic score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for high aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.-For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images. Our script enables 1k videos/s with one GPU. It also supports multiple GPUs to further accelerate the process.--### Requirement+For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.+The throughput of our code is ~1K videos/s on a single H800 GPU. It also supports running on multiple GPUs for further acceleration.+First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.```bash-# install clip+# pip installpip install git+https://github.com/openai/CLIP.gitpip install decord@@ -26,45 +26,75 @@ pip install decordwget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth```-### Usage--With `meta.csv` containing the paths to the videos, run the following command:-+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).```bash-# output: meta_aes.csvtorchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference /path/to/meta.csv --bs 1024 --num_workers 16```--This will generate multiple part files, you can use `python -m tools.datasets.csvutil DATA1.csv DATA2.csv` to merge these part files.+This will generate multiple part files. Run `python -m tools.datasets.datautil /path/to/meta_part1.csv /path/to/meta_part2.csv` to merge these part files.## Optical Flow ScoreOptical flow scores are used to assess the motion of a video. Higher optical flow scores indicate larger movement.-TODO: acknowledge UniMatch.+We use the [UniMatch](https://github.com/autonomousvision/unimatch) model for this task.-First get the pretrained model.+First, download the pretrained model to `./pretrained_model/unimatch/`+```bash+wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P ./pretrained_models/unimatch/+```+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).```bash-wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P pretrained_models/unimatch+torchrun --standalone --nproc_per_node 8 tools/scoring/optical_flow/inference.py /path/to/meta.csv```-Then run:+This should output `/path/to/meta_flow.csv` with column `flow`.++## OCR+Some videos are of dense text scenes like news broadcast and advertisement, which are not desired for training.+We apply Optical Character Recognition (OCR) to detect texts and drop samples with dense texts. Here, we use+the [DBNet++](https://arxiv.org/abs/2202.10304) model implemented by [MMOCR](https://github.com/open-mmlab/mmocr/).++First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).+For reference, we install packages of these versions.+```+torch==2.0.1+mmcv==2.0.1+mmdet==3.1.0+mmocr==1.0.1+```+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).```bash-torchrun --standalone --nproc_per_node 8 tools/scoring/optical_flow/inference.py /path/to/meta.csv+torchrun --standalone --nproc_per_node 8 tools/scoring/ocr/inference.py /path/to/meta.csv```+This should output `/path/to/meta_ocr.csv` with column `ocr`, indicating the number of text regions with detection confidence > 0.3.-The output should be `/path/to/meta_flow.csv` with column `flow`.## Matching ScoreMatching scores are calculated to evaluate the alignment between an image/video and its caption.-For videos, we compute the matching score of the middle frame and the caption.+Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.+We simply use the cosine similarity as the matching score.+For videos, we extract the middle frame and compare it with the caption.++First, install OpenAI CLIP.+```bash+pip install git+https://github.com/openai/CLIP.git+```-**Make sure** meta files contain the column `text`, which is the caption of the sample. Then run:+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample) and `text` (caption of the sample).```bashtorchrun --standalone --nproc_per_node 8 tools/scoring/matching/inference.py /path/to/meta.csv```-The output should be `/path/to/meta_match.csv` with column `match`. Higher matching scores indicate better image-text/video-text alignment.+This should output `/path/to/meta_match.csv` with column `match`. Higher matching scores indicate better image-text/video-text alignment.+++## Filtering+Once scores are obtained, it is simple to filter samples based on these scores. Here is an example to remove+samples of aesthetic score < 5.0.+```+python -m tools.datasets.datautil /path/to/meta.csv --aesmin 5.0+```+This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0\ No newline at end of file