Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

Dev/pxy (#66)

xyupengd79c8c6b47cd
diff --git a/docs/data_processing.md b/docs/data_processing.md
new file mode 100644
index 0000000..47532e8
--- /dev/null
+++ b/docs/data_processing.md
@@ -0,0 +1,9 @@
+# Data Processing
+
+## Dataset Management
+
+## Scene Detection and Video Splitting
+
+## Scoring and Filtering
+
+## Captioning
\ No newline at end of file
diff --git a/tools/scoring/README.md b/tools/scoring/README.md
index e4ccae8..06ddd17 100644
--- a/tools/scoring/README.md
+++ b/tools/scoring/README.md
@@ -1,24 +1,24 @@
-# Data Scoring and Filtering
+# Scoring and Filtering
-- [Data Scoring and Filtering](#data-scoring-and-filtering)
- - [Aesthetic Scoring](#aesthetic-scoring)
- - [Requirement](#requirement)
- - [Usage](#usage)
+- [Scoring and Filtering](#scoring-and-filtering)
+ - [Aesthetic Score](#aesthetic-score)
- [Optical Flow Score](#optical-flow-score)
+ - [OCR](#ocr)
- [Matching Score](#matching-score)
+ - [Filtering](#filtering)
-## Aesthetic Scoring
+## Aesthetic Score
-To evaluate the aesthetic quality of videos, we use a pretrained model from [CLIP+MLP Aesthetic Score Predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor). This model is trained on 176K SAC (Simulacra Aesthetic Captions) pairs, 15K LAION-Logos (Logos) pairs, and 250K AVA (The Aesthetic Visual Analysis) image-text pairs.
+To evaluate the aesthetic quality of videos, we use the scoring model from [CLIP+MLP Aesthetic Score Predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor). This model is trained on 176K SAC (Simulacra Aesthetic Captions) pairs, 15K LAION-Logos (Logos) pairs, and 250K AVA (The Aesthetic Visual Analysis) image-text pairs.
-The score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for good aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.
+The aesthetic score is between 1 and 10, where 5.5 can be considered as the threshold for fair aesthetics, and 6.5 for high aesthetics. Good text-to-image models can achieve a score of 7.0 or higher.
-For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images. Our script enables 1k videos/s with one GPU. It also supports multiple GPUs to further accelerate the process.
-
-### Requirement
+For videos, we extract the first, last, and the middle frames for evaluation. The script also supports images as input.
+The throughput of our code is ~1K videos/s on a single H800 GPU. It also supports running on multiple GPUs for further acceleration.
+First, install the required packages and download the scoring model to `./pretrained_models/aesthetic.pth`.
```bash
-# install clip
+# pip install
pip install git+https://github.com/openai/CLIP.git
pip install decord
@@ -26,45 +26,75 @@ pip install decord
wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/main/sac+logos+ava1-l14-linearMSE.pth -O pretrained_models/aesthetic.pth
```
-### Usage
-
-With `meta.csv` containing the paths to the videos, run the following command:
-
+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).
```bash
-# output: meta_aes.csv
torchrun --nproc_per_node 8 -m tools.scoring.aesthetic.inference /path/to/meta.csv --bs 1024 --num_workers 16
```
-
-This will generate multiple part files, you can use `python -m tools.datasets.csvutil DATA1.csv DATA2.csv` to merge these part files.
+This will generate multiple part files. Run `python -m tools.datasets.datautil /path/to/meta_part1.csv /path/to/meta_part2.csv` to merge these part files.
## Optical Flow Score
Optical flow scores are used to assess the motion of a video. Higher optical flow scores indicate larger movement.
-TODO: acknowledge UniMatch.
+We use the [UniMatch](https://github.com/autonomousvision/unimatch) model for this task.
-First get the pretrained model.
+First, download the pretrained model to `./pretrained_model/unimatch/`
+```bash
+wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P ./pretrained_models/unimatch/
+```
+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).
```bash
-wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P pretrained_models/unimatch
+torchrun --standalone --nproc_per_node 8 tools/scoring/optical_flow/inference.py /path/to/meta.csv
```
-Then run:
+This should output `/path/to/meta_flow.csv` with column `flow`.
+
+## OCR
+Some videos are of dense text scenes like news broadcast and advertisement, which are not desired for training.
+We apply Optical Character Recognition (OCR) to detect texts and drop samples with dense texts. Here, we use
+the [DBNet++](https://arxiv.org/abs/2202.10304) model implemented by [MMOCR](https://github.com/open-mmlab/mmocr/).
+
+First, install [MMOCR](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
+For reference, we install packages of these versions.
+```
+torch==2.0.1
+mmcv==2.0.1
+mmdet==3.1.0
+mmocr==1.0.1
+```
+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample).
```bash
-torchrun --standalone --nproc_per_node 8 tools/scoring/optical_flow/inference.py /path/to/meta.csv
+torchrun --standalone --nproc_per_node 8 tools/scoring/ocr/inference.py /path/to/meta.csv
```
+This should output `/path/to/meta_ocr.csv` with column `ocr`, indicating the number of text regions with detection confidence > 0.3.
-The output should be `/path/to/meta_flow.csv` with column `flow`.
## Matching Score
Matching scores are calculated to evaluate the alignment between an image/video and its caption.
-For videos, we compute the matching score of the middle frame and the caption.
+Here, we use the [CLIP](https://github.com/openai/CLIP) model, which is trained on image-text pairs.
+We simply use the cosine similarity as the matching score.
+For videos, we extract the middle frame and compare it with the caption.
+
+First, install OpenAI CLIP.
+```bash
+pip install git+https://github.com/openai/CLIP.git
+```
-**Make sure** meta files contain the column `text`, which is the caption of the sample. Then run:
+Then, run the following command. **Make sure** the meta file has column `path` (path to the sample) and `text` (caption of the sample).
```bash
torchrun --standalone --nproc_per_node 8 tools/scoring/matching/inference.py /path/to/meta.csv
```
-The output should be `/path/to/meta_match.csv` with column `match`. Higher matching scores indicate better image-text/video-text alignment.
+This should output `/path/to/meta_match.csv` with column `match`. Higher matching scores indicate better image-text/video-text alignment.
+
+
+## Filtering
+Once scores are obtained, it is simple to filter samples based on these scores. Here is an example to remove
+samples of aesthetic score < 5.0.
+```
+python -m tools.datasets.datautil /path/to/meta.csv --aesmin 5.0
+```
+This should output `/path/to/meta_aesmin5.0.csv` with column `aes` >= 5.0
\ No newline at end of file