Open-Sora: Democratizing Efficient Video Production for All
Python
[docs] data processing
Zangwei Zhengff15a0acfb0f
diff --git a/README.md b/README.mdindex 8000852..fd1fa7e 100644--- a/README.md+++ b/README.md@@ -191,13 +191,48 @@ To lower the memory usage, set a smaller `vae.micro_batch_size` in the config (s## Data ProcessingHigh-quality Data is the key to high-quality models. Our used datasets and data collection plan-is [here](/docs/datasets.md). We provide tools to process video data. Currently, our data processing pipeline includes+is [here](/docs/datasets.md). We provide tools to process video data. Our data processing pipeline includesthe following steps:-1. Downloading datasets. [[docs](/tools/datasets/README.md)]+1. Manage datasets. [[docs](/tools/datasets/README.md)]2. Split videos into clips. [[docs](/tools/scenedetect/README.md)]3. Generate video captions. [[docs](/tools/caption/README.md)]+Below is an example workflow to process data. However, we recommend you to read the detailed documentation for each tool, and decide which tools to use based on your needs.++```bash+# Suppose files under ~/dataset/+# 1. Convert dataset to CSV+# output: ~/dataset.csv+python -m tools.dataset.convert video ~/dataset+# filter out broken videos (broken ones num_frames=0)+python -m tools.dataset.csvutil ~/dataset.csv --video-info --fmin 2 --output ~/dataset.csv++# 2. Filter dataset by aesthetic scores+# output: ~/dataset_aesthetic.csv+python -m tools.aesthetic.inference ~/dataset.csv+# sort and examine videos by aesthetic scores+python -m tools.datasets.csvutil ~/dataset_aesthetic.csv --sort-descending aesthetic_score+# bad videos (aesthetic_score < 5)+tail ~/dataset_aesthetic.csv+# filter videos by aesthetic scores+# output: ~/dataset_aesthetic_aesmin_5.csv+python -m tools.datasets.csvutil ~/dataset_aesthetic.csv --aesmin 5++# 3. Caption dataset+# output: ~/dataset_aesthetic_aesmin_5_caption.csv+torchrun --nproc_per_node 8 --standalone -m tools.caption.caption_llava ~/dataset_aesthetic_aesmin_5.csv --tp-size 2 --dp-size 4 --bs 16+# remove empty captions and process captions (may need to re-caption lost ones)+python -m tools.datasets.csvutil ~/dataset_aesthetic_aesmin_5_caption.csv --remove-caption-prefix --remove-empty-caption++# 4. Sanity check & prepare for training+# sanity check+python -m tools.datasets.csvutil ~/dataset_aesthetic_aesmin_5_caption.csv --ext --video-info --output ~/dataset_ready.csv+# filter out videos less than 48 frames+# output: ~/dataset_ready_fmin_48.csv+python -m tools.datasets.csvutil ~/dataset_ready.csv --fmin 48+```+## TrainingTo launch training, first download [T5](https://huggingface.co/DeepFloyd/t5-v1_1-xxl/tree/main) weightsdiff --git a/tools/caption/README.md b/tools/caption/README.mdindex a980b29..fd53303 100644--- a/tools/caption/README.md+++ b/tools/caption/README.md@@ -54,7 +54,7 @@ torchrun --nproc_per_node 2 --standalone -m tools.caption.caption_llava DATA.csvAfter running the script, with `dp-size=N`, you will get `N` parts of csv files. Run the following command to merge them:```bash-python -m tools.datasets.csvutil DATA_caption_part0.csv DATA_caption_part1.csv DATA_caption_part2.csv DATA_caption_part3.csv --output DATA_caption.csv+python -m tools.datasets.csvutil DATA_caption_part*.csv --output DATA_caption.csv```### Resume@@ -63,11 +63,9 @@ Sometimes the process may be interrupted. We can resume the process by running t```bash# merge generated results-# output: DATA_caption_part0+DATA_caption_part1+DATA_caption_part2+DATA_caption_part3.csv-python -m tools.datasets.csvutil DATA_caption_part0.csv DATA_caption_part1.csv DATA_caption_part2.csv DATA_caption_part3.csv+python -m tools.datasets.csvutil DATA_caption_part*.csv --output DATA_caption.csv# get the remaining videos-# output: DATA-DATA_caption_part0+DATA_caption_part1+DATA_caption_part2+DATA_caption_part3.csv-python -m tools.datasets.csvutil DATA.csv --difference DATA_caption_part0+DATA_caption_part1+DATA_caption_part2+DATA_caption_part3.csv+python -m tools.datasets.csvutil DATA.csv --difference DATA_caption.csv --output DATA_remaining.csv```Then use the output csv file to resume the process.