Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

Dev/pxy (#67)

xyupengc6cbc7a6bfc3
diff --git a/docs/data_processing.md b/docs/data_processing.md
index 47532e8..21c55e3 100644
--- a/docs/data_processing.md
+++ b/docs/data_processing.md
@@ -1,9 +1,20 @@
# Data Processing
+We establish a complete pipeline for video/image data processing. The pipeline is shown below.
-## Dataset Management
+![pipeline](/assets/readme/report_data_pipeline.png)
-## Scene Detection and Video Splitting
+First, raw videos,
+either from the Internet or public datasets, are split into shorter clips based on scene detection.
+Then, we evaluate these videos by predicting multiple scores using existing models. We first predict the aesthetic score
+and the optical flow score for a video. We also conduct OCR to detect texts in the video. Only videos with satisfactory
+evaluation results are sent to the next step for captioning. After captioning, the matching score is also calculated as
+an assessment of video-text alignment. Finally, we filter samples based on the matching score and
+conduct camera motion detection for the remaining samples.
+In summary, our pipeline produces video-text pairs which have high aesthetic quality, large video motion and strong
+semantic consistency.
-## Scoring and Filtering
-
-## Captioning
\ No newline at end of file
+For more information, please refer to:
+- [Dataset Management](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/datasets/README.md)
+- [Scene Detection and Video Splitting](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/scene_cut/README.md)
+- [Scoring and Filtering](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/scoring/README.md)
+- [Captioning](https://github.com/hpcaitech/Open-Sora-dev/blob/dev/v1.1/tools/caption/README.md)
\ No newline at end of file