Open-Sora: Democratizing Efficient Video Production for All
Python
[docs] pllava
zhengzangwa3504b344b8f
diff --git a/docs/report_03.md b/docs/report_03.mdindex e6b0f0e..7f08798 100644--- a/docs/report_03.md+++ b/docs/report_03.md@@ -100,6 +100,8 @@ In this stage, we collect 2M video clips with 5k hours from all kinds of sources- [Vript](https://github.com/mutonix/Vript/tree/main): a densely annotated dataset.- And some other datasets.+While MiraData and Vript have captions from GPT, we use [PLLaVA](https://github.com/magic-research/PLLaVA) to caption the rest ones. We use 4 frames for each video captioning, and choose the 13B version. PLLaVA is more efficient and better finetuned for video than LLaVA. The accelerated PLLaVA is released in our tools.+We mainly train on 480p and 720p in this stage. The training config locates in [stage3.py](/configs/opensora-v1-2/train/stage3.py). We train the model for 15k steps, which is approximately 2 epochs.## Easy and effective model conditioning