Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

fix start from scratch

zhengzangw1e702a308852
diff --git a/docs/report_02.md b/docs/report_02.md
index 763d204..05ac062 100644
--- a/docs/report_02.md
+++ b/docs/report_02.md
@@ -95,8 +95,8 @@ With limited computational resources, we have to carefully monitor the training
2. **[Stage 1]** Then, we pretrain the model with gradient-checkpointing for **24k** steps, which takes **4 days** on 64 H800 GPUs. Although the number of samples seen by the model is the same, we find the model learns slowly compared to a smaller batch size. We speculate that at an early stage, the number of steps is more important for training. The most videos are in **240p** resolution, and the config is similar to [stage2.py](/configs/opensora-v1-1/train/stage2.py).
3. **[Stage 1]** To increase the number of steps, we switch to a smaller batch size without gradient-checkpointing. We also add fps conditioning at this point. We trained **40k** steps for **2 days**. The most videos are in **144p** resolution, and the config file is [stage1.py](/configs/opensora-v1-1/train/stage1.py).
4. **[Stage 1]** We find the model cannot learn well for long videos, and find a noised generation result as speculated to be half-precision problem found in Open-Sora 1.0 training. Thus, we adopt the QK-normalization to stabilize the training. We also switch iddpm-speed to iddpm. We trained for **17k** steps for **14 hours**. The most videos are in **144p** resolution, and the config file is [stage1.py](/configs/opensora-v1-1/train/stage1.py). The stage 1 training lasts for approximately one week, with total step **81k**.
-5. **[Stage 2]** We switch to a higher resolution, where most videos are in **240p and 480p** resolution ([stage2.py](/configs/opensora-v1-1/train/stage2.py)). We trained **22k** steps for **one day** on all pre-training data, and **?k** with **? hours** on high-quality data.
-6. **[Stage 3]** We switch to a higher resolution, where most videos are in **480p and 720p** resolution ([stage3.py](/configs/opensora-v1-1/train/stage3.py)). We trained **?k** with **? hours** on high-quality data.
+5. **[Stage 2]** We switch to a higher resolution, where most videos are in **240p and 480p** resolution ([stage2.py](/configs/opensora-v1-1/train/stage2.py)). We trained **22k** steps for **one day** on all pre-training data.
+6. **[Stage 3]** We switch to a higher resolution, where most videos are in **480p and 720p** resolution ([stage3.py](/configs/opensora-v1-1/train/stage3.py)). We trained **4k** with **one day** on high-quality data.
## Results and Evaluation
diff --git a/scripts/train.py b/scripts/train.py
index 33fa7f6..aed7b45 100644
--- a/scripts/train.py
+++ b/scripts/train.py
@@ -197,7 +197,7 @@ def main():
# 6.1. resume training
if cfg.load is not None:
logger.info("Loading checkpoint")
- start_epoch, start_step, sampler_start_idx = load(
+ ret = load(
booster,
model,
ema,
@@ -206,6 +206,8 @@ def main():
cfg.load,
sampler=sampler_to_io if not cfg.start_from_scratch else None,
)
+ if not cfg.start_from_scratch:
+ start_epoch, start_step, sampler_start_idx = ret
logger.info(f"Loaded checkpoint {cfg.load} at epoch {start_epoch} step {start_step}")
logger.info(f"Training for {cfg.epochs} epochs with {num_steps_per_epoch} steps per epoch")