Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation
| Name | Last commit message | Last commit date |
|---|---|---|
| assets | ||
| configs | ||
| docs | ||
| examples | ||
| src | ||
| tools | ||
| .gitignore 185 B | ||
| inference.py 1.0 KB | ||
| LICENSE 11.3 KB | ||
| pyproject.toml 1.2 KB | ||
| README.md 10.9 KB | ||
| requirements.txt 3.0 KB | ||
| run.sh 5.7 KB | ||
| train.py 953 B | ||
| uv.lock 463 KB |
Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation
Ruopeng Gao1, Yukun Huang2,†, Jiarui Hu3, Shuai Wang1, Wenbo Hu2, Ying Shan2, Limin Wang1,*
1Nanjing University 2ARC Lab, Tencent IEG 3Zhejiang University
†Project Lead *Corresponding Author
🔍 Introduction
Understanding visual evolution is essential for autoregressive video generation and world modeling. Yet temporal redundancy can encourage local temporal shortcuts during teacher-forcing training, weakening the incentive to learn underlying dynamics. To address this challenge, we introduce Future Chunk Prediction (FCP), which trains autoregressive video models to look ahead through auxiliary branches that predict temporally distant video chunks from intermediate features of the current chunk. Using only video-pixel supervision, FCP improves physical consistency in generated videos and learns more semantically structured representations.
We provide training and inference code, along with weights for 1.3B and 5B autoregressive text-and-image-to-video (TI2V) models. Stay tuned for more model sizes and variants.
🔥 News
- 2026-10-26: Conda + pip installation is now available.
- 2026-10-22: Our paper is now available on arXiv! If you find our work helpful, please consider giving it an upvote on HF Daily.
- 2026-10-21: Our model weights and demo dataset are now available on Hugging Face!
- 2026-10-21: Our project page and code are now available! Visit the project page for more demos.
🚧 Upcoming
We are continuing to improve this project and explore new directions. Watch this repository to stay up to date with our progress. The documentation is also evolving: if you have any questions about using the project, please open an issue. Where helpful, we will incorporate clarifications into future documentation updates.
⚙️ Setup
Tested on Linux with Python 3.12 and PyTorch 2.10.0+cu128 (CUDA 12.8).
You can set up the environment using any of the following methods.
uv (recommended)
Install uv, then run the following commands from the repository root.
On Ubuntu / Debian, install FFmpeg if it is not already available:
sudo apt-get update
sudo apt-get install -y ffmpeg
Install the project environment with Python 3.12:
UV_CACHE_DIR=.uv-cache uv sync --python 3.12
# Activate the virtual environment
source .venv/bin/activate
conda + pip
Run the following commands from the repository root:
conda create -n looking-ahead python=3.12 pip "ffmpeg=7.1.*" -c conda-forge -y
conda activate looking-ahead
python -m pip install torch==2.10.0 torchvision==0.25.0 \
--index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
Activate looking-ahead before training or inference. run.sh automatically
uses the activated environment. A compatible NVIDIA driver and a C++ compiler
(g++ / c++) are required.
requirements.txt pins dependencies from uv.lock.
🤗 Model Downloads
Upstream Model Weights
Download the upstream model corresponding to your selected model size. Each repository includes the base model weights, configuration, text encoder, tokenizer and VAE.
| Model | Download |
|---|---|
| Wan2.1-T2V-1.3B | 🤗 Hugging Face |
| Wan2.2-TI2V-5B | 🤗 Hugging Face |
After activating your environment, download the model from the repository root using the Hugging Face CLI:
# 1.3B
hf download Wan-AI/Wan2.1-T2V-1.3B \
--local-dir weights/Wan-AI/Wan2.1-T2V-1.3B
# 5B
hf download Wan-AI/Wan2.2-TI2V-5B \
--local-dir weights/Wan-AI/Wan2.2-TI2V-5B
These commands place the models in the default lookup directories under ./weights/Wan-AI when running from the repository root. To use another location, set WAN_ROOT to the parent directory containing the model folders.
Our Model Weights
| Model Size | Method | Task | Final Weights |
|---|---|---|---|
| 1.3B | TF (baseline) | TI2V | 🤗 Hugging Face |
| 1.3B | FCP (ours) | TI2V | 🤗 Hugging Face |
| 5B | TF (baseline) | TI2V | 🤗 Hugging Face |
| 5B | FCP (ours) | TI2V | 🤗 Hugging Face |
Download all final weights from the repository root:
hf download TencentARC/Looking-Ahead --local-dir weights/Looking-Ahead
Save the downloaded weights under weights/Looking-Ahead/, using the corresponding configuration name as the subdirectory and model_ema.pt as the filename.
weights/Looking-Ahead/
├── tf_wan21_t2v_1.3b/model_ema.pt
├── fcp_wan21_t2v_1.3b/model_ema.pt
├── tf_wan22_ti2v_5b/model_ema.pt
└── fcp_wan22_ti2v_5b/model_ema.pt
🎬 Inference
Text-and-Video-to-Video Generation
Run this single-GPU example from the repository root. It uses the 5B FCP model and the supplied milk-pouring video and prompt to generate a 5-second video, including the input context:
python inference.py \
--config fcp_wan22_ti2v_5b \
--checkpoint_path weights/Looking-Ahead/fcp_wan22_ti2v_5b/model_ema.pt \
--use_ema \
--video_path examples/inference/v2v/milk_pouring.mp4 \
--prompt_file examples/inference/v2v/milk_pouring.txt \
--duration 5s --seed 42 --sink 3 \
--output_path outputs/v2v/fcp_milk_pouring
By default, inference selects the longest valid tail segment of the input video
as the prefix. Use --v2v_context_latents to set the prefix length manually.
See the inference documentation for details.
Text-and-Image-to-Video Generation
Run the following single-GPU example from the repository root. It generates a 5-second video using the 1.3B FCP model, the provided sailboat image and its matching prompt file:
python inference.py \
--config fcp_wan21_t2v_1.3b \
--checkpoint_path weights/Looking-Ahead/fcp_wan21_t2v_1.3b/model_ema.pt \
--use_ema \
--image_path examples/inference/i2v/sailboat_lake.png \
--prompt_file examples/inference/i2v/sailboat_lake.txt \
--duration 5s --seed 42 \
--output_path outputs/example_inference
Input images are automatically resized with their aspect ratio preserved and center-cropped to match the model's resolution. See the inference documentation for more usage details.
🏋️ Training
Data Preparation
We provide a demo dataset to help you get started with training; custom datasets should follow the format below.
Download the demo dataset from 🤗 Hugging Face.
hf download TencentARC/Looking-Ahead --repo-type dataset \
--local-dir release_assets/dataset_demo
Place the downloaded demo datasets in the following directories under the repository root, preserving the source_a/ and source_b/ subdirectories:
- 1.3B:
release_assets/dataset_demo/wan21_t2v_1.3b/ - 5B:
release_assets/dataset_demo/wan22_ti2v_5b/
These paths are configured in data_sources in the corresponding YAML files under configs/ (e.g., fcp_wan21_t2v_1.3b.yaml and fcp_wan22_ti2v_5b.yaml); the TF configurations use the same dataset paths. For each source, data_path points to its latent_shards/ directory, and annotation_path points to its annotations.jsonl file. If you store the datasets elsewhere, update these fields in your selected configuration.
Training Examples
# Use all visible GPUs with W&B enabled and the default output directory
bash run.sh train fcp_wan21_t2v_1.3b
# Select GPUs
CUDA_VISIBLE_DEVICES=0,1,2,3 bash run.sh train fcp_wan21_t2v_1.3b
# Disable W&B
bash run.sh train fcp_wan21_t2v_1.3b --disable_wandb
# Specify the output directory
bash run.sh train fcp_wan21_t2v_1.3b --exp_dir outputs/my_experiment
To train another model or baseline, replace fcp_wan21_t2v_1.3b with tf_wan21_t2v_1.3b, fcp_wan22_ti2v_5b, or tf_wan22_ti2v_5b. Each name corresponds to a YAML file under configs/.
Citation
If you find this work useful, please cite:
@misc{gao2026lookingahead,
title = {Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation},
author = {Gao, Ruopeng and Huang, Yukun and Hu, Jiarui and Wang, Shuai and Hu, Wenbo and Shan, Ying and Wang, Limin},
year = {2026},
eprint = {2610.29989},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2610.29989}
}