Skip to content

[CVPR 2026] PersonaLive! : Expressive Portrait Image Animation for Live Streaming

update

Zhiyuan Li8398fa5c989f
diff --git a/README.md b/README.md
index 42d3792..505bbac 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,6 @@
<h2>Expressive Portrait Image Animation for Live Streaming</h2>
-
#### [Zhiyuan Li<sup>1,2,3</sup>](https://huai-chang.github.io/) ยท [Chi-Man Pun<sup>1</sup>](https://cmpun.github.io/) ๐Ÿ“ช ยท [Chen Fang<sup>2</sup>](http://fangchen.org/) ยท [Jue Wang<sup>2</sup>](https://scholar.google.com/citations?user=Bt4uDWMAAAAJ&hl=en) ยท [Xiaodong Cun<sup>3</sup>](https://vinthony.github.io/academic/) ๐Ÿ“ช
<sup>1</sup> University of Macau &nbsp;&nbsp; <sup>2</sup> [Dzine.ai](https://www.dzine.ai/) &nbsp;&nbsp; <sup>3</sup> [GVC Lab, Great Bay University](https://gvclab.github.io/)
@@ -14,16 +13,91 @@
## ๐Ÿ“ฃ Updates
- **[2025.12.10]** ๐Ÿ”ฅ Release `inference code`, `config` and `checkpoints`!
-## ๐Ÿคช Framework
+## โš™๏ธ Framework
<img src="assets/overview.png" alt="Image 1" width="100%">
-<p align="justify">
-We present PersonaLive, a few-step diffusion-based framework for real-time and streamable portrait animation, which achieves low-latency and stable long-term quality.
-</p>
+
+
+We present PersonaLive, a `real-time` and `streamable` diffusion framework capable of generating `infinite-length` portrait animations on a single `12GB GPU`.
## ๐Ÿš€ Getting Started
+### ๐Ÿ›  Installation
+```
+# clone this repo
+git clone https://github.com/GVCLab/PersonaLive
+cd PersonaLive
+
+# Create conda environment
+conda create -n personalive python=3.10
+conda activate personalive
+
+# Install packages with pip
+pip install -r requirements.txt
+```
+
+### โฌ Download weights
+1. Download pre-trained weight of based models and other components ([SD V1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse) and [image_encoder](https://huggingface.co/lambdalabs/sd-image-variations-diffusers/tree/main/image_encoder))
+
+1. Download [pre-trained models](https://drive.google.com/drive/folders/1GOhDBKIeowkMpBnKhGB8jgEhJt_--vbT?usp=drive_link) into the `./pretrained_weights` folder.
+
+Finally, these weights should be orgnized as follows:
+```
+pretrained_weights
+โ”œโ”€โ”€ onnx
+โ”‚ โ”œโ”€โ”€ unet_opt
+โ”‚ โ”‚ โ”œโ”€โ”€ unet_opt.onnx
+โ”‚ โ”‚ โ””โ”€โ”€ unet_opt.onnx.data
+โ”‚ โ””โ”€โ”€ unet
+โ”œโ”€โ”€ personalive
+โ”‚ โ”œโ”€โ”€ denoising_unet.pth
+โ”‚ โ”œโ”€โ”€ motion_encoder.pth
+โ”‚ โ”œโ”€โ”€ motion_extractor.pth
+โ”‚ โ”œโ”€โ”€ pose_guider.pth
+โ”‚ โ”œโ”€โ”€ reference_unet.pth
+โ”‚ โ””โ”€โ”€ temporal_module.pth
+โ”œโ”€โ”€ sd-vae-ft-mse
+โ”‚ โ”œโ”€โ”€ diffusion_pytorch_model.bin
+โ”‚ โ””โ”€โ”€ config.json
+โ””โ”€โ”€ sd-image-variations-diffusers
+โ”‚ โ”œโ”€โ”€ image_encoder
+โ”‚ โ”‚ โ”œโ”€โ”€ pytorch_model.bin
+โ”‚ โ”‚ โ””โ”€โ”€ config.json
+โ”‚ โ”œโ”€โ”€ unet
+โ”‚ โ”‚ โ”œโ”€โ”€ diffusion_pytorch_model.bin
+โ”‚ โ”‚ โ””โ”€โ”€ config.json
+โ”‚ โ””โ”€โ”€ model_index.json
+โ””โ”€โ”€ tensorrt
+ โ””โ”€โ”€ unet_work.engine
+```
+
+### ๐ŸŽฎ Inference (Offline)
+```
+python inference_offline.py
+```
+### ๐Ÿ’ป Inference (Online)
+#### Installation
+```
+# install Node.js 18+
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
+nvm install 18
+
+cd webcam
+source start.sh
+```
+
+#### TensorRT (Optional)
+Converting the model to TensorRT can significantly speed up inference (~ 2$\times$). Building the engine may take about `20 minutes` depending on your device. Note that TensorRT optimizations may lead to slight variations or a small drop in output quality.
+```
+python torch2trt.py
+```
+
+#### Run
+```
+python inference_online.py
+```
+then open `http://0.0.0.0:7860` in your browser. (*If `http://0.0.0.0:7860` does not work well, try `http://localhost:7860`)
-<!-- ## ๐Ÿ’– Citation
+<!-- ## ๐Ÿ“‹ Citation
If you find PersonaLive useful for your research, welcome to ๐ŸŒŸ this repo and cite our work using the following BibTeX:
```bibtex
diff --git a/assets/demo_1.gif b/assets/demo_1.gif
index 240c54f..277c7b2 100644
Binary files a/assets/demo_1.gif and b/assets/demo_1.gif differ
diff --git a/assets/demo_2.gif b/assets/demo_2.gif
index 5debd77..5e646e5 100644
Binary files a/assets/demo_2.gif and b/assets/demo_2.gif differ
diff --git a/inference_online.py b/inference_online.py
index a426ed4..a317dbb 100644
--- a/inference_online.py
+++ b/inference_online.py
@@ -229,11 +229,11 @@ class App:
logging.error(f"Reference image error: {e}")
raise HTTPException(status_code=500, detail="Failed to process reference image")
- if not os.path.exists("./demo_w_camera/frontend/public"):
- os.makedirs("./demo_w_camera/frontend/public")
+ if not os.path.exists("./webcam/frontend/public"):
+ os.makedirs("./webcam/frontend/public")
self.app.mount(
- "/", StaticFiles(directory="./demo_w_camera/frontend/public", html=True), name="public"
+ "/", StaticFiles(directory="./webcam/frontend/public", html=True), name="public"
)
@self.app.on_event("shutdown")
diff --git a/requirements.txt b/requirements.txt
index 6acefc5..f24e2b1 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -28,4 +28,8 @@ tqdm==4.67.1
transformers==4.36.2
uvicorn==0.37.0
xformers==0.0.22.post7
-peft==0.10.0
\ No newline at end of file
+peft==0.10.0
+python-multipart==0.0.20
+uvicorn==0.37.0
+onnxruntime==1.16.3
+colored==2.3.1
\ No newline at end of file
diff --git a/torch2trt.py b/torch2trt.py
index 6d8f89c..aa58f43 100644
--- a/torch2trt.py
+++ b/torch2trt.py
@@ -28,7 +28,7 @@ device = torch.device("cuda:0")
device = map_device(device)
dtype = torch.float16
-config_path = './configs/prompts/personalive_trt.yaml'
+config_path = './configs/prompts/personalive_online.yaml'
cfg = OmegaConf.load(config_path)
onnx_path = cfg.onnx_path
onnx_opt_path = cfg.onnx_opt_path
diff --git a/webcam/util.py b/webcam/util.py
index 359f8b2..4008e38 100644
--- a/webcam/util.py
+++ b/webcam/util.py
@@ -70,7 +70,7 @@ def read_images_from_queue(queue, num_frames_needed, device, stop_event=None, pr
if prefer_latest:
read_size = queue.qsize()
else:
- read_size = min(queue.qsize(), num_frames_needed * 2)
+ read_size = min(queue.qsize(), num_frames_needed * 3)
images = []
for _ in range(read_size):
images.append(queue.get())
diff --git a/webcam/vid2vid.py b/webcam/vid2vid.py
index e8c784d..86e28dd 100644
--- a/webcam/vid2vid.py
+++ b/webcam/vid2vid.py
@@ -133,15 +133,9 @@ def generate_process(
clear_queue(input_queue)
restart_event.clear()
print("input_queue size = ", input_queue.qsize())
- strat_time = time.time()
images = read_images_from_queue(input_queue, chunk_size, device, stop_event)
images = torch.cat(images, dim=0)
- end_time = time.time()
- print('wait_time', end_time - strat_time)
- strat_time = time.time()
video = pipeline.process_input(images)
- end_time = time.time()
- print('process_time', end_time - strat_time)
for image in video:
output_queue.put(image)
\ No newline at end of file
diff --git a/webcam/vid2vid_trt.py b/webcam/vid2vid_trt.py
index 2c01eb4..a834b74 100644
--- a/webcam/vid2vid_trt.py
+++ b/webcam/vid2vid_trt.py
@@ -132,16 +132,9 @@ def generate_process(
if restart_event.is_set():
clear_queue(input_queue)
restart_event.clear()
- print("input_queue size = ", input_queue.qsize())
- strat_time = time.time()
images = read_images_from_queue(input_queue, chunk_size, device, stop_event)
images = torch.cat(images, dim=0)
- end_time = time.time()
- print('wait_time', end_time - strat_time)
- strat_time = time.time()
video = pipeline.process_input(images)
- end_time = time.time()
- print('process_time', end_time - strat_time)
for image in video:
output_queue.put(image)
\ No newline at end of file