[CVPR 2026] PersonaLive! : Expressive Portrait Image Animation for Live Streaming
update
Zhiyuan Li8398fa5c989f
diff --git a/README.md b/README.mdindex 42d3792..505bbac 100644--- a/README.md+++ b/README.md@@ -4,7 +4,6 @@<h2>Expressive Portrait Image Animation for Live Streaming</h2>-#### [Zhiyuan Li<sup>1,2,3</sup>](https://huai-chang.github.io/) ยท [Chi-Man Pun<sup>1</sup>](https://cmpun.github.io/) ๐ช ยท [Chen Fang<sup>2</sup>](http://fangchen.org/) ยท [Jue Wang<sup>2</sup>](https://scholar.google.com/citations?user=Bt4uDWMAAAAJ&hl=en) ยท [Xiaodong Cun<sup>3</sup>](https://vinthony.github.io/academic/) ๐ช<sup>1</sup> University of Macau <sup>2</sup> [Dzine.ai](https://www.dzine.ai/) <sup>3</sup> [GVC Lab, Great Bay University](https://gvclab.github.io/)@@ -14,16 +13,91 @@## ๐ฃ Updates- **[2025.12.10]** ๐ฅ Release `inference code`, `config` and `checkpoints`!-## ๐คช Framework+## โ๏ธ Framework<img src="assets/overview.png" alt="Image 1" width="100%">-<p align="justify">-We present PersonaLive, a few-step diffusion-based framework for real-time and streamable portrait animation, which achieves low-latency and stable long-term quality.-</p>+++We present PersonaLive, a `real-time` and `streamable` diffusion framework capable of generating `infinite-length` portrait animations on a single `12GB GPU`.## ๐ Getting Started+### ๐ Installation+```+# clone this repo+git clone https://github.com/GVCLab/PersonaLive+cd PersonaLive++# Create conda environment+conda create -n personalive python=3.10+conda activate personalive++# Install packages with pip+pip install -r requirements.txt+```++### โฌ Download weights+1. Download pre-trained weight of based models and other components ([SD V1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse) and [image_encoder](https://huggingface.co/lambdalabs/sd-image-variations-diffusers/tree/main/image_encoder))++1. Download [pre-trained models](https://drive.google.com/drive/folders/1GOhDBKIeowkMpBnKhGB8jgEhJt_--vbT?usp=drive_link) into the `./pretrained_weights` folder.++Finally, these weights should be orgnized as follows:+```+pretrained_weights+โโโ onnx+โ โโโ unet_opt+โ โ โโโ unet_opt.onnx+โ โ โโโ unet_opt.onnx.data+โ โโโ unet+โโโ personalive+โ โโโ denoising_unet.pth+โ โโโ motion_encoder.pth+โ โโโ motion_extractor.pth+โ โโโ pose_guider.pth+โ โโโ reference_unet.pth+โ โโโ temporal_module.pth+โโโ sd-vae-ft-mse+โ โโโ diffusion_pytorch_model.bin+โ โโโ config.json+โโโ sd-image-variations-diffusers+โ โโโ image_encoder+โ โ โโโ pytorch_model.bin+โ โ โโโ config.json+โ โโโ unet+โ โ โโโ diffusion_pytorch_model.bin+โ โ โโโ config.json+โ โโโ model_index.json+โโโ tensorrt+ โโโ unet_work.engine+```++### ๐ฎ Inference (Offline)+```+python inference_offline.py+```+### ๐ป Inference (Online)+#### Installation+```+# install Node.js 18++curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash+nvm install 18++cd webcam+source start.sh+```++#### TensorRT (Optional)+Converting the model to TensorRT can significantly speed up inference (~ 2$\times$). Building the engine may take about `20 minutes` depending on your device. Note that TensorRT optimizations may lead to slight variations or a small drop in output quality.+```+python torch2trt.py+```++#### Run+```+python inference_online.py+```+then open `http://0.0.0.0:7860` in your browser. (*If `http://0.0.0.0:7860` does not work well, try `http://localhost:7860`)-<!-- ## ๐ Citation+<!-- ## ๐ CitationIf you find PersonaLive useful for your research, welcome to ๐ this repo and cite our work using the following BibTeX:```bibtexdiff --git a/assets/demo_1.gif b/assets/demo_1.gifindex 240c54f..277c7b2 100644Binary files a/assets/demo_1.gif and b/assets/demo_1.gif differdiff --git a/assets/demo_2.gif b/assets/demo_2.gifindex 5debd77..5e646e5 100644Binary files a/assets/demo_2.gif and b/assets/demo_2.gif differdiff --git a/inference_online.py b/inference_online.pyindex a426ed4..a317dbb 100644--- a/inference_online.py+++ b/inference_online.py@@ -229,11 +229,11 @@ class App:logging.error(f"Reference image error: {e}")raise HTTPException(status_code=500, detail="Failed to process reference image")- if not os.path.exists("./demo_w_camera/frontend/public"):- os.makedirs("./demo_w_camera/frontend/public")+ if not os.path.exists("./webcam/frontend/public"):+ os.makedirs("./webcam/frontend/public")self.app.mount(- "/", StaticFiles(directory="./demo_w_camera/frontend/public", html=True), name="public"+ "/", StaticFiles(directory="./webcam/frontend/public", html=True), name="public")@self.app.on_event("shutdown")diff --git a/requirements.txt b/requirements.txtindex 6acefc5..f24e2b1 100644--- a/requirements.txt+++ b/requirements.txt@@ -28,4 +28,8 @@ tqdm==4.67.1transformers==4.36.2uvicorn==0.37.0xformers==0.0.22.post7-peft==0.10.0\ No newline at end of file+peft==0.10.0+python-multipart==0.0.20+uvicorn==0.37.0+onnxruntime==1.16.3+colored==2.3.1\ No newline at end of filediff --git a/torch2trt.py b/torch2trt.pyindex 6d8f89c..aa58f43 100644--- a/torch2trt.py+++ b/torch2trt.py@@ -28,7 +28,7 @@ device = torch.device("cuda:0")device = map_device(device)dtype = torch.float16-config_path = './configs/prompts/personalive_trt.yaml'+config_path = './configs/prompts/personalive_online.yaml'cfg = OmegaConf.load(config_path)onnx_path = cfg.onnx_pathonnx_opt_path = cfg.onnx_opt_pathdiff --git a/webcam/util.py b/webcam/util.pyindex 359f8b2..4008e38 100644--- a/webcam/util.py+++ b/webcam/util.py@@ -70,7 +70,7 @@ def read_images_from_queue(queue, num_frames_needed, device, stop_event=None, prif prefer_latest:read_size = queue.qsize()else:- read_size = min(queue.qsize(), num_frames_needed * 2)+ read_size = min(queue.qsize(), num_frames_needed * 3)images = []for _ in range(read_size):images.append(queue.get())diff --git a/webcam/vid2vid.py b/webcam/vid2vid.pyindex e8c784d..86e28dd 100644--- a/webcam/vid2vid.py+++ b/webcam/vid2vid.py@@ -133,15 +133,9 @@ def generate_process(clear_queue(input_queue)restart_event.clear()print("input_queue size = ", input_queue.qsize())- strat_time = time.time()images = read_images_from_queue(input_queue, chunk_size, device, stop_event)images = torch.cat(images, dim=0)- end_time = time.time()- print('wait_time', end_time - strat_time)- strat_time = time.time()video = pipeline.process_input(images)- end_time = time.time()- print('process_time', end_time - strat_time)for image in video:output_queue.put(image)\ No newline at end of filediff --git a/webcam/vid2vid_trt.py b/webcam/vid2vid_trt.pyindex 2c01eb4..a834b74 100644--- a/webcam/vid2vid_trt.py+++ b/webcam/vid2vid_trt.py@@ -132,16 +132,9 @@ def generate_process(if restart_event.is_set():clear_queue(input_queue)restart_event.clear()- print("input_queue size = ", input_queue.qsize())- strat_time = time.time()images = read_images_from_queue(input_queue, chunk_size, device, stop_event)images = torch.cat(images, dim=0)- end_time = time.time()- print('wait_time', end_time - strat_time)- strat_time = time.time()video = pipeline.process_input(images)- end_time = time.time()- print('process_time', end_time - strat_time)for image in video:output_queue.put(image)\ No newline at end of file