Skip to content

[CVPR 2026] PersonaLive! : Expressive Portrait Image Animation for Live Streaming

Upload training related code

Zhiyuan Lifbaa3ddd75ea
diff --git a/README.md b/README.md
index 8b819fc..f59168c 100644
--- a/README.md
+++ b/README.md
@@ -164,6 +164,130 @@ Then open `http://0.0.0.0:7860` in your browser. (*If `http://0.0.0.0:7860` does
1. Lower the "Driving FPS" setting in the WebUI to reduce the computational workload.
2. You can increase the multiplier (e.g., set to `num_frames_needed * 4` or higher) to better match your device's inference speed. https://github.com/GVCLab/PersonaLive/blob/6953d1a8b409f360a3ee1d7325093622b29f1e22/webcam/util.py#L73
+## 🚄 Model Training
+
+PersonaLive training is organized into three stages. Approximate training time on 8x H100 with default configs: Stage 1 ~13h, Stage 2 ~15h, Stage 3 ~20h.
+
+### 1️⃣ Environment setup
+Install base dependencies first (see installation section), then install training-only packages:
+
+```bash
+pip install -r requirements_train.txt
+```
+
+If you use multi-GPU or multi-node training, configure Accelerate once before launching training:
+
+```bash
+accelerate config
+```
+
+### 2️⃣ Data preparation
+Your dataset should contain a `videos` directory and a matching `boxes` directory:
+
+```text
+Datasets
+├── VFHQ
+│ ├── videos
+│ │ ├── example1.mp4
+│ │ ├── example2.mp4
+│ │ └── ...
+│ └── boxes
+│ ├── example1.pt
+│ ├── example2.pt
+│ └── ...
+└── ...
+```
+
+Preprocessing example:
+
+```bash
+# 1) Extract face / eye / mouth boxes from each frame
+python tools/get_boxes.py --video_dir ./Datasets/VFHQ/videos --save_dir ./Datasets/VFHQ/boxes --workers 8
+
+# 2) Generate meta json: [{"video_path": ".../videos/xxx.mp4"}, ...]
+python tools/extract_meta_info.py --root_path ./Datasets/VFHQ --dataset_name VFHQ
+```
+
+Then set `data.meta_paths` in each training config:
+
+```yaml
+data:
+ meta_paths:
+ - "./data/VFHQ_meta.json"
+ - "./data/OtherDataset_meta.json"
+```
+
+### 3️⃣ Download weights
+Download the training initialization weights: [X-NeMo](https://drive.google.com/drive/folders/1RdjBYYbstO7SOchDg7oimoAwu03g_-mI), [pose_guider](https://drive.google.com/drive/folders/1F6lyYfC5RijdS2SmvTct_8L9HZweV4nF?usp=sharing), and [stylegan2_discriminator](https://drive.google.com/drive/folders/1F6lyYfC5RijdS2SmvTct_8L9HZweV4nF?usp=sharing).
+
+```text
+pretrained_weights
+├── xnemo
+│ ├── xnemo_motion_encoder.pth
+│ ├── xnemo_denoising_unet.pth
+│ ├── xnemo_reference_unet.pth
+│ └── xnemo_temporal_module.pth
+├── sd-vae-ft-mse
+│ ├── diffusion_pytorch_model.bin
+│ └── config.json
+├── sd-image-variations-diffusers
+│ ├── image_encoder
+│ │ ├── pytorch_model.bin
+│ │ └── config.json
+│ ├── unet
+│ │ ├── diffusion_pytorch_model.bin
+│ │ └── config.json
+│ └── model_index.json
+├── pose_guider.pth
+└── stylegan2_discriminator_ffhq512.pth
+```
+
+### 4️⃣ Training workflow
+
+#### Stage 1: Image-level warm-up
+Run:
+```bash
+accelerate launch train_stage1.py --config ./configs/train/personalive_stage1.yaml
+```
+
+Default output folder: `./exp_output/personalive_stage1/`
+
+#### Stage 2: Image-level adversarial refinement
+
+Update `configs/train/personalive_stage2.yaml` to point to Stage 1 outputs:
+```bash
+motion_encoder_path: './exp_output/personalive_stage1/motion_encoder-xxxxx.pth'
+denoising_unet_path: './exp_output/personalive_stage1/denoising_unet-xxxxx.pth'
+reference_unet_path: './exp_output/personalive_stage1/reference_unet-xxxxx.pth'
+pose_guider_path: './exp_output/personalive_stage1/pose_guider-xxxxx.pth'
+```
+
+Run:
+```bash
+accelerate launch train_stage2.py --config ./configs/train/personalive_stage2.yaml
+```
+
+Default output folder: `./exp_output/personalive_stage2/`
+
+#### Stage 3: Temporal module fine-tuning for streaming
+
+Update `configs/train/personalive_stage3.yaml` to point to Stage 2 outputs:
+
+```bash
+motion_encoder_path: './exp_output/personalive_stage2/motion_encoder-xxxxx.pth'
+denoising_unet_path: './exp_output/personalive_stage2/denoising_unet-xxxxx.pth'
+reference_unet_path: './exp_output/personalive_stage2/reference_unet-xxxxx.pth'
+pose_guider_path: './exp_output/personalive_stage2/pose_guider-xxxxx.pth'
+discriminator_path: './exp_output/personalive_stage2/discriminator-xxxxx.pth'
+```
+
+Run:
+```bash
+accelerate launch train_stage3.py --config ./configs/train/personalive_stage3.yaml
+```
+
+Default output folder: `./exp_output/personalive_stage3/`
+
## 📚 Community Contribution
Special thanks to the community for providing helpful setups! 🥂
diff --git a/configs/inference/inference_stage1&2.yaml b/configs/inference/inference_stage1&2.yaml
new file mode 100644
index 0000000..e351e5a
--- /dev/null
+++ b/configs/inference/inference_stage1&2.yaml
@@ -0,0 +1,47 @@
+unet_additional_kwargs:
+ use_inflated_groupnorm: true
+ unet_use_cross_frame_attention: false
+ unet_use_temporal_attention: false
+ use_motion_module: true
+ motion_module_resolutions:
+ - 1
+ - 2
+ - 4
+ - 8
+ motion_module_mid_block: true
+ motion_module_decoder_only: false
+ motion_module_type: Vanilla
+ motion_module_kwargs:
+ num_attention_heads: 8
+ num_transformer_block: 1
+ cross_attention_dim: 16
+ attention_block_types:
+ - Spatial_Cross
+ - Spatial_Cross
+ temporal_position_encoding: false
+ temporal_position_encoding_max_len: 32
+ temporal_attention_dim_div: 1
+
+ use_temporal_module: false
+ temporal_module_type: Vanilla
+ temporal_module_kwargs:
+ num_attention_heads: 8
+ num_transformer_block: 1
+ attention_block_types:
+ - Temporal_Self
+ - Temporal_Self
+ temporal_position_encoding: true
+ temporal_position_encoding_max_len: 32
+ temporal_attention_dim_div: 1
+
+
+noise_scheduler_kwargs:
+ beta_start: 0.00085
+ beta_end: 0.02
+ beta_schedule: "scaled_linear"
+ clip_sample: false
+ steps_offset: 1
+ prediction_type: "epsilon"
+ timestep_spacing: "trailing"
+
+sampler: DDIM
\ No newline at end of file
diff --git a/configs/train/personalive_stage1.yaml b/configs/train/personalive_stage1.yaml
new file mode 100644
index 0000000..83bd573
--- /dev/null
+++ b/configs/train/personalive_stage1.yaml
@@ -0,0 +1,53 @@
+data:
+ train_bs: 8
+ train_width: 512
+ train_height: 512
+ meta_paths:
+ - "./data/VFHQ_example.json"
+ sample_margin: 30
+
+solver:
+ gradient_accumulation_steps: 1
+ mixed_precision: 'fp16'
+ enable_xformers_memory_efficient_attention: True
+ gradient_checkpointing: False
+ max_train_steps: 50000
+ max_grad_norm: 1.0
+ # lr
+ learning_rate: 1e-5
+ scale_lr: False
+ lr_warmup_steps: 1
+ lr_scheduler: 'constant'
+
+ # optimizer
+ use_8bit_adam: False
+ adam_beta1: 0.9
+ adam_beta2: 0.999
+ adam_weight_decay: 1.0e-2
+ adam_epsilon: 1.0e-8
+
+val:
+ validation_steps: 1000
+
+base_model_path: './pretrained_weights/sd-image-variations-diffusers'
+vae_model_path: './pretrained_weights/sd-vae-ft-mse'
+image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'
+pose_encoder_path: './pretrained_weights/liveportrait/base_models/motion_extractor.pth'
+
+motion_encoder_path: './pretrained_weights/xnemo/xnemo_motion_encoder.pth'
+denoising_unet_path: './pretrained_weights/xnemo/xnemo_denoising_unet.pth'
+reference_unet_path: './pretrained_weights/xnemo/xnemo_reference_unet.pth'
+pose_guider_path: './pretrained_weights/pose_guider.pth'
+
+weight_dtype: 'fp16' # [fp16, fp32]
+uncond_ratio: 0.1
+noise_offset: 0.05
+window_size: 10
+snr_gamma: 5.0
+
+seed: 12580
+resume_from_checkpoint: ''
+checkpointing_steps: 10000
+save_model_epoch_interval: 5
+exp_name: 'personalive_stage1'
+output_dir: './exp_output'
\ No newline at end of file
diff --git a/configs/train/personalive_stage2.yaml b/configs/train/personalive_stage2.yaml
new file mode 100644
index 0000000..84f55f9
--- /dev/null
+++ b/configs/train/personalive_stage2.yaml
@@ -0,0 +1,56 @@
+data:
+ train_bs: 4
+ train_width: 512
+ train_height: 512
+ meta_paths:
+ - "./data/VFHQ_example.json"
+ sample_margin: 30
+
+solver:
+ gradient_accumulation_steps: 1
+ mixed_precision: 'fp16'
+ enable_xformers_memory_efficient_attention: True
+ gradient_checkpointing: False
+ max_train_steps: 50000
+ max_grad_norm: 1.0
+ # lr
+ learning_rate: 1e-5
+ scale_lr: False
+ lr_warmup_steps: 1
+ lr_scheduler: 'constant'
+
+ # optimizer
+ use_8bit_adam: True
+ adam_beta1: 0.9
+ adam_beta2: 0.999
+ adam_weight_decay: 1.0e-2
+ adam_epsilon: 1.0e-8
+
+val:
+ validation_steps: 1000
+
+base_model_path: './pretrained_weights/sd-image-variations-diffusers'
+vae_model_path: './pretrained_weights/sd-vae-ft-mse'
+image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'
+pose_encoder_path: './pretrained_weights/liveportrait/base_models/motion_extractor.pth'
+
+motion_encoder_path: './exp_output/personalive_stage1/motion_encoder-xxxxx.pth'
+denoising_unet_path: './exp_output/personalive_stage1/denoising_unet-xxxxx.pth'
+reference_unet_path: './exp_output/personalive_stage1/reference_unet-xxxxx.pth'
+pose_guider_path: './exp_output/personalive_stage1/pose_guider-xxxxx.pth'
+
+discriminator_path: './pretrained_weights/stylegan2_discriminator_ffhq512.pth'
+
+weight_dtype: 'fp16' # [fp16, fp32]
+uncond_ratio: 0.1
+noise_offset: 0.05
+window_size: 10
+snr_gamma: 5.0
+timesteps_list: [999, 666, 333, 0]
+
+seed: 12580
+resume_from_checkpoint: ''
+checkpointing_steps: 10000
+save_model_epoch_interval: 3
+exp_name: 'personalive_stage2'
+output_dir: './exp_output'
\ No newline at end of file
diff --git a/configs/train/personalive_stage3.yaml b/configs/train/personalive_stage3.yaml
new file mode 100644
index 0000000..3fa7583
--- /dev/null
+++ b/configs/train/personalive_stage3.yaml
@@ -0,0 +1,62 @@
+data:
+ train_bs: 1
+ train_width: 512
+ train_height: 512
+ meta_paths:
+ - "./data/VFHQ_example.json"
+ sample_rate: 4
+ n_sample_frames: 40
+ max_batch_frames: 10
+
+solver:
+ gradient_accumulation_steps: 1
+ mixed_precision: 'fp16'
+ enable_xformers_memory_efficient_attention: True
+ gradient_checkpointing: False
+ max_train_steps: 10000
+ max_grad_norm: 1.0
+ # lr
+ learning_rate: 1e-5
+ scale_lr: False
+ lr_warmup_steps: 1
+ lr_scheduler: 'constant'
+
+ # optimizer
+ use_8bit_adam: True
+ adam_beta1: 0.9
+ adam_beta2: 0.999
+ adam_weight_decay: 1.0e-2
+ adam_epsilon: 1.0e-8
+
+val:
+ validation_steps: 200
+
+temporal_window_size: 4
+temporal_adaptive_step: 4
+
+base_model_path: './pretrained_weights/sd-image-variations-diffusers'
+vae_model_path: './pretrained_weights/sd-vae-ft-mse'
+image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'
+pose_encoder_path: './pretrained_weights/liveportrait/base_models/motion_extractor.pth'
+
+motion_encoder_path: './exp_output/personalive_stage2/motion_encoder-xxxxx.pth'
+denoising_unet_path: './exp_output/personalive_stage2/denoising_unet-xxxxx.pth'
+reference_unet_path: './exp_output/personalive_stage2/reference_unet-xxxxx.pth'
+pose_guider_path: './exp_output/personalive_stage2/pose_guider-xxxxx.pth'
+discriminator_path: './exp_output/personalive_stage2/discriminator-xxxxx.pth'
+
+temporal_module_path: './pretrained_weights/xnemo/xnemo_temporal_module.pth'
+
+weight_dtype: 'fp16' # [fp16, fp32]
+uncond_ratio: 0.1
+noise_offset: 0.05
+window_size: 10
+snr_gamma: 5.0
+timesteps_list: [999, 666, 333, 0]
+
+seed: 12580
+resume_from_checkpoint: ''
+checkpointing_steps: 1000
+save_model_epoch_interval: 1
+exp_name: 'personalive_stage3'
+output_dir: './exp_output'
\ No newline at end of file
diff --git a/data/VFHQ_example.json b/data/VFHQ_example.json
new file mode 100644
index 0000000..db2f208
--- /dev/null
+++ b/data/VFHQ_example.json
@@ -0,0 +1,7 @@
+[
+{"video_path": "Datasets/videos/VFHQ/Clip+SWLlH7mCWs8+P2+C1+F5731-5850.mp4"},
+{"video_path": "Datasets/videos/VFHQ/Clip+yVmVOdc9glA+P2+C2+F56876-56997.mp4"},
+{"video_path": "Datasets/videos/VFHQ/Clip+tThJIOTpmP8+P0+C1+F115-418.mp4"},
+{"video_path": "Datasets/videos/VFHQ/Clip+rWOlJ_tyNB8+P0+C1+F2870-3025.mp4"},
+{"video_path": "Datasets/videos/VFHQ/Clip+0twLK4diLu8+P0+C1+F1094-1220.mp4"}
+]
\ No newline at end of file
diff --git a/requirements_base.txt b/requirements_base.txt
index 1c72db4..6c20240 100644
--- a/requirements_base.txt
+++ b/requirements_base.txt
@@ -6,12 +6,11 @@ decord==0.6.0
diffusers==0.27.0
einops==0.8.1
fastapi==0.117.1
-huggingface-hub==0.25.1
+huggingface_hub==0.25.1
mediapipe==0.10.11
markdown2==2.5.4
numpy==1.26.3
omegaconf==2.3.0
-opencv_contrib_python==4.11.0.86
opencv_python_headless==4.10.0.84
Pillow==11.3.0
polygraphy==0.49.26
diff --git a/requirements_train.txt b/requirements_train.txt
new file mode 100644
index 0000000..34533be
--- /dev/null
+++ b/requirements_train.txt
@@ -0,0 +1,10 @@
+albumentations==1.4.12
+albucore==0.0.16
+bitsandbytes==0.44.1
+lpips==0.1.4
+mlflow==3.1.4
+mlflow_skinny==3.1.4
+numpy==1.26.3
+requests==2.32.5
+protobuf==3.20.3
+setuptools==68.2.2
\ No newline at end of file
diff --git a/src/dataset/face_augmentor.py b/src/dataset/face_augmentor.py
new file mode 100644
index 0000000..5842e63
--- /dev/null
+++ b/src/dataset/face_augmentor.py
@@ -0,0 +1,46 @@
+import random
+import os
+os.environ["NO_ALBUMENTATIONS_UPDATE"] = "1"
+import numpy as np
+import albumentations as A
+import torchvision.transforms.functional as TF
+import torch
+
+class FaceAugmentor:
+ def __init__(self):
+ self.post_aug = A.Compose([
+ A.ColorJitter(brightness=(0.3, 1.3), contrast=0.3, saturation=0.3, hue=0.3, p=1.0),
+ A.PiecewiseAffine(scale=(0.02, 0.04), p=1.0),
+ A.GaussNoise(p=1),
+ ])
+
+ def random_aspect_resize(self, img, flag=None, scale=None):
+ img = torch.from_numpy(img).permute(2, 0, 1)
+ # img: torch.Tensor [C,H,W]
+ H, W = img.shape[-2:]
+ if flag is None:
+ flag = random.random()
+ if scale is None:
+ scale = random.uniform(1.0, 1.3)
+
+ if flag < 0.5:
+ scale_x = scale
+ scale_y = 1.0
+ else:
+ scale_x = 1.0
+ scale_y = scale
+
+ new_W, new_H = int(W * scale_x), int(H * scale_y)
+ img_resized = TF.resize(img, (new_H, new_W), antialias=True)
+ # 中心裁剪/填充回原尺寸
+ img_final = TF.center_crop(img_resized, (H, W))
+ return img_final.permute(1,2,0).numpy() # [H,W,C]
+
+ def __call__(self, img, random_size=True, flag=None, scale=None): # img: numpy RGB [H, W, 3]
+ h, w = img.shape[:2]
+ img_aug = img.copy()
+
+ if random_size:
+ img_aug = self.random_aspect_resize(img_aug, flag, scale)
+
+ return self.post_aug(image=img_aug)["image"]
\ No newline at end of file
diff --git a/src/dataset/portrait_image.py b/src/dataset/portrait_image.py
new file mode 100644
index 0000000..784e25b
--- /dev/null
+++ b/src/dataset/portrait_image.py
@@ -0,0 +1,168 @@
+import json
+import os
+import random
+from typing import List
+
+import numpy as np
+import torch
+import torchvision.transforms as transforms
+from decord import VideoReader
+from PIL import Image
+from torch.utils.data import Dataset
+from transformers import CLIPImageProcessor
+from pathlib import Path
+from .utils import (
+ crop_square_containing_face_patch,
+ get_bbox_param,
+ get_mask,
+)
+from .face_augmentor import FaceAugmentor
+
+class PortraitImageDataset(Dataset):
+ def __init__(
+ self,
+ img_size,
+ img_scale=(1.0, 1.0),
+ img_ratio=(0.9, 1.0),
+ data_meta_paths=["./data/fahsion_meta.json"],
+ sample_margin=30,
+ ):
+ super().__init__()
+
+ self.img_size = img_size
+ self.img_scale = img_scale
+ self.img_ratio = img_ratio
+ self.sample_margin = sample_margin
+
+ vid_meta = []
+ for data_meta_path in data_meta_paths:
+ vid_meta.extend(json.load(open(data_meta_path, "r")))
+ self.vid_meta = vid_meta
+
+ self.clip_image_processor = CLIPImageProcessor()
+
+ self.transform = transforms.Compose(
+ [
+ transforms.Resize(
+ self.img_size,
+ interpolation=transforms.InterpolationMode.BILINEAR,
+ ),
+ transforms.ToTensor(),
+ transforms.Normalize([0.5], [0.5]),
+ ]
+ )
+
+ self.expression_transform = transforms.Compose(
+ [
+ transforms.Resize(
+ (224, 224),
+ interpolation=transforms.InterpolationMode.BILINEAR,
+ ),
+ transforms.ToTensor(),
+ transforms.Normalize([0.5], [0.5]),
+ ]
+ )
+
+ self.cond_transform = transforms.Compose(
+ [
+ transforms.Resize(
+ (256, 256),
+ interpolation=transforms.InterpolationMode.BILINEAR,
+ ),
+ transforms.ToTensor(),
+ ]
+ )
+
+ self.mask_transform = transforms.Compose(
+ [
+ transforms.Resize(
+ self.img_size,
+ interpolation=transforms.InterpolationMode.NEAREST,
+ ),
+ transforms.ToTensor(),
+ ]
+ )
+
+
+ self.face_augmentor = FaceAugmentor()
+
+ def augmentation(self, image, transform, state=None):
+ if state is not None:
+ torch.set_rng_state(state)
+ return transform(image)
+
+ def __getitem__(self, index):
+ try:
+ video_meta = self.vid_meta[index]
+ video_path = video_meta["video_path"]
+
+
+ kps_path = video_path.replace("videos", "boxes").replace(".mp4", ".pt")
+ video_reader = VideoReader(video_path)
+ landmarks = torch.load(kps_path)
+
+ video_length = min(len(video_reader), len(landmarks))
+
+ margin = self.sample_margin
+ ref_img_idx = random.randint(0, video_length - 1)
+ if video_length > 2 * margin + 4:
+ left = max(0, ref_img_idx - margin)
+ right = min(video_length, ref_img_idx + margin)
+
+ # 拼接两个不重叠的区间
+ part = list(range(0, left)) + list(range(right, video_length))
+ tgt_img_idx = random.choice(part)
+ else:
+ tgt_img_idx = random.randint(0, video_length - 1)
+
+ ref_img = video_reader[ref_img_idx]
+ tgt_img = video_reader[tgt_img_idx]
+ ref_face_box = landmarks[ref_img_idx]['face']
+ tgt_face_box = landmarks[tgt_img_idx]['face']
+ tgt_left_eye_box = landmarks[tgt_img_idx]['left_eye']
+ tgt_right_eye_box = landmarks[tgt_img_idx]['right_eye']
+ tgt_mouth_box = landmarks[tgt_img_idx]['mouth']
+
+ tgt_face_patch = crop_square_containing_face_patch(tgt_img.asnumpy(), tgt_face_box)
+ bbox_params = torch.from_numpy(get_bbox_param(tgt_face_box, ref_face_box))
+
+ face_mask, local_mask = get_mask(tgt_face_box, tgt_left_eye_box, tgt_right_eye_box, tgt_mouth_box, side=512)
+
+ tgt_face_patch = self.face_augmentor(tgt_face_patch)
+
+ ref_img_pil = Image.fromarray(ref_img.asnumpy())
+ tgt_img_pil = Image.fromarray(tgt_img.asnumpy())
+ tgt_face_pil = Image.fromarray(tgt_face_patch)
+ face_mask = Image.fromarray(face_mask)
+ local_mask = Image.fromarray(local_mask)
+
+ state = torch.get_rng_state()
+ tgt_img = self.augmentation(tgt_img_pil, self.transform, state=state)
+ ref_img = self.augmentation(ref_img_pil, self.transform, state=state)
+ tgt_face = self.augmentation(tgt_face_pil, self.expression_transform, state=state)
+ tgt_pose = self.augmentation(tgt_img_pil, self.cond_transform, state=state)
+ face_mask = self.augmentation(face_mask, self.mask_transform, state=state)
+ local_mask = self.augmentation(local_mask, self.mask_transform, state=state)
+
+ clip_image = self.clip_image_processor(
+ images=ref_img_pil, return_tensors="pt"
+ ).pixel_values[0]
+
+ sample = dict(
+ video_dir=video_path,
+ img=tgt_img,
+ ref_img=ref_img,
+ tgt_face=tgt_face,
+ tgt_pose=tgt_pose,
+ bbox_params=bbox_params,
+ clip_image=clip_image,
+ face_mask=face_mask,
+ local_mask=local_mask,
+ )
+ return sample
+ except Exception as e:
+ next_index = random.randint(0, len(self.vid_meta)-1)
+ return self.__getitem__(next_index)
+
+ def __len__(self):
+ return len(self.vid_meta)
\ No newline at end of file
diff --git a/src/dataset/portrait_video.py b/src/dataset/portrait_video.py
new file mode 100644
index 0000000..be0425b
--- /dev/null
+++ b/src/dataset/portrait_video.py
@@ -0,0 +1,184 @@
+import json
+import os
+import random
+from typing import List
+
+import numpy as np
+import torch
+import torchvision.transforms as transforms
+from decord import VideoReader
+from PIL import Image
+from torch.utils.data import Dataset
+from transformers import CLIPImageProcessor
+from pathlib import Path
+from .utils import (
+ crop_square_containing_face_patch,
+ get_bbox_param,
+ get_mask,
+)
+import gc
+from .face_augmentor import FaceAugmentor
+import torchvision.transforms.functional as TF
+
+class PortraitVideoDataset(Dataset):
+ def __init__(
+ self,
+ sample_rate,
+ n_sample_frames,
+ img_size,
+ img_scale=(1.0, 1.0),
+ img_ratio=(0.9, 1.0),
+ data_meta_paths=["./data/fahsion_meta.json"],
+ ):
+ super().__init__()
+
+ self.img_size = img_size
+ self.img_scale = img_scale
+ self.img_ratio = img_ratio
+ self.sample_rate = sample_rate
+ self.n_sample_frames = n_sample_frames
+
+ vid_meta = []
+ for data_meta_path in data_meta_paths:
+ vid_meta.extend(json.load(open(data_meta_path, "r")))
+ self.vid_meta = vid_meta
+
+ self.clip_image_processor = CLIPImageProcessor()
+
+ self.transform = transforms.Compose(
+ [
+ transforms.Resize(
+ self.img_size,
+ interpolation=transforms.InterpolationMode.BILINEAR,
+ ),
+ transforms.ToTensor(),
+ transforms.Normalize([0.5], [0.5]),
+ ]
+ )
+
+ self.expression_transform = transforms.Compose(
+ [
+ transforms.Resize(
+ (224, 224),
+ interpolation=transforms.InterpolationMode.BILINEAR,
+ ),
+ transforms.ToTensor(),
+ transforms.Normalize([0.5], [0.5]),
+ ]
+ )
+
+ self.mask_transform = transforms.Compose(
+ [
+ transforms.Resize(
+ self.img_size,
+ interpolation=transforms.InterpolationMode.NEAREST,
+ ),
+ transforms.ToTensor(),
+ ]
+ )
+
+ self.pose_transform = transforms.Compose(
+ [
+ transforms.Resize(
+ (256, 256),
+ interpolation=transforms.InterpolationMode.BILINEAR,
+ ),
+ transforms.ToTensor(),
+ ]
+ )
+
+ self.face_augmentor = FaceAugmentor()
+
+ def augmentation(self, images, transform, state=None):
+ if state is not None:
+ torch.set_rng_state(state)
+ if isinstance(images, List):
+ transformed_images = [transform(img) for img in images]
+ ret_tensor = torch.stack(transformed_images, dim=0) # (f, c, h, w)
+ else:
+ ret_tensor = transform(images) # (c, h, w)
+ return ret_tensor
+
+ def __getitem__(self, index):
+ try:
+ video_meta = self.vid_meta[index]
+ video_path = video_meta["video_path"]
+
+
+ kps_path = video_path.replace("videos", "boxes").replace(".mp4", ".pt")
+ video_reader = VideoReader(video_path)
+ landmarks = torch.load(kps_path)
+
+ video_length = min(len(video_reader), len(landmarks))
+
+ ref_img_idx = random.randint(0, video_length - 1)
+ sample_rate = self.sample_rate
+ clip_length = min(video_length, (self.n_sample_frames - 1) * sample_rate + 1)
+ start_idx = random.randint(0, video_length - clip_length)
+ batch_index = np.linspace(start_idx, start_idx + clip_length - 1, self.n_sample_frames, dtype=int).tolist()
+
+ ref_img = video_reader[ref_img_idx]
+ ref_face_box = landmarks[ref_img_idx]['face']
+
+ tgt_image_list = []
+ tgt_face_list = []
+ tgt_face_mask_list = []
+ tgt_local_mask_list = []
+ bbox_params = []
+
+ face_flag = random.random()
+ face_scale = random.uniform(1.0, 1.2)
+
+ for tgt_img_idx in batch_index:
+ tgt_img = video_reader[tgt_img_idx]
+ tgt_face_box = landmarks[tgt_img_idx]['face']
+ tgt_left_eye_box = landmarks[tgt_img_idx]['left_eye']
+ tgt_right_eye_box = landmarks[tgt_img_idx]['right_eye']
+ tgt_mouth_box = landmarks[tgt_img_idx]['mouth']
+
+ tgt_face_patch = crop_square_containing_face_patch(tgt_img.asnumpy(), tgt_face_box)
+ face_mask, local_mask = get_mask(tgt_face_box, tgt_left_eye_box, tgt_right_eye_box, tgt_mouth_box, side=512)
+ tgt_face_patch = self.face_augmentor(tgt_face_patch, flag=face_flag, scale=face_scale)
+
+ bbox_param = torch.from_numpy(get_bbox_param(tgt_face_box, ref_face_box))
+ bbox_params.append(bbox_param)
+
+ tgt_image_list.append(Image.fromarray(tgt_img.asnumpy()))
+ tgt_face_list.append(Image.fromarray(tgt_face_patch))
+ tgt_face_mask_list.append(Image.fromarray(face_mask))
+ tgt_local_mask_list.append(Image.fromarray(local_mask))
+
+ bbox_params = torch.stack(bbox_params, dim=0)
+ ref_img_pil = Image.fromarray(ref_img.asnumpy())
+
+ state = torch.get_rng_state()
+ tgt_img = self.augmentation(tgt_image_list, self.transform, state=state)
+ ref_img = self.augmentation(ref_img_pil, self.transform, state=state)
+ tgt_face = self.augmentation(tgt_face_list, self.expression_transform, state=state)
+ face_mask = self.augmentation(tgt_face_mask_list, self.mask_transform, state=state)
+ local_mask = self.augmentation(tgt_local_mask_list, self.mask_transform, state=state)
+
+ tgt_pose = self.augmentation(tgt_image_list, self.pose_transform, state=state)
+
+ clip_image = self.clip_image_processor(
+ images=ref_img_pil, return_tensors="pt"
+ ).pixel_values[0]
+
+ sample = dict(
+ video_dir=video_path,
+ img=tgt_img,
+ ref_img=ref_img,
+ tgt_face=tgt_face,
+ tgt_pose=tgt_pose,
+ bbox_params=bbox_params,
+ clip_image=clip_image,
+ face_mask=face_mask,
+ local_mask=local_mask,
+ )
+ return sample
+ except Exception as e:
+ next_index = random.randint(0, len(self.vid_meta)-1)
+ return self.__getitem__(next_index)
+
+ def __len__(self):
+ return len(self.vid_meta)
\ No newline at end of file
diff --git a/src/dataset/utils.py b/src/dataset/utils.py
new file mode 100644
index 0000000..ea16951
--- /dev/null
+++ b/src/dataset/utils.py
@@ -0,0 +1,79 @@
+import numpy as np
+import cv2
+
+def crop_square_containing_face_patch(image, face_box):
+ h, w = image.shape[:2]
+ x_min, y_min, x_max, y_max = face_box
+
+ # 人脸中心和尺寸
+ face_cx = (x_min + x_max) // 2
+ face_cy = (y_min + y_max) // 2
+
+ face_w = x_max - x_min
+ face_h = y_max - y_min
+ new_w = new_h = max(face_w, face_h)
+
+ new_x_min = int(round(face_cx - new_w / 2))
+ new_x_max = int(round(face_cx + new_w / 2))
+ new_y_min = int(round(face_cy - new_h / 2))
+ new_y_max = int(round(face_cy + new_h / 2))
+
+ new_x_min = max(0, new_x_min)
+ new_y_min = max(0, new_y_min)
+ new_x_max = min(w, new_x_max)
+ new_y_max = min(h, new_y_max)
+
+ face_patch = image[new_y_min:new_y_max, new_x_min:new_x_max]
+
+ return face_patch
+
+def check_oob(bbox, size):
+ left, top, right, bot = bbox
+ return left < 0 or top < 0 or right > size[1] - 1 or bot > size[0] - 1
+
+def scale_bb(bbox, scale, size):
+ left, top, right, bot = bbox
+ width = right - left
+ height = bot - top
+ length = max(width, height) * scale
+ center_X = (left + right) * 0.5
+ center_Y = (top + bot) * 0.5
+ left, top, right, bot = [
+ center_X - length / 2,
+ center_Y - length / 2,
+ center_X + length / 2,
+ center_Y + length / 2,
+ ]
+ if check_oob((left, top, right, bot), size):
+ return bbox
+ else:
+ return np.array([left, top, right, bot])
+
+def get_bbox_param(bbox, ref_bbox):
+ left, top, right, bot = bbox
+ center = np.array([(bot + top) * 0.5, (left + right) * 0.5])
+ length = max(right - left, bot - top)
+
+ ref_left, ref_top, ref_right, ref_bot = ref_bbox
+ ref_center = np.array([(ref_bot + ref_top) * 0.5, (ref_left + ref_right) * 0.5])
+ ref_length = max(ref_right - ref_left, ref_bot - ref_top) # 最好保证bbox是正方形
+
+ return np.asarray(
+ ((center - ref_center) / ref_length).tolist() + [length / ref_length]
+ )
+
+def draw_box(box, mask, value=255):
+ x1, y1, x2, y2 = box
+ mask = cv2.rectangle(mask, (x1, y1), (x2, y2), value, -1)
+ return mask
+
+def get_mask(face_box, left_eye_box, right_eye_box, mouth_box, side):
+ face_mask = np.zeros((side, side), dtype=np.uint8)
+ local_mask = np.zeros((side, side), dtype=np.uint8)
+
+ draw_box(face_box, face_mask)
+ draw_box(left_eye_box, local_mask)
+ draw_box(right_eye_box, local_mask)
+ draw_box(mouth_box, local_mask)
+ return face_mask, local_mask
+
diff --git a/src/models/motion_encoder/FAN_temporal_feature_extractor.py b/src/models/motion_encoder/FAN_temporal_feature_extractor.py
index 01676dd..4baf1b6 100644
--- a/src/models/motion_encoder/FAN_temporal_feature_extractor.py
+++ b/src/models/motion_encoder/FAN_temporal_feature_extractor.py
@@ -173,7 +173,7 @@ class TemporalTransformer3DModel(nn.Module):
interpolation_scale=1,
)
# 在转onnx模型时,需要设置为False
- self.pos_embed.pos_embed.requires_grad = False
+ self.pos_embed.pos_embed.requires_grad = True
self.proj_out = zero_module(self.proj_out) if zero_initialize else random_module(self.proj_out)
def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, skip=True):
diff --git a/src/models/motion_encoder/encoder.py b/src/models/motion_encoder/encoder.py
index 55efbcd..c4d7a11 100644
--- a/src/models/motion_encoder/encoder.py
+++ b/src/models/motion_encoder/encoder.py
@@ -34,8 +34,16 @@ class MotEncoder(ModelMixin):
def forward(self, x):
x = x.to(self.dtype)
- latent = self.model(rearrange(x, "b c f h w -> (b f) c h w"))
+ if x.ndim == 5:
+ latent = self.model(rearrange(x, "b c f h w -> (b f) c h w"))
+ else:
+ latent = self.model(x)
+ if self.out_bn is not None:
+ latent = self.out_bn(latent.unsqueeze(-1)).squeeze(-1)
+ if self.out_drop is not None:
+ latent = self.out_drop(latent)
latent = self.final_proj(latent)
latent = rearrange(latent, "b (l c) -> b l c", c=self.out_ch) + self.pe
- latent = rearrange(latent, "(b f) l c -> b f l c", f=x.shape[2])
+ if x.ndim == 5:
+ latent = rearrange(latent, "(b f) l c -> b f l c", f=x.shape[2])
return latent
\ No newline at end of file
diff --git a/src/models/mutual_self_attention.py b/src/models/mutual_self_attention.py
index 81781a7..bbb6868 100644
--- a/src/models/mutual_self_attention.py
+++ b/src/models/mutual_self_attention.py
@@ -149,12 +149,6 @@ class ReferenceAttentionControl:
**cross_attention_kwargs,
)
if MODE == "read":
- kv_cache = norm_hidden_states.clone()
- kv_cache = rearrange(
- kv_cache, "(b t) l c -> b t l c", t=video_length
- )
- self.kv_cache = kv_cache[:,:2,:,:]
-
bank_fea = [
rearrange(
d.unsqueeze(1).repeat(1, video_length, 1, 1),
@@ -326,9 +320,7 @@ class ReferenceAttentionControl:
)
module.bank = []
- if(self.cache_kv):
- module.kv_bank = None
- module.kv_cache = None
+ module.kv_bank = None
module.attn_weight = float(i) / float(len(attn_modules))
def update(self, writer, dtype=torch.float16, drop_ratio=0.):
@@ -461,5 +453,4 @@ class ReferenceAttentionControl:
for r in reader_attn_modules:
r.bank.clear()
if self.cache_kv:
- r.kv_bank=None
- r.kv_cache=None
\ No newline at end of file
+ r.kv_bank=None
\ No newline at end of file
diff --git a/src/models/resnet.py b/src/models/resnet.py
index ea73144..326f0b1 100644
--- a/src/models/resnet.py
+++ b/src/models/resnet.py
@@ -224,10 +224,10 @@ class ResnetBlock3D(nn.Module):
if temb is not None:
video_length = hidden_states.shape[2]
temb = self.time_emb_proj(self.nonlinearity(temb))
- temb = rearrange(temb, "(b f) c -> b c f", f=video_length)[:, :, :, None, None]
-
- # if temb is not None:
- # temb = self.time_emb_proj(self.nonlinearity(temb))[:, :, None, None, None]
+ if temb.shape[0] == hidden_states.shape[0] * video_length:
+ temb = rearrange(temb, "(b f) c -> b c f", f=video_length)[:, :, :, None, None]
+ else:
+ temb = temb[:, :, None, None, None]
if temb is not None and self.time_embedding_norm == "default":
hidden_states = hidden_states + temb
diff --git a/src/pipelines/pipeline_pose2img.py b/src/pipelines/pipeline_pose2img.py
new file mode 100644
index 0000000..07977ca
--- /dev/null
+++ b/src/pipelines/pipeline_pose2img.py
@@ -0,0 +1,669 @@
+# *************************************************************************
+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
+# SPDX-License-Identifier: Apache-2.0
+#
+# This file has been modified by ByteDance Ltd. and/or its affiliates.
+#
+# Original file was released under Aniportrait, with the full license text
+# available at https://github.com/Zejun-Yang/AniPortrait/blob/main/LICENSE.
+#
+# This modified file is released under the same license.
+# *************************************************************************
+import inspect
+import math
+from dataclasses import dataclass
+from typing import Callable, List, Optional, Union
+import random
+import numpy as np
+import torch
+import torch.nn.functional as F
+import torchvision.transforms as transforms
+from diffusers import DiffusionPipeline
+from diffusers import (
+ AutoencoderKL,
+ AutoencoderKLTemporalDecoder,
+)
+from diffusers.image_processor import VaeImageProcessor
+from diffusers.schedulers import (
+ DDIMScheduler,
+ DPMSolverMultistepScheduler,
+ EulerAncestralDiscreteScheduler,
+ EulerDiscreteScheduler,
+ LMSDiscreteScheduler,
+ PNDMScheduler,
+)
+from diffusers.utils import BaseOutput, is_accelerate_available
+from diffusers.utils.torch_utils import randn_tensor
+from einops import rearrange
+from tqdm import tqdm
+from transformers import CLIPImageProcessor
+from src.pipelines.context import get_context_scheduler
+from src.pipelines.utils import get_tensor_interpolation_method
+from src.models.mutual_self_attention import ReferenceAttentionControl
+from src.utils.util import draw_keypoints
+
+@dataclass
+class Pose2VideoPipelineOutput(BaseOutput):
+ videos: Union[torch.Tensor, np.ndarray]
+
+
+class Pose2ImagePipeline(DiffusionPipeline):
+ _optional_components = []
+
+ def __init__(
+ self,
+ vae,
+ image_encoder,
+ reference_unet,
+ denoising_unet,
+ motion_encoder,
+ pose_encoder,
+ pose_guider,
+ scheduler: Union[
+ DDIMScheduler,
+ PNDMScheduler,
+ LMSDiscreteScheduler,
+ EulerDiscreteScheduler,
+ EulerAncestralDiscreteScheduler,
+ DPMSolverMultistepScheduler,
+ ],
+ ):
+ super().__init__()
+
+ self.register_modules(
+ vae=vae,
+ image_encoder=image_encoder,
+ reference_unet=reference_unet,
+ denoising_unet=denoising_unet,
+ motion_encoder=motion_encoder,
+ pose_encoder=pose_encoder,
+ pose_guider=pose_guider,
+ scheduler=scheduler,
+ )
+ self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1)
+ self.clip_image_processor = CLIPImageProcessor()
+ self.ref_image_processor = VaeImageProcessor(
+ vae_scale_factor=self.vae_scale_factor, do_convert_rgb=True
+ )
+ self.cond_image_processor = VaeImageProcessor(
+ vae_scale_factor=self.vae_scale_factor,
+ do_convert_rgb=True,
+ do_normalize=True,
+ )
+
+ def enable_vae_slicing(self):
+ self.vae.enable_slicing()
+
+ def disable_vae_slicing(self):
+ self.vae.disable_slicing()
+
+ def enable_sequential_cpu_offload(self, gpu_id=0):
+ if is_accelerate_available():
+ from accelerate import cpu_offload
+ else:
+ raise ImportError("Please install accelerate via `pip install accelerate`")
+
+ device = torch.device(f"cuda:{gpu_id}")
+
+ for cpu_offloaded_model in [self.unet, self.text_encoder, self.vae]:
+ if cpu_offloaded_model is not None:
+ cpu_offload(cpu_offloaded_model, device)
+
+ @property
+ def _execution_device(self):
+ if self.device != torch.device("meta") or not hasattr(self.unet, "_hf_hook"):
+ return self.device
+ for module in self.unet.modules():
+ if (
+ hasattr(module, "_hf_hook")
+ and hasattr(module._hf_hook, "execution_device")
+ and module._hf_hook.execution_device is not None
+ ):
+ return torch.device(module._hf_hook.execution_device)
+ return self.device
+
+ def decode_latents(self, latents, decode_chunk_size=4):
+ video_length = latents.shape[2]
+ latents = 1 / 0.18215 * latents
+ latents = rearrange(latents, "b c f h w -> (b f) c h w")
+ # video = self.vae.decode(latents).sample
+ video = []
+ for frame_idx in range(0, latents.shape[0], decode_chunk_size):
+ video.append(self.vae.decode(latents[frame_idx : frame_idx + decode_chunk_size]).sample)
+ video = torch.cat(video)
+ video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)
+ video = (video / 2 + 0.5).clamp(0, 1)
+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
+ video = video.cpu().float().numpy()
+ return video
+
+ def decode_latents_svd(
+ self, latents: torch.FloatTensor, decode_chunk_size: int = 5
+ ):
+ video_length = latents.shape[2]
+ latents = 1 / self.vae.config.scaling_factor * latents
+ latents = rearrange(latents, "b c f h w -> (b f) c h w").to(self.vae.dtype)
+
+ # decode decode_chunk_size frames at a time to avoid OOM
+ frames = []
+ for i in range(0, latents.shape[0], decode_chunk_size):
+ num_frames_in = latents[i : i + decode_chunk_size].shape[0]
+ frame = self.vae.decode(
+ latents[i : i + decode_chunk_size], num_frames_in
+ ).sample
+ frames.append(frame)
+ video = torch.cat(frames, dim=0)
+ video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)
+ video = (video / 2 + 0.5).clamp(0, 1)
+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
+ video = video.cpu().float().numpy()
+ return video
+
+ def prepare_extra_step_kwargs(self, generator, eta):
+ # prepare extra kwargs for the scheduler step, since not all schedulers have the same signature
+ # eta (η) is only used with the DDIMScheduler, it will be ignored for other schedulers.
+ # eta corresponds to η in DDIM paper: https://arxiv.org/abs/2010.02502
+ # and should be between [0, 1]
+
+ accepts_eta = "eta" in set(
+ inspect.signature(self.scheduler.step).parameters.keys()
+ )
+ extra_step_kwargs = {}
+ if accepts_eta:
+ extra_step_kwargs["eta"] = eta
+
+ # check if the scheduler accepts generator
+ accepts_generator = "generator" in set(
+ inspect.signature(self.scheduler.step).parameters.keys()
+ )
+ if accepts_generator:
+ extra_step_kwargs["generator"] = generator
+ return extra_step_kwargs
+
+ def prepare_latents(
+ self,
+ batch_size,
+ num_channels_latents,
+ width,
+ height,
+ video_length,
+ dtype,
+ device,
+ generator,
+ latents=None,
+ ):
+ shape = (
+ batch_size,
+ num_channels_latents,
+ video_length,
+ height // self.vae_scale_factor,
+ width // self.vae_scale_factor,
+ )
+ if isinstance(generator, list) and len(generator) != batch_size:
+ raise ValueError(
+ f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"
+ f" size of {batch_size}. Make sure the batch size matches the length of the generators."
+ )
+
+ if latents is None:
+ latents = randn_tensor(
+ shape, generator=generator, device=device, dtype=dtype
+ )
+ else:
+ latents = latents.to(device)
+
+ # scale the initial noise by the standard deviation required by the scheduler
+ latents = latents * self.scheduler.init_noise_sigma
+ return latents
+
+ def _encode_prompt(
+ self,
+ prompt,
+ device,
+ num_videos_per_prompt,
+ do_classifier_free_guidance,
+ negative_prompt,
+ ):
+ batch_size = len(prompt) if isinstance(prompt, list) else 1
+
+ text_inputs = self.tokenizer(
+ prompt,
+ padding="max_length",
+ max_length=self.tokenizer.model_max_length,
+ truncation=True,
+ return_tensors="pt",
+ )
+ text_input_ids = text_inputs.input_ids
+ untruncated_ids = self.tokenizer(
+ prompt, padding="longest", return_tensors="pt"
+ ).input_ids
+
+ if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(
+ text_input_ids, untruncated_ids
+ ):
+ removed_text = self.tokenizer.batch_decode(
+ untruncated_ids[:, self.tokenizer.model_max_length - 1 : -1]
+ )
+
+ if (
+ hasattr(self.text_encoder.config, "use_attention_mask")
+ and self.text_encoder.config.use_attention_mask
+ ):
+ attention_mask = text_inputs.attention_mask.to(device)
+ else:
+ attention_mask = None
+
+ text_embeddings = self.text_encoder(
+ text_input_ids.to(device),
+ attention_mask=attention_mask,
+ )
+ text_embeddings = text_embeddings[0]
+
+ # duplicate text embeddings for each generation per prompt, using mps friendly method
+ bs_embed, seq_len, _ = text_embeddings.shape
+ text_embeddings = text_embeddings.repeat(1, num_videos_per_prompt, 1)
+ text_embeddings = text_embeddings.view(
+ bs_embed * num_videos_per_prompt, seq_len, -1
+ )
+
+ # get unconditional embeddings for classifier free guidance
+ if do_classifier_free_guidance:
+ uncond_tokens: List[str]
+ if negative_prompt is None:
+ uncond_tokens = [""] * batch_size
+ elif type(prompt) is not type(negative_prompt):
+ raise TypeError(
+ f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !="
+ f" {type(prompt)}."
+ )
+ elif isinstance(negative_prompt, str):
+ uncond_tokens = [negative_prompt]
+ elif batch_size != len(negative_prompt):
+ raise ValueError(
+ f"`negative_prompt`: {negative_prompt} has batch size {len(negative_prompt)}, but `prompt`:"
+ f" {prompt} has batch size {batch_size}. Please make sure that passed `negative_prompt` matches"
+ " the batch size of `prompt`."
+ )
+ else:
+ uncond_tokens = negative_prompt
+
+ max_length = text_input_ids.shape[-1]
+ uncond_input = self.tokenizer(
+ uncond_tokens,
+ padding="max_length",
+ max_length=max_length,
+ truncation=True,
+ return_tensors="pt",
+ )
+
+ if (
+ hasattr(self.text_encoder.config, "use_attention_mask")
+ and self.text_encoder.config.use_attention_mask
+ ):
+ attention_mask = uncond_input.attention_mask.to(device)
+ else:
+ attention_mask = None
+
+ uncond_embeddings = self.text_encoder(
+ uncond_input.input_ids.to(device),
+ attention_mask=attention_mask,
+ )
+ uncond_embeddings = uncond_embeddings[0]
+
+ # duplicate unconditional embeddings for each generation per prompt, using mps friendly method
+ seq_len = uncond_embeddings.shape[1]
+ uncond_embeddings = uncond_embeddings.repeat(1, num_videos_per_prompt, 1)
+ uncond_embeddings = uncond_embeddings.view(
+ batch_size * num_videos_per_prompt, seq_len, -1
+ )
+
+ # For classifier free guidance, we need to do two forward passes.
+ # Here we concatenate the unconditional and text embeddings into a single batch
+ # to avoid doing two forward passes
+ text_embeddings = torch.cat([uncond_embeddings, text_embeddings])
+
+ return text_embeddings
+
+ def downgrade_input(self, init_latents, generator, device, dtype):
+ mask = (
+ torch.rand(
+ *init_latents.shape,
+ generator=generator,
+ device=device,
+ )
+ >= 0.5
+ ).to(dtype=dtype)
+
+ blur = transforms.GaussianBlur(kernel_size=(9, 9), sigma=(18, 18))
+ # first_frame_latents = init_latents[:, :, 0:1, :, :].clone()
+ video_length = init_latents.shape[2]
+ init_latents = rearrange(init_latents, "b c f h w -> (b f) c h w")
+ init_latents_blur = blur(init_latents)
+ init_latents = rearrange(init_latents, "(b f) c h w -> b c f h w", f=video_length)
+ init_latents_blur = rearrange(init_latents_blur, "(b f) c h w -> b c f h w", f=video_length)
+ # init_latents = init_latents * mask + init_latents_blur * (1 - mask)
+ # init_latents[:, :, 0:1, :, :] = first_frame_latents
+
+ return init_latents_blur
+
+ def interpolate_latents(
+ self, latents: torch.Tensor, interpolation_factor: int, device
+ ):
+ if interpolation_factor < 2:
+ return latents
+
+ new_latents = torch.zeros(
+ (
+ latents.shape[0],
+ latents.shape[1],
+ ((latents.shape[2] - 1) * interpolation_factor) + 1,
+ latents.shape[3],
+ latents.shape[4],
+ ),
+ device=latents.device,
+ dtype=latents.dtype,
+ )
+
+ org_video_length = latents.shape[2]
+ rate = [i / interpolation_factor for i in range(interpolation_factor)][1:]
+
+ new_index = 0
+
+ v0 = None
+ v1 = None
+
+ for i0, i1 in zip(range(org_video_length), range(org_video_length)[1:]):
+ v0 = latents[:, :, i0, :, :]
+ v1 = latents[:, :, i1, :, :]
+
+ new_latents[:, :, new_index, :, :] = v0
+ new_index += 1
+
+ for f in rate:
+ v = get_tensor_interpolation_method()(
+ v0.to(device=device), v1.to(device=device), f
+ )
+ new_latents[:, :, new_index, :, :] = v.to(latents.device)
+ new_index += 1
+
+ new_latents[:, :, new_index, :, :] = v1
+ new_index += 1
+
+ return new_latents
+
+ @torch.no_grad()
+ def __call__(
+ self,
+ tgt_images,
+ ref_image,
+ exp_images,
+ ref_exp_image,
+ width,
+ height,
+ video_length,
+ num_inference_steps,
+ guidance_scale,
+ timesteps_list=None,
+ eta: float = 0.0,
+ generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
+ output_type: Optional[str] = "tensor",
+ return_dict: bool = True,
+ callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None,
+ callback_steps: Optional[int] = 1,
+ mot_bbox_param=None,
+ context_schedule="uniform",
+ context_frames=10,
+ context_stride=1,
+ context_overlap=0,
+ context_batch_size=1,
+ **kwargs,
+ ):
+ # Default height and width to unet
+ height = height or self.unet.config.sample_size * self.vae_scale_factor
+ width = width or self.unet.config.sample_size * self.vae_scale_factor
+
+ device = self._execution_device
+
+ do_classifier_free_guidance = guidance_scale > 1.0
+
+ # Prepare timesteps
+ if timesteps_list is None:
+ self.scheduler.set_timesteps(num_inference_steps, device=device)
+ timesteps = self.scheduler.timesteps
+ else:
+ assert len(timesteps_list) == num_inference_steps, 'len(timesteps_list) != num_inference_steps'
+ timesteps = torch.tensor(timesteps_list, device=device).long()
+ self.scheduler.set_step_length(timesteps_list[0]-timesteps_list[1])
+
+ batch_size = 1
+
+ # Prepare clip image embeds
+ clip_image = self.clip_image_processor.preprocess(
+ ref_image.resize((224, 224)), return_tensors="pt"
+ ).pixel_values
+ clip_image_embeds = self.image_encoder(
+ clip_image.to(device, dtype=self.image_encoder.dtype)
+ ).image_embeds
+ image_prompt_embeds = clip_image_embeds.unsqueeze(1)
+
+ uncond_image_prompt_embeds = torch.zeros_like(image_prompt_embeds)
+
+ if do_classifier_free_guidance:
+ image_prompt_embeds = torch.cat(
+ [uncond_image_prompt_embeds, image_prompt_embeds], dim=0
+ )
+
+ reference_control_writer = ReferenceAttentionControl(
+ self.reference_unet,
+ do_classifier_free_guidance=do_classifier_free_guidance,
+ mode="write",
+ batch_size=batch_size,
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ self.denoising_unet,
+ do_classifier_free_guidance=do_classifier_free_guidance,
+ mode="read",
+ batch_size=batch_size,
+ fusion_blocks="full",
+ )
+
+ # Prepare extra step kwargs.
+ extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
+
+ # Prepare ref image latents
+ ref_image_tensor = self.ref_image_processor.preprocess(
+ ref_image, height=height, width=width
+ ) # (bs, c, width, height)
+ ref_image_tensor = ref_image_tensor.to(
+ dtype=self.vae.dtype, device=self.vae.device
+ )
+ ref_image_latents = self.vae.encode(ref_image_tensor).latent_dist.mean
+ ref_image_latents = ref_image_latents * 0.18215 # (b, 4, h, w)
+
+ repeated_latents = ref_image_latents.unsqueeze(2).repeat(1, 1, video_length, 1, 1)
+ noise = torch.randn_like(repeated_latents)
+ noisy_latents_first = []
+ for i in range(video_length):
+ noisy_latents_first.append(self.scheduler.add_noise(repeated_latents[:, :, i:i+1, :, :], noise[:, :, i:i+1, :, :], timesteps[:1]))
+ latents = torch.cat(noisy_latents_first, dim=2)
+
+ ref_cond_tensor = self.cond_image_processor.preprocess(
+ ref_image, height=256, width=256
+ ).to(device=device, dtype=self.pose_encoder.dtype) # (1, c, h, w)
+ ref_cond_tensor = ref_cond_tensor / 2 + 0.5 # to [0, 1]
+
+ # Prepare a list of pose condition images
+ exp_cond_tensor_list = []
+ mot_bbox_param = []
+ for tgt_image, exp_image in zip(tgt_images, exp_images):
+ exp_cond_tensor = self.cond_image_processor.preprocess(
+ exp_image, height=224, width=224
+ ).transpose(0, 1) # (c, 1, h, w)
+ exp_cond_tensor_list.append(exp_cond_tensor)
+
+ tgt_cond_tensor = self.cond_image_processor.preprocess(
+ tgt_image, height=256, width=256
+ ).to(device=device, dtype=self.pose_encoder.dtype) # (1, c, h, w)
+ tgt_cond_tensor = tgt_cond_tensor / 2 + 0.5 # to [0, 1]
+ ref_motion_emb, motion_emb = self.pose_encoder.inference(ref_cond_tensor, tgt_cond_tensor) # [1, c]
+ mot_bbox_param.append(motion_emb)
+ mot_bbox_param = torch.cat(mot_bbox_param, dim=0) # (t, c)
+ keypoints = draw_keypoints(mot_bbox_param).unsqueeze(2)
+ keypoints = rearrange(keypoints, 'f c b h w -> b c f h w')
+ keypoints = keypoints.to(device=device, dtype=self.pose_guider.dtype)
+ pose_feas = self.pose_guider(keypoints) # (t, c)
+
+ exp_cond_tensor = torch.cat(exp_cond_tensor_list, dim=1) # (c, t, h, w)
+
+ exp_cond_tensor = exp_cond_tensor.unsqueeze(0) # (1, c, t, h, w)
+ exp_cond_tensor = exp_cond_tensor.to(
+ device=device, dtype=self.motion_encoder.dtype
+ )
+ context_scheduler = get_context_scheduler(context_schedule)
+
+ if do_classifier_free_guidance:
+ ref_exp_cond_tensor = self.cond_image_processor.preprocess(
+ ref_exp_image, height=224, width=224
+ ).to(device=device, dtype=self.motion_encoder.dtype)
+ neg_motion_hidden_states = self.motion_encoder(ref_exp_cond_tensor) # [b, c]
+
+ ref_keypoints = draw_keypoints(ref_motion_emb).unsqueeze(2)
+ ref_keypoints = ref_keypoints.to(device=device, dtype=self.pose_guider.dtype)
+ ref_pose_fea = self.pose_guider(ref_keypoints)
+
+ # denoising loop
+ num_warmup_steps = len(timesteps) - num_inference_steps * self.scheduler.order
+ with self.progress_bar(total=num_inference_steps) as progress_bar:
+ for i, t in enumerate(timesteps):
+ if i == 0:
+ self.reference_unet(
+ ref_image_latents.repeat(
+ (2 if do_classifier_free_guidance else 1), 1, 1, 1
+ ),
+ torch.zeros_like(t),
+ encoder_hidden_states=image_prompt_embeds,
+ return_dict=False,
+ )
+ reference_control_reader.update(reference_control_writer)
+
+ noise_pred = torch.zeros(
+ (
+ latents.shape[0] * (2 if do_classifier_free_guidance else 1),
+ *latents.shape[1:],
+ ),
+ device=latents.device,
+ dtype=latents.dtype,
+ )
+ counter = torch.zeros(
+ (1, 1, latents.shape[2], 1, 1),
+ device=latents.device,
+ dtype=latents.dtype,
+ )
+
+ offset = random.randint(0, context_frames-1)
+ context_queue = list(
+ context_scheduler(
+ 0,
+ num_inference_steps,
+ latents.shape[2],
+ context_frames,
+ context_stride,
+ context_overlap,
+ True,
+ offset
+ )
+ )
+
+ num_context_batches = math.ceil(len(context_queue) / context_batch_size)
+ global_context = []
+ for i in range(num_context_batches):
+ global_context.append(
+ context_queue[
+ i * context_batch_size : (i + 1) * context_batch_size
+ ]
+ )
+
+ for context in global_context:
+ # 3.1 expand the latents if we are doing classifier free guidance
+ latent_model_input = (
+ torch.cat([latents[:, :, c] for c in context])
+ .to(device)
+ .repeat(2 if do_classifier_free_guidance else 1, 1, 1, 1, 1)
+ ).to(self.denoising_unet.dtype)
+ latent_model_input = self.scheduler.scale_model_input(
+ latent_model_input, t
+ )
+
+ pose_cond_input = torch.cat([exp_cond_tensor[:, :, c] for c in context])
+ motion_hidden_states = self.motion_encoder(pose_cond_input) # [b, c]
+
+ pose_fea = torch.cat([pose_feas[:,:,c] for c in context], dim=0)
+
+ if do_classifier_free_guidance:
+ motion_hidden_states = torch.cat(
+ [
+ neg_motion_hidden_states.unsqueeze(1).expand_as(motion_hidden_states),
+ motion_hidden_states,
+ ],
+ dim=0,
+ )
+ pose_fea = torch.cat(
+ [
+ ref_pose_fea.expand_as(pose_fea),
+ pose_fea,
+ ],
+ dim=0,
+ )
+
+ pred = self.denoising_unet(
+ latent_model_input,
+ t,
+ encoder_hidden_states=[
+ image_prompt_embeds,
+ motion_hidden_states,
+ ],
+ pose_cond_fea=pose_fea,
+ return_dict=False,
+ )[0]
+
+ for j, c in enumerate(context):
+ noise_pred[:, :, c] = noise_pred[:, :, c] + pred
+ counter[:, :, c] = counter[:, :, c] + 1
+
+ # perform guidance
+ if do_classifier_free_guidance:
+ noise_pred_uncond, noise_pred_text = (noise_pred / counter).chunk(2)
+ noise_pred = noise_pred_uncond + guidance_scale * (
+ noise_pred_text - noise_pred_uncond
+ )
+
+ latents, pred_original_sample = self.scheduler.step(
+ noise_pred, t, latents, **extra_step_kwargs, return_dict=False
+ )
+
+ if i == len(timesteps) - 1 or (
+ (i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0
+ ):
+ progress_bar.update()
+ if callback is not None and i % callback_steps == 0:
+ step_idx = i // getattr(self.scheduler, "order", 1)
+ callback(step_idx, t, latents)
+
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+
+ # Post-processing
+ if isinstance(self.vae, AutoencoderKL):
+ images = self.decode_latents(pred_original_sample) # (b, c, f, h, w)
+ elif isinstance(self.vae, AutoencoderKLTemporalDecoder):
+ images = self.decode_latents_svd(pred_original_sample)
+
+ # Convert to tensor
+ if output_type == "tensor":
+ images = torch.from_numpy(images)
+
+ if not return_dict:
+ return images
+
+ return Pose2VideoPipelineOutput(videos=images)
diff --git a/src/pipelines/pipeline_pose2vid.py b/src/pipelines/pipeline_pose2vid.py
index 2c764fc..a64a2bb 100644
--- a/src/pipelines/pipeline_pose2vid.py
+++ b/src/pipelines/pipeline_pose2vid.py
@@ -456,6 +456,7 @@ class Pose2VideoPipeline(DiffusionPipeline):
temporal_adaptive_step = 4,
temporal_kv_cache=True,
init_latents=None,
+ timesteps_list=None,
**kwargs,
):
assert num_inference_steps % temporal_adaptive_step == 0, "temporal_adaptive_step should be divisor of num_inference_steps"
@@ -469,8 +470,13 @@ class Pose2VideoPipeline(DiffusionPipeline):
do_classifier_free_guidance = guidance_scale > 1.0
# Prepare timesteps
- timesteps = torch.tensor([999, 666, 333, 0], device=device).long()
- self.scheduler.set_step_length(333)
+ if timesteps_list is not None:
+ assert len(timesteps_list) == num_inference_steps, 'len(timesteps_list) != num_inference_steps'
+ timesteps = torch.tensor(timesteps_list, device=device).long()
+ self.scheduler.set_step_length(timesteps_list[0]-timesteps_list[1])
+ else:
+ timesteps = torch.tensor([999, 666, 333, 0], device=device).long()
+ self.scheduler.set_step_length(333)
jump = num_inference_steps // temporal_adaptive_step
windows = video_length // temporal_window_size
@@ -606,7 +612,7 @@ class Pose2VideoPipeline(DiffusionPipeline):
pose_fea = pose_feas[:, :, l:r]
add_flag = False
- if l > temporal_adaptive_step * temporal_window_size * 2 and motion_bank.shape[1] < 4:
+ if l > temporal_adaptive_step * temporal_window_size * 2 and motion_bank.shape[1] < 4 and temporal_kv_cache:
add_flag, motion_bank = self.calculate_dis(motion_bank, motion_hidden_state, threshold=17.)
if do_classifier_free_guidance:
@@ -896,7 +902,7 @@ class Pose2VideoPipeline_Stream(Pose2VideoPipeline):
pose_feas = torch.cat([pose_feas[:,:,temporal_window_size:], pose_fea], dim=2)
if l > temporal_adaptive_step * temporal_window_size * 2 and motion_bank.shape[1] < 4:
- add_flag, motion_bank = self.calculate_dis(motion_bank, motion_hidden_state, threshold=17.)
+ add_flag, motion_bank = self.calculate_dis(motion_bank, motion_hidden_states, threshold=17.)
latents = ref_image_latents.unsqueeze(2).repeat(1, 1, temporal_window_size, 1, 1)
noise = torch.randn_like(latents)
diff --git a/src/stylegan2/discriminator.py b/src/stylegan2/discriminator.py
new file mode 100644
index 0000000..a54bf8e
--- /dev/null
+++ b/src/stylegan2/discriminator.py
@@ -0,0 +1,793 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+import numpy as np
+import torch
+from torch_utils import misc
+from torch_utils import persistence
+from torch_utils.ops import conv2d_resample
+from torch_utils.ops import upfirdn2d
+from torch_utils.ops import bias_act
+from torch_utils.ops import fma
+import torch.nn.functional as F
+import torch.nn as nn
+from diffusers.models.embeddings import TimestepEmbedding, Timesteps
+#----------------------------------------------------------------------------
+
+def normalize_2nd_moment(x, dim=1, eps=1e-8):
+ return x * (x.square().mean(dim=dim, keepdim=True) + eps).rsqrt()
+
+def zero_module(module):
+ # Zero out the parameters of a module and return it.
+ assert isinstance(module, nn.Conv2d) or isinstance(module, nn.Linear), type(module)
+ for p in module.parameters():
+ p.detach().zero_()
+ return module
+
+#----------------------------------------------------------------------------
+
+def modulated_conv2d(
+ x, # Input tensor of shape [batch_size, in_channels, in_height, in_width].
+ weight, # Weight tensor of shape [out_channels, in_channels, kernel_height, kernel_width].
+ styles, # Modulation coefficients of shape [batch_size, in_channels].
+ noise = None, # Optional noise tensor to add to the output activations.
+ up = 1, # Integer upsampling factor.
+ down = 1, # Integer downsampling factor.
+ padding = 0, # Padding with respect to the upsampled image.
+ resample_filter = None, # Low-pass filter to apply when resampling activations. Must be prepared beforehand by calling upfirdn2d.setup_filter().
+ demodulate = True, # Apply weight demodulation?
+ flip_weight = True, # False = convolution, True = correlation (matches torch.nn.functional.conv2d).
+ fused_modconv = True, # Perform modulation, convolution, and demodulation as a single fused operation?
+):
+ batch_size = x.shape[0]
+ out_channels, in_channels, kh, kw = weight.shape
+ misc.assert_shape(weight, [out_channels, in_channels, kh, kw]) # [OIkk]
+ misc.assert_shape(x, [batch_size, in_channels, None, None]) # [NIHW]
+ misc.assert_shape(styles, [batch_size, in_channels]) # [NI]
+
+ # Pre-normalize inputs to avoid FP16 overflow.
+ if x.dtype == torch.float16 and demodulate:
+ weight = weight * (1 / np.sqrt(in_channels * kh * kw) / weight.norm(float('inf'), dim=[1,2,3], keepdim=True)) # max_Ikk
+ styles = styles / styles.norm(float('inf'), dim=1, keepdim=True) # max_I
+
+ # Calculate per-sample weights and demodulation coefficients.
+ w = None
+ dcoefs = None
+ if demodulate or fused_modconv:
+ w = weight.unsqueeze(0) # [NOIkk]
+ w = w * styles.reshape(batch_size, 1, -1, 1, 1) # [NOIkk]
+ if demodulate:
+ dcoefs = (w.square().sum(dim=[2,3,4]) + 1e-8).rsqrt() # [NO]
+ if demodulate and fused_modconv:
+ w = w * dcoefs.reshape(batch_size, -1, 1, 1, 1) # [NOIkk]
+
+ # Execute by scaling the activations before and after the convolution.
+ if not fused_modconv:
+ x = x * styles.to(x.dtype).reshape(batch_size, -1, 1, 1)
+ x = conv2d_resample.conv2d_resample(x=x, w=weight.to(x.dtype), f=resample_filter, up=up, down=down, padding=padding, flip_weight=flip_weight)
+ if demodulate and noise is not None:
+ x = fma.fma(x, dcoefs.to(x.dtype).reshape(batch_size, -1, 1, 1), noise.to(x.dtype))
+ elif demodulate:
+ x = x * dcoefs.to(x.dtype).reshape(batch_size, -1, 1, 1)
+ elif noise is not None:
+ x = x.add_(noise.to(x.dtype))
+ return x
+
+ # Execute as one fused op using grouped convolution.
+ with misc.suppress_tracer_warnings(): # this value will be treated as a constant
+ batch_size = int(batch_size)
+ misc.assert_shape(x, [batch_size, in_channels, None, None])
+ x = x.reshape(1, -1, *x.shape[2:])
+ w = w.reshape(-1, in_channels, kh, kw)
+ x = conv2d_resample.conv2d_resample(x=x, w=w.to(x.dtype), f=resample_filter, up=up, down=down, padding=padding, groups=batch_size, flip_weight=flip_weight)
+ x = x.reshape(batch_size, -1, *x.shape[2:])
+ if noise is not None:
+ x = x.add_(noise)
+ return x
+
+#----------------------------------------------------------------------------
+
+class FullyConnectedLayer(torch.nn.Module):
+ def __init__(self,
+ in_features, # Number of input features.
+ out_features, # Number of output features.
+ bias = True, # Apply additive bias before the activation function?
+ activation = 'linear', # Activation function: 'relu', 'lrelu', etc.
+ lr_multiplier = 1, # Learning rate multiplier.
+ bias_init = 0, # Initial value for the additive bias.
+ ):
+ super().__init__()
+ self.activation = activation
+ self.weight = torch.nn.Parameter(torch.randn([out_features, in_features]) / lr_multiplier)
+ self.bias = torch.nn.Parameter(torch.full([out_features], np.float32(bias_init))) if bias else None
+ self.weight_gain = lr_multiplier / np.sqrt(in_features)
+ self.bias_gain = lr_multiplier
+
+ def forward(self, x):
+ w = self.weight.to(x.dtype) * self.weight_gain
+ b = self.bias
+ if b is not None:
+ b = b.to(x.dtype)
+ if self.bias_gain != 1:
+ b = b * self.bias_gain
+
+ if self.activation == 'linear' and b is not None:
+ x = torch.addmm(b.unsqueeze(0), x, w.t())
+ else:
+ x = x.matmul(w.t())
+ x = bias_act.bias_act(x, b, act=self.activation)
+ return x
+
+#----------------------------------------------------------------------------
+
+class Conv2dLayer(torch.nn.Module):
+ def __init__(self,
+ in_channels, # Number of input channels.
+ out_channels, # Number of output channels.
+ kernel_size, # Width and height of the convolution kernel.
+ bias = True, # Apply additive bias before the activation function?
+ activation = 'linear', # Activation function: 'relu', 'lrelu', etc.
+ up = 1, # Integer upsampling factor.
+ down = 1, # Integer downsampling factor.
+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.
+ conv_clamp = None, # Clamp the output to +-X, None = disable clamping.
+ channels_last = False, # Expect the input to have memory_format=channels_last?
+ trainable = True, # Update the weights of this layer during training?
+ ):
+ super().__init__()
+ self.activation = activation
+ self.up = up
+ self.down = down
+ self.conv_clamp = conv_clamp
+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))
+ self.padding = kernel_size // 2
+ self.weight_gain = 1 / np.sqrt(in_channels * (kernel_size ** 2))
+ self.act_gain = bias_act.activation_funcs[activation].def_gain
+
+ memory_format = torch.channels_last if channels_last else torch.contiguous_format
+ weight = torch.randn([out_channels, in_channels, kernel_size, kernel_size]).to(memory_format=memory_format)
+ bias = torch.zeros([out_channels]) if bias else None
+ if trainable:
+ self.weight = torch.nn.Parameter(weight)
+ self.bias = torch.nn.Parameter(bias) if bias is not None else None
+ else:
+ self.register_buffer('weight', weight)
+ if bias is not None:
+ self.register_buffer('bias', bias)
+ else:
+ self.bias = None
+
+ def forward(self, x, gain=1):
+ dtype = x.dtype
+ w = self.weight * self.weight_gain
+ b = self.bias.to(x.dtype) if self.bias is not None else None
+ flip_weight = (self.up == 1) # slightly faster
+ f=self.resample_filter
+ if f is not None:
+ f = f.to(dtype=torch.float32)
+ x = conv2d_resample.conv2d_resample(x=x, w=w.to(x.dtype), f=f, up=self.up, down=self.down, padding=self.padding, flip_weight=flip_weight)
+
+ x = x.to(dtype)
+ act_gain = self.act_gain * gain
+ act_clamp = self.conv_clamp * gain if self.conv_clamp is not None else None
+ x = bias_act.bias_act(x, b, act=self.activation, gain=act_gain, clamp=act_clamp)
+ return x
+
+#----------------------------------------------------------------------------
+
+class MappingNetwork(torch.nn.Module):
+ def __init__(self,
+ z_dim, # Input latent (Z) dimensionality, 0 = no latent.
+ c_dim, # Conditioning label (C) dimensionality, 0 = no label.
+ w_dim, # Intermediate latent (W) dimensionality.
+ num_ws, # Number of intermediate latents to output, None = do not broadcast.
+ num_layers = 8, # Number of mapping layers.
+ embed_features = None, # Label embedding dimensionality, None = same as w_dim.
+ layer_features = None, # Number of intermediate features in the mapping layers, None = same as w_dim.
+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.
+ lr_multiplier = 0.01, # Learning rate multiplier for the mapping layers.
+ w_avg_beta = 0.995, # Decay for tracking the moving average of W during training, None = do not track.
+ ):
+ super().__init__()
+ self.z_dim = z_dim
+ self.c_dim = c_dim
+ self.w_dim = w_dim
+ self.num_ws = num_ws
+ self.num_layers = num_layers
+ self.w_avg_beta = w_avg_beta
+
+ if embed_features is None:
+ embed_features = w_dim
+ if c_dim == 0:
+ embed_features = 0
+ if layer_features is None:
+ layer_features = w_dim
+ features_list = [z_dim + embed_features] + [layer_features] * (num_layers - 1) + [w_dim]
+
+ if c_dim > 0:
+ self.embed = FullyConnectedLayer(c_dim, embed_features)
+ for idx in range(num_layers):
+ in_features = features_list[idx]
+ out_features = features_list[idx + 1]
+ layer = FullyConnectedLayer(in_features, out_features, activation=activation, lr_multiplier=lr_multiplier)
+ setattr(self, f'fc{idx}', layer)
+
+ if num_ws is not None and w_avg_beta is not None:
+ self.register_buffer('w_avg', torch.zeros([w_dim]))
+
+ def forward(self, z, c, truncation_psi=1, truncation_cutoff=None, skip_w_avg_update=False):
+ # Embed, normalize, and concat inputs.
+ x = None
+ with torch.autograd.profiler.record_function('input'):
+ if self.z_dim > 0:
+ misc.assert_shape(z, [None, self.z_dim])
+ x = normalize_2nd_moment(z.to(torch.float32))
+ if self.c_dim > 0:
+ misc.assert_shape(c, [None, self.c_dim])
+ y = normalize_2nd_moment(self.embed(c.to(torch.float32)))
+ x = torch.cat([x, y], dim=1) if x is not None else y
+
+ # Main layers.
+ for idx in range(self.num_layers):
+ layer = getattr(self, f'fc{idx}')
+ x = layer(x)
+
+ # Update moving average of W.
+ if self.w_avg_beta is not None and self.training and not skip_w_avg_update:
+ with torch.autograd.profiler.record_function('update_w_avg'):
+ self.w_avg.copy_(x.detach().mean(dim=0).lerp(self.w_avg, self.w_avg_beta))
+
+ # Broadcast.
+ if self.num_ws is not None:
+ with torch.autograd.profiler.record_function('broadcast'):
+ x = x.unsqueeze(1).repeat([1, self.num_ws, 1])
+
+ # Apply truncation.
+ if truncation_psi != 1:
+ with torch.autograd.profiler.record_function('truncate'):
+ assert self.w_avg_beta is not None
+ if self.num_ws is None or truncation_cutoff is None:
+ x = self.w_avg.lerp(x, truncation_psi)
+ else:
+ x[:, :truncation_cutoff] = self.w_avg.lerp(x[:, :truncation_cutoff], truncation_psi)
+ return x
+
+#----------------------------------------------------------------------------
+
+class SynthesisLayer(torch.nn.Module):
+ def __init__(self,
+ in_channels, # Number of input channels.
+ out_channels, # Number of output channels.
+ w_dim, # Intermediate latent (W) dimensionality.
+ resolution, # Resolution of this layer.
+ kernel_size = 3, # Convolution kernel size.
+ up = 1, # Integer upsampling factor.
+ use_noise = True, # Enable noise input?
+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.
+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.
+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.
+ channels_last = False, # Use channels_last format for the weights?
+ ):
+ super().__init__()
+ self.resolution = resolution
+ self.up = up
+ self.use_noise = use_noise
+ self.activation = activation
+ self.conv_clamp = conv_clamp
+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))
+ self.padding = kernel_size // 2
+ self.act_gain = bias_act.activation_funcs[activation].def_gain
+
+ self.affine = FullyConnectedLayer(w_dim, in_channels, bias_init=1)
+ memory_format = torch.channels_last if channels_last else torch.contiguous_format
+ self.weight = torch.nn.Parameter(torch.randn([out_channels, in_channels, kernel_size, kernel_size]).to(memory_format=memory_format))
+ if use_noise:
+ self.register_buffer('noise_const', torch.randn([resolution, resolution]))
+ self.noise_strength = torch.nn.Parameter(torch.zeros([]))
+ self.bias = torch.nn.Parameter(torch.zeros([out_channels]))
+
+ def forward(self, x, w, noise_mode='random', fused_modconv=True, gain=1):
+ assert noise_mode in ['random', 'const', 'none']
+ in_resolution = self.resolution // self.up
+ misc.assert_shape(x, [None, self.weight.shape[1], in_resolution, in_resolution])
+ styles = self.affine(w)
+
+ noise = None
+ if self.use_noise and noise_mode == 'random':
+ noise = torch.randn([x.shape[0], 1, self.resolution, self.resolution], device=x.device) * self.noise_strength
+ if self.use_noise and noise_mode == 'const':
+ noise = self.noise_const * self.noise_strength
+
+ flip_weight = (self.up == 1) # slightly faster
+ x = modulated_conv2d(x=x, weight=self.weight, styles=styles, noise=noise, up=self.up,
+ padding=self.padding, resample_filter=self.resample_filter, flip_weight=flip_weight, fused_modconv=fused_modconv)
+
+ act_gain = self.act_gain * gain
+ act_clamp = self.conv_clamp * gain if self.conv_clamp is not None else None
+ x = bias_act.bias_act(x, self.bias.to(x.dtype), act=self.activation, gain=act_gain, clamp=act_clamp)
+ return x
+
+#----------------------------------------------------------------------------
+
+class ToRGBLayer(torch.nn.Module):
+ def __init__(self, in_channels, out_channels, w_dim, kernel_size=1, conv_clamp=None, channels_last=False):
+ super().__init__()
+ self.conv_clamp = conv_clamp
+ self.affine = FullyConnectedLayer(w_dim, in_channels, bias_init=1)
+ memory_format = torch.channels_last if channels_last else torch.contiguous_format
+ self.weight = torch.nn.Parameter(torch.randn([out_channels, in_channels, kernel_size, kernel_size]).to(memory_format=memory_format))
+ self.bias = torch.nn.Parameter(torch.zeros([out_channels]))
+ self.weight_gain = 1 / np.sqrt(in_channels * (kernel_size ** 2))
+
+ def forward(self, x, w, fused_modconv=True):
+ styles = self.affine(w) * self.weight_gain
+ x = modulated_conv2d(x=x, weight=self.weight, styles=styles, demodulate=False, fused_modconv=fused_modconv)
+ x = bias_act.bias_act(x, self.bias.to(x.dtype), clamp=self.conv_clamp)
+ return x
+
+#----------------------------------------------------------------------------
+
+class SynthesisBlock(torch.nn.Module):
+ def __init__(self,
+ in_channels, # Number of input channels, 0 = first block.
+ out_channels, # Number of output channels.
+ w_dim, # Intermediate latent (W) dimensionality.
+ resolution, # Resolution of this block.
+ img_channels, # Number of output color channels.
+ is_last, # Is this the last block?
+ architecture = 'skip', # Architecture: 'orig', 'skip', 'resnet'.
+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.
+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.
+ use_fp16 = False, # Use FP16 for this block?
+ fp16_channels_last = False, # Use channels-last memory format with FP16?
+ **layer_kwargs, # Arguments for SynthesisLayer.
+ ):
+ assert architecture in ['orig', 'skip', 'resnet']
+ super().__init__()
+ self.in_channels = in_channels
+ self.w_dim = w_dim
+ self.resolution = resolution
+ self.img_channels = img_channels
+ self.is_last = is_last
+ self.architecture = architecture
+ self.use_fp16 = use_fp16
+ self.channels_last = (use_fp16 and fp16_channels_last)
+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))
+ self.num_conv = 0
+ self.num_torgb = 0
+
+ if in_channels == 0:
+ self.const = torch.nn.Parameter(torch.randn([out_channels, resolution, resolution]))
+
+ if in_channels != 0:
+ self.conv0 = SynthesisLayer(in_channels, out_channels, w_dim=w_dim, resolution=resolution, up=2,
+ resample_filter=resample_filter, conv_clamp=conv_clamp, channels_last=self.channels_last, **layer_kwargs)
+ self.num_conv += 1
+
+ self.conv1 = SynthesisLayer(out_channels, out_channels, w_dim=w_dim, resolution=resolution,
+ conv_clamp=conv_clamp, channels_last=self.channels_last, **layer_kwargs)
+ self.num_conv += 1
+
+ if is_last or architecture == 'skip':
+ self.torgb = ToRGBLayer(out_channels, img_channels, w_dim=w_dim,
+ conv_clamp=conv_clamp, channels_last=self.channels_last)
+ self.num_torgb += 1
+
+ if in_channels != 0 and architecture == 'resnet':
+ self.skip = Conv2dLayer(in_channels, out_channels, kernel_size=1, bias=False, up=2,
+ resample_filter=resample_filter, channels_last=self.channels_last)
+
+ def forward(self, x, img, ws, force_fp32=False, fused_modconv=None, **layer_kwargs):
+ misc.assert_shape(ws, [None, self.num_conv + self.num_torgb, self.w_dim])
+ w_iter = iter(ws.unbind(dim=1))
+ dtype = torch.float16 if self.use_fp16 and not force_fp32 else torch.float32
+ memory_format = torch.channels_last if self.channels_last and not force_fp32 else torch.contiguous_format
+ if fused_modconv is None:
+ with misc.suppress_tracer_warnings(): # this value will be treated as a constant
+ fused_modconv = (not self.training) and (dtype == torch.float32 or int(x.shape[0]) == 1)
+
+ # Input.
+ if self.in_channels == 0:
+ x = self.const.to(dtype=dtype, memory_format=memory_format)
+ x = x.unsqueeze(0).repeat([ws.shape[0], 1, 1, 1])
+ else:
+ misc.assert_shape(x, [None, self.in_channels, self.resolution // 2, self.resolution // 2])
+ x = x.to(dtype=dtype, memory_format=memory_format)
+
+ # Main layers.
+ if self.in_channels == 0:
+ x = self.conv1(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)
+ elif self.architecture == 'resnet':
+ y = self.skip(x, gain=np.sqrt(0.5))
+ x = self.conv0(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)
+ x = self.conv1(x, next(w_iter), fused_modconv=fused_modconv, gain=np.sqrt(0.5), **layer_kwargs)
+ x = y.add_(x)
+ else:
+ x = self.conv0(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)
+ x = self.conv1(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)
+
+ # ToRGB.
+ if img is not None:
+ misc.assert_shape(img, [None, self.img_channels, self.resolution // 2, self.resolution // 2])
+ img = upfirdn2d.upsample2d(img, self.resample_filter)
+ if self.is_last or self.architecture == 'skip':
+ y = self.torgb(x, next(w_iter), fused_modconv=fused_modconv)
+ y = y.to(dtype=torch.float32, memory_format=torch.contiguous_format)
+ img = img.add_(y) if img is not None else y
+
+ assert x.dtype == dtype
+ assert img is None or img.dtype == torch.float32
+ return x, img
+
+#----------------------------------------------------------------------------
+
+class SynthesisNetwork(torch.nn.Module):
+ def __init__(self,
+ w_dim, # Intermediate latent (W) dimensionality.
+ img_resolution, # Output image resolution.
+ img_channels, # Number of color channels.
+ channel_base = 32768, # Overall multiplier for the number of channels.
+ channel_max = 512, # Maximum number of channels in any layer.
+ num_fp16_res = 0, # Use FP16 for the N highest resolutions.
+ **block_kwargs, # Arguments for SynthesisBlock.
+ ):
+ assert img_resolution >= 4 and img_resolution & (img_resolution - 1) == 0
+ super().__init__()
+ self.w_dim = w_dim
+ self.img_resolution = img_resolution
+ self.img_resolution_log2 = int(np.log2(img_resolution))
+ self.img_channels = img_channels
+ self.block_resolutions = [2 ** i for i in range(2, self.img_resolution_log2 + 1)]
+ channels_dict = {res: min(channel_base // res, channel_max) for res in self.block_resolutions}
+ fp16_resolution = max(2 ** (self.img_resolution_log2 + 1 - num_fp16_res), 8)
+
+ self.num_ws = 0
+ for res in self.block_resolutions:
+ in_channels = channels_dict[res // 2] if res > 4 else 0
+ out_channels = channels_dict[res]
+ use_fp16 = (res >= fp16_resolution)
+ is_last = (res == self.img_resolution)
+ block = SynthesisBlock(in_channels, out_channels, w_dim=w_dim, resolution=res,
+ img_channels=img_channels, is_last=is_last, use_fp16=use_fp16, **block_kwargs)
+ self.num_ws += block.num_conv
+ if is_last:
+ self.num_ws += block.num_torgb
+ setattr(self, f'b{res}', block)
+
+ def forward(self, ws, **block_kwargs):
+ block_ws = []
+ with torch.autograd.profiler.record_function('split_ws'):
+ misc.assert_shape(ws, [None, self.num_ws, self.w_dim])
+ ws = ws.to(torch.float32)
+ w_idx = 0
+ for res in self.block_resolutions:
+ block = getattr(self, f'b{res}')
+ block_ws.append(ws.narrow(1, w_idx, block.num_conv + block.num_torgb))
+ w_idx += block.num_conv
+
+ x = img = None
+ for res, cur_ws in zip(self.block_resolutions, block_ws):
+ block = getattr(self, f'b{res}')
+ x, img = block(x, img, cur_ws, **block_kwargs)
+ return img
+
+#----------------------------------------------------------------------------
+
+class Generator(torch.nn.Module):
+ def __init__(self,
+ z_dim, # Input latent (Z) dimensionality.
+ c_dim, # Conditioning label (C) dimensionality.
+ w_dim, # Intermediate latent (W) dimensionality.
+ img_resolution, # Output resolution.
+ img_channels, # Number of output color channels.
+ mapping_kwargs = {}, # Arguments for MappingNetwork.
+ synthesis_kwargs = {}, # Arguments for SynthesisNetwork.
+ ):
+ super().__init__()
+ self.z_dim = z_dim
+ self.c_dim = c_dim
+ self.w_dim = w_dim
+ self.img_resolution = img_resolution
+ self.img_channels = img_channels
+ self.synthesis = SynthesisNetwork(w_dim=w_dim, img_resolution=img_resolution, img_channels=img_channels, **synthesis_kwargs)
+ self.num_ws = self.synthesis.num_ws
+ self.mapping = MappingNetwork(z_dim=z_dim, c_dim=c_dim, w_dim=w_dim, num_ws=self.num_ws, **mapping_kwargs)
+
+ def forward(self, z, c, truncation_psi=1, truncation_cutoff=None, **synthesis_kwargs):
+ ws = self.mapping(z, c, truncation_psi=truncation_psi, truncation_cutoff=truncation_cutoff)
+ img = self.synthesis(ws, **synthesis_kwargs)
+ return img
+
+#----------------------------------------------------------------------------
+
+class DiscriminatorBlock(torch.nn.Module):
+ def __init__(self,
+ in_channels, # Number of input channels, 0 = first block.
+ tmp_channels, # Number of intermediate channels.
+ out_channels, # Number of output channels.
+ resolution, # Resolution of this block.
+ img_channels, # Number of input color channels.
+ first_layer_idx, # Index of the first layer.
+ architecture = 'resnet', # Architecture: 'orig', 'skip', 'resnet'.
+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.
+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.
+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.
+ use_fp16 = False, # Use FP16 for this block?
+ fp16_channels_last = False, # Use channels-last memory format with FP16?
+ freeze_layers = 0, # Freeze-D: Number of layers to freeze.
+ ):
+ assert in_channels in [0, tmp_channels]
+ assert architecture in ['orig', 'skip', 'resnet']
+ super().__init__()
+ self.in_channels = in_channels
+ self.resolution = resolution
+ self.img_channels = img_channels
+ self.first_layer_idx = first_layer_idx
+ self.architecture = architecture
+ self.use_fp16 = use_fp16
+ self.channels_last = (use_fp16 and fp16_channels_last)
+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))
+
+ self.num_layers = 0
+ def trainable_gen():
+ while True:
+ layer_idx = self.first_layer_idx + self.num_layers
+ trainable = (layer_idx >= freeze_layers)
+ self.num_layers += 1
+ yield trainable
+ trainable_iter = trainable_gen()
+
+ if in_channels == 0 or architecture == 'skip':
+ self.fromrgb = Conv2dLayer(img_channels, tmp_channels, kernel_size=1, activation=activation,
+ trainable=next(trainable_iter), conv_clamp=conv_clamp, channels_last=self.channels_last)
+
+ self.conv0 = Conv2dLayer(tmp_channels, tmp_channels, kernel_size=3, activation=activation,
+ trainable=next(trainable_iter), conv_clamp=conv_clamp, channels_last=self.channels_last)
+
+ self.conv1 = Conv2dLayer(tmp_channels, out_channels, kernel_size=3, activation=activation, down=2,
+ trainable=next(trainable_iter), resample_filter=resample_filter, conv_clamp=conv_clamp, channels_last=self.channels_last)
+
+ if architecture == 'resnet':
+ self.skip = Conv2dLayer(tmp_channels, out_channels, kernel_size=1, bias=False, down=2,
+ trainable=next(trainable_iter), resample_filter=resample_filter, channels_last=self.channels_last)
+
+ def forward(self, x, img, force_fp32=False):
+ dtype = torch.float16 if self.use_fp16 and not force_fp32 else torch.float32
+ memory_format = torch.channels_last if self.channels_last and not force_fp32 else torch.contiguous_format
+
+ # Input.
+ if x is not None:
+ misc.assert_shape(x, [None, self.in_channels, self.resolution, self.resolution])
+ x = x.to(dtype=dtype, memory_format=memory_format)
+
+ # FromRGB.
+ if self.in_channels == 0 or self.architecture == 'skip':
+ misc.assert_shape(img, [None, self.img_channels, self.resolution, self.resolution])
+ img = img.to(dtype=dtype, memory_format=memory_format)
+ y = self.fromrgb(img)
+ x = x + y if x is not None else y
+ img = upfirdn2d.downsample2d(img, self.resample_filter) if self.architecture == 'skip' else None
+
+ # Main layers.
+ if self.architecture == 'resnet':
+ y = self.skip(x, gain=np.sqrt(0.5))
+ x = self.conv0(x)
+ x = self.conv1(x, gain=np.sqrt(0.5))
+ x = y.add_(x)
+ else:
+ x = self.conv0(x)
+ x = self.conv1(x)
+
+ assert x.dtype == dtype
+ return x, img
+
+#----------------------------------------------------------------------------
+
+class MinibatchStdLayer(torch.nn.Module):
+ def __init__(self, group_size, num_channels=1):
+ super().__init__()
+ self.group_size = group_size
+ self.num_channels = num_channels
+
+ def forward(self, x):
+ N, C, H, W = x.shape
+ with misc.suppress_tracer_warnings(): # as_tensor results are registered as constants
+ G = torch.min(torch.as_tensor(self.group_size), torch.as_tensor(N)) if self.group_size is not None else N
+ F = self.num_channels
+ c = C // F
+
+ y = x.reshape(G, -1, F, c, H, W) # [GnFcHW] Split minibatch N into n groups of size G, and channels C into F groups of size c.
+ y = y - y.mean(dim=0) # [GnFcHW] Subtract mean over group.
+ y = y.square().mean(dim=0) # [nFcHW] Calc variance over group.
+ y = (y + 1e-8).sqrt() # [nFcHW] Calc stddev over group.
+ y = y.mean(dim=[2,3,4]) # [nF] Take average over channels and pixels.
+ y = y.reshape(-1, F, 1, 1) # [nF11] Add missing dimensions.
+ y = y.repeat(G, 1, H, W) # [NFHW] Replicate over group and pixels.
+ x = torch.cat([x, y], dim=1) # [NCHW] Append to input as new channels.
+ return x
+
+#----------------------------------------------------------------------------
+
+class DiscriminatorEpilogue(torch.nn.Module):
+ def __init__(self,
+ in_channels, # Number of input channels.
+ cmap_dim, # Dimensionality of mapped conditioning label, 0 = no label.
+ resolution, # Resolution of this block.
+ img_channels, # Number of input color channels.
+ architecture = 'resnet', # Architecture: 'orig', 'skip', 'resnet'.
+ mbstd_group_size = 4, # Group size for the minibatch standard deviation layer, None = entire minibatch.
+ mbstd_num_channels = 1, # Number of features for the minibatch standard deviation layer, 0 = disable.
+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.
+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.
+ ):
+ assert architecture in ['orig', 'skip', 'resnet']
+ super().__init__()
+ self.in_channels = in_channels
+ self.cmap_dim = cmap_dim
+ self.resolution = resolution
+ self.img_channels = img_channels
+ self.architecture = architecture
+
+ if architecture == 'skip':
+ self.fromrgb = Conv2dLayer(img_channels, in_channels, kernel_size=1, activation=activation)
+ self.mbstd = MinibatchStdLayer(group_size=mbstd_group_size, num_channels=mbstd_num_channels) if mbstd_num_channels > 0 else None
+ self.conv = Conv2dLayer(in_channels + mbstd_num_channels, in_channels, kernel_size=3, activation=activation, conv_clamp=conv_clamp)
+ self.fc = FullyConnectedLayer(in_channels * (resolution ** 2), in_channels, activation=activation)
+ self.out = FullyConnectedLayer(in_channels, 1 if cmap_dim == 0 else cmap_dim)
+
+ def forward(self, x, img, cmap, force_fp32=False):
+ misc.assert_shape(x, [None, self.in_channels, self.resolution, self.resolution]) # [NCHW]
+ _ = force_fp32 # unused
+ dtype = torch.float32
+ memory_format = torch.contiguous_format
+
+ # FromRGB.
+ x = x.to(dtype=dtype, memory_format=memory_format)
+ if self.architecture == 'skip':
+ misc.assert_shape(img, [None, self.img_channels, self.resolution, self.resolution])
+ img = img.to(dtype=dtype, memory_format=memory_format)
+ x = x + self.fromrgb(img)
+
+ # Main layers.
+ if self.mbstd is not None:
+ x = self.mbstd(x)
+ x = self.conv(x)
+ x = self.fc(x.flatten(1))
+ x = self.out(x)
+
+ # Conditioning.
+ if self.cmap_dim > 0:
+ misc.assert_shape(cmap, [None, self.cmap_dim])
+ x = (x * cmap).sum(dim=1, keepdim=True) * (1 / np.sqrt(self.cmap_dim))
+
+ # assert x.dtype == dtype
+ return x
+
+#----------------------------------------------------------------------------
+
+class Discriminator(torch.nn.Module):
+ def __init__(self,
+ c_dim=0, # Conditioning label (C) dimensionality.
+ img_resolution=512, # Input resolution.
+ img_channels=3, # Number of input color channels.
+ temb_channels=512, # Number of time embedding channels
+ architecture = 'resnet', # Architecture: 'orig', 'skip', 'resnet'.
+ channel_base = 32768, # Overall multiplier for the number of channels.
+ channel_max = 512, # Maximum number of channels in any layer.
+ num_fp16_res = 0, # Use FP16 for the N highest resolutions.
+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.
+ cmap_dim = None, # Dimensionality of mapped conditioning label, None = default.
+ block_kwargs = {}, # Arguments for DiscriminatorBlock.
+ mapping_kwargs = {}, # Arguments for MappingNetwork.
+ epilogue_kwargs = {}, # Arguments for DiscriminatorEpilogue.
+ ):
+ super().__init__()
+ self.c_dim = c_dim
+ self.img_resolution = img_resolution
+ self.img_resolution_log2 = int(np.log2(img_resolution))
+ self.img_channels = img_channels
+ self.block_resolutions = [2 ** i for i in range(self.img_resolution_log2, 2, -1)]
+ channels_dict = {res: min(channel_base // res, channel_max) for res in self.block_resolutions + [4]}
+ fp16_resolution = max(2 ** (self.img_resolution_log2 + 1 - num_fp16_res), 8)
+
+ if cmap_dim is None:
+ cmap_dim = channels_dict[4]
+ if c_dim == 0:
+ cmap_dim = 0
+
+ self.time_emb_proj = nn.ModuleList()
+ common_kwargs = dict(img_channels=img_channels, architecture=architecture, conv_clamp=conv_clamp)
+ cur_layer_idx = 0
+ for res in self.block_resolutions:
+ in_channels = channels_dict[res] if res < img_resolution else 0
+ tmp_channels = channels_dict[res]
+ out_channels = channels_dict[res // 2]
+ use_fp16 = (res >= fp16_resolution)
+ block = DiscriminatorBlock(in_channels, tmp_channels, out_channels, resolution=res,
+ first_layer_idx=cur_layer_idx, use_fp16=use_fp16, **block_kwargs, **common_kwargs)
+ setattr(self, f'b{res}', block)
+ cur_layer_idx += block.num_layers
+ self.time_emb_proj.append(zero_module(nn.Linear(temb_channels, out_channels)))
+ if c_dim > 0:
+ self.mapping = MappingNetwork(z_dim=0, c_dim=c_dim, w_dim=cmap_dim, num_ws=None, w_avg_beta=None, **mapping_kwargs)
+ self.b4 = DiscriminatorEpilogue(channels_dict[4], cmap_dim=cmap_dim, resolution=4, **epilogue_kwargs, **common_kwargs)
+
+ self.nonlinearity = nn.SiLU()
+ self.time_proj = Timesteps(temb_channels, True, 0)
+
+ self.time_embedding = TimestepEmbedding(temb_channels, temb_channels)
+
+ def _pass(self, img, c, timesteps, **block_kwargs):
+ x = None
+ t_emb = self.time_proj(timesteps)
+ t_emb = t_emb.to(dtype=img.dtype)
+ t_emb = self.time_embedding(t_emb)
+ for i, res in enumerate(self.block_resolutions):
+ block = getattr(self, f'b{res}')
+ x, img = block(x, img, **block_kwargs)
+ temb = self.time_emb_proj[i](self.nonlinearity(t_emb))[:, :, None, None]
+ x += temb
+
+ cmap = None
+ if self.c_dim > 0:
+ cmap = self.mapping(None, c)
+ x = self.b4(x, img, cmap)
+ return x
+
+ def forward_G(self, fake, real, timesteps, c_fake=None, c_real=None):
+
+ fake_score = self._pass(fake, c_fake, timesteps)
+ adv_loss = F.softplus(-fake_score).mean() # Hinge G
+ return adv_loss
+
+ # def r1_penalty(self, real, c_real=None, timesteps):
+ # real.requires_grad_(True)
+ # real_score, _ = self._pass(real, c_real, timesteps)
+ # grad_real = torch.autograd.grad(outputs=real_score.sum(),
+ # inputs=real,
+ # create_graph=True,
+ # retain_graph=True,
+ # only_inputs=True)[0]
+ # r1 = grad_real.pow(2).reshape(grad_real.size(0), -1).sum(1).mean()
+ # return r1
+
+ def forward_D(self, fake, real, timesteps, c_fake=None, c_real=None, global_step: int = 0):
+ fake = fake.detach()
+
+ fake_score = self._pass(fake, c_fake, timesteps)
+ real_score = self._pass(real, c_real, timesteps)
+
+ loss_D = (F.softplus(fake_score) + F.softplus(-real_score)).mean()
+
+ # R1 regularization (lazy)
+ # if self.r1_gamma > 0 and self.r1_interval > 0 and (global_step % self.r1_interval == 0):
+ # r1 = self.r1_penalty(real, c_real, timesteps)
+ # loss_D = loss_D + (self.r1_gamma * 0.5) * r1
+ return loss_D
+
+ def forward(self, fake, real, timesteps, gen=True, **kwargs):
+ if gen:
+ return self.forward_G(fake, real, timesteps, **kwargs)
+ else:
+ return self.forward_D(fake, real, timesteps, **kwargs)
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/dnnlib/__init__.py b/src/stylegan2/dnnlib/__init__.py
new file mode 100644
index 0000000..2f08cf3
--- /dev/null
+++ b/src/stylegan2/dnnlib/__init__.py
@@ -0,0 +1,9 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+from .util import EasyDict, make_cache_dir_path
diff --git a/src/stylegan2/dnnlib/util.py b/src/stylegan2/dnnlib/util.py
new file mode 100644
index 0000000..c611acc
--- /dev/null
+++ b/src/stylegan2/dnnlib/util.py
@@ -0,0 +1,468 @@
+
+import ctypes
+import fnmatch
+import importlib
+import inspect
+import numpy as np
+import os
+import shutil
+import sys
+import types
+import io
+import pickle
+import re
+import requests
+import html
+import hashlib
+import glob
+import tempfile
+import urllib
+import urllib.request
+import uuid
+
+from distutils.util import strtobool
+from typing import Any, List, Tuple, Union
+
+
+# Util classes
+# ------------------------------------------------------------------------------------------
+
+
+class EasyDict(dict):
+ """Convenience class that behaves like a dict but allows access with the attribute syntax."""
+
+ def __getattr__(self, name: str) -> Any:
+ try:
+ return self[name]
+ except KeyError:
+ raise AttributeError(name)
+
+ def __setattr__(self, name: str, value: Any) -> None:
+ self[name] = value
+
+ def __delattr__(self, name: str) -> None:
+ del self[name]
+
+
+class Logger(object):
+ """Redirect stderr to stdout, optionally print stdout to a file, and optionally force flushing on both stdout and the file."""
+
+ def __init__(self, file_name: str = None, file_mode: str = "w", should_flush: bool = True):
+ self.file = None
+
+ if file_name is not None:
+ self.file = open(file_name, file_mode)
+
+ self.should_flush = should_flush
+ self.stdout = sys.stdout
+ self.stderr = sys.stderr
+
+ sys.stdout = self
+ sys.stderr = self
+
+ def __enter__(self) -> "Logger":
+ return self
+
+ def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
+ self.close()
+
+ def write(self, text: Union[str, bytes]) -> None:
+ """Write text to stdout (and a file) and optionally flush."""
+ if isinstance(text, bytes):
+ text = text.decode()
+ if len(text) == 0: # workaround for a bug in VSCode debugger: sys.stdout.write(''); sys.stdout.flush() => crash
+ return
+
+ if self.file is not None:
+ self.file.write(text)
+
+ self.stdout.write(text)
+
+ if self.should_flush:
+ self.flush()
+
+ def flush(self) -> None:
+ """Flush written text to both stdout and a file, if open."""
+ if self.file is not None:
+ self.file.flush()
+
+ self.stdout.flush()
+
+ def close(self) -> None:
+ """Flush, close possible files, and remove stdout/stderr mirroring."""
+ self.flush()
+
+ # if using multiple loggers, prevent closing in wrong order
+ if sys.stdout is self:
+ sys.stdout = self.stdout
+ if sys.stderr is self:
+ sys.stderr = self.stderr
+
+ if self.file is not None:
+ self.file.close()
+ self.file = None
+
+
+# Cache directories
+# ------------------------------------------------------------------------------------------
+
+_dnnlib_cache_dir = None
+
+def set_cache_dir(path: str) -> None:
+ global _dnnlib_cache_dir
+ _dnnlib_cache_dir = path
+
+def make_cache_dir_path(*paths: str) -> str:
+ if _dnnlib_cache_dir is not None:
+ return os.path.join(_dnnlib_cache_dir, *paths)
+ if 'DNNLIB_CACHE_DIR' in os.environ:
+ return os.path.join(os.environ['DNNLIB_CACHE_DIR'], *paths)
+ if 'HOME' in os.environ:
+ return os.path.join(os.environ['HOME'], '.cache', 'dnnlib', *paths)
+ if 'USERPROFILE' in os.environ:
+ return os.path.join(os.environ['USERPROFILE'], '.cache', 'dnnlib', *paths)
+ return os.path.join(tempfile.gettempdir(), '.cache', 'dnnlib', *paths)
+
+# Small util functions
+# ------------------------------------------------------------------------------------------
+
+
+def format_time(seconds: Union[int, float]) -> str:
+ """Convert the seconds to human readable string with days, hours, minutes and seconds."""
+ s = int(np.rint(seconds))
+
+ if s < 60:
+ return "{0}s".format(s)
+ elif s < 60 * 60:
+ return "{0}m {1:02}s".format(s // 60, s % 60)
+ elif s < 24 * 60 * 60:
+ return "{0}h {1:02}m {2:02}s".format(s // (60 * 60), (s // 60) % 60, s % 60)
+ else:
+ return "{0}d {1:02}h {2:02}m".format(s // (24 * 60 * 60), (s // (60 * 60)) % 24, (s // 60) % 60)
+
+
+def ask_yes_no(question: str) -> bool:
+ """Ask the user the question until the user inputs a valid answer."""
+ while True:
+ try:
+ print("{0} [y/n]".format(question))
+ return strtobool(input().lower())
+ except ValueError:
+ pass
+
+
+def tuple_product(t: Tuple) -> Any:
+ """Calculate the product of the tuple elements."""
+ result = 1
+
+ for v in t:
+ result *= v
+
+ return result
+
+
+_str_to_ctype = {
+ "uint8": ctypes.c_ubyte,
+ "uint16": ctypes.c_uint16,
+ "uint32": ctypes.c_uint32,
+ "uint64": ctypes.c_uint64,
+ "int8": ctypes.c_byte,
+ "int16": ctypes.c_int16,
+ "int32": ctypes.c_int32,
+ "int64": ctypes.c_int64,
+ "float32": ctypes.c_float,
+ "float64": ctypes.c_double
+}
+
+
+def get_dtype_and_ctype(type_obj: Any) -> Tuple[np.dtype, Any]:
+ """Given a type name string (or an object having a __name__ attribute), return matching Numpy and ctypes types that have the same size in bytes."""
+ type_str = None
+
+ if isinstance(type_obj, str):
+ type_str = type_obj
+ elif hasattr(type_obj, "__name__"):
+ type_str = type_obj.__name__
+ elif hasattr(type_obj, "name"):
+ type_str = type_obj.name
+ else:
+ raise RuntimeError("Cannot infer type name from input")
+
+ assert type_str in _str_to_ctype.keys()
+
+ my_dtype = np.dtype(type_str)
+ my_ctype = _str_to_ctype[type_str]
+
+ assert my_dtype.itemsize == ctypes.sizeof(my_ctype)
+
+ return my_dtype, my_ctype
+
+
+def is_pickleable(obj: Any) -> bool:
+ try:
+ with io.BytesIO() as stream:
+ pickle.dump(obj, stream)
+ return True
+ except:
+ return False
+
+
+# Functionality to import modules/objects by name, and call functions by name
+# ------------------------------------------------------------------------------------------
+
+def get_module_from_obj_name(obj_name: str) -> Tuple[types.ModuleType, str]:
+ """Searches for the underlying module behind the name to some python object.
+ Returns the module and the object name (original name with module part removed)."""
+
+ # allow convenience shorthands, substitute them by full names
+ obj_name = re.sub("^np.", "numpy.", obj_name)
+ obj_name = re.sub("^tf.", "tensorflow.", obj_name)
+
+ # list alternatives for (module_name, local_obj_name)
+ parts = obj_name.split(".")
+ name_pairs = [(".".join(parts[:i]), ".".join(parts[i:])) for i in range(len(parts), 0, -1)]
+
+ # try each alternative in turn
+ for module_name, local_obj_name in name_pairs:
+ try:
+ module = importlib.import_module(module_name) # may raise ImportError
+ get_obj_from_module(module, local_obj_name) # may raise AttributeError
+ return module, local_obj_name
+ except:
+ pass
+
+ # maybe some of the modules themselves contain errors?
+ for module_name, _local_obj_name in name_pairs:
+ try:
+ importlib.import_module(module_name) # may raise ImportError
+ except ImportError:
+ if not str(sys.exc_info()[1]).startswith("No module named '" + module_name + "'"):
+ raise
+
+ # maybe the requested attribute is missing?
+ for module_name, local_obj_name in name_pairs:
+ try:
+ module = importlib.import_module(module_name) # may raise ImportError
+ get_obj_from_module(module, local_obj_name) # may raise AttributeError
+ except ImportError:
+ pass
+
+ # we are out of luck, but we have no idea why
+ raise ImportError(obj_name)
+
+
+def get_obj_from_module(module: types.ModuleType, obj_name: str) -> Any:
+ """Traverses the object name and returns the last (rightmost) python object."""
+ if obj_name == '':
+ return module
+ obj = module
+ for part in obj_name.split("."):
+ obj = getattr(obj, part)
+ return obj
+
+
+def get_obj_by_name(name: str) -> Any:
+ """Finds the python object with the given name."""
+ module, obj_name = get_module_from_obj_name(name)
+ return get_obj_from_module(module, obj_name)
+
+
+def call_func_by_name(*args, func_name: str = None, **kwargs) -> Any:
+ """Finds the python object with the given name and calls it as a function."""
+ assert func_name is not None
+ func_obj = get_obj_by_name(func_name)
+ assert callable(func_obj)
+ return func_obj(*args, **kwargs)
+
+
+def construct_class_by_name(*args, class_name: str = None, **kwargs) -> Any:
+ """Finds the python class with the given name and constructs it with the given arguments."""
+ return call_func_by_name(*args, func_name=class_name, **kwargs)
+
+
+def get_module_dir_by_obj_name(obj_name: str) -> str:
+ """Get the directory path of the module containing the given object name."""
+ module, _ = get_module_from_obj_name(obj_name)
+ return os.path.dirname(inspect.getfile(module))
+
+
+def is_top_level_function(obj: Any) -> bool:
+ """Determine whether the given object is a top-level function, i.e., defined at module scope using 'def'."""
+ return callable(obj) and obj.__name__ in sys.modules[obj.__module__].__dict__
+
+
+def get_top_level_function_name(obj: Any) -> str:
+ """Return the fully-qualified name of a top-level function."""
+ assert is_top_level_function(obj)
+ module = obj.__module__
+ if module == '__main__':
+ module = os.path.splitext(os.path.basename(sys.modules[module].__file__))[0]
+ return module + "." + obj.__name__
+
+
+# File system helpers
+# ------------------------------------------------------------------------------------------
+
+def list_dir_recursively_with_ignore(dir_path: str, ignores: List[str] = None, add_base_to_relative: bool = False) -> List[Tuple[str, str]]:
+ """List all files recursively in a given directory while ignoring given file and directory names.
+ Returns list of tuples containing both absolute and relative paths."""
+ assert os.path.isdir(dir_path)
+ base_name = os.path.basename(os.path.normpath(dir_path))
+
+ if ignores is None:
+ ignores = []
+
+ result = []
+
+ for root, dirs, files in os.walk(dir_path, topdown=True):
+ for ignore_ in ignores:
+ dirs_to_remove = [d for d in dirs if fnmatch.fnmatch(d, ignore_)]
+
+ # dirs need to be edited in-place
+ for d in dirs_to_remove:
+ dirs.remove(d)
+
+ files = [f for f in files if not fnmatch.fnmatch(f, ignore_)]
+
+ absolute_paths = [os.path.join(root, f) for f in files]
+ relative_paths = [os.path.relpath(p, dir_path) for p in absolute_paths]
+
+ if add_base_to_relative:
+ relative_paths = [os.path.join(base_name, p) for p in relative_paths]
+
+ assert len(absolute_paths) == len(relative_paths)
+ result += zip(absolute_paths, relative_paths)
+
+ return result
+
+
+def copy_files_and_create_dirs(files: List[Tuple[str, str]]) -> None:
+ """Takes in a list of tuples of (src, dst) paths and copies files.
+ Will create all necessary directories."""
+ for file in files:
+ target_dir_name = os.path.dirname(file[1])
+
+ # will create all intermediate-level directories
+ if not os.path.exists(target_dir_name):
+ os.makedirs(target_dir_name)
+
+ shutil.copyfile(file[0], file[1])
+
+
+# URL helpers
+# ------------------------------------------------------------------------------------------
+
+def is_url(obj: Any, allow_file_urls: bool = False) -> bool:
+ """Determine whether the given object is a valid URL string."""
+ if not isinstance(obj, str) or not "://" in obj:
+ return False
+ if allow_file_urls and obj.startswith('file://'):
+ return True
+ try:
+ res = requests.compat.urlparse(obj)
+ if not res.scheme or not res.netloc or not "." in res.netloc:
+ return False
+ res = requests.compat.urlparse(requests.compat.urljoin(obj, "/"))
+ if not res.scheme or not res.netloc or not "." in res.netloc:
+ return False
+ except:
+ return False
+ return True
+
+
+def open_url(url: str, cache_dir: str = None, num_attempts: int = 10, verbose: bool = True, return_filename: bool = False, cache: bool = True) -> Any:
+ """Download the given URL and return a binary-mode file object to access the data."""
+ assert num_attempts >= 1
+ assert not (return_filename and (not cache))
+
+ # Doesn't look like an URL scheme so interpret it as a local filename.
+ if not re.match('^[a-z]+://', url):
+ return url if return_filename else open(url, "rb")
+
+ # Handle file URLs. This code handles unusual file:// patterns that
+ # arise on Windows:
+ #
+ # file:///c:/foo.txt
+ #
+ # which would translate to a local '/c:/foo.txt' filename that's
+ # invalid. Drop the forward slash for such pathnames.
+ #
+ # If you touch this code path, you should test it on both Linux and
+ # Windows.
+ #
+ # Some internet resources suggest using urllib.request.url2pathname() but
+ # but that converts forward slashes to backslashes and this causes
+ # its own set of problems.
+ if url.startswith('file://'):
+ filename = urllib.parse.urlparse(url).path
+ if re.match(r'^/[a-zA-Z]:', filename):
+ filename = filename[1:]
+ return filename if return_filename else open(filename, "rb")
+
+ assert is_url(url)
+
+ # Lookup from cache.
+ if cache_dir is None:
+ cache_dir = make_cache_dir_path('downloads')
+
+ url_md5 = hashlib.md5(url.encode("utf-8")).hexdigest()
+ if cache:
+ cache_files = glob.glob(os.path.join(cache_dir, url_md5 + "_*"))
+ if len(cache_files) == 1:
+ filename = cache_files[0]
+ return filename if return_filename else open(filename, "rb")
+
+ # Download.
+ url_name = None
+ url_data = None
+ with requests.Session() as session:
+ if verbose:
+ print("Downloading %s ..." % url, end="", flush=True)
+ for attempts_left in reversed(range(num_attempts)):
+ try:
+ with session.get(url) as res:
+ res.raise_for_status()
+ if len(res.content) == 0:
+ raise IOError("No data received")
+
+ if len(res.content) < 8192:
+ content_str = res.content.decode("utf-8")
+ if "download_warning" in res.headers.get("Set-Cookie", ""):
+ links = [html.unescape(link) for link in content_str.split('"') if "export=download" in link]
+ if len(links) == 1:
+ url = requests.compat.urljoin(url, links[0])
+ raise IOError("Google Drive virus checker nag")
+ if "Google Drive - Quota exceeded" in content_str:
+ raise IOError("Google Drive download quota exceeded -- please try again later")
+
+ match = re.search(r'filename="([^"]*)"', res.headers.get("Content-Disposition", ""))
+ url_name = match[1] if match else url
+ url_data = res.content
+ if verbose:
+ print(" done")
+ break
+ except KeyboardInterrupt:
+ raise
+ except:
+ if not attempts_left:
+ if verbose:
+ print(" failed")
+ raise
+ if verbose:
+ print(".", end="", flush=True)
+
+ # Save to cache.
+ if cache:
+ safe_name = re.sub(r"[^0-9a-zA-Z-._]", "_", url_name)
+ cache_file = os.path.join(cache_dir, url_md5 + "_" + safe_name)
+ temp_file = os.path.join(cache_dir, "tmp_" + uuid.uuid4().hex + "_" + url_md5 + "_" + safe_name)
+ os.makedirs(cache_dir, exist_ok=True)
+ with open(temp_file, "wb") as f:
+ f.write(url_data)
+ os.replace(temp_file, cache_file) # atomic
+ if return_filename:
+ return cache_file
+
+ # Return data as file object.
+ assert not return_filename
+ return io.BytesIO(url_data)
diff --git a/src/stylegan2/torch_utils/__init__.py b/src/stylegan2/torch_utils/__init__.py
new file mode 100644
index 0000000..ece0ea0
--- /dev/null
+++ b/src/stylegan2/torch_utils/__init__.py
@@ -0,0 +1,9 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+# empty
diff --git a/src/stylegan2/torch_utils/custom_ops.py b/src/stylegan2/torch_utils/custom_ops.py
new file mode 100644
index 0000000..4cc4e43
--- /dev/null
+++ b/src/stylegan2/torch_utils/custom_ops.py
@@ -0,0 +1,126 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+import os
+import glob
+import torch
+import torch.utils.cpp_extension
+import importlib
+import hashlib
+import shutil
+from pathlib import Path
+
+from torch.utils.file_baton import FileBaton
+
+#----------------------------------------------------------------------------
+# Global options.
+
+verbosity = 'brief' # Verbosity level: 'none', 'brief', 'full'
+
+#----------------------------------------------------------------------------
+# Internal helper funcs.
+
+def _find_compiler_bindir():
+ patterns = [
+ 'C:/Program Files (x86)/Microsoft Visual Studio/*/Professional/VC/Tools/MSVC/*/bin/Hostx64/x64',
+ 'C:/Program Files (x86)/Microsoft Visual Studio/*/BuildTools/VC/Tools/MSVC/*/bin/Hostx64/x64',
+ 'C:/Program Files (x86)/Microsoft Visual Studio/*/Community/VC/Tools/MSVC/*/bin/Hostx64/x64',
+ 'C:/Program Files (x86)/Microsoft Visual Studio */vc/bin',
+ ]
+ for pattern in patterns:
+ matches = sorted(glob.glob(pattern))
+ if len(matches):
+ return matches[-1]
+ return None
+
+#----------------------------------------------------------------------------
+# Main entry point for compiling and loading C++/CUDA plugins.
+
+_cached_plugins = dict()
+
+def get_plugin(module_name, sources, **build_kwargs):
+ assert verbosity in ['none', 'brief', 'full']
+
+ # Already cached?
+ if module_name in _cached_plugins:
+ return _cached_plugins[module_name]
+
+ # Print status.
+ if verbosity == 'full':
+ print(f'Setting up PyTorch plugin "{module_name}"...')
+ elif verbosity == 'brief':
+ print(f'Setting up PyTorch plugin "{module_name}"... ', end='', flush=True)
+
+ try: # pylint: disable=too-many-nested-blocks
+ # Make sure we can find the necessary compiler binaries.
+ if os.name == 'nt' and os.system("where cl.exe >nul 2>nul") != 0:
+ compiler_bindir = _find_compiler_bindir()
+ if compiler_bindir is None:
+ raise RuntimeError(f'Could not find MSVC/GCC/CLANG installation on this computer. Check _find_compiler_bindir() in "{__file__}".')
+ os.environ['PATH'] += ';' + compiler_bindir
+
+ # Compile and load.
+ verbose_build = (verbosity == 'full')
+
+ # Incremental build md5sum trickery. Copies all the input source files
+ # into a cached build directory under a combined md5 digest of the input
+ # source files. Copying is done only if the combined digest has changed.
+ # This keeps input file timestamps and filenames the same as in previous
+ # extension builds, allowing for fast incremental rebuilds.
+ #
+ # This optimization is done only in case all the source files reside in
+ # a single directory (just for simplicity) and if the TORCH_EXTENSIONS_DIR
+ # environment variable is set (we take this as a signal that the user
+ # actually cares about this.)
+ source_dirs_set = set(os.path.dirname(source) for source in sources)
+ if len(source_dirs_set) == 1 and ('TORCH_EXTENSIONS_DIR' in os.environ):
+ all_source_files = sorted(list(x for x in Path(list(source_dirs_set)[0]).iterdir() if x.is_file()))
+
+ # Compute a combined hash digest for all source files in the same
+ # custom op directory (usually .cu, .cpp, .py and .h files).
+ hash_md5 = hashlib.md5()
+ for src in all_source_files:
+ with open(src, 'rb') as f:
+ hash_md5.update(f.read())
+ build_dir = torch.utils.cpp_extension._get_build_directory(module_name, verbose=verbose_build) # pylint: disable=protected-access
+ digest_build_dir = os.path.join(build_dir, hash_md5.hexdigest())
+
+ if not os.path.isdir(digest_build_dir):
+ os.makedirs(digest_build_dir, exist_ok=True)
+ baton = FileBaton(os.path.join(digest_build_dir, 'lock'))
+ if baton.try_acquire():
+ try:
+ for src in all_source_files:
+ shutil.copyfile(src, os.path.join(digest_build_dir, os.path.basename(src)))
+ finally:
+ baton.release()
+ else:
+ # Someone else is copying source files under the digest dir,
+ # wait until done and continue.
+ baton.wait()
+ digest_sources = [os.path.join(digest_build_dir, os.path.basename(x)) for x in sources]
+ torch.utils.cpp_extension.load(name=module_name, build_directory=build_dir,
+ verbose=verbose_build, sources=digest_sources, **build_kwargs)
+ else:
+ torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
+ module = importlib.import_module(module_name)
+
+ except:
+ if verbosity == 'brief':
+ print('Failed!')
+ raise
+
+ # Print status and add to cache.
+ if verbosity == 'full':
+ print(f'Done setting up PyTorch plugin "{module_name}".')
+ elif verbosity == 'brief':
+ print('Done.')
+ _cached_plugins[module_name] = module
+ return module
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/misc.py b/src/stylegan2/torch_utils/misc.py
new file mode 100644
index 0000000..7829f4d
--- /dev/null
+++ b/src/stylegan2/torch_utils/misc.py
@@ -0,0 +1,262 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+import re
+import contextlib
+import numpy as np
+import torch
+import warnings
+import dnnlib
+
+#----------------------------------------------------------------------------
+# Cached construction of constant tensors. Avoids CPU=>GPU copy when the
+# same constant is used multiple times.
+
+_constant_cache = dict()
+
+def constant(value, shape=None, dtype=None, device=None, memory_format=None):
+ value = np.asarray(value)
+ if shape is not None:
+ shape = tuple(shape)
+ if dtype is None:
+ dtype = torch.get_default_dtype()
+ if device is None:
+ device = torch.device('cpu')
+ if memory_format is None:
+ memory_format = torch.contiguous_format
+
+ key = (value.shape, value.dtype, value.tobytes(), shape, dtype, device, memory_format)
+ tensor = _constant_cache.get(key, None)
+ if tensor is None:
+ tensor = torch.as_tensor(value.copy(), dtype=dtype, device=device)
+ if shape is not None:
+ tensor, _ = torch.broadcast_tensors(tensor, torch.empty(shape))
+ tensor = tensor.contiguous(memory_format=memory_format)
+ _constant_cache[key] = tensor
+ return tensor
+
+#----------------------------------------------------------------------------
+# Replace NaN/Inf with specified numerical values.
+
+try:
+ nan_to_num = torch.nan_to_num # 1.8.0a0
+except AttributeError:
+ def nan_to_num(input, nan=0.0, posinf=None, neginf=None, *, out=None): # pylint: disable=redefined-builtin
+ assert isinstance(input, torch.Tensor)
+ if posinf is None:
+ posinf = torch.finfo(input.dtype).max
+ if neginf is None:
+ neginf = torch.finfo(input.dtype).min
+ assert nan == 0
+ return torch.clamp(input.unsqueeze(0).nansum(0), min=neginf, max=posinf, out=out)
+
+#----------------------------------------------------------------------------
+# Symbolic assert.
+
+try:
+ symbolic_assert = torch._assert # 1.8.0a0 # pylint: disable=protected-access
+except AttributeError:
+ symbolic_assert = torch.Assert # 1.7.0
+
+#----------------------------------------------------------------------------
+# Context manager to suppress known warnings in torch.jit.trace().
+
+class suppress_tracer_warnings(warnings.catch_warnings):
+ def __enter__(self):
+ super().__enter__()
+ warnings.simplefilter('ignore', category=torch.jit.TracerWarning)
+ return self
+
+#----------------------------------------------------------------------------
+# Assert that the shape of a tensor matches the given list of integers.
+# None indicates that the size of a dimension is allowed to vary.
+# Performs symbolic assertion when used in torch.jit.trace().
+
+def assert_shape(tensor, ref_shape):
+ if tensor.ndim != len(ref_shape):
+ raise AssertionError(f'Wrong number of dimensions: got {tensor.ndim}, expected {len(ref_shape)}')
+ for idx, (size, ref_size) in enumerate(zip(tensor.shape, ref_shape)):
+ if ref_size is None:
+ pass
+ elif isinstance(ref_size, torch.Tensor):
+ with suppress_tracer_warnings(): # as_tensor results are registered as constants
+ symbolic_assert(torch.equal(torch.as_tensor(size), ref_size), f'Wrong size for dimension {idx}')
+ elif isinstance(size, torch.Tensor):
+ with suppress_tracer_warnings(): # as_tensor results are registered as constants
+ symbolic_assert(torch.equal(size, torch.as_tensor(ref_size)), f'Wrong size for dimension {idx}: expected {ref_size}')
+ elif size != ref_size:
+ raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}')
+
+#----------------------------------------------------------------------------
+# Function decorator that calls torch.autograd.profiler.record_function().
+
+def profiled_function(fn):
+ def decorator(*args, **kwargs):
+ with torch.autograd.profiler.record_function(fn.__name__):
+ return fn(*args, **kwargs)
+ decorator.__name__ = fn.__name__
+ return decorator
+
+#----------------------------------------------------------------------------
+# Sampler for torch.utils.data.DataLoader that loops over the dataset
+# indefinitely, shuffling items as it goes.
+
+class InfiniteSampler(torch.utils.data.Sampler):
+ def __init__(self, dataset, rank=0, num_replicas=1, shuffle=True, seed=0, window_size=0.5):
+ assert len(dataset) > 0
+ assert num_replicas > 0
+ assert 0 <= rank < num_replicas
+ assert 0 <= window_size <= 1
+ super().__init__(dataset)
+ self.dataset = dataset
+ self.rank = rank
+ self.num_replicas = num_replicas
+ self.shuffle = shuffle
+ self.seed = seed
+ self.window_size = window_size
+
+ def __iter__(self):
+ order = np.arange(len(self.dataset))
+ rnd = None
+ window = 0
+ if self.shuffle:
+ rnd = np.random.RandomState(self.seed)
+ rnd.shuffle(order)
+ window = int(np.rint(order.size * self.window_size))
+
+ idx = 0
+ while True:
+ i = idx % order.size
+ if idx % self.num_replicas == self.rank:
+ yield order[i]
+ if window >= 2:
+ j = (i - rnd.randint(window)) % order.size
+ order[i], order[j] = order[j], order[i]
+ idx += 1
+
+#----------------------------------------------------------------------------
+# Utilities for operating with torch.nn.Module parameters and buffers.
+
+def params_and_buffers(module):
+ assert isinstance(module, torch.nn.Module)
+ return list(module.parameters()) + list(module.buffers())
+
+def named_params_and_buffers(module):
+ assert isinstance(module, torch.nn.Module)
+ return list(module.named_parameters()) + list(module.named_buffers())
+
+def copy_params_and_buffers(src_module, dst_module, require_all=False):
+ assert isinstance(src_module, torch.nn.Module)
+ assert isinstance(dst_module, torch.nn.Module)
+ src_tensors = {name: tensor for name, tensor in named_params_and_buffers(src_module)}
+ for name, tensor in named_params_and_buffers(dst_module):
+ assert (name in src_tensors) or (not require_all)
+ if name in src_tensors:
+ tensor.copy_(src_tensors[name].detach()).requires_grad_(tensor.requires_grad)
+
+#----------------------------------------------------------------------------
+# Context manager for easily enabling/disabling DistributedDataParallel
+# synchronization.
+
+def ddp_sync(module, sync):
+ assert isinstance(module, torch.nn.Module)
+ if sync or not isinstance(module, torch.nn.parallel.DistributedDataParallel):
+ yield
+ else:
+ with module.no_sync():
+ yield
+
+#----------------------------------------------------------------------------
+# Check DistributedDataParallel consistency across processes.
+
+def check_ddp_consistency(module, ignore_regex=None):
+ assert isinstance(module, torch.nn.Module)
+ for name, tensor in named_params_and_buffers(module):
+ fullname = type(module).__name__ + '.' + name
+ if ignore_regex is not None and re.fullmatch(ignore_regex, fullname):
+ continue
+ tensor = tensor.detach()
+ other = tensor.clone()
+ torch.distributed.broadcast(tensor=other, src=0)
+ assert (nan_to_num(tensor) == nan_to_num(other)).all(), fullname
+
+#----------------------------------------------------------------------------
+# Print summary table of module hierarchy.
+
+def print_module_summary(module, inputs, max_nesting=3, skip_redundant=True):
+ assert isinstance(module, torch.nn.Module)
+ assert not isinstance(module, torch.jit.ScriptModule)
+ assert isinstance(inputs, (tuple, list))
+
+ # Register hooks.
+ entries = []
+ nesting = [0]
+ def pre_hook(_mod, _inputs):
+ nesting[0] += 1
+ def post_hook(mod, _inputs, outputs):
+ nesting[0] -= 1
+ if nesting[0] <= max_nesting:
+ outputs = list(outputs) if isinstance(outputs, (tuple, list)) else [outputs]
+ outputs = [t for t in outputs if isinstance(t, torch.Tensor)]
+ entries.append(dnnlib.EasyDict(mod=mod, outputs=outputs))
+ hooks = [mod.register_forward_pre_hook(pre_hook) for mod in module.modules()]
+ hooks += [mod.register_forward_hook(post_hook) for mod in module.modules()]
+
+ # Run module.
+ outputs = module(*inputs)
+ for hook in hooks:
+ hook.remove()
+
+ # Identify unique outputs, parameters, and buffers.
+ tensors_seen = set()
+ for e in entries:
+ e.unique_params = [t for t in e.mod.parameters() if id(t) not in tensors_seen]
+ e.unique_buffers = [t for t in e.mod.buffers() if id(t) not in tensors_seen]
+ e.unique_outputs = [t for t in e.outputs if id(t) not in tensors_seen]
+ tensors_seen |= {id(t) for t in e.unique_params + e.unique_buffers + e.unique_outputs}
+
+ # Filter out redundant entries.
+ if skip_redundant:
+ entries = [e for e in entries if len(e.unique_params) or len(e.unique_buffers) or len(e.unique_outputs)]
+
+ # Construct table.
+ rows = [[type(module).__name__, 'Parameters', 'Buffers', 'Output shape', 'Datatype']]
+ rows += [['---'] * len(rows[0])]
+ param_total = 0
+ buffer_total = 0
+ submodule_names = {mod: name for name, mod in module.named_modules()}
+ for e in entries:
+ name = '<top-level>' if e.mod is module else submodule_names[e.mod]
+ param_size = sum(t.numel() for t in e.unique_params)
+ buffer_size = sum(t.numel() for t in e.unique_buffers)
+ output_shapes = [str(list(e.outputs[0].shape)) for t in e.outputs]
+ output_dtypes = [str(t.dtype).split('.')[-1] for t in e.outputs]
+ rows += [[
+ name + (':0' if len(e.outputs) >= 2 else ''),
+ str(param_size) if param_size else '-',
+ str(buffer_size) if buffer_size else '-',
+ (output_shapes + ['-'])[0],
+ (output_dtypes + ['-'])[0],
+ ]]
+ for idx in range(1, len(e.outputs)):
+ rows += [[name + f':{idx}', '-', '-', output_shapes[idx], output_dtypes[idx]]]
+ param_total += param_size
+ buffer_total += buffer_size
+ rows += [['---'] * len(rows[0])]
+ rows += [['Total', str(param_total), str(buffer_total), '-', '-']]
+
+ # Print table.
+ widths = [max(len(cell) for cell in column) for column in zip(*rows)]
+ print()
+ for row in rows:
+ print(' '.join(cell + ' ' * (width - len(cell)) for cell, width in zip(row, widths)))
+ print()
+ return outputs
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/__init__.py b/src/stylegan2/torch_utils/ops/__init__.py
new file mode 100644
index 0000000..ece0ea0
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/__init__.py
@@ -0,0 +1,9 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+# empty
diff --git a/src/stylegan2/torch_utils/ops/bias_act.cpp b/src/stylegan2/torch_utils/ops/bias_act.cpp
new file mode 100644
index 0000000..5d2425d
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/bias_act.cpp
@@ -0,0 +1,99 @@
+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto. Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+#include <torch/extension.h>
+#include <ATen/cuda/CUDAContext.h>
+#include <c10/cuda/CUDAGuard.h>
+#include "bias_act.h"
+
+//------------------------------------------------------------------------
+
+static bool has_same_layout(torch::Tensor x, torch::Tensor y)
+{
+ if (x.dim() != y.dim())
+ return false;
+ for (int64_t i = 0; i < x.dim(); i++)
+ {
+ if (x.size(i) != y.size(i))
+ return false;
+ if (x.size(i) >= 2 && x.stride(i) != y.stride(i))
+ return false;
+ }
+ return true;
+}
+
+//------------------------------------------------------------------------
+
+static torch::Tensor bias_act(torch::Tensor x, torch::Tensor b, torch::Tensor xref, torch::Tensor yref, torch::Tensor dy, int grad, int dim, int act, float alpha, float gain, float clamp)
+{
+ // Validate arguments.
+ TORCH_CHECK(x.is_cuda(), "x must reside on CUDA device");
+ TORCH_CHECK(b.numel() == 0 || (b.dtype() == x.dtype() && b.device() == x.device()), "b must have the same dtype and device as x");
+ TORCH_CHECK(xref.numel() == 0 || (xref.sizes() == x.sizes() && xref.dtype() == x.dtype() && xref.device() == x.device()), "xref must have the same shape, dtype, and device as x");
+ TORCH_CHECK(yref.numel() == 0 || (yref.sizes() == x.sizes() && yref.dtype() == x.dtype() && yref.device() == x.device()), "yref must have the same shape, dtype, and device as x");
+ TORCH_CHECK(dy.numel() == 0 || (dy.sizes() == x.sizes() && dy.dtype() == x.dtype() && dy.device() == x.device()), "dy must have the same dtype and device as x");
+ TORCH_CHECK(x.numel() <= INT_MAX, "x is too large");
+ TORCH_CHECK(b.dim() == 1, "b must have rank 1");
+ TORCH_CHECK(b.numel() == 0 || (dim >= 0 && dim < x.dim()), "dim is out of bounds");
+ TORCH_CHECK(b.numel() == 0 || b.numel() == x.size(dim), "b has wrong number of elements");
+ TORCH_CHECK(grad >= 0, "grad must be non-negative");
+
+ // Validate layout.
+ TORCH_CHECK(x.is_non_overlapping_and_dense(), "x must be non-overlapping and dense");
+ TORCH_CHECK(b.is_contiguous(), "b must be contiguous");
+ TORCH_CHECK(xref.numel() == 0 || has_same_layout(xref, x), "xref must have the same layout as x");
+ TORCH_CHECK(yref.numel() == 0 || has_same_layout(yref, x), "yref must have the same layout as x");
+ TORCH_CHECK(dy.numel() == 0 || has_same_layout(dy, x), "dy must have the same layout as x");
+
+ // Create output tensor.
+ const at::cuda::OptionalCUDAGuard device_guard(device_of(x));
+ torch::Tensor y = torch::empty_like(x);
+ TORCH_CHECK(has_same_layout(y, x), "y must have the same layout as x");
+
+ // Initialize CUDA kernel parameters.
+ bias_act_kernel_params p;
+ p.x = x.data_ptr();
+ p.b = (b.numel()) ? b.data_ptr() : NULL;
+ p.xref = (xref.numel()) ? xref.data_ptr() : NULL;
+ p.yref = (yref.numel()) ? yref.data_ptr() : NULL;
+ p.dy = (dy.numel()) ? dy.data_ptr() : NULL;
+ p.y = y.data_ptr();
+ p.grad = grad;
+ p.act = act;
+ p.alpha = alpha;
+ p.gain = gain;
+ p.clamp = clamp;
+ p.sizeX = (int)x.numel();
+ p.sizeB = (int)b.numel();
+ p.stepB = (b.numel()) ? (int)x.stride(dim) : 1;
+
+ // Choose CUDA kernel.
+ void* kernel;
+ AT_DISPATCH_FLOATING_TYPES_AND_HALF(x.scalar_type(), "upfirdn2d_cuda", [&]
+ {
+ kernel = choose_bias_act_kernel<scalar_t>(p);
+ });
+ TORCH_CHECK(kernel, "no CUDA kernel found for the specified activation func");
+
+ // Launch CUDA kernel.
+ p.loopX = 4;
+ int blockSize = 4 * 32;
+ int gridSize = (p.sizeX - 1) / (p.loopX * blockSize) + 1;
+ void* args[] = {&p};
+ AT_CUDA_CHECK(cudaLaunchKernel(kernel, gridSize, blockSize, args, 0, at::cuda::getCurrentCUDAStream()));
+ return y;
+}
+
+//------------------------------------------------------------------------
+
+PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
+{
+ m.def("bias_act", &bias_act);
+}
+
+//------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/bias_act.cu b/src/stylegan2/torch_utils/ops/bias_act.cu
new file mode 100644
index 0000000..dd8fc47
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/bias_act.cu
@@ -0,0 +1,173 @@
+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto. Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+#include <c10/util/Half.h>
+#include "bias_act.h"
+
+//------------------------------------------------------------------------
+// Helpers.
+
+template <class T> struct InternalType;
+template <> struct InternalType<double> { typedef double scalar_t; };
+template <> struct InternalType<float> { typedef float scalar_t; };
+template <> struct InternalType<c10::Half> { typedef float scalar_t; };
+
+//------------------------------------------------------------------------
+// CUDA kernel.
+
+template <class T, int A>
+__global__ void bias_act_kernel(bias_act_kernel_params p)
+{
+ typedef typename InternalType<T>::scalar_t scalar_t;
+ int G = p.grad;
+ scalar_t alpha = (scalar_t)p.alpha;
+ scalar_t gain = (scalar_t)p.gain;
+ scalar_t clamp = (scalar_t)p.clamp;
+ scalar_t one = (scalar_t)1;
+ scalar_t two = (scalar_t)2;
+ scalar_t expRange = (scalar_t)80;
+ scalar_t halfExpRange = (scalar_t)40;
+ scalar_t seluScale = (scalar_t)1.0507009873554804934193349852946;
+ scalar_t seluAlpha = (scalar_t)1.6732632423543772848170429916717;
+
+ // Loop over elements.
+ int xi = blockIdx.x * p.loopX * blockDim.x + threadIdx.x;
+ for (int loopIdx = 0; loopIdx < p.loopX && xi < p.sizeX; loopIdx++, xi += blockDim.x)
+ {
+ // Load.
+ scalar_t x = (scalar_t)((const T*)p.x)[xi];
+ scalar_t b = (p.b) ? (scalar_t)((const T*)p.b)[(xi / p.stepB) % p.sizeB] : 0;
+ scalar_t xref = (p.xref) ? (scalar_t)((const T*)p.xref)[xi] : 0;
+ scalar_t yref = (p.yref) ? (scalar_t)((const T*)p.yref)[xi] : 0;
+ scalar_t dy = (p.dy) ? (scalar_t)((const T*)p.dy)[xi] : one;
+ scalar_t yy = (gain != 0) ? yref / gain : 0;
+ scalar_t y = 0;
+
+ // Apply bias.
+ ((G == 0) ? x : xref) += b;
+
+ // linear
+ if (A == 1)
+ {
+ if (G == 0) y = x;
+ if (G == 1) y = x;
+ }
+
+ // relu
+ if (A == 2)
+ {
+ if (G == 0) y = (x > 0) ? x : 0;
+ if (G == 1) y = (yy > 0) ? x : 0;
+ }
+
+ // lrelu
+ if (A == 3)
+ {
+ if (G == 0) y = (x > 0) ? x : x * alpha;
+ if (G == 1) y = (yy > 0) ? x : x * alpha;
+ }
+
+ // tanh
+ if (A == 4)
+ {
+ if (G == 0) { scalar_t c = exp(x); scalar_t d = one / c; y = (x < -expRange) ? -one : (x > expRange) ? one : (c - d) / (c + d); }
+ if (G == 1) y = x * (one - yy * yy);
+ if (G == 2) y = x * (one - yy * yy) * (-two * yy);
+ }
+
+ // sigmoid
+ if (A == 5)
+ {
+ if (G == 0) y = (x < -expRange) ? 0 : one / (exp(-x) + one);
+ if (G == 1) y = x * yy * (one - yy);
+ if (G == 2) y = x * yy * (one - yy) * (one - two * yy);
+ }
+
+ // elu
+ if (A == 6)
+ {
+ if (G == 0) y = (x >= 0) ? x : exp(x) - one;
+ if (G == 1) y = (yy >= 0) ? x : x * (yy + one);
+ if (G == 2) y = (yy >= 0) ? 0 : x * (yy + one);
+ }
+
+ // selu
+ if (A == 7)
+ {
+ if (G == 0) y = (x >= 0) ? seluScale * x : (seluScale * seluAlpha) * (exp(x) - one);
+ if (G == 1) y = (yy >= 0) ? x * seluScale : x * (yy + seluScale * seluAlpha);
+ if (G == 2) y = (yy >= 0) ? 0 : x * (yy + seluScale * seluAlpha);
+ }
+
+ // softplus
+ if (A == 8)
+ {
+ if (G == 0) y = (x > expRange) ? x : log(exp(x) + one);
+ if (G == 1) y = x * (one - exp(-yy));
+ if (G == 2) { scalar_t c = exp(-yy); y = x * c * (one - c); }
+ }
+
+ // swish
+ if (A == 9)
+ {
+ if (G == 0)
+ y = (x < -expRange) ? 0 : x / (exp(-x) + one);
+ else
+ {
+ scalar_t c = exp(xref);
+ scalar_t d = c + one;
+ if (G == 1)
+ y = (xref > halfExpRange) ? x : x * c * (xref + d) / (d * d);
+ else
+ y = (xref > halfExpRange) ? 0 : x * c * (xref * (two - d) + two * d) / (d * d * d);
+ yref = (xref < -expRange) ? 0 : xref / (exp(-xref) + one) * gain;
+ }
+ }
+
+ // Apply gain.
+ y *= gain * dy;
+
+ // Clamp.
+ if (clamp >= 0)
+ {
+ if (G == 0)
+ y = (y > -clamp & y < clamp) ? y : (y >= 0) ? clamp : -clamp;
+ else
+ y = (yref > -clamp & yref < clamp) ? y : 0;
+ }
+
+ // Store.
+ ((T*)p.y)[xi] = (T)y;
+ }
+}
+
+//------------------------------------------------------------------------
+// CUDA kernel selection.
+
+template <class T> void* choose_bias_act_kernel(const bias_act_kernel_params& p)
+{
+ if (p.act == 1) return (void*)bias_act_kernel<T, 1>;
+ if (p.act == 2) return (void*)bias_act_kernel<T, 2>;
+ if (p.act == 3) return (void*)bias_act_kernel<T, 3>;
+ if (p.act == 4) return (void*)bias_act_kernel<T, 4>;
+ if (p.act == 5) return (void*)bias_act_kernel<T, 5>;
+ if (p.act == 6) return (void*)bias_act_kernel<T, 6>;
+ if (p.act == 7) return (void*)bias_act_kernel<T, 7>;
+ if (p.act == 8) return (void*)bias_act_kernel<T, 8>;
+ if (p.act == 9) return (void*)bias_act_kernel<T, 9>;
+ return NULL;
+}
+
+//------------------------------------------------------------------------
+// Template specializations.
+
+template void* choose_bias_act_kernel<double> (const bias_act_kernel_params& p);
+template void* choose_bias_act_kernel<float> (const bias_act_kernel_params& p);
+template void* choose_bias_act_kernel<c10::Half> (const bias_act_kernel_params& p);
+
+//------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/bias_act.h b/src/stylegan2/torch_utils/ops/bias_act.h
new file mode 100644
index 0000000..a32187e
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/bias_act.h
@@ -0,0 +1,38 @@
+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto. Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+//------------------------------------------------------------------------
+// CUDA kernel parameters.
+
+struct bias_act_kernel_params
+{
+ const void* x; // [sizeX]
+ const void* b; // [sizeB] or NULL
+ const void* xref; // [sizeX] or NULL
+ const void* yref; // [sizeX] or NULL
+ const void* dy; // [sizeX] or NULL
+ void* y; // [sizeX]
+
+ int grad;
+ int act;
+ float alpha;
+ float gain;
+ float clamp;
+
+ int sizeX;
+ int sizeB;
+ int stepB;
+ int loopX;
+};
+
+//------------------------------------------------------------------------
+// CUDA kernel selection.
+
+template <class T> void* choose_bias_act_kernel(const bias_act_kernel_params& p);
+
+//------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/bias_act.py b/src/stylegan2/torch_utils/ops/bias_act.py
new file mode 100644
index 0000000..5dabe69
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/bias_act.py
@@ -0,0 +1,213 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Custom PyTorch ops for efficient bias and activation."""
+
+import os
+import warnings
+import numpy as np
+import torch
+import dnnlib
+import traceback
+
+from .. import custom_ops
+from .. import misc
+
+#----------------------------------------------------------------------------
+
+activation_funcs = {
+ 'linear': dnnlib.EasyDict(func=lambda x, **_: x, def_alpha=0, def_gain=1, cuda_idx=1, ref='', has_2nd_grad=False),
+ 'relu': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.relu(x), def_alpha=0, def_gain=np.sqrt(2), cuda_idx=2, ref='y', has_2nd_grad=False),
+ 'lrelu': dnnlib.EasyDict(func=lambda x, alpha, **_: torch.nn.functional.leaky_relu(x, alpha), def_alpha=0.2, def_gain=np.sqrt(2), cuda_idx=3, ref='y', has_2nd_grad=False),
+ 'tanh': dnnlib.EasyDict(func=lambda x, **_: torch.tanh(x), def_alpha=0, def_gain=1, cuda_idx=4, ref='y', has_2nd_grad=True),
+ 'sigmoid': dnnlib.EasyDict(func=lambda x, **_: torch.sigmoid(x), def_alpha=0, def_gain=1, cuda_idx=5, ref='y', has_2nd_grad=True),
+ 'elu': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.elu(x), def_alpha=0, def_gain=1, cuda_idx=6, ref='y', has_2nd_grad=True),
+ 'selu': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.selu(x), def_alpha=0, def_gain=1, cuda_idx=7, ref='y', has_2nd_grad=True),
+ 'softplus': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.softplus(x), def_alpha=0, def_gain=1, cuda_idx=8, ref='y', has_2nd_grad=True),
+ 'swish': dnnlib.EasyDict(func=lambda x, **_: torch.sigmoid(x) * x, def_alpha=0, def_gain=np.sqrt(2), cuda_idx=9, ref='x', has_2nd_grad=True),
+}
+
+#----------------------------------------------------------------------------
+
+_inited = False
+_plugin = None
+_null_tensor = torch.empty([0])
+
+def _init():
+ global _inited, _plugin
+ if not _inited:
+ _inited = True
+ sources = ['bias_act.cpp', 'bias_act.cu']
+ sources = [os.path.join(os.path.dirname(__file__), s) for s in sources]
+ try:
+ _plugin = custom_ops.get_plugin('bias_act_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
+ except:
+ warnings.warn('Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())
+ return _plugin is not None
+
+#----------------------------------------------------------------------------
+
+def bias_act(x, b=None, dim=1, act='linear', alpha=None, gain=None, clamp=None, impl='cuda'):
+ r"""Fused bias and activation function.
+
+ Adds bias `b` to activation tensor `x`, evaluates activation function `act`,
+ and scales the result by `gain`. Each of the steps is optional. In most cases,
+ the fused op is considerably more efficient than performing the same calculation
+ using standard PyTorch ops. It supports first and second order gradients,
+ but not third order gradients.
+
+ Args:
+ x: Input activation tensor. Can be of any shape.
+ b: Bias vector, or `None` to disable. Must be a 1D tensor of the same type
+ as `x`. The shape must be known, and it must match the dimension of `x`
+ corresponding to `dim`.
+ dim: The dimension in `x` corresponding to the elements of `b`.
+ The value of `dim` is ignored if `b` is not specified.
+ act: Name of the activation function to evaluate, or `"linear"` to disable.
+ Can be e.g. `"relu"`, `"lrelu"`, `"tanh"`, `"sigmoid"`, `"swish"`, etc.
+ See `activation_funcs` for a full list. `None` is not allowed.
+ alpha: Shape parameter for the activation function, or `None` to use the default.
+ gain: Scaling factor for the output tensor, or `None` to use default.
+ See `activation_funcs` for the default scaling of each activation function.
+ If unsure, consider specifying 1.
+ clamp: Clamp the output values to `[-clamp, +clamp]`, or `None` to disable
+ the clamping (default).
+ impl: Name of the implementation to use. Can be `"ref"` or `"cuda"` (default).
+
+ Returns:
+ Tensor of the same shape and datatype as `x`.
+ """
+ assert isinstance(x, torch.Tensor)
+ assert impl in ['ref', 'cuda']
+ if impl == 'cuda' and x.device.type == 'cuda' and _init():
+ return _bias_act_cuda(dim=dim, act=act, alpha=alpha, gain=gain, clamp=clamp).apply(x, b)
+ return _bias_act_ref(x=x, b=b, dim=dim, act=act, alpha=alpha, gain=gain, clamp=clamp)
+
+#----------------------------------------------------------------------------
+
+def _bias_act_ref(x, b=None, dim=1, act='linear', alpha=None, gain=None, clamp=None):
+ """Slow reference implementation of `bias_act()` using standard TensorFlow ops.
+ """
+ assert isinstance(x, torch.Tensor)
+ assert clamp is None or clamp >= 0
+ spec = activation_funcs[act]
+ alpha = float(alpha if alpha is not None else spec.def_alpha)
+ gain = float(gain if gain is not None else spec.def_gain)
+ clamp = float(clamp if clamp is not None else -1)
+
+ # Add bias.
+ if b is not None:
+ assert isinstance(b, torch.Tensor) and b.ndim == 1
+ assert 0 <= dim < x.ndim
+ assert b.shape[0] == x.shape[dim]
+ x = x + b.reshape([-1 if i == dim else 1 for i in range(x.ndim)])
+
+ # Evaluate activation function.
+ alpha = float(alpha)
+ x = spec.func(x, alpha=alpha)
+
+ # Scale by gain.
+ gain = float(gain)
+ if gain != 1:
+ x = x * gain
+
+ # Clamp.
+ if clamp >= 0:
+ x = x.clamp(-clamp, clamp) # pylint: disable=invalid-unary-operand-type
+ return x
+
+#----------------------------------------------------------------------------
+
+_bias_act_cuda_cache = dict()
+
+def _bias_act_cuda(dim=1, act='linear', alpha=None, gain=None, clamp=None):
+ """Fast CUDA implementation of `bias_act()` using custom ops.
+ """
+ # Parse arguments.
+ assert clamp is None or clamp >= 0
+ spec = activation_funcs[act]
+ alpha = float(alpha if alpha is not None else spec.def_alpha)
+ gain = float(gain if gain is not None else spec.def_gain)
+ clamp = float(clamp if clamp is not None else -1)
+
+ # Lookup from cache.
+ key = (dim, act, alpha, gain, clamp)
+ if key in _bias_act_cuda_cache:
+ return _bias_act_cuda_cache[key]
+
+ # Forward op.
+ class BiasActCuda(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, x, b): # pylint: disable=arguments-differ
+ ctx.memory_format = torch.channels_last if x.ndim > 2 and x.stride()[1] == 1 else torch.contiguous_format
+ x = x.contiguous(memory_format=ctx.memory_format)
+ b = b.contiguous() if b is not None else _null_tensor
+ x = x.to(b.dtype)
+ y = x
+ if act != 'linear' or gain != 1 or clamp >= 0 or b is not _null_tensor:
+ y = _plugin.bias_act(x, b, _null_tensor, _null_tensor, _null_tensor, 0, dim, spec.cuda_idx, alpha, gain, clamp)
+ ctx.save_for_backward(
+ x if 'x' in spec.ref or spec.has_2nd_grad else _null_tensor,
+ b if 'x' in spec.ref or spec.has_2nd_grad else _null_tensor,
+ y if 'y' in spec.ref else _null_tensor)
+ return y
+
+ @staticmethod
+ def backward(ctx, dy): # pylint: disable=arguments-differ
+ dy = dy.contiguous(memory_format=ctx.memory_format)
+ x, b, y = ctx.saved_tensors
+ dx = None
+ db = None
+
+ if ctx.needs_input_grad[0] or ctx.needs_input_grad[1]:
+ dx = dy
+ if act != 'linear' or gain != 1 or clamp >= 0:
+ dx = BiasActCudaGrad.apply(dy, x, b, y)
+
+ if ctx.needs_input_grad[1]:
+ db = dx.sum([i for i in range(dx.ndim) if i != dim])
+
+ return dx, db
+
+ # Backward op.
+ class BiasActCudaGrad(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, dy, x, b, y): # pylint: disable=arguments-differ
+ ctx.memory_format = torch.channels_last if dy.ndim > 2 and dy.stride()[1] == 1 else torch.contiguous_format
+ dx = _plugin.bias_act(dy, b, x, y, _null_tensor, 1, dim, spec.cuda_idx, alpha, gain, clamp)
+ ctx.save_for_backward(
+ dy if spec.has_2nd_grad else _null_tensor,
+ x, b, y)
+ return dx
+
+ @staticmethod
+ def backward(ctx, d_dx): # pylint: disable=arguments-differ
+ d_dx = d_dx.contiguous(memory_format=ctx.memory_format)
+ dy, x, b, y = ctx.saved_tensors
+ d_dy = None
+ d_x = None
+ d_b = None
+ d_y = None
+
+ if ctx.needs_input_grad[0]:
+ d_dy = BiasActCudaGrad.apply(d_dx, x, b, y)
+
+ if spec.has_2nd_grad and (ctx.needs_input_grad[1] or ctx.needs_input_grad[2]):
+ d_x = _plugin.bias_act(d_dx, b, x, y, dy, 2, dim, spec.cuda_idx, alpha, gain, clamp)
+
+ if spec.has_2nd_grad and ctx.needs_input_grad[2]:
+ d_b = d_x.sum([i for i in range(d_x.ndim) if i != dim])
+
+ return d_dy, d_x, d_b, d_y
+
+ # Add to cache.
+ _bias_act_cuda_cache[key] = BiasActCuda
+ return BiasActCuda
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/conv2d_gradfix.py b/src/stylegan2/torch_utils/ops/conv2d_gradfix.py
new file mode 100644
index 0000000..e95e10d
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/conv2d_gradfix.py
@@ -0,0 +1,170 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Custom replacement for `torch.nn.functional.conv2d` that supports
+arbitrarily high order gradients with zero performance penalty."""
+
+import warnings
+import contextlib
+import torch
+
+# pylint: disable=redefined-builtin
+# pylint: disable=arguments-differ
+# pylint: disable=protected-access
+
+#----------------------------------------------------------------------------
+
+enabled = False # Enable the custom op by setting this to true.
+weight_gradients_disabled = False # Forcefully disable computation of gradients with respect to the weights.
+
+def no_weight_gradients():
+ global weight_gradients_disabled
+ old = weight_gradients_disabled
+ weight_gradients_disabled = True
+ yield
+ weight_gradients_disabled = old
+
+#----------------------------------------------------------------------------
+
+def conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
+ if _should_use_custom_op(input):
+ return _conv2d_gradfix(transpose=False, weight_shape=weight.shape, stride=stride, padding=padding, output_padding=0, dilation=dilation, groups=groups).apply(input, weight, bias)
+ return torch.nn.functional.conv2d(input=input, weight=weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
+
+def conv_transpose2d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1):
+ if _should_use_custom_op(input):
+ return _conv2d_gradfix(transpose=True, weight_shape=weight.shape, stride=stride, padding=padding, output_padding=output_padding, groups=groups, dilation=dilation).apply(input, weight, bias)
+ return torch.nn.functional.conv_transpose2d(input=input, weight=weight, bias=bias, stride=stride, padding=padding, output_padding=output_padding, groups=groups, dilation=dilation)
+
+#----------------------------------------------------------------------------
+
+def _should_use_custom_op(input):
+ assert isinstance(input, torch.Tensor)
+ if (not enabled) or (not torch.backends.cudnn.enabled):
+ return False
+ if input.device.type != 'cuda':
+ return False
+ if any(torch.__version__.startswith(x) for x in ['1.7.', '1.8.', '1.9']):
+ return True
+ warnings.warn(f'conv2d_gradfix not supported on PyTorch {torch.__version__}. Falling back to torch.nn.functional.conv2d().')
+ return False
+
+def _tuple_of_ints(xs, ndim):
+ xs = tuple(xs) if isinstance(xs, (tuple, list)) else (xs,) * ndim
+ assert len(xs) == ndim
+ assert all(isinstance(x, int) for x in xs)
+ return xs
+
+#----------------------------------------------------------------------------
+
+_conv2d_gradfix_cache = dict()
+
+def _conv2d_gradfix(transpose, weight_shape, stride, padding, output_padding, dilation, groups):
+ # Parse arguments.
+ ndim = 2
+ weight_shape = tuple(weight_shape)
+ stride = _tuple_of_ints(stride, ndim)
+ padding = _tuple_of_ints(padding, ndim)
+ output_padding = _tuple_of_ints(output_padding, ndim)
+ dilation = _tuple_of_ints(dilation, ndim)
+
+ # Lookup from cache.
+ key = (transpose, weight_shape, stride, padding, output_padding, dilation, groups)
+ if key in _conv2d_gradfix_cache:
+ return _conv2d_gradfix_cache[key]
+
+ # Validate arguments.
+ assert groups >= 1
+ assert len(weight_shape) == ndim + 2
+ assert all(stride[i] >= 1 for i in range(ndim))
+ assert all(padding[i] >= 0 for i in range(ndim))
+ assert all(dilation[i] >= 0 for i in range(ndim))
+ if not transpose:
+ assert all(output_padding[i] == 0 for i in range(ndim))
+ else: # transpose
+ assert all(0 <= output_padding[i] < max(stride[i], dilation[i]) for i in range(ndim))
+
+ # Helpers.
+ common_kwargs = dict(stride=stride, padding=padding, dilation=dilation, groups=groups)
+ def calc_output_padding(input_shape, output_shape):
+ if transpose:
+ return [0, 0]
+ return [
+ input_shape[i + 2]
+ - (output_shape[i + 2] - 1) * stride[i]
+ - (1 - 2 * padding[i])
+ - dilation[i] * (weight_shape[i + 2] - 1)
+ for i in range(ndim)
+ ]
+
+ # Forward & backward.
+ class Conv2d(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, input, weight, bias):
+ assert weight.shape == weight_shape
+ if not transpose:
+ output = torch.nn.functional.conv2d(input=input, weight=weight, bias=bias, **common_kwargs)
+ else: # transpose
+ output = torch.nn.functional.conv_transpose2d(input=input, weight=weight, bias=bias, output_padding=output_padding, **common_kwargs)
+ ctx.save_for_backward(input, weight)
+ return output
+
+ @staticmethod
+ def backward(ctx, grad_output):
+ input, weight = ctx.saved_tensors
+ grad_input = None
+ grad_weight = None
+ grad_bias = None
+
+ if ctx.needs_input_grad[0]:
+ p = calc_output_padding(input_shape=input.shape, output_shape=grad_output.shape)
+ grad_input = _conv2d_gradfix(transpose=(not transpose), weight_shape=weight_shape, output_padding=p, **common_kwargs).apply(grad_output, weight, None)
+ assert grad_input.shape == input.shape
+
+ if ctx.needs_input_grad[1] and not weight_gradients_disabled:
+ grad_weight = Conv2dGradWeight.apply(grad_output, input)
+ assert grad_weight.shape == weight_shape
+
+ if ctx.needs_input_grad[2]:
+ grad_bias = grad_output.sum([0, 2, 3])
+
+ return grad_input, grad_weight, grad_bias
+
+ # Gradient with respect to the weights.
+ class Conv2dGradWeight(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, grad_output, input):
+ op = torch._C._jit_get_operation('aten::cudnn_convolution_backward_weight' if not transpose else 'aten::cudnn_convolution_transpose_backward_weight')
+ flags = [torch.backends.cudnn.benchmark, torch.backends.cudnn.deterministic, torch.backends.cudnn.allow_tf32]
+ grad_weight = op(weight_shape, grad_output, input, padding, stride, dilation, groups, *flags)
+ assert grad_weight.shape == weight_shape
+ ctx.save_for_backward(grad_output, input)
+ return grad_weight
+
+ @staticmethod
+ def backward(ctx, grad2_grad_weight):
+ grad_output, input = ctx.saved_tensors
+ grad2_grad_output = None
+ grad2_input = None
+
+ if ctx.needs_input_grad[0]:
+ grad2_grad_output = Conv2d.apply(input, grad2_grad_weight, None)
+ assert grad2_grad_output.shape == grad_output.shape
+
+ if ctx.needs_input_grad[1]:
+ p = calc_output_padding(input_shape=input.shape, output_shape=grad_output.shape)
+ grad2_input = _conv2d_gradfix(transpose=(not transpose), weight_shape=weight_shape, output_padding=p, **common_kwargs).apply(grad_output, grad2_grad_weight, None)
+ assert grad2_input.shape == input.shape
+
+ return grad2_grad_output, grad2_input
+
+ _conv2d_gradfix_cache[key] = Conv2d
+ return Conv2d
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/conv2d_resample.py b/src/stylegan2/torch_utils/ops/conv2d_resample.py
new file mode 100644
index 0000000..cd47507
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/conv2d_resample.py
@@ -0,0 +1,156 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""2D convolution with optional up/downsampling."""
+
+import torch
+
+from .. import misc
+from . import conv2d_gradfix
+from . import upfirdn2d
+from .upfirdn2d import _parse_padding
+from .upfirdn2d import _get_filter_size
+
+#----------------------------------------------------------------------------
+
+def _get_weight_shape(w):
+ with misc.suppress_tracer_warnings(): # this value will be treated as a constant
+ shape = [int(sz) for sz in w.shape]
+ misc.assert_shape(w, shape)
+ return shape
+
+#----------------------------------------------------------------------------
+
+def _conv2d_wrapper(x, w, stride=1, padding=0, groups=1, transpose=False, flip_weight=True):
+ """Wrapper for the underlying `conv2d()` and `conv_transpose2d()` implementations.
+ """
+ out_channels, in_channels_per_group, kh, kw = _get_weight_shape(w)
+
+ # Flip weight if requested.
+ if not flip_weight: # conv2d() actually performs correlation (flip_weight=True) not convolution (flip_weight=False).
+ w = w.flip([2, 3])
+
+ # Workaround performance pitfall in cuDNN 8.0.5, triggered when using
+ # 1x1 kernel + memory_format=channels_last + less than 64 channels.
+ if kw == 1 and kh == 1 and stride == 1 and padding in [0, [0, 0], (0, 0)] and not transpose:
+ if x.stride()[1] == 1 and min(out_channels, in_channels_per_group) < 64:
+ if out_channels <= 4 and groups == 1:
+ in_shape = x.shape
+ x = w.squeeze(3).squeeze(2) @ x.reshape([in_shape[0], in_channels_per_group, -1])
+ x = x.reshape([in_shape[0], out_channels, in_shape[2], in_shape[3]])
+ else:
+ x = x.to(memory_format=torch.contiguous_format)
+ w = w.to(memory_format=torch.contiguous_format)
+ x = conv2d_gradfix.conv2d(x, w, groups=groups)
+ return x.to(memory_format=torch.channels_last)
+
+ # Otherwise => execute using conv2d_gradfix.
+ op = conv2d_gradfix.conv_transpose2d if transpose else conv2d_gradfix.conv2d
+ return op(x, w, stride=stride, padding=padding, groups=groups)
+
+#----------------------------------------------------------------------------
+
+def conv2d_resample(x, w, f=None, up=1, down=1, padding=0, groups=1, flip_weight=True, flip_filter=False):
+ r"""2D convolution with optional up/downsampling.
+
+ Padding is performed only once at the beginning, not between the operations.
+
+ Args:
+ x: Input tensor of shape
+ `[batch_size, in_channels, in_height, in_width]`.
+ w: Weight tensor of shape
+ `[out_channels, in_channels//groups, kernel_height, kernel_width]`.
+ f: Low-pass filter for up/downsampling. Must be prepared beforehand by
+ calling upfirdn2d.setup_filter(). None = identity (default).
+ up: Integer upsampling factor (default: 1).
+ down: Integer downsampling factor (default: 1).
+ padding: Padding with respect to the upsampled image. Can be a single number
+ or a list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`
+ (default: 0).
+ groups: Split input channels into N groups (default: 1).
+ flip_weight: False = convolution, True = correlation (default: True).
+ flip_filter: False = convolution, True = correlation (default: False).
+
+ Returns:
+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.
+ """
+ # Validate arguments.
+ assert isinstance(x, torch.Tensor) and (x.ndim == 4)
+ assert isinstance(w, torch.Tensor) and (w.ndim == 4) and (w.dtype == x.dtype)
+ assert f is None or (isinstance(f, torch.Tensor) and f.ndim in [1, 2] and f.dtype == torch.float32)
+ assert isinstance(up, int) and (up >= 1)
+ assert isinstance(down, int) and (down >= 1)
+ assert isinstance(groups, int) and (groups >= 1)
+ out_channels, in_channels_per_group, kh, kw = _get_weight_shape(w)
+ fw, fh = _get_filter_size(f)
+ px0, px1, py0, py1 = _parse_padding(padding)
+
+ # Adjust padding to account for up/downsampling.
+ if up > 1:
+ px0 += (fw + up - 1) // 2
+ px1 += (fw - up) // 2
+ py0 += (fh + up - 1) // 2
+ py1 += (fh - up) // 2
+ if down > 1:
+ px0 += (fw - down + 1) // 2
+ px1 += (fw - down) // 2
+ py0 += (fh - down + 1) // 2
+ py1 += (fh - down) // 2
+
+ # Fast path: 1x1 convolution with downsampling only => downsample first, then convolve.
+ if kw == 1 and kh == 1 and (down > 1 and up == 1):
+ x = upfirdn2d.upfirdn2d(x=x, f=f, down=down, padding=[px0,px1,py0,py1], flip_filter=flip_filter)
+ x = _conv2d_wrapper(x=x, w=w, groups=groups, flip_weight=flip_weight)
+ return x
+
+ # Fast path: 1x1 convolution with upsampling only => convolve first, then upsample.
+ if kw == 1 and kh == 1 and (up > 1 and down == 1):
+ x = _conv2d_wrapper(x=x, w=w, groups=groups, flip_weight=flip_weight)
+ x = upfirdn2d.upfirdn2d(x=x, f=f, up=up, padding=[px0,px1,py0,py1], gain=up**2, flip_filter=flip_filter)
+ return x
+
+ # Fast path: downsampling only => use strided convolution.
+ if down > 1 and up == 1:
+ x = upfirdn2d.upfirdn2d(x=x, f=f, padding=[px0,px1,py0,py1], flip_filter=flip_filter)
+ x = _conv2d_wrapper(x=x, w=w, stride=down, groups=groups, flip_weight=flip_weight)
+ return x
+
+ # Fast path: upsampling with optional downsampling => use transpose strided convolution.
+ if up > 1:
+ if groups == 1:
+ w = w.transpose(0, 1)
+ else:
+ w = w.reshape(groups, out_channels // groups, in_channels_per_group, kh, kw)
+ w = w.transpose(1, 2)
+ w = w.reshape(groups * in_channels_per_group, out_channels // groups, kh, kw)
+ px0 -= kw - 1
+ px1 -= kw - up
+ py0 -= kh - 1
+ py1 -= kh - up
+ pxt = max(min(-px0, -px1), 0)
+ pyt = max(min(-py0, -py1), 0)
+ x = _conv2d_wrapper(x=x, w=w, stride=up, padding=[pyt,pxt], groups=groups, transpose=True, flip_weight=(not flip_weight))
+ x = upfirdn2d.upfirdn2d(x=x, f=f, padding=[px0+pxt,px1+pxt,py0+pyt,py1+pyt], gain=up**2, flip_filter=flip_filter)
+ if down > 1:
+ x = upfirdn2d.upfirdn2d(x=x, f=f, down=down, flip_filter=flip_filter)
+ return x
+
+ # Fast path: no up/downsampling, padding supported by the underlying implementation => use plain conv2d.
+ if up == 1 and down == 1:
+ if px0 == px1 and py0 == py1 and px0 >= 0 and py0 >= 0:
+ return _conv2d_wrapper(x=x, w=w, padding=[py0,px0], groups=groups, flip_weight=flip_weight)
+
+ # Fallback: Generic reference implementation.
+ x = upfirdn2d.upfirdn2d(x=x, f=(f if up > 1 else None), up=up, padding=[px0,px1,py0,py1], gain=up**2, flip_filter=flip_filter)
+ x = _conv2d_wrapper(x=x, w=w, groups=groups, flip_weight=flip_weight)
+ if down > 1:
+ x = upfirdn2d.upfirdn2d(x=x, f=f, down=down, flip_filter=flip_filter)
+ return x
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/fma.py b/src/stylegan2/torch_utils/ops/fma.py
new file mode 100644
index 0000000..2eeac58
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/fma.py
@@ -0,0 +1,60 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Fused multiply-add, with slightly faster gradients than `torch.addcmul()`."""
+
+import torch
+
+#----------------------------------------------------------------------------
+
+def fma(a, b, c): # => a * b + c
+ return _FusedMultiplyAdd.apply(a, b, c)
+
+#----------------------------------------------------------------------------
+
+class _FusedMultiplyAdd(torch.autograd.Function): # a * b + c
+ @staticmethod
+ def forward(ctx, a, b, c): # pylint: disable=arguments-differ
+ out = torch.addcmul(c, a, b)
+ ctx.save_for_backward(a, b)
+ ctx.c_shape = c.shape
+ return out
+
+ @staticmethod
+ def backward(ctx, dout): # pylint: disable=arguments-differ
+ a, b = ctx.saved_tensors
+ c_shape = ctx.c_shape
+ da = None
+ db = None
+ dc = None
+
+ if ctx.needs_input_grad[0]:
+ da = _unbroadcast(dout * b, a.shape)
+
+ if ctx.needs_input_grad[1]:
+ db = _unbroadcast(dout * a, b.shape)
+
+ if ctx.needs_input_grad[2]:
+ dc = _unbroadcast(dout, c_shape)
+
+ return da, db, dc
+
+#----------------------------------------------------------------------------
+
+def _unbroadcast(x, shape):
+ extra_dims = x.ndim - len(shape)
+ assert extra_dims >= 0
+ dim = [i for i in range(x.ndim) if x.shape[i] > 1 and (i < extra_dims or shape[i - extra_dims] == 1)]
+ if len(dim):
+ x = x.sum(dim=dim, keepdim=True)
+ if extra_dims:
+ x = x.reshape(-1, *x.shape[extra_dims+1:])
+ assert x.shape == shape
+ return x
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/grid_sample_gradfix.py b/src/stylegan2/torch_utils/ops/grid_sample_gradfix.py
new file mode 100644
index 0000000..ca6b341
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/grid_sample_gradfix.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Custom replacement for `torch.nn.functional.grid_sample` that
+supports arbitrarily high order gradients between the input and output.
+Only works on 2D images and assumes
+`mode='bilinear'`, `padding_mode='zeros'`, `align_corners=False`."""
+
+import warnings
+import torch
+
+# pylint: disable=redefined-builtin
+# pylint: disable=arguments-differ
+# pylint: disable=protected-access
+
+#----------------------------------------------------------------------------
+
+enabled = False # Enable the custom op by setting this to true.
+
+#----------------------------------------------------------------------------
+
+def grid_sample(input, grid):
+ if _should_use_custom_op():
+ return _GridSample2dForward.apply(input, grid)
+ return torch.nn.functional.grid_sample(input=input, grid=grid, mode='bilinear', padding_mode='zeros', align_corners=False)
+
+#----------------------------------------------------------------------------
+
+def _should_use_custom_op():
+ if not enabled:
+ return False
+ if any(torch.__version__.startswith(x) for x in ['1.7.', '1.8.', '1.9']):
+ return True
+ warnings.warn(f'grid_sample_gradfix not supported on PyTorch {torch.__version__}. Falling back to torch.nn.functional.grid_sample().')
+ return False
+
+#----------------------------------------------------------------------------
+
+class _GridSample2dForward(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, input, grid):
+ assert input.ndim == 4
+ assert grid.ndim == 4
+ output = torch.nn.functional.grid_sample(input=input, grid=grid, mode='bilinear', padding_mode='zeros', align_corners=False)
+ ctx.save_for_backward(input, grid)
+ return output
+
+ @staticmethod
+ def backward(ctx, grad_output):
+ input, grid = ctx.saved_tensors
+ grad_input, grad_grid = _GridSample2dBackward.apply(grad_output, input, grid)
+ return grad_input, grad_grid
+
+#----------------------------------------------------------------------------
+
+class _GridSample2dBackward(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, grad_output, input, grid):
+ op = torch._C._jit_get_operation('aten::grid_sampler_2d_backward')
+ grad_input, grad_grid = op(grad_output, input, grid, 0, 0, False)
+ ctx.save_for_backward(grid)
+ return grad_input, grad_grid
+
+ @staticmethod
+ def backward(ctx, grad2_grad_input, grad2_grad_grid):
+ _ = grad2_grad_grid # unused
+ grid, = ctx.saved_tensors
+ grad2_grad_output = None
+ grad2_input = None
+ grad2_grid = None
+
+ if ctx.needs_input_grad[0]:
+ grad2_grad_output = _GridSample2dForward.apply(grad2_grad_input, grid)
+
+ assert not ctx.needs_input_grad[2]
+ return grad2_grad_output, grad2_input, grad2_grid
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.cpp b/src/stylegan2/torch_utils/ops/upfirdn2d.cpp
new file mode 100644
index 0000000..2d7177f
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.cpp
@@ -0,0 +1,103 @@
+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto. Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+#include <torch/extension.h>
+#include <ATen/cuda/CUDAContext.h>
+#include <c10/cuda/CUDAGuard.h>
+#include "upfirdn2d.h"
+
+//------------------------------------------------------------------------
+
+static torch::Tensor upfirdn2d(torch::Tensor x, torch::Tensor f, int upx, int upy, int downx, int downy, int padx0, int padx1, int pady0, int pady1, bool flip, float gain)
+{
+ // Validate arguments.
+ TORCH_CHECK(x.is_cuda(), "x must reside on CUDA device");
+ TORCH_CHECK(f.device() == x.device(), "f must reside on the same device as x");
+ TORCH_CHECK(f.dtype() == torch::kFloat, "f must be float32");
+ TORCH_CHECK(x.numel() <= INT_MAX, "x is too large");
+ TORCH_CHECK(f.numel() <= INT_MAX, "f is too large");
+ TORCH_CHECK(x.dim() == 4, "x must be rank 4");
+ TORCH_CHECK(f.dim() == 2, "f must be rank 2");
+ TORCH_CHECK(f.size(0) >= 1 && f.size(1) >= 1, "f must be at least 1x1");
+ TORCH_CHECK(upx >= 1 && upy >= 1, "upsampling factor must be at least 1");
+ TORCH_CHECK(downx >= 1 && downy >= 1, "downsampling factor must be at least 1");
+
+ // Create output tensor.
+ const at::cuda::OptionalCUDAGuard device_guard(device_of(x));
+ int outW = ((int)x.size(3) * upx + padx0 + padx1 - (int)f.size(1) + downx) / downx;
+ int outH = ((int)x.size(2) * upy + pady0 + pady1 - (int)f.size(0) + downy) / downy;
+ TORCH_CHECK(outW >= 1 && outH >= 1, "output must be at least 1x1");
+ torch::Tensor y = torch::empty({x.size(0), x.size(1), outH, outW}, x.options(), x.suggest_memory_format());
+ TORCH_CHECK(y.numel() <= INT_MAX, "output is too large");
+
+ // Initialize CUDA kernel parameters.
+ upfirdn2d_kernel_params p;
+ p.x = x.data_ptr();
+ p.f = f.data_ptr<float>();
+ p.y = y.data_ptr();
+ p.up = make_int2(upx, upy);
+ p.down = make_int2(downx, downy);
+ p.pad0 = make_int2(padx0, pady0);
+ p.flip = (flip) ? 1 : 0;
+ p.gain = gain;
+ p.inSize = make_int4((int)x.size(3), (int)x.size(2), (int)x.size(1), (int)x.size(0));
+ p.inStride = make_int4((int)x.stride(3), (int)x.stride(2), (int)x.stride(1), (int)x.stride(0));
+ p.filterSize = make_int2((int)f.size(1), (int)f.size(0));
+ p.filterStride = make_int2((int)f.stride(1), (int)f.stride(0));
+ p.outSize = make_int4((int)y.size(3), (int)y.size(2), (int)y.size(1), (int)y.size(0));
+ p.outStride = make_int4((int)y.stride(3), (int)y.stride(2), (int)y.stride(1), (int)y.stride(0));
+ p.sizeMajor = (p.inStride.z == 1) ? p.inSize.w : p.inSize.w * p.inSize.z;
+ p.sizeMinor = (p.inStride.z == 1) ? p.inSize.z : 1;
+
+ // Choose CUDA kernel.
+ upfirdn2d_kernel_spec spec;
+ AT_DISPATCH_FLOATING_TYPES_AND_HALF(x.scalar_type(), "upfirdn2d_cuda", [&]
+ {
+ spec = choose_upfirdn2d_kernel<scalar_t>(p);
+ });
+
+ // Set looping options.
+ p.loopMajor = (p.sizeMajor - 1) / 16384 + 1;
+ p.loopMinor = spec.loopMinor;
+ p.loopX = spec.loopX;
+ p.launchMinor = (p.sizeMinor - 1) / p.loopMinor + 1;
+ p.launchMajor = (p.sizeMajor - 1) / p.loopMajor + 1;
+
+ // Compute grid size.
+ dim3 blockSize, gridSize;
+ if (spec.tileOutW < 0) // large
+ {
+ blockSize = dim3(4, 32, 1);
+ gridSize = dim3(
+ ((p.outSize.y - 1) / blockSize.x + 1) * p.launchMinor,
+ (p.outSize.x - 1) / (blockSize.y * p.loopX) + 1,
+ p.launchMajor);
+ }
+ else // small
+ {
+ blockSize = dim3(256, 1, 1);
+ gridSize = dim3(
+ ((p.outSize.y - 1) / spec.tileOutH + 1) * p.launchMinor,
+ (p.outSize.x - 1) / (spec.tileOutW * p.loopX) + 1,
+ p.launchMajor);
+ }
+
+ // Launch CUDA kernel.
+ void* args[] = {&p};
+ AT_CUDA_CHECK(cudaLaunchKernel(spec.kernel, gridSize, blockSize, args, 0, at::cuda::getCurrentCUDAStream()));
+ return y;
+}
+
+//------------------------------------------------------------------------
+
+PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
+{
+ m.def("upfirdn2d", &upfirdn2d);
+}
+
+//------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.cu b/src/stylegan2/torch_utils/ops/upfirdn2d.cu
new file mode 100644
index 0000000..ebdd987
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.cu
@@ -0,0 +1,350 @@
+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto. Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+#include <c10/util/Half.h>
+#include "upfirdn2d.h"
+
+//------------------------------------------------------------------------
+// Helpers.
+
+template <class T> struct InternalType;
+template <> struct InternalType<double> { typedef double scalar_t; };
+template <> struct InternalType<float> { typedef float scalar_t; };
+template <> struct InternalType<c10::Half> { typedef float scalar_t; };
+
+static __device__ __forceinline__ int floor_div(int a, int b)
+{
+ int t = 1 - a / b;
+ return (a + t * b) / b - t;
+}
+
+//------------------------------------------------------------------------
+// Generic CUDA implementation for large filters.
+
+template <class T> static __global__ void upfirdn2d_kernel_large(upfirdn2d_kernel_params p)
+{
+ typedef typename InternalType<T>::scalar_t scalar_t;
+
+ // Calculate thread index.
+ int minorBase = blockIdx.x * blockDim.x + threadIdx.x;
+ int outY = minorBase / p.launchMinor;
+ minorBase -= outY * p.launchMinor;
+ int outXBase = blockIdx.y * p.loopX * blockDim.y + threadIdx.y;
+ int majorBase = blockIdx.z * p.loopMajor;
+ if (outXBase >= p.outSize.x | outY >= p.outSize.y | majorBase >= p.sizeMajor)
+ return;
+
+ // Setup Y receptive field.
+ int midY = outY * p.down.y + p.up.y - 1 - p.pad0.y;
+ int inY = min(max(floor_div(midY, p.up.y), 0), p.inSize.y);
+ int h = min(max(floor_div(midY + p.filterSize.y, p.up.y), 0), p.inSize.y) - inY;
+ int filterY = midY + p.filterSize.y - (inY + 1) * p.up.y;
+ if (p.flip)
+ filterY = p.filterSize.y - 1 - filterY;
+
+ // Loop over major, minor, and X.
+ for (int majorIdx = 0, major = majorBase; majorIdx < p.loopMajor & major < p.sizeMajor; majorIdx++, major++)
+ for (int minorIdx = 0, minor = minorBase; minorIdx < p.loopMinor & minor < p.sizeMinor; minorIdx++, minor += p.launchMinor)
+ {
+ int nc = major * p.sizeMinor + minor;
+ int n = nc / p.inSize.z;
+ int c = nc - n * p.inSize.z;
+ for (int loopX = 0, outX = outXBase; loopX < p.loopX & outX < p.outSize.x; loopX++, outX += blockDim.y)
+ {
+ // Setup X receptive field.
+ int midX = outX * p.down.x + p.up.x - 1 - p.pad0.x;
+ int inX = min(max(floor_div(midX, p.up.x), 0), p.inSize.x);
+ int w = min(max(floor_div(midX + p.filterSize.x, p.up.x), 0), p.inSize.x) - inX;
+ int filterX = midX + p.filterSize.x - (inX + 1) * p.up.x;
+ if (p.flip)
+ filterX = p.filterSize.x - 1 - filterX;
+
+ // Initialize pointers.
+ const T* xp = &((const T*)p.x)[inX * p.inStride.x + inY * p.inStride.y + c * p.inStride.z + n * p.inStride.w];
+ const float* fp = &p.f[filterX * p.filterStride.x + filterY * p.filterStride.y];
+ int filterStepX = ((p.flip) ? p.up.x : -p.up.x) * p.filterStride.x;
+ int filterStepY = ((p.flip) ? p.up.y : -p.up.y) * p.filterStride.y;
+
+ // Inner loop.
+ scalar_t v = 0;
+ for (int y = 0; y < h; y++)
+ {
+ for (int x = 0; x < w; x++)
+ {
+ v += (scalar_t)(*xp) * (scalar_t)(*fp);
+ xp += p.inStride.x;
+ fp += filterStepX;
+ }
+ xp += p.inStride.y - w * p.inStride.x;
+ fp += filterStepY - w * filterStepX;
+ }
+
+ // Store result.
+ v *= p.gain;
+ ((T*)p.y)[outX * p.outStride.x + outY * p.outStride.y + c * p.outStride.z + n * p.outStride.w] = (T)v;
+ }
+ }
+}
+
+//------------------------------------------------------------------------
+// Specialized CUDA implementation for small filters.
+
+template <class T, int upx, int upy, int downx, int downy, int filterW, int filterH, int tileOutW, int tileOutH, int loopMinor>
+static __global__ void upfirdn2d_kernel_small(upfirdn2d_kernel_params p)
+{
+ typedef typename InternalType<T>::scalar_t scalar_t;
+ const int tileInW = ((tileOutW - 1) * downx + filterW - 1) / upx + 1;
+ const int tileInH = ((tileOutH - 1) * downy + filterH - 1) / upy + 1;
+ __shared__ volatile scalar_t sf[filterH][filterW];
+ __shared__ volatile scalar_t sx[tileInH][tileInW][loopMinor];
+
+ // Calculate tile index.
+ int minorBase = blockIdx.x;
+ int tileOutY = minorBase / p.launchMinor;
+ minorBase -= tileOutY * p.launchMinor;
+ minorBase *= loopMinor;
+ tileOutY *= tileOutH;
+ int tileOutXBase = blockIdx.y * p.loopX * tileOutW;
+ int majorBase = blockIdx.z * p.loopMajor;
+ if (tileOutXBase >= p.outSize.x | tileOutY >= p.outSize.y | majorBase >= p.sizeMajor)
+ return;
+
+ // Load filter (flipped).
+ for (int tapIdx = threadIdx.x; tapIdx < filterH * filterW; tapIdx += blockDim.x)
+ {
+ int fy = tapIdx / filterW;
+ int fx = tapIdx - fy * filterW;
+ scalar_t v = 0;
+ if (fx < p.filterSize.x & fy < p.filterSize.y)
+ {
+ int ffx = (p.flip) ? fx : p.filterSize.x - 1 - fx;
+ int ffy = (p.flip) ? fy : p.filterSize.y - 1 - fy;
+ v = (scalar_t)p.f[ffx * p.filterStride.x + ffy * p.filterStride.y];
+ }
+ sf[fy][fx] = v;
+ }
+
+ // Loop over major and X.
+ for (int majorIdx = 0, major = majorBase; majorIdx < p.loopMajor & major < p.sizeMajor; majorIdx++, major++)
+ {
+ int baseNC = major * p.sizeMinor + minorBase;
+ int n = baseNC / p.inSize.z;
+ int baseC = baseNC - n * p.inSize.z;
+ for (int loopX = 0, tileOutX = tileOutXBase; loopX < p.loopX & tileOutX < p.outSize.x; loopX++, tileOutX += tileOutW)
+ {
+ // Load input pixels.
+ int tileMidX = tileOutX * downx + upx - 1 - p.pad0.x;
+ int tileMidY = tileOutY * downy + upy - 1 - p.pad0.y;
+ int tileInX = floor_div(tileMidX, upx);
+ int tileInY = floor_div(tileMidY, upy);
+ __syncthreads();
+ for (int inIdx = threadIdx.x; inIdx < tileInH * tileInW * loopMinor; inIdx += blockDim.x)
+ {
+ int relC = inIdx;
+ int relInX = relC / loopMinor;
+ int relInY = relInX / tileInW;
+ relC -= relInX * loopMinor;
+ relInX -= relInY * tileInW;
+ int c = baseC + relC;
+ int inX = tileInX + relInX;
+ int inY = tileInY + relInY;
+ scalar_t v = 0;
+ if (inX >= 0 & inY >= 0 & inX < p.inSize.x & inY < p.inSize.y & c < p.inSize.z)
+ v = (scalar_t)((const T*)p.x)[inX * p.inStride.x + inY * p.inStride.y + c * p.inStride.z + n * p.inStride.w];
+ sx[relInY][relInX][relC] = v;
+ }
+
+ // Loop over output pixels.
+ __syncthreads();
+ for (int outIdx = threadIdx.x; outIdx < tileOutH * tileOutW * loopMinor; outIdx += blockDim.x)
+ {
+ int relC = outIdx;
+ int relOutX = relC / loopMinor;
+ int relOutY = relOutX / tileOutW;
+ relC -= relOutX * loopMinor;
+ relOutX -= relOutY * tileOutW;
+ int c = baseC + relC;
+ int outX = tileOutX + relOutX;
+ int outY = tileOutY + relOutY;
+
+ // Setup receptive field.
+ int midX = tileMidX + relOutX * downx;
+ int midY = tileMidY + relOutY * downy;
+ int inX = floor_div(midX, upx);
+ int inY = floor_div(midY, upy);
+ int relInX = inX - tileInX;
+ int relInY = inY - tileInY;
+ int filterX = (inX + 1) * upx - midX - 1; // flipped
+ int filterY = (inY + 1) * upy - midY - 1; // flipped
+
+ // Inner loop.
+ if (outX < p.outSize.x & outY < p.outSize.y & c < p.outSize.z)
+ {
+ scalar_t v = 0;
+ #pragma unroll
+ for (int y = 0; y < filterH / upy; y++)
+ #pragma unroll
+ for (int x = 0; x < filterW / upx; x++)
+ v += sx[relInY + y][relInX + x][relC] * sf[filterY + y * upy][filterX + x * upx];
+ v *= p.gain;
+ ((T*)p.y)[outX * p.outStride.x + outY * p.outStride.y + c * p.outStride.z + n * p.outStride.w] = (T)v;
+ }
+ }
+ }
+ }
+}
+
+//------------------------------------------------------------------------
+// CUDA kernel selection.
+
+template <class T> upfirdn2d_kernel_spec choose_upfirdn2d_kernel(const upfirdn2d_kernel_params& p)
+{
+ int s = p.inStride.z, fx = p.filterSize.x, fy = p.filterSize.y;
+
+ upfirdn2d_kernel_spec spec = {(void*)upfirdn2d_kernel_large<T>, -1,-1,1, 4}; // contiguous
+ if (s == 1) spec = {(void*)upfirdn2d_kernel_large<T>, -1,-1,4, 1}; // channels_last
+
+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // contiguous
+ {
+ if (fx <= 7 && fy <= 7 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 7,7, 64,16,1>, 64,16,1, 1};
+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 6,6, 64,16,1>, 64,16,1, 1};
+ if (fx <= 5 && fy <= 5 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 5,5, 64,16,1>, 64,16,1, 1};
+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 64,16,1>, 64,16,1, 1};
+ if (fx <= 3 && fy <= 3 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 3,3, 64,16,1>, 64,16,1, 1};
+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 24,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 20,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 16,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 12,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 8,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,24, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,20, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,16, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,12, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,8, 32,32,1>, 32,32,1, 1};
+ }
+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // channels_last
+ {
+ if (fx <= 7 && fy <= 7 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 7,7, 16,16,8>, 16,16,8, 1};
+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};
+ if (fx <= 5 && fy <= 5 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};
+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};
+ if (fx <= 3 && fy <= 3 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};
+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 24,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 20,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 16,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 12,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 8,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,24, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,20, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,16, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,12, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,8, 1,128,16>, 1,128,16, 1};
+ }
+ if (s != 1 && p.up.x == 2 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // contiguous
+ {
+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 8,8, 64,16,1>, 64,16,1, 1};
+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 6,6, 64,16,1>, 64,16,1, 1};
+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 4,4, 64,16,1>, 64,16,1, 1};
+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 2,2, 64,16,1>, 64,16,1, 1};
+ }
+ if (s == 1 && p.up.x == 2 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // channels_last
+ {
+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 8,8, 16,16,8>, 16,16,8, 1};
+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 6,6, 16,16,8>, 16,16,8, 1};
+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 4,4, 16,16,8>, 16,16,8, 1};
+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 2,2, 16,16,8>, 16,16,8, 1};
+ }
+ if (s != 1 && p.up.x == 2 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // contiguous
+ {
+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 24,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 20,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 16,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 12,1, 128,8,1>, 128,8,1, 1};
+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 8,1, 128,8,1>, 128,8,1, 1};
+ }
+ if (s == 1 && p.up.x == 2 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // channels_last
+ {
+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 24,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 20,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 16,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 12,1, 128,1,16>, 128,1,16, 1};
+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 8,1, 128,1,16>, 128,1,16, 1};
+ }
+ if (s != 1 && p.up.x == 1 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // contiguous
+ {
+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,24, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,20, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,16, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,12, 32,32,1>, 32,32,1, 1};
+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,8, 32,32,1>, 32,32,1, 1};
+ }
+ if (s == 1 && p.up.x == 1 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // channels_last
+ {
+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,24, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,20, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,16, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,12, 1,128,16>, 1,128,16, 1};
+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,8, 1,128,16>, 1,128,16, 1};
+ }
+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 2) // contiguous
+ {
+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 8,8, 32,8,1>, 32,8,1, 1};
+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 6,6, 32,8,1>, 32,8,1, 1};
+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 4,4, 32,8,1>, 32,8,1, 1};
+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 2,2, 32,8,1>, 32,8,1, 1};
+ }
+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 2) // channels_last
+ {
+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 8,8, 8,8,8>, 8,8,8, 1};
+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 6,6, 8,8,8>, 8,8,8, 1};
+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 4,4, 8,8,8>, 8,8,8, 1};
+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 2,2, 8,8,8>, 8,8,8, 1};
+ }
+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 1) // contiguous
+ {
+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 24,1, 64,8,1>, 64,8,1, 1};
+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 20,1, 64,8,1>, 64,8,1, 1};
+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 16,1, 64,8,1>, 64,8,1, 1};
+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 12,1, 64,8,1>, 64,8,1, 1};
+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 8,1, 64,8,1>, 64,8,1, 1};
+ }
+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 1) // channels_last
+ {
+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 24,1, 64,1,8>, 64,1,8, 1};
+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 20,1, 64,1,8>, 64,1,8, 1};
+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 16,1, 64,1,8>, 64,1,8, 1};
+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 12,1, 64,1,8>, 64,1,8, 1};
+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 8,1, 64,1,8>, 64,1,8, 1};
+ }
+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 2) // contiguous
+ {
+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,24, 32,16,1>, 32,16,1, 1};
+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,20, 32,16,1>, 32,16,1, 1};
+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,16, 32,16,1>, 32,16,1, 1};
+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,12, 32,16,1>, 32,16,1, 1};
+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,8, 32,16,1>, 32,16,1, 1};
+ }
+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 2) // channels_last
+ {
+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,24, 1,64,8>, 1,64,8, 1};
+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,20, 1,64,8>, 1,64,8, 1};
+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,16, 1,64,8>, 1,64,8, 1};
+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,12, 1,64,8>, 1,64,8, 1};
+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,8, 1,64,8>, 1,64,8, 1};
+ }
+ return spec;
+}
+
+//------------------------------------------------------------------------
+// Template specializations.
+
+template upfirdn2d_kernel_spec choose_upfirdn2d_kernel<double> (const upfirdn2d_kernel_params& p);
+template upfirdn2d_kernel_spec choose_upfirdn2d_kernel<float> (const upfirdn2d_kernel_params& p);
+template upfirdn2d_kernel_spec choose_upfirdn2d_kernel<c10::Half>(const upfirdn2d_kernel_params& p);
+
+//------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.h b/src/stylegan2/torch_utils/ops/upfirdn2d.h
new file mode 100644
index 0000000..c9e2032
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.h
@@ -0,0 +1,59 @@
+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto. Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+#include <cuda_runtime.h>
+
+//------------------------------------------------------------------------
+// CUDA kernel parameters.
+
+struct upfirdn2d_kernel_params
+{
+ const void* x;
+ const float* f;
+ void* y;
+
+ int2 up;
+ int2 down;
+ int2 pad0;
+ int flip;
+ float gain;
+
+ int4 inSize; // [width, height, channel, batch]
+ int4 inStride;
+ int2 filterSize; // [width, height]
+ int2 filterStride;
+ int4 outSize; // [width, height, channel, batch]
+ int4 outStride;
+ int sizeMinor;
+ int sizeMajor;
+
+ int loopMinor;
+ int loopMajor;
+ int loopX;
+ int launchMinor;
+ int launchMajor;
+};
+
+//------------------------------------------------------------------------
+// CUDA kernel specialization.
+
+struct upfirdn2d_kernel_spec
+{
+ void* kernel;
+ int tileOutW;
+ int tileOutH;
+ int loopMinor;
+ int loopX;
+};
+
+//------------------------------------------------------------------------
+// CUDA kernel selection.
+
+template <class T> upfirdn2d_kernel_spec choose_upfirdn2d_kernel(const upfirdn2d_kernel_params& p);
+
+//------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.py b/src/stylegan2/torch_utils/ops/upfirdn2d.py
new file mode 100644
index 0000000..ceeac2b
--- /dev/null
+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.py
@@ -0,0 +1,384 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Custom PyTorch ops for efficient resampling of 2D images."""
+
+import os
+import warnings
+import numpy as np
+import torch
+import traceback
+
+from .. import custom_ops
+from .. import misc
+from . import conv2d_gradfix
+
+#----------------------------------------------------------------------------
+
+_inited = False
+_plugin = None
+
+def _init():
+ global _inited, _plugin
+ if not _inited:
+ sources = ['upfirdn2d.cpp', 'upfirdn2d.cu']
+ sources = [os.path.join(os.path.dirname(__file__), s) for s in sources]
+ try:
+ _plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
+ except:
+ warnings.warn('Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())
+ return _plugin is not None
+
+def _parse_scaling(scaling):
+ if isinstance(scaling, int):
+ scaling = [scaling, scaling]
+ assert isinstance(scaling, (list, tuple))
+ assert all(isinstance(x, int) for x in scaling)
+ sx, sy = scaling
+ assert sx >= 1 and sy >= 1
+ return sx, sy
+
+def _parse_padding(padding):
+ if isinstance(padding, int):
+ padding = [padding, padding]
+ assert isinstance(padding, (list, tuple))
+ assert all(isinstance(x, int) for x in padding)
+ if len(padding) == 2:
+ padx, pady = padding
+ padding = [padx, padx, pady, pady]
+ padx0, padx1, pady0, pady1 = padding
+ return padx0, padx1, pady0, pady1
+
+def _get_filter_size(f):
+ if f is None:
+ return 1, 1
+ assert isinstance(f, torch.Tensor) and f.ndim in [1, 2]
+ fw = f.shape[-1]
+ fh = f.shape[0]
+ with misc.suppress_tracer_warnings():
+ fw = int(fw)
+ fh = int(fh)
+ misc.assert_shape(f, [fh, fw][:f.ndim])
+ assert fw >= 1 and fh >= 1
+ return fw, fh
+
+#----------------------------------------------------------------------------
+
+def setup_filter(f, device=torch.device('cpu'), normalize=True, flip_filter=False, gain=1, separable=None):
+ r"""Convenience function to setup 2D FIR filter for `upfirdn2d()`.
+
+ Args:
+ f: Torch tensor, numpy array, or python list of the shape
+ `[filter_height, filter_width]` (non-separable),
+ `[filter_taps]` (separable),
+ `[]` (impulse), or
+ `None` (identity).
+ device: Result device (default: cpu).
+ normalize: Normalize the filter so that it retains the magnitude
+ for constant input signal (DC)? (default: True).
+ flip_filter: Flip the filter? (default: False).
+ gain: Overall scaling factor for signal magnitude (default: 1).
+ separable: Return a separable filter? (default: select automatically).
+
+ Returns:
+ Float32 tensor of the shape
+ `[filter_height, filter_width]` (non-separable) or
+ `[filter_taps]` (separable).
+ """
+ # Validate.
+ if f is None:
+ f = 1
+ f = torch.as_tensor(f, dtype=torch.float32)
+ assert f.ndim in [0, 1, 2]
+ assert f.numel() > 0
+ if f.ndim == 0:
+ f = f[np.newaxis]
+
+ # Separable?
+ if separable is None:
+ separable = (f.ndim == 1 and f.numel() >= 8)
+ if f.ndim == 1 and not separable:
+ f = f.ger(f)
+ assert f.ndim == (1 if separable else 2)
+
+ # Apply normalize, flip, gain, and device.
+ if normalize:
+ f /= f.sum()
+ if flip_filter:
+ f = f.flip(list(range(f.ndim)))
+ f = f * (gain ** (f.ndim / 2))
+ f = f.to(device=device)
+ return f
+
+#----------------------------------------------------------------------------
+
+def upfirdn2d(x, f, up=1, down=1, padding=0, flip_filter=False, gain=1, impl='cuda'):
+ r"""Pad, upsample, filter, and downsample a batch of 2D images.
+
+ Performs the following sequence of operations for each channel:
+
+ 1. Upsample the image by inserting N-1 zeros after each pixel (`up`).
+
+ 2. Pad the image with the specified number of zeros on each side (`padding`).
+ Negative padding corresponds to cropping the image.
+
+ 3. Convolve the image with the specified 2D FIR filter (`f`), shrinking it
+ so that the footprint of all output pixels lies within the input image.
+
+ 4. Downsample the image by keeping every Nth pixel (`down`).
+
+ This sequence of operations bears close resemblance to scipy.signal.upfirdn().
+ The fused op is considerably more efficient than performing the same calculation
+ using standard PyTorch ops. It supports gradients of arbitrary order.
+
+ Args:
+ x: Float32/float64/float16 input tensor of the shape
+ `[batch_size, num_channels, in_height, in_width]`.
+ f: Float32 FIR filter of the shape
+ `[filter_height, filter_width]` (non-separable),
+ `[filter_taps]` (separable), or
+ `None` (identity).
+ up: Integer upsampling factor. Can be a single int or a list/tuple
+ `[x, y]` (default: 1).
+ down: Integer downsampling factor. Can be a single int or a list/tuple
+ `[x, y]` (default: 1).
+ padding: Padding with respect to the upsampled image. Can be a single number
+ or a list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`
+ (default: 0).
+ flip_filter: False = convolution, True = correlation (default: False).
+ gain: Overall scaling factor for signal magnitude (default: 1).
+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).
+
+ Returns:
+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.
+ """
+ assert isinstance(x, torch.Tensor)
+ assert impl in ['ref', 'cuda']
+ if impl == 'cuda' and x.device.type == 'cuda' and _init():
+ return _upfirdn2d_cuda(up=up, down=down, padding=padding, flip_filter=flip_filter, gain=gain).apply(x, f)
+ return _upfirdn2d_ref(x, f, up=up, down=down, padding=padding, flip_filter=flip_filter, gain=gain)
+
+#----------------------------------------------------------------------------
+
+def _upfirdn2d_ref(x, f, up=1, down=1, padding=0, flip_filter=False, gain=1):
+ """Slow reference implementation of `upfirdn2d()` using standard PyTorch ops.
+ """
+ # Validate arguments.
+ assert isinstance(x, torch.Tensor) and x.ndim == 4
+ if f is None:
+ f = torch.ones([1, 1], dtype=torch.float32, device=x.device)
+ assert isinstance(f, torch.Tensor) and f.ndim in [1, 2]
+ assert f.dtype == torch.float32 and not f.requires_grad
+ batch_size, num_channels, in_height, in_width = x.shape
+ upx, upy = _parse_scaling(up)
+ downx, downy = _parse_scaling(down)
+ padx0, padx1, pady0, pady1 = _parse_padding(padding)
+
+ # Upsample by inserting zeros.
+ x = x.reshape([batch_size, num_channels, in_height, 1, in_width, 1])
+ x = torch.nn.functional.pad(x, [0, upx - 1, 0, 0, 0, upy - 1])
+ x = x.reshape([batch_size, num_channels, in_height * upy, in_width * upx])
+
+ # Pad or crop.
+ x = torch.nn.functional.pad(x, [max(padx0, 0), max(padx1, 0), max(pady0, 0), max(pady1, 0)])
+ x = x[:, :, max(-pady0, 0) : x.shape[2] - max(-pady1, 0), max(-padx0, 0) : x.shape[3] - max(-padx1, 0)]
+
+ # Setup filter.
+ f = f * (gain ** (f.ndim / 2))
+ f = f.to(x.dtype)
+ if not flip_filter:
+ f = f.flip(list(range(f.ndim)))
+
+ # Convolve with the filter.
+ f = f[np.newaxis, np.newaxis].repeat([num_channels, 1] + [1] * f.ndim)
+ if f.ndim == 4:
+ x = conv2d_gradfix.conv2d(input=x, weight=f, groups=num_channels)
+ else:
+ x = conv2d_gradfix.conv2d(input=x, weight=f.unsqueeze(2), groups=num_channels)
+ x = conv2d_gradfix.conv2d(input=x, weight=f.unsqueeze(3), groups=num_channels)
+
+ # Downsample by throwing away pixels.
+ x = x[:, :, ::downy, ::downx]
+ return x
+
+#----------------------------------------------------------------------------
+
+_upfirdn2d_cuda_cache = dict()
+
+def _upfirdn2d_cuda(up=1, down=1, padding=0, flip_filter=False, gain=1):
+ """Fast CUDA implementation of `upfirdn2d()` using custom ops.
+ """
+ # Parse arguments.
+ upx, upy = _parse_scaling(up)
+ downx, downy = _parse_scaling(down)
+ padx0, padx1, pady0, pady1 = _parse_padding(padding)
+
+ # Lookup from cache.
+ key = (upx, upy, downx, downy, padx0, padx1, pady0, pady1, flip_filter, gain)
+ if key in _upfirdn2d_cuda_cache:
+ return _upfirdn2d_cuda_cache[key]
+
+ # Forward op.
+ class Upfirdn2dCuda(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, x, f): # pylint: disable=arguments-differ
+ assert isinstance(x, torch.Tensor) and x.ndim == 4
+ if f is None:
+ f = torch.ones([1, 1], dtype=torch.float32, device=x.device)
+ assert isinstance(f, torch.Tensor) and f.ndim in [1, 2]
+ y = x
+ if f.ndim == 2:
+ y = _plugin.upfirdn2d(y, f, upx, upy, downx, downy, padx0, padx1, pady0, pady1, flip_filter, gain)
+ else:
+ y = _plugin.upfirdn2d(y, f.unsqueeze(0), upx, 1, downx, 1, padx0, padx1, 0, 0, flip_filter, np.sqrt(gain))
+ y = _plugin.upfirdn2d(y, f.unsqueeze(1), 1, upy, 1, downy, 0, 0, pady0, pady1, flip_filter, np.sqrt(gain))
+ ctx.save_for_backward(f)
+ ctx.x_shape = x.shape
+ return y
+
+ @staticmethod
+ def backward(ctx, dy): # pylint: disable=arguments-differ
+ f, = ctx.saved_tensors
+ _, _, ih, iw = ctx.x_shape
+ _, _, oh, ow = dy.shape
+ fw, fh = _get_filter_size(f)
+ p = [
+ fw - padx0 - 1,
+ iw * upx - ow * downx + padx0 - upx + 1,
+ fh - pady0 - 1,
+ ih * upy - oh * downy + pady0 - upy + 1,
+ ]
+ dx = None
+ df = None
+
+ if ctx.needs_input_grad[0]:
+ dx = _upfirdn2d_cuda(up=down, down=up, padding=p, flip_filter=(not flip_filter), gain=gain).apply(dy, f)
+
+ assert not ctx.needs_input_grad[1]
+ return dx, df
+
+ # Add to cache.
+ _upfirdn2d_cuda_cache[key] = Upfirdn2dCuda
+ return Upfirdn2dCuda
+
+#----------------------------------------------------------------------------
+
+def filter2d(x, f, padding=0, flip_filter=False, gain=1, impl='cuda'):
+ r"""Filter a batch of 2D images using the given 2D FIR filter.
+
+ By default, the result is padded so that its shape matches the input.
+ User-specified padding is applied on top of that, with negative values
+ indicating cropping. Pixels outside the image are assumed to be zero.
+
+ Args:
+ x: Float32/float64/float16 input tensor of the shape
+ `[batch_size, num_channels, in_height, in_width]`.
+ f: Float32 FIR filter of the shape
+ `[filter_height, filter_width]` (non-separable),
+ `[filter_taps]` (separable), or
+ `None` (identity).
+ padding: Padding with respect to the output. Can be a single number or a
+ list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`
+ (default: 0).
+ flip_filter: False = convolution, True = correlation (default: False).
+ gain: Overall scaling factor for signal magnitude (default: 1).
+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).
+
+ Returns:
+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.
+ """
+ padx0, padx1, pady0, pady1 = _parse_padding(padding)
+ fw, fh = _get_filter_size(f)
+ p = [
+ padx0 + fw // 2,
+ padx1 + (fw - 1) // 2,
+ pady0 + fh // 2,
+ pady1 + (fh - 1) // 2,
+ ]
+ return upfirdn2d(x, f, padding=p, flip_filter=flip_filter, gain=gain, impl=impl)
+
+#----------------------------------------------------------------------------
+
+def upsample2d(x, f, up=2, padding=0, flip_filter=False, gain=1, impl='cuda'):
+ r"""Upsample a batch of 2D images using the given 2D FIR filter.
+
+ By default, the result is padded so that its shape is a multiple of the input.
+ User-specified padding is applied on top of that, with negative values
+ indicating cropping. Pixels outside the image are assumed to be zero.
+
+ Args:
+ x: Float32/float64/float16 input tensor of the shape
+ `[batch_size, num_channels, in_height, in_width]`.
+ f: Float32 FIR filter of the shape
+ `[filter_height, filter_width]` (non-separable),
+ `[filter_taps]` (separable), or
+ `None` (identity).
+ up: Integer upsampling factor. Can be a single int or a list/tuple
+ `[x, y]` (default: 1).
+ padding: Padding with respect to the output. Can be a single number or a
+ list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`
+ (default: 0).
+ flip_filter: False = convolution, True = correlation (default: False).
+ gain: Overall scaling factor for signal magnitude (default: 1).
+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).
+
+ Returns:
+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.
+ """
+ upx, upy = _parse_scaling(up)
+ padx0, padx1, pady0, pady1 = _parse_padding(padding)
+ fw, fh = _get_filter_size(f)
+ p = [
+ padx0 + (fw + upx - 1) // 2,
+ padx1 + (fw - upx) // 2,
+ pady0 + (fh + upy - 1) // 2,
+ pady1 + (fh - upy) // 2,
+ ]
+ return upfirdn2d(x, f, up=up, padding=p, flip_filter=flip_filter, gain=gain*upx*upy, impl=impl)
+
+#----------------------------------------------------------------------------
+
+def downsample2d(x, f, down=2, padding=0, flip_filter=False, gain=1, impl='cuda'):
+ r"""Downsample a batch of 2D images using the given 2D FIR filter.
+
+ By default, the result is padded so that its shape is a fraction of the input.
+ User-specified padding is applied on top of that, with negative values
+ indicating cropping. Pixels outside the image are assumed to be zero.
+
+ Args:
+ x: Float32/float64/float16 input tensor of the shape
+ `[batch_size, num_channels, in_height, in_width]`.
+ f: Float32 FIR filter of the shape
+ `[filter_height, filter_width]` (non-separable),
+ `[filter_taps]` (separable), or
+ `None` (identity).
+ down: Integer downsampling factor. Can be a single int or a list/tuple
+ `[x, y]` (default: 1).
+ padding: Padding with respect to the input. Can be a single number or a
+ list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`
+ (default: 0).
+ flip_filter: False = convolution, True = correlation (default: False).
+ gain: Overall scaling factor for signal magnitude (default: 1).
+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).
+
+ Returns:
+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.
+ """
+ downx, downy = _parse_scaling(down)
+ padx0, padx1, pady0, pady1 = _parse_padding(padding)
+ fw, fh = _get_filter_size(f)
+ p = [
+ padx0 + (fw - downx + 1) // 2,
+ padx1 + (fw - downx) // 2,
+ pady0 + (fh - downy + 1) // 2,
+ pady1 + (fh - downy) // 2,
+ ]
+ return upfirdn2d(x, f, down=down, padding=p, flip_filter=flip_filter, gain=gain, impl=impl)
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/persistence.py b/src/stylegan2/torch_utils/persistence.py
new file mode 100644
index 0000000..0186cfd
--- /dev/null
+++ b/src/stylegan2/torch_utils/persistence.py
@@ -0,0 +1,251 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Facilities for pickling Python code alongside other data.
+
+The pickled code is automatically imported into a separate Python module
+during unpickling. This way, any previously exported pickles will remain
+usable even if the original code is no longer available, or if the current
+version of the code is not consistent with what was originally pickled."""
+
+import sys
+import pickle
+import io
+import inspect
+import copy
+import uuid
+import types
+import dnnlib
+
+#----------------------------------------------------------------------------
+
+_version = 6 # internal version number
+_decorators = set() # {decorator_class, ...}
+_import_hooks = [] # [hook_function, ...]
+_module_to_src_dict = dict() # {module: src, ...}
+_src_to_module_dict = dict() # {src: module, ...}
+
+#----------------------------------------------------------------------------
+
+def persistent_class(orig_class):
+ r"""Class decorator that extends a given class to save its source code
+ when pickled.
+
+ Example:
+
+ from torch_utils import persistence
+
+ @persistence.persistent_class
+ class MyNetwork(torch.nn.Module):
+ def __init__(self, num_inputs, num_outputs):
+ super().__init__()
+ self.fc = MyLayer(num_inputs, num_outputs)
+ ...
+
+ @persistence.persistent_class
+ class MyLayer(torch.nn.Module):
+ ...
+
+ When pickled, any instance of `MyNetwork` and `MyLayer` will save its
+ source code alongside other internal state (e.g., parameters, buffers,
+ and submodules). This way, any previously exported pickle will remain
+ usable even if the class definitions have been modified or are no
+ longer available.
+
+ The decorator saves the source code of the entire Python module
+ containing the decorated class. It does *not* save the source code of
+ any imported modules. Thus, the imported modules must be available
+ during unpickling, also including `torch_utils.persistence` itself.
+
+ It is ok to call functions defined in the same module from the
+ decorated class. However, if the decorated class depends on other
+ classes defined in the same module, they must be decorated as well.
+ This is illustrated in the above example in the case of `MyLayer`.
+
+ It is also possible to employ the decorator just-in-time before
+ calling the constructor. For example:
+
+ cls = MyLayer
+ if want_to_make_it_persistent:
+ cls = persistence.persistent_class(cls)
+ layer = cls(num_inputs, num_outputs)
+
+ As an additional feature, the decorator also keeps track of the
+ arguments that were used to construct each instance of the decorated
+ class. The arguments can be queried via `obj.init_args` and
+ `obj.init_kwargs`, and they are automatically pickled alongside other
+ object state. A typical use case is to first unpickle a previous
+ instance of a persistent class, and then upgrade it to use the latest
+ version of the source code:
+
+ with open('old_pickle.pkl', 'rb') as f:
+ old_net = pickle.load(f)
+ new_net = MyNetwork(*old_obj.init_args, **old_obj.init_kwargs)
+ misc.copy_params_and_buffers(old_net, new_net, require_all=True)
+ """
+ assert isinstance(orig_class, type)
+ if is_persistent(orig_class):
+ return orig_class
+
+ assert orig_class.__module__ in sys.modules
+ orig_module = sys.modules[orig_class.__module__]
+ orig_module_src = _module_to_src(orig_module)
+
+ class Decorator(orig_class):
+ _orig_module_src = orig_module_src
+ _orig_class_name = orig_class.__name__
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._init_args = copy.deepcopy(args)
+ self._init_kwargs = copy.deepcopy(kwargs)
+ assert orig_class.__name__ in orig_module.__dict__
+ _check_pickleable(self.__reduce__())
+
+ @property
+ def init_args(self):
+ return copy.deepcopy(self._init_args)
+
+ @property
+ def init_kwargs(self):
+ return dnnlib.EasyDict(copy.deepcopy(self._init_kwargs))
+
+ def __reduce__(self):
+ fields = list(super().__reduce__())
+ fields += [None] * max(3 - len(fields), 0)
+ if fields[0] is not _reconstruct_persistent_obj:
+ meta = dict(type='class', version=_version, module_src=self._orig_module_src, class_name=self._orig_class_name, state=fields[2])
+ fields[0] = _reconstruct_persistent_obj # reconstruct func
+ fields[1] = (meta,) # reconstruct args
+ fields[2] = None # state dict
+ return tuple(fields)
+
+ Decorator.__name__ = orig_class.__name__
+ _decorators.add(Decorator)
+ return Decorator
+
+#----------------------------------------------------------------------------
+
+def is_persistent(obj):
+ r"""Test whether the given object or class is persistent, i.e.,
+ whether it will save its source code when pickled.
+ """
+ try:
+ if obj in _decorators:
+ return True
+ except TypeError:
+ pass
+ return type(obj) in _decorators # pylint: disable=unidiomatic-typecheck
+
+#----------------------------------------------------------------------------
+
+def import_hook(hook):
+ r"""Register an import hook that is called whenever a persistent object
+ is being unpickled. A typical use case is to patch the pickled source
+ code to avoid errors and inconsistencies when the API of some imported
+ module has changed.
+
+ The hook should have the following signature:
+
+ hook(meta) -> modified meta
+
+ `meta` is an instance of `dnnlib.EasyDict` with the following fields:
+
+ type: Type of the persistent object, e.g. `'class'`.
+ version: Internal version number of `torch_utils.persistence`.
+ module_src Original source code of the Python module.
+ class_name: Class name in the original Python module.
+ state: Internal state of the object.
+
+ Example:
+
+ @persistence.import_hook
+ def wreck_my_network(meta):
+ if meta.class_name == 'MyNetwork':
+ print('MyNetwork is being imported. I will wreck it!')
+ meta.module_src = meta.module_src.replace("True", "False")
+ return meta
+ """
+ assert callable(hook)
+ _import_hooks.append(hook)
+
+#----------------------------------------------------------------------------
+
+def _reconstruct_persistent_obj(meta):
+ r"""Hook that is called internally by the `pickle` module to unpickle
+ a persistent object.
+ """
+ meta = dnnlib.EasyDict(meta)
+ meta.state = dnnlib.EasyDict(meta.state)
+ for hook in _import_hooks:
+ meta = hook(meta)
+ assert meta is not None
+
+ assert meta.version == _version
+ module = _src_to_module(meta.module_src)
+
+ assert meta.type == 'class'
+ orig_class = module.__dict__[meta.class_name]
+ decorator_class = persistent_class(orig_class)
+ obj = decorator_class.__new__(decorator_class)
+
+ setstate = getattr(obj, '__setstate__', None)
+ if callable(setstate):
+ setstate(meta.state) # pylint: disable=not-callable
+ else:
+ obj.__dict__.update(meta.state)
+ return obj
+
+#----------------------------------------------------------------------------
+
+def _module_to_src(module):
+ r"""Query the source code of a given Python module.
+ """
+ src = _module_to_src_dict.get(module, None)
+ if src is None:
+ src = inspect.getsource(module)
+ _module_to_src_dict[module] = src
+ _src_to_module_dict[src] = module
+ return src
+
+def _src_to_module(src):
+ r"""Get or create a Python module for the given source code.
+ """
+ module = _src_to_module_dict.get(src, None)
+ if module is None:
+ module_name = "_imported_module_" + uuid.uuid4().hex
+ module = types.ModuleType(module_name)
+ sys.modules[module_name] = module
+ _module_to_src_dict[module] = src
+ _src_to_module_dict[src] = module
+ exec(src, module.__dict__) # pylint: disable=exec-used
+ return module
+
+#----------------------------------------------------------------------------
+
+def _check_pickleable(obj):
+ r"""Check that the given object is pickleable, raising an exception if
+ it is not. This function is expected to be considerably more efficient
+ than actually pickling the object.
+ """
+ def recurse(obj):
+ if isinstance(obj, (list, tuple, set)):
+ return [recurse(x) for x in obj]
+ if isinstance(obj, dict):
+ return [[recurse(x), recurse(y)] for x, y in obj.items()]
+ if isinstance(obj, (str, int, float, bool, bytes, bytearray)):
+ return None # Python primitive types are pickleable.
+ if f'{type(obj).__module__}.{type(obj).__name__}' in ['numpy.ndarray', 'torch.Tensor']:
+ return None # NumPy arrays and PyTorch tensors are pickleable.
+ if is_persistent(obj):
+ return None # Persistent objects are pickleable, by virtue of the constructor check.
+ return obj
+ with io.BytesIO() as f:
+ pickle.dump(recurse(obj), f)
+
+#----------------------------------------------------------------------------
diff --git a/src/stylegan2/torch_utils/training_stats.py b/src/stylegan2/torch_utils/training_stats.py
new file mode 100644
index 0000000..26f467f
--- /dev/null
+++ b/src/stylegan2/torch_utils/training_stats.py
@@ -0,0 +1,268 @@
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto. Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""Facilities for reporting and collecting training statistics across
+multiple processes and devices. The interface is designed to minimize
+synchronization overhead as well as the amount of boilerplate in user
+code."""
+
+import re
+import numpy as np
+import torch
+import dnnlib
+
+from . import misc
+
+#----------------------------------------------------------------------------
+
+_num_moments = 3 # [num_scalars, sum_of_scalars, sum_of_squares]
+_reduce_dtype = torch.float32 # Data type to use for initial per-tensor reduction.
+_counter_dtype = torch.float64 # Data type to use for the internal counters.
+_rank = 0 # Rank of the current process.
+_sync_device = None # Device to use for multiprocess communication. None = single-process.
+_sync_called = False # Has _sync() been called yet?
+_counters = dict() # Running counters on each device, updated by report(): name => device => torch.Tensor
+_cumulative = dict() # Cumulative counters on the CPU, updated by _sync(): name => torch.Tensor
+
+#----------------------------------------------------------------------------
+
+def init_multiprocessing(rank, sync_device):
+ r"""Initializes `torch_utils.training_stats` for collecting statistics
+ across multiple processes.
+
+ This function must be called after
+ `torch.distributed.init_process_group()` and before `Collector.update()`.
+ The call is not necessary if multi-process collection is not needed.
+
+ Args:
+ rank: Rank of the current process.
+ sync_device: PyTorch device to use for inter-process
+ communication, or None to disable multi-process
+ collection. Typically `torch.device('cuda', rank)`.
+ """
+ global _rank, _sync_device
+ assert not _sync_called
+ _rank = rank
+ _sync_device = sync_device
+
+#----------------------------------------------------------------------------
+
+def report(name, value):
+ r"""Broadcasts the given set of scalars to all interested instances of
+ `Collector`, across device and process boundaries.
+
+ This function is expected to be extremely cheap and can be safely
+ called from anywhere in the training loop, loss function, or inside a
+ `torch.nn.Module`.
+
+ Warning: The current implementation expects the set of unique names to
+ be consistent across processes. Please make sure that `report()` is
+ called at least once for each unique name by each process, and in the
+ same order. If a given process has no scalars to broadcast, it can do
+ `report(name, [])` (empty list).
+
+ Args:
+ name: Arbitrary string specifying the name of the statistic.
+ Averages are accumulated separately for each unique name.
+ value: Arbitrary set of scalars. Can be a list, tuple,
+ NumPy array, PyTorch tensor, or Python scalar.
+
+ Returns:
+ The same `value` that was passed in.
+ """
+ if name not in _counters:
+ _counters[name] = dict()
+
+ elems = torch.as_tensor(value)
+ if elems.numel() == 0:
+ return value
+
+ elems = elems.detach().flatten().to(_reduce_dtype)
+ moments = torch.stack([
+ torch.ones_like(elems).sum(),
+ elems.sum(),
+ elems.square().sum(),
+ ])
+ assert moments.ndim == 1 and moments.shape[0] == _num_moments
+ moments = moments.to(_counter_dtype)
+
+ device = moments.device
+ if device not in _counters[name]:
+ _counters[name][device] = torch.zeros_like(moments)
+ _counters[name][device].add_(moments)
+ return value
+
+#----------------------------------------------------------------------------
+
+def report0(name, value):
+ r"""Broadcasts the given set of scalars by the first process (`rank = 0`),
+ but ignores any scalars provided by the other processes.
+ See `report()` for further details.
+ """
+ report(name, value if _rank == 0 else [])
+ return value
+
+#----------------------------------------------------------------------------
+
+class Collector:
+ r"""Collects the scalars broadcasted by `report()` and `report0()` and
+ computes their long-term averages (mean and standard deviation) over
+ user-defined periods of time.
+
+ The averages are first collected into internal counters that are not
+ directly visible to the user. They are then copied to the user-visible
+ state as a result of calling `update()` and can then be queried using
+ `mean()`, `std()`, `as_dict()`, etc. Calling `update()` also resets the
+ internal counters for the next round, so that the user-visible state
+ effectively reflects averages collected between the last two calls to
+ `update()`.
+
+ Args:
+ regex: Regular expression defining which statistics to
+ collect. The default is to collect everything.
+ keep_previous: Whether to retain the previous averages if no
+ scalars were collected on a given round
+ (default: True).
+ """
+ def __init__(self, regex='.*', keep_previous=True):
+ self._regex = re.compile(regex)
+ self._keep_previous = keep_previous
+ self._cumulative = dict()
+ self._moments = dict()
+ self.update()
+ self._moments.clear()
+
+ def names(self):
+ r"""Returns the names of all statistics broadcasted so far that
+ match the regular expression specified at construction time.
+ """
+ return [name for name in _counters if self._regex.fullmatch(name)]
+
+ def update(self):
+ r"""Copies current values of the internal counters to the
+ user-visible state and resets them for the next round.
+
+ If `keep_previous=True` was specified at construction time, the
+ operation is skipped for statistics that have received no scalars
+ since the last update, retaining their previous averages.
+
+ This method performs a number of GPU-to-CPU transfers and one
+ `torch.distributed.all_reduce()`. It is intended to be called
+ periodically in the main training loop, typically once every
+ N training steps.
+ """
+ if not self._keep_previous:
+ self._moments.clear()
+ for name, cumulative in _sync(self.names()):
+ if name not in self._cumulative:
+ self._cumulative[name] = torch.zeros([_num_moments], dtype=_counter_dtype)
+ delta = cumulative - self._cumulative[name]
+ self._cumulative[name].copy_(cumulative)
+ if float(delta[0]) != 0:
+ self._moments[name] = delta
+
+ def _get_delta(self, name):
+ r"""Returns the raw moments that were accumulated for the given
+ statistic between the last two calls to `update()`, or zero if
+ no scalars were collected.
+ """
+ assert self._regex.fullmatch(name)
+ if name not in self._moments:
+ self._moments[name] = torch.zeros([_num_moments], dtype=_counter_dtype)
+ return self._moments[name]
+
+ def num(self, name):
+ r"""Returns the number of scalars that were accumulated for the given
+ statistic between the last two calls to `update()`, or zero if
+ no scalars were collected.
+ """
+ delta = self._get_delta(name)
+ return int(delta[0])
+
+ def mean(self, name):
+ r"""Returns the mean of the scalars that were accumulated for the
+ given statistic between the last two calls to `update()`, or NaN if
+ no scalars were collected.
+ """
+ delta = self._get_delta(name)
+ if int(delta[0]) == 0:
+ return float('nan')
+ return float(delta[1] / delta[0])
+
+ def std(self, name):
+ r"""Returns the standard deviation of the scalars that were
+ accumulated for the given statistic between the last two calls to
+ `update()`, or NaN if no scalars were collected.
+ """
+ delta = self._get_delta(name)
+ if int(delta[0]) == 0 or not np.isfinite(float(delta[1])):
+ return float('nan')
+ if int(delta[0]) == 1:
+ return float(0)
+ mean = float(delta[1] / delta[0])
+ raw_var = float(delta[2] / delta[0])
+ return np.sqrt(max(raw_var - np.square(mean), 0))
+
+ def as_dict(self):
+ r"""Returns the averages accumulated between the last two calls to
+ `update()` as an `dnnlib.EasyDict`. The contents are as follows:
+
+ dnnlib.EasyDict(
+ NAME = dnnlib.EasyDict(num=FLOAT, mean=FLOAT, std=FLOAT),
+ ...
+ )
+ """
+ stats = dnnlib.EasyDict()
+ for name in self.names():
+ stats[name] = dnnlib.EasyDict(num=self.num(name), mean=self.mean(name), std=self.std(name))
+ return stats
+
+ def __getitem__(self, name):
+ r"""Convenience getter.
+ `collector[name]` is a synonym for `collector.mean(name)`.
+ """
+ return self.mean(name)
+
+#----------------------------------------------------------------------------
+
+def _sync(names):
+ r"""Synchronize the global cumulative counters across devices and
+ processes. Called internally by `Collector.update()`.
+ """
+ if len(names) == 0:
+ return []
+ global _sync_called
+ _sync_called = True
+
+ # Collect deltas within current rank.
+ deltas = []
+ device = _sync_device if _sync_device is not None else torch.device('cpu')
+ for name in names:
+ delta = torch.zeros([_num_moments], dtype=_counter_dtype, device=device)
+ for counter in _counters[name].values():
+ delta.add_(counter.to(device))
+ counter.copy_(torch.zeros_like(counter))
+ deltas.append(delta)
+ deltas = torch.stack(deltas)
+
+ # Sum deltas across ranks.
+ if _sync_device is not None:
+ torch.distributed.all_reduce(deltas)
+
+ # Update cumulative values.
+ deltas = deltas.cpu()
+ for idx, name in enumerate(names):
+ if name not in _cumulative:
+ _cumulative[name] = torch.zeros([_num_moments], dtype=_counter_dtype)
+ _cumulative[name].add_(deltas[idx])
+
+ # Return name-value pairs.
+ return [(name, _cumulative[name]) for name in names]
+
+#----------------------------------------------------------------------------
diff --git a/tools/extract_meta_info.py b/tools/extract_meta_info.py
new file mode 100644
index 0000000..8fc6423
--- /dev/null
+++ b/tools/extract_meta_info.py
@@ -0,0 +1,32 @@
+import argparse
+import json
+import os
+
+# -----
+# python tools/extract_meta_info.py --root_path /path/to/video_dir --dataset_name fashion
+# -----
+parser = argparse.ArgumentParser()
+parser.add_argument("--root_path", type=str, default='./Datasets/videos/VFHQ')
+parser.add_argument("--dataset_name", type=str, default='VFHQ')
+parser.add_argument("--meta_info_name", type=str)
+
+args = parser.parse_args()
+
+if args.meta_info_name is None:
+ args.meta_info_name = args.dataset_name
+
+# collect all video_folder paths
+video_mp4_paths = set()
+for root, dirs, files in os.walk(args.root_path):
+ for name in files:
+ if name.endswith(".pt"):
+ video_mp4_paths.add(os.path.join(root, name).replace('boxes','videos').replace('.pt','.mp4'))
+ elif name.endswith(".mp4"):
+ video_mp4_paths.add(os.path.join(root, name))
+video_mp4_paths = list(video_mp4_paths)
+
+meta_infos = []
+for video_mp4_path in video_mp4_paths:
+ meta_infos.append({"video_path": video_mp4_path})
+
+json.dump(meta_infos, open(f"./data/{args.meta_info_name}_meta.json", "w"))
diff --git a/tools/get_boxes.py b/tools/get_boxes.py
new file mode 100644
index 0000000..b0119f1
--- /dev/null
+++ b/tools/get_boxes.py
@@ -0,0 +1,80 @@
+import mediapipe as mp
+import cv2
+import os
+import torch
+import numpy as np
+from concurrent.futures import ProcessPoolExecutor
+from src.utils.util import read_frames
+import logging
+logging.getLogger('mediapipe').setLevel(logging.ERROR)
+import argparse
+from functools import partial
+
+mp_face_mesh = mp.solutions.face_mesh
+
+face_indices = list(range(468))
+left_eye_indices = [226, 230, 223, 245]
+right_eye_indices = [446, 450, 465, 443]
+mouth_indices = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291] + \
+ [78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308] + \
+ [146, 91, 181, 84, 17, 314, 405, 321, 375] + \
+ [191, 80, 81, 82, 13, 312, 311, 310, 415]
+
+def get_region_box(landmarks, indices):
+ xs = [int(landmarks[i][0]) for i in indices]
+ ys = [int(landmarks[i][1]) for i in indices]
+ x_min, x_max = min(xs), max(xs)
+ y_min, y_max = min(ys), max(ys)
+ return [x_min, y_min, x_max, y_max]
+
+def process_video(name, video_dir, save_dir):
+ video_path = os.path.join(video_dir, name)
+ save_path = os.path.join(save_dir, name.replace('.mp4', '.pt'))
+ if os.path.exists(save_path):
+ return name
+
+ video = read_frames(video_path)
+ boxes = []
+
+ with mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1) as face_mesh:
+ for image_pil in video:
+ image = np.array(image_pil)
+ h, w, _ = image.shape
+ results = face_mesh.process(image)
+
+ if results.multi_face_landmarks is not None:
+ face_landmarks = results.multi_face_landmarks[0]
+ landmarks = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]
+ face_box = get_region_box(landmarks, face_indices)
+ left_eye_box = get_region_box(landmarks, left_eye_indices)
+ right_eye_box = get_region_box(landmarks, right_eye_indices)
+ mouth_box = get_region_box(landmarks, mouth_indices)
+ boxes.append({
+ 'face': face_box,
+ 'left_eye': left_eye_box,
+ 'right_eye': right_eye_box,
+ 'mouth': mouth_box
+ })
+ else:
+ boxes.append(boxes[-1] if boxes else {
+ 'face': [],
+ 'left_eye': [],
+ 'right_eye': [],
+ 'mouth': []
+ })
+ torch.save(boxes, save_path)
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description="Extract facial boxes from videos and save as .pt files.")
+ parser.add_argument('--video_dir', type=str, default='/home/zyli/Repositories/x-nemo-inference/lv100/videos')
+ parser.add_argument('--save_dir', type=str, default='/home/zyli/Repositories/x-nemo-inference/lv100/boxes_zyli')
+ parser.add_argument('--workers', type=int, default=8)
+
+ args = parser.parse_args()
+ os.makedirs(args.save_dir, exist_ok=True)
+ video_files = [f for f in os.listdir(args.video_dir) if f.endswith('.mp4')]
+
+ process_func = partial(process_video, video_dir=args.video_dir, save_dir=args.save_dir)
+
+ with ProcessPoolExecutor(max_workers=8) as executor:
+ list(executor.map(process_func, video_files))
\ No newline at end of file
diff --git a/train_stage1.py b/train_stage1.py
new file mode 100644
index 0000000..29ea64d
--- /dev/null
+++ b/train_stage1.py
@@ -0,0 +1,833 @@
+import argparse
+import logging
+import math
+import os
+import os.path as osp
+import random
+import warnings
+from collections import OrderedDict
+from datetime import datetime
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from torchvision import transforms
+
+import cv2
+import diffusers
+from einops import rearrange
+import mlflow
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.utils.checkpoint
+import transformers
+from accelerate import Accelerator
+from accelerate.logging import get_logger
+from accelerate.utils import DistributedDataParallelKwargs
+from diffusers import AutoencoderKL, AutoencoderKLTemporalDecoder
+from src.scheduler.scheduler_ddim import DDIMScheduler
+from diffusers.optimization import get_scheduler
+from diffusers.utils import check_min_version
+from diffusers.utils.import_utils import is_xformers_available
+from omegaconf import OmegaConf
+from PIL import Image
+from tqdm.auto import tqdm
+from transformers import CLIPVisionModelWithProjection
+from decord import VideoReader
+from src.dataset.portrait_image import PortraitImageDataset
+from src.models.mutual_self_attention import ReferenceAttentionControl
+from src.models.unet_2d_condition import UNet2DConditionModel
+from src.models.unet_3d import UNet3DConditionModel
+from src.pipelines.pipeline_pose2img import Pose2ImagePipeline
+from src.utils.util import (
+ delete_additional_ckpt,
+ import_filename,
+ read_frames,
+ save_videos_grid,
+ seed_everything,
+ draw_keypoints,
+)
+from src.dataset.utils import scale_bb
+from src.models.motion_encoder.encoder import MotEncoder
+from src.models.motion_module import zero_module
+warnings.filterwarnings("ignore")
+import mediapipe as mp
+from src.liveportrait.motion_extractor import MotionExtractor
+from src.models.pose_guider import PoseGuider
+import gc
+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
+check_min_version("0.10.0.dev0")
+from torchvision.utils import save_image
+
+logger = get_logger(__name__, log_level="INFO")
+
+class Net(nn.Module):
+ def __init__(
+ self,
+ reference_unet: UNet2DConditionModel,
+ denoising_unet: UNet3DConditionModel,
+ motion_enc: MotEncoder,
+ pose_guider: PoseGuider,
+ reference_control_writer,
+ reference_control_reader,
+ ):
+ super().__init__()
+ self.reference_unet = reference_unet
+ self.denoising_unet = denoising_unet
+ self.reference_control_writer = reference_control_writer
+ self.reference_control_reader = reference_control_reader
+
+ self.motion_encoder = motion_enc
+ self.pose_guider = pose_guider
+
+ def forward(
+ self,
+ noisy_latents,
+ timesteps,
+ ref_image_latents,
+ clip_image_embeds,
+ tgt_face,
+ tgt_pose,
+ uncond_fwd: bool = False,
+ ):
+ pose_emb = self.pose_guider(tgt_pose)
+ motion_emb = self.motion_encoder(tgt_face)
+
+ if not uncond_fwd:
+ self.reference_unet(
+ ref_image_latents,
+ torch.zeros_like(timesteps[:noisy_latents.shape[0]]),
+ encoder_hidden_states=clip_image_embeds,
+ return_dict=False,
+ )
+ self.reference_control_reader.update(self.reference_control_writer, drop_ratio=0.3)
+
+ if random.random() < 0.1:
+ pose_emb = torch.zeros_like(pose_emb)
+
+ if random.random() < 0.1:
+ motion_emb = torch.zeros_like(motion_emb)
+
+ clip_image_embeds = [clip_image_embeds, motion_emb]
+
+ model_pred = self.denoising_unet(
+ noisy_latents,
+ timesteps,
+ encoder_hidden_states=clip_image_embeds,
+ pose_cond_fea=pose_emb,
+ ).sample
+
+ return model_pred
+
+def get_x0_from_eps(ddim, sample: torch.FloatTensor, model_output: torch.FloatTensor, timesteps: torch.IntTensor):
+ video_length = sample.shape[2]
+ sample = rearrange(sample, 'b c f h w -> (b f) c h w')
+ model_output = rearrange(model_output, 'b c f h w -> (b f) c h w')
+ alpha_prod = ddim.alphas_cumprod.cuda()
+ alpha_prod_t = alpha_prod[timesteps]
+ while len(alpha_prod_t.shape) < len(sample.shape):
+ alpha_prod_t = alpha_prod_t.unsqueeze(-1)
+
+ pred_original_sample = torch.sqrt(1./alpha_prod_t) * sample - torch.sqrt(1. / alpha_prod_t - 1) * model_output
+
+ pred_original_sample = rearrange(pred_original_sample, '(b f) c h w -> b c f h w', f=video_length)
+
+ return pred_original_sample
+
+def crop_face(image_pil, face_mesh):
+ image = np.array(image_pil)
+ h, w = image.shape[:2]
+ results = face_mesh.process(image)
+ face_landmarks = results.multi_face_landmarks[0]
+ coords = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]
+ xs, ys = zip(*coords)
+ x1, y1 = min(xs), min(ys)
+ x2, y2 = max(xs), max(ys)
+ face_box = (x1, y1, x2, y2)
+
+ left, top, right, bot = scale_bb(face_box, scale=1.1, size=image.shape[:2])
+
+ face_patch = image[int(top) : int(bot), int(left) : int(right)]
+
+ return face_box, face_patch
+
+def log_validation(
+ vae,
+ image_enc,
+ net,
+ scheduler,
+ accelerator,
+ width,
+ height,
+ face_mesh,
+ pose_encoder,
+ clip_length=100,
+ generator=None,
+):
+ logger.info("Running validation... ")
+
+ ori_net = accelerator.unwrap_model(net)
+ reference_unet = ori_net.reference_unet
+ denoising_unet = ori_net.denoising_unet
+ motion_enc = ori_net.motion_encoder
+ pose_gui = ori_net.pose_guider
+ pose_enc = pose_encoder
+
+ motion_enc.eval()
+
+ # generator = torch.manual_seed(42)
+ generator = torch.Generator(device='cuda').manual_seed(42)
+ # cast unet dtype
+
+ pipe = Pose2ImagePipeline(
+ vae=vae,
+ image_encoder=image_enc,
+ reference_unet=reference_unet,
+ denoising_unet=denoising_unet,
+ motion_encoder=motion_enc,
+ pose_encoder=pose_enc,
+ pose_guider=pose_gui,
+ scheduler=scheduler,
+ )
+ pipe = pipe.to(accelerator.device)
+
+ ref_image_paths = [
+ "path/to/ref_img_1.png",
+ "path/to/ref_img_2.png",
+ ]
+ videos_path = [
+ "path/to/ref_vid_1.png",
+ "path/to/ref_vid_2.png",
+ ]
+
+ pose_transform = transforms.Compose(
+ [transforms.Resize((height, width)), transforms.ToTensor()]
+ )
+ results = []
+ for idy, pose_video_path in enumerate(videos_path):
+ pose_images = read_frames(pose_video_path)
+
+ boxes_path = pose_video_path.replace("videos", "boxes").replace(".mp4", ".pt")
+ boxes = torch.load(boxes_path)
+
+ pose_face = []
+ ori_pose_images = []
+ for idx_control, pose_image_pil in enumerate(pose_images[:clip_length]):
+ face_bbox = boxes[idx_control]["face"]
+ ori_pose_images.append(pose_image_pil)
+ pose_image = np.array(pose_image_pil)
+ left, top, right, bot = scale_bb(face_bbox, scale=1.1, size=pose_image.shape[:2])
+ pose_image = pose_image[int(top) : int(bot), int(left) : int(right)]
+ pose_image_pil = Image.fromarray(pose_image).convert("RGB")
+ pose_face.append(pose_image_pil)
+
+ pose_tensor_list = []
+ ori_pose_tensor_list = []
+
+ for idx_control, pose_image_pil in enumerate(ori_pose_images):
+ pose_tensor_list.append(pose_transform(pose_face[idx_control]))
+ ori_pose_tensor_list.append(pose_transform(pose_image_pil))
+ pose_tensor = torch.stack(pose_tensor_list, dim=0) # (f, c, h, w)
+ pose_tensor = pose_tensor.transpose(0, 1).unsqueeze(0)
+
+ ori_pose_tensor = torch.stack(ori_pose_tensor_list, dim=0) # (f, c, h, w)
+ ori_pose_tensor = ori_pose_tensor.transpose(0, 1).unsqueeze(0)
+
+ for idx, ref_image_path in enumerate(ref_image_paths):
+ ref_image_pil = Image.open(ref_image_path).convert("RGB")
+ ref_bbox, ref_patch = crop_face(ref_image_pil, face_mesh)
+ ref_face = Image.fromarray(ref_patch)
+
+ ref_tensor = pose_transform(ref_image_pil).unsqueeze(0).unsqueeze(2).expand_as(pose_tensor)
+
+ ref_name = f'{idx}'
+ pose_name = f'{idy}'
+
+ with torch.cuda.amp.autocast():
+ gen_video = pipe(
+ ori_pose_images,
+ ref_image_pil,
+ pose_face,
+ ref_face,
+ width,
+ height,
+ clip_length,
+ 20,
+ 2.5,
+ generator=generator,
+ ).videos
+
+ # Concat it with pose tensor
+ video = torch.cat([ref_tensor, pose_tensor, ori_pose_tensor, gen_video], dim=0)
+
+ results.append({"name": f"{ref_name}_{pose_name}", "vid": video})
+
+ del pipe
+ torch.cuda.empty_cache()
+ gc.collect()
+ motion_enc.train()
+ return results
+
+
+def main(cfg):
+ kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
+ accelerator = Accelerator(
+ gradient_accumulation_steps=cfg.solver.gradient_accumulation_steps,
+ mixed_precision=cfg.solver.mixed_precision,
+ log_with="mlflow",
+ project_dir="./mlruns",
+ kwargs_handlers=[kwargs],
+ )
+
+ # Make one log on every process with the configuration for debugging.
+ logging.basicConfig(
+ format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+ datefmt="%m/%d/%Y %H:%M:%S",
+ level=logging.INFO,
+ )
+ logger.info(accelerator.state, main_process_only=False)
+ if accelerator.is_local_main_process:
+ transformers.utils.logging.set_verbosity_warning()
+ diffusers.utils.logging.set_verbosity_info()
+ else:
+ transformers.utils.logging.set_verbosity_error()
+ diffusers.utils.logging.set_verbosity_error()
+
+ # If passed along, set the training seed now.
+ if cfg.seed is not None:
+ seed_everything(cfg.seed)
+
+ exp_name = cfg.exp_name
+ save_dir = f"{cfg.output_dir}/{exp_name}"
+ if accelerator.is_main_process and not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+
+ inference_config_path = "./configs/inference/inference_stage1&2.yaml"
+ infer_config = OmegaConf.load(inference_config_path)
+
+ if cfg.weight_dtype == "fp16":
+ weight_dtype = torch.float16
+ elif cfg.weight_dtype == "fp32":
+ weight_dtype = torch.float32
+ else:
+ raise ValueError(
+ f"Do not support weight dtype: {cfg.weight_dtype} during training"
+ )
+
+ sched_kwargs = OmegaConf.to_container(
+ infer_config.noise_scheduler_kwargs
+ )
+
+ val_noise_scheduler = DDIMScheduler(**sched_kwargs)
+ train_noise_scheduler = DDIMScheduler(**sched_kwargs)
+ vae = AutoencoderKL.from_pretrained(cfg.vae_model_path).to(
+ "cuda", dtype=weight_dtype
+ )
+
+ image_enc = CLIPVisionModelWithProjection.from_pretrained(
+ cfg.image_encoder_path,
+ ).to(dtype=weight_dtype, device="cuda")
+
+ reference_unet = UNet2DConditionModel.from_pretrained(
+ cfg.base_model_path,
+ subfolder="unet",
+ ).to(device="cuda")
+ denoising_unet = UNet3DConditionModel.from_pretrained_2d(
+ cfg.base_model_path,
+ "",
+ subfolder="unet",
+ unet_additional_kwargs=OmegaConf.to_container(
+ infer_config.unet_additional_kwargs
+ ),
+ ).to(device="cuda")
+
+ motion_encoder = MotEncoder().to(device="cuda")
+ motion_encoder_params = torch.load(cfg.motion_encoder_path, map_location="cpu")
+ old_weight = motion_encoder_params['final_proj.weight']
+ new_weight = motion_encoder.final_proj.weight.data
+ new_weight = old_weight[:, :new_weight.shape[1]]
+ motion_encoder_params['final_proj.weight'] = new_weight
+
+ motion_encoder.load_state_dict(
+ motion_encoder_params, strict=False
+ )
+
+ pose_guider = PoseGuider().to(device="cuda")
+ pose_guider.load_state_dict(torch.load(cfg.pose_guider_path, map_location='cpu'), strict=False)
+
+ pose_encoder = MotionExtractor(num_kp=21).to(device="cuda").eval()
+ pose_encoder.load_state_dict(torch.load(cfg.pose_encoder_path, map_location='cpu'), strict=False)
+
+ denoising_unet.load_state_dict(
+ torch.load(cfg.denoising_unet_path, map_location="cpu"),
+ strict=False,
+ )
+ reference_unet.load_state_dict(
+ torch.load(cfg.reference_unet_path, map_location="cpu"),
+ )
+
+ mp_face_mesh = mp.solutions.face_mesh
+ face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1)
+
+ # Freeze
+ vae.requires_grad_(False)
+ image_enc.requires_grad_(False)
+ pose_encoder.requires_grad_(False)
+
+ pose_guider.requires_grad_(True)
+ motion_encoder.requires_grad_(True)
+ denoising_unet.requires_grad_(True)
+ reference_unet.requires_grad_(False)
+
+ for name, param in reference_unet.named_parameters():
+ if "up_blocks.3" in name:
+ param.requires_grad_(False)
+ else:
+ param.requires_grad_(True)
+
+ reference_control_writer = ReferenceAttentionControl(
+ reference_unet,
+ do_classifier_free_guidance=False,
+ mode="write",
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ denoising_unet,
+ do_classifier_free_guidance=False,
+ mode="read",
+ fusion_blocks="full",
+ )
+
+ net = Net(
+ reference_unet,
+ denoising_unet,
+ motion_encoder,
+ pose_guider,
+ reference_control_writer,
+ reference_control_reader,
+ )
+
+ if cfg.solver.enable_xformers_memory_efficient_attention:
+ if is_xformers_available():
+ reference_unet.enable_xformers_memory_efficient_attention()
+ denoising_unet.enable_xformers_memory_efficient_attention()
+ else:
+ raise ValueError(
+ "xformers is not available. Make sure it is installed correctly"
+ )
+
+ if cfg.solver.gradient_checkpointing:
+ reference_unet.enable_gradient_checkpointing()
+ denoising_unet.enable_gradient_checkpointing()
+
+ if cfg.solver.scale_lr:
+ learning_rate = (
+ cfg.solver.learning_rate
+ * cfg.solver.gradient_accumulation_steps
+ * cfg.data.train_bs
+ * accelerator.num_processes
+ )
+ else:
+ learning_rate = cfg.solver.learning_rate
+
+ # Initialize the optimizer
+ if cfg.solver.use_8bit_adam:
+ try:
+ import bitsandbytes as bnb
+ except ImportError:
+ raise ImportError(
+ "Please install bitsandbytes to use 8-bit Adam. You can do so by running `pip install bitsandbytes`"
+ )
+
+ optimizer_cls = bnb.optim.AdamW8bit
+ else:
+ optimizer_cls = torch.optim.AdamW
+
+ trainable_params = list(filter(lambda p: p.requires_grad, net.parameters()))
+
+ optimizer = optimizer_cls(
+ trainable_params,
+ lr=learning_rate,
+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),
+ weight_decay=cfg.solver.adam_weight_decay,
+ eps=cfg.solver.adam_epsilon,
+ )
+
+ # Scheduler
+ lr_scheduler = get_scheduler(
+ cfg.solver.lr_scheduler,
+ optimizer=optimizer,
+ num_warmup_steps=cfg.solver.lr_warmup_steps
+ * cfg.solver.gradient_accumulation_steps,
+ num_training_steps=cfg.solver.max_train_steps
+ * cfg.solver.gradient_accumulation_steps,
+ )
+
+ train_dataset = PortraitImageDataset(
+ img_size=(cfg.data.train_width, cfg.data.train_height),
+ img_scale=(0.9, 1.0),
+ data_meta_paths=cfg.data.meta_paths,
+ sample_margin=cfg.data.sample_margin,
+ )
+ train_dataloader = torch.utils.data.DataLoader(
+ train_dataset, batch_size=cfg.data.train_bs, shuffle=True, num_workers=8, drop_last=True
+ )
+
+ # Prepare everything with our `accelerator`.
+ (
+ net,
+ optimizer,
+ train_dataloader,
+ lr_scheduler,
+ ) = accelerator.prepare(
+ net,
+ optimizer,
+ train_dataloader,
+ lr_scheduler,
+ )
+
+ # We need to recalculate our total training steps as the size of the training dataloader may have changed.
+ num_update_steps_per_epoch = math.ceil(
+ len(train_dataloader) / cfg.solver.gradient_accumulation_steps
+ )
+ # Afterwards we recalculate our number of training epochs
+ num_train_epochs = math.ceil(
+ cfg.solver.max_train_steps / num_update_steps_per_epoch
+ )
+
+ # We need to initialize the trackers we use, and also store our configuration.
+ # The trackers initializes automatically on the main process.
+ if accelerator.is_main_process:
+ run_time = datetime.now().strftime("%Y%m%d-%H%M")
+ accelerator.init_trackers(
+ cfg.exp_name,
+ init_kwargs={"mlflow": {"run_name": run_time}},
+ )
+ # dump config file
+ mlflow.log_dict(OmegaConf.to_container(cfg), "config.yaml")
+
+ # Train!
+ total_batch_size = (
+ cfg.data.train_bs
+ * accelerator.num_processes
+ * cfg.solver.gradient_accumulation_steps
+ )
+
+ logger.info("***** Running training *****")
+ logger.info(f" Num examples = {len(train_dataset)}")
+ logger.info(f" Num Epochs = {num_train_epochs}")
+ logger.info(f" Instantaneous batch size per device = {cfg.data.train_bs}")
+ logger.info(
+ f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"
+ )
+ logger.info(
+ f" Gradient Accumulation steps = {cfg.solver.gradient_accumulation_steps}"
+ )
+ logger.info(f" Total optimization steps = {cfg.solver.max_train_steps}")
+ global_step = 0
+ first_epoch = 0
+
+ # Potentially load in the weights and states from a previous save
+ if cfg.resume_from_checkpoint:
+ if cfg.resume_from_checkpoint != "latest":
+ resume_dir = cfg.resume_from_checkpoint
+ else:
+ resume_dir = save_dir
+ # Get the most recent checkpoint
+ dirs = os.listdir(resume_dir)
+ dirs = [d for d in dirs if d.startswith("checkpoint")]
+ dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))
+ path = dirs[-1]
+ accelerator.load_state(os.path.join(resume_dir, path))
+ accelerator.print(f"Resuming from checkpoint {path}")
+ global_step = int(path.split("-")[1])
+
+ first_epoch = global_step // num_update_steps_per_epoch
+
+ # Only show the progress bar once on each machine.
+ progress_bar = tqdm(
+ range(global_step, cfg.solver.max_train_steps),
+ disable=not accelerator.is_local_main_process,
+ )
+ progress_bar.set_description("Steps")
+
+ for epoch in range(first_epoch, num_train_epochs):
+ train_loss = 0.0
+ for step, batch in enumerate(train_dataloader):
+ with accelerator.accumulate(net):
+ # Convert videos to latent space
+ pixel_values = batch["img"].to(weight_dtype) # b,c,h,w
+ tgt_face = batch["tgt_face"]
+ tgt_pose = batch["tgt_pose"]
+
+ face_mask = batch['face_mask']
+ local_mask = batch['local_mask']
+
+ with torch.no_grad():
+ latents = vae.encode(pixel_values).latent_dist.sample()
+ latents = latents.unsqueeze(2)
+ latents = latents * 0.18215 # (b f) c h w
+
+ keypoints = pose_encoder(tgt_pose)
+ tgt_pose = draw_keypoints(keypoints).unsqueeze(2).to('cuda')
+
+ if epoch == 0 and step == 0:
+ if accelerator.is_main_process:
+ save_image(tgt_pose[:1,:,0], "image_kps.png")
+ save_image((pixel_values[:1] + 1) / 2, "image_tgt.png")
+
+ noise = torch.randn_like(latents)
+ if cfg.noise_offset > 0.0:
+ noise += cfg.noise_offset * torch.randn(
+ (noise.shape[0], noise.shape[1], 1, 1, 1),
+ device=noise.device,
+ )
+
+ bsz = latents.shape[0] # (b f)
+ # Sample a random timestep for each frame
+ timesteps = torch.randint(
+ 0,
+ train_noise_scheduler.num_train_timesteps,
+ (bsz,),
+ device=latents.device,
+ )
+ timesteps = timesteps.long()
+
+ facial_mask = F.interpolate(face_mask, size=(64,64), mode='nearest').unsqueeze(2).repeat(1,4,1,1,1) + F.interpolate(local_mask, size=(64,64), mode='nearest').unsqueeze(2).repeat(1,4,1,1,1)
+
+ uncond_fwd = random.random() < cfg.uncond_ratio
+ clip_image_list = []
+ ref_image_list = []
+ for batch_idx, (ref_img, clip_img) in enumerate(
+ zip(
+ batch["ref_img"],
+ batch["clip_image"],
+ )
+ ):
+ clip_image_list.append(clip_img)
+ ref_image_list.append(ref_img)
+
+ with torch.no_grad():
+ ref_img = torch.stack(ref_image_list, dim=0).to(
+ dtype=vae.dtype, device=vae.device
+ )
+ ref_image_latents = vae.encode(
+ ref_img
+ ).latent_dist.sample() # (bs, d, 64, 64)
+ ref_image_latents = ref_image_latents * 0.18215
+
+ clip_img = torch.stack(clip_image_list, dim=0).to(
+ dtype=image_enc.dtype, device=image_enc.device
+ )
+ clip_image_embeds = image_enc(
+ clip_img.to("cuda", dtype=weight_dtype)
+ ).image_embeds
+ image_prompt_embeds = clip_image_embeds.unsqueeze(1) # (bs, 1, d)
+
+ # add noise
+ noisy_latents = train_noise_scheduler.add_noise(
+ latents, noise, timesteps
+ ) # (b f) c h w
+
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+
+ model_pred = net(
+ noisy_latents,
+ timesteps,
+ ref_image_latents,
+ image_prompt_embeds,
+ tgt_face,
+ tgt_pose,
+ uncond_fwd,
+ )
+
+ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)
+
+ loss = F.mse_loss(latents_pred.float(), latents.float(), reduction="mean")
+ loss_local = F.mse_loss(latents_pred.float() * facial_mask, latents.float() * facial_mask, reduction="mean")
+ loss = loss + loss_local
+
+ # Gather the losses across all processes for logging (if we use distributed training).
+ avg_loss = accelerator.gather(loss.repeat(cfg.data.train_bs)).mean()
+ train_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps
+
+ # Backpropagate
+ accelerator.backward(loss)
+ if accelerator.sync_gradients:
+ accelerator.clip_grad_norm_(
+ trainable_params,
+ cfg.solver.max_grad_norm,
+ )
+ optimizer.step()
+ lr_scheduler.step()
+ optimizer.zero_grad()
+
+ if accelerator.sync_gradients:
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+ progress_bar.update(1)
+ global_step += 1
+ accelerator.log({"train_loss": train_loss}, step=global_step)
+ train_loss = 0.0
+ if global_step % cfg.checkpointing_steps == 0:
+ if accelerator.is_main_process:
+ save_path = os.path.join(save_dir, f"checkpoint-{global_step}")
+ delete_additional_ckpt(save_dir, 1)
+ accelerator.save_state(save_path)
+
+ if global_step % cfg.val.validation_steps == 0 or global_step == 1:
+ if accelerator.is_main_process:
+ generator = torch.Generator(device=accelerator.device)
+ generator.manual_seed(cfg.seed)
+
+ sample_dicts = log_validation(
+ vae=vae,
+ image_enc=image_enc,
+ net=net,
+ scheduler=val_noise_scheduler,
+ accelerator=accelerator,
+ width=cfg.data.train_width,
+ height=cfg.data.train_height,
+ face_mesh=face_mesh,
+ pose_encoder=pose_encoder,
+ generator=generator,
+ )
+
+ reference_control_writer = ReferenceAttentionControl(
+ reference_unet,
+ do_classifier_free_guidance=False,
+ mode="write",
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ denoising_unet,
+ do_classifier_free_guidance=False,
+ mode="read",
+ fusion_blocks="full",
+ )
+
+ for sample_id, sample_dict in enumerate(sample_dicts):
+ sample_name = sample_dict["name"]
+ vid = sample_dict["vid"]
+ with TemporaryDirectory() as temp_dir:
+ out_file = f"{temp_dir}/{global_step:06d}-{sample_name}.mp4"
+ save_videos_grid(vid, out_file, n_rows=4, fps=25)
+ mlflow.log_artifact(out_file)
+
+ logs = {
+ "loss": loss.detach().item(),
+ "lr": lr_scheduler.get_last_lr()[0],
+ }
+ progress_bar.set_postfix(**logs)
+
+ if global_step >= cfg.solver.max_train_steps:
+ break
+
+ # save model after each epoch
+ if (
+ epoch + 1
+ ) % cfg.save_model_epoch_interval == 0 and accelerator.is_main_process:
+ unwrap_net = accelerator.unwrap_model(net)
+ save_checkpoint(
+ unwrap_net.denoising_unet,
+ save_dir,
+ "denoising_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.reference_unet,
+ save_dir,
+ "reference_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.motion_encoder,
+ save_dir,
+ "motion_encoder",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.pose_guider,
+ save_dir,
+ "pose_guider",
+ global_step,
+ total_limit=2,
+ )
+
+ unwrap_net = accelerator.unwrap_model(net)
+ save_checkpoint(
+ unwrap_net.denoising_unet,
+ save_dir,
+ "denoising_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.reference_unet,
+ save_dir,
+ "reference_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.motion_encoder,
+ save_dir,
+ "motion_encoder",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.pose_guider,
+ save_dir,
+ "pose_guider",
+ global_step,
+ total_limit=2,
+ )
+
+ # Create the pipeline using the trained modules and save it.
+ accelerator.wait_for_everyone()
+ accelerator.end_training()
+
+
+def save_checkpoint(model, save_dir, prefix, ckpt_num, total_limit=None):
+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")
+
+ if total_limit is not None:
+ checkpoints = os.listdir(save_dir)
+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]
+ checkpoints = sorted(
+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])
+ )
+
+ if len(checkpoints) >= total_limit:
+ num_to_remove = len(checkpoints) - total_limit + 1
+ removing_checkpoints = checkpoints[0:num_to_remove]
+ logger.info(
+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
+ )
+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")
+
+ for removing_checkpoint in removing_checkpoints:
+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)
+ os.remove(removing_checkpoint)
+
+ state_dict = model.state_dict()
+ torch.save(state_dict, save_path)
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--config", type=str, default="./configs/train/personalive_stage1.yaml")
+ args = parser.parse_args()
+
+ if args.config[-5:] == ".yaml":
+ config = OmegaConf.load(args.config)
+ elif args.config[-3:] == ".py":
+ config = import_filename(args.config).cfg
+ else:
+ raise ValueError("Do not support this format config file")
+ main(config)
diff --git a/train_stage2.py b/train_stage2.py
new file mode 100644
index 0000000..b498885
--- /dev/null
+++ b/train_stage2.py
@@ -0,0 +1,969 @@
+import sys
+sys.path.append('./src/stylegan2')
+
+import argparse
+import logging
+import math
+import os
+import os.path as osp
+import random
+import warnings
+from collections import OrderedDict
+from datetime import datetime
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from torchvision import transforms
+
+import cv2
+import lpips
+import diffusers
+from einops import rearrange
+import mlflow
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.utils.checkpoint
+import transformers
+from accelerate import Accelerator
+from accelerate.logging import get_logger
+from accelerate.utils import DistributedDataParallelKwargs
+from diffusers import AutoencoderKL, AutoencoderKLTemporalDecoder
+from src.scheduler.scheduler_ddim import DDIMScheduler
+from diffusers.optimization import get_scheduler
+from diffusers.utils import check_min_version
+from diffusers.utils.import_utils import is_xformers_available
+from omegaconf import OmegaConf
+from PIL import Image
+from tqdm.auto import tqdm
+from transformers import CLIPVisionModelWithProjection
+from decord import VideoReader
+from src.dataset.portrait_image import PortraitImageDataset
+from src.models.mutual_self_attention import ReferenceAttentionControl
+from src.models.unet_2d_condition import UNet2DConditionModel
+from src.models.unet_3d import UNet3DConditionModel
+from src.pipelines.pipeline_pose2img import Pose2ImagePipeline
+from src.utils.util import (
+ delete_additional_ckpt,
+ import_filename,
+ read_frames,
+ save_videos_grid,
+ seed_everything,
+ draw_keypoints,
+)
+from src.dataset.utils import scale_bb
+from src.models.motion_encoder.encoder import MotEncoder
+from src.models.motion_module import zero_module
+warnings.filterwarnings("ignore")
+import mediapipe as mp
+from src.liveportrait.motion_extractor import MotionExtractor
+from src.models.pose_guider import PoseGuider
+import gc
+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
+check_min_version("0.10.0.dev0")
+from torchvision.utils import save_image
+from src.stylegan2.discriminator import Discriminator as StyleGAN2Discriminator
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+logger = get_logger(__name__, log_level="INFO")
+
+class Net(nn.Module):
+ def __init__(
+ self,
+ reference_unet: UNet2DConditionModel,
+ denoising_unet: UNet3DConditionModel,
+ motion_enc: MotEncoder,
+ pose_guider: PoseGuider,
+ reference_control_writer,
+ reference_control_reader,
+ ):
+ super().__init__()
+ self.reference_unet = reference_unet
+ self.denoising_unet = denoising_unet
+ self.reference_control_writer = reference_control_writer
+ self.reference_control_reader = reference_control_reader
+
+ self.motion_encoder = motion_enc
+ self.pose_guider = pose_guider
+
+ def forward(
+ self,
+ noisy_latents,
+ timesteps,
+ ref_image_latents,
+ clip_image_embeds,
+ tgt_face,
+ tgt_pose,
+ ref_flag,
+ mot_flag,
+ pose_flag,
+ uncond_fwd: bool = False,
+ ):
+ pose_emb = self.pose_guider(tgt_pose)
+ motion_emb = self.motion_encoder(tgt_face)
+
+ if not uncond_fwd and ref_flag:
+ self.reference_unet(
+ ref_image_latents,
+ torch.zeros_like(timesteps[:noisy_latents.shape[0]]),
+ encoder_hidden_states=clip_image_embeds,
+ return_dict=False,
+ )
+ self.reference_control_reader.update(self.reference_control_writer, drop_ratio=0.3)
+
+ if pose_flag:
+ pose_emb = torch.zeros_like(pose_emb)
+
+ if mot_flag:
+ motion_emb = torch.zeros_like(motion_emb)
+
+ clip_image_embeds = [clip_image_embeds, motion_emb]
+
+ model_pred = self.denoising_unet(
+ noisy_latents,
+ timesteps,
+ encoder_hidden_states=clip_image_embeds,
+ pose_cond_fea=pose_emb,
+ ).sample
+
+ return model_pred
+
+ def update_reference(
+ self,
+ noisy_latents,
+ timesteps,
+ ref_image_latents,
+ clip_image_embeds,
+ ):
+ self.reference_unet(
+ ref_image_latents,
+ torch.zeros_like(timesteps[:noisy_latents.shape[0]]),
+ encoder_hidden_states=clip_image_embeds,
+ return_dict=False,
+ )
+ self.reference_control_reader.update(self.reference_control_writer, drop_ratio=0.3)
+
+def get_x0_from_eps(ddim, sample: torch.FloatTensor, model_output: torch.FloatTensor, timesteps: torch.IntTensor):
+ video_length = sample.shape[2]
+ sample = rearrange(sample, 'b c f h w -> (b f) c h w')
+ model_output = rearrange(model_output, 'b c f h w -> (b f) c h w')
+ alpha_prod = ddim.alphas_cumprod.cuda()
+ alpha_prod_t = alpha_prod[timesteps]
+ while len(alpha_prod_t.shape) < len(sample.shape):
+ alpha_prod_t = alpha_prod_t.unsqueeze(-1)
+
+ pred_original_sample = torch.sqrt(1./alpha_prod_t) * sample - torch.sqrt(1. / alpha_prod_t - 1) * model_output
+
+ pred_original_sample = rearrange(pred_original_sample, '(b f) c h w -> b c f h w', f=video_length)
+
+ return pred_original_sample
+
+def crop_face(image_pil, face_mesh):
+ image = np.array(image_pil)
+ h, w = image.shape[:2]
+ results = face_mesh.process(image)
+ face_landmarks = results.multi_face_landmarks[0]
+ coords = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]
+ xs, ys = zip(*coords)
+ x1, y1 = min(xs), min(ys)
+ x2, y2 = max(xs), max(ys)
+ face_box = (x1, y1, x2, y2)
+
+ left, top, right, bot = scale_bb(face_box, scale=1.1, size=image.shape[:2])
+
+ face_patch = image[int(top) : int(bot), int(left) : int(right)]
+
+ return face_box, face_patch
+
+def decode_latents(vae: AutoencoderKL, latents, decode_chunk_size=4):
+ latents = latents.to(vae.dtype)
+ video_length = latents.shape[2]
+ latents = 1 / 0.18215 * latents
+ latents = rearrange(latents, "b c f h w -> (b f) c h w")
+ # video = self.vae.decode(latents).sample
+ video = []
+ for frame_idx in range(0, latents.shape[0], decode_chunk_size):
+ video.append(vae.decode(latents[frame_idx : frame_idx + decode_chunk_size]).sample)
+ video = torch.cat(video)
+ # video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)
+ # video = (video / 2 + 0.5).clamp(0, 1)
+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
+ return video
+
+def log_validation(
+ vae,
+ image_enc,
+ net,
+ scheduler,
+ accelerator,
+ width,
+ height,
+ face_mesh,
+ pose_encoder,
+ timesteps_list,
+ clip_length=100,
+ generator=None,
+):
+ logger.info("Running validation... ")
+
+ ori_net = accelerator.unwrap_model(net)
+ reference_unet = ori_net.reference_unet
+ denoising_unet = ori_net.denoising_unet
+ motion_enc = ori_net.motion_encoder
+ pose_gui = ori_net.pose_guider
+ pose_enc = pose_encoder
+
+ motion_enc.eval()
+
+ # generator = torch.manual_seed(42)
+ generator = torch.Generator(device='cuda').manual_seed(42)
+ # cast unet dtype
+
+ pipe = Pose2ImagePipeline(
+ vae=vae,
+ image_encoder=image_enc,
+ reference_unet=reference_unet,
+ denoising_unet=denoising_unet,
+ motion_encoder=motion_enc,
+ pose_encoder=pose_enc,
+ pose_guider=pose_gui,
+ scheduler=scheduler,
+ )
+ pipe = pipe.to(accelerator.device)
+
+ ref_image_paths = [
+ "path/to/ref_img_1.png",
+ "path/to/ref_img_2.png",
+ ]
+ videos_path = [
+ "path/to/ref_vid_1.png",
+ "path/to/ref_vid_2.png",
+ ]
+
+ pose_transform = transforms.Compose(
+ [transforms.Resize((height, width)), transforms.ToTensor()]
+ )
+ results = []
+ for idy, pose_video_path in enumerate(videos_path):
+ pose_images = read_frames(pose_video_path)
+
+ boxes_path = pose_video_path.replace("videos", "boxes").replace(".mp4", ".pt")
+ boxes = torch.load(boxes_path)
+
+ pose_face = []
+ ori_pose_images = []
+ for idx_control, pose_image_pil in enumerate(pose_images[:clip_length]):
+ face_bbox = boxes[idx_control]["face"]
+ ori_pose_images.append(pose_image_pil)
+ pose_image = np.array(pose_image_pil)
+ left, top, right, bot = scale_bb(face_bbox, scale=1.1, size=pose_image.shape[:2])
+ pose_image = pose_image[int(top) : int(bot), int(left) : int(right)]
+ pose_image_pil = Image.fromarray(pose_image).convert("RGB")
+ pose_face.append(pose_image_pil)
+
+ pose_tensor_list = []
+ ori_pose_tensor_list = []
+
+ for idx_control, pose_image_pil in enumerate(ori_pose_images):
+ pose_tensor_list.append(pose_transform(pose_face[idx_control]))
+ ori_pose_tensor_list.append(pose_transform(pose_image_pil))
+ pose_tensor = torch.stack(pose_tensor_list, dim=0) # (f, c, h, w)
+ pose_tensor = pose_tensor.transpose(0, 1).unsqueeze(0)
+
+ ori_pose_tensor = torch.stack(ori_pose_tensor_list, dim=0) # (f, c, h, w)
+ ori_pose_tensor = ori_pose_tensor.transpose(0, 1).unsqueeze(0)
+
+ for idx, ref_image_path in enumerate(ref_image_paths):
+ ref_image_pil = Image.open(ref_image_path).convert("RGB")
+ ref_bbox, ref_patch = crop_face(ref_image_pil, face_mesh)
+ ref_face = Image.fromarray(ref_patch)
+
+ ref_tensor = pose_transform(ref_image_pil).unsqueeze(0).unsqueeze(2).expand_as(pose_tensor)
+
+ ref_name = f'{idx}'
+ pose_name = f'{idy}'
+
+ with torch.cuda.amp.autocast():
+ gen_video = pipe(
+ ori_pose_images,
+ ref_image_pil,
+ pose_face,
+ ref_face,
+ width,
+ height,
+ clip_length,
+ 4,
+ 1.0,
+ timesteps_list=timesteps_list,
+ generator=generator,
+ ).videos
+
+ # Concat it with pose tensor
+ video = torch.cat([ref_tensor, pose_tensor, ori_pose_tensor, gen_video], dim=0)
+
+ results.append({"name": f"{ref_name}_{pose_name}", "vid": video})
+
+ del pipe
+ torch.cuda.empty_cache()
+ gc.collect()
+ motion_enc.train()
+ return results
+
+
+def main(cfg):
+ kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
+ accelerator = Accelerator(
+ gradient_accumulation_steps=cfg.solver.gradient_accumulation_steps,
+ mixed_precision=cfg.solver.mixed_precision,
+ log_with="mlflow",
+ project_dir="./mlruns",
+ kwargs_handlers=[kwargs],
+ )
+
+ # Make one log on every process with the configuration for debugging.
+ logging.basicConfig(
+ format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+ datefmt="%m/%d/%Y %H:%M:%S",
+ level=logging.INFO,
+ )
+ logger.info(accelerator.state, main_process_only=False)
+ if accelerator.is_local_main_process:
+ transformers.utils.logging.set_verbosity_warning()
+ diffusers.utils.logging.set_verbosity_info()
+ else:
+ transformers.utils.logging.set_verbosity_error()
+ diffusers.utils.logging.set_verbosity_error()
+
+ # If passed along, set the training seed now.
+ if cfg.seed is not None:
+ seed_everything(cfg.seed)
+
+ exp_name = cfg.exp_name
+ save_dir = f"{cfg.output_dir}/{exp_name}"
+ if accelerator.is_main_process and not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+
+ inference_config_path = "./configs/inference/inference_stage1&2.yaml"
+ infer_config = OmegaConf.load(inference_config_path)
+
+ if cfg.weight_dtype == "fp16":
+ weight_dtype = torch.float16
+ elif cfg.weight_dtype == "bf16":
+ weight_dtype = torch.bfloat16
+ elif cfg.weight_dtype == "fp32":
+ weight_dtype = torch.float32
+ else:
+ raise ValueError(
+ f"Do not support weight dtype: {cfg.weight_dtype} during training"
+ )
+
+ sched_kwargs = OmegaConf.to_container(
+ infer_config.noise_scheduler_kwargs
+ )
+
+ val_noise_scheduler = DDIMScheduler(**sched_kwargs)
+ train_noise_scheduler = DDIMScheduler(**sched_kwargs)
+ vae = AutoencoderKL.from_pretrained(cfg.vae_model_path).to(
+ "cuda", dtype=weight_dtype
+ )
+ net_lpips = lpips.LPIPS(net='vgg').cuda()
+
+ image_enc = CLIPVisionModelWithProjection.from_pretrained(
+ cfg.image_encoder_path,
+ ).to(dtype=weight_dtype, device="cuda")
+
+ reference_unet = UNet2DConditionModel.from_pretrained(
+ cfg.base_model_path,
+ subfolder="unet",
+ ).to(device="cuda")
+ denoising_unet = UNet3DConditionModel.from_pretrained_2d(
+ cfg.base_model_path,
+ "",
+ subfolder="unet",
+ unet_additional_kwargs=OmegaConf.to_container(
+ infer_config.unet_additional_kwargs
+ ),
+ ).to(device="cuda")
+
+ motion_encoder = MotEncoder().to(device="cuda")
+ motion_encoder.load_state_dict(
+ torch.load(cfg.motion_encoder_path, map_location="cpu"), strict=False
+ )
+
+ pose_guider = PoseGuider().to(device="cuda")
+ pose_guider.load_state_dict(torch.load(cfg.pose_guider_path, map_location='cpu'), strict=False)
+
+ pose_encoder = MotionExtractor(num_kp=21).to(device="cuda").eval()
+ pose_encoder.load_state_dict(torch.load(cfg.pose_encoder_path, map_location='cpu'), strict=False)
+
+ denoising_unet.load_state_dict(
+ torch.load(cfg.denoising_unet_path, map_location="cpu"),
+ strict=False,
+ )
+ reference_unet.load_state_dict(
+ torch.load(cfg.reference_unet_path, map_location="cpu"),
+ )
+
+ mp_face_mesh = mp.solutions.face_mesh
+ face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1)
+
+ discriminator = StyleGAN2Discriminator()
+ discriminator = discriminator.to(device="cuda")
+ discriminator.load_state_dict(
+ torch.load(cfg.discriminator_path, map_location="cpu"), strict=False
+ )
+
+ # Freeze
+ vae.requires_grad_(False)
+ image_enc.requires_grad_(False)
+ net_lpips.requires_grad_(False)
+ pose_encoder.requires_grad_(False)
+
+ pose_guider.requires_grad_(True)
+ motion_encoder.requires_grad_(True)
+ denoising_unet.requires_grad_(True)
+ reference_unet.requires_grad_(False)
+ discriminator.requires_grad_(True)
+
+ for name, param in reference_unet.named_parameters():
+ if "up_blocks.3" in name:
+ param.requires_grad_(False)
+ else:
+ param.requires_grad_(True)
+
+ reference_control_writer = ReferenceAttentionControl(
+ reference_unet,
+ do_classifier_free_guidance=False,
+ mode="write",
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ denoising_unet,
+ do_classifier_free_guidance=False,
+ mode="read",
+ fusion_blocks="full",
+ )
+
+ net = Net(
+ reference_unet,
+ denoising_unet,
+ motion_encoder,
+ pose_guider,
+ reference_control_writer,
+ reference_control_reader,
+ )
+
+ if cfg.solver.enable_xformers_memory_efficient_attention:
+ if is_xformers_available():
+ reference_unet.enable_xformers_memory_efficient_attention()
+ denoising_unet.enable_xformers_memory_efficient_attention()
+ else:
+ raise ValueError(
+ "xformers is not available. Make sure it is installed correctly"
+ )
+
+ if cfg.solver.gradient_checkpointing:
+ reference_unet.enable_gradient_checkpointing()
+ denoising_unet.enable_gradient_checkpointing()
+
+ if cfg.solver.scale_lr:
+ learning_rate = (
+ cfg.solver.learning_rate
+ * cfg.solver.gradient_accumulation_steps
+ * cfg.data.train_bs
+ * accelerator.num_processes
+ )
+ else:
+ learning_rate = cfg.solver.learning_rate
+
+ # Initialize the optimizer
+ if cfg.solver.use_8bit_adam:
+ try:
+ import bitsandbytes as bnb
+ except ImportError:
+ raise ImportError(
+ "Please install bitsandbytes to use 8-bit Adam. You can do so by running `pip install bitsandbytes`"
+ )
+
+ optimizer_cls = bnb.optim.AdamW8bit
+ else:
+ optimizer_cls = torch.optim.AdamW
+
+ trainable_params = list(filter(lambda p: p.requires_grad, net.parameters()))
+
+ optimizer = optimizer_cls(
+ trainable_params,
+ lr=learning_rate,
+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),
+ weight_decay=cfg.solver.adam_weight_decay,
+ eps=cfg.solver.adam_epsilon,
+ )
+
+ dis_optimizer = optimizer_cls(
+ discriminator.parameters(),
+ lr=learning_rate * 2,
+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),
+ weight_decay=cfg.solver.adam_weight_decay,
+ eps=cfg.solver.adam_epsilon,
+ )
+
+ # Scheduler
+ lr_scheduler = get_scheduler(
+ cfg.solver.lr_scheduler,
+ optimizer=optimizer,
+ num_warmup_steps=cfg.solver.lr_warmup_steps
+ * cfg.solver.gradient_accumulation_steps,
+ num_training_steps=cfg.solver.max_train_steps
+ * cfg.solver.gradient_accumulation_steps,
+ )
+
+ train_dataset = PortraitImageDataset(
+ img_size=(cfg.data.train_width, cfg.data.train_height),
+ img_scale=(0.9, 1.0),
+ data_meta_paths=cfg.data.meta_paths,
+ sample_margin=cfg.data.sample_margin,
+ )
+ train_dataloader = torch.utils.data.DataLoader(
+ train_dataset, batch_size=cfg.data.train_bs, shuffle=True, num_workers=8, drop_last=True
+ )
+
+ # Prepare everything with our `accelerator`.
+ (
+ net,
+ optimizer,
+ discriminator,
+ dis_optimizer,
+ train_dataloader,
+ lr_scheduler,
+ ) = accelerator.prepare(
+ net,
+ optimizer,
+ discriminator,
+ dis_optimizer,
+ train_dataloader,
+ lr_scheduler,
+ )
+
+ # We need to recalculate our total training steps as the size of the training dataloader may have changed.
+ num_update_steps_per_epoch = math.ceil(
+ len(train_dataloader) / cfg.solver.gradient_accumulation_steps
+ )
+ # Afterwards we recalculate our number of training epochs
+ num_train_epochs = math.ceil(
+ cfg.solver.max_train_steps / num_update_steps_per_epoch
+ )
+
+ # We need to initialize the trackers we use, and also store our configuration.
+ # The trackers initializes automatically on the main process.
+ if accelerator.is_main_process:
+ run_time = datetime.now().strftime("%Y%m%d-%H%M")
+ accelerator.init_trackers(
+ cfg.exp_name,
+ init_kwargs={"mlflow": {"run_name": run_time}},
+ )
+ # dump config file
+ mlflow.log_dict(OmegaConf.to_container(cfg), "config.yaml")
+
+ # Train!
+ total_batch_size = (
+ cfg.data.train_bs
+ * accelerator.num_processes
+ * cfg.solver.gradient_accumulation_steps
+ )
+
+ logger.info("***** Running training *****")
+ logger.info(f" Num examples = {len(train_dataset)}")
+ logger.info(f" Num Epochs = {num_train_epochs}")
+ logger.info(f" Instantaneous batch size per device = {cfg.data.train_bs}")
+ logger.info(
+ f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"
+ )
+ logger.info(
+ f" Gradient Accumulation steps = {cfg.solver.gradient_accumulation_steps}"
+ )
+ logger.info(f" Total optimization steps = {cfg.solver.max_train_steps}")
+ global_step = 0
+ first_epoch = 0
+
+ # Potentially load in the weights and states from a previous save
+ if cfg.resume_from_checkpoint:
+ if cfg.resume_from_checkpoint != "latest":
+ resume_dir = cfg.resume_from_checkpoint
+ else:
+ resume_dir = save_dir
+ # Get the most recent checkpoint
+ dirs = os.listdir(resume_dir)
+ dirs = [d for d in dirs if d.startswith("checkpoint")]
+ dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))
+ path = dirs[-1]
+ accelerator.load_state(os.path.join(resume_dir, path))
+ accelerator.print(f"Resuming from checkpoint {path}")
+ global_step = int(path.split("-")[1])
+
+ first_epoch = global_step // num_update_steps_per_epoch
+
+ # Only show the progress bar once on each machine.
+ progress_bar = tqdm(
+ range(global_step, cfg.solver.max_train_steps),
+ disable=not accelerator.is_local_main_process,
+ )
+ progress_bar.set_description("Steps")
+
+ for epoch in range(first_epoch, num_train_epochs):
+ train_loss = 0.0
+ train_dis_loss = 0.0
+ for step, batch in enumerate(train_dataloader):
+ with accelerator.accumulate(None):
+ # Convert videos to latent space
+ pixel_values = batch["img"].to(weight_dtype) # b,c,h,w
+ tgt_face = batch["tgt_face"]
+ tgt_pose = batch["tgt_pose"]
+
+ face_mask = batch['face_mask']
+ local_mask = batch['local_mask']
+
+ with torch.no_grad():
+ latents = vae.encode(pixel_values).latent_dist.sample()
+ latents = latents.unsqueeze(2)
+ latents = latents * 0.18215 # (b f) c h w
+
+ keypoints = pose_encoder(tgt_pose)
+ tgt_pose = draw_keypoints(keypoints).unsqueeze(2).to('cuda')
+
+ if epoch == 0 and step == 0:
+ if accelerator.is_main_process:
+ save_image(tgt_pose[:1,:,0], "image_kps.png")
+ save_image((pixel_values[:1] + 1) / 2, "image_tgt.png")
+
+ noise = torch.randn_like(latents)
+
+ timesteps_list = cfg.timesteps_list
+
+ bsz = latents.shape[0] # (b f)
+ timesteps = torch.ones(bsz, device=latents.device) * timesteps_list[0]
+ timesteps = timesteps.long()
+
+ facial_mask_pix = face_mask.repeat(1,3,1,1) + local_mask.repeat(1,3,1,1)
+
+ uncond_fwd = random.random() < cfg.uncond_ratio
+ clip_image_list = []
+ ref_image_list = []
+ for batch_idx, (ref_img, clip_img) in enumerate(
+ zip(
+ batch["ref_img"],
+ batch["clip_image"],
+ )
+ ):
+ clip_image_list.append(clip_img)
+ ref_image_list.append(ref_img)
+
+ with torch.no_grad():
+ ref_img = torch.stack(ref_image_list, dim=0).to(
+ dtype=vae.dtype, device=vae.device
+ )
+ ref_image_latents = vae.encode(
+ ref_img
+ ).latent_dist.sample() # (bs, d, 64, 64)
+ ref_image_latents = ref_image_latents * 0.18215
+
+ clip_img = torch.stack(clip_image_list, dim=0).to(
+ dtype=image_enc.dtype, device=image_enc.device
+ )
+ clip_image_embeds = image_enc(
+ clip_img.to("cuda", dtype=weight_dtype)
+ ).image_embeds
+ image_prompt_embeds = clip_image_embeds.unsqueeze(1) # (bs, 1, d)
+
+ noisy_latents = noise
+
+ update_step = random.randint(1, 4)
+
+ ref_flag = True
+ mot_flag = random.random() < 0.1
+ pose_flag = random.random() < 0.1
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+
+ for i in range(update_step-1):
+ with torch.no_grad():
+ model_pred = net(
+ noisy_latents,
+ timesteps,
+ ref_image_latents,
+ image_prompt_embeds,
+ tgt_face,
+ tgt_pose,
+ ref_flag,
+ mot_flag,
+ pose_flag,
+ uncond_fwd,
+ )
+
+ ref_flag = False
+
+ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)
+ timesteps = torch.ones(bsz, device=latents.device) * timesteps_list[i+1]
+ timesteps = timesteps.long()
+
+ noise = torch.randn_like(latents_pred)
+ noisy_latents = train_noise_scheduler.add_noise(
+ latents_pred, noise, timesteps
+ ).detach()
+
+ if ref_flag == False:
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+ ref_flag = True
+
+ model_pred = net(
+ noisy_latents,
+ timesteps,
+ ref_image_latents,
+ image_prompt_embeds,
+ tgt_face,
+ tgt_pose,
+ ref_flag,
+ mot_flag,
+ pose_flag,
+ uncond_fwd,
+ )
+
+ discriminator.requires_grad_(False)
+
+ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)
+
+ image_pred = decode_latents(vae, latents_pred)
+ loss = F.mse_loss(image_pred.float(), pixel_values.float(), reduction="mean")
+ loss += F.mse_loss(image_pred.float() * facial_mask_pix, pixel_values.float() * facial_mask_pix, reduction="mean")
+ loss_lpips = net_lpips(image_pred.float(), pixel_values.float()).mean() # 输入应该是[-1,1]
+
+ if global_step > 3000: # This value may not be optimal
+ adv_loss = discriminator(image_pred.float(), pixel_values.float(), timesteps)
+ else:
+ adv_loss = torch.tensor(0.0).to(loss.device)
+ loss = loss + loss_lpips * 2 + adv_loss * 0.05
+
+ # Gather the losses across all processes for logging (if we use distributed training).
+ avg_loss = accelerator.gather(loss.repeat(cfg.data.train_bs)).mean()
+ train_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps
+
+ # Backpropagate
+ accelerator.backward(loss)
+ if accelerator.sync_gradients:
+ accelerator.clip_grad_norm_(
+ trainable_params,
+ cfg.solver.max_grad_norm,
+ )
+ optimizer.step()
+ lr_scheduler.step()
+ optimizer.zero_grad(set_to_none=True)
+
+ if global_step > 5000: # This value may not be optimal
+ discriminator.requires_grad_(True)
+ dis_loss = discriminator(image_pred.float().detach(), pixel_values.float(), timesteps, gen=False)
+ avg_loss = accelerator.gather(dis_loss.repeat(cfg.data.train_bs)).mean()
+ train_dis_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps
+
+ accelerator.backward(dis_loss)
+ if accelerator.sync_gradients:
+ accelerator.clip_grad_norm_(
+ discriminator.parameters(),
+ cfg.solver.max_grad_norm,
+ )
+ dis_optimizer.step()
+ dis_optimizer.zero_grad(set_to_none=True)
+ else:
+ dis_loss = torch.tensor(0.0).to(loss.device)
+
+
+ if accelerator.sync_gradients:
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+ progress_bar.update(1)
+ global_step += 1
+ accelerator.log({"train_loss": train_loss, "train_dis_loss": train_dis_loss}, step=global_step)
+ train_loss = 0.0
+ train_dis_loss = 0.0
+ if global_step % cfg.checkpointing_steps == 0:
+ if accelerator.is_main_process:
+ save_path = os.path.join(save_dir, f"checkpoint-{global_step}")
+ delete_additional_ckpt(save_dir, 1)
+ accelerator.save_state(save_path)
+
+ if global_step % cfg.val.validation_steps == 0 or global_step == 1:
+ accelerator.wait_for_everyone()
+ if accelerator.is_main_process:
+ generator = torch.Generator(device=accelerator.device)
+ generator.manual_seed(cfg.seed)
+
+ sample_dicts = log_validation(
+ vae=vae,
+ image_enc=image_enc,
+ net=net,
+ scheduler=val_noise_scheduler,
+ accelerator=accelerator,
+ width=cfg.data.train_width,
+ height=cfg.data.train_height,
+ face_mesh=face_mesh,
+ pose_encoder=pose_encoder,
+ timesteps_list=timesteps_list,
+ generator=generator,
+ )
+
+ reference_control_writer = ReferenceAttentionControl(
+ reference_unet,
+ do_classifier_free_guidance=False,
+ mode="write",
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ denoising_unet,
+ do_classifier_free_guidance=False,
+ mode="read",
+ fusion_blocks="full",
+ )
+
+ if accelerator.is_main_process:
+ for sample_id, sample_dict in enumerate(sample_dicts):
+ sample_name = sample_dict["name"]
+ vid = sample_dict["vid"]
+ with TemporaryDirectory() as temp_dir:
+ out_file = f"{temp_dir}/{global_step:06d}-{sample_name}.mp4"
+ save_videos_grid(vid, out_file, n_rows=4, fps=25)
+ mlflow.log_artifact(out_file)
+
+ logs = {
+ "loss": loss.detach().item(),
+ "step": update_step,
+ "dis": dis_loss.detach().item(),
+ "adv": adv_loss.detach().item(),
+ "lr": lr_scheduler.get_last_lr()[0],
+ }
+ progress_bar.set_postfix(**logs)
+
+ if global_step >= cfg.solver.max_train_steps:
+ break
+
+ # save model after each epoch
+ if (
+ epoch + 1
+ ) % cfg.save_model_epoch_interval == 0 and accelerator.is_main_process:
+ unwrap_net = accelerator.unwrap_model(net)
+ unwrap_discriminator = accelerator.unwrap_model(discriminator)
+ save_checkpoint(
+ unwrap_net.denoising_unet,
+ save_dir,
+ "denoising_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.reference_unet,
+ save_dir,
+ "reference_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.motion_encoder,
+ save_dir,
+ "motion_encoder",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.pose_guider,
+ save_dir,
+ "pose_guider",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_discriminator,
+ save_dir,
+ "discriminator",
+ global_step,
+ total_limit=2,
+ )
+
+ unwrap_net = accelerator.unwrap_model(net)
+ unwrap_discriminator = accelerator.unwrap_model(discriminator)
+ save_checkpoint(
+ unwrap_net.denoising_unet,
+ save_dir,
+ "denoising_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.reference_unet,
+ save_dir,
+ "reference_unet",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.motion_encoder,
+ save_dir,
+ "motion_encoder",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_net.pose_guider,
+ save_dir,
+ "pose_guider",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint(
+ unwrap_discriminator,
+ save_dir,
+ "discriminator",
+ global_step,
+ total_limit=2,
+ )
+
+ # Create the pipeline using the trained modules and save it.
+ accelerator.wait_for_everyone()
+ accelerator.end_training()
+
+
+def save_checkpoint(model, save_dir, prefix, ckpt_num, total_limit=None):
+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")
+
+ if total_limit is not None:
+ checkpoints = os.listdir(save_dir)
+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]
+ checkpoints = sorted(
+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])
+ )
+
+ if len(checkpoints) >= total_limit:
+ num_to_remove = len(checkpoints) - total_limit + 1
+ removing_checkpoints = checkpoints[0:num_to_remove]
+ logger.info(
+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
+ )
+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")
+
+ for removing_checkpoint in removing_checkpoints:
+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)
+ os.remove(removing_checkpoint)
+
+ state_dict = model.state_dict()
+ torch.save(state_dict, save_path)
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--config", type=str, default="./configs/train/personalive_stage2.yaml")
+ args = parser.parse_args()
+
+ if args.config[-5:] == ".yaml":
+ config = OmegaConf.load(args.config)
+ elif args.config[-3:] == ".py":
+ config = import_filename(args.config).cfg
+ else:
+ raise ValueError("Do not support this format config file")
+ main(config)
diff --git a/train_stage3.py b/train_stage3.py
new file mode 100644
index 0000000..a8407b1
--- /dev/null
+++ b/train_stage3.py
@@ -0,0 +1,954 @@
+import sys
+sys.path.append('./src/stylegan2')
+
+import argparse
+import logging
+import math
+import os
+import os.path as osp
+import random
+import warnings
+from collections import OrderedDict
+from datetime import datetime
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from torchvision import transforms
+
+import cv2
+import lpips
+import diffusers
+from einops import rearrange
+import mlflow
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.utils.checkpoint
+import transformers
+from accelerate import Accelerator
+from accelerate.logging import get_logger
+from accelerate.utils import DistributedDataParallelKwargs
+from diffusers import AutoencoderKL, AutoencoderKLTemporalDecoder, AutoencoderTiny
+from src.scheduler.scheduler_ddim import DDIMScheduler
+from diffusers.optimization import get_scheduler
+from diffusers.utils import check_min_version
+from diffusers.utils.import_utils import is_xformers_available
+from omegaconf import OmegaConf
+from PIL import Image
+from tqdm.auto import tqdm
+from transformers import CLIPVisionModelWithProjection
+from decord import VideoReader
+from src.dataset.portrait_video import PortraitVideoDataset
+from src.models.mutual_self_attention import ReferenceAttentionControl
+from src.models.unet_2d_condition import UNet2DConditionModel
+from src.models.unet_3d import UNet3DConditionModel
+from src.pipelines.pipeline_pose2vid import Pose2VideoPipeline
+from src.utils.util import (
+ delete_additional_ckpt,
+ import_filename,
+ read_frames,
+ save_videos_grid,
+ seed_everything,
+ draw_keypoints,
+)
+from src.dataset.utils import scale_bb
+from src.models.motion_encoder.encoder import MotEncoder
+from src.models.motion_module import zero_module
+warnings.filterwarnings("ignore")
+import mediapipe as mp
+from src.liveportrait.motion_extractor import MotionExtractor
+from src.models.pose_guider import PoseGuider
+import gc
+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
+check_min_version("0.10.0.dev0")
+from torchvision.utils import save_image
+from src.stylegan2.discriminator import Discriminator as StyleGAN2Discriminator
+
+logger = get_logger(__name__, log_level="INFO")
+
+class Net(nn.Module):
+ def __init__(
+ self,
+ denoising_unet: UNet3DConditionModel,
+ reference_control_reader,
+ ):
+ super().__init__()
+ self.denoising_unet = denoising_unet
+ self.reference_control_reader = reference_control_reader
+
+ def forward(
+ self,
+ noisy_latents,
+ timesteps,
+ clip_image_embeds,
+ motion,
+ pose,
+ ):
+ motion_emb = motion
+
+ clip_image_embeds = [clip_image_embeds, motion_emb]
+
+ model_pred = self.denoising_unet(
+ noisy_latents,
+ timesteps,
+ encoder_hidden_states=clip_image_embeds,
+ pose_cond_fea=pose,
+ ).sample
+
+ return model_pred
+
+def get_x0_from_eps(ddim, sample: torch.FloatTensor, model_output: torch.FloatTensor, timesteps: torch.IntTensor):
+ video_length = sample.shape[2]
+ sample = rearrange(sample, 'b c f h w -> (b f) c h w')
+ model_output = rearrange(model_output, 'b c f h w -> (b f) c h w')
+ alpha_prod = ddim.alphas_cumprod.cuda()
+ alpha_prod_t = alpha_prod[timesteps]
+ while len(alpha_prod_t.shape) < len(sample.shape):
+ alpha_prod_t = alpha_prod_t.unsqueeze(-1)
+
+ pred_original_sample = torch.sqrt(1./alpha_prod_t) * sample - torch.sqrt(1. / alpha_prod_t - 1) * model_output
+
+ pred_original_sample = rearrange(pred_original_sample, '(b f) c h w -> b c f h w', f=video_length)
+
+ return pred_original_sample
+
+def crop_face(image_pil, face_mesh):
+ image = np.array(image_pil)
+ h, w = image.shape[:2]
+ results = face_mesh.process(image)
+ face_landmarks = results.multi_face_landmarks[0]
+ coords = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]
+ xs, ys = zip(*coords)
+ x1, y1 = min(xs), min(ys)
+ x2, y2 = max(xs), max(ys)
+ face_box = (x1, y1, x2, y2)
+
+ left, top, right, bot = scale_bb(face_box, scale=1.1, size=image.shape[:2])
+
+ face_patch = image[int(top) : int(bot), int(left) : int(right)]
+
+ return face_box, face_patch
+
+def decode_latents(vae: AutoencoderKL, latents, decode_chunk_size=1):
+ latents = latents.to(vae.dtype)
+ video_length = latents.shape[2]
+ latents = 1 / 0.18215 * latents
+ latents = rearrange(latents, "b c f h w -> (b f) c h w")
+ # video = self.vae.decode(latents).sample
+ video = []
+ for frame_idx in range(0, latents.shape[0], decode_chunk_size):
+ video.append(vae.decode(latents[frame_idx : frame_idx + decode_chunk_size]).sample)
+ video = torch.cat(video)
+ video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)
+ # video = (video / 2 + 0.5).clamp(0, 1)
+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
+ return video
+
+def log_validation(
+ vae,
+ image_enc,
+ net,
+ scheduler,
+ accelerator,
+ width,
+ height,
+ face_mesh,
+ reference_unet,
+ pose_encoder,
+ pose_guider,
+ motion_encoder,
+ timesteps_list,
+ clip_length=100,
+ generator=None,
+):
+ logger.info("Running validation... ")
+
+ ori_net = accelerator.unwrap_model(net)
+ reference_unet = reference_unet
+ denoising_unet = ori_net.denoising_unet
+ motion_enc = motion_encoder
+ pose_gui = pose_guider
+ pose_enc = pose_encoder
+
+ # generator = torch.manual_seed(42)
+ generator = torch.Generator(device='cuda').manual_seed(42)
+ # cast unet dtype
+
+ pipe = Pose2VideoPipeline(
+ vae=vae,
+ image_encoder=image_enc,
+ reference_unet=reference_unet,
+ denoising_unet=denoising_unet,
+ motion_encoder=motion_enc,
+ pose_encoder=pose_enc,
+ pose_guider=pose_gui,
+ scheduler=scheduler,
+ )
+ pipe = pipe.to(accelerator.device)
+
+ ref_image_paths = [
+ "path/to/ref_img_1.png",
+ "path/to/ref_img_2.png",
+ ]
+ videos_path = [
+ "path/to/ref_vid_1.png",
+ "path/to/ref_vid_2.png",
+ ]
+
+ pose_transform = transforms.Compose(
+ [transforms.Resize((height, width)), transforms.ToTensor()]
+ )
+ results = []
+ for idy, pose_video_path in enumerate(videos_path):
+ pose_images = read_frames(pose_video_path)
+
+ boxes_path = pose_video_path.replace("videos", "boxes").replace(".mp4", ".pt")
+ boxes = torch.load(boxes_path)
+
+ pose_face = []
+ ori_pose_images = []
+ for idx_control, pose_image_pil in enumerate(pose_images[:clip_length]):
+ face_bbox = boxes[idx_control]["face"]
+ ori_pose_images.append(pose_image_pil)
+ pose_image = np.array(pose_image_pil)
+ left, top, right, bot = scale_bb(face_bbox, scale=1.1, size=pose_image.shape[:2])
+ pose_image = pose_image[int(top) : int(bot), int(left) : int(right)]
+ pose_image_pil = Image.fromarray(pose_image).convert("RGB")
+ pose_face.append(pose_image_pil)
+
+ pose_tensor_list = []
+ ori_pose_tensor_list = []
+
+ for idx_control, pose_image_pil in enumerate(ori_pose_images):
+ pose_tensor_list.append(pose_transform(pose_face[idx_control]))
+ ori_pose_tensor_list.append(pose_transform(pose_image_pil))
+ pose_tensor = torch.stack(pose_tensor_list, dim=0) # (f, c, h, w)
+ pose_tensor = pose_tensor.transpose(0, 1).unsqueeze(0)
+
+ ori_pose_tensor = torch.stack(ori_pose_tensor_list, dim=0) # (f, c, h, w)
+ ori_pose_tensor = ori_pose_tensor.transpose(0, 1).unsqueeze(0)
+
+ for idx, ref_image_path in enumerate(ref_image_paths):
+ ref_image_pil = Image.open(ref_image_path).convert("RGB")
+ ref_bbox, ref_patch = crop_face(ref_image_pil, face_mesh)
+ ref_face = Image.fromarray(ref_patch)
+
+ ref_tensor = pose_transform(ref_image_pil).unsqueeze(0).unsqueeze(2).expand_as(pose_tensor)
+
+ ref_name = f'{idx}'
+ pose_name = f'{idy}'
+
+ with torch.cuda.amp.autocast():
+ gen_video = pipe(
+ ori_pose_images,
+ ref_image_pil,
+ pose_face,
+ ref_face,
+ width,
+ height,
+ clip_length,
+ 4,
+ 1.0,
+ timesteps_list=timesteps_list,
+ generator=generator,
+ temporal_window_size = 4,
+ temporal_adaptive_step = 4,
+ temporal_kv_cache=False,
+ ).videos
+
+ # Concat it with pose tensor
+ video = torch.cat([ref_tensor, pose_tensor, ori_pose_tensor, gen_video], dim=0)
+
+ results.append({"name": f"{ref_name}_{pose_name}", "vid": video})
+
+ del pipe
+ torch.cuda.empty_cache()
+ gc.collect()
+ return results
+
+
+def main(cfg):
+ kwargs = DistributedDataParallelKwargs(find_unused_parameters=True, static_graph=True)
+ accelerator = Accelerator(
+ gradient_accumulation_steps=cfg.solver.gradient_accumulation_steps,
+ mixed_precision=cfg.solver.mixed_precision,
+ log_with="mlflow",
+ project_dir="./mlruns",
+ kwargs_handlers=[kwargs],
+ )
+
+ # Make one log on every process with the configuration for debugging.
+ logging.basicConfig(
+ format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+ datefmt="%m/%d/%Y %H:%M:%S",
+ level=logging.INFO,
+ )
+ logger.info(accelerator.state, main_process_only=False)
+ if accelerator.is_local_main_process:
+ transformers.utils.logging.set_verbosity_warning()
+ diffusers.utils.logging.set_verbosity_info()
+ else:
+ transformers.utils.logging.set_verbosity_error()
+ diffusers.utils.logging.set_verbosity_error()
+
+ # If passed along, set the training seed now.
+ if cfg.seed is not None:
+ seed_everything(cfg.seed)
+
+ exp_name = cfg.exp_name
+ save_dir = f"{cfg.output_dir}/{exp_name}"
+ if accelerator.is_main_process and not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+
+ inference_config_path = "./configs/inference/inference_stage3.yaml"
+ infer_config = OmegaConf.load(inference_config_path)
+
+ if cfg.weight_dtype == "fp16":
+ weight_dtype = torch.float16
+ elif cfg.weight_dtype == "fp32":
+ weight_dtype = torch.float32
+ else:
+ raise ValueError(
+ f"Do not support weight dtype: {cfg.weight_dtype} during training"
+ )
+
+ sched_kwargs = OmegaConf.to_container(
+ infer_config.noise_scheduler_kwargs
+ )
+
+ val_noise_scheduler = DDIMScheduler(**sched_kwargs)
+ train_noise_scheduler = DDIMScheduler(**sched_kwargs)
+ vae = AutoencoderKL.from_pretrained(cfg.vae_model_path).to(
+ "cuda", dtype=weight_dtype
+ )
+ net_lpips = lpips.LPIPS(net='vgg').cuda()
+
+ image_enc = CLIPVisionModelWithProjection.from_pretrained(
+ cfg.image_encoder_path,
+ ).to(dtype=weight_dtype, device="cuda")
+
+ reference_unet = UNet2DConditionModel.from_pretrained(
+ cfg.base_model_path,
+ subfolder="unet",
+ ).to(device="cuda", dtype=weight_dtype)
+ denoising_unet = UNet3DConditionModel.from_pretrained_2d(
+ cfg.base_model_path,
+ '',
+ subfolder="unet",
+ unet_additional_kwargs=OmegaConf.to_container(
+ infer_config.unet_additional_kwargs
+ ),
+ ).to(device="cuda")
+
+ motion_encoder = MotEncoder().to(device="cuda", dtype=weight_dtype).eval()
+ motion_encoder.load_state_dict(
+ torch.load(cfg.motion_encoder_path, map_location="cpu"), strict=True
+ )
+
+ pose_guider = PoseGuider().to(device="cuda", dtype=weight_dtype)
+ pose_guider.load_state_dict(torch.load(cfg.pose_guider_path, map_location='cpu'), strict=True)
+
+ pose_encoder = MotionExtractor(num_kp=21).to(device="cuda").eval()
+ pose_encoder.load_state_dict(torch.load(cfg.pose_encoder_path, map_location='cpu'), strict=False)
+
+ denoising_unet.load_state_dict(
+ torch.load(cfg.denoising_unet_path, map_location="cpu"),
+ strict=False,
+ )
+ denoising_unet.load_state_dict(
+ torch.load(cfg.temporal_module_path, map_location="cpu"),
+ strict=False,
+ )
+ reference_unet.load_state_dict(
+ torch.load(cfg.reference_unet_path, map_location="cpu"),
+ )
+
+ mp_face_mesh = mp.solutions.face_mesh
+ face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1)
+
+ discriminator = StyleGAN2Discriminator()
+ discriminator = discriminator.to(device="cuda")
+ discriminator.load_state_dict(
+ torch.load(cfg.discriminator_path, map_location="cpu"), strict=True
+ )
+
+ # Freeze
+ vae.requires_grad_(False)
+ image_enc.requires_grad_(False)
+ net_lpips.requires_grad_(False)
+ pose_encoder.requires_grad_(False)
+
+ pose_guider.requires_grad_(False)
+ motion_encoder.requires_grad_(False)
+ denoising_unet.requires_grad_(False)
+ reference_unet.requires_grad_(False)
+
+ discriminator.requires_grad_(True)
+ for name, module in denoising_unet.named_modules():
+ if "temporal_modules" in name:
+ for params in module.parameters():
+ params.requires_grad = True
+
+ reference_control_writer = ReferenceAttentionControl(
+ reference_unet,
+ do_classifier_free_guidance=False,
+ mode="write",
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ denoising_unet,
+ do_classifier_free_guidance=False,
+ mode="read",
+ fusion_blocks="full",
+ )
+
+ net = Net(
+ denoising_unet,
+ reference_control_reader,
+ )
+
+ if cfg.solver.enable_xformers_memory_efficient_attention:
+ if is_xformers_available():
+ reference_unet.enable_xformers_memory_efficient_attention()
+ denoising_unet.enable_xformers_memory_efficient_attention()
+ else:
+ raise ValueError(
+ "xformers is not available. Make sure it is installed correctly"
+ )
+
+ if cfg.solver.gradient_checkpointing:
+ reference_unet.enable_gradient_checkpointing()
+ denoising_unet.enable_gradient_checkpointing()
+
+ # Ensure that no out-of-memory (OOM) error occurs
+ vae.decoder.gradient_checkpointing = True
+ vae.decoder.training = True
+
+ if cfg.solver.scale_lr:
+ learning_rate = (
+ cfg.solver.learning_rate
+ * cfg.solver.gradient_accumulation_steps
+ * cfg.data.train_bs
+ * accelerator.num_processes
+ )
+ else:
+ learning_rate = cfg.solver.learning_rate
+
+ # Initialize the optimizer
+ if cfg.solver.use_8bit_adam:
+ try:
+ import bitsandbytes as bnb
+ except ImportError:
+ raise ImportError(
+ "Please install bitsandbytes to use 8-bit Adam. You can do so by running `pip install bitsandbytes`"
+ )
+
+ optimizer_cls = bnb.optim.AdamW8bit
+ else:
+ optimizer_cls = torch.optim.AdamW
+
+ trainable_params = list(filter(lambda p: p.requires_grad, net.parameters()))
+
+ optimizer = optimizer_cls(
+ trainable_params,
+ lr=learning_rate,
+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),
+ weight_decay=cfg.solver.adam_weight_decay,
+ eps=cfg.solver.adam_epsilon,
+ )
+
+ dis_optimizer = optimizer_cls(
+ discriminator.parameters(),
+ lr=learning_rate * 2,
+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),
+ weight_decay=cfg.solver.adam_weight_decay,
+ eps=cfg.solver.adam_epsilon,
+ )
+
+ # Scheduler
+ lr_scheduler = get_scheduler(
+ cfg.solver.lr_scheduler,
+ optimizer=optimizer,
+ num_warmup_steps=cfg.solver.lr_warmup_steps
+ * cfg.solver.gradient_accumulation_steps,
+ num_training_steps=cfg.solver.max_train_steps
+ * cfg.solver.gradient_accumulation_steps,
+ )
+
+ train_dataset = PortraitVideoDataset(
+ img_size=(cfg.data.train_width, cfg.data.train_height),
+ img_scale=(0.9, 1.0),
+ data_meta_paths=cfg.data.meta_paths,
+ n_sample_frames=cfg.data.n_sample_frames,
+ sample_rate=cfg.data.sample_rate,
+ )
+ train_dataloader = torch.utils.data.DataLoader(
+ train_dataset, batch_size=cfg.data.train_bs, shuffle=True, num_workers=8, drop_last=True
+ )
+
+ # Prepare everything with our `accelerator`.
+ (
+ net,
+ optimizer,
+ discriminator,
+ dis_optimizer,
+ train_dataloader,
+ lr_scheduler,
+ ) = accelerator.prepare(
+ net,
+ optimizer,
+ discriminator,
+ dis_optimizer,
+ train_dataloader,
+ lr_scheduler,
+ )
+
+ # We need to recalculate our total training steps as the size of the training dataloader may have changed.
+ num_update_steps_per_epoch = math.ceil(
+ len(train_dataloader) / cfg.solver.gradient_accumulation_steps
+ )
+ # Afterwards we recalculate our number of training epochs
+ num_train_epochs = math.ceil(
+ cfg.solver.max_train_steps / num_update_steps_per_epoch
+ )
+
+ # We need to initialize the trackers we use, and also store our configuration.
+ # The trackers initializes automatically on the main process.
+ if accelerator.is_main_process:
+ run_time = datetime.now().strftime("%Y%m%d-%H%M")
+ accelerator.init_trackers(
+ cfg.exp_name,
+ init_kwargs={"mlflow": {"run_name": run_time}},
+ )
+ # dump config file
+ mlflow.log_dict(OmegaConf.to_container(cfg), "config.yaml")
+
+ # Train!
+ total_batch_size = (
+ cfg.data.train_bs
+ * accelerator.num_processes
+ * cfg.solver.gradient_accumulation_steps
+ )
+
+ logger.info("***** Running training *****")
+ logger.info(f" Num examples = {len(train_dataset)}")
+ logger.info(f" Num Epochs = {num_train_epochs}")
+ logger.info(f" Instantaneous batch size per device = {cfg.data.train_bs}")
+ logger.info(
+ f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"
+ )
+ logger.info(
+ f" Gradient Accumulation steps = {cfg.solver.gradient_accumulation_steps}"
+ )
+ logger.info(f" Total optimization steps = {cfg.solver.max_train_steps}")
+ global_step = 0
+ first_epoch = 0
+
+ # Potentially load in the weights and states from a previous save
+ if cfg.resume_from_checkpoint:
+ if cfg.resume_from_checkpoint != "latest":
+ resume_dir = cfg.resume_from_checkpoint
+ else:
+ resume_dir = save_dir
+ # Get the most recent checkpoint
+ dirs = os.listdir(resume_dir)
+ dirs = [d for d in dirs if d.startswith("checkpoint")]
+ dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))
+ path = dirs[-1]
+ accelerator.load_state(os.path.join(resume_dir, path))
+ accelerator.print(f"Resuming from checkpoint {path}")
+ global_step = int(path.split("-")[1])
+
+ first_epoch = global_step // num_update_steps_per_epoch
+
+ # Only show the progress bar once on each machine.
+ progress_bar = tqdm(
+ range(global_step, cfg.solver.max_train_steps),
+ disable=not accelerator.is_local_main_process,
+ )
+ progress_bar.set_description("Steps")
+
+ temporal_window_size = cfg.temporal_window_size
+ temporal_adaptive_step = cfg.temporal_adaptive_step
+
+ windows = cfg.data.n_sample_frames // temporal_window_size
+
+ for epoch in range(first_epoch, num_train_epochs):
+ train_loss = 0.0
+ train_dis_loss = 0.0
+ for step, batch in enumerate(train_dataloader):
+ # Convert videos to latent space
+ pixel_values = batch["img"].to(weight_dtype) # b,c,h,w
+ video_length = pixel_values.shape[1]
+
+ tgt_face = batch["tgt_face"]
+ tgt_pose = batch["tgt_pose"]
+
+ tgt_face = rearrange(tgt_face, "b f c h w -> (b f) c h w")
+ tgt_pose = rearrange(tgt_pose, "b f c h w -> (b f) c h w")
+
+ face_mask = batch['face_mask']
+ local_mask = batch['local_mask']
+
+ face_mask = rearrange(face_mask, "b f c h w -> b c f h w")
+ local_mask = rearrange(local_mask, "b f c h w -> b c f h w")
+
+ facial_mask_pix = face_mask.repeat(1,3,1,1,1) + local_mask.repeat(1,3,1,1,1)
+
+ pixel_values = rearrange(pixel_values, "b f c h w -> (b f) c h w")
+ with torch.no_grad():
+ latents = []
+ for i in range(0, pixel_values.shape[0], cfg.data.max_batch_frames):
+ batch_latents = vae.encode(
+ pixel_values[i : i + cfg.data.max_batch_frames]
+ ).latent_dist.sample()
+ latents.append(batch_latents)
+ latents = torch.cat(latents, dim=0)
+ latents = latents * 0.18215 # (b f) c h w
+ latents = rearrange(latents, "(b f) c h w -> b c f h w", f=video_length)
+
+ pixel_values = rearrange(pixel_values, "(b f) c h w -> b c f h w", f=video_length)
+ noise = torch.randn_like(latents)
+
+ bsz = latents.shape[0] # (b f)
+ timesteps_list = cfg.timesteps_list
+ timesteps = torch.tensor(timesteps_list[::-1], device=latents.device)
+ timesteps = timesteps.repeat_interleave(temporal_window_size, dim=0).long()
+
+ uncond_fwd = random.random() < cfg.uncond_ratio
+ clip_image_list = []
+ ref_image_list = []
+ for batch_idx, (ref_img, clip_img) in enumerate(
+ zip(
+ batch["ref_img"],
+ batch["clip_image"],
+ )
+ ):
+ clip_image_list.append(clip_img)
+ ref_image_list.append(ref_img)
+
+ with torch.no_grad():
+ ref_img = torch.stack(ref_image_list, dim=0).to(
+ dtype=vae.dtype, device=vae.device
+ )
+ ref_image_latents = vae.encode(
+ ref_img
+ ).latent_dist.sample() # (bs, d, 64, 64)
+ ref_image_latents = ref_image_latents * 0.18215
+
+ clip_img = torch.stack(clip_image_list, dim=0).to(
+ dtype=image_enc.dtype, device=image_enc.device
+ )
+ clip_image_embeds = image_enc(
+ clip_img.to("cuda", dtype=weight_dtype)
+ ).image_embeds
+ image_prompt_embeds = clip_image_embeds.unsqueeze(1) # (bs, 1, d)
+
+ init_frames = (temporal_adaptive_step - 1) * temporal_window_size
+ # add noise
+ noisy_latents = train_noise_scheduler.add_noise(
+ latents[:,:,:init_frames], noise[:,:,:init_frames], timesteps[:init_frames]
+ ) # (b f) c h w
+
+ noisy_latents_all = torch.cat([noisy_latents, noise[:,:,init_frames:]], dim=2)
+
+ mot_flag = random.random() < 0.1
+ pose_flag = random.random() < 0.1
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+
+ with torch.no_grad():
+ keypoints = pose_encoder(tgt_pose)
+ tgt_pose = draw_keypoints(keypoints).to('cuda')
+ tgt_pose = rearrange(tgt_pose, '(b f) c h w -> b c f h w', b=bsz)
+ pose_emb = pose_guider(tgt_pose.to(pose_guider.dtype))
+ motion_emb = motion_encoder(tgt_face)
+
+ if mot_flag:
+ motion_emb = torch.zeros_like(motion_emb)
+ if pose_flag:
+ pose_emb = torch.zeros_like(pose_emb)
+
+ # reference feature
+ if not uncond_fwd:
+ reference_unet(
+ ref_image_latents,
+ torch.zeros_like(timesteps[:bsz]),
+ encoder_hidden_states=image_prompt_embeds,
+ return_dict=False,
+ )
+ reference_control_reader.update(reference_control_writer, drop_ratio=0.3)
+
+ for i in range(windows - temporal_adaptive_step + 1):
+ l = i * temporal_window_size
+ r = (i + temporal_adaptive_step) * temporal_window_size
+
+ noisy_latents = noisy_latents_all[:,:,l:r,:,:].detach()
+ motion = motion_emb[l:r]
+ pose = pose_emb[:,:,l:r]
+
+ if i % (temporal_adaptive_step - 1) != 0:
+ with torch.no_grad():
+ model_pred = net(
+ noisy_latents,
+ timesteps,
+ image_prompt_embeds,
+ motion,
+ pose,
+ )
+
+ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)
+ noise = torch.randn_like(model_pred[:,:,temporal_window_size:])
+ noisy_latents = train_noise_scheduler.add_noise(
+ latents_pred[:,:,temporal_window_size:], noise, timesteps[:-temporal_window_size]
+ )
+ mid_latents = torch.cat([latents_pred[:,:,:temporal_window_size], noisy_latents], dim=2)
+ noisy_latents_all[:,:,l:r,:,:] = mid_latents.detach()
+ else:
+ with accelerator.accumulate(net):
+ model_pred = net(
+ noisy_latents,
+ timesteps,
+ image_prompt_embeds,
+ motion,
+ pose,
+ )
+
+ discriminator.requires_grad_(False)
+
+ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)
+ image_pred = decode_latents(vae, latents_pred)
+
+ pixel_values_tgt = pixel_values[:, :, l:r,:,:]
+
+ if epoch == 0 and step == 0:
+ if accelerator.is_main_process:
+ save_image((ref_img[:1] + 1) / 2, "image_ref.png")
+ save_image((pixel_values_tgt[:,:,0] + 1) / 2, "image_tgt.png")
+ save_image((image_pred[:,:,0] + 1) / 2, "image_pred.png")
+ facial_mask = facial_mask_pix[:, :, l:r, :, :]
+ loss = F.mse_loss(image_pred.float(), pixel_values_tgt.float(), reduction="mean")
+ loss += F.mse_loss(image_pred.float() * facial_mask, pixel_values_tgt.float() * facial_mask, reduction="mean")
+ image_pred = rearrange(image_pred, 'b c f h w -> (b f) c h w')
+ pixel_values_tgt = rearrange(pixel_values_tgt, 'b c f h w -> (b f) c h w')
+ loss_lpips = net_lpips(image_pred.float(), pixel_values_tgt.float()).mean() # 输入应该是[-1,1]
+
+ adv_loss = discriminator(image_pred.float(), pixel_values_tgt.float(), timesteps)
+ loss = loss + loss_lpips * 2.0 + adv_loss * 0.05
+
+ noise = torch.randn_like(model_pred[:,:,temporal_window_size:])
+ noisy_latents = train_noise_scheduler.add_noise(
+ latents_pred[:,:,temporal_window_size:], noise, timesteps[:-temporal_window_size]
+ )
+ mid_latents = torch.cat([latents_pred[:,:,:temporal_window_size], noisy_latents], dim=2)
+ noisy_latents_all[:,:,l:r,:,:] = mid_latents
+
+ # Gather the losses across all processes for logging (if we use distributed training).
+ avg_loss = accelerator.gather(loss.repeat(cfg.data.train_bs)).mean()
+ train_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps
+
+ # Backpropagate
+ accelerator.backward(loss)
+ if accelerator.sync_gradients:
+ accelerator.clip_grad_norm_(
+ trainable_params,
+ cfg.solver.max_grad_norm,
+ )
+ optimizer.step()
+ lr_scheduler.step()
+ optimizer.zero_grad()
+
+ if global_step > 50: # This value may not be optimal
+ discriminator.requires_grad_(True)
+ dis_loss = discriminator(image_pred.float().detach(), pixel_values_tgt.float(), timesteps, gen=False)
+ avg_loss = accelerator.gather(dis_loss.repeat(cfg.data.train_bs)).mean()
+ train_dis_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps
+
+ accelerator.backward(dis_loss)
+ if accelerator.sync_gradients:
+ accelerator.clip_grad_norm_(
+ discriminator.parameters(),
+ cfg.solver.max_grad_norm,
+ )
+ dis_optimizer.step()
+ dis_optimizer.zero_grad(set_to_none=True)
+ else:
+ dis_loss = torch.tensor(0.0)
+
+ if accelerator.sync_gradients:
+ reference_control_reader.clear()
+ reference_control_writer.clear()
+ progress_bar.update(1)
+ global_step += 1
+ accelerator.log({"train_loss": train_loss, "train_dis_loss": train_dis_loss}, step=global_step)
+ train_loss = 0.0
+ train_dis_loss = 0.0
+ if global_step % cfg.checkpointing_steps == 0:
+ if accelerator.is_main_process:
+ save_path = os.path.join(save_dir, f"checkpoint-{global_step}")
+ delete_additional_ckpt(save_dir, 1)
+ accelerator.save_state(save_path)
+
+ if global_step % cfg.val.validation_steps == 0 or global_step == 1:
+ if accelerator.is_main_process:
+ generator = torch.Generator(device=accelerator.device)
+ generator.manual_seed(cfg.seed)
+
+ sample_dicts = log_validation(
+ vae=vae,
+ image_enc=image_enc,
+ net=net,
+ scheduler=val_noise_scheduler,
+ accelerator=accelerator,
+ width=cfg.data.train_width,
+ height=cfg.data.train_height,
+ face_mesh=face_mesh,
+ reference_unet=reference_unet,
+ generator=generator,
+ pose_encoder=pose_encoder,
+ pose_guider=pose_guider,
+ motion_encoder=motion_encoder,
+ timesteps_list=timesteps_list,
+ )
+
+ reference_control_writer = ReferenceAttentionControl(
+ reference_unet,
+ do_classifier_free_guidance=False,
+ mode="write",
+ fusion_blocks="full",
+ )
+ reference_control_reader = ReferenceAttentionControl(
+ denoising_unet,
+ do_classifier_free_guidance=False,
+ mode="read",
+ fusion_blocks="full",
+ )
+
+ for sample_id, sample_dict in enumerate(sample_dicts):
+ sample_name = sample_dict["name"]
+ vid = sample_dict["vid"]
+ with TemporaryDirectory() as temp_dir:
+ out_file = f"{temp_dir}/{global_step:06d}-{sample_name}.mp4"
+ save_videos_grid(vid, out_file, n_rows=4, fps=25)
+ mlflow.log_artifact(out_file)
+
+ logs = {
+ "loss": loss.detach().item(),
+ "dis": dis_loss.detach().item(),
+ "adv": adv_loss.detach().item(),
+ "lr": lr_scheduler.get_last_lr()[0],
+ }
+ progress_bar.set_postfix(**logs)
+
+ # save model after each epoch
+ if global_step % cfg.checkpointing_steps == 0 and accelerator.is_main_process:
+ unwrap_net = accelerator.unwrap_model(net)
+ unwrap_discriminator = accelerator.unwrap_model(discriminator)
+ save_checkpoint(
+ unwrap_net.denoising_unet,
+ save_dir,
+ "temporal_module",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint_dis(
+ unwrap_discriminator,
+ save_dir,
+ "discriminator",
+ global_step,
+ total_limit=2,
+ )
+
+ if global_step >= cfg.solver.max_train_steps:
+ break
+
+ unwrap_net = accelerator.unwrap_model(net)
+ unwrap_discriminator = accelerator.unwrap_model(discriminator)
+ save_checkpoint(
+ unwrap_net.denoising_unet,
+ save_dir,
+ "temporal_module",
+ global_step,
+ total_limit=2,
+ )
+ save_checkpoint_dis(
+ unwrap_discriminator,
+ save_dir,
+ "discriminator",
+ global_step,
+ total_limit=2,
+ )
+
+ # Create the pipeline using the trained modules and save it.
+ accelerator.wait_for_everyone()
+ accelerator.end_training()
+
+
+def save_checkpoint(model, save_dir, prefix, ckpt_num, total_limit=None):
+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")
+
+ if total_limit is not None:
+ checkpoints = os.listdir(save_dir)
+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]
+ checkpoints = sorted(
+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])
+ )
+
+ if len(checkpoints) >= total_limit:
+ num_to_remove = len(checkpoints) - total_limit + 1
+ removing_checkpoints = checkpoints[0:num_to_remove]
+ logger.info(
+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
+ )
+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")
+
+ for removing_checkpoint in removing_checkpoints:
+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)
+ os.remove(removing_checkpoint)
+
+ mm_state_dict = OrderedDict()
+ state_dict = model.state_dict()
+ for key in state_dict:
+ if "temporal_modules" in key:
+ mm_state_dict[key] = state_dict[key]
+
+ torch.save(mm_state_dict, save_path)
+
+def save_checkpoint_dis(model, save_dir, prefix, ckpt_num, total_limit=None):
+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")
+
+ if total_limit is not None:
+ checkpoints = os.listdir(save_dir)
+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]
+ checkpoints = sorted(
+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])
+ )
+
+ if len(checkpoints) >= total_limit:
+ num_to_remove = len(checkpoints) - total_limit + 1
+ removing_checkpoints = checkpoints[0:num_to_remove]
+ logger.info(
+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
+ )
+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")
+
+ for removing_checkpoint in removing_checkpoints:
+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)
+ os.remove(removing_checkpoint)
+
+ state_dict = model.state_dict()
+ torch.save(state_dict, save_path)
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--config", type=str, default="./configs/train/personalive_stage3.yaml")
+ args = parser.parse_args()
+
+ if args.config[-5:] == ".yaml":
+ config = OmegaConf.load(args.config)
+ elif args.config[-3:] == ".py":
+ config = import_filename(args.config).cfg
+ else:
+ raise ValueError("Do not support this format config file")
+ main(config)