[CVPR 2026] PersonaLive! : Expressive Portrait Image Animation for Live Streaming
Upload training related code
Zhiyuan Lifbaa3ddd75ea
diff --git a/README.md b/README.mdindex 8b819fc..f59168c 100644--- a/README.md+++ b/README.md@@ -164,6 +164,130 @@ Then open `http://0.0.0.0:7860` in your browser. (*If `http://0.0.0.0:7860` does1. Lower the "Driving FPS" setting in the WebUI to reduce the computational workload.2. You can increase the multiplier (e.g., set to `num_frames_needed * 4` or higher) to better match your device's inference speed. https://github.com/GVCLab/PersonaLive/blob/6953d1a8b409f360a3ee1d7325093622b29f1e22/webcam/util.py#L73+## 🚄 Model Training++PersonaLive training is organized into three stages. Approximate training time on 8x H100 with default configs: Stage 1 ~13h, Stage 2 ~15h, Stage 3 ~20h.++### 1️⃣ Environment setup+Install base dependencies first (see installation section), then install training-only packages:++```bash+pip install -r requirements_train.txt+```++If you use multi-GPU or multi-node training, configure Accelerate once before launching training:++```bash+accelerate config+```++### 2️⃣ Data preparation+Your dataset should contain a `videos` directory and a matching `boxes` directory:++```text+Datasets+├── VFHQ+│ ├── videos+│ │ ├── example1.mp4+│ │ ├── example2.mp4+│ │ └── ...+│ └── boxes+│ ├── example1.pt+│ ├── example2.pt+│ └── ...+└── ...+```++Preprocessing example:++```bash+# 1) Extract face / eye / mouth boxes from each frame+python tools/get_boxes.py --video_dir ./Datasets/VFHQ/videos --save_dir ./Datasets/VFHQ/boxes --workers 8++# 2) Generate meta json: [{"video_path": ".../videos/xxx.mp4"}, ...]+python tools/extract_meta_info.py --root_path ./Datasets/VFHQ --dataset_name VFHQ+```++Then set `data.meta_paths` in each training config:++```yaml+data:+ meta_paths:+ - "./data/VFHQ_meta.json"+ - "./data/OtherDataset_meta.json"+```++### 3️⃣ Download weights+Download the training initialization weights: [X-NeMo](https://drive.google.com/drive/folders/1RdjBYYbstO7SOchDg7oimoAwu03g_-mI), [pose_guider](https://drive.google.com/drive/folders/1F6lyYfC5RijdS2SmvTct_8L9HZweV4nF?usp=sharing), and [stylegan2_discriminator](https://drive.google.com/drive/folders/1F6lyYfC5RijdS2SmvTct_8L9HZweV4nF?usp=sharing).++```text+pretrained_weights+├── xnemo+│ ├── xnemo_motion_encoder.pth+│ ├── xnemo_denoising_unet.pth+│ ├── xnemo_reference_unet.pth+│ └── xnemo_temporal_module.pth+├── sd-vae-ft-mse+│ ├── diffusion_pytorch_model.bin+│ └── config.json+├── sd-image-variations-diffusers+│ ├── image_encoder+│ │ ├── pytorch_model.bin+│ │ └── config.json+│ ├── unet+│ │ ├── diffusion_pytorch_model.bin+│ │ └── config.json+│ └── model_index.json+├── pose_guider.pth+└── stylegan2_discriminator_ffhq512.pth+```++### 4️⃣ Training workflow++#### Stage 1: Image-level warm-up+Run:+```bash+accelerate launch train_stage1.py --config ./configs/train/personalive_stage1.yaml+```++Default output folder: `./exp_output/personalive_stage1/`++#### Stage 2: Image-level adversarial refinement++Update `configs/train/personalive_stage2.yaml` to point to Stage 1 outputs:+```bash+motion_encoder_path: './exp_output/personalive_stage1/motion_encoder-xxxxx.pth'+denoising_unet_path: './exp_output/personalive_stage1/denoising_unet-xxxxx.pth'+reference_unet_path: './exp_output/personalive_stage1/reference_unet-xxxxx.pth'+pose_guider_path: './exp_output/personalive_stage1/pose_guider-xxxxx.pth'+```++Run:+```bash+accelerate launch train_stage2.py --config ./configs/train/personalive_stage2.yaml+```++Default output folder: `./exp_output/personalive_stage2/`++#### Stage 3: Temporal module fine-tuning for streaming++Update `configs/train/personalive_stage3.yaml` to point to Stage 2 outputs:++```bash+motion_encoder_path: './exp_output/personalive_stage2/motion_encoder-xxxxx.pth'+denoising_unet_path: './exp_output/personalive_stage2/denoising_unet-xxxxx.pth'+reference_unet_path: './exp_output/personalive_stage2/reference_unet-xxxxx.pth'+pose_guider_path: './exp_output/personalive_stage2/pose_guider-xxxxx.pth'+discriminator_path: './exp_output/personalive_stage2/discriminator-xxxxx.pth'+```++Run:+```bash+accelerate launch train_stage3.py --config ./configs/train/personalive_stage3.yaml+```++Default output folder: `./exp_output/personalive_stage3/`+## 📚 Community ContributionSpecial thanks to the community for providing helpful setups! 🥂diff --git a/configs/inference/inference_stage1&2.yaml b/configs/inference/inference_stage1&2.yamlnew file mode 100644index 0000000..e351e5a--- /dev/null+++ b/configs/inference/inference_stage1&2.yaml@@ -0,0 +1,47 @@+unet_additional_kwargs:+ use_inflated_groupnorm: true+ unet_use_cross_frame_attention: false+ unet_use_temporal_attention: false+ use_motion_module: true+ motion_module_resolutions:+ - 1+ - 2+ - 4+ - 8+ motion_module_mid_block: true+ motion_module_decoder_only: false+ motion_module_type: Vanilla+ motion_module_kwargs:+ num_attention_heads: 8+ num_transformer_block: 1+ cross_attention_dim: 16+ attention_block_types:+ - Spatial_Cross+ - Spatial_Cross+ temporal_position_encoding: false+ temporal_position_encoding_max_len: 32+ temporal_attention_dim_div: 1++ use_temporal_module: false+ temporal_module_type: Vanilla+ temporal_module_kwargs:+ num_attention_heads: 8+ num_transformer_block: 1+ attention_block_types:+ - Temporal_Self+ - Temporal_Self+ temporal_position_encoding: true+ temporal_position_encoding_max_len: 32+ temporal_attention_dim_div: 1+++noise_scheduler_kwargs:+ beta_start: 0.00085+ beta_end: 0.02+ beta_schedule: "scaled_linear"+ clip_sample: false+ steps_offset: 1+ prediction_type: "epsilon"+ timestep_spacing: "trailing"++sampler: DDIM\ No newline at end of filediff --git a/configs/train/personalive_stage1.yaml b/configs/train/personalive_stage1.yamlnew file mode 100644index 0000000..83bd573--- /dev/null+++ b/configs/train/personalive_stage1.yaml@@ -0,0 +1,53 @@+data:+ train_bs: 8+ train_width: 512+ train_height: 512+ meta_paths:+ - "./data/VFHQ_example.json"+ sample_margin: 30++solver:+ gradient_accumulation_steps: 1+ mixed_precision: 'fp16'+ enable_xformers_memory_efficient_attention: True+ gradient_checkpointing: False+ max_train_steps: 50000+ max_grad_norm: 1.0+ # lr+ learning_rate: 1e-5+ scale_lr: False+ lr_warmup_steps: 1+ lr_scheduler: 'constant'++ # optimizer+ use_8bit_adam: False+ adam_beta1: 0.9+ adam_beta2: 0.999+ adam_weight_decay: 1.0e-2+ adam_epsilon: 1.0e-8++val:+ validation_steps: 1000++base_model_path: './pretrained_weights/sd-image-variations-diffusers'+vae_model_path: './pretrained_weights/sd-vae-ft-mse'+image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'+pose_encoder_path: './pretrained_weights/liveportrait/base_models/motion_extractor.pth'++motion_encoder_path: './pretrained_weights/xnemo/xnemo_motion_encoder.pth'+denoising_unet_path: './pretrained_weights/xnemo/xnemo_denoising_unet.pth'+reference_unet_path: './pretrained_weights/xnemo/xnemo_reference_unet.pth'+pose_guider_path: './pretrained_weights/pose_guider.pth'++weight_dtype: 'fp16' # [fp16, fp32]+uncond_ratio: 0.1+noise_offset: 0.05+window_size: 10+snr_gamma: 5.0++seed: 12580+resume_from_checkpoint: ''+checkpointing_steps: 10000+save_model_epoch_interval: 5+exp_name: 'personalive_stage1'+output_dir: './exp_output'\ No newline at end of filediff --git a/configs/train/personalive_stage2.yaml b/configs/train/personalive_stage2.yamlnew file mode 100644index 0000000..84f55f9--- /dev/null+++ b/configs/train/personalive_stage2.yaml@@ -0,0 +1,56 @@+data:+ train_bs: 4+ train_width: 512+ train_height: 512+ meta_paths:+ - "./data/VFHQ_example.json"+ sample_margin: 30++solver:+ gradient_accumulation_steps: 1+ mixed_precision: 'fp16'+ enable_xformers_memory_efficient_attention: True+ gradient_checkpointing: False+ max_train_steps: 50000+ max_grad_norm: 1.0+ # lr+ learning_rate: 1e-5+ scale_lr: False+ lr_warmup_steps: 1+ lr_scheduler: 'constant'++ # optimizer+ use_8bit_adam: True+ adam_beta1: 0.9+ adam_beta2: 0.999+ adam_weight_decay: 1.0e-2+ adam_epsilon: 1.0e-8++val:+ validation_steps: 1000++base_model_path: './pretrained_weights/sd-image-variations-diffusers'+vae_model_path: './pretrained_weights/sd-vae-ft-mse'+image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'+pose_encoder_path: './pretrained_weights/liveportrait/base_models/motion_extractor.pth'++motion_encoder_path: './exp_output/personalive_stage1/motion_encoder-xxxxx.pth'+denoising_unet_path: './exp_output/personalive_stage1/denoising_unet-xxxxx.pth'+reference_unet_path: './exp_output/personalive_stage1/reference_unet-xxxxx.pth'+pose_guider_path: './exp_output/personalive_stage1/pose_guider-xxxxx.pth'++discriminator_path: './pretrained_weights/stylegan2_discriminator_ffhq512.pth'++weight_dtype: 'fp16' # [fp16, fp32]+uncond_ratio: 0.1+noise_offset: 0.05+window_size: 10+snr_gamma: 5.0+timesteps_list: [999, 666, 333, 0]++seed: 12580+resume_from_checkpoint: ''+checkpointing_steps: 10000+save_model_epoch_interval: 3+exp_name: 'personalive_stage2'+output_dir: './exp_output'\ No newline at end of filediff --git a/configs/train/personalive_stage3.yaml b/configs/train/personalive_stage3.yamlnew file mode 100644index 0000000..3fa7583--- /dev/null+++ b/configs/train/personalive_stage3.yaml@@ -0,0 +1,62 @@+data:+ train_bs: 1+ train_width: 512+ train_height: 512+ meta_paths:+ - "./data/VFHQ_example.json"+ sample_rate: 4+ n_sample_frames: 40+ max_batch_frames: 10++solver:+ gradient_accumulation_steps: 1+ mixed_precision: 'fp16'+ enable_xformers_memory_efficient_attention: True+ gradient_checkpointing: False+ max_train_steps: 10000+ max_grad_norm: 1.0+ # lr+ learning_rate: 1e-5+ scale_lr: False+ lr_warmup_steps: 1+ lr_scheduler: 'constant'++ # optimizer+ use_8bit_adam: True+ adam_beta1: 0.9+ adam_beta2: 0.999+ adam_weight_decay: 1.0e-2+ adam_epsilon: 1.0e-8++val:+ validation_steps: 200++temporal_window_size: 4+temporal_adaptive_step: 4++base_model_path: './pretrained_weights/sd-image-variations-diffusers'+vae_model_path: './pretrained_weights/sd-vae-ft-mse'+image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'+pose_encoder_path: './pretrained_weights/liveportrait/base_models/motion_extractor.pth'++motion_encoder_path: './exp_output/personalive_stage2/motion_encoder-xxxxx.pth'+denoising_unet_path: './exp_output/personalive_stage2/denoising_unet-xxxxx.pth'+reference_unet_path: './exp_output/personalive_stage2/reference_unet-xxxxx.pth'+pose_guider_path: './exp_output/personalive_stage2/pose_guider-xxxxx.pth'+discriminator_path: './exp_output/personalive_stage2/discriminator-xxxxx.pth'++temporal_module_path: './pretrained_weights/xnemo/xnemo_temporal_module.pth'++weight_dtype: 'fp16' # [fp16, fp32]+uncond_ratio: 0.1+noise_offset: 0.05+window_size: 10+snr_gamma: 5.0+timesteps_list: [999, 666, 333, 0]++seed: 12580+resume_from_checkpoint: ''+checkpointing_steps: 1000+save_model_epoch_interval: 1+exp_name: 'personalive_stage3'+output_dir: './exp_output'\ No newline at end of filediff --git a/data/VFHQ_example.json b/data/VFHQ_example.jsonnew file mode 100644index 0000000..db2f208--- /dev/null+++ b/data/VFHQ_example.json@@ -0,0 +1,7 @@+[+{"video_path": "Datasets/videos/VFHQ/Clip+SWLlH7mCWs8+P2+C1+F5731-5850.mp4"},+{"video_path": "Datasets/videos/VFHQ/Clip+yVmVOdc9glA+P2+C2+F56876-56997.mp4"},+{"video_path": "Datasets/videos/VFHQ/Clip+tThJIOTpmP8+P0+C1+F115-418.mp4"},+{"video_path": "Datasets/videos/VFHQ/Clip+rWOlJ_tyNB8+P0+C1+F2870-3025.mp4"},+{"video_path": "Datasets/videos/VFHQ/Clip+0twLK4diLu8+P0+C1+F1094-1220.mp4"}+]\ No newline at end of filediff --git a/requirements_base.txt b/requirements_base.txtindex 1c72db4..6c20240 100644--- a/requirements_base.txt+++ b/requirements_base.txt@@ -6,12 +6,11 @@ decord==0.6.0diffusers==0.27.0einops==0.8.1fastapi==0.117.1-huggingface-hub==0.25.1+huggingface_hub==0.25.1mediapipe==0.10.11markdown2==2.5.4numpy==1.26.3omegaconf==2.3.0-opencv_contrib_python==4.11.0.86opencv_python_headless==4.10.0.84Pillow==11.3.0polygraphy==0.49.26diff --git a/requirements_train.txt b/requirements_train.txtnew file mode 100644index 0000000..34533be--- /dev/null+++ b/requirements_train.txt@@ -0,0 +1,10 @@+albumentations==1.4.12+albucore==0.0.16+bitsandbytes==0.44.1+lpips==0.1.4+mlflow==3.1.4+mlflow_skinny==3.1.4+numpy==1.26.3+requests==2.32.5+protobuf==3.20.3+setuptools==68.2.2\ No newline at end of filediff --git a/src/dataset/face_augmentor.py b/src/dataset/face_augmentor.pynew file mode 100644index 0000000..5842e63--- /dev/null+++ b/src/dataset/face_augmentor.py@@ -0,0 +1,46 @@+import random+import os+os.environ["NO_ALBUMENTATIONS_UPDATE"] = "1"+import numpy as np+import albumentations as A+import torchvision.transforms.functional as TF+import torch++class FaceAugmentor:+ def __init__(self):+ self.post_aug = A.Compose([+ A.ColorJitter(brightness=(0.3, 1.3), contrast=0.3, saturation=0.3, hue=0.3, p=1.0),+ A.PiecewiseAffine(scale=(0.02, 0.04), p=1.0),+ A.GaussNoise(p=1),+ ])++ def random_aspect_resize(self, img, flag=None, scale=None):+ img = torch.from_numpy(img).permute(2, 0, 1)+ # img: torch.Tensor [C,H,W]+ H, W = img.shape[-2:]+ if flag is None:+ flag = random.random()+ if scale is None:+ scale = random.uniform(1.0, 1.3)++ if flag < 0.5:+ scale_x = scale+ scale_y = 1.0+ else:+ scale_x = 1.0+ scale_y = scale++ new_W, new_H = int(W * scale_x), int(H * scale_y)+ img_resized = TF.resize(img, (new_H, new_W), antialias=True)+ # 中心裁剪/填充回原尺寸+ img_final = TF.center_crop(img_resized, (H, W))+ return img_final.permute(1,2,0).numpy() # [H,W,C]++ def __call__(self, img, random_size=True, flag=None, scale=None): # img: numpy RGB [H, W, 3]+ h, w = img.shape[:2]+ img_aug = img.copy()++ if random_size:+ img_aug = self.random_aspect_resize(img_aug, flag, scale)++ return self.post_aug(image=img_aug)["image"]\ No newline at end of filediff --git a/src/dataset/portrait_image.py b/src/dataset/portrait_image.pynew file mode 100644index 0000000..784e25b--- /dev/null+++ b/src/dataset/portrait_image.py@@ -0,0 +1,168 @@+import json+import os+import random+from typing import List++import numpy as np+import torch+import torchvision.transforms as transforms+from decord import VideoReader+from PIL import Image+from torch.utils.data import Dataset+from transformers import CLIPImageProcessor+from pathlib import Path+from .utils import (+ crop_square_containing_face_patch,+ get_bbox_param,+ get_mask,+)+from .face_augmentor import FaceAugmentor++class PortraitImageDataset(Dataset):+ def __init__(+ self,+ img_size,+ img_scale=(1.0, 1.0),+ img_ratio=(0.9, 1.0),+ data_meta_paths=["./data/fahsion_meta.json"],+ sample_margin=30,+ ):+ super().__init__()++ self.img_size = img_size+ self.img_scale = img_scale+ self.img_ratio = img_ratio+ self.sample_margin = sample_margin++ vid_meta = []+ for data_meta_path in data_meta_paths:+ vid_meta.extend(json.load(open(data_meta_path, "r")))+ self.vid_meta = vid_meta++ self.clip_image_processor = CLIPImageProcessor()++ self.transform = transforms.Compose(+ [+ transforms.Resize(+ self.img_size,+ interpolation=transforms.InterpolationMode.BILINEAR,+ ),+ transforms.ToTensor(),+ transforms.Normalize([0.5], [0.5]),+ ]+ )++ self.expression_transform = transforms.Compose(+ [+ transforms.Resize(+ (224, 224),+ interpolation=transforms.InterpolationMode.BILINEAR,+ ),+ transforms.ToTensor(),+ transforms.Normalize([0.5], [0.5]),+ ]+ )++ self.cond_transform = transforms.Compose(+ [+ transforms.Resize(+ (256, 256),+ interpolation=transforms.InterpolationMode.BILINEAR,+ ),+ transforms.ToTensor(),+ ]+ )++ self.mask_transform = transforms.Compose(+ [+ transforms.Resize(+ self.img_size,+ interpolation=transforms.InterpolationMode.NEAREST,+ ),+ transforms.ToTensor(),+ ]+ )+++ self.face_augmentor = FaceAugmentor()++ def augmentation(self, image, transform, state=None):+ if state is not None:+ torch.set_rng_state(state)+ return transform(image)++ def __getitem__(self, index):+ try:+ video_meta = self.vid_meta[index]+ video_path = video_meta["video_path"]+++ kps_path = video_path.replace("videos", "boxes").replace(".mp4", ".pt")+ video_reader = VideoReader(video_path)+ landmarks = torch.load(kps_path)++ video_length = min(len(video_reader), len(landmarks))++ margin = self.sample_margin+ ref_img_idx = random.randint(0, video_length - 1)+ if video_length > 2 * margin + 4:+ left = max(0, ref_img_idx - margin)+ right = min(video_length, ref_img_idx + margin)++ # 拼接两个不重叠的区间+ part = list(range(0, left)) + list(range(right, video_length))+ tgt_img_idx = random.choice(part)+ else:+ tgt_img_idx = random.randint(0, video_length - 1)++ ref_img = video_reader[ref_img_idx]+ tgt_img = video_reader[tgt_img_idx]+ ref_face_box = landmarks[ref_img_idx]['face']+ tgt_face_box = landmarks[tgt_img_idx]['face']+ tgt_left_eye_box = landmarks[tgt_img_idx]['left_eye']+ tgt_right_eye_box = landmarks[tgt_img_idx]['right_eye']+ tgt_mouth_box = landmarks[tgt_img_idx]['mouth']++ tgt_face_patch = crop_square_containing_face_patch(tgt_img.asnumpy(), tgt_face_box)+ bbox_params = torch.from_numpy(get_bbox_param(tgt_face_box, ref_face_box))++ face_mask, local_mask = get_mask(tgt_face_box, tgt_left_eye_box, tgt_right_eye_box, tgt_mouth_box, side=512)++ tgt_face_patch = self.face_augmentor(tgt_face_patch)++ ref_img_pil = Image.fromarray(ref_img.asnumpy())+ tgt_img_pil = Image.fromarray(tgt_img.asnumpy())+ tgt_face_pil = Image.fromarray(tgt_face_patch)+ face_mask = Image.fromarray(face_mask)+ local_mask = Image.fromarray(local_mask)++ state = torch.get_rng_state()+ tgt_img = self.augmentation(tgt_img_pil, self.transform, state=state)+ ref_img = self.augmentation(ref_img_pil, self.transform, state=state)+ tgt_face = self.augmentation(tgt_face_pil, self.expression_transform, state=state)+ tgt_pose = self.augmentation(tgt_img_pil, self.cond_transform, state=state)+ face_mask = self.augmentation(face_mask, self.mask_transform, state=state)+ local_mask = self.augmentation(local_mask, self.mask_transform, state=state)++ clip_image = self.clip_image_processor(+ images=ref_img_pil, return_tensors="pt"+ ).pixel_values[0]++ sample = dict(+ video_dir=video_path,+ img=tgt_img,+ ref_img=ref_img,+ tgt_face=tgt_face,+ tgt_pose=tgt_pose,+ bbox_params=bbox_params,+ clip_image=clip_image,+ face_mask=face_mask,+ local_mask=local_mask,+ )+ return sample+ except Exception as e:+ next_index = random.randint(0, len(self.vid_meta)-1)+ return self.__getitem__(next_index)++ def __len__(self):+ return len(self.vid_meta)\ No newline at end of filediff --git a/src/dataset/portrait_video.py b/src/dataset/portrait_video.pynew file mode 100644index 0000000..be0425b--- /dev/null+++ b/src/dataset/portrait_video.py@@ -0,0 +1,184 @@+import json+import os+import random+from typing import List++import numpy as np+import torch+import torchvision.transforms as transforms+from decord import VideoReader+from PIL import Image+from torch.utils.data import Dataset+from transformers import CLIPImageProcessor+from pathlib import Path+from .utils import (+ crop_square_containing_face_patch,+ get_bbox_param,+ get_mask,+)+import gc+from .face_augmentor import FaceAugmentor+import torchvision.transforms.functional as TF++class PortraitVideoDataset(Dataset):+ def __init__(+ self,+ sample_rate,+ n_sample_frames,+ img_size,+ img_scale=(1.0, 1.0),+ img_ratio=(0.9, 1.0),+ data_meta_paths=["./data/fahsion_meta.json"],+ ):+ super().__init__()++ self.img_size = img_size+ self.img_scale = img_scale+ self.img_ratio = img_ratio+ self.sample_rate = sample_rate+ self.n_sample_frames = n_sample_frames++ vid_meta = []+ for data_meta_path in data_meta_paths:+ vid_meta.extend(json.load(open(data_meta_path, "r")))+ self.vid_meta = vid_meta++ self.clip_image_processor = CLIPImageProcessor()++ self.transform = transforms.Compose(+ [+ transforms.Resize(+ self.img_size,+ interpolation=transforms.InterpolationMode.BILINEAR,+ ),+ transforms.ToTensor(),+ transforms.Normalize([0.5], [0.5]),+ ]+ )++ self.expression_transform = transforms.Compose(+ [+ transforms.Resize(+ (224, 224),+ interpolation=transforms.InterpolationMode.BILINEAR,+ ),+ transforms.ToTensor(),+ transforms.Normalize([0.5], [0.5]),+ ]+ )++ self.mask_transform = transforms.Compose(+ [+ transforms.Resize(+ self.img_size,+ interpolation=transforms.InterpolationMode.NEAREST,+ ),+ transforms.ToTensor(),+ ]+ )++ self.pose_transform = transforms.Compose(+ [+ transforms.Resize(+ (256, 256),+ interpolation=transforms.InterpolationMode.BILINEAR,+ ),+ transforms.ToTensor(),+ ]+ )++ self.face_augmentor = FaceAugmentor()++ def augmentation(self, images, transform, state=None):+ if state is not None:+ torch.set_rng_state(state)+ if isinstance(images, List):+ transformed_images = [transform(img) for img in images]+ ret_tensor = torch.stack(transformed_images, dim=0) # (f, c, h, w)+ else:+ ret_tensor = transform(images) # (c, h, w)+ return ret_tensor++ def __getitem__(self, index):+ try:+ video_meta = self.vid_meta[index]+ video_path = video_meta["video_path"]+++ kps_path = video_path.replace("videos", "boxes").replace(".mp4", ".pt")+ video_reader = VideoReader(video_path)+ landmarks = torch.load(kps_path)++ video_length = min(len(video_reader), len(landmarks))++ ref_img_idx = random.randint(0, video_length - 1)+ sample_rate = self.sample_rate+ clip_length = min(video_length, (self.n_sample_frames - 1) * sample_rate + 1)+ start_idx = random.randint(0, video_length - clip_length)+ batch_index = np.linspace(start_idx, start_idx + clip_length - 1, self.n_sample_frames, dtype=int).tolist()++ ref_img = video_reader[ref_img_idx]+ ref_face_box = landmarks[ref_img_idx]['face']++ tgt_image_list = []+ tgt_face_list = []+ tgt_face_mask_list = []+ tgt_local_mask_list = []+ bbox_params = []++ face_flag = random.random()+ face_scale = random.uniform(1.0, 1.2)++ for tgt_img_idx in batch_index:+ tgt_img = video_reader[tgt_img_idx]+ tgt_face_box = landmarks[tgt_img_idx]['face']+ tgt_left_eye_box = landmarks[tgt_img_idx]['left_eye']+ tgt_right_eye_box = landmarks[tgt_img_idx]['right_eye']+ tgt_mouth_box = landmarks[tgt_img_idx]['mouth']++ tgt_face_patch = crop_square_containing_face_patch(tgt_img.asnumpy(), tgt_face_box)+ face_mask, local_mask = get_mask(tgt_face_box, tgt_left_eye_box, tgt_right_eye_box, tgt_mouth_box, side=512)+ tgt_face_patch = self.face_augmentor(tgt_face_patch, flag=face_flag, scale=face_scale)++ bbox_param = torch.from_numpy(get_bbox_param(tgt_face_box, ref_face_box))+ bbox_params.append(bbox_param)++ tgt_image_list.append(Image.fromarray(tgt_img.asnumpy()))+ tgt_face_list.append(Image.fromarray(tgt_face_patch))+ tgt_face_mask_list.append(Image.fromarray(face_mask))+ tgt_local_mask_list.append(Image.fromarray(local_mask))++ bbox_params = torch.stack(bbox_params, dim=0)+ ref_img_pil = Image.fromarray(ref_img.asnumpy())++ state = torch.get_rng_state()+ tgt_img = self.augmentation(tgt_image_list, self.transform, state=state)+ ref_img = self.augmentation(ref_img_pil, self.transform, state=state)+ tgt_face = self.augmentation(tgt_face_list, self.expression_transform, state=state)+ face_mask = self.augmentation(tgt_face_mask_list, self.mask_transform, state=state)+ local_mask = self.augmentation(tgt_local_mask_list, self.mask_transform, state=state)++ tgt_pose = self.augmentation(tgt_image_list, self.pose_transform, state=state)++ clip_image = self.clip_image_processor(+ images=ref_img_pil, return_tensors="pt"+ ).pixel_values[0]++ sample = dict(+ video_dir=video_path,+ img=tgt_img,+ ref_img=ref_img,+ tgt_face=tgt_face,+ tgt_pose=tgt_pose,+ bbox_params=bbox_params,+ clip_image=clip_image,+ face_mask=face_mask,+ local_mask=local_mask,+ )+ return sample+ except Exception as e:+ next_index = random.randint(0, len(self.vid_meta)-1)+ return self.__getitem__(next_index)++ def __len__(self):+ return len(self.vid_meta)\ No newline at end of filediff --git a/src/dataset/utils.py b/src/dataset/utils.pynew file mode 100644index 0000000..ea16951--- /dev/null+++ b/src/dataset/utils.py@@ -0,0 +1,79 @@+import numpy as np+import cv2++def crop_square_containing_face_patch(image, face_box):+ h, w = image.shape[:2]+ x_min, y_min, x_max, y_max = face_box++ # 人脸中心和尺寸+ face_cx = (x_min + x_max) // 2+ face_cy = (y_min + y_max) // 2++ face_w = x_max - x_min+ face_h = y_max - y_min+ new_w = new_h = max(face_w, face_h)++ new_x_min = int(round(face_cx - new_w / 2))+ new_x_max = int(round(face_cx + new_w / 2))+ new_y_min = int(round(face_cy - new_h / 2))+ new_y_max = int(round(face_cy + new_h / 2))++ new_x_min = max(0, new_x_min)+ new_y_min = max(0, new_y_min)+ new_x_max = min(w, new_x_max)+ new_y_max = min(h, new_y_max)++ face_patch = image[new_y_min:new_y_max, new_x_min:new_x_max]++ return face_patch++def check_oob(bbox, size):+ left, top, right, bot = bbox+ return left < 0 or top < 0 or right > size[1] - 1 or bot > size[0] - 1++def scale_bb(bbox, scale, size):+ left, top, right, bot = bbox+ width = right - left+ height = bot - top+ length = max(width, height) * scale+ center_X = (left + right) * 0.5+ center_Y = (top + bot) * 0.5+ left, top, right, bot = [+ center_X - length / 2,+ center_Y - length / 2,+ center_X + length / 2,+ center_Y + length / 2,+ ]+ if check_oob((left, top, right, bot), size):+ return bbox+ else:+ return np.array([left, top, right, bot])++def get_bbox_param(bbox, ref_bbox):+ left, top, right, bot = bbox+ center = np.array([(bot + top) * 0.5, (left + right) * 0.5])+ length = max(right - left, bot - top)++ ref_left, ref_top, ref_right, ref_bot = ref_bbox+ ref_center = np.array([(ref_bot + ref_top) * 0.5, (ref_left + ref_right) * 0.5])+ ref_length = max(ref_right - ref_left, ref_bot - ref_top) # 最好保证bbox是正方形++ return np.asarray(+ ((center - ref_center) / ref_length).tolist() + [length / ref_length]+ )++def draw_box(box, mask, value=255):+ x1, y1, x2, y2 = box+ mask = cv2.rectangle(mask, (x1, y1), (x2, y2), value, -1)+ return mask++def get_mask(face_box, left_eye_box, right_eye_box, mouth_box, side):+ face_mask = np.zeros((side, side), dtype=np.uint8)+ local_mask = np.zeros((side, side), dtype=np.uint8)++ draw_box(face_box, face_mask)+ draw_box(left_eye_box, local_mask)+ draw_box(right_eye_box, local_mask)+ draw_box(mouth_box, local_mask)+ return face_mask, local_mask+diff --git a/src/models/motion_encoder/FAN_temporal_feature_extractor.py b/src/models/motion_encoder/FAN_temporal_feature_extractor.pyindex 01676dd..4baf1b6 100644--- a/src/models/motion_encoder/FAN_temporal_feature_extractor.py+++ b/src/models/motion_encoder/FAN_temporal_feature_extractor.py@@ -173,7 +173,7 @@ class TemporalTransformer3DModel(nn.Module):interpolation_scale=1,)# 在转onnx模型时,需要设置为False- self.pos_embed.pos_embed.requires_grad = False+ self.pos_embed.pos_embed.requires_grad = Trueself.proj_out = zero_module(self.proj_out) if zero_initialize else random_module(self.proj_out)def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, skip=True):diff --git a/src/models/motion_encoder/encoder.py b/src/models/motion_encoder/encoder.pyindex 55efbcd..c4d7a11 100644--- a/src/models/motion_encoder/encoder.py+++ b/src/models/motion_encoder/encoder.py@@ -34,8 +34,16 @@ class MotEncoder(ModelMixin):def forward(self, x):x = x.to(self.dtype)- latent = self.model(rearrange(x, "b c f h w -> (b f) c h w"))+ if x.ndim == 5:+ latent = self.model(rearrange(x, "b c f h w -> (b f) c h w"))+ else:+ latent = self.model(x)+ if self.out_bn is not None:+ latent = self.out_bn(latent.unsqueeze(-1)).squeeze(-1)+ if self.out_drop is not None:+ latent = self.out_drop(latent)latent = self.final_proj(latent)latent = rearrange(latent, "b (l c) -> b l c", c=self.out_ch) + self.pe- latent = rearrange(latent, "(b f) l c -> b f l c", f=x.shape[2])+ if x.ndim == 5:+ latent = rearrange(latent, "(b f) l c -> b f l c", f=x.shape[2])return latent\ No newline at end of filediff --git a/src/models/mutual_self_attention.py b/src/models/mutual_self_attention.pyindex 81781a7..bbb6868 100644--- a/src/models/mutual_self_attention.py+++ b/src/models/mutual_self_attention.py@@ -149,12 +149,6 @@ class ReferenceAttentionControl:**cross_attention_kwargs,)if MODE == "read":- kv_cache = norm_hidden_states.clone()- kv_cache = rearrange(- kv_cache, "(b t) l c -> b t l c", t=video_length- )- self.kv_cache = kv_cache[:,:2,:,:]-bank_fea = [rearrange(d.unsqueeze(1).repeat(1, video_length, 1, 1),@@ -326,9 +320,7 @@ class ReferenceAttentionControl:)module.bank = []- if(self.cache_kv):- module.kv_bank = None- module.kv_cache = None+ module.kv_bank = Nonemodule.attn_weight = float(i) / float(len(attn_modules))def update(self, writer, dtype=torch.float16, drop_ratio=0.):@@ -461,5 +453,4 @@ class ReferenceAttentionControl:for r in reader_attn_modules:r.bank.clear()if self.cache_kv:- r.kv_bank=None- r.kv_cache=None\ No newline at end of file+ r.kv_bank=None\ No newline at end of filediff --git a/src/models/resnet.py b/src/models/resnet.pyindex ea73144..326f0b1 100644--- a/src/models/resnet.py+++ b/src/models/resnet.py@@ -224,10 +224,10 @@ class ResnetBlock3D(nn.Module):if temb is not None:video_length = hidden_states.shape[2]temb = self.time_emb_proj(self.nonlinearity(temb))- temb = rearrange(temb, "(b f) c -> b c f", f=video_length)[:, :, :, None, None]-- # if temb is not None:- # temb = self.time_emb_proj(self.nonlinearity(temb))[:, :, None, None, None]+ if temb.shape[0] == hidden_states.shape[0] * video_length:+ temb = rearrange(temb, "(b f) c -> b c f", f=video_length)[:, :, :, None, None]+ else:+ temb = temb[:, :, None, None, None]if temb is not None and self.time_embedding_norm == "default":hidden_states = hidden_states + tembdiff --git a/src/pipelines/pipeline_pose2img.py b/src/pipelines/pipeline_pose2img.pynew file mode 100644index 0000000..07977ca--- /dev/null+++ b/src/pipelines/pipeline_pose2img.py@@ -0,0 +1,669 @@+# *************************************************************************+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates+# SPDX-License-Identifier: Apache-2.0+#+# This file has been modified by ByteDance Ltd. and/or its affiliates.+#+# Original file was released under Aniportrait, with the full license text+# available at https://github.com/Zejun-Yang/AniPortrait/blob/main/LICENSE.+#+# This modified file is released under the same license.+# *************************************************************************+import inspect+import math+from dataclasses import dataclass+from typing import Callable, List, Optional, Union+import random+import numpy as np+import torch+import torch.nn.functional as F+import torchvision.transforms as transforms+from diffusers import DiffusionPipeline+from diffusers import (+ AutoencoderKL,+ AutoencoderKLTemporalDecoder,+)+from diffusers.image_processor import VaeImageProcessor+from diffusers.schedulers import (+ DDIMScheduler,+ DPMSolverMultistepScheduler,+ EulerAncestralDiscreteScheduler,+ EulerDiscreteScheduler,+ LMSDiscreteScheduler,+ PNDMScheduler,+)+from diffusers.utils import BaseOutput, is_accelerate_available+from diffusers.utils.torch_utils import randn_tensor+from einops import rearrange+from tqdm import tqdm+from transformers import CLIPImageProcessor+from src.pipelines.context import get_context_scheduler+from src.pipelines.utils import get_tensor_interpolation_method+from src.models.mutual_self_attention import ReferenceAttentionControl+from src.utils.util import draw_keypoints++@dataclass+class Pose2VideoPipelineOutput(BaseOutput):+ videos: Union[torch.Tensor, np.ndarray]+++class Pose2ImagePipeline(DiffusionPipeline):+ _optional_components = []++ def __init__(+ self,+ vae,+ image_encoder,+ reference_unet,+ denoising_unet,+ motion_encoder,+ pose_encoder,+ pose_guider,+ scheduler: Union[+ DDIMScheduler,+ PNDMScheduler,+ LMSDiscreteScheduler,+ EulerDiscreteScheduler,+ EulerAncestralDiscreteScheduler,+ DPMSolverMultistepScheduler,+ ],+ ):+ super().__init__()++ self.register_modules(+ vae=vae,+ image_encoder=image_encoder,+ reference_unet=reference_unet,+ denoising_unet=denoising_unet,+ motion_encoder=motion_encoder,+ pose_encoder=pose_encoder,+ pose_guider=pose_guider,+ scheduler=scheduler,+ )+ self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1)+ self.clip_image_processor = CLIPImageProcessor()+ self.ref_image_processor = VaeImageProcessor(+ vae_scale_factor=self.vae_scale_factor, do_convert_rgb=True+ )+ self.cond_image_processor = VaeImageProcessor(+ vae_scale_factor=self.vae_scale_factor,+ do_convert_rgb=True,+ do_normalize=True,+ )++ def enable_vae_slicing(self):+ self.vae.enable_slicing()++ def disable_vae_slicing(self):+ self.vae.disable_slicing()++ def enable_sequential_cpu_offload(self, gpu_id=0):+ if is_accelerate_available():+ from accelerate import cpu_offload+ else:+ raise ImportError("Please install accelerate via `pip install accelerate`")++ device = torch.device(f"cuda:{gpu_id}")++ for cpu_offloaded_model in [self.unet, self.text_encoder, self.vae]:+ if cpu_offloaded_model is not None:+ cpu_offload(cpu_offloaded_model, device)++ @property+ def _execution_device(self):+ if self.device != torch.device("meta") or not hasattr(self.unet, "_hf_hook"):+ return self.device+ for module in self.unet.modules():+ if (+ hasattr(module, "_hf_hook")+ and hasattr(module._hf_hook, "execution_device")+ and module._hf_hook.execution_device is not None+ ):+ return torch.device(module._hf_hook.execution_device)+ return self.device++ def decode_latents(self, latents, decode_chunk_size=4):+ video_length = latents.shape[2]+ latents = 1 / 0.18215 * latents+ latents = rearrange(latents, "b c f h w -> (b f) c h w")+ # video = self.vae.decode(latents).sample+ video = []+ for frame_idx in range(0, latents.shape[0], decode_chunk_size):+ video.append(self.vae.decode(latents[frame_idx : frame_idx + decode_chunk_size]).sample)+ video = torch.cat(video)+ video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)+ video = (video / 2 + 0.5).clamp(0, 1)+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16+ video = video.cpu().float().numpy()+ return video++ def decode_latents_svd(+ self, latents: torch.FloatTensor, decode_chunk_size: int = 5+ ):+ video_length = latents.shape[2]+ latents = 1 / self.vae.config.scaling_factor * latents+ latents = rearrange(latents, "b c f h w -> (b f) c h w").to(self.vae.dtype)++ # decode decode_chunk_size frames at a time to avoid OOM+ frames = []+ for i in range(0, latents.shape[0], decode_chunk_size):+ num_frames_in = latents[i : i + decode_chunk_size].shape[0]+ frame = self.vae.decode(+ latents[i : i + decode_chunk_size], num_frames_in+ ).sample+ frames.append(frame)+ video = torch.cat(frames, dim=0)+ video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)+ video = (video / 2 + 0.5).clamp(0, 1)+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16+ video = video.cpu().float().numpy()+ return video++ def prepare_extra_step_kwargs(self, generator, eta):+ # prepare extra kwargs for the scheduler step, since not all schedulers have the same signature+ # eta (η) is only used with the DDIMScheduler, it will be ignored for other schedulers.+ # eta corresponds to η in DDIM paper: https://arxiv.org/abs/2010.02502+ # and should be between [0, 1]++ accepts_eta = "eta" in set(+ inspect.signature(self.scheduler.step).parameters.keys()+ )+ extra_step_kwargs = {}+ if accepts_eta:+ extra_step_kwargs["eta"] = eta++ # check if the scheduler accepts generator+ accepts_generator = "generator" in set(+ inspect.signature(self.scheduler.step).parameters.keys()+ )+ if accepts_generator:+ extra_step_kwargs["generator"] = generator+ return extra_step_kwargs++ def prepare_latents(+ self,+ batch_size,+ num_channels_latents,+ width,+ height,+ video_length,+ dtype,+ device,+ generator,+ latents=None,+ ):+ shape = (+ batch_size,+ num_channels_latents,+ video_length,+ height // self.vae_scale_factor,+ width // self.vae_scale_factor,+ )+ if isinstance(generator, list) and len(generator) != batch_size:+ raise ValueError(+ f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"+ f" size of {batch_size}. Make sure the batch size matches the length of the generators."+ )++ if latents is None:+ latents = randn_tensor(+ shape, generator=generator, device=device, dtype=dtype+ )+ else:+ latents = latents.to(device)++ # scale the initial noise by the standard deviation required by the scheduler+ latents = latents * self.scheduler.init_noise_sigma+ return latents++ def _encode_prompt(+ self,+ prompt,+ device,+ num_videos_per_prompt,+ do_classifier_free_guidance,+ negative_prompt,+ ):+ batch_size = len(prompt) if isinstance(prompt, list) else 1++ text_inputs = self.tokenizer(+ prompt,+ padding="max_length",+ max_length=self.tokenizer.model_max_length,+ truncation=True,+ return_tensors="pt",+ )+ text_input_ids = text_inputs.input_ids+ untruncated_ids = self.tokenizer(+ prompt, padding="longest", return_tensors="pt"+ ).input_ids++ if untruncated_ids.shape[-1] >= text_input_ids.shape[-1] and not torch.equal(+ text_input_ids, untruncated_ids+ ):+ removed_text = self.tokenizer.batch_decode(+ untruncated_ids[:, self.tokenizer.model_max_length - 1 : -1]+ )++ if (+ hasattr(self.text_encoder.config, "use_attention_mask")+ and self.text_encoder.config.use_attention_mask+ ):+ attention_mask = text_inputs.attention_mask.to(device)+ else:+ attention_mask = None++ text_embeddings = self.text_encoder(+ text_input_ids.to(device),+ attention_mask=attention_mask,+ )+ text_embeddings = text_embeddings[0]++ # duplicate text embeddings for each generation per prompt, using mps friendly method+ bs_embed, seq_len, _ = text_embeddings.shape+ text_embeddings = text_embeddings.repeat(1, num_videos_per_prompt, 1)+ text_embeddings = text_embeddings.view(+ bs_embed * num_videos_per_prompt, seq_len, -1+ )++ # get unconditional embeddings for classifier free guidance+ if do_classifier_free_guidance:+ uncond_tokens: List[str]+ if negative_prompt is None:+ uncond_tokens = [""] * batch_size+ elif type(prompt) is not type(negative_prompt):+ raise TypeError(+ f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !="+ f" {type(prompt)}."+ )+ elif isinstance(negative_prompt, str):+ uncond_tokens = [negative_prompt]+ elif batch_size != len(negative_prompt):+ raise ValueError(+ f"`negative_prompt`: {negative_prompt} has batch size {len(negative_prompt)}, but `prompt`:"+ f" {prompt} has batch size {batch_size}. Please make sure that passed `negative_prompt` matches"+ " the batch size of `prompt`."+ )+ else:+ uncond_tokens = negative_prompt++ max_length = text_input_ids.shape[-1]+ uncond_input = self.tokenizer(+ uncond_tokens,+ padding="max_length",+ max_length=max_length,+ truncation=True,+ return_tensors="pt",+ )++ if (+ hasattr(self.text_encoder.config, "use_attention_mask")+ and self.text_encoder.config.use_attention_mask+ ):+ attention_mask = uncond_input.attention_mask.to(device)+ else:+ attention_mask = None++ uncond_embeddings = self.text_encoder(+ uncond_input.input_ids.to(device),+ attention_mask=attention_mask,+ )+ uncond_embeddings = uncond_embeddings[0]++ # duplicate unconditional embeddings for each generation per prompt, using mps friendly method+ seq_len = uncond_embeddings.shape[1]+ uncond_embeddings = uncond_embeddings.repeat(1, num_videos_per_prompt, 1)+ uncond_embeddings = uncond_embeddings.view(+ batch_size * num_videos_per_prompt, seq_len, -1+ )++ # For classifier free guidance, we need to do two forward passes.+ # Here we concatenate the unconditional and text embeddings into a single batch+ # to avoid doing two forward passes+ text_embeddings = torch.cat([uncond_embeddings, text_embeddings])++ return text_embeddings++ def downgrade_input(self, init_latents, generator, device, dtype):+ mask = (+ torch.rand(+ *init_latents.shape,+ generator=generator,+ device=device,+ )+ >= 0.5+ ).to(dtype=dtype)++ blur = transforms.GaussianBlur(kernel_size=(9, 9), sigma=(18, 18))+ # first_frame_latents = init_latents[:, :, 0:1, :, :].clone()+ video_length = init_latents.shape[2]+ init_latents = rearrange(init_latents, "b c f h w -> (b f) c h w")+ init_latents_blur = blur(init_latents)+ init_latents = rearrange(init_latents, "(b f) c h w -> b c f h w", f=video_length)+ init_latents_blur = rearrange(init_latents_blur, "(b f) c h w -> b c f h w", f=video_length)+ # init_latents = init_latents * mask + init_latents_blur * (1 - mask)+ # init_latents[:, :, 0:1, :, :] = first_frame_latents++ return init_latents_blur++ def interpolate_latents(+ self, latents: torch.Tensor, interpolation_factor: int, device+ ):+ if interpolation_factor < 2:+ return latents++ new_latents = torch.zeros(+ (+ latents.shape[0],+ latents.shape[1],+ ((latents.shape[2] - 1) * interpolation_factor) + 1,+ latents.shape[3],+ latents.shape[4],+ ),+ device=latents.device,+ dtype=latents.dtype,+ )++ org_video_length = latents.shape[2]+ rate = [i / interpolation_factor for i in range(interpolation_factor)][1:]++ new_index = 0++ v0 = None+ v1 = None++ for i0, i1 in zip(range(org_video_length), range(org_video_length)[1:]):+ v0 = latents[:, :, i0, :, :]+ v1 = latents[:, :, i1, :, :]++ new_latents[:, :, new_index, :, :] = v0+ new_index += 1++ for f in rate:+ v = get_tensor_interpolation_method()(+ v0.to(device=device), v1.to(device=device), f+ )+ new_latents[:, :, new_index, :, :] = v.to(latents.device)+ new_index += 1++ new_latents[:, :, new_index, :, :] = v1+ new_index += 1++ return new_latents++ @torch.no_grad()+ def __call__(+ self,+ tgt_images,+ ref_image,+ exp_images,+ ref_exp_image,+ width,+ height,+ video_length,+ num_inference_steps,+ guidance_scale,+ timesteps_list=None,+ eta: float = 0.0,+ generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,+ output_type: Optional[str] = "tensor",+ return_dict: bool = True,+ callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None,+ callback_steps: Optional[int] = 1,+ mot_bbox_param=None,+ context_schedule="uniform",+ context_frames=10,+ context_stride=1,+ context_overlap=0,+ context_batch_size=1,+ **kwargs,+ ):+ # Default height and width to unet+ height = height or self.unet.config.sample_size * self.vae_scale_factor+ width = width or self.unet.config.sample_size * self.vae_scale_factor++ device = self._execution_device++ do_classifier_free_guidance = guidance_scale > 1.0++ # Prepare timesteps+ if timesteps_list is None:+ self.scheduler.set_timesteps(num_inference_steps, device=device)+ timesteps = self.scheduler.timesteps+ else:+ assert len(timesteps_list) == num_inference_steps, 'len(timesteps_list) != num_inference_steps'+ timesteps = torch.tensor(timesteps_list, device=device).long()+ self.scheduler.set_step_length(timesteps_list[0]-timesteps_list[1])++ batch_size = 1++ # Prepare clip image embeds+ clip_image = self.clip_image_processor.preprocess(+ ref_image.resize((224, 224)), return_tensors="pt"+ ).pixel_values+ clip_image_embeds = self.image_encoder(+ clip_image.to(device, dtype=self.image_encoder.dtype)+ ).image_embeds+ image_prompt_embeds = clip_image_embeds.unsqueeze(1)++ uncond_image_prompt_embeds = torch.zeros_like(image_prompt_embeds)++ if do_classifier_free_guidance:+ image_prompt_embeds = torch.cat(+ [uncond_image_prompt_embeds, image_prompt_embeds], dim=0+ )++ reference_control_writer = ReferenceAttentionControl(+ self.reference_unet,+ do_classifier_free_guidance=do_classifier_free_guidance,+ mode="write",+ batch_size=batch_size,+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ self.denoising_unet,+ do_classifier_free_guidance=do_classifier_free_guidance,+ mode="read",+ batch_size=batch_size,+ fusion_blocks="full",+ )++ # Prepare extra step kwargs.+ extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)++ # Prepare ref image latents+ ref_image_tensor = self.ref_image_processor.preprocess(+ ref_image, height=height, width=width+ ) # (bs, c, width, height)+ ref_image_tensor = ref_image_tensor.to(+ dtype=self.vae.dtype, device=self.vae.device+ )+ ref_image_latents = self.vae.encode(ref_image_tensor).latent_dist.mean+ ref_image_latents = ref_image_latents * 0.18215 # (b, 4, h, w)++ repeated_latents = ref_image_latents.unsqueeze(2).repeat(1, 1, video_length, 1, 1)+ noise = torch.randn_like(repeated_latents)+ noisy_latents_first = []+ for i in range(video_length):+ noisy_latents_first.append(self.scheduler.add_noise(repeated_latents[:, :, i:i+1, :, :], noise[:, :, i:i+1, :, :], timesteps[:1]))+ latents = torch.cat(noisy_latents_first, dim=2)++ ref_cond_tensor = self.cond_image_processor.preprocess(+ ref_image, height=256, width=256+ ).to(device=device, dtype=self.pose_encoder.dtype) # (1, c, h, w)+ ref_cond_tensor = ref_cond_tensor / 2 + 0.5 # to [0, 1]++ # Prepare a list of pose condition images+ exp_cond_tensor_list = []+ mot_bbox_param = []+ for tgt_image, exp_image in zip(tgt_images, exp_images):+ exp_cond_tensor = self.cond_image_processor.preprocess(+ exp_image, height=224, width=224+ ).transpose(0, 1) # (c, 1, h, w)+ exp_cond_tensor_list.append(exp_cond_tensor)++ tgt_cond_tensor = self.cond_image_processor.preprocess(+ tgt_image, height=256, width=256+ ).to(device=device, dtype=self.pose_encoder.dtype) # (1, c, h, w)+ tgt_cond_tensor = tgt_cond_tensor / 2 + 0.5 # to [0, 1]+ ref_motion_emb, motion_emb = self.pose_encoder.inference(ref_cond_tensor, tgt_cond_tensor) # [1, c]+ mot_bbox_param.append(motion_emb)+ mot_bbox_param = torch.cat(mot_bbox_param, dim=0) # (t, c)+ keypoints = draw_keypoints(mot_bbox_param).unsqueeze(2)+ keypoints = rearrange(keypoints, 'f c b h w -> b c f h w')+ keypoints = keypoints.to(device=device, dtype=self.pose_guider.dtype)+ pose_feas = self.pose_guider(keypoints) # (t, c)++ exp_cond_tensor = torch.cat(exp_cond_tensor_list, dim=1) # (c, t, h, w)++ exp_cond_tensor = exp_cond_tensor.unsqueeze(0) # (1, c, t, h, w)+ exp_cond_tensor = exp_cond_tensor.to(+ device=device, dtype=self.motion_encoder.dtype+ )+ context_scheduler = get_context_scheduler(context_schedule)++ if do_classifier_free_guidance:+ ref_exp_cond_tensor = self.cond_image_processor.preprocess(+ ref_exp_image, height=224, width=224+ ).to(device=device, dtype=self.motion_encoder.dtype)+ neg_motion_hidden_states = self.motion_encoder(ref_exp_cond_tensor) # [b, c]++ ref_keypoints = draw_keypoints(ref_motion_emb).unsqueeze(2)+ ref_keypoints = ref_keypoints.to(device=device, dtype=self.pose_guider.dtype)+ ref_pose_fea = self.pose_guider(ref_keypoints)++ # denoising loop+ num_warmup_steps = len(timesteps) - num_inference_steps * self.scheduler.order+ with self.progress_bar(total=num_inference_steps) as progress_bar:+ for i, t in enumerate(timesteps):+ if i == 0:+ self.reference_unet(+ ref_image_latents.repeat(+ (2 if do_classifier_free_guidance else 1), 1, 1, 1+ ),+ torch.zeros_like(t),+ encoder_hidden_states=image_prompt_embeds,+ return_dict=False,+ )+ reference_control_reader.update(reference_control_writer)++ noise_pred = torch.zeros(+ (+ latents.shape[0] * (2 if do_classifier_free_guidance else 1),+ *latents.shape[1:],+ ),+ device=latents.device,+ dtype=latents.dtype,+ )+ counter = torch.zeros(+ (1, 1, latents.shape[2], 1, 1),+ device=latents.device,+ dtype=latents.dtype,+ )++ offset = random.randint(0, context_frames-1)+ context_queue = list(+ context_scheduler(+ 0,+ num_inference_steps,+ latents.shape[2],+ context_frames,+ context_stride,+ context_overlap,+ True,+ offset+ )+ )++ num_context_batches = math.ceil(len(context_queue) / context_batch_size)+ global_context = []+ for i in range(num_context_batches):+ global_context.append(+ context_queue[+ i * context_batch_size : (i + 1) * context_batch_size+ ]+ )++ for context in global_context:+ # 3.1 expand the latents if we are doing classifier free guidance+ latent_model_input = (+ torch.cat([latents[:, :, c] for c in context])+ .to(device)+ .repeat(2 if do_classifier_free_guidance else 1, 1, 1, 1, 1)+ ).to(self.denoising_unet.dtype)+ latent_model_input = self.scheduler.scale_model_input(+ latent_model_input, t+ )++ pose_cond_input = torch.cat([exp_cond_tensor[:, :, c] for c in context])+ motion_hidden_states = self.motion_encoder(pose_cond_input) # [b, c]++ pose_fea = torch.cat([pose_feas[:,:,c] for c in context], dim=0)++ if do_classifier_free_guidance:+ motion_hidden_states = torch.cat(+ [+ neg_motion_hidden_states.unsqueeze(1).expand_as(motion_hidden_states),+ motion_hidden_states,+ ],+ dim=0,+ )+ pose_fea = torch.cat(+ [+ ref_pose_fea.expand_as(pose_fea),+ pose_fea,+ ],+ dim=0,+ )++ pred = self.denoising_unet(+ latent_model_input,+ t,+ encoder_hidden_states=[+ image_prompt_embeds,+ motion_hidden_states,+ ],+ pose_cond_fea=pose_fea,+ return_dict=False,+ )[0]++ for j, c in enumerate(context):+ noise_pred[:, :, c] = noise_pred[:, :, c] + pred+ counter[:, :, c] = counter[:, :, c] + 1++ # perform guidance+ if do_classifier_free_guidance:+ noise_pred_uncond, noise_pred_text = (noise_pred / counter).chunk(2)+ noise_pred = noise_pred_uncond + guidance_scale * (+ noise_pred_text - noise_pred_uncond+ )++ latents, pred_original_sample = self.scheduler.step(+ noise_pred, t, latents, **extra_step_kwargs, return_dict=False+ )++ if i == len(timesteps) - 1 or (+ (i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0+ ):+ progress_bar.update()+ if callback is not None and i % callback_steps == 0:+ step_idx = i // getattr(self.scheduler, "order", 1)+ callback(step_idx, t, latents)++ reference_control_reader.clear()+ reference_control_writer.clear()++ # Post-processing+ if isinstance(self.vae, AutoencoderKL):+ images = self.decode_latents(pred_original_sample) # (b, c, f, h, w)+ elif isinstance(self.vae, AutoencoderKLTemporalDecoder):+ images = self.decode_latents_svd(pred_original_sample)++ # Convert to tensor+ if output_type == "tensor":+ images = torch.from_numpy(images)++ if not return_dict:+ return images++ return Pose2VideoPipelineOutput(videos=images)diff --git a/src/pipelines/pipeline_pose2vid.py b/src/pipelines/pipeline_pose2vid.pyindex 2c764fc..a64a2bb 100644--- a/src/pipelines/pipeline_pose2vid.py+++ b/src/pipelines/pipeline_pose2vid.py@@ -456,6 +456,7 @@ class Pose2VideoPipeline(DiffusionPipeline):temporal_adaptive_step = 4,temporal_kv_cache=True,init_latents=None,+ timesteps_list=None,**kwargs,):assert num_inference_steps % temporal_adaptive_step == 0, "temporal_adaptive_step should be divisor of num_inference_steps"@@ -469,8 +470,13 @@ class Pose2VideoPipeline(DiffusionPipeline):do_classifier_free_guidance = guidance_scale > 1.0# Prepare timesteps- timesteps = torch.tensor([999, 666, 333, 0], device=device).long()- self.scheduler.set_step_length(333)+ if timesteps_list is not None:+ assert len(timesteps_list) == num_inference_steps, 'len(timesteps_list) != num_inference_steps'+ timesteps = torch.tensor(timesteps_list, device=device).long()+ self.scheduler.set_step_length(timesteps_list[0]-timesteps_list[1])+ else:+ timesteps = torch.tensor([999, 666, 333, 0], device=device).long()+ self.scheduler.set_step_length(333)jump = num_inference_steps // temporal_adaptive_stepwindows = video_length // temporal_window_size@@ -606,7 +612,7 @@ class Pose2VideoPipeline(DiffusionPipeline):pose_fea = pose_feas[:, :, l:r]add_flag = False- if l > temporal_adaptive_step * temporal_window_size * 2 and motion_bank.shape[1] < 4:+ if l > temporal_adaptive_step * temporal_window_size * 2 and motion_bank.shape[1] < 4 and temporal_kv_cache:add_flag, motion_bank = self.calculate_dis(motion_bank, motion_hidden_state, threshold=17.)if do_classifier_free_guidance:@@ -896,7 +902,7 @@ class Pose2VideoPipeline_Stream(Pose2VideoPipeline):pose_feas = torch.cat([pose_feas[:,:,temporal_window_size:], pose_fea], dim=2)if l > temporal_adaptive_step * temporal_window_size * 2 and motion_bank.shape[1] < 4:- add_flag, motion_bank = self.calculate_dis(motion_bank, motion_hidden_state, threshold=17.)+ add_flag, motion_bank = self.calculate_dis(motion_bank, motion_hidden_states, threshold=17.)latents = ref_image_latents.unsqueeze(2).repeat(1, 1, temporal_window_size, 1, 1)noise = torch.randn_like(latents)diff --git a/src/stylegan2/discriminator.py b/src/stylegan2/discriminator.pynew file mode 100644index 0000000..a54bf8e--- /dev/null+++ b/src/stylegan2/discriminator.py@@ -0,0 +1,793 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++import numpy as np+import torch+from torch_utils import misc+from torch_utils import persistence+from torch_utils.ops import conv2d_resample+from torch_utils.ops import upfirdn2d+from torch_utils.ops import bias_act+from torch_utils.ops import fma+import torch.nn.functional as F+import torch.nn as nn+from diffusers.models.embeddings import TimestepEmbedding, Timesteps+#----------------------------------------------------------------------------+[email protected]_function+def normalize_2nd_moment(x, dim=1, eps=1e-8):+ return x * (x.square().mean(dim=dim, keepdim=True) + eps).rsqrt()++def zero_module(module):+ # Zero out the parameters of a module and return it.+ assert isinstance(module, nn.Conv2d) or isinstance(module, nn.Linear), type(module)+ for p in module.parameters():+ p.detach().zero_()+ return module++#----------------------------------------------------------------------------+[email protected]_function+def modulated_conv2d(+ x, # Input tensor of shape [batch_size, in_channels, in_height, in_width].+ weight, # Weight tensor of shape [out_channels, in_channels, kernel_height, kernel_width].+ styles, # Modulation coefficients of shape [batch_size, in_channels].+ noise = None, # Optional noise tensor to add to the output activations.+ up = 1, # Integer upsampling factor.+ down = 1, # Integer downsampling factor.+ padding = 0, # Padding with respect to the upsampled image.+ resample_filter = None, # Low-pass filter to apply when resampling activations. Must be prepared beforehand by calling upfirdn2d.setup_filter().+ demodulate = True, # Apply weight demodulation?+ flip_weight = True, # False = convolution, True = correlation (matches torch.nn.functional.conv2d).+ fused_modconv = True, # Perform modulation, convolution, and demodulation as a single fused operation?+):+ batch_size = x.shape[0]+ out_channels, in_channels, kh, kw = weight.shape+ misc.assert_shape(weight, [out_channels, in_channels, kh, kw]) # [OIkk]+ misc.assert_shape(x, [batch_size, in_channels, None, None]) # [NIHW]+ misc.assert_shape(styles, [batch_size, in_channels]) # [NI]++ # Pre-normalize inputs to avoid FP16 overflow.+ if x.dtype == torch.float16 and demodulate:+ weight = weight * (1 / np.sqrt(in_channels * kh * kw) / weight.norm(float('inf'), dim=[1,2,3], keepdim=True)) # max_Ikk+ styles = styles / styles.norm(float('inf'), dim=1, keepdim=True) # max_I++ # Calculate per-sample weights and demodulation coefficients.+ w = None+ dcoefs = None+ if demodulate or fused_modconv:+ w = weight.unsqueeze(0) # [NOIkk]+ w = w * styles.reshape(batch_size, 1, -1, 1, 1) # [NOIkk]+ if demodulate:+ dcoefs = (w.square().sum(dim=[2,3,4]) + 1e-8).rsqrt() # [NO]+ if demodulate and fused_modconv:+ w = w * dcoefs.reshape(batch_size, -1, 1, 1, 1) # [NOIkk]++ # Execute by scaling the activations before and after the convolution.+ if not fused_modconv:+ x = x * styles.to(x.dtype).reshape(batch_size, -1, 1, 1)+ x = conv2d_resample.conv2d_resample(x=x, w=weight.to(x.dtype), f=resample_filter, up=up, down=down, padding=padding, flip_weight=flip_weight)+ if demodulate and noise is not None:+ x = fma.fma(x, dcoefs.to(x.dtype).reshape(batch_size, -1, 1, 1), noise.to(x.dtype))+ elif demodulate:+ x = x * dcoefs.to(x.dtype).reshape(batch_size, -1, 1, 1)+ elif noise is not None:+ x = x.add_(noise.to(x.dtype))+ return x++ # Execute as one fused op using grouped convolution.+ with misc.suppress_tracer_warnings(): # this value will be treated as a constant+ batch_size = int(batch_size)+ misc.assert_shape(x, [batch_size, in_channels, None, None])+ x = x.reshape(1, -1, *x.shape[2:])+ w = w.reshape(-1, in_channels, kh, kw)+ x = conv2d_resample.conv2d_resample(x=x, w=w.to(x.dtype), f=resample_filter, up=up, down=down, padding=padding, groups=batch_size, flip_weight=flip_weight)+ x = x.reshape(batch_size, -1, *x.shape[2:])+ if noise is not None:+ x = x.add_(noise)+ return x++#----------------------------------------------------------------------------+[email protected]_class+class FullyConnectedLayer(torch.nn.Module):+ def __init__(self,+ in_features, # Number of input features.+ out_features, # Number of output features.+ bias = True, # Apply additive bias before the activation function?+ activation = 'linear', # Activation function: 'relu', 'lrelu', etc.+ lr_multiplier = 1, # Learning rate multiplier.+ bias_init = 0, # Initial value for the additive bias.+ ):+ super().__init__()+ self.activation = activation+ self.weight = torch.nn.Parameter(torch.randn([out_features, in_features]) / lr_multiplier)+ self.bias = torch.nn.Parameter(torch.full([out_features], np.float32(bias_init))) if bias else None+ self.weight_gain = lr_multiplier / np.sqrt(in_features)+ self.bias_gain = lr_multiplier++ def forward(self, x):+ w = self.weight.to(x.dtype) * self.weight_gain+ b = self.bias+ if b is not None:+ b = b.to(x.dtype)+ if self.bias_gain != 1:+ b = b * self.bias_gain++ if self.activation == 'linear' and b is not None:+ x = torch.addmm(b.unsqueeze(0), x, w.t())+ else:+ x = x.matmul(w.t())+ x = bias_act.bias_act(x, b, act=self.activation)+ return x++#----------------------------------------------------------------------------+[email protected]_class+class Conv2dLayer(torch.nn.Module):+ def __init__(self,+ in_channels, # Number of input channels.+ out_channels, # Number of output channels.+ kernel_size, # Width and height of the convolution kernel.+ bias = True, # Apply additive bias before the activation function?+ activation = 'linear', # Activation function: 'relu', 'lrelu', etc.+ up = 1, # Integer upsampling factor.+ down = 1, # Integer downsampling factor.+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.+ conv_clamp = None, # Clamp the output to +-X, None = disable clamping.+ channels_last = False, # Expect the input to have memory_format=channels_last?+ trainable = True, # Update the weights of this layer during training?+ ):+ super().__init__()+ self.activation = activation+ self.up = up+ self.down = down+ self.conv_clamp = conv_clamp+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))+ self.padding = kernel_size // 2+ self.weight_gain = 1 / np.sqrt(in_channels * (kernel_size ** 2))+ self.act_gain = bias_act.activation_funcs[activation].def_gain++ memory_format = torch.channels_last if channels_last else torch.contiguous_format+ weight = torch.randn([out_channels, in_channels, kernel_size, kernel_size]).to(memory_format=memory_format)+ bias = torch.zeros([out_channels]) if bias else None+ if trainable:+ self.weight = torch.nn.Parameter(weight)+ self.bias = torch.nn.Parameter(bias) if bias is not None else None+ else:+ self.register_buffer('weight', weight)+ if bias is not None:+ self.register_buffer('bias', bias)+ else:+ self.bias = None++ def forward(self, x, gain=1):+ dtype = x.dtype+ w = self.weight * self.weight_gain+ b = self.bias.to(x.dtype) if self.bias is not None else None+ flip_weight = (self.up == 1) # slightly faster+ f=self.resample_filter+ if f is not None:+ f = f.to(dtype=torch.float32)+ x = conv2d_resample.conv2d_resample(x=x, w=w.to(x.dtype), f=f, up=self.up, down=self.down, padding=self.padding, flip_weight=flip_weight)++ x = x.to(dtype)+ act_gain = self.act_gain * gain+ act_clamp = self.conv_clamp * gain if self.conv_clamp is not None else None+ x = bias_act.bias_act(x, b, act=self.activation, gain=act_gain, clamp=act_clamp)+ return x++#----------------------------------------------------------------------------+[email protected]_class+class MappingNetwork(torch.nn.Module):+ def __init__(self,+ z_dim, # Input latent (Z) dimensionality, 0 = no latent.+ c_dim, # Conditioning label (C) dimensionality, 0 = no label.+ w_dim, # Intermediate latent (W) dimensionality.+ num_ws, # Number of intermediate latents to output, None = do not broadcast.+ num_layers = 8, # Number of mapping layers.+ embed_features = None, # Label embedding dimensionality, None = same as w_dim.+ layer_features = None, # Number of intermediate features in the mapping layers, None = same as w_dim.+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.+ lr_multiplier = 0.01, # Learning rate multiplier for the mapping layers.+ w_avg_beta = 0.995, # Decay for tracking the moving average of W during training, None = do not track.+ ):+ super().__init__()+ self.z_dim = z_dim+ self.c_dim = c_dim+ self.w_dim = w_dim+ self.num_ws = num_ws+ self.num_layers = num_layers+ self.w_avg_beta = w_avg_beta++ if embed_features is None:+ embed_features = w_dim+ if c_dim == 0:+ embed_features = 0+ if layer_features is None:+ layer_features = w_dim+ features_list = [z_dim + embed_features] + [layer_features] * (num_layers - 1) + [w_dim]++ if c_dim > 0:+ self.embed = FullyConnectedLayer(c_dim, embed_features)+ for idx in range(num_layers):+ in_features = features_list[idx]+ out_features = features_list[idx + 1]+ layer = FullyConnectedLayer(in_features, out_features, activation=activation, lr_multiplier=lr_multiplier)+ setattr(self, f'fc{idx}', layer)++ if num_ws is not None and w_avg_beta is not None:+ self.register_buffer('w_avg', torch.zeros([w_dim]))++ def forward(self, z, c, truncation_psi=1, truncation_cutoff=None, skip_w_avg_update=False):+ # Embed, normalize, and concat inputs.+ x = None+ with torch.autograd.profiler.record_function('input'):+ if self.z_dim > 0:+ misc.assert_shape(z, [None, self.z_dim])+ x = normalize_2nd_moment(z.to(torch.float32))+ if self.c_dim > 0:+ misc.assert_shape(c, [None, self.c_dim])+ y = normalize_2nd_moment(self.embed(c.to(torch.float32)))+ x = torch.cat([x, y], dim=1) if x is not None else y++ # Main layers.+ for idx in range(self.num_layers):+ layer = getattr(self, f'fc{idx}')+ x = layer(x)++ # Update moving average of W.+ if self.w_avg_beta is not None and self.training and not skip_w_avg_update:+ with torch.autograd.profiler.record_function('update_w_avg'):+ self.w_avg.copy_(x.detach().mean(dim=0).lerp(self.w_avg, self.w_avg_beta))++ # Broadcast.+ if self.num_ws is not None:+ with torch.autograd.profiler.record_function('broadcast'):+ x = x.unsqueeze(1).repeat([1, self.num_ws, 1])++ # Apply truncation.+ if truncation_psi != 1:+ with torch.autograd.profiler.record_function('truncate'):+ assert self.w_avg_beta is not None+ if self.num_ws is None or truncation_cutoff is None:+ x = self.w_avg.lerp(x, truncation_psi)+ else:+ x[:, :truncation_cutoff] = self.w_avg.lerp(x[:, :truncation_cutoff], truncation_psi)+ return x++#----------------------------------------------------------------------------+[email protected]_class+class SynthesisLayer(torch.nn.Module):+ def __init__(self,+ in_channels, # Number of input channels.+ out_channels, # Number of output channels.+ w_dim, # Intermediate latent (W) dimensionality.+ resolution, # Resolution of this layer.+ kernel_size = 3, # Convolution kernel size.+ up = 1, # Integer upsampling factor.+ use_noise = True, # Enable noise input?+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.+ channels_last = False, # Use channels_last format for the weights?+ ):+ super().__init__()+ self.resolution = resolution+ self.up = up+ self.use_noise = use_noise+ self.activation = activation+ self.conv_clamp = conv_clamp+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))+ self.padding = kernel_size // 2+ self.act_gain = bias_act.activation_funcs[activation].def_gain++ self.affine = FullyConnectedLayer(w_dim, in_channels, bias_init=1)+ memory_format = torch.channels_last if channels_last else torch.contiguous_format+ self.weight = torch.nn.Parameter(torch.randn([out_channels, in_channels, kernel_size, kernel_size]).to(memory_format=memory_format))+ if use_noise:+ self.register_buffer('noise_const', torch.randn([resolution, resolution]))+ self.noise_strength = torch.nn.Parameter(torch.zeros([]))+ self.bias = torch.nn.Parameter(torch.zeros([out_channels]))++ def forward(self, x, w, noise_mode='random', fused_modconv=True, gain=1):+ assert noise_mode in ['random', 'const', 'none']+ in_resolution = self.resolution // self.up+ misc.assert_shape(x, [None, self.weight.shape[1], in_resolution, in_resolution])+ styles = self.affine(w)++ noise = None+ if self.use_noise and noise_mode == 'random':+ noise = torch.randn([x.shape[0], 1, self.resolution, self.resolution], device=x.device) * self.noise_strength+ if self.use_noise and noise_mode == 'const':+ noise = self.noise_const * self.noise_strength++ flip_weight = (self.up == 1) # slightly faster+ x = modulated_conv2d(x=x, weight=self.weight, styles=styles, noise=noise, up=self.up,+ padding=self.padding, resample_filter=self.resample_filter, flip_weight=flip_weight, fused_modconv=fused_modconv)++ act_gain = self.act_gain * gain+ act_clamp = self.conv_clamp * gain if self.conv_clamp is not None else None+ x = bias_act.bias_act(x, self.bias.to(x.dtype), act=self.activation, gain=act_gain, clamp=act_clamp)+ return x++#----------------------------------------------------------------------------+[email protected]_class+class ToRGBLayer(torch.nn.Module):+ def __init__(self, in_channels, out_channels, w_dim, kernel_size=1, conv_clamp=None, channels_last=False):+ super().__init__()+ self.conv_clamp = conv_clamp+ self.affine = FullyConnectedLayer(w_dim, in_channels, bias_init=1)+ memory_format = torch.channels_last if channels_last else torch.contiguous_format+ self.weight = torch.nn.Parameter(torch.randn([out_channels, in_channels, kernel_size, kernel_size]).to(memory_format=memory_format))+ self.bias = torch.nn.Parameter(torch.zeros([out_channels]))+ self.weight_gain = 1 / np.sqrt(in_channels * (kernel_size ** 2))++ def forward(self, x, w, fused_modconv=True):+ styles = self.affine(w) * self.weight_gain+ x = modulated_conv2d(x=x, weight=self.weight, styles=styles, demodulate=False, fused_modconv=fused_modconv)+ x = bias_act.bias_act(x, self.bias.to(x.dtype), clamp=self.conv_clamp)+ return x++#----------------------------------------------------------------------------+[email protected]_class+class SynthesisBlock(torch.nn.Module):+ def __init__(self,+ in_channels, # Number of input channels, 0 = first block.+ out_channels, # Number of output channels.+ w_dim, # Intermediate latent (W) dimensionality.+ resolution, # Resolution of this block.+ img_channels, # Number of output color channels.+ is_last, # Is this the last block?+ architecture = 'skip', # Architecture: 'orig', 'skip', 'resnet'.+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.+ use_fp16 = False, # Use FP16 for this block?+ fp16_channels_last = False, # Use channels-last memory format with FP16?+ **layer_kwargs, # Arguments for SynthesisLayer.+ ):+ assert architecture in ['orig', 'skip', 'resnet']+ super().__init__()+ self.in_channels = in_channels+ self.w_dim = w_dim+ self.resolution = resolution+ self.img_channels = img_channels+ self.is_last = is_last+ self.architecture = architecture+ self.use_fp16 = use_fp16+ self.channels_last = (use_fp16 and fp16_channels_last)+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))+ self.num_conv = 0+ self.num_torgb = 0++ if in_channels == 0:+ self.const = torch.nn.Parameter(torch.randn([out_channels, resolution, resolution]))++ if in_channels != 0:+ self.conv0 = SynthesisLayer(in_channels, out_channels, w_dim=w_dim, resolution=resolution, up=2,+ resample_filter=resample_filter, conv_clamp=conv_clamp, channels_last=self.channels_last, **layer_kwargs)+ self.num_conv += 1++ self.conv1 = SynthesisLayer(out_channels, out_channels, w_dim=w_dim, resolution=resolution,+ conv_clamp=conv_clamp, channels_last=self.channels_last, **layer_kwargs)+ self.num_conv += 1++ if is_last or architecture == 'skip':+ self.torgb = ToRGBLayer(out_channels, img_channels, w_dim=w_dim,+ conv_clamp=conv_clamp, channels_last=self.channels_last)+ self.num_torgb += 1++ if in_channels != 0 and architecture == 'resnet':+ self.skip = Conv2dLayer(in_channels, out_channels, kernel_size=1, bias=False, up=2,+ resample_filter=resample_filter, channels_last=self.channels_last)++ def forward(self, x, img, ws, force_fp32=False, fused_modconv=None, **layer_kwargs):+ misc.assert_shape(ws, [None, self.num_conv + self.num_torgb, self.w_dim])+ w_iter = iter(ws.unbind(dim=1))+ dtype = torch.float16 if self.use_fp16 and not force_fp32 else torch.float32+ memory_format = torch.channels_last if self.channels_last and not force_fp32 else torch.contiguous_format+ if fused_modconv is None:+ with misc.suppress_tracer_warnings(): # this value will be treated as a constant+ fused_modconv = (not self.training) and (dtype == torch.float32 or int(x.shape[0]) == 1)++ # Input.+ if self.in_channels == 0:+ x = self.const.to(dtype=dtype, memory_format=memory_format)+ x = x.unsqueeze(0).repeat([ws.shape[0], 1, 1, 1])+ else:+ misc.assert_shape(x, [None, self.in_channels, self.resolution // 2, self.resolution // 2])+ x = x.to(dtype=dtype, memory_format=memory_format)++ # Main layers.+ if self.in_channels == 0:+ x = self.conv1(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)+ elif self.architecture == 'resnet':+ y = self.skip(x, gain=np.sqrt(0.5))+ x = self.conv0(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)+ x = self.conv1(x, next(w_iter), fused_modconv=fused_modconv, gain=np.sqrt(0.5), **layer_kwargs)+ x = y.add_(x)+ else:+ x = self.conv0(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)+ x = self.conv1(x, next(w_iter), fused_modconv=fused_modconv, **layer_kwargs)++ # ToRGB.+ if img is not None:+ misc.assert_shape(img, [None, self.img_channels, self.resolution // 2, self.resolution // 2])+ img = upfirdn2d.upsample2d(img, self.resample_filter)+ if self.is_last or self.architecture == 'skip':+ y = self.torgb(x, next(w_iter), fused_modconv=fused_modconv)+ y = y.to(dtype=torch.float32, memory_format=torch.contiguous_format)+ img = img.add_(y) if img is not None else y++ assert x.dtype == dtype+ assert img is None or img.dtype == torch.float32+ return x, img++#----------------------------------------------------------------------------+[email protected]_class+class SynthesisNetwork(torch.nn.Module):+ def __init__(self,+ w_dim, # Intermediate latent (W) dimensionality.+ img_resolution, # Output image resolution.+ img_channels, # Number of color channels.+ channel_base = 32768, # Overall multiplier for the number of channels.+ channel_max = 512, # Maximum number of channels in any layer.+ num_fp16_res = 0, # Use FP16 for the N highest resolutions.+ **block_kwargs, # Arguments for SynthesisBlock.+ ):+ assert img_resolution >= 4 and img_resolution & (img_resolution - 1) == 0+ super().__init__()+ self.w_dim = w_dim+ self.img_resolution = img_resolution+ self.img_resolution_log2 = int(np.log2(img_resolution))+ self.img_channels = img_channels+ self.block_resolutions = [2 ** i for i in range(2, self.img_resolution_log2 + 1)]+ channels_dict = {res: min(channel_base // res, channel_max) for res in self.block_resolutions}+ fp16_resolution = max(2 ** (self.img_resolution_log2 + 1 - num_fp16_res), 8)++ self.num_ws = 0+ for res in self.block_resolutions:+ in_channels = channels_dict[res // 2] if res > 4 else 0+ out_channels = channels_dict[res]+ use_fp16 = (res >= fp16_resolution)+ is_last = (res == self.img_resolution)+ block = SynthesisBlock(in_channels, out_channels, w_dim=w_dim, resolution=res,+ img_channels=img_channels, is_last=is_last, use_fp16=use_fp16, **block_kwargs)+ self.num_ws += block.num_conv+ if is_last:+ self.num_ws += block.num_torgb+ setattr(self, f'b{res}', block)++ def forward(self, ws, **block_kwargs):+ block_ws = []+ with torch.autograd.profiler.record_function('split_ws'):+ misc.assert_shape(ws, [None, self.num_ws, self.w_dim])+ ws = ws.to(torch.float32)+ w_idx = 0+ for res in self.block_resolutions:+ block = getattr(self, f'b{res}')+ block_ws.append(ws.narrow(1, w_idx, block.num_conv + block.num_torgb))+ w_idx += block.num_conv++ x = img = None+ for res, cur_ws in zip(self.block_resolutions, block_ws):+ block = getattr(self, f'b{res}')+ x, img = block(x, img, cur_ws, **block_kwargs)+ return img++#----------------------------------------------------------------------------+[email protected]_class+class Generator(torch.nn.Module):+ def __init__(self,+ z_dim, # Input latent (Z) dimensionality.+ c_dim, # Conditioning label (C) dimensionality.+ w_dim, # Intermediate latent (W) dimensionality.+ img_resolution, # Output resolution.+ img_channels, # Number of output color channels.+ mapping_kwargs = {}, # Arguments for MappingNetwork.+ synthesis_kwargs = {}, # Arguments for SynthesisNetwork.+ ):+ super().__init__()+ self.z_dim = z_dim+ self.c_dim = c_dim+ self.w_dim = w_dim+ self.img_resolution = img_resolution+ self.img_channels = img_channels+ self.synthesis = SynthesisNetwork(w_dim=w_dim, img_resolution=img_resolution, img_channels=img_channels, **synthesis_kwargs)+ self.num_ws = self.synthesis.num_ws+ self.mapping = MappingNetwork(z_dim=z_dim, c_dim=c_dim, w_dim=w_dim, num_ws=self.num_ws, **mapping_kwargs)++ def forward(self, z, c, truncation_psi=1, truncation_cutoff=None, **synthesis_kwargs):+ ws = self.mapping(z, c, truncation_psi=truncation_psi, truncation_cutoff=truncation_cutoff)+ img = self.synthesis(ws, **synthesis_kwargs)+ return img++#----------------------------------------------------------------------------+[email protected]_class+class DiscriminatorBlock(torch.nn.Module):+ def __init__(self,+ in_channels, # Number of input channels, 0 = first block.+ tmp_channels, # Number of intermediate channels.+ out_channels, # Number of output channels.+ resolution, # Resolution of this block.+ img_channels, # Number of input color channels.+ first_layer_idx, # Index of the first layer.+ architecture = 'resnet', # Architecture: 'orig', 'skip', 'resnet'.+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.+ resample_filter = [1,3,3,1], # Low-pass filter to apply when resampling activations.+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.+ use_fp16 = False, # Use FP16 for this block?+ fp16_channels_last = False, # Use channels-last memory format with FP16?+ freeze_layers = 0, # Freeze-D: Number of layers to freeze.+ ):+ assert in_channels in [0, tmp_channels]+ assert architecture in ['orig', 'skip', 'resnet']+ super().__init__()+ self.in_channels = in_channels+ self.resolution = resolution+ self.img_channels = img_channels+ self.first_layer_idx = first_layer_idx+ self.architecture = architecture+ self.use_fp16 = use_fp16+ self.channels_last = (use_fp16 and fp16_channels_last)+ self.register_buffer('resample_filter', upfirdn2d.setup_filter(resample_filter))++ self.num_layers = 0+ def trainable_gen():+ while True:+ layer_idx = self.first_layer_idx + self.num_layers+ trainable = (layer_idx >= freeze_layers)+ self.num_layers += 1+ yield trainable+ trainable_iter = trainable_gen()++ if in_channels == 0 or architecture == 'skip':+ self.fromrgb = Conv2dLayer(img_channels, tmp_channels, kernel_size=1, activation=activation,+ trainable=next(trainable_iter), conv_clamp=conv_clamp, channels_last=self.channels_last)++ self.conv0 = Conv2dLayer(tmp_channels, tmp_channels, kernel_size=3, activation=activation,+ trainable=next(trainable_iter), conv_clamp=conv_clamp, channels_last=self.channels_last)++ self.conv1 = Conv2dLayer(tmp_channels, out_channels, kernel_size=3, activation=activation, down=2,+ trainable=next(trainable_iter), resample_filter=resample_filter, conv_clamp=conv_clamp, channels_last=self.channels_last)++ if architecture == 'resnet':+ self.skip = Conv2dLayer(tmp_channels, out_channels, kernel_size=1, bias=False, down=2,+ trainable=next(trainable_iter), resample_filter=resample_filter, channels_last=self.channels_last)++ def forward(self, x, img, force_fp32=False):+ dtype = torch.float16 if self.use_fp16 and not force_fp32 else torch.float32+ memory_format = torch.channels_last if self.channels_last and not force_fp32 else torch.contiguous_format++ # Input.+ if x is not None:+ misc.assert_shape(x, [None, self.in_channels, self.resolution, self.resolution])+ x = x.to(dtype=dtype, memory_format=memory_format)++ # FromRGB.+ if self.in_channels == 0 or self.architecture == 'skip':+ misc.assert_shape(img, [None, self.img_channels, self.resolution, self.resolution])+ img = img.to(dtype=dtype, memory_format=memory_format)+ y = self.fromrgb(img)+ x = x + y if x is not None else y+ img = upfirdn2d.downsample2d(img, self.resample_filter) if self.architecture == 'skip' else None++ # Main layers.+ if self.architecture == 'resnet':+ y = self.skip(x, gain=np.sqrt(0.5))+ x = self.conv0(x)+ x = self.conv1(x, gain=np.sqrt(0.5))+ x = y.add_(x)+ else:+ x = self.conv0(x)+ x = self.conv1(x)++ assert x.dtype == dtype+ return x, img++#----------------------------------------------------------------------------+[email protected]_class+class MinibatchStdLayer(torch.nn.Module):+ def __init__(self, group_size, num_channels=1):+ super().__init__()+ self.group_size = group_size+ self.num_channels = num_channels++ def forward(self, x):+ N, C, H, W = x.shape+ with misc.suppress_tracer_warnings(): # as_tensor results are registered as constants+ G = torch.min(torch.as_tensor(self.group_size), torch.as_tensor(N)) if self.group_size is not None else N+ F = self.num_channels+ c = C // F++ y = x.reshape(G, -1, F, c, H, W) # [GnFcHW] Split minibatch N into n groups of size G, and channels C into F groups of size c.+ y = y - y.mean(dim=0) # [GnFcHW] Subtract mean over group.+ y = y.square().mean(dim=0) # [nFcHW] Calc variance over group.+ y = (y + 1e-8).sqrt() # [nFcHW] Calc stddev over group.+ y = y.mean(dim=[2,3,4]) # [nF] Take average over channels and pixels.+ y = y.reshape(-1, F, 1, 1) # [nF11] Add missing dimensions.+ y = y.repeat(G, 1, H, W) # [NFHW] Replicate over group and pixels.+ x = torch.cat([x, y], dim=1) # [NCHW] Append to input as new channels.+ return x++#----------------------------------------------------------------------------+[email protected]_class+class DiscriminatorEpilogue(torch.nn.Module):+ def __init__(self,+ in_channels, # Number of input channels.+ cmap_dim, # Dimensionality of mapped conditioning label, 0 = no label.+ resolution, # Resolution of this block.+ img_channels, # Number of input color channels.+ architecture = 'resnet', # Architecture: 'orig', 'skip', 'resnet'.+ mbstd_group_size = 4, # Group size for the minibatch standard deviation layer, None = entire minibatch.+ mbstd_num_channels = 1, # Number of features for the minibatch standard deviation layer, 0 = disable.+ activation = 'lrelu', # Activation function: 'relu', 'lrelu', etc.+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.+ ):+ assert architecture in ['orig', 'skip', 'resnet']+ super().__init__()+ self.in_channels = in_channels+ self.cmap_dim = cmap_dim+ self.resolution = resolution+ self.img_channels = img_channels+ self.architecture = architecture++ if architecture == 'skip':+ self.fromrgb = Conv2dLayer(img_channels, in_channels, kernel_size=1, activation=activation)+ self.mbstd = MinibatchStdLayer(group_size=mbstd_group_size, num_channels=mbstd_num_channels) if mbstd_num_channels > 0 else None+ self.conv = Conv2dLayer(in_channels + mbstd_num_channels, in_channels, kernel_size=3, activation=activation, conv_clamp=conv_clamp)+ self.fc = FullyConnectedLayer(in_channels * (resolution ** 2), in_channels, activation=activation)+ self.out = FullyConnectedLayer(in_channels, 1 if cmap_dim == 0 else cmap_dim)++ def forward(self, x, img, cmap, force_fp32=False):+ misc.assert_shape(x, [None, self.in_channels, self.resolution, self.resolution]) # [NCHW]+ _ = force_fp32 # unused+ dtype = torch.float32+ memory_format = torch.contiguous_format++ # FromRGB.+ x = x.to(dtype=dtype, memory_format=memory_format)+ if self.architecture == 'skip':+ misc.assert_shape(img, [None, self.img_channels, self.resolution, self.resolution])+ img = img.to(dtype=dtype, memory_format=memory_format)+ x = x + self.fromrgb(img)++ # Main layers.+ if self.mbstd is not None:+ x = self.mbstd(x)+ x = self.conv(x)+ x = self.fc(x.flatten(1))+ x = self.out(x)++ # Conditioning.+ if self.cmap_dim > 0:+ misc.assert_shape(cmap, [None, self.cmap_dim])+ x = (x * cmap).sum(dim=1, keepdim=True) * (1 / np.sqrt(self.cmap_dim))++ # assert x.dtype == dtype+ return x++#----------------------------------------------------------------------------+[email protected]_class+class Discriminator(torch.nn.Module):+ def __init__(self,+ c_dim=0, # Conditioning label (C) dimensionality.+ img_resolution=512, # Input resolution.+ img_channels=3, # Number of input color channels.+ temb_channels=512, # Number of time embedding channels+ architecture = 'resnet', # Architecture: 'orig', 'skip', 'resnet'.+ channel_base = 32768, # Overall multiplier for the number of channels.+ channel_max = 512, # Maximum number of channels in any layer.+ num_fp16_res = 0, # Use FP16 for the N highest resolutions.+ conv_clamp = None, # Clamp the output of convolution layers to +-X, None = disable clamping.+ cmap_dim = None, # Dimensionality of mapped conditioning label, None = default.+ block_kwargs = {}, # Arguments for DiscriminatorBlock.+ mapping_kwargs = {}, # Arguments for MappingNetwork.+ epilogue_kwargs = {}, # Arguments for DiscriminatorEpilogue.+ ):+ super().__init__()+ self.c_dim = c_dim+ self.img_resolution = img_resolution+ self.img_resolution_log2 = int(np.log2(img_resolution))+ self.img_channels = img_channels+ self.block_resolutions = [2 ** i for i in range(self.img_resolution_log2, 2, -1)]+ channels_dict = {res: min(channel_base // res, channel_max) for res in self.block_resolutions + [4]}+ fp16_resolution = max(2 ** (self.img_resolution_log2 + 1 - num_fp16_res), 8)++ if cmap_dim is None:+ cmap_dim = channels_dict[4]+ if c_dim == 0:+ cmap_dim = 0++ self.time_emb_proj = nn.ModuleList()+ common_kwargs = dict(img_channels=img_channels, architecture=architecture, conv_clamp=conv_clamp)+ cur_layer_idx = 0+ for res in self.block_resolutions:+ in_channels = channels_dict[res] if res < img_resolution else 0+ tmp_channels = channels_dict[res]+ out_channels = channels_dict[res // 2]+ use_fp16 = (res >= fp16_resolution)+ block = DiscriminatorBlock(in_channels, tmp_channels, out_channels, resolution=res,+ first_layer_idx=cur_layer_idx, use_fp16=use_fp16, **block_kwargs, **common_kwargs)+ setattr(self, f'b{res}', block)+ cur_layer_idx += block.num_layers+ self.time_emb_proj.append(zero_module(nn.Linear(temb_channels, out_channels)))+ if c_dim > 0:+ self.mapping = MappingNetwork(z_dim=0, c_dim=c_dim, w_dim=cmap_dim, num_ws=None, w_avg_beta=None, **mapping_kwargs)+ self.b4 = DiscriminatorEpilogue(channels_dict[4], cmap_dim=cmap_dim, resolution=4, **epilogue_kwargs, **common_kwargs)++ self.nonlinearity = nn.SiLU()+ self.time_proj = Timesteps(temb_channels, True, 0)++ self.time_embedding = TimestepEmbedding(temb_channels, temb_channels)++ def _pass(self, img, c, timesteps, **block_kwargs):+ x = None+ t_emb = self.time_proj(timesteps)+ t_emb = t_emb.to(dtype=img.dtype)+ t_emb = self.time_embedding(t_emb)+ for i, res in enumerate(self.block_resolutions):+ block = getattr(self, f'b{res}')+ x, img = block(x, img, **block_kwargs)+ temb = self.time_emb_proj[i](self.nonlinearity(t_emb))[:, :, None, None]+ x += temb++ cmap = None+ if self.c_dim > 0:+ cmap = self.mapping(None, c)+ x = self.b4(x, img, cmap)+ return x++ def forward_G(self, fake, real, timesteps, c_fake=None, c_real=None):++ fake_score = self._pass(fake, c_fake, timesteps)+ adv_loss = F.softplus(-fake_score).mean() # Hinge G+ return adv_loss++ # def r1_penalty(self, real, c_real=None, timesteps):+ # real.requires_grad_(True)+ # real_score, _ = self._pass(real, c_real, timesteps)+ # grad_real = torch.autograd.grad(outputs=real_score.sum(),+ # inputs=real,+ # create_graph=True,+ # retain_graph=True,+ # only_inputs=True)[0]+ # r1 = grad_real.pow(2).reshape(grad_real.size(0), -1).sum(1).mean()+ # return r1++ def forward_D(self, fake, real, timesteps, c_fake=None, c_real=None, global_step: int = 0):+ fake = fake.detach()++ fake_score = self._pass(fake, c_fake, timesteps)+ real_score = self._pass(real, c_real, timesteps)++ loss_D = (F.softplus(fake_score) + F.softplus(-real_score)).mean()++ # R1 regularization (lazy)+ # if self.r1_gamma > 0 and self.r1_interval > 0 and (global_step % self.r1_interval == 0):+ # r1 = self.r1_penalty(real, c_real, timesteps)+ # loss_D = loss_D + (self.r1_gamma * 0.5) * r1+ return loss_D++ def forward(self, fake, real, timesteps, gen=True, **kwargs):+ if gen:+ return self.forward_G(fake, real, timesteps, **kwargs)+ else:+ return self.forward_D(fake, real, timesteps, **kwargs)++#----------------------------------------------------------------------------diff --git a/src/stylegan2/dnnlib/__init__.py b/src/stylegan2/dnnlib/__init__.pynew file mode 100644index 0000000..2f08cf3--- /dev/null+++ b/src/stylegan2/dnnlib/__init__.py@@ -0,0 +1,9 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++from .util import EasyDict, make_cache_dir_pathdiff --git a/src/stylegan2/dnnlib/util.py b/src/stylegan2/dnnlib/util.pynew file mode 100644index 0000000..c611acc--- /dev/null+++ b/src/stylegan2/dnnlib/util.py@@ -0,0 +1,468 @@++import ctypes+import fnmatch+import importlib+import inspect+import numpy as np+import os+import shutil+import sys+import types+import io+import pickle+import re+import requests+import html+import hashlib+import glob+import tempfile+import urllib+import urllib.request+import uuid++from distutils.util import strtobool+from typing import Any, List, Tuple, Union+++# Util classes+# ------------------------------------------------------------------------------------------+++class EasyDict(dict):+ """Convenience class that behaves like a dict but allows access with the attribute syntax."""++ def __getattr__(self, name: str) -> Any:+ try:+ return self[name]+ except KeyError:+ raise AttributeError(name)++ def __setattr__(self, name: str, value: Any) -> None:+ self[name] = value++ def __delattr__(self, name: str) -> None:+ del self[name]+++class Logger(object):+ """Redirect stderr to stdout, optionally print stdout to a file, and optionally force flushing on both stdout and the file."""++ def __init__(self, file_name: str = None, file_mode: str = "w", should_flush: bool = True):+ self.file = None++ if file_name is not None:+ self.file = open(file_name, file_mode)++ self.should_flush = should_flush+ self.stdout = sys.stdout+ self.stderr = sys.stderr++ sys.stdout = self+ sys.stderr = self++ def __enter__(self) -> "Logger":+ return self++ def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:+ self.close()++ def write(self, text: Union[str, bytes]) -> None:+ """Write text to stdout (and a file) and optionally flush."""+ if isinstance(text, bytes):+ text = text.decode()+ if len(text) == 0: # workaround for a bug in VSCode debugger: sys.stdout.write(''); sys.stdout.flush() => crash+ return++ if self.file is not None:+ self.file.write(text)++ self.stdout.write(text)++ if self.should_flush:+ self.flush()++ def flush(self) -> None:+ """Flush written text to both stdout and a file, if open."""+ if self.file is not None:+ self.file.flush()++ self.stdout.flush()++ def close(self) -> None:+ """Flush, close possible files, and remove stdout/stderr mirroring."""+ self.flush()++ # if using multiple loggers, prevent closing in wrong order+ if sys.stdout is self:+ sys.stdout = self.stdout+ if sys.stderr is self:+ sys.stderr = self.stderr++ if self.file is not None:+ self.file.close()+ self.file = None+++# Cache directories+# ------------------------------------------------------------------------------------------++_dnnlib_cache_dir = None++def set_cache_dir(path: str) -> None:+ global _dnnlib_cache_dir+ _dnnlib_cache_dir = path++def make_cache_dir_path(*paths: str) -> str:+ if _dnnlib_cache_dir is not None:+ return os.path.join(_dnnlib_cache_dir, *paths)+ if 'DNNLIB_CACHE_DIR' in os.environ:+ return os.path.join(os.environ['DNNLIB_CACHE_DIR'], *paths)+ if 'HOME' in os.environ:+ return os.path.join(os.environ['HOME'], '.cache', 'dnnlib', *paths)+ if 'USERPROFILE' in os.environ:+ return os.path.join(os.environ['USERPROFILE'], '.cache', 'dnnlib', *paths)+ return os.path.join(tempfile.gettempdir(), '.cache', 'dnnlib', *paths)++# Small util functions+# ------------------------------------------------------------------------------------------+++def format_time(seconds: Union[int, float]) -> str:+ """Convert the seconds to human readable string with days, hours, minutes and seconds."""+ s = int(np.rint(seconds))++ if s < 60:+ return "{0}s".format(s)+ elif s < 60 * 60:+ return "{0}m {1:02}s".format(s // 60, s % 60)+ elif s < 24 * 60 * 60:+ return "{0}h {1:02}m {2:02}s".format(s // (60 * 60), (s // 60) % 60, s % 60)+ else:+ return "{0}d {1:02}h {2:02}m".format(s // (24 * 60 * 60), (s // (60 * 60)) % 24, (s // 60) % 60)+++def ask_yes_no(question: str) -> bool:+ """Ask the user the question until the user inputs a valid answer."""+ while True:+ try:+ print("{0} [y/n]".format(question))+ return strtobool(input().lower())+ except ValueError:+ pass+++def tuple_product(t: Tuple) -> Any:+ """Calculate the product of the tuple elements."""+ result = 1++ for v in t:+ result *= v++ return result+++_str_to_ctype = {+ "uint8": ctypes.c_ubyte,+ "uint16": ctypes.c_uint16,+ "uint32": ctypes.c_uint32,+ "uint64": ctypes.c_uint64,+ "int8": ctypes.c_byte,+ "int16": ctypes.c_int16,+ "int32": ctypes.c_int32,+ "int64": ctypes.c_int64,+ "float32": ctypes.c_float,+ "float64": ctypes.c_double+}+++def get_dtype_and_ctype(type_obj: Any) -> Tuple[np.dtype, Any]:+ """Given a type name string (or an object having a __name__ attribute), return matching Numpy and ctypes types that have the same size in bytes."""+ type_str = None++ if isinstance(type_obj, str):+ type_str = type_obj+ elif hasattr(type_obj, "__name__"):+ type_str = type_obj.__name__+ elif hasattr(type_obj, "name"):+ type_str = type_obj.name+ else:+ raise RuntimeError("Cannot infer type name from input")++ assert type_str in _str_to_ctype.keys()++ my_dtype = np.dtype(type_str)+ my_ctype = _str_to_ctype[type_str]++ assert my_dtype.itemsize == ctypes.sizeof(my_ctype)++ return my_dtype, my_ctype+++def is_pickleable(obj: Any) -> bool:+ try:+ with io.BytesIO() as stream:+ pickle.dump(obj, stream)+ return True+ except:+ return False+++# Functionality to import modules/objects by name, and call functions by name+# ------------------------------------------------------------------------------------------++def get_module_from_obj_name(obj_name: str) -> Tuple[types.ModuleType, str]:+ """Searches for the underlying module behind the name to some python object.+ Returns the module and the object name (original name with module part removed)."""++ # allow convenience shorthands, substitute them by full names+ obj_name = re.sub("^np.", "numpy.", obj_name)+ obj_name = re.sub("^tf.", "tensorflow.", obj_name)++ # list alternatives for (module_name, local_obj_name)+ parts = obj_name.split(".")+ name_pairs = [(".".join(parts[:i]), ".".join(parts[i:])) for i in range(len(parts), 0, -1)]++ # try each alternative in turn+ for module_name, local_obj_name in name_pairs:+ try:+ module = importlib.import_module(module_name) # may raise ImportError+ get_obj_from_module(module, local_obj_name) # may raise AttributeError+ return module, local_obj_name+ except:+ pass++ # maybe some of the modules themselves contain errors?+ for module_name, _local_obj_name in name_pairs:+ try:+ importlib.import_module(module_name) # may raise ImportError+ except ImportError:+ if not str(sys.exc_info()[1]).startswith("No module named '" + module_name + "'"):+ raise++ # maybe the requested attribute is missing?+ for module_name, local_obj_name in name_pairs:+ try:+ module = importlib.import_module(module_name) # may raise ImportError+ get_obj_from_module(module, local_obj_name) # may raise AttributeError+ except ImportError:+ pass++ # we are out of luck, but we have no idea why+ raise ImportError(obj_name)+++def get_obj_from_module(module: types.ModuleType, obj_name: str) -> Any:+ """Traverses the object name and returns the last (rightmost) python object."""+ if obj_name == '':+ return module+ obj = module+ for part in obj_name.split("."):+ obj = getattr(obj, part)+ return obj+++def get_obj_by_name(name: str) -> Any:+ """Finds the python object with the given name."""+ module, obj_name = get_module_from_obj_name(name)+ return get_obj_from_module(module, obj_name)+++def call_func_by_name(*args, func_name: str = None, **kwargs) -> Any:+ """Finds the python object with the given name and calls it as a function."""+ assert func_name is not None+ func_obj = get_obj_by_name(func_name)+ assert callable(func_obj)+ return func_obj(*args, **kwargs)+++def construct_class_by_name(*args, class_name: str = None, **kwargs) -> Any:+ """Finds the python class with the given name and constructs it with the given arguments."""+ return call_func_by_name(*args, func_name=class_name, **kwargs)+++def get_module_dir_by_obj_name(obj_name: str) -> str:+ """Get the directory path of the module containing the given object name."""+ module, _ = get_module_from_obj_name(obj_name)+ return os.path.dirname(inspect.getfile(module))+++def is_top_level_function(obj: Any) -> bool:+ """Determine whether the given object is a top-level function, i.e., defined at module scope using 'def'."""+ return callable(obj) and obj.__name__ in sys.modules[obj.__module__].__dict__+++def get_top_level_function_name(obj: Any) -> str:+ """Return the fully-qualified name of a top-level function."""+ assert is_top_level_function(obj)+ module = obj.__module__+ if module == '__main__':+ module = os.path.splitext(os.path.basename(sys.modules[module].__file__))[0]+ return module + "." + obj.__name__+++# File system helpers+# ------------------------------------------------------------------------------------------++def list_dir_recursively_with_ignore(dir_path: str, ignores: List[str] = None, add_base_to_relative: bool = False) -> List[Tuple[str, str]]:+ """List all files recursively in a given directory while ignoring given file and directory names.+ Returns list of tuples containing both absolute and relative paths."""+ assert os.path.isdir(dir_path)+ base_name = os.path.basename(os.path.normpath(dir_path))++ if ignores is None:+ ignores = []++ result = []++ for root, dirs, files in os.walk(dir_path, topdown=True):+ for ignore_ in ignores:+ dirs_to_remove = [d for d in dirs if fnmatch.fnmatch(d, ignore_)]++ # dirs need to be edited in-place+ for d in dirs_to_remove:+ dirs.remove(d)++ files = [f for f in files if not fnmatch.fnmatch(f, ignore_)]++ absolute_paths = [os.path.join(root, f) for f in files]+ relative_paths = [os.path.relpath(p, dir_path) for p in absolute_paths]++ if add_base_to_relative:+ relative_paths = [os.path.join(base_name, p) for p in relative_paths]++ assert len(absolute_paths) == len(relative_paths)+ result += zip(absolute_paths, relative_paths)++ return result+++def copy_files_and_create_dirs(files: List[Tuple[str, str]]) -> None:+ """Takes in a list of tuples of (src, dst) paths and copies files.+ Will create all necessary directories."""+ for file in files:+ target_dir_name = os.path.dirname(file[1])++ # will create all intermediate-level directories+ if not os.path.exists(target_dir_name):+ os.makedirs(target_dir_name)++ shutil.copyfile(file[0], file[1])+++# URL helpers+# ------------------------------------------------------------------------------------------++def is_url(obj: Any, allow_file_urls: bool = False) -> bool:+ """Determine whether the given object is a valid URL string."""+ if not isinstance(obj, str) or not "://" in obj:+ return False+ if allow_file_urls and obj.startswith('file://'):+ return True+ try:+ res = requests.compat.urlparse(obj)+ if not res.scheme or not res.netloc or not "." in res.netloc:+ return False+ res = requests.compat.urlparse(requests.compat.urljoin(obj, "/"))+ if not res.scheme or not res.netloc or not "." in res.netloc:+ return False+ except:+ return False+ return True+++def open_url(url: str, cache_dir: str = None, num_attempts: int = 10, verbose: bool = True, return_filename: bool = False, cache: bool = True) -> Any:+ """Download the given URL and return a binary-mode file object to access the data."""+ assert num_attempts >= 1+ assert not (return_filename and (not cache))++ # Doesn't look like an URL scheme so interpret it as a local filename.+ if not re.match('^[a-z]+://', url):+ return url if return_filename else open(url, "rb")++ # Handle file URLs. This code handles unusual file:// patterns that+ # arise on Windows:+ #+ # file:///c:/foo.txt+ #+ # which would translate to a local '/c:/foo.txt' filename that's+ # invalid. Drop the forward slash for such pathnames.+ #+ # If you touch this code path, you should test it on both Linux and+ # Windows.+ #+ # Some internet resources suggest using urllib.request.url2pathname() but+ # but that converts forward slashes to backslashes and this causes+ # its own set of problems.+ if url.startswith('file://'):+ filename = urllib.parse.urlparse(url).path+ if re.match(r'^/[a-zA-Z]:', filename):+ filename = filename[1:]+ return filename if return_filename else open(filename, "rb")++ assert is_url(url)++ # Lookup from cache.+ if cache_dir is None:+ cache_dir = make_cache_dir_path('downloads')++ url_md5 = hashlib.md5(url.encode("utf-8")).hexdigest()+ if cache:+ cache_files = glob.glob(os.path.join(cache_dir, url_md5 + "_*"))+ if len(cache_files) == 1:+ filename = cache_files[0]+ return filename if return_filename else open(filename, "rb")++ # Download.+ url_name = None+ url_data = None+ with requests.Session() as session:+ if verbose:+ print("Downloading %s ..." % url, end="", flush=True)+ for attempts_left in reversed(range(num_attempts)):+ try:+ with session.get(url) as res:+ res.raise_for_status()+ if len(res.content) == 0:+ raise IOError("No data received")++ if len(res.content) < 8192:+ content_str = res.content.decode("utf-8")+ if "download_warning" in res.headers.get("Set-Cookie", ""):+ links = [html.unescape(link) for link in content_str.split('"') if "export=download" in link]+ if len(links) == 1:+ url = requests.compat.urljoin(url, links[0])+ raise IOError("Google Drive virus checker nag")+ if "Google Drive - Quota exceeded" in content_str:+ raise IOError("Google Drive download quota exceeded -- please try again later")++ match = re.search(r'filename="([^"]*)"', res.headers.get("Content-Disposition", ""))+ url_name = match[1] if match else url+ url_data = res.content+ if verbose:+ print(" done")+ break+ except KeyboardInterrupt:+ raise+ except:+ if not attempts_left:+ if verbose:+ print(" failed")+ raise+ if verbose:+ print(".", end="", flush=True)++ # Save to cache.+ if cache:+ safe_name = re.sub(r"[^0-9a-zA-Z-._]", "_", url_name)+ cache_file = os.path.join(cache_dir, url_md5 + "_" + safe_name)+ temp_file = os.path.join(cache_dir, "tmp_" + uuid.uuid4().hex + "_" + url_md5 + "_" + safe_name)+ os.makedirs(cache_dir, exist_ok=True)+ with open(temp_file, "wb") as f:+ f.write(url_data)+ os.replace(temp_file, cache_file) # atomic+ if return_filename:+ return cache_file++ # Return data as file object.+ assert not return_filename+ return io.BytesIO(url_data)diff --git a/src/stylegan2/torch_utils/__init__.py b/src/stylegan2/torch_utils/__init__.pynew file mode 100644index 0000000..ece0ea0--- /dev/null+++ b/src/stylegan2/torch_utils/__init__.py@@ -0,0 +1,9 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++# emptydiff --git a/src/stylegan2/torch_utils/custom_ops.py b/src/stylegan2/torch_utils/custom_ops.pynew file mode 100644index 0000000..4cc4e43--- /dev/null+++ b/src/stylegan2/torch_utils/custom_ops.py@@ -0,0 +1,126 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++import os+import glob+import torch+import torch.utils.cpp_extension+import importlib+import hashlib+import shutil+from pathlib import Path++from torch.utils.file_baton import FileBaton++#----------------------------------------------------------------------------+# Global options.++verbosity = 'brief' # Verbosity level: 'none', 'brief', 'full'++#----------------------------------------------------------------------------+# Internal helper funcs.++def _find_compiler_bindir():+ patterns = [+ 'C:/Program Files (x86)/Microsoft Visual Studio/*/Professional/VC/Tools/MSVC/*/bin/Hostx64/x64',+ 'C:/Program Files (x86)/Microsoft Visual Studio/*/BuildTools/VC/Tools/MSVC/*/bin/Hostx64/x64',+ 'C:/Program Files (x86)/Microsoft Visual Studio/*/Community/VC/Tools/MSVC/*/bin/Hostx64/x64',+ 'C:/Program Files (x86)/Microsoft Visual Studio */vc/bin',+ ]+ for pattern in patterns:+ matches = sorted(glob.glob(pattern))+ if len(matches):+ return matches[-1]+ return None++#----------------------------------------------------------------------------+# Main entry point for compiling and loading C++/CUDA plugins.++_cached_plugins = dict()++def get_plugin(module_name, sources, **build_kwargs):+ assert verbosity in ['none', 'brief', 'full']++ # Already cached?+ if module_name in _cached_plugins:+ return _cached_plugins[module_name]++ # Print status.+ if verbosity == 'full':+ print(f'Setting up PyTorch plugin "{module_name}"...')+ elif verbosity == 'brief':+ print(f'Setting up PyTorch plugin "{module_name}"... ', end='', flush=True)++ try: # pylint: disable=too-many-nested-blocks+ # Make sure we can find the necessary compiler binaries.+ if os.name == 'nt' and os.system("where cl.exe >nul 2>nul") != 0:+ compiler_bindir = _find_compiler_bindir()+ if compiler_bindir is None:+ raise RuntimeError(f'Could not find MSVC/GCC/CLANG installation on this computer. Check _find_compiler_bindir() in "{__file__}".')+ os.environ['PATH'] += ';' + compiler_bindir++ # Compile and load.+ verbose_build = (verbosity == 'full')++ # Incremental build md5sum trickery. Copies all the input source files+ # into a cached build directory under a combined md5 digest of the input+ # source files. Copying is done only if the combined digest has changed.+ # This keeps input file timestamps and filenames the same as in previous+ # extension builds, allowing for fast incremental rebuilds.+ #+ # This optimization is done only in case all the source files reside in+ # a single directory (just for simplicity) and if the TORCH_EXTENSIONS_DIR+ # environment variable is set (we take this as a signal that the user+ # actually cares about this.)+ source_dirs_set = set(os.path.dirname(source) for source in sources)+ if len(source_dirs_set) == 1 and ('TORCH_EXTENSIONS_DIR' in os.environ):+ all_source_files = sorted(list(x for x in Path(list(source_dirs_set)[0]).iterdir() if x.is_file()))++ # Compute a combined hash digest for all source files in the same+ # custom op directory (usually .cu, .cpp, .py and .h files).+ hash_md5 = hashlib.md5()+ for src in all_source_files:+ with open(src, 'rb') as f:+ hash_md5.update(f.read())+ build_dir = torch.utils.cpp_extension._get_build_directory(module_name, verbose=verbose_build) # pylint: disable=protected-access+ digest_build_dir = os.path.join(build_dir, hash_md5.hexdigest())++ if not os.path.isdir(digest_build_dir):+ os.makedirs(digest_build_dir, exist_ok=True)+ baton = FileBaton(os.path.join(digest_build_dir, 'lock'))+ if baton.try_acquire():+ try:+ for src in all_source_files:+ shutil.copyfile(src, os.path.join(digest_build_dir, os.path.basename(src)))+ finally:+ baton.release()+ else:+ # Someone else is copying source files under the digest dir,+ # wait until done and continue.+ baton.wait()+ digest_sources = [os.path.join(digest_build_dir, os.path.basename(x)) for x in sources]+ torch.utils.cpp_extension.load(name=module_name, build_directory=build_dir,+ verbose=verbose_build, sources=digest_sources, **build_kwargs)+ else:+ torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)+ module = importlib.import_module(module_name)++ except:+ if verbosity == 'brief':+ print('Failed!')+ raise++ # Print status and add to cache.+ if verbosity == 'full':+ print(f'Done setting up PyTorch plugin "{module_name}".')+ elif verbosity == 'brief':+ print('Done.')+ _cached_plugins[module_name] = module+ return module++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/misc.py b/src/stylegan2/torch_utils/misc.pynew file mode 100644index 0000000..7829f4d--- /dev/null+++ b/src/stylegan2/torch_utils/misc.py@@ -0,0 +1,262 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++import re+import contextlib+import numpy as np+import torch+import warnings+import dnnlib++#----------------------------------------------------------------------------+# Cached construction of constant tensors. Avoids CPU=>GPU copy when the+# same constant is used multiple times.++_constant_cache = dict()++def constant(value, shape=None, dtype=None, device=None, memory_format=None):+ value = np.asarray(value)+ if shape is not None:+ shape = tuple(shape)+ if dtype is None:+ dtype = torch.get_default_dtype()+ if device is None:+ device = torch.device('cpu')+ if memory_format is None:+ memory_format = torch.contiguous_format++ key = (value.shape, value.dtype, value.tobytes(), shape, dtype, device, memory_format)+ tensor = _constant_cache.get(key, None)+ if tensor is None:+ tensor = torch.as_tensor(value.copy(), dtype=dtype, device=device)+ if shape is not None:+ tensor, _ = torch.broadcast_tensors(tensor, torch.empty(shape))+ tensor = tensor.contiguous(memory_format=memory_format)+ _constant_cache[key] = tensor+ return tensor++#----------------------------------------------------------------------------+# Replace NaN/Inf with specified numerical values.++try:+ nan_to_num = torch.nan_to_num # 1.8.0a0+except AttributeError:+ def nan_to_num(input, nan=0.0, posinf=None, neginf=None, *, out=None): # pylint: disable=redefined-builtin+ assert isinstance(input, torch.Tensor)+ if posinf is None:+ posinf = torch.finfo(input.dtype).max+ if neginf is None:+ neginf = torch.finfo(input.dtype).min+ assert nan == 0+ return torch.clamp(input.unsqueeze(0).nansum(0), min=neginf, max=posinf, out=out)++#----------------------------------------------------------------------------+# Symbolic assert.++try:+ symbolic_assert = torch._assert # 1.8.0a0 # pylint: disable=protected-access+except AttributeError:+ symbolic_assert = torch.Assert # 1.7.0++#----------------------------------------------------------------------------+# Context manager to suppress known warnings in torch.jit.trace().++class suppress_tracer_warnings(warnings.catch_warnings):+ def __enter__(self):+ super().__enter__()+ warnings.simplefilter('ignore', category=torch.jit.TracerWarning)+ return self++#----------------------------------------------------------------------------+# Assert that the shape of a tensor matches the given list of integers.+# None indicates that the size of a dimension is allowed to vary.+# Performs symbolic assertion when used in torch.jit.trace().++def assert_shape(tensor, ref_shape):+ if tensor.ndim != len(ref_shape):+ raise AssertionError(f'Wrong number of dimensions: got {tensor.ndim}, expected {len(ref_shape)}')+ for idx, (size, ref_size) in enumerate(zip(tensor.shape, ref_shape)):+ if ref_size is None:+ pass+ elif isinstance(ref_size, torch.Tensor):+ with suppress_tracer_warnings(): # as_tensor results are registered as constants+ symbolic_assert(torch.equal(torch.as_tensor(size), ref_size), f'Wrong size for dimension {idx}')+ elif isinstance(size, torch.Tensor):+ with suppress_tracer_warnings(): # as_tensor results are registered as constants+ symbolic_assert(torch.equal(size, torch.as_tensor(ref_size)), f'Wrong size for dimension {idx}: expected {ref_size}')+ elif size != ref_size:+ raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}')++#----------------------------------------------------------------------------+# Function decorator that calls torch.autograd.profiler.record_function().++def profiled_function(fn):+ def decorator(*args, **kwargs):+ with torch.autograd.profiler.record_function(fn.__name__):+ return fn(*args, **kwargs)+ decorator.__name__ = fn.__name__+ return decorator++#----------------------------------------------------------------------------+# Sampler for torch.utils.data.DataLoader that loops over the dataset+# indefinitely, shuffling items as it goes.++class InfiniteSampler(torch.utils.data.Sampler):+ def __init__(self, dataset, rank=0, num_replicas=1, shuffle=True, seed=0, window_size=0.5):+ assert len(dataset) > 0+ assert num_replicas > 0+ assert 0 <= rank < num_replicas+ assert 0 <= window_size <= 1+ super().__init__(dataset)+ self.dataset = dataset+ self.rank = rank+ self.num_replicas = num_replicas+ self.shuffle = shuffle+ self.seed = seed+ self.window_size = window_size++ def __iter__(self):+ order = np.arange(len(self.dataset))+ rnd = None+ window = 0+ if self.shuffle:+ rnd = np.random.RandomState(self.seed)+ rnd.shuffle(order)+ window = int(np.rint(order.size * self.window_size))++ idx = 0+ while True:+ i = idx % order.size+ if idx % self.num_replicas == self.rank:+ yield order[i]+ if window >= 2:+ j = (i - rnd.randint(window)) % order.size+ order[i], order[j] = order[j], order[i]+ idx += 1++#----------------------------------------------------------------------------+# Utilities for operating with torch.nn.Module parameters and buffers.++def params_and_buffers(module):+ assert isinstance(module, torch.nn.Module)+ return list(module.parameters()) + list(module.buffers())++def named_params_and_buffers(module):+ assert isinstance(module, torch.nn.Module)+ return list(module.named_parameters()) + list(module.named_buffers())++def copy_params_and_buffers(src_module, dst_module, require_all=False):+ assert isinstance(src_module, torch.nn.Module)+ assert isinstance(dst_module, torch.nn.Module)+ src_tensors = {name: tensor for name, tensor in named_params_and_buffers(src_module)}+ for name, tensor in named_params_and_buffers(dst_module):+ assert (name in src_tensors) or (not require_all)+ if name in src_tensors:+ tensor.copy_(src_tensors[name].detach()).requires_grad_(tensor.requires_grad)++#----------------------------------------------------------------------------+# Context manager for easily enabling/disabling DistributedDataParallel+# synchronization.++def ddp_sync(module, sync):+ assert isinstance(module, torch.nn.Module)+ if sync or not isinstance(module, torch.nn.parallel.DistributedDataParallel):+ yield+ else:+ with module.no_sync():+ yield++#----------------------------------------------------------------------------+# Check DistributedDataParallel consistency across processes.++def check_ddp_consistency(module, ignore_regex=None):+ assert isinstance(module, torch.nn.Module)+ for name, tensor in named_params_and_buffers(module):+ fullname = type(module).__name__ + '.' + name+ if ignore_regex is not None and re.fullmatch(ignore_regex, fullname):+ continue+ tensor = tensor.detach()+ other = tensor.clone()+ torch.distributed.broadcast(tensor=other, src=0)+ assert (nan_to_num(tensor) == nan_to_num(other)).all(), fullname++#----------------------------------------------------------------------------+# Print summary table of module hierarchy.++def print_module_summary(module, inputs, max_nesting=3, skip_redundant=True):+ assert isinstance(module, torch.nn.Module)+ assert not isinstance(module, torch.jit.ScriptModule)+ assert isinstance(inputs, (tuple, list))++ # Register hooks.+ entries = []+ nesting = [0]+ def pre_hook(_mod, _inputs):+ nesting[0] += 1+ def post_hook(mod, _inputs, outputs):+ nesting[0] -= 1+ if nesting[0] <= max_nesting:+ outputs = list(outputs) if isinstance(outputs, (tuple, list)) else [outputs]+ outputs = [t for t in outputs if isinstance(t, torch.Tensor)]+ entries.append(dnnlib.EasyDict(mod=mod, outputs=outputs))+ hooks = [mod.register_forward_pre_hook(pre_hook) for mod in module.modules()]+ hooks += [mod.register_forward_hook(post_hook) for mod in module.modules()]++ # Run module.+ outputs = module(*inputs)+ for hook in hooks:+ hook.remove()++ # Identify unique outputs, parameters, and buffers.+ tensors_seen = set()+ for e in entries:+ e.unique_params = [t for t in e.mod.parameters() if id(t) not in tensors_seen]+ e.unique_buffers = [t for t in e.mod.buffers() if id(t) not in tensors_seen]+ e.unique_outputs = [t for t in e.outputs if id(t) not in tensors_seen]+ tensors_seen |= {id(t) for t in e.unique_params + e.unique_buffers + e.unique_outputs}++ # Filter out redundant entries.+ if skip_redundant:+ entries = [e for e in entries if len(e.unique_params) or len(e.unique_buffers) or len(e.unique_outputs)]++ # Construct table.+ rows = [[type(module).__name__, 'Parameters', 'Buffers', 'Output shape', 'Datatype']]+ rows += [['---'] * len(rows[0])]+ param_total = 0+ buffer_total = 0+ submodule_names = {mod: name for name, mod in module.named_modules()}+ for e in entries:+ name = '<top-level>' if e.mod is module else submodule_names[e.mod]+ param_size = sum(t.numel() for t in e.unique_params)+ buffer_size = sum(t.numel() for t in e.unique_buffers)+ output_shapes = [str(list(e.outputs[0].shape)) for t in e.outputs]+ output_dtypes = [str(t.dtype).split('.')[-1] for t in e.outputs]+ rows += [[+ name + (':0' if len(e.outputs) >= 2 else ''),+ str(param_size) if param_size else '-',+ str(buffer_size) if buffer_size else '-',+ (output_shapes + ['-'])[0],+ (output_dtypes + ['-'])[0],+ ]]+ for idx in range(1, len(e.outputs)):+ rows += [[name + f':{idx}', '-', '-', output_shapes[idx], output_dtypes[idx]]]+ param_total += param_size+ buffer_total += buffer_size+ rows += [['---'] * len(rows[0])]+ rows += [['Total', str(param_total), str(buffer_total), '-', '-']]++ # Print table.+ widths = [max(len(cell) for cell in column) for column in zip(*rows)]+ print()+ for row in rows:+ print(' '.join(cell + ' ' * (width - len(cell)) for cell, width in zip(row, widths)))+ print()+ return outputs++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/__init__.py b/src/stylegan2/torch_utils/ops/__init__.pynew file mode 100644index 0000000..ece0ea0--- /dev/null+++ b/src/stylegan2/torch_utils/ops/__init__.py@@ -0,0 +1,9 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++# emptydiff --git a/src/stylegan2/torch_utils/ops/bias_act.cpp b/src/stylegan2/torch_utils/ops/bias_act.cppnew file mode 100644index 0000000..5d2425d--- /dev/null+++ b/src/stylegan2/torch_utils/ops/bias_act.cpp@@ -0,0 +1,99 @@+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+//+// NVIDIA CORPORATION and its licensors retain all intellectual property+// and proprietary rights in and to this software, related documentation+// and any modifications thereto. Any use, reproduction, disclosure or+// distribution of this software and related documentation without an express+// license agreement from NVIDIA CORPORATION is strictly prohibited.++#include <torch/extension.h>+#include <ATen/cuda/CUDAContext.h>+#include <c10/cuda/CUDAGuard.h>+#include "bias_act.h"++//------------------------------------------------------------------------++static bool has_same_layout(torch::Tensor x, torch::Tensor y)+{+ if (x.dim() != y.dim())+ return false;+ for (int64_t i = 0; i < x.dim(); i++)+ {+ if (x.size(i) != y.size(i))+ return false;+ if (x.size(i) >= 2 && x.stride(i) != y.stride(i))+ return false;+ }+ return true;+}++//------------------------------------------------------------------------++static torch::Tensor bias_act(torch::Tensor x, torch::Tensor b, torch::Tensor xref, torch::Tensor yref, torch::Tensor dy, int grad, int dim, int act, float alpha, float gain, float clamp)+{+ // Validate arguments.+ TORCH_CHECK(x.is_cuda(), "x must reside on CUDA device");+ TORCH_CHECK(b.numel() == 0 || (b.dtype() == x.dtype() && b.device() == x.device()), "b must have the same dtype and device as x");+ TORCH_CHECK(xref.numel() == 0 || (xref.sizes() == x.sizes() && xref.dtype() == x.dtype() && xref.device() == x.device()), "xref must have the same shape, dtype, and device as x");+ TORCH_CHECK(yref.numel() == 0 || (yref.sizes() == x.sizes() && yref.dtype() == x.dtype() && yref.device() == x.device()), "yref must have the same shape, dtype, and device as x");+ TORCH_CHECK(dy.numel() == 0 || (dy.sizes() == x.sizes() && dy.dtype() == x.dtype() && dy.device() == x.device()), "dy must have the same dtype and device as x");+ TORCH_CHECK(x.numel() <= INT_MAX, "x is too large");+ TORCH_CHECK(b.dim() == 1, "b must have rank 1");+ TORCH_CHECK(b.numel() == 0 || (dim >= 0 && dim < x.dim()), "dim is out of bounds");+ TORCH_CHECK(b.numel() == 0 || b.numel() == x.size(dim), "b has wrong number of elements");+ TORCH_CHECK(grad >= 0, "grad must be non-negative");++ // Validate layout.+ TORCH_CHECK(x.is_non_overlapping_and_dense(), "x must be non-overlapping and dense");+ TORCH_CHECK(b.is_contiguous(), "b must be contiguous");+ TORCH_CHECK(xref.numel() == 0 || has_same_layout(xref, x), "xref must have the same layout as x");+ TORCH_CHECK(yref.numel() == 0 || has_same_layout(yref, x), "yref must have the same layout as x");+ TORCH_CHECK(dy.numel() == 0 || has_same_layout(dy, x), "dy must have the same layout as x");++ // Create output tensor.+ const at::cuda::OptionalCUDAGuard device_guard(device_of(x));+ torch::Tensor y = torch::empty_like(x);+ TORCH_CHECK(has_same_layout(y, x), "y must have the same layout as x");++ // Initialize CUDA kernel parameters.+ bias_act_kernel_params p;+ p.x = x.data_ptr();+ p.b = (b.numel()) ? b.data_ptr() : NULL;+ p.xref = (xref.numel()) ? xref.data_ptr() : NULL;+ p.yref = (yref.numel()) ? yref.data_ptr() : NULL;+ p.dy = (dy.numel()) ? dy.data_ptr() : NULL;+ p.y = y.data_ptr();+ p.grad = grad;+ p.act = act;+ p.alpha = alpha;+ p.gain = gain;+ p.clamp = clamp;+ p.sizeX = (int)x.numel();+ p.sizeB = (int)b.numel();+ p.stepB = (b.numel()) ? (int)x.stride(dim) : 1;++ // Choose CUDA kernel.+ void* kernel;+ AT_DISPATCH_FLOATING_TYPES_AND_HALF(x.scalar_type(), "upfirdn2d_cuda", [&]+ {+ kernel = choose_bias_act_kernel<scalar_t>(p);+ });+ TORCH_CHECK(kernel, "no CUDA kernel found for the specified activation func");++ // Launch CUDA kernel.+ p.loopX = 4;+ int blockSize = 4 * 32;+ int gridSize = (p.sizeX - 1) / (p.loopX * blockSize) + 1;+ void* args[] = {&p};+ AT_CUDA_CHECK(cudaLaunchKernel(kernel, gridSize, blockSize, args, 0, at::cuda::getCurrentCUDAStream()));+ return y;+}++//------------------------------------------------------------------------++PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)+{+ m.def("bias_act", &bias_act);+}++//------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/bias_act.cu b/src/stylegan2/torch_utils/ops/bias_act.cunew file mode 100644index 0000000..dd8fc47--- /dev/null+++ b/src/stylegan2/torch_utils/ops/bias_act.cu@@ -0,0 +1,173 @@+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+//+// NVIDIA CORPORATION and its licensors retain all intellectual property+// and proprietary rights in and to this software, related documentation+// and any modifications thereto. Any use, reproduction, disclosure or+// distribution of this software and related documentation without an express+// license agreement from NVIDIA CORPORATION is strictly prohibited.++#include <c10/util/Half.h>+#include "bias_act.h"++//------------------------------------------------------------------------+// Helpers.++template <class T> struct InternalType;+template <> struct InternalType<double> { typedef double scalar_t; };+template <> struct InternalType<float> { typedef float scalar_t; };+template <> struct InternalType<c10::Half> { typedef float scalar_t; };++//------------------------------------------------------------------------+// CUDA kernel.++template <class T, int A>+__global__ void bias_act_kernel(bias_act_kernel_params p)+{+ typedef typename InternalType<T>::scalar_t scalar_t;+ int G = p.grad;+ scalar_t alpha = (scalar_t)p.alpha;+ scalar_t gain = (scalar_t)p.gain;+ scalar_t clamp = (scalar_t)p.clamp;+ scalar_t one = (scalar_t)1;+ scalar_t two = (scalar_t)2;+ scalar_t expRange = (scalar_t)80;+ scalar_t halfExpRange = (scalar_t)40;+ scalar_t seluScale = (scalar_t)1.0507009873554804934193349852946;+ scalar_t seluAlpha = (scalar_t)1.6732632423543772848170429916717;++ // Loop over elements.+ int xi = blockIdx.x * p.loopX * blockDim.x + threadIdx.x;+ for (int loopIdx = 0; loopIdx < p.loopX && xi < p.sizeX; loopIdx++, xi += blockDim.x)+ {+ // Load.+ scalar_t x = (scalar_t)((const T*)p.x)[xi];+ scalar_t b = (p.b) ? (scalar_t)((const T*)p.b)[(xi / p.stepB) % p.sizeB] : 0;+ scalar_t xref = (p.xref) ? (scalar_t)((const T*)p.xref)[xi] : 0;+ scalar_t yref = (p.yref) ? (scalar_t)((const T*)p.yref)[xi] : 0;+ scalar_t dy = (p.dy) ? (scalar_t)((const T*)p.dy)[xi] : one;+ scalar_t yy = (gain != 0) ? yref / gain : 0;+ scalar_t y = 0;++ // Apply bias.+ ((G == 0) ? x : xref) += b;++ // linear+ if (A == 1)+ {+ if (G == 0) y = x;+ if (G == 1) y = x;+ }++ // relu+ if (A == 2)+ {+ if (G == 0) y = (x > 0) ? x : 0;+ if (G == 1) y = (yy > 0) ? x : 0;+ }++ // lrelu+ if (A == 3)+ {+ if (G == 0) y = (x > 0) ? x : x * alpha;+ if (G == 1) y = (yy > 0) ? x : x * alpha;+ }++ // tanh+ if (A == 4)+ {+ if (G == 0) { scalar_t c = exp(x); scalar_t d = one / c; y = (x < -expRange) ? -one : (x > expRange) ? one : (c - d) / (c + d); }+ if (G == 1) y = x * (one - yy * yy);+ if (G == 2) y = x * (one - yy * yy) * (-two * yy);+ }++ // sigmoid+ if (A == 5)+ {+ if (G == 0) y = (x < -expRange) ? 0 : one / (exp(-x) + one);+ if (G == 1) y = x * yy * (one - yy);+ if (G == 2) y = x * yy * (one - yy) * (one - two * yy);+ }++ // elu+ if (A == 6)+ {+ if (G == 0) y = (x >= 0) ? x : exp(x) - one;+ if (G == 1) y = (yy >= 0) ? x : x * (yy + one);+ if (G == 2) y = (yy >= 0) ? 0 : x * (yy + one);+ }++ // selu+ if (A == 7)+ {+ if (G == 0) y = (x >= 0) ? seluScale * x : (seluScale * seluAlpha) * (exp(x) - one);+ if (G == 1) y = (yy >= 0) ? x * seluScale : x * (yy + seluScale * seluAlpha);+ if (G == 2) y = (yy >= 0) ? 0 : x * (yy + seluScale * seluAlpha);+ }++ // softplus+ if (A == 8)+ {+ if (G == 0) y = (x > expRange) ? x : log(exp(x) + one);+ if (G == 1) y = x * (one - exp(-yy));+ if (G == 2) { scalar_t c = exp(-yy); y = x * c * (one - c); }+ }++ // swish+ if (A == 9)+ {+ if (G == 0)+ y = (x < -expRange) ? 0 : x / (exp(-x) + one);+ else+ {+ scalar_t c = exp(xref);+ scalar_t d = c + one;+ if (G == 1)+ y = (xref > halfExpRange) ? x : x * c * (xref + d) / (d * d);+ else+ y = (xref > halfExpRange) ? 0 : x * c * (xref * (two - d) + two * d) / (d * d * d);+ yref = (xref < -expRange) ? 0 : xref / (exp(-xref) + one) * gain;+ }+ }++ // Apply gain.+ y *= gain * dy;++ // Clamp.+ if (clamp >= 0)+ {+ if (G == 0)+ y = (y > -clamp & y < clamp) ? y : (y >= 0) ? clamp : -clamp;+ else+ y = (yref > -clamp & yref < clamp) ? y : 0;+ }++ // Store.+ ((T*)p.y)[xi] = (T)y;+ }+}++//------------------------------------------------------------------------+// CUDA kernel selection.++template <class T> void* choose_bias_act_kernel(const bias_act_kernel_params& p)+{+ if (p.act == 1) return (void*)bias_act_kernel<T, 1>;+ if (p.act == 2) return (void*)bias_act_kernel<T, 2>;+ if (p.act == 3) return (void*)bias_act_kernel<T, 3>;+ if (p.act == 4) return (void*)bias_act_kernel<T, 4>;+ if (p.act == 5) return (void*)bias_act_kernel<T, 5>;+ if (p.act == 6) return (void*)bias_act_kernel<T, 6>;+ if (p.act == 7) return (void*)bias_act_kernel<T, 7>;+ if (p.act == 8) return (void*)bias_act_kernel<T, 8>;+ if (p.act == 9) return (void*)bias_act_kernel<T, 9>;+ return NULL;+}++//------------------------------------------------------------------------+// Template specializations.++template void* choose_bias_act_kernel<double> (const bias_act_kernel_params& p);+template void* choose_bias_act_kernel<float> (const bias_act_kernel_params& p);+template void* choose_bias_act_kernel<c10::Half> (const bias_act_kernel_params& p);++//------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/bias_act.h b/src/stylegan2/torch_utils/ops/bias_act.hnew file mode 100644index 0000000..a32187e--- /dev/null+++ b/src/stylegan2/torch_utils/ops/bias_act.h@@ -0,0 +1,38 @@+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+//+// NVIDIA CORPORATION and its licensors retain all intellectual property+// and proprietary rights in and to this software, related documentation+// and any modifications thereto. Any use, reproduction, disclosure or+// distribution of this software and related documentation without an express+// license agreement from NVIDIA CORPORATION is strictly prohibited.++//------------------------------------------------------------------------+// CUDA kernel parameters.++struct bias_act_kernel_params+{+ const void* x; // [sizeX]+ const void* b; // [sizeB] or NULL+ const void* xref; // [sizeX] or NULL+ const void* yref; // [sizeX] or NULL+ const void* dy; // [sizeX] or NULL+ void* y; // [sizeX]++ int grad;+ int act;+ float alpha;+ float gain;+ float clamp;++ int sizeX;+ int sizeB;+ int stepB;+ int loopX;+};++//------------------------------------------------------------------------+// CUDA kernel selection.++template <class T> void* choose_bias_act_kernel(const bias_act_kernel_params& p);++//------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/bias_act.py b/src/stylegan2/torch_utils/ops/bias_act.pynew file mode 100644index 0000000..5dabe69--- /dev/null+++ b/src/stylegan2/torch_utils/ops/bias_act.py@@ -0,0 +1,213 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Custom PyTorch ops for efficient bias and activation."""++import os+import warnings+import numpy as np+import torch+import dnnlib+import traceback++from .. import custom_ops+from .. import misc++#----------------------------------------------------------------------------++activation_funcs = {+ 'linear': dnnlib.EasyDict(func=lambda x, **_: x, def_alpha=0, def_gain=1, cuda_idx=1, ref='', has_2nd_grad=False),+ 'relu': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.relu(x), def_alpha=0, def_gain=np.sqrt(2), cuda_idx=2, ref='y', has_2nd_grad=False),+ 'lrelu': dnnlib.EasyDict(func=lambda x, alpha, **_: torch.nn.functional.leaky_relu(x, alpha), def_alpha=0.2, def_gain=np.sqrt(2), cuda_idx=3, ref='y', has_2nd_grad=False),+ 'tanh': dnnlib.EasyDict(func=lambda x, **_: torch.tanh(x), def_alpha=0, def_gain=1, cuda_idx=4, ref='y', has_2nd_grad=True),+ 'sigmoid': dnnlib.EasyDict(func=lambda x, **_: torch.sigmoid(x), def_alpha=0, def_gain=1, cuda_idx=5, ref='y', has_2nd_grad=True),+ 'elu': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.elu(x), def_alpha=0, def_gain=1, cuda_idx=6, ref='y', has_2nd_grad=True),+ 'selu': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.selu(x), def_alpha=0, def_gain=1, cuda_idx=7, ref='y', has_2nd_grad=True),+ 'softplus': dnnlib.EasyDict(func=lambda x, **_: torch.nn.functional.softplus(x), def_alpha=0, def_gain=1, cuda_idx=8, ref='y', has_2nd_grad=True),+ 'swish': dnnlib.EasyDict(func=lambda x, **_: torch.sigmoid(x) * x, def_alpha=0, def_gain=np.sqrt(2), cuda_idx=9, ref='x', has_2nd_grad=True),+}++#----------------------------------------------------------------------------++_inited = False+_plugin = None+_null_tensor = torch.empty([0])++def _init():+ global _inited, _plugin+ if not _inited:+ _inited = True+ sources = ['bias_act.cpp', 'bias_act.cu']+ sources = [os.path.join(os.path.dirname(__file__), s) for s in sources]+ try:+ _plugin = custom_ops.get_plugin('bias_act_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])+ except:+ warnings.warn('Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())+ return _plugin is not None++#----------------------------------------------------------------------------++def bias_act(x, b=None, dim=1, act='linear', alpha=None, gain=None, clamp=None, impl='cuda'):+ r"""Fused bias and activation function.++ Adds bias `b` to activation tensor `x`, evaluates activation function `act`,+ and scales the result by `gain`. Each of the steps is optional. In most cases,+ the fused op is considerably more efficient than performing the same calculation+ using standard PyTorch ops. It supports first and second order gradients,+ but not third order gradients.++ Args:+ x: Input activation tensor. Can be of any shape.+ b: Bias vector, or `None` to disable. Must be a 1D tensor of the same type+ as `x`. The shape must be known, and it must match the dimension of `x`+ corresponding to `dim`.+ dim: The dimension in `x` corresponding to the elements of `b`.+ The value of `dim` is ignored if `b` is not specified.+ act: Name of the activation function to evaluate, or `"linear"` to disable.+ Can be e.g. `"relu"`, `"lrelu"`, `"tanh"`, `"sigmoid"`, `"swish"`, etc.+ See `activation_funcs` for a full list. `None` is not allowed.+ alpha: Shape parameter for the activation function, or `None` to use the default.+ gain: Scaling factor for the output tensor, or `None` to use default.+ See `activation_funcs` for the default scaling of each activation function.+ If unsure, consider specifying 1.+ clamp: Clamp the output values to `[-clamp, +clamp]`, or `None` to disable+ the clamping (default).+ impl: Name of the implementation to use. Can be `"ref"` or `"cuda"` (default).++ Returns:+ Tensor of the same shape and datatype as `x`.+ """+ assert isinstance(x, torch.Tensor)+ assert impl in ['ref', 'cuda']+ if impl == 'cuda' and x.device.type == 'cuda' and _init():+ return _bias_act_cuda(dim=dim, act=act, alpha=alpha, gain=gain, clamp=clamp).apply(x, b)+ return _bias_act_ref(x=x, b=b, dim=dim, act=act, alpha=alpha, gain=gain, clamp=clamp)++#----------------------------------------------------------------------------+[email protected]_function+def _bias_act_ref(x, b=None, dim=1, act='linear', alpha=None, gain=None, clamp=None):+ """Slow reference implementation of `bias_act()` using standard TensorFlow ops.+ """+ assert isinstance(x, torch.Tensor)+ assert clamp is None or clamp >= 0+ spec = activation_funcs[act]+ alpha = float(alpha if alpha is not None else spec.def_alpha)+ gain = float(gain if gain is not None else spec.def_gain)+ clamp = float(clamp if clamp is not None else -1)++ # Add bias.+ if b is not None:+ assert isinstance(b, torch.Tensor) and b.ndim == 1+ assert 0 <= dim < x.ndim+ assert b.shape[0] == x.shape[dim]+ x = x + b.reshape([-1 if i == dim else 1 for i in range(x.ndim)])++ # Evaluate activation function.+ alpha = float(alpha)+ x = spec.func(x, alpha=alpha)++ # Scale by gain.+ gain = float(gain)+ if gain != 1:+ x = x * gain++ # Clamp.+ if clamp >= 0:+ x = x.clamp(-clamp, clamp) # pylint: disable=invalid-unary-operand-type+ return x++#----------------------------------------------------------------------------++_bias_act_cuda_cache = dict()++def _bias_act_cuda(dim=1, act='linear', alpha=None, gain=None, clamp=None):+ """Fast CUDA implementation of `bias_act()` using custom ops.+ """+ # Parse arguments.+ assert clamp is None or clamp >= 0+ spec = activation_funcs[act]+ alpha = float(alpha if alpha is not None else spec.def_alpha)+ gain = float(gain if gain is not None else spec.def_gain)+ clamp = float(clamp if clamp is not None else -1)++ # Lookup from cache.+ key = (dim, act, alpha, gain, clamp)+ if key in _bias_act_cuda_cache:+ return _bias_act_cuda_cache[key]++ # Forward op.+ class BiasActCuda(torch.autograd.Function):+ @staticmethod+ def forward(ctx, x, b): # pylint: disable=arguments-differ+ ctx.memory_format = torch.channels_last if x.ndim > 2 and x.stride()[1] == 1 else torch.contiguous_format+ x = x.contiguous(memory_format=ctx.memory_format)+ b = b.contiguous() if b is not None else _null_tensor+ x = x.to(b.dtype)+ y = x+ if act != 'linear' or gain != 1 or clamp >= 0 or b is not _null_tensor:+ y = _plugin.bias_act(x, b, _null_tensor, _null_tensor, _null_tensor, 0, dim, spec.cuda_idx, alpha, gain, clamp)+ ctx.save_for_backward(+ x if 'x' in spec.ref or spec.has_2nd_grad else _null_tensor,+ b if 'x' in spec.ref or spec.has_2nd_grad else _null_tensor,+ y if 'y' in spec.ref else _null_tensor)+ return y++ @staticmethod+ def backward(ctx, dy): # pylint: disable=arguments-differ+ dy = dy.contiguous(memory_format=ctx.memory_format)+ x, b, y = ctx.saved_tensors+ dx = None+ db = None++ if ctx.needs_input_grad[0] or ctx.needs_input_grad[1]:+ dx = dy+ if act != 'linear' or gain != 1 or clamp >= 0:+ dx = BiasActCudaGrad.apply(dy, x, b, y)++ if ctx.needs_input_grad[1]:+ db = dx.sum([i for i in range(dx.ndim) if i != dim])++ return dx, db++ # Backward op.+ class BiasActCudaGrad(torch.autograd.Function):+ @staticmethod+ def forward(ctx, dy, x, b, y): # pylint: disable=arguments-differ+ ctx.memory_format = torch.channels_last if dy.ndim > 2 and dy.stride()[1] == 1 else torch.contiguous_format+ dx = _plugin.bias_act(dy, b, x, y, _null_tensor, 1, dim, spec.cuda_idx, alpha, gain, clamp)+ ctx.save_for_backward(+ dy if spec.has_2nd_grad else _null_tensor,+ x, b, y)+ return dx++ @staticmethod+ def backward(ctx, d_dx): # pylint: disable=arguments-differ+ d_dx = d_dx.contiguous(memory_format=ctx.memory_format)+ dy, x, b, y = ctx.saved_tensors+ d_dy = None+ d_x = None+ d_b = None+ d_y = None++ if ctx.needs_input_grad[0]:+ d_dy = BiasActCudaGrad.apply(d_dx, x, b, y)++ if spec.has_2nd_grad and (ctx.needs_input_grad[1] or ctx.needs_input_grad[2]):+ d_x = _plugin.bias_act(d_dx, b, x, y, dy, 2, dim, spec.cuda_idx, alpha, gain, clamp)++ if spec.has_2nd_grad and ctx.needs_input_grad[2]:+ d_b = d_x.sum([i for i in range(d_x.ndim) if i != dim])++ return d_dy, d_x, d_b, d_y++ # Add to cache.+ _bias_act_cuda_cache[key] = BiasActCuda+ return BiasActCuda++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/conv2d_gradfix.py b/src/stylegan2/torch_utils/ops/conv2d_gradfix.pynew file mode 100644index 0000000..e95e10d--- /dev/null+++ b/src/stylegan2/torch_utils/ops/conv2d_gradfix.py@@ -0,0 +1,170 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Custom replacement for `torch.nn.functional.conv2d` that supports+arbitrarily high order gradients with zero performance penalty."""++import warnings+import contextlib+import torch++# pylint: disable=redefined-builtin+# pylint: disable=arguments-differ+# pylint: disable=protected-access++#----------------------------------------------------------------------------++enabled = False # Enable the custom op by setting this to true.+weight_gradients_disabled = False # Forcefully disable computation of gradients with respect to the weights.++def no_weight_gradients():+ global weight_gradients_disabled+ old = weight_gradients_disabled+ weight_gradients_disabled = True+ yield+ weight_gradients_disabled = old++#----------------------------------------------------------------------------++def conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):+ if _should_use_custom_op(input):+ return _conv2d_gradfix(transpose=False, weight_shape=weight.shape, stride=stride, padding=padding, output_padding=0, dilation=dilation, groups=groups).apply(input, weight, bias)+ return torch.nn.functional.conv2d(input=input, weight=weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)++def conv_transpose2d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1):+ if _should_use_custom_op(input):+ return _conv2d_gradfix(transpose=True, weight_shape=weight.shape, stride=stride, padding=padding, output_padding=output_padding, groups=groups, dilation=dilation).apply(input, weight, bias)+ return torch.nn.functional.conv_transpose2d(input=input, weight=weight, bias=bias, stride=stride, padding=padding, output_padding=output_padding, groups=groups, dilation=dilation)++#----------------------------------------------------------------------------++def _should_use_custom_op(input):+ assert isinstance(input, torch.Tensor)+ if (not enabled) or (not torch.backends.cudnn.enabled):+ return False+ if input.device.type != 'cuda':+ return False+ if any(torch.__version__.startswith(x) for x in ['1.7.', '1.8.', '1.9']):+ return True+ warnings.warn(f'conv2d_gradfix not supported on PyTorch {torch.__version__}. Falling back to torch.nn.functional.conv2d().')+ return False++def _tuple_of_ints(xs, ndim):+ xs = tuple(xs) if isinstance(xs, (tuple, list)) else (xs,) * ndim+ assert len(xs) == ndim+ assert all(isinstance(x, int) for x in xs)+ return xs++#----------------------------------------------------------------------------++_conv2d_gradfix_cache = dict()++def _conv2d_gradfix(transpose, weight_shape, stride, padding, output_padding, dilation, groups):+ # Parse arguments.+ ndim = 2+ weight_shape = tuple(weight_shape)+ stride = _tuple_of_ints(stride, ndim)+ padding = _tuple_of_ints(padding, ndim)+ output_padding = _tuple_of_ints(output_padding, ndim)+ dilation = _tuple_of_ints(dilation, ndim)++ # Lookup from cache.+ key = (transpose, weight_shape, stride, padding, output_padding, dilation, groups)+ if key in _conv2d_gradfix_cache:+ return _conv2d_gradfix_cache[key]++ # Validate arguments.+ assert groups >= 1+ assert len(weight_shape) == ndim + 2+ assert all(stride[i] >= 1 for i in range(ndim))+ assert all(padding[i] >= 0 for i in range(ndim))+ assert all(dilation[i] >= 0 for i in range(ndim))+ if not transpose:+ assert all(output_padding[i] == 0 for i in range(ndim))+ else: # transpose+ assert all(0 <= output_padding[i] < max(stride[i], dilation[i]) for i in range(ndim))++ # Helpers.+ common_kwargs = dict(stride=stride, padding=padding, dilation=dilation, groups=groups)+ def calc_output_padding(input_shape, output_shape):+ if transpose:+ return [0, 0]+ return [+ input_shape[i + 2]+ - (output_shape[i + 2] - 1) * stride[i]+ - (1 - 2 * padding[i])+ - dilation[i] * (weight_shape[i + 2] - 1)+ for i in range(ndim)+ ]++ # Forward & backward.+ class Conv2d(torch.autograd.Function):+ @staticmethod+ def forward(ctx, input, weight, bias):+ assert weight.shape == weight_shape+ if not transpose:+ output = torch.nn.functional.conv2d(input=input, weight=weight, bias=bias, **common_kwargs)+ else: # transpose+ output = torch.nn.functional.conv_transpose2d(input=input, weight=weight, bias=bias, output_padding=output_padding, **common_kwargs)+ ctx.save_for_backward(input, weight)+ return output++ @staticmethod+ def backward(ctx, grad_output):+ input, weight = ctx.saved_tensors+ grad_input = None+ grad_weight = None+ grad_bias = None++ if ctx.needs_input_grad[0]:+ p = calc_output_padding(input_shape=input.shape, output_shape=grad_output.shape)+ grad_input = _conv2d_gradfix(transpose=(not transpose), weight_shape=weight_shape, output_padding=p, **common_kwargs).apply(grad_output, weight, None)+ assert grad_input.shape == input.shape++ if ctx.needs_input_grad[1] and not weight_gradients_disabled:+ grad_weight = Conv2dGradWeight.apply(grad_output, input)+ assert grad_weight.shape == weight_shape++ if ctx.needs_input_grad[2]:+ grad_bias = grad_output.sum([0, 2, 3])++ return grad_input, grad_weight, grad_bias++ # Gradient with respect to the weights.+ class Conv2dGradWeight(torch.autograd.Function):+ @staticmethod+ def forward(ctx, grad_output, input):+ op = torch._C._jit_get_operation('aten::cudnn_convolution_backward_weight' if not transpose else 'aten::cudnn_convolution_transpose_backward_weight')+ flags = [torch.backends.cudnn.benchmark, torch.backends.cudnn.deterministic, torch.backends.cudnn.allow_tf32]+ grad_weight = op(weight_shape, grad_output, input, padding, stride, dilation, groups, *flags)+ assert grad_weight.shape == weight_shape+ ctx.save_for_backward(grad_output, input)+ return grad_weight++ @staticmethod+ def backward(ctx, grad2_grad_weight):+ grad_output, input = ctx.saved_tensors+ grad2_grad_output = None+ grad2_input = None++ if ctx.needs_input_grad[0]:+ grad2_grad_output = Conv2d.apply(input, grad2_grad_weight, None)+ assert grad2_grad_output.shape == grad_output.shape++ if ctx.needs_input_grad[1]:+ p = calc_output_padding(input_shape=input.shape, output_shape=grad_output.shape)+ grad2_input = _conv2d_gradfix(transpose=(not transpose), weight_shape=weight_shape, output_padding=p, **common_kwargs).apply(grad_output, grad2_grad_weight, None)+ assert grad2_input.shape == input.shape++ return grad2_grad_output, grad2_input++ _conv2d_gradfix_cache[key] = Conv2d+ return Conv2d++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/conv2d_resample.py b/src/stylegan2/torch_utils/ops/conv2d_resample.pynew file mode 100644index 0000000..cd47507--- /dev/null+++ b/src/stylegan2/torch_utils/ops/conv2d_resample.py@@ -0,0 +1,156 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""2D convolution with optional up/downsampling."""++import torch++from .. import misc+from . import conv2d_gradfix+from . import upfirdn2d+from .upfirdn2d import _parse_padding+from .upfirdn2d import _get_filter_size++#----------------------------------------------------------------------------++def _get_weight_shape(w):+ with misc.suppress_tracer_warnings(): # this value will be treated as a constant+ shape = [int(sz) for sz in w.shape]+ misc.assert_shape(w, shape)+ return shape++#----------------------------------------------------------------------------++def _conv2d_wrapper(x, w, stride=1, padding=0, groups=1, transpose=False, flip_weight=True):+ """Wrapper for the underlying `conv2d()` and `conv_transpose2d()` implementations.+ """+ out_channels, in_channels_per_group, kh, kw = _get_weight_shape(w)++ # Flip weight if requested.+ if not flip_weight: # conv2d() actually performs correlation (flip_weight=True) not convolution (flip_weight=False).+ w = w.flip([2, 3])++ # Workaround performance pitfall in cuDNN 8.0.5, triggered when using+ # 1x1 kernel + memory_format=channels_last + less than 64 channels.+ if kw == 1 and kh == 1 and stride == 1 and padding in [0, [0, 0], (0, 0)] and not transpose:+ if x.stride()[1] == 1 and min(out_channels, in_channels_per_group) < 64:+ if out_channels <= 4 and groups == 1:+ in_shape = x.shape+ x = w.squeeze(3).squeeze(2) @ x.reshape([in_shape[0], in_channels_per_group, -1])+ x = x.reshape([in_shape[0], out_channels, in_shape[2], in_shape[3]])+ else:+ x = x.to(memory_format=torch.contiguous_format)+ w = w.to(memory_format=torch.contiguous_format)+ x = conv2d_gradfix.conv2d(x, w, groups=groups)+ return x.to(memory_format=torch.channels_last)++ # Otherwise => execute using conv2d_gradfix.+ op = conv2d_gradfix.conv_transpose2d if transpose else conv2d_gradfix.conv2d+ return op(x, w, stride=stride, padding=padding, groups=groups)++#----------------------------------------------------------------------------+[email protected]_function+def conv2d_resample(x, w, f=None, up=1, down=1, padding=0, groups=1, flip_weight=True, flip_filter=False):+ r"""2D convolution with optional up/downsampling.++ Padding is performed only once at the beginning, not between the operations.++ Args:+ x: Input tensor of shape+ `[batch_size, in_channels, in_height, in_width]`.+ w: Weight tensor of shape+ `[out_channels, in_channels//groups, kernel_height, kernel_width]`.+ f: Low-pass filter for up/downsampling. Must be prepared beforehand by+ calling upfirdn2d.setup_filter(). None = identity (default).+ up: Integer upsampling factor (default: 1).+ down: Integer downsampling factor (default: 1).+ padding: Padding with respect to the upsampled image. Can be a single number+ or a list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`+ (default: 0).+ groups: Split input channels into N groups (default: 1).+ flip_weight: False = convolution, True = correlation (default: True).+ flip_filter: False = convolution, True = correlation (default: False).++ Returns:+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.+ """+ # Validate arguments.+ assert isinstance(x, torch.Tensor) and (x.ndim == 4)+ assert isinstance(w, torch.Tensor) and (w.ndim == 4) and (w.dtype == x.dtype)+ assert f is None or (isinstance(f, torch.Tensor) and f.ndim in [1, 2] and f.dtype == torch.float32)+ assert isinstance(up, int) and (up >= 1)+ assert isinstance(down, int) and (down >= 1)+ assert isinstance(groups, int) and (groups >= 1)+ out_channels, in_channels_per_group, kh, kw = _get_weight_shape(w)+ fw, fh = _get_filter_size(f)+ px0, px1, py0, py1 = _parse_padding(padding)++ # Adjust padding to account for up/downsampling.+ if up > 1:+ px0 += (fw + up - 1) // 2+ px1 += (fw - up) // 2+ py0 += (fh + up - 1) // 2+ py1 += (fh - up) // 2+ if down > 1:+ px0 += (fw - down + 1) // 2+ px1 += (fw - down) // 2+ py0 += (fh - down + 1) // 2+ py1 += (fh - down) // 2++ # Fast path: 1x1 convolution with downsampling only => downsample first, then convolve.+ if kw == 1 and kh == 1 and (down > 1 and up == 1):+ x = upfirdn2d.upfirdn2d(x=x, f=f, down=down, padding=[px0,px1,py0,py1], flip_filter=flip_filter)+ x = _conv2d_wrapper(x=x, w=w, groups=groups, flip_weight=flip_weight)+ return x++ # Fast path: 1x1 convolution with upsampling only => convolve first, then upsample.+ if kw == 1 and kh == 1 and (up > 1 and down == 1):+ x = _conv2d_wrapper(x=x, w=w, groups=groups, flip_weight=flip_weight)+ x = upfirdn2d.upfirdn2d(x=x, f=f, up=up, padding=[px0,px1,py0,py1], gain=up**2, flip_filter=flip_filter)+ return x++ # Fast path: downsampling only => use strided convolution.+ if down > 1 and up == 1:+ x = upfirdn2d.upfirdn2d(x=x, f=f, padding=[px0,px1,py0,py1], flip_filter=flip_filter)+ x = _conv2d_wrapper(x=x, w=w, stride=down, groups=groups, flip_weight=flip_weight)+ return x++ # Fast path: upsampling with optional downsampling => use transpose strided convolution.+ if up > 1:+ if groups == 1:+ w = w.transpose(0, 1)+ else:+ w = w.reshape(groups, out_channels // groups, in_channels_per_group, kh, kw)+ w = w.transpose(1, 2)+ w = w.reshape(groups * in_channels_per_group, out_channels // groups, kh, kw)+ px0 -= kw - 1+ px1 -= kw - up+ py0 -= kh - 1+ py1 -= kh - up+ pxt = max(min(-px0, -px1), 0)+ pyt = max(min(-py0, -py1), 0)+ x = _conv2d_wrapper(x=x, w=w, stride=up, padding=[pyt,pxt], groups=groups, transpose=True, flip_weight=(not flip_weight))+ x = upfirdn2d.upfirdn2d(x=x, f=f, padding=[px0+pxt,px1+pxt,py0+pyt,py1+pyt], gain=up**2, flip_filter=flip_filter)+ if down > 1:+ x = upfirdn2d.upfirdn2d(x=x, f=f, down=down, flip_filter=flip_filter)+ return x++ # Fast path: no up/downsampling, padding supported by the underlying implementation => use plain conv2d.+ if up == 1 and down == 1:+ if px0 == px1 and py0 == py1 and px0 >= 0 and py0 >= 0:+ return _conv2d_wrapper(x=x, w=w, padding=[py0,px0], groups=groups, flip_weight=flip_weight)++ # Fallback: Generic reference implementation.+ x = upfirdn2d.upfirdn2d(x=x, f=(f if up > 1 else None), up=up, padding=[px0,px1,py0,py1], gain=up**2, flip_filter=flip_filter)+ x = _conv2d_wrapper(x=x, w=w, groups=groups, flip_weight=flip_weight)+ if down > 1:+ x = upfirdn2d.upfirdn2d(x=x, f=f, down=down, flip_filter=flip_filter)+ return x++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/fma.py b/src/stylegan2/torch_utils/ops/fma.pynew file mode 100644index 0000000..2eeac58--- /dev/null+++ b/src/stylegan2/torch_utils/ops/fma.py@@ -0,0 +1,60 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Fused multiply-add, with slightly faster gradients than `torch.addcmul()`."""++import torch++#----------------------------------------------------------------------------++def fma(a, b, c): # => a * b + c+ return _FusedMultiplyAdd.apply(a, b, c)++#----------------------------------------------------------------------------++class _FusedMultiplyAdd(torch.autograd.Function): # a * b + c+ @staticmethod+ def forward(ctx, a, b, c): # pylint: disable=arguments-differ+ out = torch.addcmul(c, a, b)+ ctx.save_for_backward(a, b)+ ctx.c_shape = c.shape+ return out++ @staticmethod+ def backward(ctx, dout): # pylint: disable=arguments-differ+ a, b = ctx.saved_tensors+ c_shape = ctx.c_shape+ da = None+ db = None+ dc = None++ if ctx.needs_input_grad[0]:+ da = _unbroadcast(dout * b, a.shape)++ if ctx.needs_input_grad[1]:+ db = _unbroadcast(dout * a, b.shape)++ if ctx.needs_input_grad[2]:+ dc = _unbroadcast(dout, c_shape)++ return da, db, dc++#----------------------------------------------------------------------------++def _unbroadcast(x, shape):+ extra_dims = x.ndim - len(shape)+ assert extra_dims >= 0+ dim = [i for i in range(x.ndim) if x.shape[i] > 1 and (i < extra_dims or shape[i - extra_dims] == 1)]+ if len(dim):+ x = x.sum(dim=dim, keepdim=True)+ if extra_dims:+ x = x.reshape(-1, *x.shape[extra_dims+1:])+ assert x.shape == shape+ return x++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/grid_sample_gradfix.py b/src/stylegan2/torch_utils/ops/grid_sample_gradfix.pynew file mode 100644index 0000000..ca6b341--- /dev/null+++ b/src/stylegan2/torch_utils/ops/grid_sample_gradfix.py@@ -0,0 +1,83 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Custom replacement for `torch.nn.functional.grid_sample` that+supports arbitrarily high order gradients between the input and output.+Only works on 2D images and assumes+`mode='bilinear'`, `padding_mode='zeros'`, `align_corners=False`."""++import warnings+import torch++# pylint: disable=redefined-builtin+# pylint: disable=arguments-differ+# pylint: disable=protected-access++#----------------------------------------------------------------------------++enabled = False # Enable the custom op by setting this to true.++#----------------------------------------------------------------------------++def grid_sample(input, grid):+ if _should_use_custom_op():+ return _GridSample2dForward.apply(input, grid)+ return torch.nn.functional.grid_sample(input=input, grid=grid, mode='bilinear', padding_mode='zeros', align_corners=False)++#----------------------------------------------------------------------------++def _should_use_custom_op():+ if not enabled:+ return False+ if any(torch.__version__.startswith(x) for x in ['1.7.', '1.8.', '1.9']):+ return True+ warnings.warn(f'grid_sample_gradfix not supported on PyTorch {torch.__version__}. Falling back to torch.nn.functional.grid_sample().')+ return False++#----------------------------------------------------------------------------++class _GridSample2dForward(torch.autograd.Function):+ @staticmethod+ def forward(ctx, input, grid):+ assert input.ndim == 4+ assert grid.ndim == 4+ output = torch.nn.functional.grid_sample(input=input, grid=grid, mode='bilinear', padding_mode='zeros', align_corners=False)+ ctx.save_for_backward(input, grid)+ return output++ @staticmethod+ def backward(ctx, grad_output):+ input, grid = ctx.saved_tensors+ grad_input, grad_grid = _GridSample2dBackward.apply(grad_output, input, grid)+ return grad_input, grad_grid++#----------------------------------------------------------------------------++class _GridSample2dBackward(torch.autograd.Function):+ @staticmethod+ def forward(ctx, grad_output, input, grid):+ op = torch._C._jit_get_operation('aten::grid_sampler_2d_backward')+ grad_input, grad_grid = op(grad_output, input, grid, 0, 0, False)+ ctx.save_for_backward(grid)+ return grad_input, grad_grid++ @staticmethod+ def backward(ctx, grad2_grad_input, grad2_grad_grid):+ _ = grad2_grad_grid # unused+ grid, = ctx.saved_tensors+ grad2_grad_output = None+ grad2_input = None+ grad2_grid = None++ if ctx.needs_input_grad[0]:+ grad2_grad_output = _GridSample2dForward.apply(grad2_grad_input, grid)++ assert not ctx.needs_input_grad[2]+ return grad2_grad_output, grad2_input, grad2_grid++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.cpp b/src/stylegan2/torch_utils/ops/upfirdn2d.cppnew file mode 100644index 0000000..2d7177f--- /dev/null+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.cpp@@ -0,0 +1,103 @@+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+//+// NVIDIA CORPORATION and its licensors retain all intellectual property+// and proprietary rights in and to this software, related documentation+// and any modifications thereto. Any use, reproduction, disclosure or+// distribution of this software and related documentation without an express+// license agreement from NVIDIA CORPORATION is strictly prohibited.++#include <torch/extension.h>+#include <ATen/cuda/CUDAContext.h>+#include <c10/cuda/CUDAGuard.h>+#include "upfirdn2d.h"++//------------------------------------------------------------------------++static torch::Tensor upfirdn2d(torch::Tensor x, torch::Tensor f, int upx, int upy, int downx, int downy, int padx0, int padx1, int pady0, int pady1, bool flip, float gain)+{+ // Validate arguments.+ TORCH_CHECK(x.is_cuda(), "x must reside on CUDA device");+ TORCH_CHECK(f.device() == x.device(), "f must reside on the same device as x");+ TORCH_CHECK(f.dtype() == torch::kFloat, "f must be float32");+ TORCH_CHECK(x.numel() <= INT_MAX, "x is too large");+ TORCH_CHECK(f.numel() <= INT_MAX, "f is too large");+ TORCH_CHECK(x.dim() == 4, "x must be rank 4");+ TORCH_CHECK(f.dim() == 2, "f must be rank 2");+ TORCH_CHECK(f.size(0) >= 1 && f.size(1) >= 1, "f must be at least 1x1");+ TORCH_CHECK(upx >= 1 && upy >= 1, "upsampling factor must be at least 1");+ TORCH_CHECK(downx >= 1 && downy >= 1, "downsampling factor must be at least 1");++ // Create output tensor.+ const at::cuda::OptionalCUDAGuard device_guard(device_of(x));+ int outW = ((int)x.size(3) * upx + padx0 + padx1 - (int)f.size(1) + downx) / downx;+ int outH = ((int)x.size(2) * upy + pady0 + pady1 - (int)f.size(0) + downy) / downy;+ TORCH_CHECK(outW >= 1 && outH >= 1, "output must be at least 1x1");+ torch::Tensor y = torch::empty({x.size(0), x.size(1), outH, outW}, x.options(), x.suggest_memory_format());+ TORCH_CHECK(y.numel() <= INT_MAX, "output is too large");++ // Initialize CUDA kernel parameters.+ upfirdn2d_kernel_params p;+ p.x = x.data_ptr();+ p.f = f.data_ptr<float>();+ p.y = y.data_ptr();+ p.up = make_int2(upx, upy);+ p.down = make_int2(downx, downy);+ p.pad0 = make_int2(padx0, pady0);+ p.flip = (flip) ? 1 : 0;+ p.gain = gain;+ p.inSize = make_int4((int)x.size(3), (int)x.size(2), (int)x.size(1), (int)x.size(0));+ p.inStride = make_int4((int)x.stride(3), (int)x.stride(2), (int)x.stride(1), (int)x.stride(0));+ p.filterSize = make_int2((int)f.size(1), (int)f.size(0));+ p.filterStride = make_int2((int)f.stride(1), (int)f.stride(0));+ p.outSize = make_int4((int)y.size(3), (int)y.size(2), (int)y.size(1), (int)y.size(0));+ p.outStride = make_int4((int)y.stride(3), (int)y.stride(2), (int)y.stride(1), (int)y.stride(0));+ p.sizeMajor = (p.inStride.z == 1) ? p.inSize.w : p.inSize.w * p.inSize.z;+ p.sizeMinor = (p.inStride.z == 1) ? p.inSize.z : 1;++ // Choose CUDA kernel.+ upfirdn2d_kernel_spec spec;+ AT_DISPATCH_FLOATING_TYPES_AND_HALF(x.scalar_type(), "upfirdn2d_cuda", [&]+ {+ spec = choose_upfirdn2d_kernel<scalar_t>(p);+ });++ // Set looping options.+ p.loopMajor = (p.sizeMajor - 1) / 16384 + 1;+ p.loopMinor = spec.loopMinor;+ p.loopX = spec.loopX;+ p.launchMinor = (p.sizeMinor - 1) / p.loopMinor + 1;+ p.launchMajor = (p.sizeMajor - 1) / p.loopMajor + 1;++ // Compute grid size.+ dim3 blockSize, gridSize;+ if (spec.tileOutW < 0) // large+ {+ blockSize = dim3(4, 32, 1);+ gridSize = dim3(+ ((p.outSize.y - 1) / blockSize.x + 1) * p.launchMinor,+ (p.outSize.x - 1) / (blockSize.y * p.loopX) + 1,+ p.launchMajor);+ }+ else // small+ {+ blockSize = dim3(256, 1, 1);+ gridSize = dim3(+ ((p.outSize.y - 1) / spec.tileOutH + 1) * p.launchMinor,+ (p.outSize.x - 1) / (spec.tileOutW * p.loopX) + 1,+ p.launchMajor);+ }++ // Launch CUDA kernel.+ void* args[] = {&p};+ AT_CUDA_CHECK(cudaLaunchKernel(spec.kernel, gridSize, blockSize, args, 0, at::cuda::getCurrentCUDAStream()));+ return y;+}++//------------------------------------------------------------------------++PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)+{+ m.def("upfirdn2d", &upfirdn2d);+}++//------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.cu b/src/stylegan2/torch_utils/ops/upfirdn2d.cunew file mode 100644index 0000000..ebdd987--- /dev/null+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.cu@@ -0,0 +1,350 @@+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+//+// NVIDIA CORPORATION and its licensors retain all intellectual property+// and proprietary rights in and to this software, related documentation+// and any modifications thereto. Any use, reproduction, disclosure or+// distribution of this software and related documentation without an express+// license agreement from NVIDIA CORPORATION is strictly prohibited.++#include <c10/util/Half.h>+#include "upfirdn2d.h"++//------------------------------------------------------------------------+// Helpers.++template <class T> struct InternalType;+template <> struct InternalType<double> { typedef double scalar_t; };+template <> struct InternalType<float> { typedef float scalar_t; };+template <> struct InternalType<c10::Half> { typedef float scalar_t; };++static __device__ __forceinline__ int floor_div(int a, int b)+{+ int t = 1 - a / b;+ return (a + t * b) / b - t;+}++//------------------------------------------------------------------------+// Generic CUDA implementation for large filters.++template <class T> static __global__ void upfirdn2d_kernel_large(upfirdn2d_kernel_params p)+{+ typedef typename InternalType<T>::scalar_t scalar_t;++ // Calculate thread index.+ int minorBase = blockIdx.x * blockDim.x + threadIdx.x;+ int outY = minorBase / p.launchMinor;+ minorBase -= outY * p.launchMinor;+ int outXBase = blockIdx.y * p.loopX * blockDim.y + threadIdx.y;+ int majorBase = blockIdx.z * p.loopMajor;+ if (outXBase >= p.outSize.x | outY >= p.outSize.y | majorBase >= p.sizeMajor)+ return;++ // Setup Y receptive field.+ int midY = outY * p.down.y + p.up.y - 1 - p.pad0.y;+ int inY = min(max(floor_div(midY, p.up.y), 0), p.inSize.y);+ int h = min(max(floor_div(midY + p.filterSize.y, p.up.y), 0), p.inSize.y) - inY;+ int filterY = midY + p.filterSize.y - (inY + 1) * p.up.y;+ if (p.flip)+ filterY = p.filterSize.y - 1 - filterY;++ // Loop over major, minor, and X.+ for (int majorIdx = 0, major = majorBase; majorIdx < p.loopMajor & major < p.sizeMajor; majorIdx++, major++)+ for (int minorIdx = 0, minor = minorBase; minorIdx < p.loopMinor & minor < p.sizeMinor; minorIdx++, minor += p.launchMinor)+ {+ int nc = major * p.sizeMinor + minor;+ int n = nc / p.inSize.z;+ int c = nc - n * p.inSize.z;+ for (int loopX = 0, outX = outXBase; loopX < p.loopX & outX < p.outSize.x; loopX++, outX += blockDim.y)+ {+ // Setup X receptive field.+ int midX = outX * p.down.x + p.up.x - 1 - p.pad0.x;+ int inX = min(max(floor_div(midX, p.up.x), 0), p.inSize.x);+ int w = min(max(floor_div(midX + p.filterSize.x, p.up.x), 0), p.inSize.x) - inX;+ int filterX = midX + p.filterSize.x - (inX + 1) * p.up.x;+ if (p.flip)+ filterX = p.filterSize.x - 1 - filterX;++ // Initialize pointers.+ const T* xp = &((const T*)p.x)[inX * p.inStride.x + inY * p.inStride.y + c * p.inStride.z + n * p.inStride.w];+ const float* fp = &p.f[filterX * p.filterStride.x + filterY * p.filterStride.y];+ int filterStepX = ((p.flip) ? p.up.x : -p.up.x) * p.filterStride.x;+ int filterStepY = ((p.flip) ? p.up.y : -p.up.y) * p.filterStride.y;++ // Inner loop.+ scalar_t v = 0;+ for (int y = 0; y < h; y++)+ {+ for (int x = 0; x < w; x++)+ {+ v += (scalar_t)(*xp) * (scalar_t)(*fp);+ xp += p.inStride.x;+ fp += filterStepX;+ }+ xp += p.inStride.y - w * p.inStride.x;+ fp += filterStepY - w * filterStepX;+ }++ // Store result.+ v *= p.gain;+ ((T*)p.y)[outX * p.outStride.x + outY * p.outStride.y + c * p.outStride.z + n * p.outStride.w] = (T)v;+ }+ }+}++//------------------------------------------------------------------------+// Specialized CUDA implementation for small filters.++template <class T, int upx, int upy, int downx, int downy, int filterW, int filterH, int tileOutW, int tileOutH, int loopMinor>+static __global__ void upfirdn2d_kernel_small(upfirdn2d_kernel_params p)+{+ typedef typename InternalType<T>::scalar_t scalar_t;+ const int tileInW = ((tileOutW - 1) * downx + filterW - 1) / upx + 1;+ const int tileInH = ((tileOutH - 1) * downy + filterH - 1) / upy + 1;+ __shared__ volatile scalar_t sf[filterH][filterW];+ __shared__ volatile scalar_t sx[tileInH][tileInW][loopMinor];++ // Calculate tile index.+ int minorBase = blockIdx.x;+ int tileOutY = minorBase / p.launchMinor;+ minorBase -= tileOutY * p.launchMinor;+ minorBase *= loopMinor;+ tileOutY *= tileOutH;+ int tileOutXBase = blockIdx.y * p.loopX * tileOutW;+ int majorBase = blockIdx.z * p.loopMajor;+ if (tileOutXBase >= p.outSize.x | tileOutY >= p.outSize.y | majorBase >= p.sizeMajor)+ return;++ // Load filter (flipped).+ for (int tapIdx = threadIdx.x; tapIdx < filterH * filterW; tapIdx += blockDim.x)+ {+ int fy = tapIdx / filterW;+ int fx = tapIdx - fy * filterW;+ scalar_t v = 0;+ if (fx < p.filterSize.x & fy < p.filterSize.y)+ {+ int ffx = (p.flip) ? fx : p.filterSize.x - 1 - fx;+ int ffy = (p.flip) ? fy : p.filterSize.y - 1 - fy;+ v = (scalar_t)p.f[ffx * p.filterStride.x + ffy * p.filterStride.y];+ }+ sf[fy][fx] = v;+ }++ // Loop over major and X.+ for (int majorIdx = 0, major = majorBase; majorIdx < p.loopMajor & major < p.sizeMajor; majorIdx++, major++)+ {+ int baseNC = major * p.sizeMinor + minorBase;+ int n = baseNC / p.inSize.z;+ int baseC = baseNC - n * p.inSize.z;+ for (int loopX = 0, tileOutX = tileOutXBase; loopX < p.loopX & tileOutX < p.outSize.x; loopX++, tileOutX += tileOutW)+ {+ // Load input pixels.+ int tileMidX = tileOutX * downx + upx - 1 - p.pad0.x;+ int tileMidY = tileOutY * downy + upy - 1 - p.pad0.y;+ int tileInX = floor_div(tileMidX, upx);+ int tileInY = floor_div(tileMidY, upy);+ __syncthreads();+ for (int inIdx = threadIdx.x; inIdx < tileInH * tileInW * loopMinor; inIdx += blockDim.x)+ {+ int relC = inIdx;+ int relInX = relC / loopMinor;+ int relInY = relInX / tileInW;+ relC -= relInX * loopMinor;+ relInX -= relInY * tileInW;+ int c = baseC + relC;+ int inX = tileInX + relInX;+ int inY = tileInY + relInY;+ scalar_t v = 0;+ if (inX >= 0 & inY >= 0 & inX < p.inSize.x & inY < p.inSize.y & c < p.inSize.z)+ v = (scalar_t)((const T*)p.x)[inX * p.inStride.x + inY * p.inStride.y + c * p.inStride.z + n * p.inStride.w];+ sx[relInY][relInX][relC] = v;+ }++ // Loop over output pixels.+ __syncthreads();+ for (int outIdx = threadIdx.x; outIdx < tileOutH * tileOutW * loopMinor; outIdx += blockDim.x)+ {+ int relC = outIdx;+ int relOutX = relC / loopMinor;+ int relOutY = relOutX / tileOutW;+ relC -= relOutX * loopMinor;+ relOutX -= relOutY * tileOutW;+ int c = baseC + relC;+ int outX = tileOutX + relOutX;+ int outY = tileOutY + relOutY;++ // Setup receptive field.+ int midX = tileMidX + relOutX * downx;+ int midY = tileMidY + relOutY * downy;+ int inX = floor_div(midX, upx);+ int inY = floor_div(midY, upy);+ int relInX = inX - tileInX;+ int relInY = inY - tileInY;+ int filterX = (inX + 1) * upx - midX - 1; // flipped+ int filterY = (inY + 1) * upy - midY - 1; // flipped++ // Inner loop.+ if (outX < p.outSize.x & outY < p.outSize.y & c < p.outSize.z)+ {+ scalar_t v = 0;+ #pragma unroll+ for (int y = 0; y < filterH / upy; y++)+ #pragma unroll+ for (int x = 0; x < filterW / upx; x++)+ v += sx[relInY + y][relInX + x][relC] * sf[filterY + y * upy][filterX + x * upx];+ v *= p.gain;+ ((T*)p.y)[outX * p.outStride.x + outY * p.outStride.y + c * p.outStride.z + n * p.outStride.w] = (T)v;+ }+ }+ }+ }+}++//------------------------------------------------------------------------+// CUDA kernel selection.++template <class T> upfirdn2d_kernel_spec choose_upfirdn2d_kernel(const upfirdn2d_kernel_params& p)+{+ int s = p.inStride.z, fx = p.filterSize.x, fy = p.filterSize.y;++ upfirdn2d_kernel_spec spec = {(void*)upfirdn2d_kernel_large<T>, -1,-1,1, 4}; // contiguous+ if (s == 1) spec = {(void*)upfirdn2d_kernel_large<T>, -1,-1,4, 1}; // channels_last++ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // contiguous+ {+ if (fx <= 7 && fy <= 7 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 7,7, 64,16,1>, 64,16,1, 1};+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 6,6, 64,16,1>, 64,16,1, 1};+ if (fx <= 5 && fy <= 5 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 5,5, 64,16,1>, 64,16,1, 1};+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 64,16,1>, 64,16,1, 1};+ if (fx <= 3 && fy <= 3 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 3,3, 64,16,1>, 64,16,1, 1};+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 24,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 20,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 16,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 12,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 8,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,24, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,20, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,16, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,12, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,8, 32,32,1>, 32,32,1, 1};+ }+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // channels_last+ {+ if (fx <= 7 && fy <= 7 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 7,7, 16,16,8>, 16,16,8, 1};+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};+ if (fx <= 5 && fy <= 5 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};+ if (fx <= 3 && fy <= 3 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 4,4, 16,16,8>, 16,16,8, 1};+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 24,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 20,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 16,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 12,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 8,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,24, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,20, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,16, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,12, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,1, 1,8, 1,128,16>, 1,128,16, 1};+ }+ if (s != 1 && p.up.x == 2 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // contiguous+ {+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 8,8, 64,16,1>, 64,16,1, 1};+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 6,6, 64,16,1>, 64,16,1, 1};+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 4,4, 64,16,1>, 64,16,1, 1};+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 2,2, 64,16,1>, 64,16,1, 1};+ }+ if (s == 1 && p.up.x == 2 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // channels_last+ {+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 8,8, 16,16,8>, 16,16,8, 1};+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 6,6, 16,16,8>, 16,16,8, 1};+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 4,4, 16,16,8>, 16,16,8, 1};+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,2, 1,1, 2,2, 16,16,8>, 16,16,8, 1};+ }+ if (s != 1 && p.up.x == 2 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // contiguous+ {+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 24,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 20,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 16,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 12,1, 128,8,1>, 128,8,1, 1};+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 8,1, 128,8,1>, 128,8,1, 1};+ }+ if (s == 1 && p.up.x == 2 && p.up.y == 1 && p.down.x == 1 && p.down.y == 1) // channels_last+ {+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 24,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 20,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 16,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 12,1, 128,1,16>, 128,1,16, 1};+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 2,1, 1,1, 8,1, 128,1,16>, 128,1,16, 1};+ }+ if (s != 1 && p.up.x == 1 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // contiguous+ {+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,24, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,20, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,16, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,12, 32,32,1>, 32,32,1, 1};+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,8, 32,32,1>, 32,32,1, 1};+ }+ if (s == 1 && p.up.x == 1 && p.up.y == 2 && p.down.x == 1 && p.down.y == 1) // channels_last+ {+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,24, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,20, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,16, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,12, 1,128,16>, 1,128,16, 1};+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,2, 1,1, 1,8, 1,128,16>, 1,128,16, 1};+ }+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 2) // contiguous+ {+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 8,8, 32,8,1>, 32,8,1, 1};+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 6,6, 32,8,1>, 32,8,1, 1};+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 4,4, 32,8,1>, 32,8,1, 1};+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 2,2, 32,8,1>, 32,8,1, 1};+ }+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 2) // channels_last+ {+ if (fx <= 8 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 8,8, 8,8,8>, 8,8,8, 1};+ if (fx <= 6 && fy <= 6 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 6,6, 8,8,8>, 8,8,8, 1};+ if (fx <= 4 && fy <= 4 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 4,4, 8,8,8>, 8,8,8, 1};+ if (fx <= 2 && fy <= 2 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,2, 2,2, 8,8,8>, 8,8,8, 1};+ }+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 1) // contiguous+ {+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 24,1, 64,8,1>, 64,8,1, 1};+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 20,1, 64,8,1>, 64,8,1, 1};+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 16,1, 64,8,1>, 64,8,1, 1};+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 12,1, 64,8,1>, 64,8,1, 1};+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 8,1, 64,8,1>, 64,8,1, 1};+ }+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 2 && p.down.y == 1) // channels_last+ {+ if (fx <= 24 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 24,1, 64,1,8>, 64,1,8, 1};+ if (fx <= 20 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 20,1, 64,1,8>, 64,1,8, 1};+ if (fx <= 16 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 16,1, 64,1,8>, 64,1,8, 1};+ if (fx <= 12 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 12,1, 64,1,8>, 64,1,8, 1};+ if (fx <= 8 && fy <= 1 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 2,1, 8,1, 64,1,8>, 64,1,8, 1};+ }+ if (s != 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 2) // contiguous+ {+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,24, 32,16,1>, 32,16,1, 1};+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,20, 32,16,1>, 32,16,1, 1};+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,16, 32,16,1>, 32,16,1, 1};+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,12, 32,16,1>, 32,16,1, 1};+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,8, 32,16,1>, 32,16,1, 1};+ }+ if (s == 1 && p.up.x == 1 && p.up.y == 1 && p.down.x == 1 && p.down.y == 2) // channels_last+ {+ if (fx <= 1 && fy <= 24) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,24, 1,64,8>, 1,64,8, 1};+ if (fx <= 1 && fy <= 20) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,20, 1,64,8>, 1,64,8, 1};+ if (fx <= 1 && fy <= 16) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,16, 1,64,8>, 1,64,8, 1};+ if (fx <= 1 && fy <= 12) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,12, 1,64,8>, 1,64,8, 1};+ if (fx <= 1 && fy <= 8 ) spec = {(void*)upfirdn2d_kernel_small<T, 1,1, 1,2, 1,8, 1,64,8>, 1,64,8, 1};+ }+ return spec;+}++//------------------------------------------------------------------------+// Template specializations.++template upfirdn2d_kernel_spec choose_upfirdn2d_kernel<double> (const upfirdn2d_kernel_params& p);+template upfirdn2d_kernel_spec choose_upfirdn2d_kernel<float> (const upfirdn2d_kernel_params& p);+template upfirdn2d_kernel_spec choose_upfirdn2d_kernel<c10::Half>(const upfirdn2d_kernel_params& p);++//------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.h b/src/stylegan2/torch_utils/ops/upfirdn2d.hnew file mode 100644index 0000000..c9e2032--- /dev/null+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.h@@ -0,0 +1,59 @@+// Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+//+// NVIDIA CORPORATION and its licensors retain all intellectual property+// and proprietary rights in and to this software, related documentation+// and any modifications thereto. Any use, reproduction, disclosure or+// distribution of this software and related documentation without an express+// license agreement from NVIDIA CORPORATION is strictly prohibited.++#include <cuda_runtime.h>++//------------------------------------------------------------------------+// CUDA kernel parameters.++struct upfirdn2d_kernel_params+{+ const void* x;+ const float* f;+ void* y;++ int2 up;+ int2 down;+ int2 pad0;+ int flip;+ float gain;++ int4 inSize; // [width, height, channel, batch]+ int4 inStride;+ int2 filterSize; // [width, height]+ int2 filterStride;+ int4 outSize; // [width, height, channel, batch]+ int4 outStride;+ int sizeMinor;+ int sizeMajor;++ int loopMinor;+ int loopMajor;+ int loopX;+ int launchMinor;+ int launchMajor;+};++//------------------------------------------------------------------------+// CUDA kernel specialization.++struct upfirdn2d_kernel_spec+{+ void* kernel;+ int tileOutW;+ int tileOutH;+ int loopMinor;+ int loopX;+};++//------------------------------------------------------------------------+// CUDA kernel selection.++template <class T> upfirdn2d_kernel_spec choose_upfirdn2d_kernel(const upfirdn2d_kernel_params& p);++//------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/ops/upfirdn2d.py b/src/stylegan2/torch_utils/ops/upfirdn2d.pynew file mode 100644index 0000000..ceeac2b--- /dev/null+++ b/src/stylegan2/torch_utils/ops/upfirdn2d.py@@ -0,0 +1,384 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Custom PyTorch ops for efficient resampling of 2D images."""++import os+import warnings+import numpy as np+import torch+import traceback++from .. import custom_ops+from .. import misc+from . import conv2d_gradfix++#----------------------------------------------------------------------------++_inited = False+_plugin = None++def _init():+ global _inited, _plugin+ if not _inited:+ sources = ['upfirdn2d.cpp', 'upfirdn2d.cu']+ sources = [os.path.join(os.path.dirname(__file__), s) for s in sources]+ try:+ _plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])+ except:+ warnings.warn('Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:\n\n' + traceback.format_exc())+ return _plugin is not None++def _parse_scaling(scaling):+ if isinstance(scaling, int):+ scaling = [scaling, scaling]+ assert isinstance(scaling, (list, tuple))+ assert all(isinstance(x, int) for x in scaling)+ sx, sy = scaling+ assert sx >= 1 and sy >= 1+ return sx, sy++def _parse_padding(padding):+ if isinstance(padding, int):+ padding = [padding, padding]+ assert isinstance(padding, (list, tuple))+ assert all(isinstance(x, int) for x in padding)+ if len(padding) == 2:+ padx, pady = padding+ padding = [padx, padx, pady, pady]+ padx0, padx1, pady0, pady1 = padding+ return padx0, padx1, pady0, pady1++def _get_filter_size(f):+ if f is None:+ return 1, 1+ assert isinstance(f, torch.Tensor) and f.ndim in [1, 2]+ fw = f.shape[-1]+ fh = f.shape[0]+ with misc.suppress_tracer_warnings():+ fw = int(fw)+ fh = int(fh)+ misc.assert_shape(f, [fh, fw][:f.ndim])+ assert fw >= 1 and fh >= 1+ return fw, fh++#----------------------------------------------------------------------------++def setup_filter(f, device=torch.device('cpu'), normalize=True, flip_filter=False, gain=1, separable=None):+ r"""Convenience function to setup 2D FIR filter for `upfirdn2d()`.++ Args:+ f: Torch tensor, numpy array, or python list of the shape+ `[filter_height, filter_width]` (non-separable),+ `[filter_taps]` (separable),+ `[]` (impulse), or+ `None` (identity).+ device: Result device (default: cpu).+ normalize: Normalize the filter so that it retains the magnitude+ for constant input signal (DC)? (default: True).+ flip_filter: Flip the filter? (default: False).+ gain: Overall scaling factor for signal magnitude (default: 1).+ separable: Return a separable filter? (default: select automatically).++ Returns:+ Float32 tensor of the shape+ `[filter_height, filter_width]` (non-separable) or+ `[filter_taps]` (separable).+ """+ # Validate.+ if f is None:+ f = 1+ f = torch.as_tensor(f, dtype=torch.float32)+ assert f.ndim in [0, 1, 2]+ assert f.numel() > 0+ if f.ndim == 0:+ f = f[np.newaxis]++ # Separable?+ if separable is None:+ separable = (f.ndim == 1 and f.numel() >= 8)+ if f.ndim == 1 and not separable:+ f = f.ger(f)+ assert f.ndim == (1 if separable else 2)++ # Apply normalize, flip, gain, and device.+ if normalize:+ f /= f.sum()+ if flip_filter:+ f = f.flip(list(range(f.ndim)))+ f = f * (gain ** (f.ndim / 2))+ f = f.to(device=device)+ return f++#----------------------------------------------------------------------------++def upfirdn2d(x, f, up=1, down=1, padding=0, flip_filter=False, gain=1, impl='cuda'):+ r"""Pad, upsample, filter, and downsample a batch of 2D images.++ Performs the following sequence of operations for each channel:++ 1. Upsample the image by inserting N-1 zeros after each pixel (`up`).++ 2. Pad the image with the specified number of zeros on each side (`padding`).+ Negative padding corresponds to cropping the image.++ 3. Convolve the image with the specified 2D FIR filter (`f`), shrinking it+ so that the footprint of all output pixels lies within the input image.++ 4. Downsample the image by keeping every Nth pixel (`down`).++ This sequence of operations bears close resemblance to scipy.signal.upfirdn().+ The fused op is considerably more efficient than performing the same calculation+ using standard PyTorch ops. It supports gradients of arbitrary order.++ Args:+ x: Float32/float64/float16 input tensor of the shape+ `[batch_size, num_channels, in_height, in_width]`.+ f: Float32 FIR filter of the shape+ `[filter_height, filter_width]` (non-separable),+ `[filter_taps]` (separable), or+ `None` (identity).+ up: Integer upsampling factor. Can be a single int or a list/tuple+ `[x, y]` (default: 1).+ down: Integer downsampling factor. Can be a single int or a list/tuple+ `[x, y]` (default: 1).+ padding: Padding with respect to the upsampled image. Can be a single number+ or a list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`+ (default: 0).+ flip_filter: False = convolution, True = correlation (default: False).+ gain: Overall scaling factor for signal magnitude (default: 1).+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).++ Returns:+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.+ """+ assert isinstance(x, torch.Tensor)+ assert impl in ['ref', 'cuda']+ if impl == 'cuda' and x.device.type == 'cuda' and _init():+ return _upfirdn2d_cuda(up=up, down=down, padding=padding, flip_filter=flip_filter, gain=gain).apply(x, f)+ return _upfirdn2d_ref(x, f, up=up, down=down, padding=padding, flip_filter=flip_filter, gain=gain)++#----------------------------------------------------------------------------+[email protected]_function+def _upfirdn2d_ref(x, f, up=1, down=1, padding=0, flip_filter=False, gain=1):+ """Slow reference implementation of `upfirdn2d()` using standard PyTorch ops.+ """+ # Validate arguments.+ assert isinstance(x, torch.Tensor) and x.ndim == 4+ if f is None:+ f = torch.ones([1, 1], dtype=torch.float32, device=x.device)+ assert isinstance(f, torch.Tensor) and f.ndim in [1, 2]+ assert f.dtype == torch.float32 and not f.requires_grad+ batch_size, num_channels, in_height, in_width = x.shape+ upx, upy = _parse_scaling(up)+ downx, downy = _parse_scaling(down)+ padx0, padx1, pady0, pady1 = _parse_padding(padding)++ # Upsample by inserting zeros.+ x = x.reshape([batch_size, num_channels, in_height, 1, in_width, 1])+ x = torch.nn.functional.pad(x, [0, upx - 1, 0, 0, 0, upy - 1])+ x = x.reshape([batch_size, num_channels, in_height * upy, in_width * upx])++ # Pad or crop.+ x = torch.nn.functional.pad(x, [max(padx0, 0), max(padx1, 0), max(pady0, 0), max(pady1, 0)])+ x = x[:, :, max(-pady0, 0) : x.shape[2] - max(-pady1, 0), max(-padx0, 0) : x.shape[3] - max(-padx1, 0)]++ # Setup filter.+ f = f * (gain ** (f.ndim / 2))+ f = f.to(x.dtype)+ if not flip_filter:+ f = f.flip(list(range(f.ndim)))++ # Convolve with the filter.+ f = f[np.newaxis, np.newaxis].repeat([num_channels, 1] + [1] * f.ndim)+ if f.ndim == 4:+ x = conv2d_gradfix.conv2d(input=x, weight=f, groups=num_channels)+ else:+ x = conv2d_gradfix.conv2d(input=x, weight=f.unsqueeze(2), groups=num_channels)+ x = conv2d_gradfix.conv2d(input=x, weight=f.unsqueeze(3), groups=num_channels)++ # Downsample by throwing away pixels.+ x = x[:, :, ::downy, ::downx]+ return x++#----------------------------------------------------------------------------++_upfirdn2d_cuda_cache = dict()++def _upfirdn2d_cuda(up=1, down=1, padding=0, flip_filter=False, gain=1):+ """Fast CUDA implementation of `upfirdn2d()` using custom ops.+ """+ # Parse arguments.+ upx, upy = _parse_scaling(up)+ downx, downy = _parse_scaling(down)+ padx0, padx1, pady0, pady1 = _parse_padding(padding)++ # Lookup from cache.+ key = (upx, upy, downx, downy, padx0, padx1, pady0, pady1, flip_filter, gain)+ if key in _upfirdn2d_cuda_cache:+ return _upfirdn2d_cuda_cache[key]++ # Forward op.+ class Upfirdn2dCuda(torch.autograd.Function):+ @staticmethod+ def forward(ctx, x, f): # pylint: disable=arguments-differ+ assert isinstance(x, torch.Tensor) and x.ndim == 4+ if f is None:+ f = torch.ones([1, 1], dtype=torch.float32, device=x.device)+ assert isinstance(f, torch.Tensor) and f.ndim in [1, 2]+ y = x+ if f.ndim == 2:+ y = _plugin.upfirdn2d(y, f, upx, upy, downx, downy, padx0, padx1, pady0, pady1, flip_filter, gain)+ else:+ y = _plugin.upfirdn2d(y, f.unsqueeze(0), upx, 1, downx, 1, padx0, padx1, 0, 0, flip_filter, np.sqrt(gain))+ y = _plugin.upfirdn2d(y, f.unsqueeze(1), 1, upy, 1, downy, 0, 0, pady0, pady1, flip_filter, np.sqrt(gain))+ ctx.save_for_backward(f)+ ctx.x_shape = x.shape+ return y++ @staticmethod+ def backward(ctx, dy): # pylint: disable=arguments-differ+ f, = ctx.saved_tensors+ _, _, ih, iw = ctx.x_shape+ _, _, oh, ow = dy.shape+ fw, fh = _get_filter_size(f)+ p = [+ fw - padx0 - 1,+ iw * upx - ow * downx + padx0 - upx + 1,+ fh - pady0 - 1,+ ih * upy - oh * downy + pady0 - upy + 1,+ ]+ dx = None+ df = None++ if ctx.needs_input_grad[0]:+ dx = _upfirdn2d_cuda(up=down, down=up, padding=p, flip_filter=(not flip_filter), gain=gain).apply(dy, f)++ assert not ctx.needs_input_grad[1]+ return dx, df++ # Add to cache.+ _upfirdn2d_cuda_cache[key] = Upfirdn2dCuda+ return Upfirdn2dCuda++#----------------------------------------------------------------------------++def filter2d(x, f, padding=0, flip_filter=False, gain=1, impl='cuda'):+ r"""Filter a batch of 2D images using the given 2D FIR filter.++ By default, the result is padded so that its shape matches the input.+ User-specified padding is applied on top of that, with negative values+ indicating cropping. Pixels outside the image are assumed to be zero.++ Args:+ x: Float32/float64/float16 input tensor of the shape+ `[batch_size, num_channels, in_height, in_width]`.+ f: Float32 FIR filter of the shape+ `[filter_height, filter_width]` (non-separable),+ `[filter_taps]` (separable), or+ `None` (identity).+ padding: Padding with respect to the output. Can be a single number or a+ list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`+ (default: 0).+ flip_filter: False = convolution, True = correlation (default: False).+ gain: Overall scaling factor for signal magnitude (default: 1).+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).++ Returns:+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.+ """+ padx0, padx1, pady0, pady1 = _parse_padding(padding)+ fw, fh = _get_filter_size(f)+ p = [+ padx0 + fw // 2,+ padx1 + (fw - 1) // 2,+ pady0 + fh // 2,+ pady1 + (fh - 1) // 2,+ ]+ return upfirdn2d(x, f, padding=p, flip_filter=flip_filter, gain=gain, impl=impl)++#----------------------------------------------------------------------------++def upsample2d(x, f, up=2, padding=0, flip_filter=False, gain=1, impl='cuda'):+ r"""Upsample a batch of 2D images using the given 2D FIR filter.++ By default, the result is padded so that its shape is a multiple of the input.+ User-specified padding is applied on top of that, with negative values+ indicating cropping. Pixels outside the image are assumed to be zero.++ Args:+ x: Float32/float64/float16 input tensor of the shape+ `[batch_size, num_channels, in_height, in_width]`.+ f: Float32 FIR filter of the shape+ `[filter_height, filter_width]` (non-separable),+ `[filter_taps]` (separable), or+ `None` (identity).+ up: Integer upsampling factor. Can be a single int or a list/tuple+ `[x, y]` (default: 1).+ padding: Padding with respect to the output. Can be a single number or a+ list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`+ (default: 0).+ flip_filter: False = convolution, True = correlation (default: False).+ gain: Overall scaling factor for signal magnitude (default: 1).+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).++ Returns:+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.+ """+ upx, upy = _parse_scaling(up)+ padx0, padx1, pady0, pady1 = _parse_padding(padding)+ fw, fh = _get_filter_size(f)+ p = [+ padx0 + (fw + upx - 1) // 2,+ padx1 + (fw - upx) // 2,+ pady0 + (fh + upy - 1) // 2,+ pady1 + (fh - upy) // 2,+ ]+ return upfirdn2d(x, f, up=up, padding=p, flip_filter=flip_filter, gain=gain*upx*upy, impl=impl)++#----------------------------------------------------------------------------++def downsample2d(x, f, down=2, padding=0, flip_filter=False, gain=1, impl='cuda'):+ r"""Downsample a batch of 2D images using the given 2D FIR filter.++ By default, the result is padded so that its shape is a fraction of the input.+ User-specified padding is applied on top of that, with negative values+ indicating cropping. Pixels outside the image are assumed to be zero.++ Args:+ x: Float32/float64/float16 input tensor of the shape+ `[batch_size, num_channels, in_height, in_width]`.+ f: Float32 FIR filter of the shape+ `[filter_height, filter_width]` (non-separable),+ `[filter_taps]` (separable), or+ `None` (identity).+ down: Integer downsampling factor. Can be a single int or a list/tuple+ `[x, y]` (default: 1).+ padding: Padding with respect to the input. Can be a single number or a+ list/tuple `[x, y]` or `[x_before, x_after, y_before, y_after]`+ (default: 0).+ flip_filter: False = convolution, True = correlation (default: False).+ gain: Overall scaling factor for signal magnitude (default: 1).+ impl: Implementation to use. Can be `'ref'` or `'cuda'` (default: `'cuda'`).++ Returns:+ Tensor of the shape `[batch_size, num_channels, out_height, out_width]`.+ """+ downx, downy = _parse_scaling(down)+ padx0, padx1, pady0, pady1 = _parse_padding(padding)+ fw, fh = _get_filter_size(f)+ p = [+ padx0 + (fw - downx + 1) // 2,+ padx1 + (fw - downx) // 2,+ pady0 + (fh - downy + 1) // 2,+ pady1 + (fh - downy) // 2,+ ]+ return upfirdn2d(x, f, down=down, padding=p, flip_filter=flip_filter, gain=gain, impl=impl)++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/persistence.py b/src/stylegan2/torch_utils/persistence.pynew file mode 100644index 0000000..0186cfd--- /dev/null+++ b/src/stylegan2/torch_utils/persistence.py@@ -0,0 +1,251 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Facilities for pickling Python code alongside other data.++The pickled code is automatically imported into a separate Python module+during unpickling. This way, any previously exported pickles will remain+usable even if the original code is no longer available, or if the current+version of the code is not consistent with what was originally pickled."""++import sys+import pickle+import io+import inspect+import copy+import uuid+import types+import dnnlib++#----------------------------------------------------------------------------++_version = 6 # internal version number+_decorators = set() # {decorator_class, ...}+_import_hooks = [] # [hook_function, ...]+_module_to_src_dict = dict() # {module: src, ...}+_src_to_module_dict = dict() # {src: module, ...}++#----------------------------------------------------------------------------++def persistent_class(orig_class):+ r"""Class decorator that extends a given class to save its source code+ when pickled.++ Example:++ from torch_utils import persistence++ @persistence.persistent_class+ class MyNetwork(torch.nn.Module):+ def __init__(self, num_inputs, num_outputs):+ super().__init__()+ self.fc = MyLayer(num_inputs, num_outputs)+ ...++ @persistence.persistent_class+ class MyLayer(torch.nn.Module):+ ...++ When pickled, any instance of `MyNetwork` and `MyLayer` will save its+ source code alongside other internal state (e.g., parameters, buffers,+ and submodules). This way, any previously exported pickle will remain+ usable even if the class definitions have been modified or are no+ longer available.++ The decorator saves the source code of the entire Python module+ containing the decorated class. It does *not* save the source code of+ any imported modules. Thus, the imported modules must be available+ during unpickling, also including `torch_utils.persistence` itself.++ It is ok to call functions defined in the same module from the+ decorated class. However, if the decorated class depends on other+ classes defined in the same module, they must be decorated as well.+ This is illustrated in the above example in the case of `MyLayer`.++ It is also possible to employ the decorator just-in-time before+ calling the constructor. For example:++ cls = MyLayer+ if want_to_make_it_persistent:+ cls = persistence.persistent_class(cls)+ layer = cls(num_inputs, num_outputs)++ As an additional feature, the decorator also keeps track of the+ arguments that were used to construct each instance of the decorated+ class. The arguments can be queried via `obj.init_args` and+ `obj.init_kwargs`, and they are automatically pickled alongside other+ object state. A typical use case is to first unpickle a previous+ instance of a persistent class, and then upgrade it to use the latest+ version of the source code:++ with open('old_pickle.pkl', 'rb') as f:+ old_net = pickle.load(f)+ new_net = MyNetwork(*old_obj.init_args, **old_obj.init_kwargs)+ misc.copy_params_and_buffers(old_net, new_net, require_all=True)+ """+ assert isinstance(orig_class, type)+ if is_persistent(orig_class):+ return orig_class++ assert orig_class.__module__ in sys.modules+ orig_module = sys.modules[orig_class.__module__]+ orig_module_src = _module_to_src(orig_module)++ class Decorator(orig_class):+ _orig_module_src = orig_module_src+ _orig_class_name = orig_class.__name__++ def __init__(self, *args, **kwargs):+ super().__init__(*args, **kwargs)+ self._init_args = copy.deepcopy(args)+ self._init_kwargs = copy.deepcopy(kwargs)+ assert orig_class.__name__ in orig_module.__dict__+ _check_pickleable(self.__reduce__())++ @property+ def init_args(self):+ return copy.deepcopy(self._init_args)++ @property+ def init_kwargs(self):+ return dnnlib.EasyDict(copy.deepcopy(self._init_kwargs))++ def __reduce__(self):+ fields = list(super().__reduce__())+ fields += [None] * max(3 - len(fields), 0)+ if fields[0] is not _reconstruct_persistent_obj:+ meta = dict(type='class', version=_version, module_src=self._orig_module_src, class_name=self._orig_class_name, state=fields[2])+ fields[0] = _reconstruct_persistent_obj # reconstruct func+ fields[1] = (meta,) # reconstruct args+ fields[2] = None # state dict+ return tuple(fields)++ Decorator.__name__ = orig_class.__name__+ _decorators.add(Decorator)+ return Decorator++#----------------------------------------------------------------------------++def is_persistent(obj):+ r"""Test whether the given object or class is persistent, i.e.,+ whether it will save its source code when pickled.+ """+ try:+ if obj in _decorators:+ return True+ except TypeError:+ pass+ return type(obj) in _decorators # pylint: disable=unidiomatic-typecheck++#----------------------------------------------------------------------------++def import_hook(hook):+ r"""Register an import hook that is called whenever a persistent object+ is being unpickled. A typical use case is to patch the pickled source+ code to avoid errors and inconsistencies when the API of some imported+ module has changed.++ The hook should have the following signature:++ hook(meta) -> modified meta++ `meta` is an instance of `dnnlib.EasyDict` with the following fields:++ type: Type of the persistent object, e.g. `'class'`.+ version: Internal version number of `torch_utils.persistence`.+ module_src Original source code of the Python module.+ class_name: Class name in the original Python module.+ state: Internal state of the object.++ Example:++ @persistence.import_hook+ def wreck_my_network(meta):+ if meta.class_name == 'MyNetwork':+ print('MyNetwork is being imported. I will wreck it!')+ meta.module_src = meta.module_src.replace("True", "False")+ return meta+ """+ assert callable(hook)+ _import_hooks.append(hook)++#----------------------------------------------------------------------------++def _reconstruct_persistent_obj(meta):+ r"""Hook that is called internally by the `pickle` module to unpickle+ a persistent object.+ """+ meta = dnnlib.EasyDict(meta)+ meta.state = dnnlib.EasyDict(meta.state)+ for hook in _import_hooks:+ meta = hook(meta)+ assert meta is not None++ assert meta.version == _version+ module = _src_to_module(meta.module_src)++ assert meta.type == 'class'+ orig_class = module.__dict__[meta.class_name]+ decorator_class = persistent_class(orig_class)+ obj = decorator_class.__new__(decorator_class)++ setstate = getattr(obj, '__setstate__', None)+ if callable(setstate):+ setstate(meta.state) # pylint: disable=not-callable+ else:+ obj.__dict__.update(meta.state)+ return obj++#----------------------------------------------------------------------------++def _module_to_src(module):+ r"""Query the source code of a given Python module.+ """+ src = _module_to_src_dict.get(module, None)+ if src is None:+ src = inspect.getsource(module)+ _module_to_src_dict[module] = src+ _src_to_module_dict[src] = module+ return src++def _src_to_module(src):+ r"""Get or create a Python module for the given source code.+ """+ module = _src_to_module_dict.get(src, None)+ if module is None:+ module_name = "_imported_module_" + uuid.uuid4().hex+ module = types.ModuleType(module_name)+ sys.modules[module_name] = module+ _module_to_src_dict[module] = src+ _src_to_module_dict[src] = module+ exec(src, module.__dict__) # pylint: disable=exec-used+ return module++#----------------------------------------------------------------------------++def _check_pickleable(obj):+ r"""Check that the given object is pickleable, raising an exception if+ it is not. This function is expected to be considerably more efficient+ than actually pickling the object.+ """+ def recurse(obj):+ if isinstance(obj, (list, tuple, set)):+ return [recurse(x) for x in obj]+ if isinstance(obj, dict):+ return [[recurse(x), recurse(y)] for x, y in obj.items()]+ if isinstance(obj, (str, int, float, bool, bytes, bytearray)):+ return None # Python primitive types are pickleable.+ if f'{type(obj).__module__}.{type(obj).__name__}' in ['numpy.ndarray', 'torch.Tensor']:+ return None # NumPy arrays and PyTorch tensors are pickleable.+ if is_persistent(obj):+ return None # Persistent objects are pickleable, by virtue of the constructor check.+ return obj+ with io.BytesIO() as f:+ pickle.dump(recurse(obj), f)++#----------------------------------------------------------------------------diff --git a/src/stylegan2/torch_utils/training_stats.py b/src/stylegan2/torch_utils/training_stats.pynew file mode 100644index 0000000..26f467f--- /dev/null+++ b/src/stylegan2/torch_utils/training_stats.py@@ -0,0 +1,268 @@+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.+#+# NVIDIA CORPORATION and its licensors retain all intellectual property+# and proprietary rights in and to this software, related documentation+# and any modifications thereto. Any use, reproduction, disclosure or+# distribution of this software and related documentation without an express+# license agreement from NVIDIA CORPORATION is strictly prohibited.++"""Facilities for reporting and collecting training statistics across+multiple processes and devices. The interface is designed to minimize+synchronization overhead as well as the amount of boilerplate in user+code."""++import re+import numpy as np+import torch+import dnnlib++from . import misc++#----------------------------------------------------------------------------++_num_moments = 3 # [num_scalars, sum_of_scalars, sum_of_squares]+_reduce_dtype = torch.float32 # Data type to use for initial per-tensor reduction.+_counter_dtype = torch.float64 # Data type to use for the internal counters.+_rank = 0 # Rank of the current process.+_sync_device = None # Device to use for multiprocess communication. None = single-process.+_sync_called = False # Has _sync() been called yet?+_counters = dict() # Running counters on each device, updated by report(): name => device => torch.Tensor+_cumulative = dict() # Cumulative counters on the CPU, updated by _sync(): name => torch.Tensor++#----------------------------------------------------------------------------++def init_multiprocessing(rank, sync_device):+ r"""Initializes `torch_utils.training_stats` for collecting statistics+ across multiple processes.++ This function must be called after+ `torch.distributed.init_process_group()` and before `Collector.update()`.+ The call is not necessary if multi-process collection is not needed.++ Args:+ rank: Rank of the current process.+ sync_device: PyTorch device to use for inter-process+ communication, or None to disable multi-process+ collection. Typically `torch.device('cuda', rank)`.+ """+ global _rank, _sync_device+ assert not _sync_called+ _rank = rank+ _sync_device = sync_device++#----------------------------------------------------------------------------+[email protected]_function+def report(name, value):+ r"""Broadcasts the given set of scalars to all interested instances of+ `Collector`, across device and process boundaries.++ This function is expected to be extremely cheap and can be safely+ called from anywhere in the training loop, loss function, or inside a+ `torch.nn.Module`.++ Warning: The current implementation expects the set of unique names to+ be consistent across processes. Please make sure that `report()` is+ called at least once for each unique name by each process, and in the+ same order. If a given process has no scalars to broadcast, it can do+ `report(name, [])` (empty list).++ Args:+ name: Arbitrary string specifying the name of the statistic.+ Averages are accumulated separately for each unique name.+ value: Arbitrary set of scalars. Can be a list, tuple,+ NumPy array, PyTorch tensor, or Python scalar.++ Returns:+ The same `value` that was passed in.+ """+ if name not in _counters:+ _counters[name] = dict()++ elems = torch.as_tensor(value)+ if elems.numel() == 0:+ return value++ elems = elems.detach().flatten().to(_reduce_dtype)+ moments = torch.stack([+ torch.ones_like(elems).sum(),+ elems.sum(),+ elems.square().sum(),+ ])+ assert moments.ndim == 1 and moments.shape[0] == _num_moments+ moments = moments.to(_counter_dtype)++ device = moments.device+ if device not in _counters[name]:+ _counters[name][device] = torch.zeros_like(moments)+ _counters[name][device].add_(moments)+ return value++#----------------------------------------------------------------------------++def report0(name, value):+ r"""Broadcasts the given set of scalars by the first process (`rank = 0`),+ but ignores any scalars provided by the other processes.+ See `report()` for further details.+ """+ report(name, value if _rank == 0 else [])+ return value++#----------------------------------------------------------------------------++class Collector:+ r"""Collects the scalars broadcasted by `report()` and `report0()` and+ computes their long-term averages (mean and standard deviation) over+ user-defined periods of time.++ The averages are first collected into internal counters that are not+ directly visible to the user. They are then copied to the user-visible+ state as a result of calling `update()` and can then be queried using+ `mean()`, `std()`, `as_dict()`, etc. Calling `update()` also resets the+ internal counters for the next round, so that the user-visible state+ effectively reflects averages collected between the last two calls to+ `update()`.++ Args:+ regex: Regular expression defining which statistics to+ collect. The default is to collect everything.+ keep_previous: Whether to retain the previous averages if no+ scalars were collected on a given round+ (default: True).+ """+ def __init__(self, regex='.*', keep_previous=True):+ self._regex = re.compile(regex)+ self._keep_previous = keep_previous+ self._cumulative = dict()+ self._moments = dict()+ self.update()+ self._moments.clear()++ def names(self):+ r"""Returns the names of all statistics broadcasted so far that+ match the regular expression specified at construction time.+ """+ return [name for name in _counters if self._regex.fullmatch(name)]++ def update(self):+ r"""Copies current values of the internal counters to the+ user-visible state and resets them for the next round.++ If `keep_previous=True` was specified at construction time, the+ operation is skipped for statistics that have received no scalars+ since the last update, retaining their previous averages.++ This method performs a number of GPU-to-CPU transfers and one+ `torch.distributed.all_reduce()`. It is intended to be called+ periodically in the main training loop, typically once every+ N training steps.+ """+ if not self._keep_previous:+ self._moments.clear()+ for name, cumulative in _sync(self.names()):+ if name not in self._cumulative:+ self._cumulative[name] = torch.zeros([_num_moments], dtype=_counter_dtype)+ delta = cumulative - self._cumulative[name]+ self._cumulative[name].copy_(cumulative)+ if float(delta[0]) != 0:+ self._moments[name] = delta++ def _get_delta(self, name):+ r"""Returns the raw moments that were accumulated for the given+ statistic between the last two calls to `update()`, or zero if+ no scalars were collected.+ """+ assert self._regex.fullmatch(name)+ if name not in self._moments:+ self._moments[name] = torch.zeros([_num_moments], dtype=_counter_dtype)+ return self._moments[name]++ def num(self, name):+ r"""Returns the number of scalars that were accumulated for the given+ statistic between the last two calls to `update()`, or zero if+ no scalars were collected.+ """+ delta = self._get_delta(name)+ return int(delta[0])++ def mean(self, name):+ r"""Returns the mean of the scalars that were accumulated for the+ given statistic between the last two calls to `update()`, or NaN if+ no scalars were collected.+ """+ delta = self._get_delta(name)+ if int(delta[0]) == 0:+ return float('nan')+ return float(delta[1] / delta[0])++ def std(self, name):+ r"""Returns the standard deviation of the scalars that were+ accumulated for the given statistic between the last two calls to+ `update()`, or NaN if no scalars were collected.+ """+ delta = self._get_delta(name)+ if int(delta[0]) == 0 or not np.isfinite(float(delta[1])):+ return float('nan')+ if int(delta[0]) == 1:+ return float(0)+ mean = float(delta[1] / delta[0])+ raw_var = float(delta[2] / delta[0])+ return np.sqrt(max(raw_var - np.square(mean), 0))++ def as_dict(self):+ r"""Returns the averages accumulated between the last two calls to+ `update()` as an `dnnlib.EasyDict`. The contents are as follows:++ dnnlib.EasyDict(+ NAME = dnnlib.EasyDict(num=FLOAT, mean=FLOAT, std=FLOAT),+ ...+ )+ """+ stats = dnnlib.EasyDict()+ for name in self.names():+ stats[name] = dnnlib.EasyDict(num=self.num(name), mean=self.mean(name), std=self.std(name))+ return stats++ def __getitem__(self, name):+ r"""Convenience getter.+ `collector[name]` is a synonym for `collector.mean(name)`.+ """+ return self.mean(name)++#----------------------------------------------------------------------------++def _sync(names):+ r"""Synchronize the global cumulative counters across devices and+ processes. Called internally by `Collector.update()`.+ """+ if len(names) == 0:+ return []+ global _sync_called+ _sync_called = True++ # Collect deltas within current rank.+ deltas = []+ device = _sync_device if _sync_device is not None else torch.device('cpu')+ for name in names:+ delta = torch.zeros([_num_moments], dtype=_counter_dtype, device=device)+ for counter in _counters[name].values():+ delta.add_(counter.to(device))+ counter.copy_(torch.zeros_like(counter))+ deltas.append(delta)+ deltas = torch.stack(deltas)++ # Sum deltas across ranks.+ if _sync_device is not None:+ torch.distributed.all_reduce(deltas)++ # Update cumulative values.+ deltas = deltas.cpu()+ for idx, name in enumerate(names):+ if name not in _cumulative:+ _cumulative[name] = torch.zeros([_num_moments], dtype=_counter_dtype)+ _cumulative[name].add_(deltas[idx])++ # Return name-value pairs.+ return [(name, _cumulative[name]) for name in names]++#----------------------------------------------------------------------------diff --git a/tools/extract_meta_info.py b/tools/extract_meta_info.pynew file mode 100644index 0000000..8fc6423--- /dev/null+++ b/tools/extract_meta_info.py@@ -0,0 +1,32 @@+import argparse+import json+import os++# -----+# python tools/extract_meta_info.py --root_path /path/to/video_dir --dataset_name fashion+# -----+parser = argparse.ArgumentParser()+parser.add_argument("--root_path", type=str, default='./Datasets/videos/VFHQ')+parser.add_argument("--dataset_name", type=str, default='VFHQ')+parser.add_argument("--meta_info_name", type=str)++args = parser.parse_args()++if args.meta_info_name is None:+ args.meta_info_name = args.dataset_name++# collect all video_folder paths+video_mp4_paths = set()+for root, dirs, files in os.walk(args.root_path):+ for name in files:+ if name.endswith(".pt"):+ video_mp4_paths.add(os.path.join(root, name).replace('boxes','videos').replace('.pt','.mp4'))+ elif name.endswith(".mp4"):+ video_mp4_paths.add(os.path.join(root, name))+video_mp4_paths = list(video_mp4_paths)++meta_infos = []+for video_mp4_path in video_mp4_paths:+ meta_infos.append({"video_path": video_mp4_path})++json.dump(meta_infos, open(f"./data/{args.meta_info_name}_meta.json", "w"))diff --git a/tools/get_boxes.py b/tools/get_boxes.pynew file mode 100644index 0000000..b0119f1--- /dev/null+++ b/tools/get_boxes.py@@ -0,0 +1,80 @@+import mediapipe as mp+import cv2+import os+import torch+import numpy as np+from concurrent.futures import ProcessPoolExecutor+from src.utils.util import read_frames+import logging+logging.getLogger('mediapipe').setLevel(logging.ERROR)+import argparse+from functools import partial++mp_face_mesh = mp.solutions.face_mesh++face_indices = list(range(468))+left_eye_indices = [226, 230, 223, 245]+right_eye_indices = [446, 450, 465, 443]+mouth_indices = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291] + \+ [78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308] + \+ [146, 91, 181, 84, 17, 314, 405, 321, 375] + \+ [191, 80, 81, 82, 13, 312, 311, 310, 415]++def get_region_box(landmarks, indices):+ xs = [int(landmarks[i][0]) for i in indices]+ ys = [int(landmarks[i][1]) for i in indices]+ x_min, x_max = min(xs), max(xs)+ y_min, y_max = min(ys), max(ys)+ return [x_min, y_min, x_max, y_max]++def process_video(name, video_dir, save_dir):+ video_path = os.path.join(video_dir, name)+ save_path = os.path.join(save_dir, name.replace('.mp4', '.pt'))+ if os.path.exists(save_path):+ return name++ video = read_frames(video_path)+ boxes = []++ with mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1) as face_mesh:+ for image_pil in video:+ image = np.array(image_pil)+ h, w, _ = image.shape+ results = face_mesh.process(image)++ if results.multi_face_landmarks is not None:+ face_landmarks = results.multi_face_landmarks[0]+ landmarks = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]+ face_box = get_region_box(landmarks, face_indices)+ left_eye_box = get_region_box(landmarks, left_eye_indices)+ right_eye_box = get_region_box(landmarks, right_eye_indices)+ mouth_box = get_region_box(landmarks, mouth_indices)+ boxes.append({+ 'face': face_box,+ 'left_eye': left_eye_box,+ 'right_eye': right_eye_box,+ 'mouth': mouth_box+ })+ else:+ boxes.append(boxes[-1] if boxes else {+ 'face': [],+ 'left_eye': [],+ 'right_eye': [],+ 'mouth': []+ })+ torch.save(boxes, save_path)++if __name__ == "__main__":+ parser = argparse.ArgumentParser(description="Extract facial boxes from videos and save as .pt files.")+ parser.add_argument('--video_dir', type=str, default='/home/zyli/Repositories/x-nemo-inference/lv100/videos')+ parser.add_argument('--save_dir', type=str, default='/home/zyli/Repositories/x-nemo-inference/lv100/boxes_zyli')+ parser.add_argument('--workers', type=int, default=8)++ args = parser.parse_args()+ os.makedirs(args.save_dir, exist_ok=True)+ video_files = [f for f in os.listdir(args.video_dir) if f.endswith('.mp4')]++ process_func = partial(process_video, video_dir=args.video_dir, save_dir=args.save_dir)++ with ProcessPoolExecutor(max_workers=8) as executor:+ list(executor.map(process_func, video_files))\ No newline at end of filediff --git a/train_stage1.py b/train_stage1.pynew file mode 100644index 0000000..29ea64d--- /dev/null+++ b/train_stage1.py@@ -0,0 +1,833 @@+import argparse+import logging+import math+import os+import os.path as osp+import random+import warnings+from collections import OrderedDict+from datetime import datetime+from pathlib import Path+from tempfile import TemporaryDirectory+from torchvision import transforms++import cv2+import diffusers+from einops import rearrange+import mlflow+import numpy as np+import torch+import torch.nn as nn+import torch.nn.functional as F+import torch.utils.checkpoint+import transformers+from accelerate import Accelerator+from accelerate.logging import get_logger+from accelerate.utils import DistributedDataParallelKwargs+from diffusers import AutoencoderKL, AutoencoderKLTemporalDecoder+from src.scheduler.scheduler_ddim import DDIMScheduler+from diffusers.optimization import get_scheduler+from diffusers.utils import check_min_version+from diffusers.utils.import_utils import is_xformers_available+from omegaconf import OmegaConf+from PIL import Image+from tqdm.auto import tqdm+from transformers import CLIPVisionModelWithProjection+from decord import VideoReader+from src.dataset.portrait_image import PortraitImageDataset+from src.models.mutual_self_attention import ReferenceAttentionControl+from src.models.unet_2d_condition import UNet2DConditionModel+from src.models.unet_3d import UNet3DConditionModel+from src.pipelines.pipeline_pose2img import Pose2ImagePipeline+from src.utils.util import (+ delete_additional_ckpt,+ import_filename,+ read_frames,+ save_videos_grid,+ seed_everything,+ draw_keypoints,+)+from src.dataset.utils import scale_bb+from src.models.motion_encoder.encoder import MotEncoder+from src.models.motion_module import zero_module+warnings.filterwarnings("ignore")+import mediapipe as mp+from src.liveportrait.motion_extractor import MotionExtractor+from src.models.pose_guider import PoseGuider+import gc+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.+check_min_version("0.10.0.dev0")+from torchvision.utils import save_image++logger = get_logger(__name__, log_level="INFO")++class Net(nn.Module):+ def __init__(+ self,+ reference_unet: UNet2DConditionModel,+ denoising_unet: UNet3DConditionModel,+ motion_enc: MotEncoder,+ pose_guider: PoseGuider,+ reference_control_writer,+ reference_control_reader,+ ):+ super().__init__()+ self.reference_unet = reference_unet+ self.denoising_unet = denoising_unet+ self.reference_control_writer = reference_control_writer+ self.reference_control_reader = reference_control_reader++ self.motion_encoder = motion_enc+ self.pose_guider = pose_guider++ def forward(+ self,+ noisy_latents,+ timesteps,+ ref_image_latents,+ clip_image_embeds,+ tgt_face,+ tgt_pose,+ uncond_fwd: bool = False,+ ):+ pose_emb = self.pose_guider(tgt_pose)+ motion_emb = self.motion_encoder(tgt_face)++ if not uncond_fwd:+ self.reference_unet(+ ref_image_latents,+ torch.zeros_like(timesteps[:noisy_latents.shape[0]]),+ encoder_hidden_states=clip_image_embeds,+ return_dict=False,+ )+ self.reference_control_reader.update(self.reference_control_writer, drop_ratio=0.3)++ if random.random() < 0.1:+ pose_emb = torch.zeros_like(pose_emb)++ if random.random() < 0.1:+ motion_emb = torch.zeros_like(motion_emb)++ clip_image_embeds = [clip_image_embeds, motion_emb]++ model_pred = self.denoising_unet(+ noisy_latents,+ timesteps,+ encoder_hidden_states=clip_image_embeds,+ pose_cond_fea=pose_emb,+ ).sample++ return model_pred++def get_x0_from_eps(ddim, sample: torch.FloatTensor, model_output: torch.FloatTensor, timesteps: torch.IntTensor):+ video_length = sample.shape[2]+ sample = rearrange(sample, 'b c f h w -> (b f) c h w')+ model_output = rearrange(model_output, 'b c f h w -> (b f) c h w')+ alpha_prod = ddim.alphas_cumprod.cuda()+ alpha_prod_t = alpha_prod[timesteps]+ while len(alpha_prod_t.shape) < len(sample.shape):+ alpha_prod_t = alpha_prod_t.unsqueeze(-1)++ pred_original_sample = torch.sqrt(1./alpha_prod_t) * sample - torch.sqrt(1. / alpha_prod_t - 1) * model_output++ pred_original_sample = rearrange(pred_original_sample, '(b f) c h w -> b c f h w', f=video_length)++ return pred_original_sample++def crop_face(image_pil, face_mesh):+ image = np.array(image_pil)+ h, w = image.shape[:2]+ results = face_mesh.process(image)+ face_landmarks = results.multi_face_landmarks[0]+ coords = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]+ xs, ys = zip(*coords)+ x1, y1 = min(xs), min(ys)+ x2, y2 = max(xs), max(ys)+ face_box = (x1, y1, x2, y2)++ left, top, right, bot = scale_bb(face_box, scale=1.1, size=image.shape[:2])++ face_patch = image[int(top) : int(bot), int(left) : int(right)]++ return face_box, face_patch++def log_validation(+ vae,+ image_enc,+ net,+ scheduler,+ accelerator,+ width,+ height,+ face_mesh,+ pose_encoder,+ clip_length=100,+ generator=None,+):+ logger.info("Running validation... ")++ ori_net = accelerator.unwrap_model(net)+ reference_unet = ori_net.reference_unet+ denoising_unet = ori_net.denoising_unet+ motion_enc = ori_net.motion_encoder+ pose_gui = ori_net.pose_guider+ pose_enc = pose_encoder++ motion_enc.eval()++ # generator = torch.manual_seed(42)+ generator = torch.Generator(device='cuda').manual_seed(42)+ # cast unet dtype++ pipe = Pose2ImagePipeline(+ vae=vae,+ image_encoder=image_enc,+ reference_unet=reference_unet,+ denoising_unet=denoising_unet,+ motion_encoder=motion_enc,+ pose_encoder=pose_enc,+ pose_guider=pose_gui,+ scheduler=scheduler,+ )+ pipe = pipe.to(accelerator.device)++ ref_image_paths = [+ "path/to/ref_img_1.png",+ "path/to/ref_img_2.png",+ ]+ videos_path = [+ "path/to/ref_vid_1.png",+ "path/to/ref_vid_2.png",+ ]++ pose_transform = transforms.Compose(+ [transforms.Resize((height, width)), transforms.ToTensor()]+ )+ results = []+ for idy, pose_video_path in enumerate(videos_path):+ pose_images = read_frames(pose_video_path)++ boxes_path = pose_video_path.replace("videos", "boxes").replace(".mp4", ".pt")+ boxes = torch.load(boxes_path)++ pose_face = []+ ori_pose_images = []+ for idx_control, pose_image_pil in enumerate(pose_images[:clip_length]):+ face_bbox = boxes[idx_control]["face"]+ ori_pose_images.append(pose_image_pil)+ pose_image = np.array(pose_image_pil)+ left, top, right, bot = scale_bb(face_bbox, scale=1.1, size=pose_image.shape[:2])+ pose_image = pose_image[int(top) : int(bot), int(left) : int(right)]+ pose_image_pil = Image.fromarray(pose_image).convert("RGB")+ pose_face.append(pose_image_pil)++ pose_tensor_list = []+ ori_pose_tensor_list = []++ for idx_control, pose_image_pil in enumerate(ori_pose_images):+ pose_tensor_list.append(pose_transform(pose_face[idx_control]))+ ori_pose_tensor_list.append(pose_transform(pose_image_pil))+ pose_tensor = torch.stack(pose_tensor_list, dim=0) # (f, c, h, w)+ pose_tensor = pose_tensor.transpose(0, 1).unsqueeze(0)++ ori_pose_tensor = torch.stack(ori_pose_tensor_list, dim=0) # (f, c, h, w)+ ori_pose_tensor = ori_pose_tensor.transpose(0, 1).unsqueeze(0)++ for idx, ref_image_path in enumerate(ref_image_paths):+ ref_image_pil = Image.open(ref_image_path).convert("RGB")+ ref_bbox, ref_patch = crop_face(ref_image_pil, face_mesh)+ ref_face = Image.fromarray(ref_patch)++ ref_tensor = pose_transform(ref_image_pil).unsqueeze(0).unsqueeze(2).expand_as(pose_tensor)++ ref_name = f'{idx}'+ pose_name = f'{idy}'++ with torch.cuda.amp.autocast():+ gen_video = pipe(+ ori_pose_images,+ ref_image_pil,+ pose_face,+ ref_face,+ width,+ height,+ clip_length,+ 20,+ 2.5,+ generator=generator,+ ).videos++ # Concat it with pose tensor+ video = torch.cat([ref_tensor, pose_tensor, ori_pose_tensor, gen_video], dim=0)++ results.append({"name": f"{ref_name}_{pose_name}", "vid": video})++ del pipe+ torch.cuda.empty_cache()+ gc.collect()+ motion_enc.train()+ return results+++def main(cfg):+ kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)+ accelerator = Accelerator(+ gradient_accumulation_steps=cfg.solver.gradient_accumulation_steps,+ mixed_precision=cfg.solver.mixed_precision,+ log_with="mlflow",+ project_dir="./mlruns",+ kwargs_handlers=[kwargs],+ )++ # Make one log on every process with the configuration for debugging.+ logging.basicConfig(+ format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",+ datefmt="%m/%d/%Y %H:%M:%S",+ level=logging.INFO,+ )+ logger.info(accelerator.state, main_process_only=False)+ if accelerator.is_local_main_process:+ transformers.utils.logging.set_verbosity_warning()+ diffusers.utils.logging.set_verbosity_info()+ else:+ transformers.utils.logging.set_verbosity_error()+ diffusers.utils.logging.set_verbosity_error()++ # If passed along, set the training seed now.+ if cfg.seed is not None:+ seed_everything(cfg.seed)++ exp_name = cfg.exp_name+ save_dir = f"{cfg.output_dir}/{exp_name}"+ if accelerator.is_main_process and not os.path.exists(save_dir):+ os.makedirs(save_dir)++ inference_config_path = "./configs/inference/inference_stage1&2.yaml"+ infer_config = OmegaConf.load(inference_config_path)++ if cfg.weight_dtype == "fp16":+ weight_dtype = torch.float16+ elif cfg.weight_dtype == "fp32":+ weight_dtype = torch.float32+ else:+ raise ValueError(+ f"Do not support weight dtype: {cfg.weight_dtype} during training"+ )++ sched_kwargs = OmegaConf.to_container(+ infer_config.noise_scheduler_kwargs+ )++ val_noise_scheduler = DDIMScheduler(**sched_kwargs)+ train_noise_scheduler = DDIMScheduler(**sched_kwargs)+ vae = AutoencoderKL.from_pretrained(cfg.vae_model_path).to(+ "cuda", dtype=weight_dtype+ )++ image_enc = CLIPVisionModelWithProjection.from_pretrained(+ cfg.image_encoder_path,+ ).to(dtype=weight_dtype, device="cuda")++ reference_unet = UNet2DConditionModel.from_pretrained(+ cfg.base_model_path,+ subfolder="unet",+ ).to(device="cuda")+ denoising_unet = UNet3DConditionModel.from_pretrained_2d(+ cfg.base_model_path,+ "",+ subfolder="unet",+ unet_additional_kwargs=OmegaConf.to_container(+ infer_config.unet_additional_kwargs+ ),+ ).to(device="cuda")++ motion_encoder = MotEncoder().to(device="cuda")+ motion_encoder_params = torch.load(cfg.motion_encoder_path, map_location="cpu")+ old_weight = motion_encoder_params['final_proj.weight']+ new_weight = motion_encoder.final_proj.weight.data+ new_weight = old_weight[:, :new_weight.shape[1]]+ motion_encoder_params['final_proj.weight'] = new_weight++ motion_encoder.load_state_dict(+ motion_encoder_params, strict=False+ )++ pose_guider = PoseGuider().to(device="cuda")+ pose_guider.load_state_dict(torch.load(cfg.pose_guider_path, map_location='cpu'), strict=False)++ pose_encoder = MotionExtractor(num_kp=21).to(device="cuda").eval()+ pose_encoder.load_state_dict(torch.load(cfg.pose_encoder_path, map_location='cpu'), strict=False)++ denoising_unet.load_state_dict(+ torch.load(cfg.denoising_unet_path, map_location="cpu"),+ strict=False,+ )+ reference_unet.load_state_dict(+ torch.load(cfg.reference_unet_path, map_location="cpu"),+ )++ mp_face_mesh = mp.solutions.face_mesh+ face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1)++ # Freeze+ vae.requires_grad_(False)+ image_enc.requires_grad_(False)+ pose_encoder.requires_grad_(False)++ pose_guider.requires_grad_(True)+ motion_encoder.requires_grad_(True)+ denoising_unet.requires_grad_(True)+ reference_unet.requires_grad_(False)++ for name, param in reference_unet.named_parameters():+ if "up_blocks.3" in name:+ param.requires_grad_(False)+ else:+ param.requires_grad_(True)++ reference_control_writer = ReferenceAttentionControl(+ reference_unet,+ do_classifier_free_guidance=False,+ mode="write",+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ denoising_unet,+ do_classifier_free_guidance=False,+ mode="read",+ fusion_blocks="full",+ )++ net = Net(+ reference_unet,+ denoising_unet,+ motion_encoder,+ pose_guider,+ reference_control_writer,+ reference_control_reader,+ )++ if cfg.solver.enable_xformers_memory_efficient_attention:+ if is_xformers_available():+ reference_unet.enable_xformers_memory_efficient_attention()+ denoising_unet.enable_xformers_memory_efficient_attention()+ else:+ raise ValueError(+ "xformers is not available. Make sure it is installed correctly"+ )++ if cfg.solver.gradient_checkpointing:+ reference_unet.enable_gradient_checkpointing()+ denoising_unet.enable_gradient_checkpointing()++ if cfg.solver.scale_lr:+ learning_rate = (+ cfg.solver.learning_rate+ * cfg.solver.gradient_accumulation_steps+ * cfg.data.train_bs+ * accelerator.num_processes+ )+ else:+ learning_rate = cfg.solver.learning_rate++ # Initialize the optimizer+ if cfg.solver.use_8bit_adam:+ try:+ import bitsandbytes as bnb+ except ImportError:+ raise ImportError(+ "Please install bitsandbytes to use 8-bit Adam. You can do so by running `pip install bitsandbytes`"+ )++ optimizer_cls = bnb.optim.AdamW8bit+ else:+ optimizer_cls = torch.optim.AdamW++ trainable_params = list(filter(lambda p: p.requires_grad, net.parameters()))++ optimizer = optimizer_cls(+ trainable_params,+ lr=learning_rate,+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),+ weight_decay=cfg.solver.adam_weight_decay,+ eps=cfg.solver.adam_epsilon,+ )++ # Scheduler+ lr_scheduler = get_scheduler(+ cfg.solver.lr_scheduler,+ optimizer=optimizer,+ num_warmup_steps=cfg.solver.lr_warmup_steps+ * cfg.solver.gradient_accumulation_steps,+ num_training_steps=cfg.solver.max_train_steps+ * cfg.solver.gradient_accumulation_steps,+ )++ train_dataset = PortraitImageDataset(+ img_size=(cfg.data.train_width, cfg.data.train_height),+ img_scale=(0.9, 1.0),+ data_meta_paths=cfg.data.meta_paths,+ sample_margin=cfg.data.sample_margin,+ )+ train_dataloader = torch.utils.data.DataLoader(+ train_dataset, batch_size=cfg.data.train_bs, shuffle=True, num_workers=8, drop_last=True+ )++ # Prepare everything with our `accelerator`.+ (+ net,+ optimizer,+ train_dataloader,+ lr_scheduler,+ ) = accelerator.prepare(+ net,+ optimizer,+ train_dataloader,+ lr_scheduler,+ )++ # We need to recalculate our total training steps as the size of the training dataloader may have changed.+ num_update_steps_per_epoch = math.ceil(+ len(train_dataloader) / cfg.solver.gradient_accumulation_steps+ )+ # Afterwards we recalculate our number of training epochs+ num_train_epochs = math.ceil(+ cfg.solver.max_train_steps / num_update_steps_per_epoch+ )++ # We need to initialize the trackers we use, and also store our configuration.+ # The trackers initializes automatically on the main process.+ if accelerator.is_main_process:+ run_time = datetime.now().strftime("%Y%m%d-%H%M")+ accelerator.init_trackers(+ cfg.exp_name,+ init_kwargs={"mlflow": {"run_name": run_time}},+ )+ # dump config file+ mlflow.log_dict(OmegaConf.to_container(cfg), "config.yaml")++ # Train!+ total_batch_size = (+ cfg.data.train_bs+ * accelerator.num_processes+ * cfg.solver.gradient_accumulation_steps+ )++ logger.info("***** Running training *****")+ logger.info(f" Num examples = {len(train_dataset)}")+ logger.info(f" Num Epochs = {num_train_epochs}")+ logger.info(f" Instantaneous batch size per device = {cfg.data.train_bs}")+ logger.info(+ f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"+ )+ logger.info(+ f" Gradient Accumulation steps = {cfg.solver.gradient_accumulation_steps}"+ )+ logger.info(f" Total optimization steps = {cfg.solver.max_train_steps}")+ global_step = 0+ first_epoch = 0++ # Potentially load in the weights and states from a previous save+ if cfg.resume_from_checkpoint:+ if cfg.resume_from_checkpoint != "latest":+ resume_dir = cfg.resume_from_checkpoint+ else:+ resume_dir = save_dir+ # Get the most recent checkpoint+ dirs = os.listdir(resume_dir)+ dirs = [d for d in dirs if d.startswith("checkpoint")]+ dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))+ path = dirs[-1]+ accelerator.load_state(os.path.join(resume_dir, path))+ accelerator.print(f"Resuming from checkpoint {path}")+ global_step = int(path.split("-")[1])++ first_epoch = global_step // num_update_steps_per_epoch++ # Only show the progress bar once on each machine.+ progress_bar = tqdm(+ range(global_step, cfg.solver.max_train_steps),+ disable=not accelerator.is_local_main_process,+ )+ progress_bar.set_description("Steps")++ for epoch in range(first_epoch, num_train_epochs):+ train_loss = 0.0+ for step, batch in enumerate(train_dataloader):+ with accelerator.accumulate(net):+ # Convert videos to latent space+ pixel_values = batch["img"].to(weight_dtype) # b,c,h,w+ tgt_face = batch["tgt_face"]+ tgt_pose = batch["tgt_pose"]++ face_mask = batch['face_mask']+ local_mask = batch['local_mask']++ with torch.no_grad():+ latents = vae.encode(pixel_values).latent_dist.sample()+ latents = latents.unsqueeze(2)+ latents = latents * 0.18215 # (b f) c h w++ keypoints = pose_encoder(tgt_pose)+ tgt_pose = draw_keypoints(keypoints).unsqueeze(2).to('cuda')++ if epoch == 0 and step == 0:+ if accelerator.is_main_process:+ save_image(tgt_pose[:1,:,0], "image_kps.png")+ save_image((pixel_values[:1] + 1) / 2, "image_tgt.png")++ noise = torch.randn_like(latents)+ if cfg.noise_offset > 0.0:+ noise += cfg.noise_offset * torch.randn(+ (noise.shape[0], noise.shape[1], 1, 1, 1),+ device=noise.device,+ )++ bsz = latents.shape[0] # (b f)+ # Sample a random timestep for each frame+ timesteps = torch.randint(+ 0,+ train_noise_scheduler.num_train_timesteps,+ (bsz,),+ device=latents.device,+ )+ timesteps = timesteps.long()++ facial_mask = F.interpolate(face_mask, size=(64,64), mode='nearest').unsqueeze(2).repeat(1,4,1,1,1) + F.interpolate(local_mask, size=(64,64), mode='nearest').unsqueeze(2).repeat(1,4,1,1,1)++ uncond_fwd = random.random() < cfg.uncond_ratio+ clip_image_list = []+ ref_image_list = []+ for batch_idx, (ref_img, clip_img) in enumerate(+ zip(+ batch["ref_img"],+ batch["clip_image"],+ )+ ):+ clip_image_list.append(clip_img)+ ref_image_list.append(ref_img)++ with torch.no_grad():+ ref_img = torch.stack(ref_image_list, dim=0).to(+ dtype=vae.dtype, device=vae.device+ )+ ref_image_latents = vae.encode(+ ref_img+ ).latent_dist.sample() # (bs, d, 64, 64)+ ref_image_latents = ref_image_latents * 0.18215++ clip_img = torch.stack(clip_image_list, dim=0).to(+ dtype=image_enc.dtype, device=image_enc.device+ )+ clip_image_embeds = image_enc(+ clip_img.to("cuda", dtype=weight_dtype)+ ).image_embeds+ image_prompt_embeds = clip_image_embeds.unsqueeze(1) # (bs, 1, d)++ # add noise+ noisy_latents = train_noise_scheduler.add_noise(+ latents, noise, timesteps+ ) # (b f) c h w++ reference_control_reader.clear()+ reference_control_writer.clear()++ model_pred = net(+ noisy_latents,+ timesteps,+ ref_image_latents,+ image_prompt_embeds,+ tgt_face,+ tgt_pose,+ uncond_fwd,+ )++ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)++ loss = F.mse_loss(latents_pred.float(), latents.float(), reduction="mean")+ loss_local = F.mse_loss(latents_pred.float() * facial_mask, latents.float() * facial_mask, reduction="mean")+ loss = loss + loss_local++ # Gather the losses across all processes for logging (if we use distributed training).+ avg_loss = accelerator.gather(loss.repeat(cfg.data.train_bs)).mean()+ train_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps++ # Backpropagate+ accelerator.backward(loss)+ if accelerator.sync_gradients:+ accelerator.clip_grad_norm_(+ trainable_params,+ cfg.solver.max_grad_norm,+ )+ optimizer.step()+ lr_scheduler.step()+ optimizer.zero_grad()++ if accelerator.sync_gradients:+ reference_control_reader.clear()+ reference_control_writer.clear()+ progress_bar.update(1)+ global_step += 1+ accelerator.log({"train_loss": train_loss}, step=global_step)+ train_loss = 0.0+ if global_step % cfg.checkpointing_steps == 0:+ if accelerator.is_main_process:+ save_path = os.path.join(save_dir, f"checkpoint-{global_step}")+ delete_additional_ckpt(save_dir, 1)+ accelerator.save_state(save_path)++ if global_step % cfg.val.validation_steps == 0 or global_step == 1:+ if accelerator.is_main_process:+ generator = torch.Generator(device=accelerator.device)+ generator.manual_seed(cfg.seed)++ sample_dicts = log_validation(+ vae=vae,+ image_enc=image_enc,+ net=net,+ scheduler=val_noise_scheduler,+ accelerator=accelerator,+ width=cfg.data.train_width,+ height=cfg.data.train_height,+ face_mesh=face_mesh,+ pose_encoder=pose_encoder,+ generator=generator,+ )++ reference_control_writer = ReferenceAttentionControl(+ reference_unet,+ do_classifier_free_guidance=False,+ mode="write",+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ denoising_unet,+ do_classifier_free_guidance=False,+ mode="read",+ fusion_blocks="full",+ )++ for sample_id, sample_dict in enumerate(sample_dicts):+ sample_name = sample_dict["name"]+ vid = sample_dict["vid"]+ with TemporaryDirectory() as temp_dir:+ out_file = f"{temp_dir}/{global_step:06d}-{sample_name}.mp4"+ save_videos_grid(vid, out_file, n_rows=4, fps=25)+ mlflow.log_artifact(out_file)++ logs = {+ "loss": loss.detach().item(),+ "lr": lr_scheduler.get_last_lr()[0],+ }+ progress_bar.set_postfix(**logs)++ if global_step >= cfg.solver.max_train_steps:+ break++ # save model after each epoch+ if (+ epoch + 1+ ) % cfg.save_model_epoch_interval == 0 and accelerator.is_main_process:+ unwrap_net = accelerator.unwrap_model(net)+ save_checkpoint(+ unwrap_net.denoising_unet,+ save_dir,+ "denoising_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.reference_unet,+ save_dir,+ "reference_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.motion_encoder,+ save_dir,+ "motion_encoder",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.pose_guider,+ save_dir,+ "pose_guider",+ global_step,+ total_limit=2,+ )++ unwrap_net = accelerator.unwrap_model(net)+ save_checkpoint(+ unwrap_net.denoising_unet,+ save_dir,+ "denoising_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.reference_unet,+ save_dir,+ "reference_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.motion_encoder,+ save_dir,+ "motion_encoder",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.pose_guider,+ save_dir,+ "pose_guider",+ global_step,+ total_limit=2,+ )++ # Create the pipeline using the trained modules and save it.+ accelerator.wait_for_everyone()+ accelerator.end_training()+++def save_checkpoint(model, save_dir, prefix, ckpt_num, total_limit=None):+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")++ if total_limit is not None:+ checkpoints = os.listdir(save_dir)+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]+ checkpoints = sorted(+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])+ )++ if len(checkpoints) >= total_limit:+ num_to_remove = len(checkpoints) - total_limit + 1+ removing_checkpoints = checkpoints[0:num_to_remove]+ logger.info(+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"+ )+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")++ for removing_checkpoint in removing_checkpoints:+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)+ os.remove(removing_checkpoint)++ state_dict = model.state_dict()+ torch.save(state_dict, save_path)+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--config", type=str, default="./configs/train/personalive_stage1.yaml")+ args = parser.parse_args()++ if args.config[-5:] == ".yaml":+ config = OmegaConf.load(args.config)+ elif args.config[-3:] == ".py":+ config = import_filename(args.config).cfg+ else:+ raise ValueError("Do not support this format config file")+ main(config)diff --git a/train_stage2.py b/train_stage2.pynew file mode 100644index 0000000..b498885--- /dev/null+++ b/train_stage2.py@@ -0,0 +1,969 @@+import sys+sys.path.append('./src/stylegan2')++import argparse+import logging+import math+import os+import os.path as osp+import random+import warnings+from collections import OrderedDict+from datetime import datetime+from pathlib import Path+from tempfile import TemporaryDirectory+from torchvision import transforms++import cv2+import lpips+import diffusers+from einops import rearrange+import mlflow+import numpy as np+import torch+import torch.nn as nn+import torch.nn.functional as F+import torch.utils.checkpoint+import transformers+from accelerate import Accelerator+from accelerate.logging import get_logger+from accelerate.utils import DistributedDataParallelKwargs+from diffusers import AutoencoderKL, AutoencoderKLTemporalDecoder+from src.scheduler.scheduler_ddim import DDIMScheduler+from diffusers.optimization import get_scheduler+from diffusers.utils import check_min_version+from diffusers.utils.import_utils import is_xformers_available+from omegaconf import OmegaConf+from PIL import Image+from tqdm.auto import tqdm+from transformers import CLIPVisionModelWithProjection+from decord import VideoReader+from src.dataset.portrait_image import PortraitImageDataset+from src.models.mutual_self_attention import ReferenceAttentionControl+from src.models.unet_2d_condition import UNet2DConditionModel+from src.models.unet_3d import UNet3DConditionModel+from src.pipelines.pipeline_pose2img import Pose2ImagePipeline+from src.utils.util import (+ delete_additional_ckpt,+ import_filename,+ read_frames,+ save_videos_grid,+ seed_everything,+ draw_keypoints,+)+from src.dataset.utils import scale_bb+from src.models.motion_encoder.encoder import MotEncoder+from src.models.motion_module import zero_module+warnings.filterwarnings("ignore")+import mediapipe as mp+from src.liveportrait.motion_extractor import MotionExtractor+from src.models.pose_guider import PoseGuider+import gc+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.+check_min_version("0.10.0.dev0")+from torchvision.utils import save_image+from src.stylegan2.discriminator import Discriminator as StyleGAN2Discriminator+from torch.nn.parallel import DistributedDataParallel as DDP++logger = get_logger(__name__, log_level="INFO")++class Net(nn.Module):+ def __init__(+ self,+ reference_unet: UNet2DConditionModel,+ denoising_unet: UNet3DConditionModel,+ motion_enc: MotEncoder,+ pose_guider: PoseGuider,+ reference_control_writer,+ reference_control_reader,+ ):+ super().__init__()+ self.reference_unet = reference_unet+ self.denoising_unet = denoising_unet+ self.reference_control_writer = reference_control_writer+ self.reference_control_reader = reference_control_reader++ self.motion_encoder = motion_enc+ self.pose_guider = pose_guider++ def forward(+ self,+ noisy_latents,+ timesteps,+ ref_image_latents,+ clip_image_embeds,+ tgt_face,+ tgt_pose,+ ref_flag,+ mot_flag,+ pose_flag,+ uncond_fwd: bool = False,+ ):+ pose_emb = self.pose_guider(tgt_pose)+ motion_emb = self.motion_encoder(tgt_face)++ if not uncond_fwd and ref_flag:+ self.reference_unet(+ ref_image_latents,+ torch.zeros_like(timesteps[:noisy_latents.shape[0]]),+ encoder_hidden_states=clip_image_embeds,+ return_dict=False,+ )+ self.reference_control_reader.update(self.reference_control_writer, drop_ratio=0.3)++ if pose_flag:+ pose_emb = torch.zeros_like(pose_emb)++ if mot_flag:+ motion_emb = torch.zeros_like(motion_emb)++ clip_image_embeds = [clip_image_embeds, motion_emb]++ model_pred = self.denoising_unet(+ noisy_latents,+ timesteps,+ encoder_hidden_states=clip_image_embeds,+ pose_cond_fea=pose_emb,+ ).sample++ return model_pred++ def update_reference(+ self,+ noisy_latents,+ timesteps,+ ref_image_latents,+ clip_image_embeds,+ ):+ self.reference_unet(+ ref_image_latents,+ torch.zeros_like(timesteps[:noisy_latents.shape[0]]),+ encoder_hidden_states=clip_image_embeds,+ return_dict=False,+ )+ self.reference_control_reader.update(self.reference_control_writer, drop_ratio=0.3)++def get_x0_from_eps(ddim, sample: torch.FloatTensor, model_output: torch.FloatTensor, timesteps: torch.IntTensor):+ video_length = sample.shape[2]+ sample = rearrange(sample, 'b c f h w -> (b f) c h w')+ model_output = rearrange(model_output, 'b c f h w -> (b f) c h w')+ alpha_prod = ddim.alphas_cumprod.cuda()+ alpha_prod_t = alpha_prod[timesteps]+ while len(alpha_prod_t.shape) < len(sample.shape):+ alpha_prod_t = alpha_prod_t.unsqueeze(-1)++ pred_original_sample = torch.sqrt(1./alpha_prod_t) * sample - torch.sqrt(1. / alpha_prod_t - 1) * model_output++ pred_original_sample = rearrange(pred_original_sample, '(b f) c h w -> b c f h w', f=video_length)++ return pred_original_sample++def crop_face(image_pil, face_mesh):+ image = np.array(image_pil)+ h, w = image.shape[:2]+ results = face_mesh.process(image)+ face_landmarks = results.multi_face_landmarks[0]+ coords = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]+ xs, ys = zip(*coords)+ x1, y1 = min(xs), min(ys)+ x2, y2 = max(xs), max(ys)+ face_box = (x1, y1, x2, y2)++ left, top, right, bot = scale_bb(face_box, scale=1.1, size=image.shape[:2])++ face_patch = image[int(top) : int(bot), int(left) : int(right)]++ return face_box, face_patch++def decode_latents(vae: AutoencoderKL, latents, decode_chunk_size=4):+ latents = latents.to(vae.dtype)+ video_length = latents.shape[2]+ latents = 1 / 0.18215 * latents+ latents = rearrange(latents, "b c f h w -> (b f) c h w")+ # video = self.vae.decode(latents).sample+ video = []+ for frame_idx in range(0, latents.shape[0], decode_chunk_size):+ video.append(vae.decode(latents[frame_idx : frame_idx + decode_chunk_size]).sample)+ video = torch.cat(video)+ # video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)+ # video = (video / 2 + 0.5).clamp(0, 1)+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16+ return video++def log_validation(+ vae,+ image_enc,+ net,+ scheduler,+ accelerator,+ width,+ height,+ face_mesh,+ pose_encoder,+ timesteps_list,+ clip_length=100,+ generator=None,+):+ logger.info("Running validation... ")++ ori_net = accelerator.unwrap_model(net)+ reference_unet = ori_net.reference_unet+ denoising_unet = ori_net.denoising_unet+ motion_enc = ori_net.motion_encoder+ pose_gui = ori_net.pose_guider+ pose_enc = pose_encoder++ motion_enc.eval()++ # generator = torch.manual_seed(42)+ generator = torch.Generator(device='cuda').manual_seed(42)+ # cast unet dtype++ pipe = Pose2ImagePipeline(+ vae=vae,+ image_encoder=image_enc,+ reference_unet=reference_unet,+ denoising_unet=denoising_unet,+ motion_encoder=motion_enc,+ pose_encoder=pose_enc,+ pose_guider=pose_gui,+ scheduler=scheduler,+ )+ pipe = pipe.to(accelerator.device)++ ref_image_paths = [+ "path/to/ref_img_1.png",+ "path/to/ref_img_2.png",+ ]+ videos_path = [+ "path/to/ref_vid_1.png",+ "path/to/ref_vid_2.png",+ ]++ pose_transform = transforms.Compose(+ [transforms.Resize((height, width)), transforms.ToTensor()]+ )+ results = []+ for idy, pose_video_path in enumerate(videos_path):+ pose_images = read_frames(pose_video_path)++ boxes_path = pose_video_path.replace("videos", "boxes").replace(".mp4", ".pt")+ boxes = torch.load(boxes_path)++ pose_face = []+ ori_pose_images = []+ for idx_control, pose_image_pil in enumerate(pose_images[:clip_length]):+ face_bbox = boxes[idx_control]["face"]+ ori_pose_images.append(pose_image_pil)+ pose_image = np.array(pose_image_pil)+ left, top, right, bot = scale_bb(face_bbox, scale=1.1, size=pose_image.shape[:2])+ pose_image = pose_image[int(top) : int(bot), int(left) : int(right)]+ pose_image_pil = Image.fromarray(pose_image).convert("RGB")+ pose_face.append(pose_image_pil)++ pose_tensor_list = []+ ori_pose_tensor_list = []++ for idx_control, pose_image_pil in enumerate(ori_pose_images):+ pose_tensor_list.append(pose_transform(pose_face[idx_control]))+ ori_pose_tensor_list.append(pose_transform(pose_image_pil))+ pose_tensor = torch.stack(pose_tensor_list, dim=0) # (f, c, h, w)+ pose_tensor = pose_tensor.transpose(0, 1).unsqueeze(0)++ ori_pose_tensor = torch.stack(ori_pose_tensor_list, dim=0) # (f, c, h, w)+ ori_pose_tensor = ori_pose_tensor.transpose(0, 1).unsqueeze(0)++ for idx, ref_image_path in enumerate(ref_image_paths):+ ref_image_pil = Image.open(ref_image_path).convert("RGB")+ ref_bbox, ref_patch = crop_face(ref_image_pil, face_mesh)+ ref_face = Image.fromarray(ref_patch)++ ref_tensor = pose_transform(ref_image_pil).unsqueeze(0).unsqueeze(2).expand_as(pose_tensor)++ ref_name = f'{idx}'+ pose_name = f'{idy}'++ with torch.cuda.amp.autocast():+ gen_video = pipe(+ ori_pose_images,+ ref_image_pil,+ pose_face,+ ref_face,+ width,+ height,+ clip_length,+ 4,+ 1.0,+ timesteps_list=timesteps_list,+ generator=generator,+ ).videos++ # Concat it with pose tensor+ video = torch.cat([ref_tensor, pose_tensor, ori_pose_tensor, gen_video], dim=0)++ results.append({"name": f"{ref_name}_{pose_name}", "vid": video})++ del pipe+ torch.cuda.empty_cache()+ gc.collect()+ motion_enc.train()+ return results+++def main(cfg):+ kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)+ accelerator = Accelerator(+ gradient_accumulation_steps=cfg.solver.gradient_accumulation_steps,+ mixed_precision=cfg.solver.mixed_precision,+ log_with="mlflow",+ project_dir="./mlruns",+ kwargs_handlers=[kwargs],+ )++ # Make one log on every process with the configuration for debugging.+ logging.basicConfig(+ format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",+ datefmt="%m/%d/%Y %H:%M:%S",+ level=logging.INFO,+ )+ logger.info(accelerator.state, main_process_only=False)+ if accelerator.is_local_main_process:+ transformers.utils.logging.set_verbosity_warning()+ diffusers.utils.logging.set_verbosity_info()+ else:+ transformers.utils.logging.set_verbosity_error()+ diffusers.utils.logging.set_verbosity_error()++ # If passed along, set the training seed now.+ if cfg.seed is not None:+ seed_everything(cfg.seed)++ exp_name = cfg.exp_name+ save_dir = f"{cfg.output_dir}/{exp_name}"+ if accelerator.is_main_process and not os.path.exists(save_dir):+ os.makedirs(save_dir)++ inference_config_path = "./configs/inference/inference_stage1&2.yaml"+ infer_config = OmegaConf.load(inference_config_path)++ if cfg.weight_dtype == "fp16":+ weight_dtype = torch.float16+ elif cfg.weight_dtype == "bf16":+ weight_dtype = torch.bfloat16+ elif cfg.weight_dtype == "fp32":+ weight_dtype = torch.float32+ else:+ raise ValueError(+ f"Do not support weight dtype: {cfg.weight_dtype} during training"+ )++ sched_kwargs = OmegaConf.to_container(+ infer_config.noise_scheduler_kwargs+ )++ val_noise_scheduler = DDIMScheduler(**sched_kwargs)+ train_noise_scheduler = DDIMScheduler(**sched_kwargs)+ vae = AutoencoderKL.from_pretrained(cfg.vae_model_path).to(+ "cuda", dtype=weight_dtype+ )+ net_lpips = lpips.LPIPS(net='vgg').cuda()++ image_enc = CLIPVisionModelWithProjection.from_pretrained(+ cfg.image_encoder_path,+ ).to(dtype=weight_dtype, device="cuda")++ reference_unet = UNet2DConditionModel.from_pretrained(+ cfg.base_model_path,+ subfolder="unet",+ ).to(device="cuda")+ denoising_unet = UNet3DConditionModel.from_pretrained_2d(+ cfg.base_model_path,+ "",+ subfolder="unet",+ unet_additional_kwargs=OmegaConf.to_container(+ infer_config.unet_additional_kwargs+ ),+ ).to(device="cuda")++ motion_encoder = MotEncoder().to(device="cuda")+ motion_encoder.load_state_dict(+ torch.load(cfg.motion_encoder_path, map_location="cpu"), strict=False+ )++ pose_guider = PoseGuider().to(device="cuda")+ pose_guider.load_state_dict(torch.load(cfg.pose_guider_path, map_location='cpu'), strict=False)++ pose_encoder = MotionExtractor(num_kp=21).to(device="cuda").eval()+ pose_encoder.load_state_dict(torch.load(cfg.pose_encoder_path, map_location='cpu'), strict=False)++ denoising_unet.load_state_dict(+ torch.load(cfg.denoising_unet_path, map_location="cpu"),+ strict=False,+ )+ reference_unet.load_state_dict(+ torch.load(cfg.reference_unet_path, map_location="cpu"),+ )++ mp_face_mesh = mp.solutions.face_mesh+ face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1)++ discriminator = StyleGAN2Discriminator()+ discriminator = discriminator.to(device="cuda")+ discriminator.load_state_dict(+ torch.load(cfg.discriminator_path, map_location="cpu"), strict=False+ )++ # Freeze+ vae.requires_grad_(False)+ image_enc.requires_grad_(False)+ net_lpips.requires_grad_(False)+ pose_encoder.requires_grad_(False)++ pose_guider.requires_grad_(True)+ motion_encoder.requires_grad_(True)+ denoising_unet.requires_grad_(True)+ reference_unet.requires_grad_(False)+ discriminator.requires_grad_(True)++ for name, param in reference_unet.named_parameters():+ if "up_blocks.3" in name:+ param.requires_grad_(False)+ else:+ param.requires_grad_(True)++ reference_control_writer = ReferenceAttentionControl(+ reference_unet,+ do_classifier_free_guidance=False,+ mode="write",+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ denoising_unet,+ do_classifier_free_guidance=False,+ mode="read",+ fusion_blocks="full",+ )++ net = Net(+ reference_unet,+ denoising_unet,+ motion_encoder,+ pose_guider,+ reference_control_writer,+ reference_control_reader,+ )++ if cfg.solver.enable_xformers_memory_efficient_attention:+ if is_xformers_available():+ reference_unet.enable_xformers_memory_efficient_attention()+ denoising_unet.enable_xformers_memory_efficient_attention()+ else:+ raise ValueError(+ "xformers is not available. Make sure it is installed correctly"+ )++ if cfg.solver.gradient_checkpointing:+ reference_unet.enable_gradient_checkpointing()+ denoising_unet.enable_gradient_checkpointing()++ if cfg.solver.scale_lr:+ learning_rate = (+ cfg.solver.learning_rate+ * cfg.solver.gradient_accumulation_steps+ * cfg.data.train_bs+ * accelerator.num_processes+ )+ else:+ learning_rate = cfg.solver.learning_rate++ # Initialize the optimizer+ if cfg.solver.use_8bit_adam:+ try:+ import bitsandbytes as bnb+ except ImportError:+ raise ImportError(+ "Please install bitsandbytes to use 8-bit Adam. You can do so by running `pip install bitsandbytes`"+ )++ optimizer_cls = bnb.optim.AdamW8bit+ else:+ optimizer_cls = torch.optim.AdamW++ trainable_params = list(filter(lambda p: p.requires_grad, net.parameters()))++ optimizer = optimizer_cls(+ trainable_params,+ lr=learning_rate,+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),+ weight_decay=cfg.solver.adam_weight_decay,+ eps=cfg.solver.adam_epsilon,+ )++ dis_optimizer = optimizer_cls(+ discriminator.parameters(),+ lr=learning_rate * 2,+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),+ weight_decay=cfg.solver.adam_weight_decay,+ eps=cfg.solver.adam_epsilon,+ )++ # Scheduler+ lr_scheduler = get_scheduler(+ cfg.solver.lr_scheduler,+ optimizer=optimizer,+ num_warmup_steps=cfg.solver.lr_warmup_steps+ * cfg.solver.gradient_accumulation_steps,+ num_training_steps=cfg.solver.max_train_steps+ * cfg.solver.gradient_accumulation_steps,+ )++ train_dataset = PortraitImageDataset(+ img_size=(cfg.data.train_width, cfg.data.train_height),+ img_scale=(0.9, 1.0),+ data_meta_paths=cfg.data.meta_paths,+ sample_margin=cfg.data.sample_margin,+ )+ train_dataloader = torch.utils.data.DataLoader(+ train_dataset, batch_size=cfg.data.train_bs, shuffle=True, num_workers=8, drop_last=True+ )++ # Prepare everything with our `accelerator`.+ (+ net,+ optimizer,+ discriminator,+ dis_optimizer,+ train_dataloader,+ lr_scheduler,+ ) = accelerator.prepare(+ net,+ optimizer,+ discriminator,+ dis_optimizer,+ train_dataloader,+ lr_scheduler,+ )++ # We need to recalculate our total training steps as the size of the training dataloader may have changed.+ num_update_steps_per_epoch = math.ceil(+ len(train_dataloader) / cfg.solver.gradient_accumulation_steps+ )+ # Afterwards we recalculate our number of training epochs+ num_train_epochs = math.ceil(+ cfg.solver.max_train_steps / num_update_steps_per_epoch+ )++ # We need to initialize the trackers we use, and also store our configuration.+ # The trackers initializes automatically on the main process.+ if accelerator.is_main_process:+ run_time = datetime.now().strftime("%Y%m%d-%H%M")+ accelerator.init_trackers(+ cfg.exp_name,+ init_kwargs={"mlflow": {"run_name": run_time}},+ )+ # dump config file+ mlflow.log_dict(OmegaConf.to_container(cfg), "config.yaml")++ # Train!+ total_batch_size = (+ cfg.data.train_bs+ * accelerator.num_processes+ * cfg.solver.gradient_accumulation_steps+ )++ logger.info("***** Running training *****")+ logger.info(f" Num examples = {len(train_dataset)}")+ logger.info(f" Num Epochs = {num_train_epochs}")+ logger.info(f" Instantaneous batch size per device = {cfg.data.train_bs}")+ logger.info(+ f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"+ )+ logger.info(+ f" Gradient Accumulation steps = {cfg.solver.gradient_accumulation_steps}"+ )+ logger.info(f" Total optimization steps = {cfg.solver.max_train_steps}")+ global_step = 0+ first_epoch = 0++ # Potentially load in the weights and states from a previous save+ if cfg.resume_from_checkpoint:+ if cfg.resume_from_checkpoint != "latest":+ resume_dir = cfg.resume_from_checkpoint+ else:+ resume_dir = save_dir+ # Get the most recent checkpoint+ dirs = os.listdir(resume_dir)+ dirs = [d for d in dirs if d.startswith("checkpoint")]+ dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))+ path = dirs[-1]+ accelerator.load_state(os.path.join(resume_dir, path))+ accelerator.print(f"Resuming from checkpoint {path}")+ global_step = int(path.split("-")[1])++ first_epoch = global_step // num_update_steps_per_epoch++ # Only show the progress bar once on each machine.+ progress_bar = tqdm(+ range(global_step, cfg.solver.max_train_steps),+ disable=not accelerator.is_local_main_process,+ )+ progress_bar.set_description("Steps")++ for epoch in range(first_epoch, num_train_epochs):+ train_loss = 0.0+ train_dis_loss = 0.0+ for step, batch in enumerate(train_dataloader):+ with accelerator.accumulate(None):+ # Convert videos to latent space+ pixel_values = batch["img"].to(weight_dtype) # b,c,h,w+ tgt_face = batch["tgt_face"]+ tgt_pose = batch["tgt_pose"]++ face_mask = batch['face_mask']+ local_mask = batch['local_mask']++ with torch.no_grad():+ latents = vae.encode(pixel_values).latent_dist.sample()+ latents = latents.unsqueeze(2)+ latents = latents * 0.18215 # (b f) c h w++ keypoints = pose_encoder(tgt_pose)+ tgt_pose = draw_keypoints(keypoints).unsqueeze(2).to('cuda')++ if epoch == 0 and step == 0:+ if accelerator.is_main_process:+ save_image(tgt_pose[:1,:,0], "image_kps.png")+ save_image((pixel_values[:1] + 1) / 2, "image_tgt.png")++ noise = torch.randn_like(latents)++ timesteps_list = cfg.timesteps_list++ bsz = latents.shape[0] # (b f)+ timesteps = torch.ones(bsz, device=latents.device) * timesteps_list[0]+ timesteps = timesteps.long()++ facial_mask_pix = face_mask.repeat(1,3,1,1) + local_mask.repeat(1,3,1,1)++ uncond_fwd = random.random() < cfg.uncond_ratio+ clip_image_list = []+ ref_image_list = []+ for batch_idx, (ref_img, clip_img) in enumerate(+ zip(+ batch["ref_img"],+ batch["clip_image"],+ )+ ):+ clip_image_list.append(clip_img)+ ref_image_list.append(ref_img)++ with torch.no_grad():+ ref_img = torch.stack(ref_image_list, dim=0).to(+ dtype=vae.dtype, device=vae.device+ )+ ref_image_latents = vae.encode(+ ref_img+ ).latent_dist.sample() # (bs, d, 64, 64)+ ref_image_latents = ref_image_latents * 0.18215++ clip_img = torch.stack(clip_image_list, dim=0).to(+ dtype=image_enc.dtype, device=image_enc.device+ )+ clip_image_embeds = image_enc(+ clip_img.to("cuda", dtype=weight_dtype)+ ).image_embeds+ image_prompt_embeds = clip_image_embeds.unsqueeze(1) # (bs, 1, d)++ noisy_latents = noise++ update_step = random.randint(1, 4)++ ref_flag = True+ mot_flag = random.random() < 0.1+ pose_flag = random.random() < 0.1+ reference_control_reader.clear()+ reference_control_writer.clear()++ for i in range(update_step-1):+ with torch.no_grad():+ model_pred = net(+ noisy_latents,+ timesteps,+ ref_image_latents,+ image_prompt_embeds,+ tgt_face,+ tgt_pose,+ ref_flag,+ mot_flag,+ pose_flag,+ uncond_fwd,+ )++ ref_flag = False++ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)+ timesteps = torch.ones(bsz, device=latents.device) * timesteps_list[i+1]+ timesteps = timesteps.long()++ noise = torch.randn_like(latents_pred)+ noisy_latents = train_noise_scheduler.add_noise(+ latents_pred, noise, timesteps+ ).detach()++ if ref_flag == False:+ reference_control_reader.clear()+ reference_control_writer.clear()+ ref_flag = True++ model_pred = net(+ noisy_latents,+ timesteps,+ ref_image_latents,+ image_prompt_embeds,+ tgt_face,+ tgt_pose,+ ref_flag,+ mot_flag,+ pose_flag,+ uncond_fwd,+ )++ discriminator.requires_grad_(False)++ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)++ image_pred = decode_latents(vae, latents_pred)+ loss = F.mse_loss(image_pred.float(), pixel_values.float(), reduction="mean")+ loss += F.mse_loss(image_pred.float() * facial_mask_pix, pixel_values.float() * facial_mask_pix, reduction="mean")+ loss_lpips = net_lpips(image_pred.float(), pixel_values.float()).mean() # 输入应该是[-1,1]++ if global_step > 3000: # This value may not be optimal+ adv_loss = discriminator(image_pred.float(), pixel_values.float(), timesteps)+ else:+ adv_loss = torch.tensor(0.0).to(loss.device)+ loss = loss + loss_lpips * 2 + adv_loss * 0.05++ # Gather the losses across all processes for logging (if we use distributed training).+ avg_loss = accelerator.gather(loss.repeat(cfg.data.train_bs)).mean()+ train_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps++ # Backpropagate+ accelerator.backward(loss)+ if accelerator.sync_gradients:+ accelerator.clip_grad_norm_(+ trainable_params,+ cfg.solver.max_grad_norm,+ )+ optimizer.step()+ lr_scheduler.step()+ optimizer.zero_grad(set_to_none=True)++ if global_step > 5000: # This value may not be optimal+ discriminator.requires_grad_(True)+ dis_loss = discriminator(image_pred.float().detach(), pixel_values.float(), timesteps, gen=False)+ avg_loss = accelerator.gather(dis_loss.repeat(cfg.data.train_bs)).mean()+ train_dis_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps++ accelerator.backward(dis_loss)+ if accelerator.sync_gradients:+ accelerator.clip_grad_norm_(+ discriminator.parameters(),+ cfg.solver.max_grad_norm,+ )+ dis_optimizer.step()+ dis_optimizer.zero_grad(set_to_none=True)+ else:+ dis_loss = torch.tensor(0.0).to(loss.device)+++ if accelerator.sync_gradients:+ reference_control_reader.clear()+ reference_control_writer.clear()+ progress_bar.update(1)+ global_step += 1+ accelerator.log({"train_loss": train_loss, "train_dis_loss": train_dis_loss}, step=global_step)+ train_loss = 0.0+ train_dis_loss = 0.0+ if global_step % cfg.checkpointing_steps == 0:+ if accelerator.is_main_process:+ save_path = os.path.join(save_dir, f"checkpoint-{global_step}")+ delete_additional_ckpt(save_dir, 1)+ accelerator.save_state(save_path)++ if global_step % cfg.val.validation_steps == 0 or global_step == 1:+ accelerator.wait_for_everyone()+ if accelerator.is_main_process:+ generator = torch.Generator(device=accelerator.device)+ generator.manual_seed(cfg.seed)++ sample_dicts = log_validation(+ vae=vae,+ image_enc=image_enc,+ net=net,+ scheduler=val_noise_scheduler,+ accelerator=accelerator,+ width=cfg.data.train_width,+ height=cfg.data.train_height,+ face_mesh=face_mesh,+ pose_encoder=pose_encoder,+ timesteps_list=timesteps_list,+ generator=generator,+ )++ reference_control_writer = ReferenceAttentionControl(+ reference_unet,+ do_classifier_free_guidance=False,+ mode="write",+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ denoising_unet,+ do_classifier_free_guidance=False,+ mode="read",+ fusion_blocks="full",+ )++ if accelerator.is_main_process:+ for sample_id, sample_dict in enumerate(sample_dicts):+ sample_name = sample_dict["name"]+ vid = sample_dict["vid"]+ with TemporaryDirectory() as temp_dir:+ out_file = f"{temp_dir}/{global_step:06d}-{sample_name}.mp4"+ save_videos_grid(vid, out_file, n_rows=4, fps=25)+ mlflow.log_artifact(out_file)++ logs = {+ "loss": loss.detach().item(),+ "step": update_step,+ "dis": dis_loss.detach().item(),+ "adv": adv_loss.detach().item(),+ "lr": lr_scheduler.get_last_lr()[0],+ }+ progress_bar.set_postfix(**logs)++ if global_step >= cfg.solver.max_train_steps:+ break++ # save model after each epoch+ if (+ epoch + 1+ ) % cfg.save_model_epoch_interval == 0 and accelerator.is_main_process:+ unwrap_net = accelerator.unwrap_model(net)+ unwrap_discriminator = accelerator.unwrap_model(discriminator)+ save_checkpoint(+ unwrap_net.denoising_unet,+ save_dir,+ "denoising_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.reference_unet,+ save_dir,+ "reference_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.motion_encoder,+ save_dir,+ "motion_encoder",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.pose_guider,+ save_dir,+ "pose_guider",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_discriminator,+ save_dir,+ "discriminator",+ global_step,+ total_limit=2,+ )++ unwrap_net = accelerator.unwrap_model(net)+ unwrap_discriminator = accelerator.unwrap_model(discriminator)+ save_checkpoint(+ unwrap_net.denoising_unet,+ save_dir,+ "denoising_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.reference_unet,+ save_dir,+ "reference_unet",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.motion_encoder,+ save_dir,+ "motion_encoder",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_net.pose_guider,+ save_dir,+ "pose_guider",+ global_step,+ total_limit=2,+ )+ save_checkpoint(+ unwrap_discriminator,+ save_dir,+ "discriminator",+ global_step,+ total_limit=2,+ )++ # Create the pipeline using the trained modules and save it.+ accelerator.wait_for_everyone()+ accelerator.end_training()+++def save_checkpoint(model, save_dir, prefix, ckpt_num, total_limit=None):+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")++ if total_limit is not None:+ checkpoints = os.listdir(save_dir)+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]+ checkpoints = sorted(+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])+ )++ if len(checkpoints) >= total_limit:+ num_to_remove = len(checkpoints) - total_limit + 1+ removing_checkpoints = checkpoints[0:num_to_remove]+ logger.info(+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"+ )+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")++ for removing_checkpoint in removing_checkpoints:+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)+ os.remove(removing_checkpoint)++ state_dict = model.state_dict()+ torch.save(state_dict, save_path)+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--config", type=str, default="./configs/train/personalive_stage2.yaml")+ args = parser.parse_args()++ if args.config[-5:] == ".yaml":+ config = OmegaConf.load(args.config)+ elif args.config[-3:] == ".py":+ config = import_filename(args.config).cfg+ else:+ raise ValueError("Do not support this format config file")+ main(config)diff --git a/train_stage3.py b/train_stage3.pynew file mode 100644index 0000000..a8407b1--- /dev/null+++ b/train_stage3.py@@ -0,0 +1,954 @@+import sys+sys.path.append('./src/stylegan2')++import argparse+import logging+import math+import os+import os.path as osp+import random+import warnings+from collections import OrderedDict+from datetime import datetime+from pathlib import Path+from tempfile import TemporaryDirectory+from torchvision import transforms++import cv2+import lpips+import diffusers+from einops import rearrange+import mlflow+import numpy as np+import torch+import torch.nn as nn+import torch.nn.functional as F+import torch.utils.checkpoint+import transformers+from accelerate import Accelerator+from accelerate.logging import get_logger+from accelerate.utils import DistributedDataParallelKwargs+from diffusers import AutoencoderKL, AutoencoderKLTemporalDecoder, AutoencoderTiny+from src.scheduler.scheduler_ddim import DDIMScheduler+from diffusers.optimization import get_scheduler+from diffusers.utils import check_min_version+from diffusers.utils.import_utils import is_xformers_available+from omegaconf import OmegaConf+from PIL import Image+from tqdm.auto import tqdm+from transformers import CLIPVisionModelWithProjection+from decord import VideoReader+from src.dataset.portrait_video import PortraitVideoDataset+from src.models.mutual_self_attention import ReferenceAttentionControl+from src.models.unet_2d_condition import UNet2DConditionModel+from src.models.unet_3d import UNet3DConditionModel+from src.pipelines.pipeline_pose2vid import Pose2VideoPipeline+from src.utils.util import (+ delete_additional_ckpt,+ import_filename,+ read_frames,+ save_videos_grid,+ seed_everything,+ draw_keypoints,+)+from src.dataset.utils import scale_bb+from src.models.motion_encoder.encoder import MotEncoder+from src.models.motion_module import zero_module+warnings.filterwarnings("ignore")+import mediapipe as mp+from src.liveportrait.motion_extractor import MotionExtractor+from src.models.pose_guider import PoseGuider+import gc+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.+check_min_version("0.10.0.dev0")+from torchvision.utils import save_image+from src.stylegan2.discriminator import Discriminator as StyleGAN2Discriminator++logger = get_logger(__name__, log_level="INFO")++class Net(nn.Module):+ def __init__(+ self,+ denoising_unet: UNet3DConditionModel,+ reference_control_reader,+ ):+ super().__init__()+ self.denoising_unet = denoising_unet+ self.reference_control_reader = reference_control_reader++ def forward(+ self,+ noisy_latents,+ timesteps,+ clip_image_embeds,+ motion,+ pose,+ ):+ motion_emb = motion++ clip_image_embeds = [clip_image_embeds, motion_emb]++ model_pred = self.denoising_unet(+ noisy_latents,+ timesteps,+ encoder_hidden_states=clip_image_embeds,+ pose_cond_fea=pose,+ ).sample++ return model_pred++def get_x0_from_eps(ddim, sample: torch.FloatTensor, model_output: torch.FloatTensor, timesteps: torch.IntTensor):+ video_length = sample.shape[2]+ sample = rearrange(sample, 'b c f h w -> (b f) c h w')+ model_output = rearrange(model_output, 'b c f h w -> (b f) c h w')+ alpha_prod = ddim.alphas_cumprod.cuda()+ alpha_prod_t = alpha_prod[timesteps]+ while len(alpha_prod_t.shape) < len(sample.shape):+ alpha_prod_t = alpha_prod_t.unsqueeze(-1)++ pred_original_sample = torch.sqrt(1./alpha_prod_t) * sample - torch.sqrt(1. / alpha_prod_t - 1) * model_output++ pred_original_sample = rearrange(pred_original_sample, '(b f) c h w -> b c f h w', f=video_length)++ return pred_original_sample++def crop_face(image_pil, face_mesh):+ image = np.array(image_pil)+ h, w = image.shape[:2]+ results = face_mesh.process(image)+ face_landmarks = results.multi_face_landmarks[0]+ coords = [(int(l.x * w), int(l.y * h)) for l in face_landmarks.landmark]+ xs, ys = zip(*coords)+ x1, y1 = min(xs), min(ys)+ x2, y2 = max(xs), max(ys)+ face_box = (x1, y1, x2, y2)++ left, top, right, bot = scale_bb(face_box, scale=1.1, size=image.shape[:2])++ face_patch = image[int(top) : int(bot), int(left) : int(right)]++ return face_box, face_patch++def decode_latents(vae: AutoencoderKL, latents, decode_chunk_size=1):+ latents = latents.to(vae.dtype)+ video_length = latents.shape[2]+ latents = 1 / 0.18215 * latents+ latents = rearrange(latents, "b c f h w -> (b f) c h w")+ # video = self.vae.decode(latents).sample+ video = []+ for frame_idx in range(0, latents.shape[0], decode_chunk_size):+ video.append(vae.decode(latents[frame_idx : frame_idx + decode_chunk_size]).sample)+ video = torch.cat(video)+ video = rearrange(video, "(b f) c h w -> b c f h w", f=video_length)+ # video = (video / 2 + 0.5).clamp(0, 1)+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16+ return video++def log_validation(+ vae,+ image_enc,+ net,+ scheduler,+ accelerator,+ width,+ height,+ face_mesh,+ reference_unet,+ pose_encoder,+ pose_guider,+ motion_encoder,+ timesteps_list,+ clip_length=100,+ generator=None,+):+ logger.info("Running validation... ")++ ori_net = accelerator.unwrap_model(net)+ reference_unet = reference_unet+ denoising_unet = ori_net.denoising_unet+ motion_enc = motion_encoder+ pose_gui = pose_guider+ pose_enc = pose_encoder++ # generator = torch.manual_seed(42)+ generator = torch.Generator(device='cuda').manual_seed(42)+ # cast unet dtype++ pipe = Pose2VideoPipeline(+ vae=vae,+ image_encoder=image_enc,+ reference_unet=reference_unet,+ denoising_unet=denoising_unet,+ motion_encoder=motion_enc,+ pose_encoder=pose_enc,+ pose_guider=pose_gui,+ scheduler=scheduler,+ )+ pipe = pipe.to(accelerator.device)++ ref_image_paths = [+ "path/to/ref_img_1.png",+ "path/to/ref_img_2.png",+ ]+ videos_path = [+ "path/to/ref_vid_1.png",+ "path/to/ref_vid_2.png",+ ]++ pose_transform = transforms.Compose(+ [transforms.Resize((height, width)), transforms.ToTensor()]+ )+ results = []+ for idy, pose_video_path in enumerate(videos_path):+ pose_images = read_frames(pose_video_path)++ boxes_path = pose_video_path.replace("videos", "boxes").replace(".mp4", ".pt")+ boxes = torch.load(boxes_path)++ pose_face = []+ ori_pose_images = []+ for idx_control, pose_image_pil in enumerate(pose_images[:clip_length]):+ face_bbox = boxes[idx_control]["face"]+ ori_pose_images.append(pose_image_pil)+ pose_image = np.array(pose_image_pil)+ left, top, right, bot = scale_bb(face_bbox, scale=1.1, size=pose_image.shape[:2])+ pose_image = pose_image[int(top) : int(bot), int(left) : int(right)]+ pose_image_pil = Image.fromarray(pose_image).convert("RGB")+ pose_face.append(pose_image_pil)++ pose_tensor_list = []+ ori_pose_tensor_list = []++ for idx_control, pose_image_pil in enumerate(ori_pose_images):+ pose_tensor_list.append(pose_transform(pose_face[idx_control]))+ ori_pose_tensor_list.append(pose_transform(pose_image_pil))+ pose_tensor = torch.stack(pose_tensor_list, dim=0) # (f, c, h, w)+ pose_tensor = pose_tensor.transpose(0, 1).unsqueeze(0)++ ori_pose_tensor = torch.stack(ori_pose_tensor_list, dim=0) # (f, c, h, w)+ ori_pose_tensor = ori_pose_tensor.transpose(0, 1).unsqueeze(0)++ for idx, ref_image_path in enumerate(ref_image_paths):+ ref_image_pil = Image.open(ref_image_path).convert("RGB")+ ref_bbox, ref_patch = crop_face(ref_image_pil, face_mesh)+ ref_face = Image.fromarray(ref_patch)++ ref_tensor = pose_transform(ref_image_pil).unsqueeze(0).unsqueeze(2).expand_as(pose_tensor)++ ref_name = f'{idx}'+ pose_name = f'{idy}'++ with torch.cuda.amp.autocast():+ gen_video = pipe(+ ori_pose_images,+ ref_image_pil,+ pose_face,+ ref_face,+ width,+ height,+ clip_length,+ 4,+ 1.0,+ timesteps_list=timesteps_list,+ generator=generator,+ temporal_window_size = 4,+ temporal_adaptive_step = 4,+ temporal_kv_cache=False,+ ).videos++ # Concat it with pose tensor+ video = torch.cat([ref_tensor, pose_tensor, ori_pose_tensor, gen_video], dim=0)++ results.append({"name": f"{ref_name}_{pose_name}", "vid": video})++ del pipe+ torch.cuda.empty_cache()+ gc.collect()+ return results+++def main(cfg):+ kwargs = DistributedDataParallelKwargs(find_unused_parameters=True, static_graph=True)+ accelerator = Accelerator(+ gradient_accumulation_steps=cfg.solver.gradient_accumulation_steps,+ mixed_precision=cfg.solver.mixed_precision,+ log_with="mlflow",+ project_dir="./mlruns",+ kwargs_handlers=[kwargs],+ )++ # Make one log on every process with the configuration for debugging.+ logging.basicConfig(+ format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",+ datefmt="%m/%d/%Y %H:%M:%S",+ level=logging.INFO,+ )+ logger.info(accelerator.state, main_process_only=False)+ if accelerator.is_local_main_process:+ transformers.utils.logging.set_verbosity_warning()+ diffusers.utils.logging.set_verbosity_info()+ else:+ transformers.utils.logging.set_verbosity_error()+ diffusers.utils.logging.set_verbosity_error()++ # If passed along, set the training seed now.+ if cfg.seed is not None:+ seed_everything(cfg.seed)++ exp_name = cfg.exp_name+ save_dir = f"{cfg.output_dir}/{exp_name}"+ if accelerator.is_main_process and not os.path.exists(save_dir):+ os.makedirs(save_dir)++ inference_config_path = "./configs/inference/inference_stage3.yaml"+ infer_config = OmegaConf.load(inference_config_path)++ if cfg.weight_dtype == "fp16":+ weight_dtype = torch.float16+ elif cfg.weight_dtype == "fp32":+ weight_dtype = torch.float32+ else:+ raise ValueError(+ f"Do not support weight dtype: {cfg.weight_dtype} during training"+ )++ sched_kwargs = OmegaConf.to_container(+ infer_config.noise_scheduler_kwargs+ )++ val_noise_scheduler = DDIMScheduler(**sched_kwargs)+ train_noise_scheduler = DDIMScheduler(**sched_kwargs)+ vae = AutoencoderKL.from_pretrained(cfg.vae_model_path).to(+ "cuda", dtype=weight_dtype+ )+ net_lpips = lpips.LPIPS(net='vgg').cuda()++ image_enc = CLIPVisionModelWithProjection.from_pretrained(+ cfg.image_encoder_path,+ ).to(dtype=weight_dtype, device="cuda")++ reference_unet = UNet2DConditionModel.from_pretrained(+ cfg.base_model_path,+ subfolder="unet",+ ).to(device="cuda", dtype=weight_dtype)+ denoising_unet = UNet3DConditionModel.from_pretrained_2d(+ cfg.base_model_path,+ '',+ subfolder="unet",+ unet_additional_kwargs=OmegaConf.to_container(+ infer_config.unet_additional_kwargs+ ),+ ).to(device="cuda")++ motion_encoder = MotEncoder().to(device="cuda", dtype=weight_dtype).eval()+ motion_encoder.load_state_dict(+ torch.load(cfg.motion_encoder_path, map_location="cpu"), strict=True+ )++ pose_guider = PoseGuider().to(device="cuda", dtype=weight_dtype)+ pose_guider.load_state_dict(torch.load(cfg.pose_guider_path, map_location='cpu'), strict=True)++ pose_encoder = MotionExtractor(num_kp=21).to(device="cuda").eval()+ pose_encoder.load_state_dict(torch.load(cfg.pose_encoder_path, map_location='cpu'), strict=False)++ denoising_unet.load_state_dict(+ torch.load(cfg.denoising_unet_path, map_location="cpu"),+ strict=False,+ )+ denoising_unet.load_state_dict(+ torch.load(cfg.temporal_module_path, map_location="cpu"),+ strict=False,+ )+ reference_unet.load_state_dict(+ torch.load(cfg.reference_unet_path, map_location="cpu"),+ )++ mp_face_mesh = mp.solutions.face_mesh+ face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=1)++ discriminator = StyleGAN2Discriminator()+ discriminator = discriminator.to(device="cuda")+ discriminator.load_state_dict(+ torch.load(cfg.discriminator_path, map_location="cpu"), strict=True+ )++ # Freeze+ vae.requires_grad_(False)+ image_enc.requires_grad_(False)+ net_lpips.requires_grad_(False)+ pose_encoder.requires_grad_(False)++ pose_guider.requires_grad_(False)+ motion_encoder.requires_grad_(False)+ denoising_unet.requires_grad_(False)+ reference_unet.requires_grad_(False)++ discriminator.requires_grad_(True)+ for name, module in denoising_unet.named_modules():+ if "temporal_modules" in name:+ for params in module.parameters():+ params.requires_grad = True++ reference_control_writer = ReferenceAttentionControl(+ reference_unet,+ do_classifier_free_guidance=False,+ mode="write",+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ denoising_unet,+ do_classifier_free_guidance=False,+ mode="read",+ fusion_blocks="full",+ )++ net = Net(+ denoising_unet,+ reference_control_reader,+ )++ if cfg.solver.enable_xformers_memory_efficient_attention:+ if is_xformers_available():+ reference_unet.enable_xformers_memory_efficient_attention()+ denoising_unet.enable_xformers_memory_efficient_attention()+ else:+ raise ValueError(+ "xformers is not available. Make sure it is installed correctly"+ )++ if cfg.solver.gradient_checkpointing:+ reference_unet.enable_gradient_checkpointing()+ denoising_unet.enable_gradient_checkpointing()++ # Ensure that no out-of-memory (OOM) error occurs+ vae.decoder.gradient_checkpointing = True+ vae.decoder.training = True++ if cfg.solver.scale_lr:+ learning_rate = (+ cfg.solver.learning_rate+ * cfg.solver.gradient_accumulation_steps+ * cfg.data.train_bs+ * accelerator.num_processes+ )+ else:+ learning_rate = cfg.solver.learning_rate++ # Initialize the optimizer+ if cfg.solver.use_8bit_adam:+ try:+ import bitsandbytes as bnb+ except ImportError:+ raise ImportError(+ "Please install bitsandbytes to use 8-bit Adam. You can do so by running `pip install bitsandbytes`"+ )++ optimizer_cls = bnb.optim.AdamW8bit+ else:+ optimizer_cls = torch.optim.AdamW++ trainable_params = list(filter(lambda p: p.requires_grad, net.parameters()))++ optimizer = optimizer_cls(+ trainable_params,+ lr=learning_rate,+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),+ weight_decay=cfg.solver.adam_weight_decay,+ eps=cfg.solver.adam_epsilon,+ )++ dis_optimizer = optimizer_cls(+ discriminator.parameters(),+ lr=learning_rate * 2,+ betas=(cfg.solver.adam_beta1, cfg.solver.adam_beta2),+ weight_decay=cfg.solver.adam_weight_decay,+ eps=cfg.solver.adam_epsilon,+ )++ # Scheduler+ lr_scheduler = get_scheduler(+ cfg.solver.lr_scheduler,+ optimizer=optimizer,+ num_warmup_steps=cfg.solver.lr_warmup_steps+ * cfg.solver.gradient_accumulation_steps,+ num_training_steps=cfg.solver.max_train_steps+ * cfg.solver.gradient_accumulation_steps,+ )++ train_dataset = PortraitVideoDataset(+ img_size=(cfg.data.train_width, cfg.data.train_height),+ img_scale=(0.9, 1.0),+ data_meta_paths=cfg.data.meta_paths,+ n_sample_frames=cfg.data.n_sample_frames,+ sample_rate=cfg.data.sample_rate,+ )+ train_dataloader = torch.utils.data.DataLoader(+ train_dataset, batch_size=cfg.data.train_bs, shuffle=True, num_workers=8, drop_last=True+ )++ # Prepare everything with our `accelerator`.+ (+ net,+ optimizer,+ discriminator,+ dis_optimizer,+ train_dataloader,+ lr_scheduler,+ ) = accelerator.prepare(+ net,+ optimizer,+ discriminator,+ dis_optimizer,+ train_dataloader,+ lr_scheduler,+ )++ # We need to recalculate our total training steps as the size of the training dataloader may have changed.+ num_update_steps_per_epoch = math.ceil(+ len(train_dataloader) / cfg.solver.gradient_accumulation_steps+ )+ # Afterwards we recalculate our number of training epochs+ num_train_epochs = math.ceil(+ cfg.solver.max_train_steps / num_update_steps_per_epoch+ )++ # We need to initialize the trackers we use, and also store our configuration.+ # The trackers initializes automatically on the main process.+ if accelerator.is_main_process:+ run_time = datetime.now().strftime("%Y%m%d-%H%M")+ accelerator.init_trackers(+ cfg.exp_name,+ init_kwargs={"mlflow": {"run_name": run_time}},+ )+ # dump config file+ mlflow.log_dict(OmegaConf.to_container(cfg), "config.yaml")++ # Train!+ total_batch_size = (+ cfg.data.train_bs+ * accelerator.num_processes+ * cfg.solver.gradient_accumulation_steps+ )++ logger.info("***** Running training *****")+ logger.info(f" Num examples = {len(train_dataset)}")+ logger.info(f" Num Epochs = {num_train_epochs}")+ logger.info(f" Instantaneous batch size per device = {cfg.data.train_bs}")+ logger.info(+ f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}"+ )+ logger.info(+ f" Gradient Accumulation steps = {cfg.solver.gradient_accumulation_steps}"+ )+ logger.info(f" Total optimization steps = {cfg.solver.max_train_steps}")+ global_step = 0+ first_epoch = 0++ # Potentially load in the weights and states from a previous save+ if cfg.resume_from_checkpoint:+ if cfg.resume_from_checkpoint != "latest":+ resume_dir = cfg.resume_from_checkpoint+ else:+ resume_dir = save_dir+ # Get the most recent checkpoint+ dirs = os.listdir(resume_dir)+ dirs = [d for d in dirs if d.startswith("checkpoint")]+ dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))+ path = dirs[-1]+ accelerator.load_state(os.path.join(resume_dir, path))+ accelerator.print(f"Resuming from checkpoint {path}")+ global_step = int(path.split("-")[1])++ first_epoch = global_step // num_update_steps_per_epoch++ # Only show the progress bar once on each machine.+ progress_bar = tqdm(+ range(global_step, cfg.solver.max_train_steps),+ disable=not accelerator.is_local_main_process,+ )+ progress_bar.set_description("Steps")++ temporal_window_size = cfg.temporal_window_size+ temporal_adaptive_step = cfg.temporal_adaptive_step++ windows = cfg.data.n_sample_frames // temporal_window_size++ for epoch in range(first_epoch, num_train_epochs):+ train_loss = 0.0+ train_dis_loss = 0.0+ for step, batch in enumerate(train_dataloader):+ # Convert videos to latent space+ pixel_values = batch["img"].to(weight_dtype) # b,c,h,w+ video_length = pixel_values.shape[1]++ tgt_face = batch["tgt_face"]+ tgt_pose = batch["tgt_pose"]++ tgt_face = rearrange(tgt_face, "b f c h w -> (b f) c h w")+ tgt_pose = rearrange(tgt_pose, "b f c h w -> (b f) c h w")++ face_mask = batch['face_mask']+ local_mask = batch['local_mask']++ face_mask = rearrange(face_mask, "b f c h w -> b c f h w")+ local_mask = rearrange(local_mask, "b f c h w -> b c f h w")++ facial_mask_pix = face_mask.repeat(1,3,1,1,1) + local_mask.repeat(1,3,1,1,1)++ pixel_values = rearrange(pixel_values, "b f c h w -> (b f) c h w")+ with torch.no_grad():+ latents = []+ for i in range(0, pixel_values.shape[0], cfg.data.max_batch_frames):+ batch_latents = vae.encode(+ pixel_values[i : i + cfg.data.max_batch_frames]+ ).latent_dist.sample()+ latents.append(batch_latents)+ latents = torch.cat(latents, dim=0)+ latents = latents * 0.18215 # (b f) c h w+ latents = rearrange(latents, "(b f) c h w -> b c f h w", f=video_length)++ pixel_values = rearrange(pixel_values, "(b f) c h w -> b c f h w", f=video_length)+ noise = torch.randn_like(latents)++ bsz = latents.shape[0] # (b f)+ timesteps_list = cfg.timesteps_list+ timesteps = torch.tensor(timesteps_list[::-1], device=latents.device)+ timesteps = timesteps.repeat_interleave(temporal_window_size, dim=0).long()++ uncond_fwd = random.random() < cfg.uncond_ratio+ clip_image_list = []+ ref_image_list = []+ for batch_idx, (ref_img, clip_img) in enumerate(+ zip(+ batch["ref_img"],+ batch["clip_image"],+ )+ ):+ clip_image_list.append(clip_img)+ ref_image_list.append(ref_img)++ with torch.no_grad():+ ref_img = torch.stack(ref_image_list, dim=0).to(+ dtype=vae.dtype, device=vae.device+ )+ ref_image_latents = vae.encode(+ ref_img+ ).latent_dist.sample() # (bs, d, 64, 64)+ ref_image_latents = ref_image_latents * 0.18215++ clip_img = torch.stack(clip_image_list, dim=0).to(+ dtype=image_enc.dtype, device=image_enc.device+ )+ clip_image_embeds = image_enc(+ clip_img.to("cuda", dtype=weight_dtype)+ ).image_embeds+ image_prompt_embeds = clip_image_embeds.unsqueeze(1) # (bs, 1, d)++ init_frames = (temporal_adaptive_step - 1) * temporal_window_size+ # add noise+ noisy_latents = train_noise_scheduler.add_noise(+ latents[:,:,:init_frames], noise[:,:,:init_frames], timesteps[:init_frames]+ ) # (b f) c h w++ noisy_latents_all = torch.cat([noisy_latents, noise[:,:,init_frames:]], dim=2)++ mot_flag = random.random() < 0.1+ pose_flag = random.random() < 0.1+ reference_control_reader.clear()+ reference_control_writer.clear()++ with torch.no_grad():+ keypoints = pose_encoder(tgt_pose)+ tgt_pose = draw_keypoints(keypoints).to('cuda')+ tgt_pose = rearrange(tgt_pose, '(b f) c h w -> b c f h w', b=bsz)+ pose_emb = pose_guider(tgt_pose.to(pose_guider.dtype))+ motion_emb = motion_encoder(tgt_face)++ if mot_flag:+ motion_emb = torch.zeros_like(motion_emb)+ if pose_flag:+ pose_emb = torch.zeros_like(pose_emb)++ # reference feature+ if not uncond_fwd:+ reference_unet(+ ref_image_latents,+ torch.zeros_like(timesteps[:bsz]),+ encoder_hidden_states=image_prompt_embeds,+ return_dict=False,+ )+ reference_control_reader.update(reference_control_writer, drop_ratio=0.3)++ for i in range(windows - temporal_adaptive_step + 1):+ l = i * temporal_window_size+ r = (i + temporal_adaptive_step) * temporal_window_size++ noisy_latents = noisy_latents_all[:,:,l:r,:,:].detach()+ motion = motion_emb[l:r]+ pose = pose_emb[:,:,l:r]++ if i % (temporal_adaptive_step - 1) != 0:+ with torch.no_grad():+ model_pred = net(+ noisy_latents,+ timesteps,+ image_prompt_embeds,+ motion,+ pose,+ )++ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)+ noise = torch.randn_like(model_pred[:,:,temporal_window_size:])+ noisy_latents = train_noise_scheduler.add_noise(+ latents_pred[:,:,temporal_window_size:], noise, timesteps[:-temporal_window_size]+ )+ mid_latents = torch.cat([latents_pred[:,:,:temporal_window_size], noisy_latents], dim=2)+ noisy_latents_all[:,:,l:r,:,:] = mid_latents.detach()+ else:+ with accelerator.accumulate(net):+ model_pred = net(+ noisy_latents,+ timesteps,+ image_prompt_embeds,+ motion,+ pose,+ )++ discriminator.requires_grad_(False)++ latents_pred = get_x0_from_eps(train_noise_scheduler, noisy_latents, model_pred, timesteps)+ image_pred = decode_latents(vae, latents_pred)++ pixel_values_tgt = pixel_values[:, :, l:r,:,:]++ if epoch == 0 and step == 0:+ if accelerator.is_main_process:+ save_image((ref_img[:1] + 1) / 2, "image_ref.png")+ save_image((pixel_values_tgt[:,:,0] + 1) / 2, "image_tgt.png")+ save_image((image_pred[:,:,0] + 1) / 2, "image_pred.png")+ facial_mask = facial_mask_pix[:, :, l:r, :, :]+ loss = F.mse_loss(image_pred.float(), pixel_values_tgt.float(), reduction="mean")+ loss += F.mse_loss(image_pred.float() * facial_mask, pixel_values_tgt.float() * facial_mask, reduction="mean")+ image_pred = rearrange(image_pred, 'b c f h w -> (b f) c h w')+ pixel_values_tgt = rearrange(pixel_values_tgt, 'b c f h w -> (b f) c h w')+ loss_lpips = net_lpips(image_pred.float(), pixel_values_tgt.float()).mean() # 输入应该是[-1,1]++ adv_loss = discriminator(image_pred.float(), pixel_values_tgt.float(), timesteps)+ loss = loss + loss_lpips * 2.0 + adv_loss * 0.05++ noise = torch.randn_like(model_pred[:,:,temporal_window_size:])+ noisy_latents = train_noise_scheduler.add_noise(+ latents_pred[:,:,temporal_window_size:], noise, timesteps[:-temporal_window_size]+ )+ mid_latents = torch.cat([latents_pred[:,:,:temporal_window_size], noisy_latents], dim=2)+ noisy_latents_all[:,:,l:r,:,:] = mid_latents++ # Gather the losses across all processes for logging (if we use distributed training).+ avg_loss = accelerator.gather(loss.repeat(cfg.data.train_bs)).mean()+ train_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps++ # Backpropagate+ accelerator.backward(loss)+ if accelerator.sync_gradients:+ accelerator.clip_grad_norm_(+ trainable_params,+ cfg.solver.max_grad_norm,+ )+ optimizer.step()+ lr_scheduler.step()+ optimizer.zero_grad()++ if global_step > 50: # This value may not be optimal+ discriminator.requires_grad_(True)+ dis_loss = discriminator(image_pred.float().detach(), pixel_values_tgt.float(), timesteps, gen=False)+ avg_loss = accelerator.gather(dis_loss.repeat(cfg.data.train_bs)).mean()+ train_dis_loss += avg_loss.item() / cfg.solver.gradient_accumulation_steps++ accelerator.backward(dis_loss)+ if accelerator.sync_gradients:+ accelerator.clip_grad_norm_(+ discriminator.parameters(),+ cfg.solver.max_grad_norm,+ )+ dis_optimizer.step()+ dis_optimizer.zero_grad(set_to_none=True)+ else:+ dis_loss = torch.tensor(0.0)++ if accelerator.sync_gradients:+ reference_control_reader.clear()+ reference_control_writer.clear()+ progress_bar.update(1)+ global_step += 1+ accelerator.log({"train_loss": train_loss, "train_dis_loss": train_dis_loss}, step=global_step)+ train_loss = 0.0+ train_dis_loss = 0.0+ if global_step % cfg.checkpointing_steps == 0:+ if accelerator.is_main_process:+ save_path = os.path.join(save_dir, f"checkpoint-{global_step}")+ delete_additional_ckpt(save_dir, 1)+ accelerator.save_state(save_path)++ if global_step % cfg.val.validation_steps == 0 or global_step == 1:+ if accelerator.is_main_process:+ generator = torch.Generator(device=accelerator.device)+ generator.manual_seed(cfg.seed)++ sample_dicts = log_validation(+ vae=vae,+ image_enc=image_enc,+ net=net,+ scheduler=val_noise_scheduler,+ accelerator=accelerator,+ width=cfg.data.train_width,+ height=cfg.data.train_height,+ face_mesh=face_mesh,+ reference_unet=reference_unet,+ generator=generator,+ pose_encoder=pose_encoder,+ pose_guider=pose_guider,+ motion_encoder=motion_encoder,+ timesteps_list=timesteps_list,+ )++ reference_control_writer = ReferenceAttentionControl(+ reference_unet,+ do_classifier_free_guidance=False,+ mode="write",+ fusion_blocks="full",+ )+ reference_control_reader = ReferenceAttentionControl(+ denoising_unet,+ do_classifier_free_guidance=False,+ mode="read",+ fusion_blocks="full",+ )++ for sample_id, sample_dict in enumerate(sample_dicts):+ sample_name = sample_dict["name"]+ vid = sample_dict["vid"]+ with TemporaryDirectory() as temp_dir:+ out_file = f"{temp_dir}/{global_step:06d}-{sample_name}.mp4"+ save_videos_grid(vid, out_file, n_rows=4, fps=25)+ mlflow.log_artifact(out_file)++ logs = {+ "loss": loss.detach().item(),+ "dis": dis_loss.detach().item(),+ "adv": adv_loss.detach().item(),+ "lr": lr_scheduler.get_last_lr()[0],+ }+ progress_bar.set_postfix(**logs)++ # save model after each epoch+ if global_step % cfg.checkpointing_steps == 0 and accelerator.is_main_process:+ unwrap_net = accelerator.unwrap_model(net)+ unwrap_discriminator = accelerator.unwrap_model(discriminator)+ save_checkpoint(+ unwrap_net.denoising_unet,+ save_dir,+ "temporal_module",+ global_step,+ total_limit=2,+ )+ save_checkpoint_dis(+ unwrap_discriminator,+ save_dir,+ "discriminator",+ global_step,+ total_limit=2,+ )++ if global_step >= cfg.solver.max_train_steps:+ break++ unwrap_net = accelerator.unwrap_model(net)+ unwrap_discriminator = accelerator.unwrap_model(discriminator)+ save_checkpoint(+ unwrap_net.denoising_unet,+ save_dir,+ "temporal_module",+ global_step,+ total_limit=2,+ )+ save_checkpoint_dis(+ unwrap_discriminator,+ save_dir,+ "discriminator",+ global_step,+ total_limit=2,+ )++ # Create the pipeline using the trained modules and save it.+ accelerator.wait_for_everyone()+ accelerator.end_training()+++def save_checkpoint(model, save_dir, prefix, ckpt_num, total_limit=None):+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")++ if total_limit is not None:+ checkpoints = os.listdir(save_dir)+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]+ checkpoints = sorted(+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])+ )++ if len(checkpoints) >= total_limit:+ num_to_remove = len(checkpoints) - total_limit + 1+ removing_checkpoints = checkpoints[0:num_to_remove]+ logger.info(+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"+ )+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")++ for removing_checkpoint in removing_checkpoints:+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)+ os.remove(removing_checkpoint)++ mm_state_dict = OrderedDict()+ state_dict = model.state_dict()+ for key in state_dict:+ if "temporal_modules" in key:+ mm_state_dict[key] = state_dict[key]++ torch.save(mm_state_dict, save_path)++def save_checkpoint_dis(model, save_dir, prefix, ckpt_num, total_limit=None):+ save_path = osp.join(save_dir, f"{prefix}-{ckpt_num}.pth")++ if total_limit is not None:+ checkpoints = os.listdir(save_dir)+ checkpoints = [d for d in checkpoints if d.startswith(prefix)]+ checkpoints = sorted(+ checkpoints, key=lambda x: int(x.split("-")[1].split(".")[0])+ )++ if len(checkpoints) >= total_limit:+ num_to_remove = len(checkpoints) - total_limit + 1+ removing_checkpoints = checkpoints[0:num_to_remove]+ logger.info(+ f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"+ )+ logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")++ for removing_checkpoint in removing_checkpoints:+ removing_checkpoint = os.path.join(save_dir, removing_checkpoint)+ os.remove(removing_checkpoint)++ state_dict = model.state_dict()+ torch.save(state_dict, save_path)+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--config", type=str, default="./configs/train/personalive_stage3.yaml")+ args = parser.parse_args()++ if args.config[-5:] == ".yaml":+ config = OmegaConf.load(args.config)+ elif args.config[-3:] == ".py":+ config = import_filename(args.config).cfg+ else:+ raise ValueError("Do not support this format config file")+ main(config)