Python
Added Preview mode and support Sky Reels v2 Diffusion Forcing
deepbeepmeep7c447ea36cc3
diff --git a/README.md b/README.mdindex 6207f6a..9523cd2 100644--- a/README.md+++ b/README.md@@ -10,6 +10,7 @@## 🔥 Latest News!!+* April 25 2025: 👋 Wan 2.1GP v4.3: Added preview mode and support for Sky Reels v2 Diffusion Forcing for high quality "infinite length videos" (see Windo siding section below)* April 18 2025: 👋 Wan 2.1GP v4.2: FLF2V model support, official support from Wan for image2video start and end frames specialized for 720p.* April 17 2025: 👋 Wan 2.1GP v4.1: Recam Master model support, view a video from a different angle. The video to process must be at least 81 frames long and you should set at least 15 steps denoising to get good results.* April 13 2025: 👋 Wan 2.1GP v4.0: lots of goodies for you !@@ -302,18 +303,22 @@ There is also a guide that describes the various combination of hints (https://gIt seems you will get better results if you turn on "Skip Layer Guidance" with its default configuration-### VACE Slidig Window-With this mode (that works for the moment only with Vace) you can merge mutiple Videos to form a very long video (up to 1 min). What is this very nice a about this feature is that the resulting video can be driven by the same control video. For instance the first 0-4s of the control video will be used to generate the first window then the next 4-8s of the control video will be used to generate the second window, and so on. So if your control video contains a person walking, your generate video could contain up to one minute of this person walking.+### VACE and Sky Reels v2 Diffusion Forcing Slidig Window+With this mode (that works for the moment only with Vace and Sky Reels v2) you can merge mutiple Videos to form a very long video (up to 1 min).-To turn on sliding window, you need to go in the Advanced Settings Tab *Sliding Window* and set the iteration number to a number greater than 1. This number corresponds to the default number of windows. You can still increase the number during the genreation by clicking the "One More Sample, Please !" button.+When combined with Vace this feature can use the same control video to generate the full Video that results from concatenining the different windows. For instance the first 0-4s of the control video will be used to generate the first window then the next 4-8s of the control video will be used to generate the second window, and so on. So if your control video contains a person walking, your generate video could contain up to one minute of this person walking.-Each window duration will be set by the *Number of frames (16 = 1s)* form field. However the actual number of frames generated by each iteration will be less, because the *overlap frames* and *discard last frames*:-- *overlap frames* : the first frames ofa new window are filled with last frames of the previous window in order to ensure continuity between the two windows-- *discard last frames* : quite often the last frames of a window have a worse quality. You decide here how many ending frames of a new window should be dropped.+When combined with Sky Reels V2, you can extend an existing video indefinetely.-Number of Generated = [Number of iterations] * ([Number of frames] - [Overlap Frames] - [Discard Last Frames]) + [Overlap Frames]+Sliding Windows are turned on by default and are triggered as soon as you try to generate a Video longer than the Window Size. You can go in the Advanced Settings Tab *Sliding Window* to set this Window Size. You can make the Video even longer during the generation process by adding one more Window to generate each time you click "Extend the Video Sample, Please !" button.-Experimental: if your prompt is broken into multiple lines (each line separated by a carriage return), then each line of the prompt will be used for a new window. If there are more windows to generate than prompt lines, the last prompt line will be repeated.+Although the window duration is set by the *Sliding Window Size* form field, the actual number of frames generated by each iteration will be less, because of the *overlap frames* and *discard last frames*:+- *overlap frames* : the first frames of a new window are filled with last frames of the previous window in order to ensure continuity between the two windows+- *discard last frames* : quite often (Vace model Only) the last frames of a window have a worse quality. You can decide here how many ending frames of a new window should be dropped.+s+Number of Generated Frames = [Number of Windows - 1] * ([Window Size] - [Overlap Frames] - [Discard Last Frames]) + [Window Size]++Experimental: if your prompt is broken into multiple lines (each line separated by a carriage return), then each line of the prompt will be used for a new window. If there are more windows to generate than prompt lines, the last prompt line will be repeated.### Command line parameters for Gradio Server--i2v : launch the image to video generator\diff --git a/requirements.txt b/requirements.txtindex c8eb6b2..8dc178c 100644--- a/requirements.txt+++ b/requirements.txt@@ -12,7 +12,7 @@ ftfydashscopeimageio-ffmpeg# flash_attn-gradio>=5.0.0+gradio==5.23.0numpy>=1.23.5,<2einopsmoviepy==1.0.3diff --git a/wan/__init__.py b/wan/__init__.pyindex df36ebe..54004dd 100644--- a/wan/__init__.py+++ b/wan/__init__.py@@ -1,3 +1,4 @@from . import configs, distributed, modulesfrom .image2video import WanI2Vfrom .text2video import WanT2V+from .diffusion_forcing import DTT2V\ No newline at end of filediff --git a/wan/image2video.py b/wan/image2video.pyindex 99e0959..da48e67 100644--- a/wan/image2video.py+++ b/wan/image2video.py@@ -352,7 +352,7 @@ class WanI2V:# self.model.to(self.device)if callback != None:- callback(-1, True)+ callback(-1, None, True)for i, t in enumerate(tqdm(timesteps)):offload.set_step_no_for_lora(self.model, i)@@ -426,7 +426,7 @@ class WanI2V:del timestepif callback is not None:- callback(i, False)+ callback(i, latent, False)x0 = [latent.to(self.device, dtype=self.dtype)]diff --git a/wan/modules/model.py b/wan/modules/model.pyindex 37b8989..5346fac 100644--- a/wan/modules/model.py+++ b/wan/modules/model.py@@ -10,6 +10,7 @@ import numpy as npfrom typing import Union,Optionalfrom mmgp import offloadfrom .attention import pay_attention+from torch.backends.cuda import sdp_kernel__all__ = ['WanModel']@@ -27,6 +28,10 @@ def sinusoidal_embedding_1d(dim, position):return x+def reshape_latent(latent, latent_frames):+ if latent_frames == latent.shape[0]:+ return latent+ return latent.reshape(latent_frames, -1, latent.shape[-1] )def identify_k( b: float, d: int, N: int):@@ -167,7 +172,7 @@ class WanSelfAttention(nn.Module):self.norm_q = WanRMSNorm(dim, eps=eps) if qk_norm else nn.Identity()self.norm_k = WanRMSNorm(dim, eps=eps) if qk_norm else nn.Identity()- def forward(self, xlist, grid_sizes, freqs):+ def forward(self, xlist, grid_sizes, freqs, block_mask = None):r"""Args:x(Tensor): Shape [B, L, num_heads, C / num_heads]@@ -190,12 +195,44 @@ class WanSelfAttention(nn.Module):del xqklist = [q,k]del q,k+q,k = apply_rotary_emb(qklist, freqs, head_first=False)qkv_list = [q,k,v]del q,k,v- x = pay_attention(- qkv_list,- window_size=self.window_size)+ if block_mask == None:+ x = pay_attention(+ qkv_list,+ window_size=self.window_size)+ else:+ with sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):+ x = (+ torch.nn.functional.scaled_dot_product_attention(+ q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), attn_mask=block_mask+ )+ .transpose(1, 2)+ .contiguous()+ )++ # if not self._flag_ar_attention:+ # q = rope_apply(q, grid_sizes, freqs)+ # k = rope_apply(k, grid_sizes, freqs)+ # x = flash_attention(q=q, k=k, v=v, window_size=self.window_size)+ # else:+ # q = rope_apply(q, grid_sizes, freqs)+ # k = rope_apply(k, grid_sizes, freqs)+ # q = q.to(torch.bfloat16)+ # k = k.to(torch.bfloat16)+ # v = v.to(torch.bfloat16)++ # with sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):+ # x = (+ # torch.nn.functional.scaled_dot_product_attention(+ # q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), attn_mask=block_mask+ # )+ # .transpose(1, 2)+ # .contiguous()+ # )+# outputx = x.flatten(2)x = self.o(x)@@ -360,7 +397,8 @@ class WanAttentionBlock(nn.Module):context,hints= None,context_scale=1.0,- cam_emb= None+ cam_emb= None,+ block_mask = None):r"""Args:@@ -381,13 +419,14 @@ class WanAttentionBlock(nn.Module):hint = self.vace(hints, x, **kwargs)else:hint = self.vace(hints, None, **kwargs)-+ latent_frames = e.shape[0]e = (self.modulation + e).chunk(6, dim=1)-# self-attentionx_mod = self.norm1(x)+ x_mod = reshape_latent(x_mod , latent_frames)x_mod *= 1 + e[1]x_mod += e[0]+ x_mod = reshape_latent(x_mod , 1)if cam_emb != None:cam_emb = self.cam_encoder(cam_emb)cam_emb = cam_emb.repeat(1, 2, 1)@@ -397,12 +436,13 @@ class WanAttentionBlock(nn.Module):xlist = [x_mod]del x_mod- y = self.self_attn( xlist, grid_sizes, freqs)+ y = self.self_attn( xlist, grid_sizes, freqs, block_mask)if cam_emb != None:y = self.projector(y)- # x = x + gate_msa * self.projector(self.self_attn(input_x, freqs))+ x, y = reshape_latent(x , latent_frames), reshape_latent(y , latent_frames)x.addcmul_(y, e[2])+ x, y = reshape_latent(x , 1), reshape_latent(y , 1)del yy = self.norm3(x)ylist= [y]@@ -410,8 +450,10 @@ class WanAttentionBlock(nn.Module):x += self.cross_attn(ylist, context)y = self.norm2(x)+ y = reshape_latent(y , latent_frames)y *= 1 + e[4]y += e[3]+ y = reshape_latent(y , 1)ffn = self.ffn[0]gelu = self.ffn[1]@@ -428,7 +470,9 @@ class WanAttentionBlock(nn.Module):del mlp_chunky = y.view(y_shape)+ x, y = reshape_latent(x , latent_frames), reshape_latent(y , latent_frames)x.addcmul_(y, e[5])+ x, y = reshape_latent(x , 1), reshape_latent(y , 1)if hint is not None:if context_scale == 1:@@ -500,10 +544,14 @@ class Head(nn.Module):"""# assert e.dtype == torch.float32dtype = x.dtype++ latent_frames = e.shape[0]e = (self.modulation + e.unsqueeze(1)).chunk(2, dim=1)x = self.norm(x).to(dtype)+ x = reshape_latent(x , latent_frames)x *= (1 + e[1])x += e[0]+ x = reshape_latent(x , 1)x = self.head(x)return x@@ -552,7 +600,8 @@ class WanModel(ModelMixin, ConfigMixin):qk_norm=True,cross_attn_norm=True,eps=1e-6,- recammaster = False+ recammaster = False,+ inject_sample_info = False,):r"""Initialize the diffusion model backbone.@@ -609,6 +658,10 @@ class WanModel(ModelMixin, ConfigMixin):self.qk_norm = qk_normself.cross_attn_norm = cross_attn_normself.eps = eps+ self.num_frame_per_block = 1+ self.flag_causal_attention = False+ self.block_mask = None+ self.inject_sample_info = inject_sample_info# embeddingsself.patch_embedding = nn.Conv3d(@@ -617,6 +670,10 @@ class WanModel(ModelMixin, ConfigMixin):nn.Linear(text_dim, dim), nn.GELU(approximate='tanh'),nn.Linear(dim, dim))+ if inject_sample_info:+ self.fps_embedding = nn.Embedding(2, dim)+ self.fps_projection = nn.Sequential(nn.Linear(dim, dim), nn.SiLU(), nn.Linear(dim, dim * 6))+self.time_embedding = nn.Sequential(nn.Linear(freq_dim, dim), nn.SiLU(), nn.Linear(dim, dim))self.time_projection = nn.Sequential(nn.SiLU(), nn.Linear(dim, dim * 6))@@ -678,12 +735,13 @@ class WanModel(ModelMixin, ConfigMixin):block.projector.bias = nn.Parameter(torch.zeros(dim))- def compute_teacache_threshold(self, start_step, timesteps = None, speed_factor =0):+ def compute_teacache_threshold(self, start_step, timesteps = None, speed_factor =0):rescale_func = np.poly1d(self.coefficients)e_list = []for t in timesteps:t = torch.stack([t])- e_list.append(self.time_embedding( sinusoidal_embedding_1d(self.freq_dim, t)))+ time_emb = self.time_embedding( sinusoidal_embedding_1d(self.freq_dim, t.flatten()).to(self.patch_embedding.weight.dtype) ) # b, dim+ e_list.append(time_emb)best_threshold = 0.01best_diff = 1000@@ -695,16 +753,13 @@ class WanModel(ModelMixin, ConfigMixin):nb_steps = 0diff = 1000for i, t in enumerate(timesteps):- skip = False+ skip = Falseif not (i<=start_step or i== len(timesteps)):- accumulated_rel_l1_distance += rescale_func(((e_list[i]-previous_modulated_input).abs().mean() / previous_modulated_input.abs().mean()).cpu().item())- # self.accumulated_rel_l1_distance_even += rescale_func(((e_list[i]-self.previous_e0_even).abs().mean() / self.previous_e0_even.abs().mean()).cpu().item())-+ accumulated_rel_l1_distance += abs(rescale_func(((e_list[i]-e_list[i-1]).abs().mean() / e_list[i-1].abs().mean()).cpu().item()))if accumulated_rel_l1_distance < threshold:skip = Trueelse:accumulated_rel_l1_distance = 0- previous_modulated_input = e_list[i]if not skip:nb_steps += 1signed_diff = target_nb_steps - nb_steps@@ -739,6 +794,9 @@ class WanModel(ModelMixin, ConfigMixin):slg_layers=None,callback = None,cam_emb: torch.Tensor = None,+ fps = None,+ causal_block_size = 1,+ causal_attention = False,):if self.model_type == 'i2v':@@ -752,26 +810,53 @@ class WanModel(ModelMixin, ConfigMixin):x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)]# embeddings- x = [self.patch_embedding(u.unsqueeze(0)) for u in x]+ x = [self.patch_embedding(u.unsqueeze(0)) for u in x]# grid_sizes = torch.stack(# [torch.tensor(u.shape[2:], dtype=torch.long) for u in x])grid_sizes = [ list(u.shape[2:]) for u in x]embed_sizes = grid_sizes[0]+ if causal_attention : #causal_block_size > 0:+ frame_num = embed_sizes[0]+ height = embed_sizes[1]+ width = embed_sizes[2]+ block_num = frame_num // causal_block_size+ range_tensor = torch.arange(block_num).view(-1, 1)+ range_tensor = range_tensor.repeat(1, causal_block_size).flatten()+ causal_mask = range_tensor.unsqueeze(0) <= range_tensor.unsqueeze(1) # f, f+ causal_mask = causal_mask.view(frame_num, 1, 1, frame_num, 1, 1).to(x[0].device)+ causal_mask = causal_mask.repeat(1, height, width, 1, height, width)+ causal_mask = causal_mask.reshape(frame_num * height * width, frame_num * height * width)+ block_mask = causal_mask.unsqueeze(0).unsqueeze(0)+ del causal_maskoffload.shared_state["embed_sizes"] = embed_sizesoffload.shared_state["step_no"] = current_stepoffload.shared_state["max_steps"] = max_steps-x = [u.flatten(2).transpose(1, 2) for u in x]x = x[0]- # time embeddings+ if t.dim() == 2:+ b, f = t.shape+ _flag_df = True+ else:+ _flag_df = False+e = self.time_embedding(- sinusoidal_embedding_1d(self.freq_dim, t))+ sinusoidal_embedding_1d(self.freq_dim, t.flatten()).to(self.patch_embedding.weight.dtype)+ ) # b, dime0 = self.time_projection(e).unflatten(1, (6, self.dim)).to(e.dtype)+ if self.inject_sample_info:+ fps = torch.tensor(fps, dtype=torch.long, device=device)++ fps_emb = self.fps_embedding(fps).float()+ if _flag_df:+ e0 = e0 + self.fps_projection(fps_emb).unflatten(1, (6, self.dim)).repeat(t.shape[1], 1, 1)+ else:+ e0 = e0 + self.fps_projection(fps_emb).unflatten(1, (6, self.dim))+# contextcontext = self.text_embedding(torch.stack([@@ -833,7 +918,7 @@ class WanModel(ModelMixin, ConfigMixin):self.accumulated_rel_l1_distance = 0else:rescale_func = np.poly1d(self.coefficients)- self.accumulated_rel_l1_distance += rescale_func(((e-self.previous_modulated_input).abs().mean() / self.previous_modulated_input.abs().mean()).cpu().item())+ self.accumulated_rel_l1_distance += abs(rescale_func(((e-self.previous_modulated_input).abs().mean() / self.previous_modulated_input.abs().mean()).cpu().item()))if self.accumulated_rel_l1_distance < self.rel_l1_thresh:should_calc = Falseself.teacache_skipped_steps += 1@@ -858,7 +943,7 @@ class WanModel(ModelMixin, ConfigMixin):for block_idx, block in enumerate(self.blocks):offload.shared_state["layer"] = block_idxif callback != None:- callback(-1, False, True)+ callback(-1, None, False, True)if pipeline._interrupt:if joint_pass:return None, Nonediff --git a/wan/modules/sage2_core.py b/wan/modules/sage2_core.pyindex 2b7da50..e023a28 100644--- a/wan/modules/sage2_core.py+++ b/wan/modules/sage2_core.py@@ -1075,13 +1075,13 @@ def sageattn_qk_int8_pv_fp8_cuda_sm90(q_int8, q_scale, k_int8, k_scale = per_thread_int8_triton(q, k, km, tensor_layout=tensor_layout, BLKQ=64, WARPQ=16, BLKK=128, WARPK=128)q_size = q.size()+ kv_len = k.size(seq_dim)q_device = q.devicedel q,k# pad v to multiple of 128# TODO: modify per_channel_fp8 kernel to handle this- kv_len = k.size(seq_dim)v_pad_len = 128 - (kv_len % 128) if kv_len % 128 != 0 else 0if v_pad_len > 0:if tensor_layout == "HND":diff --git a/wan/text2video.py b/wan/text2video.pyindex 725d51f..45f4149 100644--- a/wan/text2video.py+++ b/wan/text2video.py@@ -49,40 +49,14 @@ class WanT2V:config,checkpoint_dir,rank=0,- t5_fsdp=False,- dit_fsdp=False,- use_usp=False,- t5_cpu=False,model_filename = None,text_encoder_filename = None,quantizeTransformer = False,dtype = torch.bfloat16):- r"""- Initializes the Wan text-to-video generation model components.-- Args:- config (EasyDict):- Object containing model parameters initialized from config.py- checkpoint_dir (`str`):- Path to directory containing model checkpoints- device_id (`int`, *optional*, defaults to 0):- Id of target GPU device- rank (`int`, *optional*, defaults to 0):- Process rank for distributed training- t5_fsdp (`bool`, *optional*, defaults to False):- Enable FSDP sharding for T5 model- dit_fsdp (`bool`, *optional*, defaults to False):- Enable FSDP sharding for DiT model- use_usp (`bool`, *optional*, defaults to False):- Enable distribution strategy of USP.- t5_cpu (`bool`, *optional*, defaults to False):- Whether to place T5 model on CPU. Only works without t5_fsdp.- """self.device = torch.device(f"cuda")self.config = configself.rank = rank- self.t5_cpu = t5_cpuself.dtype = dtypeself.num_train_timesteps = config.num_train_timestepsself.param_dtype = config.param_dtype@@ -419,9 +393,9 @@ class WanT2V:freqs = get_rotary_pos_embed(shape, enable_RIFLEx= False)else:freqs = get_rotary_pos_embed(latents[0].shape[1:], enable_RIFLEx= enable_RIFLEx)- arg_c = {'context': context, 'freqs': freqs, 'pipeline': self}- arg_null = {'context': context_null, 'freqs': freqs, 'pipeline': self}- arg_both = {'context': context, 'context2': context_null, 'freqs': freqs, 'pipeline': self}+ arg_c = {'context': context, 'freqs': freqs, 'pipeline': self, 'callback': callback}+ arg_null = {'context': context_null, 'freqs': freqs, 'pipeline': self, 'callback': callback}+ arg_both = {'context': context, 'context2': context_null, 'freqs': freqs, 'pipeline': self, 'callback': callback}if target_camera != None:recam_dict = {'cam_emb': cam_emb}@@ -438,7 +412,7 @@ class WanT2V:if self.model.enable_teacache:self.model.compute_teacache_threshold(self.model.teacache_start_step, timesteps, self.model.teacache_multiplier)if callback != None:- callback(-1, True)+ callback(-1, None, True)for i, t in enumerate(tqdm(timesteps)):if target_camera != None:latent_model_input = [torch.cat([u,v], dim=1) for u,v in zip(latents,source_latents )]@@ -494,7 +468,7 @@ class WanT2V:del temp_x0if callback is not None:- callback(i, False)+ callback(i, latents[0], False)x0 = latentsdiff --git a/wgp.py b/wgp.pyindex 141cee0..3285d8e 100644--- a/wgp.py+++ b/wgp.py@@ -115,6 +115,14 @@ def pil_to_base64_uri(pil_image, format="png", quality=75):print(f"Error converting PIL to base64: {e}")return None+def is_integer(n):+ try:+ float(n)+ except ValueError:+ return False+ else:+ return float(n).is_integer()+def process_prompt_and_add_tasks(state, model_choice):@@ -172,17 +180,67 @@ def process_prompt_and_add_tasks(state, model_choice):gr.Info("You must use the 14B model to generate videos with a resolution equivalent to 720P")return- sliding_window_repeat = inputs["sliding_window_repeat"]- sliding_window = sliding_window_repeat > 0+ if "diffusion_forcing" in model_filename or "Vace" in model_filename:+ video_length = inputs["video_length"]+ sliding_window_size = inputs["sliding_window_size"]+ if video_length > sliding_window_size:+ gr.Info(f"The Number of Frames to generate ({video_length}) is greater than the Sliding Window Size ({sliding_window_size}) , multiple Windows will be generated")- if "recam" in model_filename:++ if "diffusion_forcing" in model_filename:+ image_start = inputs["image_start"]+ video_source = inputs["video_source"]+ keep_frames_video_source = inputs["keep_frames_video_source"]+ image_prompt_type = inputs["image_prompt_type"]++ if len(keep_frames_video_source) > 0:+ if not is_integer(keep_frames_video_source):+ gr.Info("The number of frames to keep must be an integer")+ return++ if "V" in image_prompt_type:+ if video_source == None or len(video_source) == 0:+ gr.Info("You must provide a Video to continue")+ return+ image_start = None++ if "S" in image_prompt_type:+ if image_start == None :+ gr.Info("You must provide a Start Image")+ return+ if len(image_start) > 1:+ gr.Info("Only one Start Image is supported for the moment")+ return+ if isinstance(image_start[0][0], str) :+ gr.Info("Start Image should be an Image")+ return++ image_start = [ convert_image(tup[0]) for tup in image_start ]+ video_source = None++ if "T" in image_prompt_type:+ image_start = None+ video_source = None++ if len(prompts) > 0:+ prompts = ["\n".join(prompts)]++ for single_prompt in prompts:+ extra_inputs = {+ "prompt" : single_prompt,+ "image_start" : image_start,+ "video_source" : video_source,+ }+ inputs.update(extra_inputs)+ add_video_task(**inputs)+ elif "recam" in model_filename:video_source = inputs["video_source"]if video_source == None:gr.Info("You must provide a Source Video")return- frames = get_resampled_video(video_source, 0, 81)+ frames = get_resampled_video(video_source, 0, 81, 16)if len(frames)<81:- gr.Info("Recammaster source video should be at least 81 frames one the resampling at 16 fps has been done")+ gr.Info("Recammaster source video should be at least 81 frames once the resampling at 16 fps has been done")returnfor single_prompt in prompts:extra_inputs = {@@ -198,14 +256,6 @@ def process_prompt_and_add_tasks(state, model_choice):video_guide = inputs["video_guide"]video_mask = inputs["video_mask"]- if sliding_window:- if inputs["repeat_generation"]!=1:- gr.Info("Only one Video generated per Prompt is supported when Sliding windows is used")- return- if inputs["sliding_window_overlap"]>=inputs["video_length"] :- gr.Info("The number of frames of the Sliding Window Overlap must be less than the Number of Frames to Generate")- return-if "1.3B" in model_filename :resolution_reformated = str(height) + "*" + str(width)if not resolution_reformated in VACE_SIZE_CONFIGS:@@ -214,7 +264,7 @@ def process_prompt_and_add_tasks(state, model_choice):returnif "I" in video_prompt_type:if image_refs == None:- gr.Info("You must provide at one Refererence Image")+ gr.Info("You must provide at least one Refererence Image")returnelse:image_refs = None@@ -231,17 +281,17 @@ def process_prompt_and_add_tasks(state, model_choice):else:video_mask = Noneif "O" in video_prompt_type :- keep_frames= inputs["keep_frames"]+ keep_frames_video_guide= inputs["keep_frames_video_guide"]video_length = inputs["video_length"]- if len(keep_frames) ==0:+ if len(keep_frames_video_guide) ==0:gr.Info(f"Warning : you have asked to reuse all the frames of the control Video in the Alternate Video Ending it. Please make sure the number of frames of the control Video is lower than the total number of frames to generate otherwise it won't make a difference.")# elif keep_frames >= video_length:- # gr.Info(f"The number of frames in the control Video to reuse ({keep_frames}) in Alternate Video Ending can not be bigger than the total number of frames ({video_length}) to generate.")+ # gr.Info(f"The number of frames in the control Video to reuse ({keep_frames_video_guide}) in Alternate Video Ending can not be bigger than the total number of frames ({video_length}) to generate.")# returnelif "V" in video_prompt_type:- keep_frames= inputs["keep_frames"]+ keep_frames_video_guide= inputs["keep_frames_video_guide"]video_length = inputs["video_length"]- _, error = parse_keep_frames(keep_frames, video_length)+ _, error = parse_keep_frames_video_guide(keep_frames_video_guide, video_length)if len(error) > 0:gr.Info(f"Invalid Keep Frames property: {error}")return@@ -254,7 +304,7 @@ def process_prompt_and_add_tasks(state, model_choice):image_refs = resize_and_remove_background(image_refs, width, height, inputs["remove_background_image_ref"] ==1)- if sliding_window and len(prompts) > 0:+ if len(prompts) > 0:prompts = ["\n".join(prompts)]for single_prompt in prompts:@@ -275,15 +325,20 @@ def process_prompt_and_add_tasks(state, model_choice):returnif not "E" in image_prompt_type:image_end = None- if isinstance(image_start, list):- image_start = [ convert_image(tup[0]) for tup in image_start ]- else:- image_start = [convert_image(image_start)]+ if not isinstance(image_start, list):+ image_start = [image_start]+ if not all( not isinstance(img[0], str) for img in image_start) :+ gr.Info("Start Image should be an Image")+ return+ image_start = [ convert_image(tup[0]) for tup in image_start ]+if image_end != None:- if isinstance(image_end , list):- image_end = [ convert_image(tup[0]) for tup in image_end ]- else:- image_end = [convert_image(image_end) ]+ if not isinstance(image_end , list):+ image_end = [image_end]+ if not all( not isinstance(img[0], str) for img in image_end) :+ gr.Info("End Image should be an Image")+ return+ image_end = [ convert_image(tup[0]) for tup in image_end ]if len(image_start) != len(image_end):gr.Info("The number of start and end images should be the same ")return@@ -573,16 +628,29 @@ def save_queue_action(state):finally:zip_buffer.close()-def load_queue_action(filepath, state):+def load_queue_action(filepath, state, evt:gr.EventData):global task_id+gen = get_gen_info(state)original_queue = gen.get("queue", [])+ delete_autoqueue_file = False+ if evt.target == None:++ if original_queue or not Path(AUTOSAVE_FILENAME).is_file():+ return+ print(f"Autoloading queue from {AUTOSAVE_FILENAME}...")+ filename = AUTOSAVE_FILENAME+ delete_autoqueue_file = True+ else:+ if not filepath or not hasattr(filepath, 'name') or not Path(filepath.name).is_file():+ print("[load_queue_action] Warning: No valid file selected or file not found.")+ return update_queue_data(original_queue)+ filename = filepath.name++save_path_base = server_config.get("save_path", "outputs")loaded_cache_dir = os.path.join(save_path_base, "_loaded_queue_cache")- if not filepath or not hasattr(filepath, 'name') or not Path(filepath.name).is_file():- print("[load_queue_action] Warning: No valid file selected or file not found.")- return update_queue_data(original_queue)newly_loaded_queue = []max_id_in_file = 0@@ -590,14 +658,14 @@ def load_queue_action(filepath, state):local_queue_copy_for_global_ref = Nonetry:- print(f"[load_queue_action] Attempting to load queue from: {filepath.name}")+ print(f"[load_queue_action] Attempting to load queue from: {filename}")os.makedirs(loaded_cache_dir, exist_ok=True)print(f"[load_queue_action] Using cache directory: {loaded_cache_dir}")with tempfile.TemporaryDirectory() as tmpdir:- with zipfile.ZipFile(filepath.name, 'r') as zf:+ with zipfile.ZipFile(filename, 'r') as zf:if "queue.json" not in zf.namelist(): raise ValueError("queue.json not found in zip file")- print(f"[load_queue_action] Extracting {filepath.name} to {tmpdir}")+ print(f"[load_queue_action] Extracting {filename} to {tmpdir}")zf.extractall(tmpdir)print(f"[load_queue_action] Extraction complete.")@@ -677,8 +745,8 @@ def load_queue_action(filepath, state):primary_preview_pil_list, secondary_preview_pil_list = get_preview_images(params)- start_b64 = [pil_to_base64_uri(primary_preview_pil_list[0], format="jpeg", quality=70)] if primary_preview_pil_list[0] else None- end_b64 = [pil_to_base64_uri(secondary_preview_pil_list[0], format="jpeg", quality=70)] if secondary_preview_pil_list[0] else None+ start_b64 = [pil_to_base64_uri(primary_preview_pil_list[0], format="jpeg", quality=70)] if primary_preview_pil_list and primary_preview_pil_list[0] else None+ end_b64 = [pil_to_base64_uri(secondary_preview_pil_list[0], format="jpeg", quality=70)] if secondary_preview_pil_list and secondary_preview_pil_list[0] else Nonetop_level_start_image = params.get("image_start") or params.get("image_refs")top_level_end_image = params.get("image_end")@@ -732,6 +800,11 @@ def load_queue_action(filepath, state):print("[load_queue_action] Load failed. Returning DataFrame update for original queue.")return update_queue_data(original_queue)finally:+ if delete_autoqueue_file:+ if os.path.isfile(filename):+ os.remove(filename)+ print(f"Clear Queue: Deleted autosave file '{filename}'.")+if filepath and hasattr(filepath, 'name') and filepath.name and os.path.exists(filepath.name):if tempfile.gettempdir() in os.path.abspath(filepath.name):try:@@ -919,78 +992,6 @@ def autosave_queue():print(f"Error during autosave: {e}")traceback.print_exc()--def autoload_queue(state):- global task_id- try:- gen = get_gen_info(state)- original_queue = gen.get("queue", [])- except AttributeError:- print("[autoload_queue] Error: Initial state is not a dictionary. Cannot autoload.")- return gr.update(visible=False), False, state-- loaded_flag = False- dataframe_update = update_queue_data(original_queue)-- if not original_queue and Path(AUTOSAVE_FILENAME).is_file():- print(f"Autoloading queue from {AUTOSAVE_FILENAME}...")- class MockFile:- def __init__(self, name):- self.name = name- mock_filepath = MockFile(AUTOSAVE_FILENAME)- dataframe_update = load_queue_action(mock_filepath, state)-- gen = get_gen_info(state)- loaded_queue_after_action = gen.get("queue", [])-- if loaded_queue_after_action:- print(f"Autoload successful. Loaded {len(loaded_queue_after_action)} tasks into state.")- loaded_flag = True- else:- print("Autoload attempted but queue in state remains empty (file might be empty or invalid).")- with lock:- gen["queue"] = []- gen["prompts_max"] = 0- update_global_queue_ref([])- dataframe_update = update_queue_data([])-- # need to remove queue otherwise every new tab will be processed it again- try:- if os.path.isfile(AUTOSAVE_FILENAME):- os.remove(AUTOSAVE_FILENAME)- print(f"Clear Queue: Deleted autosave file '{AUTOSAVE_FILENAME}'.")- except OSError as e:- print(f"Clear Queue: Error deleting autosave file '{AUTOSAVE_FILENAME}': {e}")- gr.Warning(f"Could not delete the autosave file '{AUTOSAVE_FILENAME}'. You may need to remove it manually.")-- else:- if original_queue:- print("Autoload skipped: Queue is not empty.")- update_global_queue_ref(original_queue)- dataframe_update = update_queue_data(original_queue)- else:- # print(f"Autoload skipped: {AUTOSAVE_FILENAME} not found.")- update_global_queue_ref([])- dataframe_update = update_queue_data([])--- return dataframe_update, loaded_flag, state--def run_autoload_and_prepare_ui(current_state):- df_update, loaded_flag, modified_state = autoload_queue(current_state)- should_start_processing = loaded_flag- accordion_update = gr.Accordion(open=True) if loaded_flag else gr.update()- return df_update, gr.update(visible=loaded_flag), accordion_update, should_start_processing, modified_state--def start_processing_if_needed(should_start, current_state):- if not isinstance(current_state, dict) or 'gen' not in current_state:- yield "Error: Invalid state received before processing."- return- if should_start:- yield from process_tasks(current_state)- else:- yield None-def finalize_generation_with_state(current_state):if not isinstance(current_state, dict) or 'gen' not in current_state:return gr.update(), gr.update(interactive=True), gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False, value=""), gr.update(), current_state@@ -1040,10 +1041,6 @@ def update_queue_data(queue):update_global_queue_ref(queue)data = get_queue_table(queue)- # if len(data) == 0:- # return gr.HTML(visible=False)- # else:- # return gr.HTML(value=data, visible= True)if len(data) == 0:return gr.DataFrame(visible=False)else:@@ -1365,7 +1362,8 @@ quantizeTransformer = args.quantize_transformercheck_loras = args.check_loras ==1advanced = args.advanced-transformer_choices_t2v=["ckpts/wan2.1_text2video_1.3B_bf16.safetensors", "ckpts/wan2.1_text2video_14B_bf16.safetensors", "ckpts/wan2.1_text2video_14B_quanto_int8.safetensors", "ckpts/wan2.1_Vace_1.3B_preview_bf16.safetensors", "ckpts/wan2.1_recammaster_1.3B_bf16.safetensors"]+transformer_choices_t2v=["ckpts/wan2.1_text2video_1.3B_bf16.safetensors", "ckpts/wan2.1_text2video_14B_bf16.safetensors", "ckpts/wan2.1_text2video_14B_quanto_int8.safetensors", "ckpts/wan2.1_Vace_1.3B_preview_bf16.safetensors",+ "ckpts/wan2.1_recammaster_1.3B_bf16.safetensors", "ckpts/sky_reels2_diffusion_forcing_1.3B_bf16.safetensors", "ckpts/sky_reels2_diffusion_forcing_14B_bf16.safetensors", "ckpts/sky_reels2_diffusion_forcing_14B_quanto_int8.safetensors"]transformer_choices_i2v=["ckpts/wan2.1_image2video_480p_14B_bf16.safetensors", "ckpts/wan2.1_image2video_480p_14B_quanto_int8.safetensors", "ckpts/wan2.1_image2video_720p_14B_bf16.safetensors", "ckpts/wan2.1_image2video_720p_14B_quanto_int8.safetensors", "ckpts/wan2.1_Fun_InP_1.3B_bf16.safetensors", "ckpts/wan2.1_Fun_InP_14B_bf16.safetensors", "ckpts/wan2.1_Fun_InP_14B_quanto_int8.safetensors", "ckpts/wan2.1_FLF2V_720p_14B_bf16.safetensors", "ckpts/wan2.1_FLF2V_720p_14B_quanto_int8.safetensors"]transformer_choices = transformer_choices_t2v + transformer_choices_i2vtext_encoder_choices = ["ckpts/models_t5_umt5-xxl-enc-bf16.safetensors", "ckpts/models_t5_umt5-xxl-enc-quanto_int8.safetensors"]@@ -1404,10 +1402,10 @@ else:server_config = json.loads(text)-model_types = [ "t2v_1.3B", "vace_1.3B", "fun_inp_1.3B", "t2v", "i2v", "i2v_720p", "fun_inp", "recam_1.3B", "flf2v_720p"]+model_types = [ "t2v_1.3B", "vace_1.3B", "fun_inp_1.3B", "t2v", "i2v", "i2v_720p", "fun_inp", "recam_1.3B", "flf2v_720p", "sky_df_1.3B", "sky_df_14B"]model_signatures = {"t2v": "text2video_14B", "t2v_1.3B" : "text2video_1.3B", "fun_inp_1.3B" : "Fun_InP_1.3B", "fun_inp" : "Fun_InP_14B","i2v" : "image2video_480p", "i2v_720p" : "image2video_720p" , "vace_1.3B" : "Vace_1.3B", "recam_1.3B": "recammaster_1.3B",- "flf2v_720p" : "FLF2V_720p" }+ "flf2v_720p" : "FLF2V_720p", "sky_df_1.3B" : "sky_reels2_diffusion_forcing_1.3B", "sky_df_14B" : "sky_reels2_diffusion_forcing_14B" }def get_model_type(model_filename):@@ -1435,6 +1433,9 @@ def get_model_name(model_filename):elif "FLF2V" in model_filename:model_name = "Wan2.1 FLF2V"model_name += " 720p" if "720p" in model_filename else " 480p"+ elif "sky_reels2_diffusion_forcing" in model_filename:+ model_name = "SkyReels2 diffusion forcing"+ model_name += " 14B" if "14B" in model_filename else " 1.3B"else:model_name = "Wan2.1 text2video"model_name += " 14B" if "14B" in model_filename else " 1.3B"@@ -1491,6 +1492,14 @@ def get_default_settings(filename):"slg_start_perc": 10,"slg_end_perc": 90}++ if get_model_type(filename) in ("sky_df_1.3B", "sky_df_14B"):+ ui_defaults.update({+ "guidance_scale": 6.0,+ "flow_shift": 8,+ "sliding_window_discard_last_frames" : 0+ })+with open(defaults_filename, "w", encoding="utf-8") as f:json.dump(ui_defaults, f, indent=4)else:@@ -1669,7 +1678,7 @@ for file_name in to_remove:download_models(transformer_filename, text_encoder_filename)def sanitize_file_name(file_name, rep =""):- return file_name.replace("/",rep).replace("\\",rep).replace(":",rep).replace("|",rep).replace("?",rep).replace("<",rep).replace(">",rep).replace("\"",rep)+ return file_name.replace("/",rep).replace("\\",rep).replace(":",rep).replace("|",rep).replace("?",rep).replace("<",rep).replace(">",rep).replace("\"",rep).replace("\n",rep).replace("\r",rep)def extract_preset(model_filename, lset_name, loras):loras_choices = []@@ -1759,14 +1768,14 @@ def load_t2v_model(model_filename, value, quantizeTransformer = False, dtype = tcfg = WAN_CONFIGS['t2v-14B']# cfg = WAN_CONFIGS['t2v-1.3B']print(f"Loading '{model_filename}' model...")+ if get_model_type(model_filename) in ("sky_df_1.3B", "sky_df_14B"):+ model_factory = wan.DTT2V+ else:+ model_factory = wan.WanT2V- wan_model = wan.WanT2V(+ wan_model = model_factory(config=cfg,checkpoint_dir="ckpts",- rank=0,- t5_fsdp=False,- dit_fsdp=False,- use_usp=False,model_filename=model_filename,text_encoder_filename= text_encoder_filename,quantizeTransformer = quantizeTransformer,@@ -1922,7 +1931,7 @@ def apply_changes( state,"compile" : compile_choice,"profile" : profile_choice,"vae_config" : vae_config_choice,- "metadata_choice": metadata_choice,+ "metadata_type": metadata_choice,"transformer_quantization" : quantization_choice,"boost" : boost_choice,"clear_file_list" : clear_file_list,@@ -1967,7 +1976,7 @@ def apply_changes( state,model_transformer_type = transformer_types[0] if len(transformer_types) > 0 else model_types[0]model_filename = get_model_filename(model_transformer_type, transformer_quantization)state["model_filename"] = model_filename- if all(change in ["attention_mode", "vae_config", "boost", "save_path", "metadata_choice", "clear_file_list"] for change in changes ):+ if all(change in ["attention_mode", "vae_config", "boost", "save_path", "metadata_type", "clear_file_list"] for change in changes ):model_choice = gr.Dropdown()else:reload_needed = True@@ -1995,9 +2004,10 @@ def get_gen_info(state):state["gen"] = cachereturn cache-def build_callback(state, pipe, progress, status, num_inference_steps):- def callback(step_idx, force_refresh, read_state = False):- gen = get_gen_info(state)+def build_callback(state, pipe, send_cmd, status, num_inference_steps):+ gen = get_gen_info(state)+ gen["num_inference_steps"] = num_inference_steps+ def callback(step_idx, latent, force_refresh, read_state = False, override_num_inference_steps = -1):refresh_id = gen.get("refresh", -1)if force_refresh or step_idx >= 0:pass@@ -2008,7 +2018,10 @@ def build_callback(state, pipe, progress, status, num_inference_steps):UI_refresh = state.get("refresh", 0)if UI_refresh >= refresh_id:return-+ if override_num_inference_steps > 0:+ gen["num_inference_steps"] = override_num_inference_steps++ num_inference_steps = gen.get("num_inference_steps", 0)status = gen["progress_status"]state["refresh"] = refresh_idif read_state:@@ -2029,8 +2042,12 @@ def build_callback(state, pipe, progress, status, num_inference_steps):else:progress_args = [0, status_msg]- progress(*progress_args)- gen["progress_args"] = progress_args+ # progress(*progress_args)+ send_cmd("progress", progress_args)+ if latent != None:+ send_cmd("preview", latent.to("cpu", non_blocking=True))++ # gen["progress_args"] = progress_argsreturn callbackdef abort_generation(state):@@ -2042,17 +2059,18 @@ def abort_generation(state):if wan_model != None:wan_model._interrupt= Truemsg = "Processing Request to abort Current Generation"+ gen["status"] = msggr.Info(msg)- return msg, gr.Button(interactive= False)+ return gr.Button(interactive= False)else:- return "", gr.Button(interactive= True)+ return gr.Button(interactive= True)-def refresh_gallery(state, msg):+def refresh_gallery(state): #, msggen = get_gen_info(state)- gen["last_msg"] = msg+ # gen["last_msg"] = msgfile_list = gen.get("file_list", None)choice = gen.get("selected",0)in_progress = "in_progress" in gen@@ -2063,20 +2081,22 @@ def refresh_gallery(state, msg):queue = gen.get("queue", [])abort_interactive = not gen.get("abort", False)if not in_progress or len(queue) == 0:- return gr.Gallery(selected_index=choice, value = file_list), gr.HTML("", visible= False), gr.Button(visible=True), gr.Button(visible=False), gr.Row(visible=False), update_queue_data(queue), gr.Button(interactive= abort_interactive)+ return gr.Gallery(selected_index=choice, value = file_list), gr.HTML("", visible= False), gr.Button(visible=True), gr.Button(visible=False), gr.Row(visible=False), update_queue_data(queue), gr.Button(interactive= abort_interactive), gr.Button(visible= False)else:task = queue[0]start_img_md = ""end_img_md = ""prompt = task["prompt"]params = task["params"]- if "\n" in prompt and params.get("sliding_window_repeat", 0) > 0:+ model_filename = params["model_filename"]+ onemorewindow_visible = "Vace" in model_filename or "diffusion_forcing" in model_filename+ if "\n" in prompt :prompts = prompt.split("\n")- repeat_no= gen.get("repeat_no",1)- if repeat_no > len(prompts):- repeat_no = len(prompts)- repeat_no -= 1- prompts[repeat_no]="<B>" + prompts[repeat_no] + "</B>"+ window_no= gen.get("window_no",1)+ if window_no > len(prompts):+ window_no = len(prompts)+ window_no -= 1+ prompts[window_no]="<B>" + prompts[window_no] + "</B>"prompt = "<BR>".join(prompts)start_img_uri = task.get('start_image_data_base64')@@ -2099,7 +2119,7 @@ def refresh_gallery(state, msg):html += "</TR></TABLE>"html_output = gr.HTML(html, visible= True)- return gr.Gallery(selected_index=choice, value = file_list), html_output, gr.Button(visible=False), gr.Button(visible=True), gr.Row(visible=True), update_queue_data(queue), gr.Button(interactive= abort_interactive)+ return gr.Gallery(selected_index=choice, value = file_list), html_output, gr.Button(visible=False), gr.Button(visible=True), gr.Row(visible=True), update_queue_data(queue), gr.Button(interactive= abort_interactive), gr.Button(visible= onemorewindow_visible)@@ -2153,7 +2173,7 @@ def convert_image(image):image = image.convert('RGB')return cast(Image, ImageOps.exif_transpose(image))-def get_resampled_video(video_in, start_frame, max_frames):+def get_resampled_video(video_in, start_frame, max_frames, target_fps):from wan.utils.utils import resampleimport decord@@ -2162,13 +2182,13 @@ def get_resampled_video(video_in, start_frame, max_frames):fps = reader.get_avg_fps()- frame_nos = resample(fps, len(reader), max_target_frames_count= max_frames, target_fps=16, start_target_frame= start_frame)+ frame_nos = resample(fps, len(reader), max_target_frames_count= max_frames, target_fps=target_fps, start_target_frame= start_frame)frames_list = reader.get_batch(frame_nos)return frames_list-def preprocess_video(process_type, height, width, video_in, max_frames, start_frame=0, fit_canvas = False):+def preprocess_video(process_type, height, width, video_in, max_frames, start_frame=0, fit_canvas = False, target_fps = 16):- frames_list = get_resampled_video(video_in, start_frame, max_frames)+ frames_list = get_resampled_video(video_in, start_frame, max_frames, target_fps)if len(frames_list) == 0:return None@@ -2227,14 +2247,8 @@ def preprocess_video(process_type, height, width, video_in, max_frames, start_frreturn torch.stack(torch_frames)-def parse_keep_frames(keep_frames, video_length):- def is_integer(n):- try:- float(n)- except ValueError:- return False- else:- return float(n).is_integer()++def parse_keep_frames_video_guide(keep_frames, video_length):def absolute(n):if n==0:@@ -2280,7 +2294,7 @@ def parse_keep_frames(keep_frames, video_length):def generate_video(task_id,- progress,+ send_cmd,prompt,negative_prompt,resolution,@@ -2299,14 +2313,15 @@ def generate_video(image_prompt_type,image_start,image_end,+ model_mode,+ video_source,+ keep_frames_video_source,video_prompt_type,image_refs,video_guide,+ keep_frames_video_guide,video_mask,- camera_type,- video_source,- keep_frames,- sliding_window_repeat,+ sliding_window_size,sliding_window_overlap,sliding_window_discard_last_frames,remove_background_image_ref,@@ -2323,7 +2338,6 @@ def generate_video(model_filename):-global wan_model, offloadobj, reload_neededgen = get_gen_info(state)@@ -2345,9 +2359,9 @@ def generate_video(offloadobj.release()offloadobj = Nonegc.collect()- yield f"Loading model {get_model_name(model_filename)}..."+ send_cmd("status", f"Loading model {get_model_name(model_filename)}...")wan_model, offloadobj, trans = load_models(model_filename)- yield f"Model loaded"+ send_cmd("status", "Model loaded")reload_needed= Falseif attention_mode == "auto":@@ -2355,7 +2369,8 @@ def generate_video(elif attention_mode in attention_modes_supported:attn = attention_modeelse:- gr.Info(f"You have selected attention mode '{attention_mode}'. However it is not installed or supported on your system. You should either install it or switch to the default 'sdpa' attention.")+ send_cmd("info", f"You have selected attention mode '{attention_mode}'. However it is not installed or supported on your system. You should either install it or switch to the default 'sdpa' attention.")+ send_cmd("exit")returnwidth, height = resolution.split("x")@@ -2447,19 +2462,15 @@ def generate_video(if image2video:if '480p' in model_filename:- # teacache_thresholds = [0.13, .19, 0.26]trans.coefficients = [-3.02331670e+02, 2.23948934e+02, -5.25463970e+01, 5.87348440e+00, -2.01973289e-01]elif '720p' in model_filename:- teacache_thresholds = [0.18, 0.2 , 0.3]trans.coefficients = [-114.36346466, 65.26524496, -18.82220707, 4.91518089, -0.23412683]else:raise gr.Error("Teacache not supported for this model")else:if '1.3B' in model_filename:- # teacache_thresholds= [0.05, 0.07, 0.08]trans.coefficients = [2.39676752e+03, -1.31110545e+03, 2.01331979e+02, -8.29855975e+00, 1.37887774e-01]elif '14B' in model_filename:- # teacache_thresholds = [0.14, 0.15, 0.2]trans.coefficients = [-5784.54975374, 5449.50911966, -1811.16591783, 256.27178429, -13.02252404]else:raise gr.Error("Teacache not supported for this model")@@ -2467,14 +2478,13 @@ def generate_video(target_camera = Noneif "recam" in model_filename:source_video = preprocess_video("", width=width, height=height,video_in=video_source, max_frames= video_length, start_frame = 0, fit_canvas= True)- target_camera = camera_type+ target_camera = model_modeimport randomif seed == None or seed <0:seed = random.randint(0, 999999999)global save_pathos.makedirs(save_path, exist_ok=True)- video_no = 0abort = Falsegc.collect()torch.cuda.empty_cache()@@ -2483,307 +2493,364 @@ def generate_video(gen["prompt"] = promptrepeat_no = 0extra_generation = 0- start_frame = 0- sliding_window = sliding_window_repeat > 0+ initial_total_windows = 0+ max_frames_to_generate = video_length+ diffusion_forcing = "diffusion_forcing" in model_filename+ vace = "Vace" in model_filename+ if diffusion_forcing or vace:+ reuse_frames = min(sliding_window_size - 4, sliding_window_overlap)+ if diffusion_forcing and source_video != None:+ video_length += sliding_window_overlap+ sliding_window = ("Vace" in model_filename or diffusion_forcing) and video_length > sliding_window_size+if sliding_window:- reuse_frames = sliding_window_overlap- discard_last_frames = sliding_window_discard_last_frames #4- repeat_generation = sliding_window_repeat+ discard_last_frames = sliding_window_discard_last_frames+ left_after_first_window = video_length - sliding_window_size + discard_last_frames+ initial_total_windows= 1 + math.ceil(left_after_first_window / (sliding_window_size - discard_last_frames - reuse_frames))+ video_length = sliding_window_sizeprompts = prompt.split("\n")prompts = [part for part in prompts if len(prompt)>0]+ else:+ initial_total_windows = 1-+ first_window_video_length = video_length+ fps = 24 if diffusion_forcing else 16+gen["sliding_window"] = sliding_window- frames_already_processed = None- pre_video_guide = None-- while True:+ while not abort:extra_generation += gen.get("extra_orders",0)gen["extra_orders"] = 0total_generation = repeat_generation + extra_generationgen["total_generation"] = total_generation- if abort or repeat_no >= total_generation:+ if repeat_no >= total_generation:break-- if "Vace" in model_filename and (repeat_no == 0 or sliding_window):- if sliding_window:- prompt = prompts[repeat_no] if repeat_no < len(prompts) else prompts[-1]-- # video_prompt_type = video_prompt_type +"G"- image_refs_copy = image_refs.copy() if image_refs != None else None # required since prepare_source do inplace modifications- video_guide_copy = video_guide- video_mask_copy = video_mask- if any(process in video_prompt_type for process in ("P", "D", "G")) :- prompts_max = gen["prompts_max"]-- status = get_generation_status(prompt_no, prompts_max, 1, 1, sliding_window)- preprocess_type = None- if "P" in video_prompt_type :- progress_args = [0, status + " - Extracting Open Pose Information"]- preprocess_type = "pose"- elif "D" in video_prompt_type :- progress_args = [0, status + " - Extracting Depth Information"]- preprocess_type = "depth"- elif "G" in video_prompt_type :- progress_args = [0, status + " - Extracting Gray Level Information"]- preprocess_type = "gray"-- if preprocess_type != None :- progress(*progress_args )- gen["progress_args"] = progress_args- video_guide_copy = preprocess_video(preprocess_type, width=width, height=height,video_in=video_guide, max_frames= video_length if repeat_no ==0 else video_length - reuse_frames, start_frame = start_frame)- keep_frames_parsed, error = parse_keep_frames(keep_frames, video_length)- if len(error) > 0:- raise gr.Error(f"invalid keep frames {keep_frames}")- if repeat_no == 0:- image_size = VACE_SIZE_CONFIGS[resolution_reformated] # default frame dimensions until it is set by video_src (if there is any)- src_video, src_mask, src_ref_images = wan_model.prepare_source([video_guide_copy],- [video_mask_copy ],- [image_refs_copy],- video_length, image_size = image_size, device ="cpu",- original_video= "O" in video_prompt_type,- keep_frames=keep_frames_parsed,- start_frame = start_frame,- pre_src_video = [pre_video_guide]- )- if repeat_no == 0 and src_video != None and len(src_video) > 0:- image_size = src_video[0].shape[-2:]-- else:- src_video, src_mask, src_ref_images = None, None, None--repeat_no +=1gen["repeat_no"] = repeat_no- prompts_max = gen["prompts_max"]- status = get_generation_status(prompt_no, prompts_max, repeat_no, total_generation, sliding_window)--- gen["progress_status"] = status- gen["progress_phase"] = (" - Encoding Prompt", -1 )- callback = build_callback(state, trans, progress, status, num_inference_steps)- progress_args = [0, status + " - Encoding Prompt"]- progress(*progress_args )- gen["progress_args"] = progress_args-- try:- start_time = time.time()+ src_video, src_mask, src_ref_images = None, None, None+ prefix_video = None+ prefix_video_frames_count = 0+ frames_already_processed = None+ pre_video_guide = None+ window_no = 0+ extra_windows = 0+ guide_start_frame = 0+ video_length = first_window_video_length+ while not abort:+ if sliding_window:+ prompt = prompts[window_no] if window_no < len(prompts) else prompts[-1]+ extra_windows += gen.get("extra_windows",0)+ if extra_windows > 0:+ video_length = sliding_window_size+ gen["extra_windows"] = 0+ total_windows = initial_total_windows + extra_windows+ gen["total_windows"] = total_windows+ if window_no >= total_windows:+ break+ window_no += 1+ gen["window_no"] = window_no++ if diffusion_forcing:+ if video_source != None and len(video_source) > 0 and window_no == 1:+ keep_frames_video_source= 1000 if len(keep_frames_video_source) ==0 else int(keep_frames_video_source)+ prefix_video = preprocess_video(None, width=width, height=height,video_in=video_source, max_frames= keep_frames_video_source , start_frame = 0, fit_canvas= True, target_fps = fps)+ prefix_video = prefix_video .permute(3, 0, 1, 2)+ prefix_video = prefix_video .float().div_(127.5).sub_(1.) # c, f, h, w+ prefix_video_frames_count = prefix_video.shape[1]+ pre_video_guide = prefix_video[:, -reuse_frames:]++ if vace:+ # video_prompt_type = video_prompt_type +"G"+ image_refs_copy = image_refs.copy() if image_refs != None else None # required since prepare_source do inplace modifications+ video_guide_copy = video_guide+ video_mask_copy = video_mask+ if any(process in video_prompt_type for process in ("P", "D", "G")) :+ prompts_max = gen["prompts_max"]++ status = get_latest_status(state)++ preprocess_type = None+ if "P" in video_prompt_type :+ progress_args = [0, status + " - Extracting Open Pose Information"]+ preprocess_type = "pose"+ elif "D" in video_prompt_type :+ progress_args = [0, status + " - Extracting Depth Information"]+ preprocess_type = "depth"+ elif "G" in video_prompt_type :+ progress_args = [0, status + " - Extracting Gray Level Information"]+ preprocess_type = "gray"++ if preprocess_type != None :+ send_cmd("progress", progress_args)+ video_guide_copy = preprocess_video(preprocess_type, width=width, height=height,video_in=video_guide, max_frames= video_length if window_no == 1 else video_length - reuse_frames, start_frame = guide_start_frame, target_fps = fps)+ keep_frames_parsed, error = parse_keep_frames_video_guide(keep_frames_video_guide, max_frames_to_generate)+ if len(error) > 0:+ raise gr.Error(f"invalid keep frames {keep_frames_video_guide}")+ keep_frames_parsed = keep_frames_parsed[guide_start_frame: guide_start_frame + video_length]+ if window_no == 1:+ image_size = VACE_SIZE_CONFIGS[resolution_reformated] # default frame dimensions until it is set by video_src (if there is any)+ src_video, src_mask, src_ref_images = wan_model.prepare_source([video_guide_copy],+ [video_mask_copy ],+ [image_refs_copy],+ video_length, image_size = image_size, device ="cpu",+ original_video= "O" in video_prompt_type,+ keep_frames=keep_frames_parsed,+ start_frame = guide_start_frame,+ pre_src_video = [pre_video_guide]+ )+ if window_no == 1 and src_video != None and len(src_video) > 0:+ image_size = src_video[0].shape[-2:]+ prompts_max = gen["prompts_max"]+ status = get_latest_status(state)+++ gen["progress_status"] = status+ gen["progress_phase"] = (" - Encoding Prompt", -1 )+ callback = build_callback(state, trans, send_cmd, status, num_inference_steps)+ progress_args = [0, status + " - Encoding Prompt"]+ send_cmd("progress", progress_args)++ samples = torch.empty( (1,2)) #for testing+ # if False:++ try:+ if trans.enable_teacache:+ trans.teacache_counter = 0+ trans.num_steps = num_inference_steps+ trans.teacache_skipped_steps = 0+ trans.previous_residual_uncond = None+ trans.previous_residual_cond = None++ if image2video:+ samples = wan_model.generate(+ prompt,+ image_start,+ image_end if image_end != None else None,+ frame_num=(video_length // 4)* 4 + 1,+ max_area=MAX_AREA_CONFIGS[resolution_reformated],+ shift=flow_shift,+ sampling_steps=num_inference_steps,+ guide_scale=guidance_scale,+ n_prompt=negative_prompt,+ seed=seed,+ offload_model=False,+ callback=callback,+ enable_RIFLEx = enable_RIFLEx,+ VAE_tile_size = VAE_tile_size,+ joint_pass = joint_pass,+ slg_layers = slg_layers,+ slg_start = slg_start_perc/100,+ slg_end = slg_end_perc/100,+ cfg_star_switch = cfg_star_switch,+ cfg_zero_step = cfg_zero_step,+ add_frames_for_end_image = "image2video" in model_filename+ )+ elif diffusion_forcing:+ samples = wan_model.generate(+ prompt = prompt,+ negative_prompt = negative_prompt,+ image = image_start,+ input_video= pre_video_guide,+ height = height,+ width = width,+ seed = seed,+ num_frames = (video_length // 4)* 4 + 1, #377+ num_inference_steps = num_inference_steps,+ shift = flow_shift,+ guidance_scale= guidance_scale,+ callback= callback,+ VAE_tile_size = VAE_tile_size,+ joint_pass = joint_pass,+ addnoise_condition = 20,+ ar_step = model_mode, #5+ causal_block_size = 5,+ causal_attention = True,+ fps = fps,+ )+ else:+ samples = wan_model.generate(+ prompt,+ input_frames = src_video,+ input_ref_images= src_ref_images,+ input_masks = src_mask,+ source_video= source_video,+ target_camera= target_camera,+ frame_num=(video_length // 4)* 4 + 1,+ size=(width, height),+ shift=flow_shift,+ sampling_steps=num_inference_steps,+ guide_scale=guidance_scale,+ n_prompt=negative_prompt,+ seed=seed,+ offload_model=False,+ callback=callback,+ enable_RIFLEx = enable_RIFLEx,+ VAE_tile_size = VAE_tile_size,+ joint_pass = joint_pass,+ slg_layers = slg_layers,+ slg_start = slg_start_perc/100,+ slg_end = slg_end_perc/100,+ cfg_star_switch = cfg_star_switch,+ cfg_zero_step = cfg_zero_step,+ )+ except Exception as e:+ if temp_filename!= None and os.path.isfile(temp_filename):+ os.remove(temp_filename)+ offload.last_offload_obj.unload_all()+ offload.unload_loras_from_model(trans)+ # if compile:+ # cache_size = torch._dynamo.config.cache_size_limit+ # torch.compiler.reset()+ # torch._dynamo.config.cache_size_limit = cache_size++ gc.collect()+ torch.cuda.empty_cache()+ s = str(e)+ keyword_list = {"CUDA out of memory" : "VRAM", "Tried to allocate":"VRAM", "CUDA error: out of memory": "RAM", "CUDA error: too many resources requested": "RAM"}+ crash_type = ""+ for keyword, tp in keyword_list.items():+ if keyword in s:+ crash_type = tp+ break+ state["prompt"] = ""+ if crash_type == "VRAM":+ new_error = "The generation of the video has encountered an error: it is likely that you have unsufficient VRAM and you should therefore reduce the video resolution or its number of frames."+ elif crash_type == "RAM":+ new_error = "The generation of the video has encountered an error: it is likely that you have unsufficient RAM and / or Reserved RAM allocation should be reduced using 'perc_reserved_mem_max' or using a different Profile."+ else:+ new_error = gr.Error(f"The generation of the video has encountered an error, please check your terminal for more information. '{s}'")+ tb = traceback.format_exc().split('\n')[:-1]+ print('\n'.join(tb))+ send_cmd("error", new_error)+ returnif trans.enable_teacache:- trans.teacache_counter = 0- trans.num_steps = num_inference_steps- trans.teacache_skipped_steps = 0+ print(f"Teacache Skipped Steps:{trans.teacache_skipped_steps}/{num_inference_steps}" )trans.previous_residual_uncond = Nonetrans.previous_residual_cond = None- video_no += 1- if image2video:- samples = wan_model.generate(- prompt,- image_start,- image_end if image_end != None else None,- frame_num=(video_length // 4)* 4 + 1,- max_area=MAX_AREA_CONFIGS[resolution_reformated],- shift=flow_shift,- sampling_steps=num_inference_steps,- guide_scale=guidance_scale,- n_prompt=negative_prompt,- seed=seed,- offload_model=False,- callback=callback,- enable_RIFLEx = enable_RIFLEx,- VAE_tile_size = VAE_tile_size,- joint_pass = joint_pass,- slg_layers = slg_layers,- slg_start = slg_start_perc/100,- slg_end = slg_end_perc/100,- cfg_star_switch = cfg_star_switch,- cfg_zero_step = cfg_zero_step,- add_frames_for_end_image = "image2video" in model_filename- )- else:- samples = wan_model.generate(- prompt,- input_frames = src_video,- input_ref_images= src_ref_images,- input_masks = src_mask,- source_video= source_video,- target_camera= target_camera,- frame_num=(video_length // 4)* 4 + 1,- size=(width, height),- shift=flow_shift,- sampling_steps=num_inference_steps,- guide_scale=guidance_scale,- n_prompt=negative_prompt,- seed=seed,- offload_model=False,- callback=callback,- enable_RIFLEx = enable_RIFLEx,- VAE_tile_size = VAE_tile_size,- joint_pass = joint_pass,- slg_layers = slg_layers,- slg_start = slg_start_perc/100,- slg_end = slg_end_perc/100,- cfg_star_switch = cfg_star_switch,- cfg_zero_step = cfg_zero_step,- )- # samples = torch.empty( (1,2)) #for testing- except Exception as e:- if temp_filename!= None and os.path.isfile(temp_filename):- os.remove(temp_filename)+ if samples != None:+ samples = samples.to("cpu")offload.last_offload_obj.unload_all()- offload.unload_loras_from_model(trans)- # if compile:- # cache_size = torch._dynamo.config.cache_size_limit- # torch.compiler.reset()- # torch._dynamo.config.cache_size_limit = cache_size-gc.collect()torch.cuda.empty_cache()- s = str(e)- keyword_list = {"CUDA out of memory" : "VRAM", "Tried to allocate":"VRAM", "CUDA error: out of memory": "RAM", "CUDA error: too many resources requested": "RAM"}- crash_type = ""- for keyword, tp in keyword_list.items():- if keyword in s:- crash_type = tp- break- state["prompt"] = ""- if crash_type == "VRAM":- new_error = "The generation of the video has encountered an error: it is likely that you have unsufficient VRAM and you should therefore reduce the video resolution or its number of frames."- elif crash_type == "RAM":- new_error = "The generation of the video has encountered an error: it is likely that you have unsufficient RAM and / or Reserved RAM allocation should be reduced using 'perc_reserved_mem_max' or using a different Profile."- else:- new_error = gr.Error(f"The generation of the video has encountered an error, please check your terminal for more information. '{s}'")- tb = traceback.format_exc().split('\n')[:-1]- print('\n'.join(tb))- raise gr.Error(new_error, print_exception= False)-- finally:- pass- # with tracker_lock:- # if task_id in progress_tracker:- # del progress_tracker[task_id]-- if trans.enable_teacache:- print(f"Teacache Skipped Steps:{trans.teacache_skipped_steps}/{num_inference_steps}" )- trans.previous_residual_uncond = None- trans.previous_residual_cond = None-- if samples != None:- samples = samples.to("cpu")- offload.last_offload_obj.unload_all()- gc.collect()- torch.cuda.empty_cache()- if samples == None:- end_time = time.time()- abort = True- state["prompt"] = ""- # yield f"Video generation was aborted. Total Generation Time: {end_time-start_time:.1f}s"- else:- sample = samples.cpu()- if sliding_window :- start_frame += video_length- if discard_last_frames > 0:- sample = sample[: , :-discard_last_frames]- start_frame -= discard_last_frames- pre_video_guide = sample[:, -reuse_frames:]- if repeat_no > 1:+ if samples == None:+ abort = True+ state["prompt"] = ""+ else:+ sample = samples.cpu()+ if True: # for testing+ torch.save(sample, "output.pt")+ else:+ sample =torch.load("output.pt")++ if sliding_window :+ guide_start_frame += video_length+ if discard_last_frames > 0:+ sample = sample[: , :-discard_last_frames]+ guide_start_frame -= discard_last_frames+ pre_video_guide = sample[:, -reuse_frames:]+ if prefix_video != None:+ sample = torch.cat([ prefix_video[:, :-reuse_frames], sample], dim = 1)+ prefix_video = None+ if sliding_window and window_no > 1:sample = sample[: , reuse_frames:]- start_frame -= reuse_frames+ guide_start_frame -= reuse_frames- time_flag = datetime.fromtimestamp(time.time()).strftime("%Y-%m-%d-%Hh%Mm%Ss")- if os.name == 'nt':- file_name = f"{time_flag}_seed{seed}_{sanitize_file_name(prompt[:50]).strip()}.mp4"- else:- file_name = f"{time_flag}_seed{seed}_{sanitize_file_name(prompt[:100]).strip()}.mp4"- video_path = os.path.join(save_path, file_name)- # if False: # for testing- # torch.save(sample, "output.pt")- # else:- # sample =torch.load("output.pt")- exp = 0- fps = 16-- if len(temporal_upsampling) > 0 or len(spatial_upsampling) > 0:- progress_args = [(num_inference_steps , num_inference_steps) , status + " - Upsampling" , num_inference_steps]- progress(*progress_args )- gen["progress_args"] = progress_args-- if temporal_upsampling == "rife2":- exp = 1- elif temporal_upsampling == "rife4":- exp = 2-- if exp > 0:- from rife.inference import temporal_interpolation- if sliding_window and repeat_no > 1:- sample = torch.cat([frames_already_processed[:, -2:-1], sample], dim=1)- sample = temporal_interpolation( os.path.join("ckpts", "flownet.pkl"), sample, exp, device=processing_device)- sample = sample[:, 1:]+ time_flag = datetime.fromtimestamp(time.time()).strftime("%Y-%m-%d-%Hh%Mm%Ss")+ if os.name == 'nt':+ file_name = f"{time_flag}_seed{seed}_{sanitize_file_name(prompt[:50]).strip()}.mp4"else:- sample = temporal_interpolation( os.path.join("ckpts", "flownet.pkl"), sample, exp, device=processing_device)+ file_name = f"{time_flag}_seed{seed}_{sanitize_file_name(prompt[:100]).strip()}.mp4"+ video_path = os.path.join(save_path, file_name)+ exp = 0++ if len(temporal_upsampling) > 0 or len(spatial_upsampling) > 0:+ progress_args = [(num_inference_steps , num_inference_steps) , status + " - Upsampling" , num_inference_steps]+ send_cmd("progress", progress_args)++ if temporal_upsampling == "rife2":+ exp = 1+ elif temporal_upsampling == "rife4":+ exp = 2++ if exp > 0:+ from rife.inference import temporal_interpolation+ if sliding_window and window_no > 1:+ sample = torch.cat([frames_already_processed[:, -2:-1], sample], dim=1)+ sample = temporal_interpolation( os.path.join("ckpts", "flownet.pkl"), sample, exp, device=processing_device)+ sample = sample[:, 1:]+ else:+ sample = temporal_interpolation( os.path.join("ckpts", "flownet.pkl"), sample, exp, device=processing_device)- fps = fps * 2**exp+ fps = fps * 2**exp- if len(spatial_upsampling) > 0:- from wan.utils.utils import resize_lanczos # need multithreading or to do lanczos with cuda- if spatial_upsampling == "lanczos1.5":- scale = 1.5- else:- scale = 2- sample = (sample + 1) / 2- h, w = sample.shape[-2:]- h *= scale- w *= scale- h = int(h)- w = int(w)- new_frames =[]- for i in range( sample.shape[1] ):- frame = sample[:, i]- frame = resize_lanczos(frame, h, w)- frame = frame.unsqueeze(1)- new_frames.append(frame)- sample = torch.cat(new_frames, dim=1)- new_frames = None- sample = sample * 2 - 1-- if sliding_window :- if repeat_no == 1:+ if len(spatial_upsampling) > 0:+ from wan.utils.utils import resize_lanczos # need multithreading or to do lanczos with cuda+ if spatial_upsampling == "lanczos1.5":+ scale = 1.5+ else:+ scale = 2+ sample = (sample + 1) / 2+ h, w = sample.shape[-2:]+ h *= scale+ w *= scale+ h = int(h)+ w = int(w)+ new_frames =[]+ for i in range( sample.shape[1] ):+ frame = sample[:, i]+ frame = resize_lanczos(frame, h, w)+ frame = frame.unsqueeze(1)+ new_frames.append(frame)+ sample = torch.cat(new_frames, dim=1)+ new_frames = None+ sample = sample * 2 - 1++ if sliding_window :+ if frames_already_processed == None:+ frames_already_processed = sample+ else:+ sample = torch.cat([frames_already_processed, sample], dim=1)frames_already_processed = sample++ cache_video(+ tensor=sample[None],+ save_file=video_path,+ fps=fps,+ nrow=1,+ normalize=True,+ value_range=(-1, 1))++ inputs = get_function_arguments(generate_video, locals())+ inputs.pop("send_cmd")+ configs = prepare_inputs_dict("metadata", inputs)++ metadata_choice = server_config.get("metadata_type","metadata")+ if metadata_choice == "json":+ with open(video_path.replace('.mp4', '.json'), 'w') as f:+ json.dump(configs, f, indent=4)+ elif metadata_choice == "metadata":+ from mutagen.mp4 import MP4+ file = MP4(video_path)+ file.tags['©cmt'] = [json.dumps(configs)]+ file.save()++ print(f"New video saved to Path: "+video_path)+ file_list.append(video_path)+ state['update_gallery'] = True+ send_cmd("output")+ if sliding_window :+ if max_frames_to_generate > 0 and extra_windows == 0:+ current_length = sample.shape[1]+ if (current_length - prefix_video_frames_count)>= max_frames_to_generate:+ break+ video_length = min(sliding_window_size, ((max_frames_to_generate - (current_length - prefix_video_frames_count) + reuse_frames + discard_last_frames) // 4) * 4 + 1 )else:- sample = torch.cat([frames_already_processed, sample], dim=1)- frames_already_processed = sample-- cache_video(- tensor=sample[None],- save_file=video_path,- fps=fps,- nrow=1,- normalize=True,- value_range=(-1, 1))-- inputs = get_function_arguments(generate_video, locals())- inputs.pop("progress")- configs = prepare_inputs_dict("metadata", inputs)-- metadata_choice = server_config.get("metadata_choice","metadata")- if metadata_choice == "json":- with open(video_path.replace('.mp4', '.json'), 'w') as f:- json.dump(configs, f, indent=4)- elif metadata_choice == "metadata":- from mutagen.mp4 import MP4- file = MP4(video_path)- file.tags['©cmt'] = [json.dumps(configs)]- file.save()-- print(f"New video saved to Path: "+video_path)- file_list.append(video_path)- state['update_gallery'] = True- if not sliding_window:- seed += 1- yield status+ break++ seed += 1if temp_filename!= None and os.path.isfile(temp_filename):os.remove(temp_filename)@@ -2795,12 +2862,73 @@ def prepare_generate_video(state):else:return gr.Button(visible= False), gr.Button(visible= True), gr.Column(visible= True)+def generate_preview(latents):+ import einops++ latent_channels = 16+ latent_dimensions = 3+ latents = latents.unsqueeze(0)+ latent_rgb_factors = [+ [-0.1299, -0.1692, 0.2932],+ [ 0.0671, 0.0406, 0.0442],+ [ 0.3568, 0.2548, 0.1747],+ [ 0.0372, 0.2344, 0.1420],+ [ 0.0313, 0.0189, -0.0328],+ [ 0.0296, -0.0956, -0.0665],+ [-0.3477, -0.4059, -0.2925],+ [ 0.0166, 0.1902, 0.1975],+ [-0.0412, 0.0267, -0.1364],+ [-0.1293, 0.0740, 0.1636],+ [ 0.0680, 0.3019, 0.1128],+ [ 0.0032, 0.0581, 0.0639],+ [-0.1251, 0.0927, 0.1699],+ [ 0.0060, -0.0633, 0.0005],+ [ 0.3477, 0.2275, 0.2950],+ [ 0.1984, 0.0913, 0.1861]+ ]++ # credits for the rgb factors to ComfyUI ?++ latent_rgb_factors_bias = [-0.1835, -0.0868, -0.3360]++ latent_rgb_factors_bias = [0.0259, -0.0192, -0.0761]+ nb_latents = latents.shape[2]+ latents_to_preview = 4+ latents_to_preview = min(nb_latents, latents_to_preview)+ skip_latent = nb_latents / latents_to_preview+ latent_no = 0+ selected_latents = []+ while latent_no < nb_latents:+ selected_latents.append( latents[:, : , int(latent_no): int(latent_no)+1])+ latent_no += skip_latent++ latents = torch.cat(selected_latents, dim = 2)+ weight = torch.tensor(latent_rgb_factors, device=latents.device, dtype=latents.dtype).transpose(0, 1)[:, :, None, None, None]+ bias = torch.tensor(latent_rgb_factors_bias, device=latents.device, dtype=latents.dtype)++ images = torch.nn.functional.conv3d(latents, weight, bias=bias, stride=1, padding=0, dilation=1, groups=1)+ images = images.clamp(0.0, 1.0)+++ images = (images * 255.0).detach().cpu().numpy().clip(0, 255).astype(np.uint8)+ images = einops.rearrange(images, 'b c t h w -> (b h) (t w) c')+ h, w, _ = images.shape+ scale = 200 / h+ images= Image.fromarray(images)+ images = images.resize(( int(w*scale),int(h*scale)), resample=Image.Resampling.BILINEAR)++ return images+++def process_tasks(state):+ from wan.utils.thread_utils import AsyncStream, async_run-def process_tasks(state, progress=gr.Progress()):gen = get_gen_info(state)queue = gen.get("queue", [])+ progress = Noneif len(queue) == 0:+ gen["status_display"] = Falsereturngen = get_gen_info(state)clear_file_list = server_config.get("clear_file_list", 0)@@ -2824,8 +2952,9 @@ def process_tasks(state, progress=gr.Progress()):gen_in_progress = Truegen["in_progress"] = True- yield "Generating Video"-+ gen["preview"] = None+ gen["status"] = "Generating Video"+ yield time.time(), time.time()prompt_no = 0while len(queue) > 0:prompt_no += 1@@ -2833,27 +2962,57 @@ def process_tasks(state, progress=gr.Progress()):task = queue[0]task_id = task["id"]params = task['params']- iterator = iter(generate_video(task_id, progress, **params))- while True:++ com_stream = AsyncStream()+ send_cmd = com_stream.output_queue.push+ def generate_video_error_handler():try:- ok = False- status = next(iterator, "#")- ok = True- if status == "#":- break+ generate_video(task_id, send_cmd, **params)except Exception as e:- _ , exc_value, exc_traceback = sys.exc_info()- raise exc_value.with_traceback(exc_traceback)+ tb = traceback.format_exc().split('\n')[:-1]+ print('\n'.join(tb))+ send_cmd("error",str(e))finally:- if not ok:- queue.clear()- gen["prompts_max"] = 0- gen["prompt"] = ""- yield status+ send_cmd("exit", None)+++ async_run(generate_video_error_handler)++ while True:+ cmd, data = com_stream.output_queue.next()+ if cmd == "exit":+ break+ elif cmd == "info":+ gr.Info(data)+ elif cmd == "error":+ queue.clear()+ gen["prompts_max"] = 0+ gen["prompt"] = ""+ gen["status_display"] = False++ raise gr.Error(data, print_exception= False)+ elif cmd == "status":+ gen["status"] = data+ elif cmd == "output":+ gen["preview"] = None+ yield time.time() , time.time()+ elif cmd == "progress":+ gen["progress_args"] = data+ # progress(*data)+ elif cmd == "preview":+ preview= None if data== None else generate_preview(data)+ gen["preview"] = preview+ yield time.time() , gr.Text()+ else:+ raise Exception(f"unknown command {cmd}")+abort = gen.get("abort", False)if abort:gen["abort"] = False- yield "Video Generation Aborted"+ status = "Video Generation Aborted", "Video Generation Aborted"+ yield gr.Text(), gr.Text()+ gen["status"] = status+queue[:] = [item for item in queue if item['id'] != task['id']]update_global_queue_ref(queue)@@ -2861,25 +3020,29 @@ def process_tasks(state, progress=gr.Progress()):gen["prompt"] = ""end_time = time.time()if abort:- yield f"Video generation was aborted. Total Generation Time: {end_time-start_time:.1f}s"+ status = f"Video generation was aborted. Total Generation Time: {end_time-start_time:.1f}s"else:- yield f"Total Generation Time: {end_time-start_time:.1f}s"+ status = f"Total Generation Time: {end_time-start_time:.1f}s"+ gen["status"] = status+ gen["status_display"] = False-def get_generation_status(prompt_no, prompts_max, repeat_no, repeat_max, sliding_window):- item = "Sliding Window" if sliding_window else "Sample"+def get_generation_status(prompt_no, prompts_max, repeat_no, repeat_max, window_no, total_windows):if prompts_max == 1:if repeat_max == 1:- return "Video"+ status = "Video"else:- return f"{item} {repeat_no}/{repeat_max}"+ status = f"Sample {repeat_no}/{repeat_max}"else:if repeat_max == 1:- return f"Prompt {prompt_no}/{prompts_max}"+ status = f"Prompt {prompt_no}/{prompts_max}"else:- return f"Prompt {prompt_no}/{prompts_max}, {item} {repeat_no}/{repeat_max}"+ status = f"Prompt {prompt_no}/{prompts_max}, Sample {repeat_no}/{repeat_max}"+ if total_windows > 1:+ status += f", Sliding Window {window_no}/{total_windows}"+ return statusrefresh_id = 0@@ -2888,15 +3051,22 @@ def get_new_refresh_id():refresh_id += 1return refresh_id-def update_status(state):+def get_latest_status(state):gen = get_gen_info(state)prompt_no = gen["prompt_no"]prompts_max = gen.get("prompts_max",0)total_generation = gen.get("total_generation", 1)repeat_no = gen["repeat_no"]- sliding_window = gen["sliding_window"]- status = get_generation_status(prompt_no, prompts_max, repeat_no, total_generation, sliding_window)- gen["progress_status"] = status+ total_generation += gen.get("extra_orders", 0)+ total_windows = gen.get("total_windows", 0)+ total_windows += gen.get("extra_windows", 0)+ window_no = gen.get("window_no", 0)+ status = get_generation_status(prompt_no, prompts_max, repeat_no, total_generation, window_no, total_windows)+ return status++def update_status(state):+ gen = get_gen_info(state)+ gen["progress_status"] = get_latest_status(state)gen["refresh"] = get_new_refresh_id()@@ -2908,16 +3078,25 @@ def one_more_sample(state):in_progress = gen.get("in_progress", False)if not in_progress :return state- prompt_no = gen["prompt_no"]- prompts_max = gen.get("prompts_max",0)- total_generation = gen["total_generation"] + extra_orders- repeat_no = gen["repeat_no"]- status = get_generation_status(prompt_no, prompts_max, repeat_no, total_generation, gen.get("sliding_window",False))+ total_generation = gen.get("total_generation", 0) + extra_orders+ gen["progress_status"] = get_latest_status(state)+ gen["refresh"] = get_new_refresh_id()+ gr.Info(f"An extra sample generation is planned for a total of {total_generation} videos for this prompt")+ return state- gen["progress_status"] = status+def one_more_window(state):+ gen = get_gen_info(state)+ extra_windows = gen.get("extra_windows", 0)+ extra_windows += 1+ gen["extra_windows"]= extra_windows+ in_progress = gen.get("in_progress", False)+ if not in_progress :+ return state+ total_windows = gen.get("total_windows", 0) + extra_windows+ gen["progress_status"] = get_latest_status(state)gen["refresh"] = get_new_refresh_id()- gr.Info(f"An extra sample generation is planned for a total of {total_generation} videos for this prompt")+ gr.Info(f"An extra window generation is planned for a total of {total_windows} videos for this sample")return state@@ -3234,11 +3413,21 @@ def prepare_inputs_dict(target, inputs ):inputs.pop("image_prompt_type")- if not "recam" in model_filename:- inputs.pop("camera_type")+ if not "recam" in model_filename or not "diffusion_forcing" in model_filename:+ inputs.pop("model_mode")if not "Vace" in model_filename:- unsaved_params = ["video_prompt_type", "keep_frames", "remove_background_image_ref", "sliding_window_repeat", "sliding_window_overlap", "sliding_window_discard_last_frames"]+ unsaved_params = ["keep_frames_video_guide", "video_prompt_type", "remove_background_image_ref"]+ for k in unsaved_params:+ inputs.pop(k)++ if not "diffusion_forcing" in model_filename:+ unsaved_params = ["keep_frames_video_source"]+ for k in unsaved_params:+ inputs.pop(k)++ if not "Vace" in model_filename or "diffusion_forcing" in model_filename:+ unsaved_params = [ "sliding_window_size", "sliding_window_overlap", "sliding_window_discard_last_frames"]for k in unsaved_params:inputs.pop(k)@@ -3276,14 +3465,15 @@ def save_inputs(image_prompt_type,image_start,image_end,- video_prompt_type,+ model_mode,+ video_source,+ keep_frames_video_source,+ video_prompt_type,image_refs,video_guide,+ keep_frames_video_guide,video_mask,- camera_type,- video_source,- keep_frames,- sliding_window_repeat,+ sliding_window_size,sliding_window_overlap,sliding_window_discard_last_frames,remove_background_image_ref,@@ -3344,10 +3534,7 @@ def download_loras():returndef refresh_image_prompt_type(state, image_prompt_type):- if args.multiple_images:- return gr.Gallery(visible = "S" in image_prompt_type ), gr.Gallery(visible = "E" in image_prompt_type )- else:- return gr.Image(visible = "S" in image_prompt_type ), gr.Image(visible = "E" in image_prompt_type )+ return gr.update(visible = "S" in image_prompt_type ), gr.update(visible = "E" in image_prompt_type ), gr.update(visible = "V" in image_prompt_type) , gr.update(visible = "V" in image_prompt_type )def refresh_video_prompt_type(state, video_prompt_type):return gr.Gallery(visible = "I" in video_prompt_type), gr.Video(visible= "V" in video_prompt_type),gr.Video(visible= "M" in video_prompt_type ), gr.Text(visible= "V" in video_prompt_type) , gr.Checkbox(visible= "I" in video_prompt_type)@@ -3482,8 +3669,22 @@ def refresh_video_prompt_video_guide_trigger(video_prompt_type, video_prompt_typreturn video_prompt_type, video_prompt_type_video_guide, gr.update(visible= "V" in video_prompt_type ), gr.update(visible= "M" in video_prompt_type) , gr.update(visible= "V" in video_prompt_type )--def generate_video_tab(update_form = False, state_dict = None, ui_defaults = None, model_choice = None, header = None):+def refresh_preview(state):+ gen = get_gen_info(state)+ preview = gen.get("preview", None)+ return preview++def init_process_queue_if_any(state):+ gen = get_gen_info(state)+ if bool(gen.get("queue",[])):+ state["validate_success"] = 1+ return gr.Button(visible=False), gr.Button(visible=True), gr.Column(visible=True)+ else:+ return gr.Button(visible=True), gr.Button(visible=False), gr.Column(visible=False)++++def generate_video_tab(update_form = False, state_dict = None, ui_defaults = None, model_choice = None, header = None, main = None):global inputs_names #, advancedif update_form:@@ -3572,45 +3773,69 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonif not update_form:state = gr.State(state_dict)trigger_refresh_input_type = gr.Text(interactive= False, visible= False)- with gr.Column(visible= test_class_i2v(model_filename) ) as image_prompt_column:- image_prompt_type_value= ui_defaults.get("image_prompt_type","S")- image_prompt_type = gr.Radio( [("Use only a Start Image", "S"),("Use both a Start and an End Image", "SE")], value =image_prompt_type_value, label="Location", show_label= False, scale= 3)-- if args.multiple_images:+ diffusion_forcing = "diffusion_forcing" in model_filename+ recammaster = "recam" in model_filename+ vace = "Vace" in model_filename+ with gr.Column(visible= test_class_i2v(model_filename) or diffusion_forcing or recammaster) as image_prompt_column:+ if diffusion_forcing:+ image_prompt_type_value= ui_defaults.get("image_prompt_type","S")+ image_prompt_type = gr.Radio( [("Start Video with Image", "S"),("Continue Video", "V"),("Text Prompt Only", "T")], value =image_prompt_type_value, label="Location", show_label= False, visible= True, scale= 3)+ # image_start = gr.Image(label= "Image as a starting point for a new video", type ="pil",value= ui_defaults.get("image_start", None), visible= "S" in image_prompt_type_value )image_start = gr.Gallery(label="Images as starting points for new videos", type ="pil", #file_types= "image",- columns=[3], rows=[1], object_fit="contain", height="auto", selected_index=0, interactive= True, value= ui_defaults.get("image_start", None), visible= "S" in image_prompt_type_value)+ columns=[3], rows=[1], object_fit="contain", height="auto", selected_index=0, interactive= True, value= ui_defaults.get("image_start", None), visible= "S" in image_prompt_type_value)+ image_end = gr.Gallery(visible=False)+ video_source = gr.Video(label= "Video to Continue", visible= "V" in image_prompt_type_value, value= ui_defaults.get("video_source", None),)+ model_mode = gr.Dropdown(+ choices=[+ ("Synchronous", 0),+ ("Asynchronous (better quality but around 50% extra steps added)", 5),+ ],+ value=ui_defaults.get("model_mode", 0),+ label="Generation Type", scale = 3,+ visible= True+ )+ keep_frames_video_source = gr.Text(value=ui_defaults.get("keep_frames_video_source","") , visible= "V" in image_prompt_type_value, scale = 2, label= "Truncate Video beyond this number of Frames of Video (empty=Keep All)" )+ elif recammaster:+ image_prompt_type = gr.Radio(visible= False)+ image_start = gr.Gallery(visible = False)+ image_end = gr.Gallery(visible=False)+ video_source = gr.Video(label= "Video Source", visible = True, value= ui_defaults.get("video_source", None),)+ model_mode = gr.Dropdown(+ choices=[+ ("Pan Right", 1),+ ("Pan Left", 2),+ ("Tilt Up", 3),+ ("Tilt Down", 4),+ ("Zoom In", 5),+ ("Zoom Out", 6),+ ("Translate Up (with rotation)", 7),+ ("Translate Down (with rotation)", 8),+ ("Arc Left (with rotation)", 9),+ ("Arc Right (with rotation)", 10),+ ],+ value=ui_defaults.get("model_mode", 1),+ label="Camera Movement Type", scale = 3,+ visible= True+ )+ keep_frames_video_source = gr.Text(visible=False)else:- image_start = gr.Image(label= "Image as a starting point for a new video", type ="pil",value= ui_defaults.get("image_start", None), visible= "S" in image_prompt_type_value )+ image_prompt_type_value= ui_defaults.get("image_prompt_type","S")+ image_prompt_type = gr.Radio( [("Use only a Start Image", "S"),("Use both a Start and an End Image", "SE")], value =image_prompt_type_value, label="Location", show_label= False, visible= True, scale= 3)++ image_start = gr.Gallery(+ label="Images as starting points for new videos", type ="pil", #file_types= "image",+ columns=[3], rows=[1], object_fit="contain", height="auto", selected_index=0, interactive= True, value= ui_defaults.get("image_start", None), visible= "S" in image_prompt_type_value)- if args.multiple_images:image_end = gr.Gallery(label="Images as ending points for new videos", type ="pil", #file_types= "image",columns=[3], rows=[1], object_fit="contain", height="auto", selected_index=0, interactive= True, visible="E" in image_prompt_type_value, value= ui_defaults.get("image_end", None))- else:- image_end = gr.Image(label= "Last Image for a new video", type ="pil", visible="E" in image_prompt_type_value, value= ui_defaults.get("image_end", None))-- with gr.Column(visible= "recam" in model_filename ) as recam_column:- camera_type = gr.Dropdown(- choices=[- ("Pan Right", 1),- ("Pan Left", 2),- ("Tilt Up", 3),- ("Tilt Down", 4),- ("Zoom In", 5),- ("Zoom Out", 6),- ("Translate Up (with rotation)", 7),- ("Translate Down (with rotation)", 8),- ("Arc Left (with rotation)", 9),- ("Arc Right (with rotation)", 10),- ],- value=ui_defaults.get("camera_type", 1),- label="Camera Movement Type", scale = 3- )- video_source = gr.Video(label= "Video Source", value= ui_defaults.get("video_source", None),)--- with gr.Column(visible= "Vace" in model_filename ) as video_prompt_column:++ video_source = gr.Video(visible=False)+ model_mode = gr.Dropdown(visible=False)+ keep_frames_video_source = gr.Text(visible=False)++ with gr.Column(visible= vace ) as video_prompt_column:video_prompt_type_value= ui_defaults.get("video_prompt_type","")video_prompt_type = gr.Text(value= video_prompt_type_value, visible= False)with gr.Row():@@ -3639,9 +3864,8 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Non)# video_prompt_type_image_refs = gr.Checkbox(value="I" in video_prompt_type_value , label= "Use References Images (Faces, Objects) to customize New Video", scale =1 )-video_guide = gr.Video(label= "Control Video", visible= "V" in video_prompt_type_value, value= ui_defaults.get("video_guide", None),)- keep_frames = gr.Text(value=ui_defaults.get("keep_frames","") , visible= "V" in video_prompt_type_value, scale = 2, label= "Frames to keep in Control Video (empty=All, 1=first, a:b for a range, space to separate values)" ) #, -1=last+ keep_frames_video_guide = gr.Text(value=ui_defaults.get("keep_frames_video_guide","") , visible= "V" in video_prompt_type_value, scale = 2, label= "Frames to keep in Control Video (empty=All, 1=first, a:b for a range, space to separate values)" ) #, -1=lastimage_refs = gr.Gallery( label ="Reference Images",type ="pil", show_label= True,columns=[3], rows=[1], object_fit="contain", height="auto", selected_index=0, interactive= True, visible= "I" in video_prompt_type_value,@@ -3716,11 +3940,15 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonlabel="Resolution")with gr.Row():- if "recam" in model_filename:+ if recammaster:video_length = gr.Slider(5, 193, value=ui_defaults.get("video_length", 81), step=4, label="Number of frames (16 = 1s), locked", interactive= False)+ elif diffusion_forcing:+ video_length = gr.Slider(17, 737, value=ui_defaults.get("video_length", 97), step=20, label="Number of frames (24 = 1s)", interactive= True)+ elif vace:+ video_length = gr.Slider(17, 737, value=ui_defaults.get("video_length", 81), step=4, label="Number of frames (16 = 1s)", interactive= True)else:video_length = gr.Slider(5, 193, value=ui_defaults.get("video_length", 81), step=4, label="Number of frames (16 = 1s)", interactive= True)-+ with gr.Row():num_inference_steps = gr.Slider(1, 100, value=ui_defaults.get("num_inference_steps",30), step=1, label="Number of Inference Steps")@@ -3737,7 +3965,7 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonchoices=[("Generate every combination of images and texts", 0),("Match images and text prompts", 1),- ], visible= args.multiple_images, label= "Multiple Images as Texts Prompts"+ ], visible= True, label= "Multiple Images as Texts Prompts")with gr.Row():guidance_scale = gr.Slider(1.0, 20.0, value=ui_defaults.get("guidance_scale",5), step=0.5, label="Guidance Scale", visible=True)@@ -3848,14 +4076,20 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonwith gr.Row():cfg_zero_step = gr.Slider(-1, 39, value=ui_defaults.get("cfg_zero_step",-1), step=1, label="CFG Zero below this Layer (Extra Process)")- with gr.Tab("Sliding Window", visible= "Vace" in model_filename ) as sliding_window_tab:+ with gr.Tab("Sliding Window", visible= "Vace" in model_filename or "diffusion_forcing" in model_filename) as sliding_window_tab:with gr.Column():gr.Markdown("<B>A Sliding Window allows you to generate video with a duration not limited by the Model</B>")+ gr.Markdown("<B>It is automatically turned on if the number of frames to generate is higher than the Window Size</B>")+ if diffusion_forcing:+ sliding_window_size = gr.Slider(37, 137, value=ui_defaults.get("sliding_window_size", 97), step=20, label="Sliding Window Size")+ sliding_window_overlap = gr.Slider(17, 97, value=ui_defaults.get("sliding_window_overlap",17), step=20, label="Windows Frames Overlap (needed to maintain continuity between windows, a higher value will require more windows)")+ sliding_window_discard_last_frames = gr.Slider(0, 10, value=ui_defaults.get("sliding_window_discard_last_frames", 0), step=1, visible = False)+ else:+ sliding_window_size = gr.Slider(5, 137, value=ui_defaults.get("sliding_window_size", 81), step=4, label="Sliding Window Size")+ sliding_window_overlap = gr.Slider(1, 97, value=ui_defaults.get("sliding_window_overlap",17), step=4, label="Windows Frames Overlap (needed to maintain continuity between windows, a higher value will require more windows)")+ sliding_window_discard_last_frames = gr.Slider(0, 10, value=ui_defaults.get("sliding_window_discard_last_frames", 16), step=1, label="Discard Last Frames of a Window (that may have bad quality)", visible = True)- sliding_window_repeat = gr.Slider(0, 20, value=ui_defaults.get("sliding_window_repeat", 0), step=1, label="Sliding Window Iterations (O=Disabled)")- sliding_window_overlap = gr.Slider(1, 32, value=ui_defaults.get("sliding_window_overlap",16), step=1, label="Windows Frames Overlap (needed to maintain continuity between windows, a higher value will require more windows)")- sliding_window_discard_last_frames = gr.Slider(1, 10, value=ui_defaults.get("sliding_window_discard_last_frames", 4), step=1, label="Discard Last Frames of a Window (that may have bad quality)")with gr.Tab("Miscellaneous", visible= not "recam" in model_filename):gr.Markdown("<B>With Riflex you can generate videos longer than 5s which is the default duration of videos used to train the model</B>")@@ -3875,203 +4109,54 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonif not update_form:with gr.Column():gen_status = gr.Text(interactive= False, label = "Status")+ status_trigger = gr.Text(interactive= False, visible=False)output = gr.Gallery( label="Generated videos", show_label=False, elem_id="gallery" , columns=[3], rows=[1], object_fit="contain", height=450, selected_index=0, interactive= False)+ output_trigger = gr.Text(interactive= False, visible=False)+generate_btn = gr.Button("Generate")add_to_queue_btn = gr.Button("Add New Prompt To Queue", visible = False)with gr.Column(visible= False) as current_gen_column:+ with gr.Accordion("Preview", open=False) as queue_accordion:+ preview = gr.Image(label="Preview", height=200, show_label= False)+ preview_trigger = gr.Text(visible= False)+ gen_info = gr.HTML(visible=False, min_height=1)with gr.Row():- gen_info = gr.HTML(visible=False, min_height=1)- with gr.Row():- onemore_btn = gr.Button("One More Sample Please !")+ onemoresample_btn = gr.Button("One More Sample Please !")+ onemorewindow_btn = gr.Button("Extend this Sample Please !", visible = False)abort_btn = gr.Button("Abort")with gr.Accordion("Queue Management", open=False) as queue_accordion:- queue_df = gr.DataFrame(- headers=["Qty","Prompt", "Length","Steps","", "", "", "", ""],- datatype=[ "str","markdown","str", "markdown", "markdown", "markdown", "str", "str", "str"],- column_widths= ["5%", None, "7%", "7%", "10%", "10%", "3%", "3%", "34"],- interactive=False,- col_count=(9, "fixed"),- wrap=True,- value=[],- line_breaks= True,- visible= True,- elem_id="queue_df"- )- with gr.Row():- queue_zip_base64_output = gr.Text(visible=False)- save_queue_btn = gr.DownloadButton("Save Queue", size="sm")- load_queue_btn = gr.UploadButton("Load Queue", file_types=[".zip"], size="sm")- clear_queue_btn = gr.Button("Clear Queue", size="sm", variant="stop")- quit_button = gr.Button("Save and Quit", size="sm", variant="secondary")- with gr.Row(visible=False) as quit_confirmation_row:- confirm_quit_button = gr.Button("Confirm", elem_id="comfirm_quit_btn_hidden", size="sm", variant="stop")- cancel_quit_button = gr.Button("Cancel", size="sm", variant="secondary")- hidden_force_quit_trigger = gr.Button("force_quit", visible=False, elem_id="force_quit_btn_hidden")- hidden_countdown_state = gr.Number(value=-1, visible=False, elem_id="hidden_countdown_state_num")- single_hidden_trigger_btn = gr.Button("trigger_countdown", visible=False, elem_id="trigger_info_single_btn")-- start_quit_timer_js = """- () => {- function findAndClickGradioButton(elemId) {- const gradioApp = document.querySelector('gradio-app') || document;- const button = gradioApp.querySelector(`#${elemId}`);- if (button) { button.click(); }- }-- if (window.quitCountdownTimeoutId) clearTimeout(window.quitCountdownTimeoutId);-- let js_click_count = 0;- const max_clicks = 5;-- function countdownStep() {- if (js_click_count < max_clicks) {- findAndClickGradioButton('trigger_info_single_btn');- js_click_count++;- window.quitCountdownTimeoutId = setTimeout(countdownStep, 1000);- } else {- findAndClickGradioButton('force_quit_btn_hidden');- }- }-- countdownStep();- }- """-- cancel_quit_timer_js = """- () => {- if (window.quitCountdownTimeoutId) {- clearTimeout(window.quitCountdownTimeoutId);- window.quitCountdownTimeoutId = null;- console.log("Quit countdown cancelled (single trigger).");- }- }- """-- trigger_zip_download_js = """- (base64String) => {- if (!base64String) {- console.log("No base64 zip data received, skipping download.");- return;- }- try {- const byteCharacters = atob(base64String);- const byteNumbers = new Array(byteCharacters.length);- for (let i = 0; i < byteCharacters.length; i++) {- byteNumbers[i] = byteCharacters.charCodeAt(i);- }- const byteArray = new Uint8Array(byteNumbers);- const blob = new Blob([byteArray], { type: 'application/zip' });-- const url = URL.createObjectURL(blob);- const a = document.createElement('a');- a.style.display = 'none';- a.href = url;- a.download = 'queue.zip';- document.body.appendChild(a);- a.click();-- window.URL.revokeObjectURL(url);- document.body.removeChild(a);- console.log("Zip download triggered.");- } catch (e) {- console.error("Error processing base64 data or triggering download:", e);- }- }- """-- single_hidden_trigger_btn.click(- fn=show_countdown_info_from_state,- inputs=[hidden_countdown_state],- outputs=[hidden_countdown_state]- )- quit_button.click(- fn=start_quit_process,- inputs=[],- outputs=[hidden_countdown_state, quit_button, quit_confirmation_row]- ).then(- fn=None, inputs=None, outputs=None, js=start_quit_timer_js- )-- confirm_quit_button.click(- fn=quit_application,- inputs=[],- outputs=[]- ).then(- fn=None, inputs=None, outputs=None, js=cancel_quit_timer_js- )-- cancel_quit_button.click(- fn=cancel_quit_process,- inputs=[],- outputs=[hidden_countdown_state, quit_button, quit_confirmation_row]- ).then(- fn=None, inputs=None, outputs=None, js=cancel_quit_timer_js- )-- hidden_force_quit_trigger.click(- fn=quit_application,- inputs=[],- outputs=[]- )+ with gr.Row( ):+ queue_df = gr.DataFrame(+ headers=["Qty","Prompt", "Length","Steps","", "", "", "", ""],+ datatype=[ "str","markdown","str", "markdown", "markdown", "markdown", "str", "str", "str"],+ column_widths= ["5%", None, "7%", "7%", "10%", "10%", "3%", "3%", "34"],+ interactive=False,+ col_count=(9, "fixed"),+ wrap=True,+ value=[],+ line_breaks= True,+ visible= True,+ elem_id="queue_df",+ max_height= 1000- save_queue_btn.click(- fn=save_queue_action,- inputs=[state],- outputs=[queue_zip_base64_output]- ).then(- fn=None,- inputs=[queue_zip_base64_output],- outputs=None,- js=trigger_zip_download_js- )-- should_start_flag = gr.State(False)- load_queue_btn.upload(- fn=load_queue_action,- inputs=[load_queue_btn, state],- outputs=[queue_df]- ).then(- fn=lambda s: (gr.update(visible=bool(get_gen_info(s).get("queue",[]))), gr.Accordion(open=True)) if bool(get_gen_info(s).get("queue",[])) else (gr.update(visible=False), gr.update()),- inputs=[state],- outputs=[current_gen_column, queue_accordion]- ).then(- fn=lambda s: (- (gr.Button(visible=False), gr.Button(visible=True), gr.Column(visible=True), True)- if bool(get_gen_info(s).get("queue",[]))- else (gr.Button(visible=True), gr.Button(visible=False), gr.Column(visible=False), False)- ),- inputs=[state],- outputs=[generate_btn, add_to_queue_btn, current_gen_column, should_start_flag]- ).then(- fn=start_processing_if_needed,- inputs=[should_start_flag, state],- outputs=[gen_status],- trigger_mode="once"- ).then(- fn=finalize_generation_with_state,- inputs=[state],- outputs=[output, abort_btn, generate_btn, add_to_queue_btn, current_gen_column, gen_info, queue_accordion, state],- trigger_mode="always_last"- ).then(- unload_model_if_needed,- inputs= [state],- outputs= []- )-- clear_queue_btn.click(- fn=clear_queue_action,- inputs=[state],- outputs=[queue_df]- ).then(- fn=lambda: (gr.update(visible=False), gr.Accordion(open=False)),- inputs=None,- outputs=[current_gen_column, queue_accordion]- )+ )+ with gr.Row(visible= True):+ queue_zip_base64_output = gr.Text(visible=False)+ save_queue_btn = gr.DownloadButton("Save Queue", size="sm")+ load_queue_btn = gr.UploadButton("Load Queue", file_types=[".zip"], size="sm")+ clear_queue_btn = gr.Button("Clear Queue", size="sm", variant="stop")+ quit_button = gr.Button("Save and Quit", size="sm", variant="secondary")+ with gr.Row(visible=False) as quit_confirmation_row:+ confirm_quit_button = gr.Button("Confirm", elem_id="comfirm_quit_btn_hidden", size="sm", variant="stop")+ cancel_quit_button = gr.Button("Cancel", size="sm", variant="secondary")+ hidden_force_quit_trigger = gr.Button("force_quit", visible=False, elem_id="force_quit_btn_hidden")+ hidden_countdown_state = gr.Number(value=-1, visible=False, elem_id="hidden_countdown_state_num")+ single_hidden_trigger_btn = gr.Button("trigger_countdown", visible=False, elem_id="trigger_info_single_btn")extra_inputs = prompt_vars + [wizard_prompt, wizard_variables_var, wizard_prompt_activated_var, video_prompt_column, image_prompt_column,prompt_column_advanced, prompt_column_wizard_vars, prompt_column_wizard, lset_name, advanced_row, sliding_window_tab,- video_prompt_type_video_guide, video_prompt_type_image_refs, recam_column] # show_advanced presets_column,+ video_prompt_type_video_guide, video_prompt_type_image_refs] # show_advanced presets_column,if update_form:locals_dict = locals()gen_inputs = [state_dict if k=="state" else locals_dict[k] for k in inputs_names] + [state_dict] + extra_inputs@@ -4080,10 +4165,10 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nontarget_state = gr.Text(value = "state", interactive= False, visible= False)target_settings = gr.Text(value = "settings", interactive= False, visible= False)- image_prompt_type.change(fn=refresh_image_prompt_type, inputs=[state, image_prompt_type], outputs=[image_start, image_end])- video_prompt_video_guide_trigger.change(fn=refresh_video_prompt_video_guide_trigger, inputs=[video_prompt_type, video_prompt_video_guide_trigger], outputs=[video_prompt_type, video_prompt_type_video_guide, video_guide, video_mask, keep_frames])+ image_prompt_type.change(fn=refresh_image_prompt_type, inputs=[state, image_prompt_type], outputs=[image_start, image_end, video_source, keep_frames_video_source] )+ video_prompt_video_guide_trigger.change(fn=refresh_video_prompt_video_guide_trigger, inputs=[video_prompt_type, video_prompt_video_guide_trigger], outputs=[video_prompt_type, video_prompt_type_video_guide, video_guide, video_mask, keep_frames_video_guide])video_prompt_type_image_refs.input(fn=refresh_video_prompt_type_image_refs, inputs = [video_prompt_type, video_prompt_type_image_refs], outputs = [video_prompt_type, image_refs, remove_background_image_ref ])- video_prompt_type_video_guide.input(fn=refresh_video_prompt_type_video_guide, inputs = [video_prompt_type, video_prompt_type_video_guide], outputs = [video_prompt_type, video_guide, keep_frames, video_mask])+ video_prompt_type_video_guide.input(fn=refresh_video_prompt_type_video_guide, inputs = [video_prompt_type, video_prompt_type_video_guide], outputs = [video_prompt_type, video_guide, keep_frames_video_guide, video_mask])show_advanced.change(fn=switch_advanced, inputs=[state, show_advanced, lset_name], outputs=[advanced_row, preset_buttons_rows, refresh_lora_btn, refresh2_row ,lset_name ]).then(fn=switch_prompt_type, inputs = [state, wizard_prompt_activated_var, wizard_variables_var, prompt, wizard_prompt, *prompt_vars], outputs = [wizard_prompt_activated_var, wizard_variables_var, prompt, wizard_prompt, prompt_column_advanced, prompt_column_wizard, prompt_column_wizard_vars, *prompt_vars])@@ -4100,14 +4185,43 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonrefresh_lora_btn.click(refresh_lora_list, inputs=[state, lset_name,loras_choices], outputs=[lset_name, loras_choices])refresh_lora_btn2.click(refresh_lora_list, inputs=[state, lset_name,loras_choices], outputs=[lset_name, loras_choices])output.select(select_video, state, None )-- gen_status.change(refresh_gallery,- inputs = [state, gen_status],- outputs = [output, gen_info, generate_btn, add_to_queue_btn, current_gen_column, queue_df, abort_btn])-+ preview_trigger.change(refresh_preview, inputs= [state], outputs= [preview])++ def refresh_status_async(state, progress=gr.Progress()):+ gen = get_gen_info(state)+ gen["progress"] = progress++ while True:+ progress_args= gen.get("progress_args", None)+ if progress_args != None:+ progress(*progress_args)+ gen["progress_args"] = None+ status= gen.get("status","")+ if status == None or len(status) > 0:+ yield status+ gen["status"]= ""+ if not gen.get("status_display", False):+ return+ time.sleep(0.5)++ def activate_status(state):+ if state.get("validate_success",0) != 1:+ return+ gen = get_gen_info(state)+ gen["status_display"] = True+ return time.time()++ status_trigger.change(refresh_status_async, inputs= [state] , outputs= [gen_status], show_progress_on= [gen_status])++ output_trigger.change(refresh_gallery,+ inputs = [state],+ outputs = [output, gen_info, generate_btn, add_to_queue_btn, current_gen_column, queue_df, abort_btn, onemorewindow_btn])++- abort_btn.click(abort_generation, [state], [gen_status, abort_btn] ) #.then(refresh_gallery, inputs = [state, gen_info], outputs = [output, gen_info, queue_df] )- onemore_btn.click(fn=one_more_sample,inputs=[state], outputs= [state])+ abort_btn.click(abort_generation, [state], [ abort_btn] ) #.then(refresh_gallery, inputs = [state, gen_info], outputs = [output, gen_info, queue_df] )+ onemoresample_btn.click(fn=one_more_sample,inputs=[state], outputs= [state])+ onemorewindow_btn.click(fn=one_more_window,inputs=[state], outputs= [state])inputs_names= list(inspect.signature(save_inputs).parameters)[1:-1]locals_dict = locals()@@ -4144,9 +4258,12 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Non).then(fn=prepare_generate_video,inputs= [state],outputs= [generate_btn, add_to_queue_btn, current_gen_column]+ ).then(fn=activate_status,+ inputs= [state],+ outputs= [status_trigger],).then(fn=process_tasks,inputs= [state],- outputs= [gen_status],+ outputs= [preview_trigger, output_trigger],).then(finalize_generation,inputs= [state],outputs= [output, abort_btn, generate_btn, add_to_queue_btn, current_gen_column, gen_info]@@ -4159,6 +4276,97 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonoutputs= [])+ gr.on(triggers=[load_queue_btn.upload, main.load],+ fn=load_queue_action,+ inputs=[load_queue_btn, state],+ outputs=[queue_df]+ ).then(+ fn=lambda s: (gr.update(visible=bool(get_gen_info(s).get("queue",[]))), gr.Accordion(open=True)) if bool(get_gen_info(s).get("queue",[])) else (gr.update(visible=False), gr.update()),+ inputs=[state],+ outputs=[current_gen_column, queue_accordion]+ ).then(+ fn=init_process_queue_if_any,+ inputs=[state],+ outputs=[generate_btn, add_to_queue_btn, current_gen_column, ]+ ).then(fn=activate_status,+ inputs= [state],+ outputs= [status_trigger],+ ).then(+ fn=process_tasks,+ inputs=[state],+ outputs=[preview_trigger, output_trigger],+ trigger_mode="once"+ ).then(+ fn=finalize_generation_with_state,+ inputs=[state],+ outputs=[output, abort_btn, generate_btn, add_to_queue_btn, current_gen_column, gen_info, queue_accordion, state],+ trigger_mode="always_last"+ ).then(+ unload_model_if_needed,+ inputs= [state],+ outputs= []+ )+++ start_quit_timer_js, cancel_quit_timer_js, trigger_zip_download_js = get_timer_js()++ single_hidden_trigger_btn.click(+ fn=show_countdown_info_from_state,+ inputs=[hidden_countdown_state],+ outputs=[hidden_countdown_state]+ )+ quit_button.click(+ fn=start_quit_process,+ inputs=[],+ outputs=[hidden_countdown_state, quit_button, quit_confirmation_row]+ ).then(+ fn=None, inputs=None, outputs=None, js=start_quit_timer_js+ )++ confirm_quit_button.click(+ fn=quit_application,+ inputs=[],+ outputs=[]+ ).then(+ fn=None, inputs=None, outputs=None, js=cancel_quit_timer_js+ )++ cancel_quit_button.click(+ fn=cancel_quit_process,+ inputs=[],+ outputs=[hidden_countdown_state, quit_button, quit_confirmation_row]+ ).then(+ fn=None, inputs=None, outputs=None, js=cancel_quit_timer_js+ )++ hidden_force_quit_trigger.click(+ fn=quit_application,+ inputs=[],+ outputs=[]+ )++ save_queue_btn.click(+ fn=save_queue_action,+ inputs=[state],+ outputs=[queue_zip_base64_output]+ ).then(+ fn=None,+ inputs=[queue_zip_base64_output],+ outputs=None,+ js=trigger_zip_download_js+ )++ clear_queue_btn.click(+ fn=clear_queue_action,+ inputs=[state],+ outputs=[queue_df]+ ).then(+ fn=lambda: (gr.update(visible=False), gr.Accordion(open=False)),+ inputs=None,+ outputs=[current_gen_column, queue_accordion]+ )++add_to_queue_btn.click(fn=validate_wizard_prompt,inputs =[state, wizard_prompt_activated_var, wizard_variables_var, prompt, wizard_prompt, *prompt_vars] ,outputs= [prompt]@@ -4183,10 +4391,8 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonoutputs=[modal_container])- return ( state,- loras_choices, lset_name, state, queue_df, current_gen_column,- gen_status, output, abort_btn, generate_btn, add_to_queue_btn,- gen_info, queue_accordion, video_guide, video_mask, video_prompt_video_guide_trigger+ return ( state, loras_choices, lset_name, state,+ video_guide, video_mask, video_prompt_video_guide_trigger)@@ -4437,6 +4643,77 @@ def select_tab(tab_state, evt:gr.SelectData):tab_state["tab_no"] = new_tab_noreturn gr.Tabs()+def get_timer_js():+ start_quit_timer_js = """+ () => {+ function findAndClickGradioButton(elemId) {+ const gradioApp = document.querySelector('gradio-app') || document;+ const button = gradioApp.querySelector(`#${elemId}`);+ if (button) { button.click(); }+ }++ if (window.quitCountdownTimeoutId) clearTimeout(window.quitCountdownTimeoutId);++ let js_click_count = 0;+ const max_clicks = 5;++ function countdownStep() {+ if (js_click_count < max_clicks) {+ findAndClickGradioButton('trigger_info_single_btn');+ js_click_count++;+ window.quitCountdownTimeoutId = setTimeout(countdownStep, 1000);+ } else {+ findAndClickGradioButton('force_quit_btn_hidden');+ }+ }++ countdownStep();+ }+ """++ cancel_quit_timer_js = """+ () => {+ if (window.quitCountdownTimeoutId) {+ clearTimeout(window.quitCountdownTimeoutId);+ window.quitCountdownTimeoutId = null;+ console.log("Quit countdown cancelled (single trigger).");+ }+ }+ """++ trigger_zip_download_js = """+ (base64String) => {+ if (!base64String) {+ console.log("No base64 zip data received, skipping download.");+ return;+ }+ try {+ const byteCharacters = atob(base64String);+ const byteNumbers = new Array(byteCharacters.length);+ for (let i = 0; i < byteCharacters.length; i++) {+ byteNumbers[i] = byteCharacters.charCodeAt(i);+ }+ const byteArray = new Uint8Array(byteNumbers);+ const blob = new Blob([byteArray], { type: 'application/zip' });++ const url = URL.createObjectURL(blob);+ const a = document.createElement('a');+ a.style.display = 'none';+ a.href = url;+ a.download = 'queue.zip';+ document.body.appendChild(a);+ a.click();++ window.URL.revokeObjectURL(url);+ document.body.removeChild(a);+ console.log("Zip download triggered.");+ } catch (e) {+ console.error("Error processing base64 data or triggering download:", e);+ }+ }+ """+ return start_quit_timer_js, cancel_quit_timer_js, trigger_zip_download_js+def create_demo():global vmc_event_handlercss = """@@ -4670,6 +4947,7 @@ def create_demo():z-index: 2;pointer-events: none;}+"""UI_theme = server_config.get("UI_theme", "default")UI_theme = args.theme if len(args.theme) > 0 else UI_theme@@ -4678,8 +4956,8 @@ def create_demo():else:theme = gr.themes.Soft(font=["Verdana"], primary_hue="sky", neutral_hue="slate", text_size="md")- with gr.Blocks(css=css, theme=theme, title= "Wan2GP") as demo:- gr.Markdown("<div align=center><H1>Wan<SUP>GP</SUP> v4.2 <FONT SIZE=4>by <I>DeepBeepMeep</I></FONT> <FONT SIZE=3>") # (<A HREF='https://github.com/deepbeepmeep/Wan2GP'>Updates</A>)</FONT SIZE=3></H1></div>")+ with gr.Blocks(css=css, theme=theme, title= "Wan2GP") as main:+ gr.Markdown("<div align=center><H1>Wan<SUP>GP</SUP> v4.3 <FONT SIZE=4>by <I>DeepBeepMeep</I></FONT> <FONT SIZE=3>") # (<A HREF='https://github.com/deepbeepmeep/Wan2GP'>Updates</A>)</FONT SIZE=3></H1></div>")global model_listtab_state = gr.State({ "tab_no":0 })@@ -4697,11 +4975,9 @@ def create_demo():with gr.Row():header = gr.Markdown(generate_header(transformer_filename, compile, attention_mode), visible= True)with gr.Row():- ( state,- loras_choices, lset_name, state, queue_df, current_gen_column,- gen_status, output, abort_btn, generate_btn, add_to_queue_btn,- gen_info, queue_accordion, video_guide, video_mask, video_prompt_type_video_trigger- ) = generate_video_tab(model_choice=model_choice, header=header)+ ( state, loras_choices, lset_name, state,+ video_guide, video_mask, video_prompt_type_video_trigger+ ) = generate_video_tab(model_choice=model_choice, header=header, main = main)with gr.Tab("Informations", id="info"):generate_info_tab()with gr.Tab("Video Mask Creator", id="video_mask_creator") as video_mask_creator:@@ -4713,30 +4989,12 @@ def create_demo():with gr.Tab("Downloads", id="downloads") as downloads_tab:generate_download_tab(lset_name, loras_choices, state)with gr.Tab("Configuration", id="configuration"):- generate_configuration_tab(state, demo, header, model_choice)+ generate_configuration_tab(state, main, header, model_choice)with gr.Tab("About"):generate_about_tab()- should_start_flag = gr.State(False)-- demo.load(- fn=run_autoload_and_prepare_ui,- inputs=[state],- outputs=[queue_df, current_gen_column, queue_accordion, should_start_flag, state]- ).then(- fn=start_processing_if_needed,- inputs=[should_start_flag, state],- outputs=[gen_status],- trigger_mode="once"- ).then(- fn=finalize_generation_with_state,- inputs=[state],- outputs=[output, abort_btn, generate_btn, add_to_queue_btn, current_gen_column, gen_info, queue_accordion, state],- trigger_mode="always_last"- )-main_tabs.select(fn=select_tab, inputs= [tab_state], outputs= main_tabs)- return demo+ return mainif __name__ == "__main__":atexit.register(autosave_queue)