Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

migrate some new files (#64)

Frank Leec4c5d64e49b1
diff --git a/assets/texts/imagenet_id.txt b/assets/texts/imagenet_id.txt
new file mode 100644
index 0000000..9085aa0
--- /dev/null
+++ b/assets/texts/imagenet_id.txt
@@ -0,0 +1,8 @@
+207
+360
+387
+974
+88
+979
+417
+279
diff --git a/assets/texts/imagenet_labels.txt b/assets/texts/imagenet_labels.txt
new file mode 100644
index 0000000..6493fdb
--- /dev/null
+++ b/assets/texts/imagenet_labels.txt
@@ -0,0 +1,8 @@
+golden retriever
+otter
+lesser panda
+geyser
+macaw
+valley
+balloon
+golden panda
diff --git a/assets/texts/t2i_samples.txt b/assets/texts/t2i_samples.txt
new file mode 100644
index 0000000..0fd1284
--- /dev/null
+++ b/assets/texts/t2i_samples.txt
@@ -0,0 +1,8 @@
+A small cactus with a happy face in the Sahara desert.
+Bright scene, aerial view,ancient city, fantasy, gorgeous light, mirror reflection, high detail, wide angle lens.
+Nature vs human nature, surreal, UHD, 8k, hyper details, rich colors, photograph.
+Poster of a mechanical cat, techical Schematics viewed from front.
+Luffy from ONEPIECE, handsome face, fantasy.
+Real beautiful woman.
+A alpaca made of colorful building blocks, cyberpunk.
+artistic
diff --git a/assets/texts/t2v_latte.txt b/assets/texts/t2v_latte.txt
new file mode 100644
index 0000000..a61359c
--- /dev/null
+++ b/assets/texts/t2v_latte.txt
@@ -0,0 +1,7 @@
+Yellow and black tropical fish dart through the sea.
+An epic tornado attacking above aglowing city at night.
+Slow pan upward of blazing oak fire in an indoor fireplace.
+a cat wearing sunglasses and working as a lifeguard at pool.
+Sunset over the sea.
+A dog in astronaut suit and sunglasses floating in space.
+A astronaut in flying in space, 4k, high resolution
diff --git a/assets/texts/t2v_samples.txt b/assets/texts/t2v_samples.txt
new file mode 100644
index 0000000..862a8c5
--- /dev/null
+++ b/assets/texts/t2v_samples.txt
@@ -0,0 +1,3 @@
+The majestic beauty of a waterfall cascading down a cliff into a serene lake. The waterfall, with its powerful flow, is the central focus of the video. The surrounding landscape is lush and green, with trees and foliage adding to the natural beauty of the scene. The camera angle provides a bird's eye view of the waterfall, allowing viewers to appreciate the full height and grandeur of the waterfall. The video is a stunning representation of nature's power and beauty.
+A majestic lion in its natural habitat. The lion, with its golden fur, is seen walking through a lush green field. The field is dotted with bushes and trees, providing a serene and natural backdrop. The lion's movement is captured in three frames, each showing the lion in a different position, giving a sense of its journey through the field. The overall style of the video is realistic and naturalistic, capturing the beauty of the lion in its environment.
+A lively scene in a park, where a large flock of pigeons is seen in the grassy field. The birds are scattered across the field, some standing, some walking, and some pecking at the ground. The park is lush with green grass and trees, providing a natural habitat for the birds. In the background, there are playground equipment and a building, indicating that the park is located in an urban area. The video is shot in daylight, with the sunlight casting a warm glow on the scene. The overall style of the video is a real-life, candid capture of a moment in the park, showcasing the interaction between the birds and their environment.
diff --git a/assets/texts/t2v_sora.txt b/assets/texts/t2v_sora.txt
new file mode 100644
index 0000000..eeb887b
--- /dev/null
+++ b/assets/texts/t2v_sora.txt
@@ -0,0 +1,48 @@
+A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.
+Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field.
+A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors.
+Drone view of waves crashing against the rugged cliffs along Big Sur’s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff’s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.
+Animated scene features a close-up of a short fluffy monster kneeling beside a melting red candle. The art style is 3D and realistic, with a focus on lighting and texture. The mood of the painting is one of wonder and curiosity, as the monster gazes at the flame with wide eyes and open mouth. Its pose and expression convey a sense of innocence and playfulness, as if it is exploring the world around it for the first time. The use of warm colors and dramatic lighting further enhances the cozy atmosphere of the image.
+A gorgeously rendered papercraft world of a coral reef, rife with colorful fish and sea creatures.
+This close-up shot of a Victoria crowned pigeon showcases its striking blue plumage and red chest. Its crest is made of delicate, lacy feathers, while its eye is a striking red color. The bird’s head is tilted slightly to the side, giving the impression of it looking regal and majestic. The background is blurred, drawing attention to the bird’s striking appearance.
+Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee.
+A young man at his 20s is sitting on a piece of cloud in the sky, reading a book.
+Historical footage of California during the gold rush.
+A close up view of a glass sphere that has a zen garden within it. There is a small dwarf in the sphere who is raking the zen garden and creating patterns in the sand.
+Extreme close up of a 24 year old woman’s eye blinking, standing in Marrakech during magic hour, cinematic film shot in 70mm, depth of field, vivid colors, cinematic
+A cartoon kangaroo disco dances.
+A beautiful homemade video showing the people of Lagos, Nigeria in the year 2056. Shot with a mobile phone camera.
+A petri dish with a bamboo forest growing within it that has tiny red pandas running around.
+The camera rotates around a large stack of vintage televisions all showing different programs — 1950s sci-fi movies, horror movies, news, static, a 1970s sitcom, etc, set inside a large New York museum gallery.
+3D animation of a small, round, fluffy creature with big, expressive eyes explores a vibrant, enchanted forest. The creature, a whimsical blend of a rabbit and a squirrel, has soft blue fur and a bushy, striped tail. It hops along a sparkling stream, its eyes wide with wonder. The forest is alive with magical elements: flowers that glow and change colors, trees with leaves in shades of purple and silver, and small floating lights that resemble fireflies. The creature stops to interact playfully with a group of tiny, fairy-like beings dancing around a mushroom ring. The creature looks up in awe at a large, glowing tree that seems to be the heart of the forest.
+The camera follows behind a white vintage SUV with a black roof rack as it speeds up a steep dirt road surrounded by pine trees on a steep mountain slope, dust kicks up from it’s tires, the sunlight shines on the SUV as it speeds along the dirt road, casting a warm glow over the scene. The dirt road curves gently into the distance, with no other cars or vehicles in sight. The trees on either side of the road are redwoods, with patches of greenery scattered throughout. The car is seen from the rear following the curve with ease, making it seem as if it is on a rugged drive through the rugged terrain. The dirt road itself is surrounded by steep hills and mountains, with a clear blue sky above with wispy clouds.
+Reflections in the window of a train traveling through the Tokyo suburbs.
+A drone camera circles around a beautiful historic church built on a rocky outcropping along the Amalfi Coast, the view showcases historic and magnificent architectural details and tiered pathways and patios, waves are seen crashing against the rocks below as the view overlooks the horizon of the coastal waters and hilly landscapes of the Amalfi Coast Italy, several distant people are seen walking and enjoying vistas on patios of the dramatic ocean views, the warm glow of the afternoon sun creates a magical and romantic feeling to the scene, the view is stunning captured with beautiful photography.
+A large orange octopus is seen resting on the bottom of the ocean floor, blending in with the sandy and rocky terrain. Its tentacles are spread out around its body, and its eyes are closed. The octopus is unaware of a king crab that is crawling towards it from behind a rock, its claws raised and ready to attack. The crab is brown and spiny, with long legs and antennae. The scene is captured from a wide angle, showing the vastness and depth of the ocean. The water is clear and blue, with rays of sunlight filtering through. The shot is sharp and crisp, with a high dynamic range. The octopus and the crab are in focus, while the background is slightly blurred, creating a depth of field effect.
+A flock of paper airplanes flutters through a dense jungle, weaving around trees as if they were migrating birds.
+A cat waking up its sleeping owner demanding breakfast. The owner tries to ignore the cat, but the cat tries new tactics and finally the owner pulls out a secret stash of treats from under the pillow to hold the cat off a little longer.
+Borneo wildlife on the Kinabatangan River
+A Chinese Lunar New Year celebration video with Chinese Dragon.
+Tour of an art gallery with many beautiful works of art in different styles.
+Beautiful, snowy Tokyo city is bustling. The camera moves through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls. Gorgeous sakura petals are flying through the wind along with snowflakes.
+A stop motion animation of a flower growing out of the windowsill of a suburban house.
+The story of a robot’s life in a cyberpunk setting.
+An extreme close-up of an gray-haired man with a beard in his 60s, he is deep in thought pondering the history of the universe as he sits at a cafe in Paris, his eyes focus on people offscreen as they walk as he sits mostly motionless, he is dressed in a wool coat suit coat with a button-down shirt , he wears a brown beret and glasses and has a very professorial appearance, and the end he offers a subtle closed-mouth smile as if he found the answer to the mystery of life, the lighting is very cinematic with the golden light and the Parisian streets and city in the background, depth of field, cinematic 35mm film.
+A beautiful silhouette animation shows a wolf howling at the moon, feeling lonely, until it finds its pack.
+New York City submerged like Atlantis. Fish, whales, sea turtles and sharks swim through the streets of New York.
+A litter of golden retriever puppies playing in the snow. Their heads pop out of the snow, covered in.
+Step-printing scene of a person running, cinematic film shot in 35mm.
+Five gray wolf pups frolicking and chasing each other around a remote gravel road, surrounded by grass. The pups run and leap, chasing each other, and nipping at each other, playing.
+Basketball through hoop then explodes.
+Archeologists discover a generic plastic chair in the desert, excavating and dusting it with great care.
+A grandmother with neatly combed grey hair stands behind a colorful birthday cake with numerous candles at a wood dining room table, expression is one of pure joy and happiness, with a happy glow in her eye. She leans forward and blows out the candles with a gentle puff, the cake has pink frosting and sprinkles and the candles cease to flicker, the grandmother wears a light blue blouse adorned with floral patterns, several happy friends and family sitting at the table can be seen celebrating, out of focus. The scene is beautifully captured, cinematic, showing a 3/4 view of the grandmother and the dining room. Warm color tones and soft lighting enhance the mood.
+The camera directly faces colorful buildings in Burano Italy. An adorable dalmation looks through a window on a building on the ground floor. Many people are walking and cycling along the canal streets in front of the buildings.
+An adorable happy otter confidently stands on a surfboard wearing a yellow lifejacket, riding along turquoise tropical waters near lush tropical islands, 3D digital render art style.
+This close-up shot of a chameleon showcases its striking color changing capabilities. The background is blurred, drawing attention to the animal’s striking appearance.
+A corgi vlogging itself in tropical Maui.
+A white and orange tabby cat is seen happily darting through a dense garden, as if chasing something. Its eyes are wide and happy as it jogs forward, scanning the branches, flowers, and leaves as it walks. The path is narrow as it makes its way between all the plants. the scene is captured from a ground-level angle, following the cat closely, giving a low and intimate perspective. The image is cinematic with warm tones and a grainy texture. The scattered daylight between the leaves and plants above creates a warm contrast, accentuating the cat’s orange fur. The shot is clear and sharp, with a shallow depth of field.
+Aerial view of Santorini during the blue hour, showcasing the stunning architecture of white Cycladic buildings with blue domes. The caldera views are breathtaking, and the lighting creates a beautiful, serene atmosphere.
+Tiltshift of a construction site filled with workers, equipment, and heavy machinery.
+A giant, towering cloud in the shape of a man looms over the earth. The cloud man shoots lighting bolts down to the earth.
+A Samoyed and a Golden Retriever dog are playfully romping through a futuristic neon city at night. The neon lights emitted from the nearby buildings glistens off of their fur.
+The Glenfinnan Viaduct is a historic railway bridge in Scotland, UK, that crosses over the west highland line between the towns of Mallaig and Fort William. It is a stunning sight as a steam train leaves the bridge, traveling over the arch-covered viaduct. The landscape is dotted with lush greenery and rocky mountains, creating a picturesque backdrop for the train journey. The sky is blue and the sun is shining, making for a beautiful day to explore this majestic spot.
diff --git a/assets/texts/ucf101_id.txt b/assets/texts/ucf101_id.txt
new file mode 100644
index 0000000..e8371f0
--- /dev/null
+++ b/assets/texts/ucf101_id.txt
@@ -0,0 +1,6 @@
+0
+1
+2
+3
+4
+5
diff --git a/assets/texts/ucf101_labels.txt b/assets/texts/ucf101_labels.txt
new file mode 100644
index 0000000..264dbfd
--- /dev/null
+++ b/assets/texts/ucf101_labels.txt
@@ -0,0 +1,6 @@
+Apply Eye Makeup
+Apply Lipstick
+Archery
+Baby Crawling
+Balance Beam
+Band Marching
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..6e8196d
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,29 @@
+# should be installed by users
+torch>=1.13
+colossalai
+flash_attn # optional
+apex # optional
+
+# packages
+accelerate
+diffusers
+timm
+transformers
+xformers # necessary for PixArt
+
+# config & logging
+gdown
+pre-commit
+matplotlib
+mmengine
+tensorboard
+tqdm
+wandb
+
+# video
+pyav
+
+# text
+clip
+ftfy
+
diff --git a/scripts/inference.py b/scripts/inference.py
new file mode 100644
index 0000000..b36e70b
--- /dev/null
+++ b/scripts/inference.py
@@ -0,0 +1,95 @@
+import os
+
+import torch
+from mmengine.runner import set_random_seed
+
+from opensora.datasets import save_sample
+from opensora.registry import MODELS, SCHEDULERS, build_module
+from opensora.utils.config_utils import parse_configs
+from opensora.utils.misc import to_torch_dtype
+
+
+def load_prompts(prompt_path):
+ with open(prompt_path, "r") as f:
+ prompts = [line.strip() for line in f.readlines()]
+ return prompts
+
+
+def main():
+ # ======================================================
+ # 1. args & cfg
+ # ======================================================
+ cfg = parse_configs(training=False)
+ print(cfg)
+
+ # ======================================================
+ # 2. runtime variables
+ # ======================================================
+ torch.set_grad_enabled(False)
+ torch.backends.cuda.matmul.allow_tf32 = True
+ torch.backends.cudnn.allow_tf32 = True
+ device = "cuda" if torch.cuda.is_available() else "cpu"
+ dtype = to_torch_dtype(cfg.dtype)
+ set_random_seed(seed=cfg.seed)
+ prompts = load_prompts(cfg.prompt_path)
+
+ # ======================================================
+ # 3. build model & load weights
+ # ======================================================
+ # 3.1. build model
+ input_size = (cfg.num_frames, *cfg.image_size)
+ vae = build_module(cfg.vae, MODELS)
+ latent_size = vae.get_latent_size(input_size)
+ text_encoder = build_module(cfg.text_encoder, MODELS, device=device) # T5 must be fp32
+ model = build_module(
+ cfg.model,
+ MODELS,
+ input_size=latent_size,
+ in_channels=vae.out_channels,
+ caption_channels=text_encoder.output_dim,
+ model_max_length=text_encoder.model_max_length,
+ dtype=dtype,
+ )
+ text_encoder.y_embedder = model.y_embedder # hack for classifier-free guidance
+
+ # 3.2. move to device & eval
+ vae = vae.to(device, dtype).eval()
+ model = model.to(device, dtype).eval()
+
+ # 3.3. build scheduler
+ scheduler = build_module(cfg.scheduler, SCHEDULERS)
+
+ # 3.4. support for multi-resolution
+ model_args = dict()
+ if cfg.multi_resolution:
+ image_size = cfg.image_size
+ hw = torch.tensor([image_size], device=device, dtype=dtype).repeat(cfg.batch_size, 1)
+ ar = torch.tensor([[image_size[0] / image_size[1]]], device=device, dtype=dtype).repeat(cfg.batch_size, 1)
+ model_args["data_info"] = dict(ar=ar, hw=hw)
+
+ # ======================================================
+ # 4. inference
+ # ======================================================
+ sample_idx = 0
+ save_dir = cfg.save_dir
+ os.makedirs(save_dir, exist_ok=True)
+ for i in range(0, len(prompts), cfg.batch_size):
+ batch_prompts = prompts[i : i + cfg.batch_size]
+ samples = scheduler.sample(
+ model,
+ text_encoder,
+ z_size=(vae.out_channels, *latent_size),
+ prompts=batch_prompts,
+ device=device,
+ additional_args=model_args,
+ )
+ samples = vae.decode(samples.to(dtype))
+ for idx, sample in enumerate(samples):
+ print(f"Prompt: {batch_prompts[idx]}")
+ save_path = os.path.join(save_dir, f"sample_{sample_idx}")
+ save_sample(sample, fps=cfg.fps, save_path=save_path)
+ sample_idx += 1
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/train.py b/scripts/train.py
new file mode 100644
index 0000000..b750748
--- /dev/null
+++ b/scripts/train.py
@@ -0,0 +1,287 @@
+from copy import deepcopy
+
+import colossalai
+import torch
+import torch.distributed as dist
+import wandb
+from colossalai.booster import Booster
+from colossalai.booster.plugin import LowLevelZeroPlugin
+from colossalai.cluster import DistCoordinator
+from colossalai.nn.optimizer import HybridAdam
+from colossalai.utils import get_current_device
+from tqdm import tqdm
+
+from opensora.acceleration.parallel_states import (
+ get_data_parallel_group,
+ set_data_parallel_group,
+ set_sequence_parallel_group,
+)
+from opensora.acceleration.plugin import ZeroSeqParallelPlugin
+from opensora.acceleration.checkpoint import set_grad_checkpoint
+from opensora.datasets import DatasetFromCSV, get_transforms_image, get_transforms_video, prepare_dataloader
+from opensora.registry import MODELS, SCHEDULERS, build_module
+from opensora.utils.ckpt_utils import create_logger, load, model_sharding, record_model_param_shape, save
+from opensora.utils.config_utils import (
+ create_experiment_workspace,
+ create_tensorboard_writer,
+ parse_configs,
+ save_training_config,
+)
+from opensora.utils.misc import all_reduce_mean, format_numel_str, get_model_numel, requires_grad, to_torch_dtype
+from opensora.utils.train_utils import update_ema
+
+
+def main():
+ # ======================================================
+ # 1. args & cfg
+ # ======================================================
+ cfg = parse_configs(training=True)
+ print(cfg)
+ exp_name, exp_dir = create_experiment_workspace(cfg)
+ save_training_config(cfg._cfg_dict, exp_dir)
+
+ # ======================================================
+ # 2. runtime variables & colossalai launch
+ # ======================================================
+ assert torch.cuda.is_available(), "Training currently requires at least one GPU."
+ assert cfg.dtype in ["fp16", "bf16"], f"Unknown mixed precision {cfg.dtype}"
+
+ # 2.1. colossalai init distributed training
+ colossalai.launch_from_torch({})
+ coordinator = DistCoordinator()
+ device = get_current_device()
+ dtype = to_torch_dtype(cfg.dtype)
+
+ # 2.2. init logger, tensorboard & wandb
+ if not coordinator.is_master():
+ logger = create_logger(None)
+ else:
+ logger = create_logger(exp_dir)
+ logger.info(f"Experiment directory created at {exp_dir}")
+
+ writer = create_tensorboard_writer(exp_dir)
+ if cfg.wandb:
+ wandb.init(project="minisora", name=exp_name, config=cfg._cfg_dict)
+
+ # 2.3. initialize ColossalAI booster
+ if cfg.plugin == "zero2":
+ plugin = LowLevelZeroPlugin(
+ stage=2,
+ precision=cfg.dtype,
+ initial_scale=2**16,
+ max_norm=cfg.grad_clip,
+ )
+ set_data_parallel_group(dist.group.WORLD)
+ elif cfg.plugin == "zero2-seq":
+ plugin = ZeroSeqParallelPlugin(
+ sp_size=cfg.sp_size,
+ stage=2,
+ precision=cfg.dtype,
+ initial_scale=2**16,
+ max_norm=cfg.grad_clip,
+ )
+ set_sequence_parallel_group(plugin.sp_group)
+ set_data_parallel_group(plugin.dp_group)
+ else:
+ raise ValueError(f"Unknown plugin {cfg.plugin}")
+ booster = Booster(plugin=plugin)
+
+ # ======================================================
+ # 3. build dataset and dataloader
+ # ======================================================
+ dataset = DatasetFromCSV(
+ cfg.data_path,
+ # TODO: change transforms
+ transform=(
+ get_transforms_video(cfg.image_size[0])
+ if not cfg.use_image_transform
+ else get_transforms_image(cfg.image_size[0])
+ ),
+ num_frames=cfg.num_frames,
+ frame_interval=cfg.frame_interval,
+ root=cfg.root,
+ )
+
+ # TODO: use plugin's prepare dataloader
+ # a batch contains:
+ # {
+ # "video": torch.Tensor, # [B, C, T, H, W],
+ # "text": List[str],
+ # }
+ dataloader = prepare_dataloader(
+ dataset,
+ batch_size=cfg.batch_size,
+ num_workers=cfg.num_workers,
+ shuffle=True,
+ drop_last=True,
+ pin_memory=True,
+ process_group=get_data_parallel_group(),
+ )
+ logger.info(f"Dataset contains {len(dataset):,} videos ({cfg.data_path})")
+
+ total_batch_size = cfg.batch_size * dist.get_world_size() // cfg.sp_size
+ logger.info(f"Total batch size: {total_batch_size}")
+
+ # ======================================================
+ # 4. build model
+ # ======================================================
+ # 4.1. build model
+ input_size = (cfg.num_frames, *cfg.image_size)
+ vae = build_module(cfg.vae, MODELS)
+ latent_size = vae.get_latent_size(input_size)
+ text_encoder = build_module(cfg.text_encoder, MODELS, device=device) # T5 must be fp32
+ model = build_module(
+ cfg.model,
+ MODELS,
+ input_size=latent_size,
+ in_channels=vae.out_channels,
+ caption_channels=text_encoder.output_dim,
+ model_max_length=text_encoder.model_max_length,
+ dtype=dtype,
+ )
+ model_numel, model_numel_trainable = get_model_numel(model)
+ logger.info(
+ f"Trainable model params: {format_numel_str(model_numel_trainable)}, Total model params: {format_numel_str(model_numel)}"
+ )
+
+ # 4.2. create ema
+ ema = deepcopy(model).to(torch.float32).to(device)
+ requires_grad(ema, False)
+ ema_shape_dict = record_model_param_shape(ema)
+
+ # 4.3. move to device
+ vae = vae.to(device, dtype)
+ model = model.to(device, dtype)
+
+ # 4.4. build scheduler
+ scheduler = build_module(cfg.scheduler, SCHEDULERS)
+
+ # 4.5. setup optimizer
+ optimizer = HybridAdam(
+ filter(lambda p: p.requires_grad, model.parameters()), lr=cfg.lr, weight_decay=0, adamw_mode=True
+ )
+ lr_scheduler = None
+
+ # 4.6. prepare for training
+ if cfg.grad_checkpoint:
+ set_grad_checkpoint(model)
+ model.train()
+ update_ema(ema, model, decay=0, sharded=False)
+ ema.eval()
+
+ # =======================================================
+ # 5. boost model for distributed training with colossalai
+ # =======================================================
+ torch.set_default_dtype(dtype)
+ model, optimizer, _, dataloader, lr_scheduler = booster.boost(
+ model=model, optimizer=optimizer, lr_scheduler=lr_scheduler, dataloader=dataloader
+ )
+ torch.set_default_dtype(torch.float)
+ num_steps_per_epoch = len(dataloader)
+ logger.info("Boost model for distributed training")
+
+ # =======================================================
+ # 6. training loop
+ # =======================================================
+ start_epoch = start_step = log_step = sampler_start_idx = 0
+ running_loss = 0.0
+
+ # 6.1. resume training
+ if cfg.load is not None:
+ logger.info("Loading checkpoint")
+ start_epoch, start_step, sampler_start_idx = load(booster, model, ema, optimizer, lr_scheduler, cfg.load)
+ logger.info(f"Loaded checkpoint {cfg.load} at epoch {start_epoch} step {start_step}")
+ logger.info(f"Training for {cfg.epochs} epochs with {num_steps_per_epoch} steps per epoch")
+
+ dataloader.sampler.set_start_index(sampler_start_idx)
+ model_sharding(ema)
+
+ # 6.2. training loop
+ for epoch in range(start_epoch, cfg.epochs):
+ dataloader.sampler.set_epoch(epoch)
+ dataloader_iter = iter(dataloader)
+ logger.info(f"Beginning epoch {epoch}...")
+
+ with tqdm(
+ range(start_step, num_steps_per_epoch),
+ desc=f"Epoch {epoch}",
+ disable=not coordinator.is_master(),
+ total=num_steps_per_epoch,
+ initial=start_step,
+ ) as pbar:
+ for step in pbar:
+ batch = next(dataloader_iter)
+ x = batch["video"].to(device, dtype) # [B, C, T, H, W]
+ y = batch["text"]
+
+ with torch.no_grad():
+ # Prepare visual inputs
+ x = vae.encode(x) # [B, C, T, H/P, W/P]
+ # Prepare text inputs
+ model_args = text_encoder.encode(y)
+
+ # Diffusion
+ t = torch.randint(0, scheduler.num_timesteps, (x.shape[0],), device=device)
+ loss_dict = scheduler.training_losses(model, x, t, model_args)
+
+ # Backward & update
+ loss = loss_dict["loss"].mean()
+ booster.backward(loss=loss, optimizer=optimizer)
+ optimizer.step()
+ optimizer.zero_grad()
+
+ # Update EMA
+ update_ema(ema, model.module, optimizer=optimizer)
+
+ # Log loss values:
+ all_reduce_mean(loss)
+ running_loss += loss.item()
+ global_step = epoch * num_steps_per_epoch + step
+ log_step += 1
+
+ # Log to tensorboard
+ if coordinator.is_master() and (global_step + 1) % cfg.log_every == 0:
+ avg_loss = running_loss / log_step
+ pbar.set_postfix({"loss": avg_loss, "step": step, "global_step": global_step})
+ running_loss = 0
+ log_step = 0
+ writer.add_scalar("loss", loss.item(), global_step)
+ if cfg.wandb:
+ wandb.log(
+ {
+ "iter": global_step,
+ "num_samples": global_step * total_batch_size,
+ "epoch": epoch,
+ "loss": loss.item(),
+ "avg_loss": avg_loss,
+ },
+ step=global_step,
+ )
+
+ # Save checkpoint
+ if cfg.ckpt_every > 0 and (global_step + 1) % cfg.ckpt_every == 0:
+ save(
+ booster,
+ model,
+ ema,
+ optimizer,
+ lr_scheduler,
+ epoch,
+ step + 1,
+ global_step + 1,
+ cfg.batch_size,
+ coordinator,
+ exp_dir,
+ ema_shape_dict,
+ )
+ logger.info(
+ f"Saved checkpoint at epoch {epoch} step {step + 1} global_step {global_step + 1} to {exp_dir}"
+ )
+
+ # the continue epochs are not resumed, so we need to reset the sampler start index and start step
+ dataloader.sampler.set_start_index(0)
+ start_step = 0
+
+
+if __name__ == "__main__":
+ main()
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..e1717af
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,55 @@
+from typing import List
+
+from setuptools import find_packages, setup
+
+
+def fetch_requirements(path) -> List[str]:
+ """
+ This function reads the requirements file.
+
+ Args:
+ path (str): the path to the requirements file.
+
+ Returns:
+ The lines in the requirements file.
+ """
+ with open(path, "r") as fd:
+ return [r.strip() for r in fd.readlines()]
+
+
+def fetch_readme() -> str:
+ """
+ This function reads the README.md file in the current directory.
+
+ Returns:
+ The lines in the README file.
+ """
+ with open("README.md", encoding="utf-8") as f:
+ return f.read()
+
+
+setup(
+ name="opensora",
+ version="0.1.0",
+ packages=find_packages(
+ exclude=(
+ "videos",
+ "tests",
+ "figure",
+ "*.egg-info",
+ )
+ ),
+ description="OpenDiT",
+ long_description=fetch_readme(),
+ long_description_content_type="text/markdown",
+ license="Apache Software License 2.0",
+ install_requires=fetch_requirements("requirements.txt"),
+ python_requires=">=3.6",
+ classifiers=[
+ "Programming Language :: Python :: 3",
+ "License :: OSI Approved :: Apache Software License",
+ "Environment :: GPU :: NVIDIA CUDA",
+ "Topic :: Scientific/Engineering :: Artificial Intelligence",
+ "Topic :: System :: Distributed Computing",
+ ],
+)
diff --git a/tests/test_seq_parallel_attention.py b/tests/test_seq_parallel_attention.py
new file mode 100644
index 0000000..67ce7dc
--- /dev/null
+++ b/tests/test_seq_parallel_attention.py
@@ -0,0 +1,152 @@
+import colossalai
+import torch
+import torch.distributed as dist
+from colossalai.testing import spawn
+
+from opensora.acceleration.communications import gather_forward_split_backward, split_forward_gather_backward
+from opensora.acceleration.parallel_states import set_sequence_parallel_group
+from opensora.models.layers.blocks import (
+ Attention,
+ MultiHeadCrossAttention,
+ SeqParallelAttention,
+ SeqParallelMultiHeadCrossAttention,
+)
+
+
+def run_attention(rank, world_size):
+ # create model
+ torch.manual_seed(1024)
+ set_sequence_parallel_group(dist.group.WORLD)
+
+ seq_parallel_attention = SeqParallelAttention(
+ dim=256,
+ num_heads=4,
+ qkv_bias=True,
+ enable_flashattn=False
+ ).cuda()
+
+ torch.manual_seed(1024)
+ attention = Attention(
+ dim=256,
+ num_heads=4,
+ qkv_bias=True,
+ enable_flashattn=False,
+ ).cuda()
+
+ # create inputs
+ torch.manual_seed(1024)
+ x = torch.randn(4, 64, 256).cuda()
+ seq_x = x.clone().detach()
+
+ x.requires_grad = True
+ x.retain_grad()
+ seq_x.requires_grad = True
+ seq_x.retain_grad()
+
+ sub_seq_x = split_forward_gather_backward(seq_x, dist.group.WORLD, dim=1, grad_scale="down")
+
+ # run model
+ out = attention(x)
+ sub_seq_out = seq_parallel_attention(sub_seq_x)
+ seq_out = gather_forward_split_backward(sub_seq_out, dist.group.WORLD, dim=1, grad_scale="up")
+
+ assert torch.allclose(seq_out, out, atol=1e-7), f"{seq_out}\nvs\n{out}"
+
+ # run backward
+ seq_out.mean().backward()
+ out.mean().backward()
+
+ # all reduce gradient for sp
+ for p in seq_parallel_attention.parameters():
+ if p.grad is not None:
+ dist.all_reduce(p.grad, group=dist.group.WORLD)
+ p.grad.div_(world_size)
+
+ # check grad
+ for p1, p2 in zip(seq_parallel_attention.parameters(), attention.parameters()):
+ assert torch.allclose(p1.grad, p2.grad, atol=1e-7), f"{p1.grad}\nvs\n{p2.grad}"
+
+ # check input grad
+ assert torch.allclose(x.grad, seq_x.grad, atol=1e-7), f"{x.grad}\nvs\n{seq_x.grad}"
+
+
+def run_cross_attention(rank, world_size):
+ # create model
+ torch.manual_seed(1024)
+ set_sequence_parallel_group(dist.group.WORLD)
+ seq_parallel_attention = SeqParallelMultiHeadCrossAttention(
+ d_model=256,
+ num_heads=4,
+ ).cuda()
+
+ torch.manual_seed(1024)
+ attention = MultiHeadCrossAttention(
+ d_model=256,
+ num_heads=4,
+ ).cuda()
+
+ # create inputs
+ torch.manual_seed(1024)
+ x = torch.randn(4, 64, 256).cuda()
+ y = torch.randn(4, 32, 256).cuda()
+
+ mask = [2, 10, 8, 16]
+ mask = None
+ seq_x = x.clone().detach()
+ seq_y = y.clone().detach()
+
+ # set grad
+ x.requires_grad = True
+ x.retain_grad()
+ seq_x.requires_grad = True
+ seq_x.retain_grad()
+ y.requires_grad = True
+ y.retain_grad()
+ seq_y.requires_grad = True
+ seq_y.retain_grad()
+
+ # split by sequence
+ sub_seq_x = split_forward_gather_backward(seq_x, dist.group.WORLD, dim=1, grad_scale="down")
+
+ # run model
+ out = attention(x, y, mask)
+ sub_seq_out = seq_parallel_attention(sub_seq_x, seq_y, mask)
+ seq_out = gather_forward_split_backward(sub_seq_out, dist.group.WORLD, dim=1, grad_scale="up")
+
+ assert torch.allclose(seq_out, out, rtol=1e-5, atol=1e-6), f"\n{seq_out}\nvs\n{out}"
+
+ # run backward
+ seq_out.mean().backward()
+ out.mean().backward()
+
+ # all reduce gradient for sp
+ for name, p in seq_parallel_attention.named_parameters():
+ if p.grad is not None:
+ dist.all_reduce(p.grad, group=dist.group.WORLD)
+ p.grad.div_(world_size)
+ else:
+ print(f"grad of {name} is None")
+
+ # # check grad
+ for p1, p2 in zip(seq_parallel_attention.named_parameters(), attention.named_parameters()):
+ # if not torch.allclose(p1[1].grad, p2[1].grad, atol=1e-7):
+ # print(p1[0], p2[0])
+ assert torch.allclose(p1[1].grad, p2[1].grad, atol=1e-7), f"\n{p1[0]}\nvs\n{p2[0]}:\n{p1[1].grad}\nvs\n{p2[1].grad}"
+
+ # # check input grad
+ assert torch.allclose(x.grad, seq_x.grad, atol=1e-7), f"{x.grad}\nvs\n{seq_x.grad}"
+ assert torch.allclose(y.grad, seq_y.grad, atol=1e-7), f"{y.grad}\nvs\n{seq_y.grad}"
+
+
+
+
+def run_dist(rank, world_size, port):
+ colossalai.launch({}, rank=rank, world_size=world_size, host="localhost", port=port)
+ # run_attention(rank, world_size)
+ run_cross_attention(rank, world_size)
+
+def test_seq_parallel_attention():
+ spawn(run_dist, nprocs=2)
+
+if __name__ == '__main__':
+ test_seq_parallel_attention()
diff --git a/tests/test_t5_shardformer.py b/tests/test_t5_shardformer.py
new file mode 100644
index 0000000..7e32e98
--- /dev/null
+++ b/tests/test_t5_shardformer.py
@@ -0,0 +1,69 @@
+import time
+from copy import deepcopy
+
+import colossalai
+import torch
+from colossalai.shardformer import ShardConfig, ShardFormer
+from colossalai.testing import spawn
+
+from opensora.acceleration.shardformer.policy.t5_encoder import T5EncoderPolicy
+from opensora.models.text_encoder.t5 import T5Embedder
+
+
+def run_t5_encoder(rank, world_size, port):
+ colossalai.launch({}, rank=rank, world_size=world_size, port=port, host="localhost")
+
+ # t5 embedder
+ t5_path = "./pretrained_models/t5_ckpts"
+ hf_t5 = T5Embedder(device="cuda", local_cache=True, cache_dir=t5_path, torch_dtype=torch.float)
+ sf_t5 = deepcopy(hf_t5)
+
+ # create huggingface model as normal
+ shard_config = ShardConfig(tensor_parallel_process_group=None,
+ pipeline_stage_manager=None,
+ enable_tensor_parallelism=False,
+ enable_fused_normalization=False,
+ enable_flash_attention=False,
+ enable_jit_fused=True,
+ enable_sequence_parallelism=False,
+ enable_sequence_overlap=False)
+ shard_former = ShardFormer(shard_config=shard_config)
+ sharded_model, _ = shard_former.optimize(sf_t5.model, policy=T5EncoderPolicy())
+ sf_t5.model = sharded_model
+
+ # test t5 embedder
+ texts = ["Who is the best player in the history of NBA?", "How to study computer science?"]
+ for i in range(5):
+ hf_embs, hf_masks = hf_t5.get_text_embeddings(texts)
+ sf_embs, sf_masks = sf_t5.get_text_embeddings(texts)
+
+ # check accuracy
+ assert torch.allclose(hf_embs, sf_embs, rtol=1e-4, atol=1e-5), f"{hf_embs} \nvs\n{sf_embs}"
+ assert torch.allclose(hf_masks, sf_masks), f"{hf_masks} \nvs\n{sf_masks}"
+
+ # measure perf
+ torch.cuda.synchronize()
+ hf_start = time.time()
+ for i in range(20):
+ hf_embs, hf_masks = hf_t5.get_text_embeddings(texts)
+ torch.cuda.synchronize()
+ hf_end = time.time()
+
+ # convert sf to fp16
+ hf_t5.model = hf_t5.model.half()
+ torch.cuda.synchronize()
+ sf_start = time.time()
+ for i in range(20):
+ hf_embs, hf_masks = hf_t5.get_text_embeddings(texts)
+ torch.cuda.synchronize()
+ sf_end = time.time()
+
+ print(f"[Performance] native: {hf_end - hf_start}s, shardformer: {sf_end - sf_start} s")
+
+
+def test_t5_encoder():
+ spawn(run_t5_encoder)
+
+
+if __name__ == '__main__':
+ test_t5_encoder()