Open-Sora: Democratizing Efficient Video Production for All
Python
update frame extraction (#31)
Zheng Zangwei (Alex Zheng)1bb49f75e77a
diff --git a/docs/structure.md b/docs/structure.mdindex 9978c33..ac865c9 100644--- a/docs/structure.md+++ b/docs/structure.md@@ -177,3 +177,38 @@ batch_size = 4lr = 2e-5grad_clip = 1.0 # gradient clipping```++## Bucket Configs++To enable dynamic training (for STDiT2), use `VariableVideoText` dataset, and set the `bucket_config` in the config. An example is:++```python+bucket_config = {+ "240p": {16: (1.0, 16), 32: (1.0, 8), 64: (1.0, 4), 128: (1.0, 2)},+ "256": {1: (1.0, 256)},+ "512": {1: (1.0, 80)},+ "480p": {1: (1.0, 52), 16: (0.5, 4), 32: (0.0, None)},+ "720p": {16: (1.0, 2), 32: (0.0, None)},+ "1024": {1: (1.0, 20)},+ "1080p": {1: (1.0, 8)},+}+```++This looks a bit difficult to understand at the first glance. Let's understand this config step by step.++### Why bucket?++Dynamic training needs to support training on different resolution (HxW), different aspect ratio (H/W), and different frame length. There are several possible ways to achieve this:++- NaViT: support dynamic size within the same batch by masking, without efficiency loss. However, the system is a bit complex to implement, and may not benefit from optimized kernels such as flash attention.+- FiT: support dynamic size within the same batch by padding. However, padding different resolutions to the same size is not efficient.+- PixArt: support dynamic size in different batches by bucketing, but the size must be the same within the same batch, and only a fixed number of size can be applied. With the same size in a batch, we do not need to implement complex masking or padding.++Bucketing means we pre-define some fixed resolution, and allocate different samples to different bucket. The concern for bucketing is listed below. But we can see that the concern is not a big issue in our case.++- The bucket size is limited to a fixed number: First, in real-world applications, only a few aspect ratios (9:16, 3:4) and resolutions (240p, 1080p) are commonly used. Second, we find trained models can generalize well to unseen resolutions.+- The size in each batch is the same, breaks the i.i.d. assumption: Since we are using multiple GPUs, the local batches on different GPUs have different sizes. We did not see a significant performance drop due to this issue.+- The may not be enough samples to fill each bucket and the distribution may be biased: First, our dataset is large enough to fill each bucket when local batch size is not too large. Second, we should analyze the data's distribution on sizes and define the bucket size accordingly. Third, an unbalanced distribution did not affect the training process significantly.+- Different resolutions and frame lengths may have different processing speed: Different from PixArt, which only deals with aspect ratios of similar resolutions (similar token numbers), we need to consider the processing speed of different resolutions and frame lengths. We can use the `bucket_config` to define the batch size for each bucket to ensure the processing speed is similar.++### Three-level bucketdiff --git a/tools/datasets/README.md b/tools/datasets/README.mdindex 3774000..158da68 100644--- a/tools/datasets/README.md+++ b/tools/datasets/README.md@@ -6,10 +6,13 @@- [Manage datasets](#manage-datasets)- [Requirement](#requirement)- [Usage](#usage)+ - [Transform datasets](#transform-datasets)+ - [Resize](#resize)+ - [Frame extraction](#frame-extraction)+ - [Crop Midjourney 4 grid](#crop-midjourney-4-grid)- [Analyze datasets](#analyze-datasets)- [Frame extraction speed](#frame-extraction-speed)-After preparing the raw dataset according to the [instructions](/docs/datasets.md), you can use the following commands to manage the dataset.## Dataset Format@@ -183,6 +186,38 @@ python -m tools.datasets.csvutil DATA.csv --aesmin 0.5python -m tools.datasets.csvutil DATA.csv --matchmin 0.5```+## Transform datasets++The `tools.datasets.transform` module provides a set of tools to transform the dataset. The general usage is as follows:++```bash+python -m tools.datasets.transform TRANSFORM_TYPE META.csv ORIGINAL_DATA_FOLDER DATA_FOLDER_TO_SAVE_RESULTS --additional-args+```++### Resize++Sometimes you may need to resize the images or videos to a specific resolution. You can use the following commands to resize the dataset:++```bash+python -m tools.datasets.transform meta.csv /path/to/raw/data /path/to/new/data --length 2160+```++### Frame extraction++To extract frames from videos, you can use the following commands:++```bash+python -m tools.datasets.transform vid_frame_extract meta.csv /path/to/raw/data /path/to/new/data --points 0.1 0.5 0.9+```++### Crop Midjourney 4 grid++Randomly select one of the 4 images in the 4 grid generated by Midjourney.++```bash+python -m tools.datasets.transform img_rand_crop meta.csv /path/to/raw/data /path/to/new/data+```+## Analyze datasetsSince the dataset is provided in a CSV file, you can easily analyze the dataset using pandas (after applying `--info`). Here are some examples:@@ -196,14 +231,6 @@ data.hist(column="resolution")plt.savefig('info.jpg')```-## Resize datasets--Sometimes you may need to resize the images or videos to a specific resolution. You can use the following commands to resize the dataset:--```bash-python -m tools.datasets.resize meta.csv /path/to/raw/data /path/to/new/data --length 2160-```-## Frame extraction speedWe use three libraries to extract frames from videos: `opencv`, `pyav` and `decord`. Our benchmark results of loading 256 video's middle frames are as follows:diff --git a/tools/datasets/csvutil.py b/tools/datasets/csvutil.pyindex 7078257..a93d252 100644--- a/tools/datasets/csvutil.py+++ b/tools/datasets/csvutil.py@@ -20,10 +20,10 @@ except ImportError:pandas_has_parallel = False-def apply(df, func):+def apply(df, func, **kwargs):if pandas_has_parallel:- return df.parallel_apply(func)- return df.progress_apply(func)+ return df.parallel_apply(func, **kwargs)+ return df.progress_apply(func, **kwargs)IMG_EXTENSIONS = (diff --git a/tools/datasets/resize.py b/tools/datasets/resize.pydeleted file mode 100644index 352be53..0000000--- a/tools/datasets/resize.py+++ /dev/null@@ -1,87 +0,0 @@-import argparse-import os--import cv2-import pandas as pd-from tqdm import tqdm--tqdm.pandas()--try:- from pandarallel import pandarallel-- pandarallel.initialize(progress_bar=True)- pandas_has_parallel = True-except ImportError:- pandas_has_parallel = False---def apply(df, func):- if pandas_has_parallel:- return df.parallel_apply(func)- return df.progress_apply(func)---IMG_EXTENSIONS = (- ".jpg",- ".jpeg",- ".png",- ".ppm",- ".bmp",- ".pgm",- ".tif",- ".tiff",- ".webp",-)---def get_new_path(path, input_dir, output):- path_new = os.path.join(output, os.path.relpath(path, input_dir))- os.makedirs(os.path.dirname(path_new), exist_ok=True)- return path_new---def resize(path, length, input_dir, output):- path_new = get_new_path(path, input_dir, output)- ext = os.path.splitext(path)[1].lower()- if ext in IMG_EXTENSIONS:- img = cv2.imread(path)- h, w = img.shape[:2]- if min(h, w) > length:- if h > w:- new_h = length- new_w = int(w * new_h / h)- else:- new_w = length- new_h = int(h * new_w / w)- img = cv2.resize(img, (new_w, new_h))- cv2.imwrite(path_new, img)- else:- pass- return path_new---def main(args):- data = pd.read_csv(args.input)- data["path"] = apply(data["path"], lambda x: resize(x, args.length, args.input_dir, args.output))- output_csv = args.input.replace(".csv", f"_resized{args.length}.csv")- data.to_csv(output_csv, index=False)- print(f"Saved to {output_csv}")---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument("input", type=str)- parser.add_argument("input_dir", type=str)- parser.add_argument("output", type=str)- parser.add_argument("--disable-parallel", action="store_true")- parser.add_argument("--length", type=int, default=2160)- args = parser.parse_args()- return args---if __name__ == "__main__":- args = parse_args()- if args.disable_parallel:- pandas_has_parallel = False- main(args)diff --git a/tools/datasets/transform.py b/tools/datasets/transform.pynew file mode 100644index 0000000..87673ad--- /dev/null+++ b/tools/datasets/transform.py@@ -0,0 +1,147 @@+import argparse+import os+import random++import cv2+import decord+import numpy as np+import pandas as pd+from PIL import Image+from tqdm import tqdm++tqdm.pandas()++try:+ from pandarallel import pandarallel++ pandarallel.initialize(progress_bar=True)+ pandas_has_parallel = True+except ImportError:+ pandas_has_parallel = False+++def apply(df, func, **kwargs):+ if pandas_has_parallel:+ return df.parallel_apply(func, **kwargs)+ return df.progress_apply(func, **kwargs)+++IMG_EXTENSIONS = (+ ".jpg",+ ".jpeg",+ ".png",+ ".ppm",+ ".bmp",+ ".pgm",+ ".tif",+ ".tiff",+ ".webp",+)+++def get_new_path(path, input_dir, output):+ path_new = os.path.join(output, os.path.relpath(path, input_dir))+ os.makedirs(os.path.dirname(path_new), exist_ok=True)+ return path_new+++def resize(path, length, input_dir, output):+ path_new = get_new_path(path, input_dir, output)+ ext = os.path.splitext(path)[1].lower()+ assert ext in IMG_EXTENSIONS+ img = cv2.imread(path)+ h, w = img.shape[:2]+ if min(h, w) > length:+ if h > w:+ new_h = length+ new_w = int(w * new_h / h)+ else:+ new_w = length+ new_h = int(h * new_w / w)+ img = cv2.resize(img, (new_w, new_h))+ cv2.imwrite(path_new, img)+ return path_new+++def rand_crop(path, input_dir, output):+ ext = os.path.splitext(path)[1].lower()+ path_new = get_new_path(path, input_dir, output)+ assert ext in IMG_EXTENSIONS+ img = cv2.imread(path)+ h, w = img.shape[:2]+ width, height, _ = img.shape+ pos = random.randint(0, 3)+ if pos == 0:+ img_cropped = img[: width // 2, : height // 2]+ elif pos == 1:+ img_cropped = img[width // 2 :, : height // 2]+ elif pos == 2:+ img_cropped = img[: width // 2, height // 2 :]+ else:+ img_cropped = img[width // 2 :, height // 2 :]+ cv2.imwrite(path_new, img_cropped)+ return path_new+++def extract_frames(video_path, input_dir, output, point):+ point = round(point)+ points = [point]+ path_new = get_new_path(video_path, input_dir, output)++ container = decord.VideoReader(video_path, num_threads=1)+ total_frames = len(container)+ frame_inds = np.array(points).astype(np.int32)+ frame_inds[frame_inds >= total_frames] = total_frames - 1+ frames = container.get_batch(frame_inds).asnumpy()+ frames_pil = Image.fromarray(frames[0])++ os.makedirs(path_new, exist_ok=True)+ path_new = os.path.join(path_new, f"{point}.jpg")+ frames_pil.save(path_new)+ return path_new+++def main(args):+ data = pd.read_csv(args.input)+ if args.method == "img_rand_crop":+ data["path"] = apply(data["path"], lambda x: rand_crop(x, args.input_dir, args.output))+ elif args.method == "img_resize":+ data["path"] = apply(data["path"], lambda x: resize(x, args.length, args.input_dir, args.output))+ elif args.method == "vid_frame_extract":+ points = args.points if args.points is not None else args.points_index+ data = pd.DataFrame(np.repeat(data.values, 3, axis=0), columns=data.columns)+ num_points = len(points)+ data["point"] = np.nan+ for i, point in enumerate(points):+ if isinstance(point, int):+ data.loc[i::num_points, "point"] = point+ else:+ data.loc[i::num_points, "point"] = data.loc[i::num_points, "num_frames"] * point+ data["path"] = apply(data, lambda x: extract_frames(x["path"], args.input_dir, args.output, x["point"]), axis=1)++ output_csv = args.input.replace(".csv", f"_resized{args.length}.csv")+ data.to_csv(output_csv, index=False)+ print(f"Saved to {output_csv}")+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("method", type=str, choices=["img_resize", "img_rand_crop", "vid_frame_extract"])+ parser.add_argument("input", type=str)+ parser.add_argument("input_dir", type=str)+ parser.add_argument("output", type=str)+ parser.add_argument("--disable-parallel", action="store_true")+ parser.add_argument("--length", type=int, default=2160)+ parser.add_argument("--seed", type=int, default=42, help="seed for random")+ parser.add_argument("--points", nargs="+", type=float, default=None)+ parser.add_argument("--points_index", nargs="+", type=int, default=None)+ args = parser.parse_args()+ return args+++if __name__ == "__main__":+ args = parse_args()+ random.seed(args.seed)+ if args.disable_parallel:+ pandas_has_parallel = False+ main(args)