Open-Sora: Democratizing Efficient Video Production for All
Python
Dev/pxy (#26)
xyupengb66a6e39d15f
diff --git a/tools/scoring/README.md b/tools/scoring/README.mdindex 1460f3c..d3270e1 100644--- a/tools/scoring/README.md+++ b/tools/scoring/README.md@@ -36,25 +36,27 @@ python -m tools.scoring.aesthetic.inference meta.csv```## Optical Flow Score+Optical flow scores are used to assess the motion of a video. Higher optical flow scores indicate larger movement.+TODO: acknowledge UniMatch.First get the pretrained model.-```bashwget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P pretrained_models/unimatch```-With `meta.csv` containing the paths to the videos, run the following command:--```bash-python -m tools.scoring.optical_flow.inference /path/to/meta.csv-# or run in parallel-torchrun --standalone --nproc_per_node 8 -m tools.scoring.optical_flow.inference_parallel /path/to/meta.csv+Then run:```--The output should be `/path/to/meta_flow.csv` with column `flow`. Higher optical flow scores indicate larger movement.+python tools/scoring/optical_flow/inference.py /path/to/meta.csv+```+The output should be `/path/to/meta_flow.csv` with column `flow`.## Matching Score+Matching scores are calculated to evaluate the alignment between an image/video and its caption.+For videos, we compute the matching score of the middle frame and the caption.-Require column `text` in meta files, which is the caption of the sample.+**Make sure** meta files contain the column `text`, which is the caption of the sample. Then run:-TODO.+```+python tools/scoring/matching/inference.py /path/to/meta.csv+```+The output should be `/path/to/meta_match.csv` with column `match`. Higher matching scores indicate better image-text/video-text alignment.diff --git a/tools/scoring/matching/inference.py b/tools/scoring/matching/inference.pyindex 8d4ab65..f162cc2 100644--- a/tools/scoring/matching/inference.py+++ b/tools/scoring/matching/inference.py@@ -6,10 +6,21 @@ import numpy as npimport pandas as pdimport torchimport torch.nn.functional as F+from torchvision.datasets.folder import pil_loader+from tqdm import tqdmimport clip+IMG_EXTENSIONS = (".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp")+VID_EXTENSIONS = (".mp4", ".avi", ".mov", ".mkv")+++def is_video(filename):+ ext = os.path.splitext(filename)[-1].lower()+ return ext in VID_EXTENSIONS++def extract_frames(video_path, points=[0.5]):container = av.open(video_path)total_frames = container.streams.video[0].frames@@ -31,7 +42,13 @@ class VideoTextDataset(torch.utils.data.Dataset):def __getitem__(self, index):row = self.meta.iloc[index]- img = extract_frames(row["path"], points=[0.5])[0]+ path = row['path']++ if is_video(path):+ img = extract_frames(path, points=[0.5])[0]+ else:+ img = pil_loader(path)+img = self.transform(img)text = row['text']