Skip to content

Open-Sora: Democratizing Efficient Video Production for All

Python

Dev/pxy (#26)

xyupengb66a6e39d15f
diff --git a/tools/scoring/README.md b/tools/scoring/README.md
index 1460f3c..d3270e1 100644
--- a/tools/scoring/README.md
+++ b/tools/scoring/README.md
@@ -36,25 +36,27 @@ python -m tools.scoring.aesthetic.inference meta.csv
```
## Optical Flow Score
+Optical flow scores are used to assess the motion of a video. Higher optical flow scores indicate larger movement.
+TODO: acknowledge UniMatch.
First get the pretrained model.
-
```bash
wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth -P pretrained_models/unimatch
```
-With `meta.csv` containing the paths to the videos, run the following command:
-
-```bash
-python -m tools.scoring.optical_flow.inference /path/to/meta.csv
-# or run in parallel
-torchrun --standalone --nproc_per_node 8 -m tools.scoring.optical_flow.inference_parallel /path/to/meta.csv
+Then run:
```
-
-The output should be `/path/to/meta_flow.csv` with column `flow`. Higher optical flow scores indicate larger movement.
+python tools/scoring/optical_flow/inference.py /path/to/meta.csv
+```
+The output should be `/path/to/meta_flow.csv` with column `flow`.
## Matching Score
+Matching scores are calculated to evaluate the alignment between an image/video and its caption.
+For videos, we compute the matching score of the middle frame and the caption.
-Require column `text` in meta files, which is the caption of the sample.
+**Make sure** meta files contain the column `text`, which is the caption of the sample. Then run:
-TODO.
+```
+python tools/scoring/matching/inference.py /path/to/meta.csv
+```
+The output should be `/path/to/meta_match.csv` with column `match`. Higher matching scores indicate better image-text/video-text alignment.
diff --git a/tools/scoring/matching/inference.py b/tools/scoring/matching/inference.py
index 8d4ab65..f162cc2 100644
--- a/tools/scoring/matching/inference.py
+++ b/tools/scoring/matching/inference.py
@@ -6,10 +6,21 @@ import numpy as np
import pandas as pd
import torch
import torch.nn.functional as F
+from torchvision.datasets.folder import pil_loader
+
from tqdm import tqdm
import clip
+IMG_EXTENSIONS = (".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp")
+VID_EXTENSIONS = (".mp4", ".avi", ".mov", ".mkv")
+
+
+def is_video(filename):
+ ext = os.path.splitext(filename)[-1].lower()
+ return ext in VID_EXTENSIONS
+
+
def extract_frames(video_path, points=[0.5]):
container = av.open(video_path)
total_frames = container.streams.video[0].frames
@@ -31,7 +42,13 @@ class VideoTextDataset(torch.utils.data.Dataset):
def __getitem__(self, index):
row = self.meta.iloc[index]
- img = extract_frames(row["path"], points=[0.5])[0]
+ path = row['path']
+
+ if is_video(path):
+ img = extract_frames(path, points=[0.5])[0]
+ else:
+ img = pil_loader(path)
+
img = self.transform(img)
text = row['text']