Open-Sora: Democratizing Efficient Video Production for All
Python
Dev/v1.0.1 (#58)
Zheng Zangwei (Alex Zheng)341b12f9bf44
diff --git a/.gitignore b/.gitignoreindex a64a359..2b9e525 100644--- a/.gitignore+++ b/.gitignore@@ -173,6 +173,7 @@ samples/sampleslogs/pretrained_models/+evaluation_results/*.swp# Secret filesdiff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex d606846..2acbec4 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -8,7 +8,6 @@ To contribute to Open-Sora, we would like to first guide you to set up a properYou can refer to the [Installation Section](./README.md#installation) and replace `pip install -v .` with `pip install -v -e .`.-### Code StyleWe have some static checks when you commit your code change, please make sure you can pass all the tests and make sure the coding style meets our requirements. We use pre-commit hook to make sure the code is aligned with the writing standard. To set up the code style checking, you need to follow the steps below.@@ -21,7 +20,6 @@ pre-commit installCode format checking will be automatically executed when you commit your changes.-## Contribution GuideYou need to follow these steps below to make contribution to the main repository via pull request. You can learn about the details of pull request [here](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests).@@ -68,7 +66,6 @@ git push origin mainYou should not make changes to the `main` branch of your forked repository as this might make upstream synchronization difficult. You can create a new branch with the appropriate name. General branch name format should start with `hotfix/` and `feature/`. `hotfix` is for bug fix and `feature` is for addition of a new feature.-```shellgit checkout -b <NEW-BRANCH-NAME>```@@ -89,3 +86,15 @@ git push -u origin <NEW-BRANCH-NAME>You can now create a pull request on the GitHub webpage of your repository. The source branch is `<NEW-BRANCH-NAME>` of your repository and the target branch should be `main` of `hpcaitech/Open-Sora`. After creating this pull request, you should be able to see it [here](https://github.com/hpcaitech/Open-Sora/pulls).The Open-Sora team will review your code change and merge your code if applicable.++## FQA++1. `pylint` cannot recognize some members:++Add this into your `settings.json` in VSCode:++```json+"pylint.args": [+ "--generated-members=numpy.* ,torch.*,cv2.*",+],+```diff --git a/assets/texts/VBench/all_category.txt b/assets/texts/VBench/all_category.txtnew file mode 100644index 0000000..fd797ed--- /dev/null+++ b/assets/texts/VBench/all_category.txt@@ -0,0 +1,800 @@+a black dog wearing halloween costume+spider making a web+bat eating fruits while hanging+a snake crawling on a wooden flooring+a close up video of a dragonfly+macro shot of ladybug on green leaf plant+chameleon eating ant+a bee feeding on nectars+bird nests on a tree captured with moving camera+a squirrel eating nuts+close up video of snail+top view of a hermit crab crawling on a wooden surface+cat licking another cat+red dragonfly perched on green leaf+close up view of a brown caterpillar crawling on green leaf+ants eating dead spider+an eagle on a tree branch+a frog eating an ant+white rabbit near the fence+a gorilla eating a carrot+close up of wolf+a meerkat looking around+a hyena in a zoo+lemur eating grass leaves+an owl being trained by a man+a lizard on a bamboo+brown chicken hunting for its food+video of parrots perched on bird stand+underwater footage of an octopus in a coral reef+a cute pomeranian dog playing with a soccer ball+white fox on rock+close up footage of a horse figurine+giraffe feeding on a tree in a savannah+curious cat sitting and looking around+hummingbird hawk moth flying near pink flowers+close up of a scorpion on a rock+close up on fish in net+koala eating leaves from a branch+a pod of dolphins swirling in the sea catching forage fish+low angle view of a hawk perched on a tree branch+a lion standing on wild grass+deer grazing in the field+elephant herd in a savanna+close up on lobster under water+hedgehog crossing road in forest+a sheep eating yellow flowers from behind a wire fence+twin sisters and a turtle+a pig wallowing in mud+flock of goose eating on the lake water+cow in a field irritated with flies+a close up shot of a fly+cheetah lying on the grass+close up of a lemur+close up shot of a kangaroo itching in the sand+a tortoise covered with algae+turkey in cage+a great blue heron bird in the lakeside+crab with shell in aquarium+a seagull walking on shore+an american crocodile+a tiger walking inside a cage+alligator in the nature+a raccoon climbing a tree+wild rabbit in a green meadow+group of ring tailed lemurs+a clouded leopard on a tree branch+duck grooming its feathers+an african penguin walking on a beach+a video of a peacock+close up shot of a wild bear+baby rhino plays with mom+porcupine climbs tree branches+close up of a natterjack toad on a rock+a sleeping orangutan+mother whale swimming with babies+a bear wearing red jersey+pink jellyfish swimming underwater in a blue sea+beautiful clown fish swimming+animation of disposable objects shaped as a whale+paper cut out of a pair of hands a whale and a heart+vertical video of camel roaming in the field during daytime+a still video of mosquito biting human+a curious sloth hanging from a tree branch+a plastic flamingo bird stumbles from the wind+a wolf in its natural habitat+a monkey sitting in the stone and scratching his head+bat hanging upside down+a red panda eating leaves+snake on ground+a harbour seal swimming near the shore+shark swimming in the sea+otter on branch while eating+goat standing over a rock+a troop of monkey on top of a mountain+a zebra eating grass on the field+a colorful butterfly perching on a bud+a snail crawling on a leaf+zookeeper showering a baby elephant+a beetle emerging from the sand+a nine banded armadillo searching for food+an apartment building with balcony+asian garden and medieval castle+illuminated tower in berlin+a wooden house overseeing the lake+a crowd of people in a plaza in front of a government building+a church interior+jewish friends posing with hanukkah menorah in a cabin house+a destroyed building after a missile attack in ukraine+abandoned building in the woods+drone video of an abandoned school building in pripyat ukraine+elegant university building+architecture and designs of buildings in central london+a pancake tower with chocolate syrup and strawberries on top+an ancient white building+friends hanging out at a coffee house+house front door with christmas decorations+city night dark building+a bird house hanging on a tree branch+sacred sculpture in a temple+high angle shot of a clock tower+modern wooden house interior+the interior of an abandoned building+opera house overlooking sea+a concrete structure near the green trees+dome like building in scotland+low angle shot of a building+tower on hill+a miniature house+eiffel tower from the seine river+low angle footage of an apartment building+island with pier and antique building+asian historic architecture+drone footage of a beautiful mansion+mosque in the middle east+building a tent and hammock in the forest camping site+top view of a high rise building+house covered in snow+skyscraper at night+house in village+a casino with people outside the building+silhouette of a building+a woman climbing a tree house+drone view of house near lake during golden hour+an under construction concrete house+a watch tower by the sea+exterior view of arabic style building+video of a hotel building+red paper lantern decorations hanging outside a building+house on seashore+aerial footage of the palace of culture and science building in warsaw poland+aerial video of stuttgart tv tower in germany+aerial view of the highway and building in a city+drone shot of a skyscraper san francisco california usa+waterfall and house+view of the sky through a building+drone footage of a house on top of the mountain+abandoned house in the nature+clouds hovering over a mansion+light house on the ocean+buddhist temple at sunrise+people walking by a graveyard near a mosque at sunset+view of lifeguard tower on the beach+scenic view of a house in the mountains+the landscape in front of a government building+aerial footage of a building and its surrounding landscape in winter+time lapse of a cloudy sky behind a transmission tower+blue ocean near the brown castle+fog over temple+house in countryside top view+building under construction+turkish flag waving on old tower+the georgian building+close up shot of a steel structure+the atrium and interior design of a multi floor building+city view reflected on a glass building+aerial view of a luxurious house with pool+an unpaved road leading to the house+drone footage of a lookout tower in mountain landscape+wind turbines on hill behind building+time lapse footage of the sun light in front of a small house porch+a building built with lots of stairways+overcast over house on seashore+the view of the sydney opera house from the other side of the harbor+candle on a jar and a house figurine on a surface+video of a farm and house+a dilapidated building made of bricks+a view of a unique building from a moving vehicle+aerial footage of a tall building in cambodia+push in shot of a huge house+a beach house built over a seawall protected from the sea waves+exotic house surrounded by trees+drone video of a house surrounded by tropical vegetation+drone footage of a building beside a pond+observation tower on hill in forest+a tree house in the woods+a video of vessel structure during daytime+fire in front of illuminated building at night+a footage of a wooden house on a wheat field+tilt shot of a solar panel below a light tower+water tower on the desert+freshly baked finger looking cookies+video of fake blood in wine glass+halloween food art+a person slicing a vegetable+a serving of pumpkin dish in a plate+close up view of green leafy vegetable+a birthday cake in the plate+video of a slice papaya fruit+a muffin with a burning candle and a love sign by a ceramic mug+a jack o lantern designed cookie+baked bread with chocolate+a broccoli soup on wooden table+a freshly brewed coffee on a pink mug+grabbing sourdough neapolitan style pizza slices+person cooking mushrooms in frying pan+rice grains placed on a reusable cloth bag+slices of kiwi fruit+grilling a steak on a pan grill+close up of bread popping out of a toaster+man eating noodle+preparing a cocktail drink+close up pasta with bacon on plate+milk and cinnamon rolls+boy getting a dumpling using chopsticks+a mother preparing food with her kids+man using his phone while eating+fresh salmon salad on a plate+cutting cucumbers into long thin slices as ingredient for sushi roll+a steaming cup of tea by the window+a glass filled with beer+a kid eating popcorn while watching tv+close up shot of fried fish on the plate+a man eating a donut+person making a vegetarian dish+spreading cheese on bagel+close up view of a man drinking red wine+a couple having breakfast in a restaurant+a student eating her sandwich+girl peeling a banana+red rice in a small bowl+pancake with blueberry on the top+green apple fruit on white wooden table+a man eating a taco by the bar+making of a burrito+squeezing lemon into salad+a chef cutting sushi rolls+video of a delicious dessert+deep frying a crab on a wok in high fire+close up video of a orange juice+video of a cooked chicken breast+woman holding a pineapple+a woman eating a bar of chocolate+decorating christmas cookie+squeezing a slice of fruit+tuna sashimi on a plate+a strawberry fruit mixed in an alcoholic drink+preparing hot dogs in a grill+a woman cutting a tomato+an orange fruit cut in half+a coconut fruit with drinking straw+woman holding a dragon fruit+a woman pouring hot beverage on a cup+waffles with whipped cream and fruit+focus shot of an insect at the bottom of a fruit+preparing a healthy broccoli dish+man eating snack at picnic+close up video of a grilled shrimp skewer+a woman mixing a smoothie drinks+close up video of woman having a bite of jelly+businessman drinking whiskey at the bar counter of a hotel lounge+cutting an onion with a knife over a wooden chopping board+fresh lemonade in bottles+grilling a meat on a charcoal grill+people enjoying asian cuisine+close up footage of a hot dish on a clay pot+pork ribs dish+waffle with strawberry and syrup for breakfast+tofu dish with rose garnish+uncooked pork meat+egg yolk being dumped over gourmet dish+tasty brunch dish close up+little boy pretending to eat the watermelon+slicing roasted beef+close up of a chef adding teriyaki sauce to a dish+flat lay mexican dish+a person placing an octopus dish on a marble surface+close up of tea leaves brewing in a glass kettle+adding fresh herbs to soup dish+a scoop of roasted coffee beans+fresh dim sum set up on a bamboo steam tray for cooking+a girl putting ketchup on food at the kitchen+cooking on electric stove+a woman with a slice of a pie+grapes and wine on a wooden board+man taking picture of his food+hamburger and fries on restaurant table+close up video of japanese food+a cracker sandwich with cheese filling for snack+barista preparing matcha tea+close up of onion rings being deep fried+people carving a pumpkin+people sitting on a sofa+a man with a muertos face painting+man walking in the dark+men in front of their computer editing photos+men loading christmas tree on tow truck+woman washing the dishes+woman adding honey to the cinnamon rolls+two women kissing and smiling+three women looking at watercolor paintings+a family wearing paper bag masks+a family posing for the camera+a boy covering a rose flower with a dome glass+boy sitting on grass petting a dog+a girl in her tennis sportswear+a girl coloring the cardboard+silhouette of the couple during sunset+couple dancing with body paint+a child playing with water+a woman with her child sitting on a couch in the living room+a group of friend place doing hand gestures of agreement+friends having a group selfie+friends talking while on the basketball court+group of people protesting+a group of campers with a cute dog+a group of photographers taking pictures at the north western gardens in llandudno north wales+a group of students laughing and talking+a group of martial artist warming up+a person playing golf+a person walking on a wet wooden bridge+person doing a leg exercise+ice hockey athlete on rink+a young athlete training in swimming+chess player dusting a chessboard+baseball player holding his bat+a bearded man putting a vinyl record on a vinyl player+an orchestra finishes a performance+people applauding the performance of the kids+band performance at the recording studio+father and his children playing jenga game+people playing a board game+man playing a video game+a man video recording the movie in theater+man and a woman eating while watching a movie+movie crew talking together+a director explaining the movie scene+man and woman listening to music on car+man playing music+couple dancing slow dance with sun glare+a ballerina practicing in the dance studio+father and son holding hands+father and daughter talking together+a mother and her kids engaged in a video call+mother and daughter reading a book together+a mother teaching her daughter playing a violin+kid in a halloween costume+a happy kid playing the ukulele+a chef slicing a cucumber+chef wearing his gloves properly+brother and sister using hammock+girl applying sunblock to her brother+a girl pushing the chair while her sister is on the chair+colleagues talking in office building+fighter practice kicking+a woman fighter in her cosplay costume+an engineer holding blueprints while talking with her colleague+a young woman looking at vr controllers with her friend+workmates teasing a colleague in the work+a male police officer talking on the radio+teacher holding a marker while talking+teacher writing on her notebook+a young student attending her online classes+a student showing his classmates his wand+a male vendor selling fruits+a shirtless male climber+a sound engineer listening to music+female talking to a psychiatrist in a therapy session+young female activist posing with flag+a man in a hoodie and woman with a red bandana talking to each other and smiling+a medium close up of women wearing kimonos+a male interviewer listening to a person talking+a social worker having a conversation with the foster parents+a farm worker harvesting onions+worker packing street food+worker and client at barber shop+elderly man lifting kettlebell+mom assisting son in riding a bicycle+dad watching her daughter eat+young guy with vr headset+pregnant woman exercising with trainer+a fortune teller talking to a client+wizard doing a ritual on a woman+a footage of an actor on a movie scene+a man holding a best actor trophy+a singer of a music band+a young singer performing on stage+young dancer practicing at home+seller showing room to a couple+cab driver talking to passenger+a policeman talking to the car driver+kids celebrating halloween at home+little boy helping mother in kitchen+video of a indoor green plant+a girl arranges a christmas garland hanging by the kitchen cabinet+candle burning in dark room+couple having fun and goofing around the bedroom+girls jumping up and down in the bedroom+woman and man in pajamas working from home+a muslim family sitting and talking in the living room+family enjoying snack time while sitting in the living room+woman holding an animal puppet and a little girl playing together at the living room+kids playing in the indoor tent+young people celebrating new year at the office+a woman writing on the sticky note in the office+a woman exercising at home over a yoga mat+girls preparing easter decorations at home+dog on floor in room+turning on a fluorescent light inside a room+colleagues talking to each other near the office windows+a woman recording herself while exercising at home+music room+different kind of tools kept in a utility room+sofa beds and other furniture+a girl finding her brother reading a book in the bedroom+an elegant ceramic plant pot and hanging plant on indoor+furniture inside a bedroom+interior design of the bar section+living room with party decoration+firewood burning in dark room+a young woman playing the ukulele at home+woman painting at home+a woman in a locker room+video of a bathroom interior+the interior design of a jewish synagogue+a woman in protective suit disinfecting the kitchen+modern minimalist home interior+modern interior design of a coffee shop+person arranging minimalist furniture+aerial shot of interior of the warehouse+a room of a manufacturing facility+interior of catholic+interior design of a restaurant+a female model in a changing room looking herself in mirror+men walking in the office hallway+people sitting in a conference room+the interior design of a shopping mall+chandeliers in room+lucerne railway station interior+a female fencer posing in a foggy room+a toolbox and a paint roller beside a huge package in a room+bedroom in hotel+a woman lying in the operating room+a chef holding and checking kitchen utensils+a couple singing in the shower room together+a woman cleaning mess in the living room+an empty meeting room with natural light+person dancing in a dark room+close up on blood in hospital room+a couple resting on their home floor+a young female staff at courier office+a man entering the gym locker room+a bored man sitting by the tv at home+woman dancing in indoor garden+rubble in the interior of an abandoned house+indoor farm in a greenhouse+man doing handstand in indoor garden+an abandoned indoor swimming pool+home decorations on top of a cabinet+graffiti art on the interior walls of an abandoned mansion+indoor wall climbing activity+sunlight inside a room+teenage girl roller skating at indoor rink+home deco with lighted+baby in the shower room+men enjoying office christmas party+a bedroom with a brick wall+actors prepping in the dressing room+kids playing at an indoor playground+a person sanitizing an office space using smoke machine+mother and daughter choosing clothes at home+a woman sitting by the indoor fire pit+man standing on the corner of the room while looking around+person assembling furniture+a family stacking cardboard boxes in a room+family having fun in the dining room+person disinfecting a room+a woman washing strawberries in the kitchen sink+modern office waiting room+close up view of a person slicing with a kitchen knife+boiling coffee on a stove in the kitchen+modern equipment used in a home studio+interior of a recording studio+people working in a call center office+band performing at a home concert+a group of people watching a concert in a room+people packing their furniture+young employees in office holding a certificate+a criminal inside a dark room handcuffed in a table+couple browsing and looking for furniture in the store+workspace at home+video of a indoor green plant+close up view of a plant+close up shot of a burning plant+plucking leaves from plant+a plant on gold pot with glass lid+a branch of a tree and a plant+a leafless tree+close up shot of fern leaf+close up video of strawberry plant+plant with blooming flowers+close up video of flower petals+watering yellow plant+beautiful flower decoration+cannabis flower in a jar+a footage of the tree leaves+a red leaf plant+close up view of a white christmas tree+snow pouring on a tree+close up shot of white flowers on the tree+leaves in the trees daytime+a dead tree lying on a grass field+tree branches in a flowing river+purple flowers with leaves+a coconut tree by the house+close up on flower in winter+bamboo leaves backlit by the sun+close up video of a wet flower+a man putting a flower in a box+dropping flower petals on a wooden bowl+a close up shot of gypsophila flower+variety of succulent plants on a garden+variety of trees and plants in a botanical garden+forest of deciduous trees+a stack of dried leaves burning in a forest+tall forest trees on a misty morning+close up view of dewdrops on a leaf+close up view of white petaled flower+removing a pineapple leaf+a dragonfly perched on a leaf+butterfly pollinating flower+person visiting and checking a corn plant+woman picking beans from a plant+woman plucking mint leaves+single tree in the middle of farmland+a plant on a soil+drone footage of a tree on farm field+a tractor harvesting lavender flower+people putting christmas ornaments on a christmas tree+jack o lantern hanging on a tree+tree with halloween decoration+flower field near the waterfall+truck carrying the tree logs+raindrops falling on leaves+shot of a palm tree swaying with the wind+squirrels on a tree branch+person holding a flower+a fallen tree trunk+tree with golden leaves+cherry tree+wind blows through leaves of the tree in autumn+a leaf on a glass+the long trunks of tall trees in the forest+trees in the forest during sunny day+close up video of tree bark+reflection of tree branches+trunks of many trees in the forest+tree leaves providing shades from the sun+leaves swaying in the wind+low angle shot of baobab tree+bare trees in forest+a plant surrounded by fallen leaves+a couple preparing food and pruning a plant+a man cutting a tree bark+oranges on a tree branch+plant connected on the stones+video of a sawmill machine cutting tree log+women drying flower petals+macro view of an agave plant+a video of a person tying a plant on a string+green moss in forest nature+coconut tree near sea under blue sky+the canopy of a coconut tree+a man leaning on a tree at the beach+a full grown plant on a pot+candle wax dripping on flower petals+close up of leaves in autumn+a woman opening a book with a flower inside+a man holding leaves looking at the camera+a shadow of a swaying plant+a tree and concrete structure under a blue and cloudy sky+trimming excess leaves on a potted plant+the changing color of the tree leaves during autumn season+a gooseberry tree swayed by the wind+forest trees and a medieval castle at sunset+woman cut down tree+an old oak tree in a park across the street from a hotel+wild flowers growing in a forest ground+a mossy fountain and green plants in a botanical garden+mansion with beautiful garden+ants on a dragon fruit flower+scenery of desert landscape+landscape agriculture farm tractor+burning slash piles in the forest+graveyard at sunset+view of a jack o lantern with pumpkins in a smoky garden+sun view through a spider web+view of the sea from an abandoned building+close up view of a full moon+close up view of lighted candles+close up view of swaying white flowers and leaves+scenery of a relaxing beach+selective focus video of grass during sunny day+aerial view of brown dry landscape+fireworks display in the sky at night+a bonfire near river+mountain view+waterfalls in between mountain+a picturesque view of nature+exotic view of a riverfront city+tall trees in the forest under the clear sky+snow on branches in forest+stream in the nature+an airplane flying above the sea of clouds+scenic video of sunset+view of houses with bush fence under a blue and cloudy sky+scenic view from wooden pathway+scenic view of a tropical beach+drone footage of waves crashing on beach shore+a scenic view of the golden hour at norway+time lapse video of foggy mountain forest+brown mountain during fall season+video of ocean during daytime+boat sailing in the ocean+top view of yachts+beautiful scenery of flowing waterfalls and river+wild ducks paddling on the lake surface+a relaxing scenery of beach view under cloudy sky+natural rock formations on beach under cloudy sky+a palm tree against blue sky+video of sailboat on a lake during sunset+aerial view of snow piles+time lapse of a sunset sky in the countryside+aerial footage of a statue+time lapse video of a farm during sunset+clouds formation in the sky at sunset+aerial shot of a village+drone shot of a beautiful sunrise at the mountains+time lapse video of foggy morning during sunrise+sun shining between tree leaves at sunrise+video of lake during dawn+vehicles traveling on roadway under cloudy sky+view of golden domed church+a monument under the blue sky+firecrackers in the sky+view of fruit signage in the farm+a dark clouds over shadowing the full moon+view of the amazon river+a big river swamp in a dense forest+a blooming cherry blossom tree under a blue sky with white clouds+a river waterfall cascading down the plunge basin+flooded landscape with palm trees+a blurry waterfall background+waterfall in the mountains+aerial footage of a city at night+pond by small waterfall in forest+aerial view of farmlands at the bay of lake+rice terraces in the countryside+a highway built across an agricultural area in the countryside+gloomy morning in the countryside+drone shot of an abandoned coliseum on a snowy mountain top+boat sailing in the middle of ocean+drone shot of the grass field+natural landscape of mountain and sea with islets developed into a community+aerial view of zaporizhia in ukraine+aerial footage of a herd+an aerial footage of a red sky+grass and plants growing in the remains of an abandoned house+view from hill on city+aerial view on orthodox church+aerial view of bay in croatia+a footage of a frozen river+overlooking view of a city at daylight+view outside the cemetery+clear sky with moon over meadow+clouds over railway+aerial footage of moving vehicles on the road at night+aerial view of town and park+top view of skyscrapers+top view of the empire state building in manhattan+top view of the central park in new york city+sheep running in a grass field+clear sky over factory+smoke and fire in birds eye view+view of a pathway with snow melting on its side+ferry under bridge on river near city in malaysia+mountain slopes covered in green vegetation+panoramic view of a town surrounded by snow covered mountains+aerial view of a palace+top view of vehicles driving on the intersection+a graveyard by a church in a mountain landscape+a modern railway station in malaysia use for public transportation+drone footage of amsterdam metro station+train arriving at a station+red vehicle driving on field+close up view of flashing emergency vehicle lighting+vehicle with fertilizer on field+a highway built across an agricultural area in the countryside+drone footage of motorcycles driving on country road between agricultural fields+a road in the woods under fog+footage of a car driving through a wheat field+vehicle stops for an ambulance passing through city traffic+emergency vehicle parked outside the casino+zombies attacking a woman and a boy inside a car+woman seating inside the car while chewing+video of passengers riding a double decker bus during night+traffic in london street at night+elderly couple checking engine of automobile+a green vintage automobile with an open hood parked in a parking area+close up of a prototype automobile with exposed engine on the back seat of the car+aerial view of road in forest+train departing from station+aerial view of a train passing by a bridge+video of a train tracks+video footage of a subway+video of blinking traffic lights+couple walking out on the subway+time lapse of a subway tunnel+monitor board inside the subway+metro train at night+zoom in video of a tram passing by city+young man using laptop in the tram+man reading a book at bus stop+close up shot of a moving taxi+night travel in london street on a public bus+red bus in a rainy city+flow of traffic in the city+close up shot of a yellow taxi turning left+two women calling for a taxi+drone view of an illuminated bridge across a river+policeman in police car talking on radio+airplane taking off at night+view through window in airplane+an airplane in the sky+helicopter landing on the street+a pilot getting out of a helicopter+a helicopter flying under blue sky+boat sailing in the middle of the ocean+girl playing with a toy boat+silhouette of a boat on sea during golden hour+a boat travelling around the lake+road on mountain ridge+ship sailing on danube river+slow motion video of a ship water trail in the sea+drone footage of a wreck ship on shore+a white yacht traveling on a river and passing under the bridge+female teenagers drinking champagne in the yacht+video of yacht sailing in the ocean+red combine harvester on road on field+a woman sitting on a bicycle while using a mobile phone+a woman sitting on a motorcycle looking around+three teenagers fixing a bicycle+a woman in a halloween costume posing on a motorcycle+a parked motorcycle on a foggy roadside+cable car near sea shore+a truck travelling in the road+footage of the road without any traffic+a road sign+love padlocks on a bridge+camera moving at highway construction site+vehicles driving on highway+a motorbike on highway at timelapse mode+point of view of a car driving through a tunnel+time lapse of heavy traffic on an avenue+ferry boat on city canal+black vintage car in museum+a zigzag road across a forest+people crossing the road+video of a kayak boat in a river+a person paddling a wooden boat in a lake+a car charging in the parking area+cars parked on the road+footage of the street with people and vehicle passing by in the rain+traffic on busy city street+a woman getting out of the car to walk with their dog+yacht sailing through the ocean+people in queue to military ship+man wearing motorcycle helmet looking at the camera+empty seats in the bus+empty boat on the water+cargo train traveling on the mountainside+cruise ship in harbor+counting down at traffic lights+pressing the car ignition+fire truck driving on the road+a footage of a broken bicycle+drone footage of an ambulance on the road+slow motion footage of a racing car+ship sailing on sea against sunset+big cargo ship passing on the shore+back view of man and woman walking on unpaved roaddiff --git a/assets/texts/VBench/all_dimension.txt b/assets/texts/VBench/all_dimension.txtnew file mode 100644index 0000000..f26fbf8--- /dev/null+++ b/assets/texts/VBench/all_dimension.txt@@ -0,0 +1,946 @@+In a still frame, a stop sign+a toilet, frozen in time+a laptop, frozen in time+A tranquil tableau of alley+A tranquil tableau of bar+A tranquil tableau of barn+A tranquil tableau of bathroom+A tranquil tableau of bedroom+A tranquil tableau of cliff+In a still frame, courtyard+In a still frame, gas station+A tranquil tableau of house+indoor gymnasium, frozen in time+A tranquil tableau of indoor library+A tranquil tableau of kitchen+A tranquil tableau of palace+In a still frame, parking lot+In a still frame, phone booth+A tranquil tableau of restaurant+A tranquil tableau of tower+A tranquil tableau of a bowl+A tranquil tableau of an apple+A tranquil tableau of a bench+A tranquil tableau of a bed+A tranquil tableau of a chair+A tranquil tableau of a cup+A tranquil tableau of a dining table+In a still frame, a pear+A tranquil tableau of a bunch of grapes+A tranquil tableau of a bowl on the kitchen counter+A tranquil tableau of a beautiful, handcrafted ceramic bowl+A tranquil tableau of an antique bowl+A tranquil tableau of an exquisite mahogany dining table+A tranquil tableau of a wooden bench in the park+A tranquil tableau of a beautiful wrought-iron bench surrounded by blooming flowers+In a still frame, a park bench with a view of the lake+A tranquil tableau of a vintage rocking chair was placed on the porch+A tranquil tableau of the jail cell was small and dimly lit, with cold, steel bars+A tranquil tableau of the phone booth was tucked away in a quiet alley+a dilapidated phone booth stood as a relic of a bygone era on the sidewalk, frozen in time+A tranquil tableau of the old red barn stood weathered and iconic against the backdrop of the countryside+A tranquil tableau of a picturesque barn was painted a warm shade of red and nestled in a picturesque meadow+In a still frame, within the desolate desert, an oasis unfolded, characterized by the stoic presence of palm trees and a motionless, glassy pool of water+In a still frame, the Parthenon's majestic Doric columns stand in serene solitude atop the Acropolis, framed by the tranquil Athenian landscape+In a still frame, the Temple of Hephaestus, with its timeless Doric grace, stands stoically against the backdrop of a quiet Athens+In a still frame, the ornate Victorian streetlamp stands solemnly, adorned with intricate ironwork and stained glass panels+A tranquil tableau of the Stonehenge presented itself as an enigmatic puzzle, each colossal stone meticulously placed against the backdrop of tranquility+In a still frame, in the vast desert, an oasis nestled among dunes, featuring tall palm trees and an air of serenity+static view on a desert scene with an oasis, palm trees, and a clear, calm pool of water+A tranquil tableau of an ornate Victorian streetlamp standing on a cobblestone street corner, illuminating the empty night+A tranquil tableau of a tranquil lakeside cabin nestled among tall pines, its reflection mirrored perfectly in the calm water+In a still frame, a vintage gas lantern, adorned with intricate details, gracing a historic cobblestone square+In a still frame, a tranquil Japanese tea ceremony room, with tatami mats, a delicate tea set, and a bonsai tree in the corner+A tranquil tableau of the Parthenon stands resolute in its classical elegance, a timeless symbol of Athens' cultural legacy+A tranquil tableau of in the heart of Plaka, the neoclassical architecture of the old city harmonizes with the ancient ruins+A tranquil tableau of in the desolate beauty of the American Southwest, Chaco Canyon's ancient ruins whispered tales of an enigmatic civilization that once thrived amidst the arid landscapes+A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades+In a still frame, amidst the cobblestone streets, an Art Nouveau lamppost stood tall+A tranquil tableau of in the quaint village square, a traditional wrought-iron streetlamp featured delicate filigree patterns and amber-hued glass panels+A tranquil tableau of the lampposts were adorned with Art Deco motifs, their geometric shapes and frosted glass creating a sense of vintage glamour+In a still frame, in the picturesque square, a Gothic-style lamppost adorned with intricate stone carvings added a touch of medieval charm to the setting+In a still frame, in the heart of the old city, a row of ornate lantern-style streetlamps bathed the narrow alleyway in a warm, welcoming light+A tranquil tableau of in the heart of the Utah desert, a massive sandstone arch spanned the horizon+A tranquil tableau of in the Arizona desert, a massive stone bridge arched across a rugged canyon+A tranquil tableau of in the corner of the minimalist tea room, a bonsai tree added a touch of nature's beauty to the otherwise simple and elegant space+In a still frame, amidst the hushed ambiance of the traditional tea room, a meticulously arranged tea set awaited, with porcelain cups, a bamboo whisk+In a still frame, nestled in the Zen garden, a rustic teahouse featured tatami seating and a traditional charcoal brazier+A tranquil tableau of a country estate's library featured elegant wooden shelves+A tranquil tableau of beneath the shade of a solitary oak tree, an old wooden park bench sat patiently+A tranquil tableau of beside a tranquil pond, a weeping willow tree draped its branches gracefully over the water's surface, creating a serene tableau of reflection and calm+A tranquil tableau of in the Zen garden, a perfectly raked gravel path led to a serene rock garden+In a still frame, a tranquil pond was fringed by weeping cherry trees, their blossoms drifting lazily onto the glassy surface+In a still frame, within the historic library's reading room, rows of antique leather chairs and mahogany tables offered a serene haven for literary contemplation+A tranquil tableau of a peaceful orchid garden showcased a variety of delicate blooms+A tranquil tableau of in the serene courtyard, a centuries-old stone well stood as a symbol of a bygone era, its mossy stones bearing witness to the passage of time+a bird and a cat+a cat and a dog+a dog and a horse+a horse and a sheep+a sheep and a cow+a cow and an elephant+an elephant and a bear+a bear and a zebra+a zebra and a giraffe+a giraffe and a bird+a chair and a couch+a couch and a potted plant+a potted plant and a tv+a tv and a laptop+a laptop and a remote+a remote and a keyboard+a keyboard and a cell phone+a cell phone and a book+a book and a clock+a clock and a backpack+a backpack and an umbrella+an umbrella and a handbag+a handbag and a tie+a tie and a suitcase+a suitcase and a vase+a vase and scissors+scissors and a teddy bear+a teddy bear and a frisbee+a frisbee and skis+skis and a snowboard+a snowboard and a sports ball+a sports ball and a kite+a kite and a baseball bat+a baseball bat and a baseball glove+a baseball glove and a skateboard+a skateboard and a surfboard+a surfboard and a tennis racket+a tennis racket and a bottle+a bottle and a chair+an airplane and a train+a train and a boat+a boat and an airplane+a bicycle and a car+a car and a motorcycle+a motorcycle and a bus+a bus and a traffic light+a traffic light and a fire hydrant+a fire hydrant and a stop sign+a stop sign and a parking meter+a parking meter and a truck+a truck and a bicycle+a toilet and a hair drier+a hair drier and a toothbrush+a toothbrush and a sink+a sink and a toilet+a wine glass and a chair+a cup and a couch+a fork and a potted plant+a knife and a tv+a spoon and a laptop+a bowl and a remote+a banana and a keyboard+an apple and a cell phone+a sandwich and a book+an orange and a clock+broccoli and a backpack+a carrot and an umbrella+a hot dog and a handbag+a pizza and a tie+a donut and a suitcase+a cake and a vase+an oven and scissors+a toaster and a teddy bear+a microwave and a frisbee+a refrigerator and skis+a bicycle and an airplane+a car and a train+a motorcycle and a boat+a person and a toilet+a person and a hair drier+a person and a toothbrush+a person and a sink+A person is riding a bike+A person is marching+A person is roller skating+A person is tasting beer+A person is clapping+A person is drawing+A person is petting animal (not cat)+A person is eating watermelon+A person is playing harp+A person is wrestling+A person is riding scooter+A person is sweeping floor+A person is skateboarding+A person is dunking basketball+A person is playing flute+A person is stretching leg+A person is tying tie+A person is skydiving+A person is shooting goal (soccer)+A person is playing piano+A person is finger snapping+A person is canoeing or kayaking+A person is laughing+A person is digging+A person is clay pottery making+A person is shooting basketball+A person is bending back+A person is shaking hands+A person is bandaging+A person is push up+A person is catching or throwing frisbee+A person is playing trumpet+A person is flying kite+A person is filling eyebrows+A person is shuffling cards+A person is folding clothes+A person is smoking+A person is tai chi+A person is squat+A person is playing controller+A person is throwing axe+A person is giving or receiving award+A person is air drumming+A person is taking a shower+A person is planting trees+A person is sharpening knives+A person is robot dancing+A person is rock climbing+A person is hula hooping+A person is writing+A person is bungee jumping+A person is pushing cart+A person is cleaning windows+A person is cutting watermelon+A person is cheerleading+A person is washing hands+A person is ironing+A person is cutting nails+A person is hugging+A person is trimming or shaving beard+A person is jogging+A person is making bed+A person is washing dishes+A person is grooming dog+A person is doing laundry+A person is knitting+A person is reading book+A person is baby waking up+A person is massaging legs+A person is brushing teeth+A person is crawling baby+A person is motorcycling+A person is driving car+A person is sticking tongue out+A person is shaking head+A person is sword fighting+A person is doing aerobics+A person is strumming guitar+A person is riding or walking with horse+A person is archery+A person is catching or throwing baseball+A person is playing chess+A person is rock scissors paper+A person is using computer+A person is arranging flowers+A person is bending metal+A person is ice skating+A person is climbing a rope+A person is crying+A person is dancing ballet+A person is getting a haircut+A person is running on treadmill+A person is kissing+A person is counting money+A person is barbequing+A person is peeling apples+A person is milking cow+A person is shining shoes+A person is making snowman+A person is sailing+a person swimming in ocean+a person giving a presentation to a room full of colleagues+a person washing the dishes+a person eating a burger+a person walking in the snowstorm+a person drinking coffee in a cafe+a person playing guitar+a bicycle leaning against a tree+a bicycle gliding through a snowy field+a bicycle slowing down to stop+a bicycle accelerating to gain speed+a car stuck in traffic during rush hour+a car turning a corner+a car slowing down to stop+a car accelerating to gain speed+a motorcycle cruising along a coastal highway+a motorcycle turning a corner+a motorcycle slowing down to stop+a motorcycle gliding through a snowy field+a motorcycle accelerating to gain speed+an airplane soaring through a clear blue sky+an airplane taking off+an airplane landing smoothly on a runway+an airplane accelerating to gain speed+a bus turning a corner+a bus stuck in traffic during rush hour+a bus accelerating to gain speed+a train speeding down the tracks+a train crossing over a tall bridge+a train accelerating to gain speed+a truck turning a corner+a truck anchored in a tranquil bay+a truck stuck in traffic during rush hour+a truck slowing down to stop+a truck accelerating to gain speed+a boat sailing smoothly on a calm lake+a boat slowing down to stop+a boat accelerating to gain speed+a bird soaring gracefully in the sky+a bird building a nest from twigs and leaves+a bird flying over a snowy forest+a cat grooming itself meticulously with its tongue+a cat playing in park+a cat drinking water+a cat running happily+a dog enjoying a peaceful walk+a dog playing in park+a dog drinking water+a dog running happily+a horse bending down to drink water from a river+a horse galloping across an open field+a horse taking a peaceful walk+a horse running to join a herd of its kind+a sheep bending down to drink water from a river+a sheep taking a peaceful walk+a sheep running to join a herd of its kind+a cow bending down to drink water from a river+a cow chewing cud while resting in a tranquil barn+a cow running to join a herd of its kind+an elephant spraying itself with water using its trunk to cool down+an elephant taking a peaceful walk+an elephant running to join a herd of its kind+a bear catching a salmon in its powerful jaws+a bear sniffing the air for scents of food+a bear climbing a tree+a bear hunting for prey+a zebra bending down to drink water from a river+a zebra running to join a herd of its kind+a zebra taking a peaceful walk+a giraffe bending down to drink water from a river+a giraffe taking a peaceful walk+a giraffe running to join a herd of its kind+a person+a bicycle+a car+a motorcycle+an airplane+a bus+a train+a truck+a boat+a traffic light+a fire hydrant+a stop sign+a parking meter+a bench+a bird+a cat+a dog+a horse+a sheep+a cow+an elephant+a bear+a zebra+a giraffe+a backpack+an umbrella+a handbag+a tie+a suitcase+a frisbee+skis+a snowboard+a sports ball+a kite+a baseball bat+a baseball glove+a skateboard+a surfboard+a tennis racket+a bottle+a wine glass+a cup+a fork+a knife+a spoon+a bowl+a banana+an apple+a sandwich+an orange+broccoli+a carrot+a hot dog+a pizza+a donut+a cake+a chair+a couch+a potted plant+a bed+a dining table+a toilet+a tv+a laptop+a remote+a keyboard+a cell phone+a microwave+an oven+a toaster+a sink+a refrigerator+a book+a clock+a vase+scissors+a teddy bear+a hair drier+a toothbrush+a red bicycle+a green bicycle+a blue bicycle+a yellow bicycle+an orange bicycle+a purple bicycle+a pink bicycle+a black bicycle+a white bicycle+a red car+a green car+a blue car+a yellow car+an orange car+a purple car+a pink car+a black car+a white car+a red bird+a green bird+a blue bird+a yellow bird+an orange bird+a purple bird+a pink bird+a black bird+a white bird+a black cat+a white cat+an orange cat+a yellow cat+a red umbrella+a green umbrella+a blue umbrella+a yellow umbrella+an orange umbrella+a purple umbrella+a pink umbrella+a black umbrella+a white umbrella+a red suitcase+a green suitcase+a blue suitcase+a yellow suitcase+an orange suitcase+a purple suitcase+a pink suitcase+a black suitcase+a white suitcase+a red bowl+a green bowl+a blue bowl+a yellow bowl+an orange bowl+a purple bowl+a pink bowl+a black bowl+a white bowl+a red chair+a green chair+a blue chair+a yellow chair+an orange chair+a purple chair+a pink chair+a black chair+a white chair+a red clock+a green clock+a blue clock+a yellow clock+an orange clock+a purple clock+a pink clock+a black clock+a white clock+a red vase+a green vase+a blue vase+a yellow vase+an orange vase+a purple vase+a pink vase+a black vase+a white vase+A beautiful coastal beach in spring, waves lapping on sand, Van Gogh style+A beautiful coastal beach in spring, waves lapping on sand, oil painting+A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo+A beautiful coastal beach in spring, waves lapping on sand, black and white+A beautiful coastal beach in spring, waves lapping on sand, pixel art+A beautiful coastal beach in spring, waves lapping on sand, in cyberpunk style+A beautiful coastal beach in spring, waves lapping on sand, animated style+A beautiful coastal beach in spring, waves lapping on sand, watercolor painting+A beautiful coastal beach in spring, waves lapping on sand, surrealism style+The bund Shanghai, Van Gogh style+The bund Shanghai, oil painting+The bund Shanghai by Hokusai, in the style of Ukiyo+The bund Shanghai, black and white+The bund Shanghai, pixel art+The bund Shanghai, in cyberpunk style+The bund Shanghai, animated style+The bund Shanghai, watercolor painting+The bund Shanghai, surrealism style+a shark is swimming in the ocean, Van Gogh style+a shark is swimming in the ocean, oil painting+a shark is swimming in the ocean by Hokusai, in the style of Ukiyo+a shark is swimming in the ocean, black and white+a shark is swimming in the ocean, pixel art+a shark is swimming in the ocean, in cyberpunk style+a shark is swimming in the ocean, animated style+a shark is swimming in the ocean, watercolor painting+a shark is swimming in the ocean, surrealism style+A panda drinking coffee in a cafe in Paris, Van Gogh style+A panda drinking coffee in a cafe in Paris, oil painting+A panda drinking coffee in a cafe in Paris by Hokusai, in the style of Ukiyo+A panda drinking coffee in a cafe in Paris, black and white+A panda drinking coffee in a cafe in Paris, pixel art+A panda drinking coffee in a cafe in Paris, in cyberpunk style+A panda drinking coffee in a cafe in Paris, animated style+A panda drinking coffee in a cafe in Paris, watercolor painting+A panda drinking coffee in a cafe in Paris, surrealism style+A cute happy Corgi playing in park, sunset, Van Gogh style+A cute happy Corgi playing in park, sunset, oil painting+A cute happy Corgi playing in park, sunset by Hokusai, in the style of Ukiyo+A cute happy Corgi playing in park, sunset, black and white+A cute happy Corgi playing in park, sunset, pixel art+A cute happy Corgi playing in park, sunset, in cyberpunk style+A cute happy Corgi playing in park, sunset, animated style+A cute happy Corgi playing in park, sunset, watercolor painting+A cute happy Corgi playing in park, sunset, surrealism style+Gwen Stacy reading a book, Van Gogh style+Gwen Stacy reading a book, oil painting+Gwen Stacy reading a book by Hokusai, in the style of Ukiyo+Gwen Stacy reading a book, black and white+Gwen Stacy reading a book, pixel art+Gwen Stacy reading a book, in cyberpunk style+Gwen Stacy reading a book, animated style+Gwen Stacy reading a book, watercolor painting+Gwen Stacy reading a book, surrealism style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, Van Gogh style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, oil painting+A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Hokusai, in the style of Ukiyo+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, black and white+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pixel art+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, in cyberpunk style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, animated style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, watercolor painting+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, surrealism style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, Van Gogh style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, oil painting+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas by Hokusai, in the style of Ukiyo+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, black and white+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pixel art+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, in cyberpunk style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, animated style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, watercolor painting+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, surrealism style+An astronaut flying in space, Van Gogh style+An astronaut flying in space, oil painting+An astronaut flying in space by Hokusai, in the style of Ukiyo+An astronaut flying in space, black and white+An astronaut flying in space, pixel art+An astronaut flying in space, in cyberpunk style+An astronaut flying in space, animated style+An astronaut flying in space, watercolor painting+An astronaut flying in space, surrealism style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, Van Gogh style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, oil painting+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks by Hokusai, in the style of Ukiyo+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, black and white+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pixel art+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, in cyberpunk style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, animated style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, watercolor painting+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, surrealism style+A beautiful coastal beach in spring, waves lapping on sand, in super slow motion+A beautiful coastal beach in spring, waves lapping on sand, zoom in+A beautiful coastal beach in spring, waves lapping on sand, zoom out+A beautiful coastal beach in spring, waves lapping on sand, pan left+A beautiful coastal beach in spring, waves lapping on sand, pan right+A beautiful coastal beach in spring, waves lapping on sand, tilt up+A beautiful coastal beach in spring, waves lapping on sand, tilt down+A beautiful coastal beach in spring, waves lapping on sand, with an intense shaking effect+A beautiful coastal beach in spring, waves lapping on sand, featuring a steady and smooth perspective+A beautiful coastal beach in spring, waves lapping on sand, racking focus+The bund Shanghai, in super slow motion+The bund Shanghai, zoom in+The bund Shanghai, zoom out+The bund Shanghai, pan left+The bund Shanghai, pan right+The bund Shanghai, tilt up+The bund Shanghai, tilt down+The bund Shanghai, with an intense shaking effect+The bund Shanghai, featuring a steady and smooth perspective+The bund Shanghai, racking focus+a shark is swimming in the ocean, in super slow motion+a shark is swimming in the ocean, zoom in+a shark is swimming in the ocean, zoom out+a shark is swimming in the ocean, pan left+a shark is swimming in the ocean, pan right+a shark is swimming in the ocean, tilt up+a shark is swimming in the ocean, tilt down+a shark is swimming in the ocean, with an intense shaking effect+a shark is swimming in the ocean, featuring a steady and smooth perspective+a shark is swimming in the ocean, racking focus+A panda drinking coffee in a cafe in Paris, in super slow motion+A panda drinking coffee in a cafe in Paris, zoom in+A panda drinking coffee in a cafe in Paris, zoom out+A panda drinking coffee in a cafe in Paris, pan left+A panda drinking coffee in a cafe in Paris, pan right+A panda drinking coffee in a cafe in Paris, tilt up+A panda drinking coffee in a cafe in Paris, tilt down+A panda drinking coffee in a cafe in Paris, with an intense shaking effect+A panda drinking coffee in a cafe in Paris, featuring a steady and smooth perspective+A panda drinking coffee in a cafe in Paris, racking focus+A cute happy Corgi playing in park, sunset, in super slow motion+A cute happy Corgi playing in park, sunset, zoom in+A cute happy Corgi playing in park, sunset, zoom out+A cute happy Corgi playing in park, sunset, pan left+A cute happy Corgi playing in park, sunset, pan right+A cute happy Corgi playing in park, sunset, tilt up+A cute happy Corgi playing in park, sunset, tilt down+A cute happy Corgi playing in park, sunset, with an intense shaking effect+A cute happy Corgi playing in park, sunset, featuring a steady and smooth perspective+A cute happy Corgi playing in park, sunset, racking focus+Gwen Stacy reading a book, in super slow motion+Gwen Stacy reading a book, zoom in+Gwen Stacy reading a book, zoom out+Gwen Stacy reading a book, pan left+Gwen Stacy reading a book, pan right+Gwen Stacy reading a book, tilt up+Gwen Stacy reading a book, tilt down+Gwen Stacy reading a book, with an intense shaking effect+Gwen Stacy reading a book, featuring a steady and smooth perspective+Gwen Stacy reading a book, racking focus+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, in super slow motion+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, zoom in+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, zoom out+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pan left+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pan right+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, tilt up+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, tilt down+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, with an intense shaking effect+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, featuring a steady and smooth perspective+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, racking focus+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, in super slow motion+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, zoom in+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, zoom out+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pan left+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pan right+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, tilt up+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, tilt down+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, with an intense shaking effect+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, featuring a steady and smooth perspective+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, racking focus+An astronaut flying in space, in super slow motion+An astronaut flying in space, zoom in+An astronaut flying in space, zoom out+An astronaut flying in space, pan left+An astronaut flying in space, pan right+An astronaut flying in space, tilt up+An astronaut flying in space, tilt down+An astronaut flying in space, with an intense shaking effect+An astronaut flying in space, featuring a steady and smooth perspective+An astronaut flying in space, racking focus+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, in super slow motion+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, zoom in+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, zoom out+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pan left+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pan right+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, tilt up+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, tilt down+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, with an intense shaking effect+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, featuring a steady and smooth perspective+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, racking focus+Close up of grapes on a rotating table.+Turtle swimming in ocean.+A storm trooper vacuuming the beach.+A panda standing on a surfboard in the ocean in sunset.+An astronaut feeding ducks on a sunny afternoon, reflection from the water.+Two pandas discussing an academic paper.+Sunset time lapse at the beach with moving clouds and colors in the sky.+A fat rabbit wearing a purple robe walking through a fantasy landscape.+A koala bear playing piano in the forest.+An astronaut flying in space.+Fireworks.+An animated painting of fluffy white clouds moving in sky.+Flying through fantasy landscapes.+A bigfoot walking in the snowstorm.+A squirrel eating a burger.+A cat wearing sunglasses and working as a lifeguard at a pool.+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks.+Splash of turquoise water in extreme slow motion, alpha channel included.+an ice cream is melting on the table.+a drone flying over a snowy forest.+a shark is swimming in the ocean.+Aerial panoramic video from a drone of a fantasy land.+a teddy bear is swimming in the ocean.+time lapse of sunrise on mars.+golden fish swimming in the ocean.+An artist brush painting on a canvas close up.+A drone view of celebration with Christmas tree and fireworks, starry sky - background.+happy dog wearing a yellow turtleneck, studio, portrait, facing camera, dark background+Origami dancers in white paper, 3D render, on white background, studio shot, dancing modern dance.+Campfire at night in a snowy forest with starry sky in the background.+a fantasy landscape+A 3D model of a 1800s victorian house.+this is how I do makeup in the morning.+A raccoon that looks like a turtle, digital art.+Robot dancing in Times Square.+Busy freeway at night.+Balloon full of water exploding in extreme slow motion.+An astronaut is riding a horse in the space in a photorealistic style.+Macro slo-mo. Slow motion cropped closeup of roasted coffee beans falling into an empty bowl.+Sewing machine, old sewing machine working.+Motion colour drop in water, ink swirling in water, colourful ink in water, abstraction fancy dream cloud of ink.+Few big purple plums rotating on the turntable. water drops appear on the skin during rotation. isolated on the white background. close-up. macro.+Vampire makeup face of beautiful girl, red contact lenses.+Ashtray full of butts on table, smoke flowing on black background, close-up+Pacific coast, carmel by the sea ocean and waves.+A teddy bear is playing drum kit in NYC Times Square.+A corgi is playing drum kit.+An Iron man is playing the electronic guitar, high electronic guitar.+A raccoon is playing the electronic guitar.+A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh+A corgi's head depicted as an explosion of a nebula+A fantasy landscape+A future where humans have achieved teleportation technology+A jellyfish floating through the ocean, with bioluminescent tentacles+A Mars rover moving on Mars+A panda drinking coffee in a cafe in Paris+A space shuttle launching into orbit, with flames and smoke billowing out from the engines+A steam train moving on a mountainside+A super cool giant robot in Cyberpunk Beijing+A tropical beach at sunrise, with palm trees and crystal-clear water in the foreground+Cinematic shot of Van Gogh's selfie, Van Gogh style+Gwen Stacy reading a book+Iron Man flying in the sky+The bund Shanghai, oil painting+Yoda playing guitar on the stage+A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo+A beautiful coastal beach in spring, waves lapping on sand by Vincent van Gogh+A boat sailing leisurely along the Seine River with the Eiffel Tower in background+A car moving slowly on an empty street, rainy evening+A cat eating food out of a bowl+A cat wearing sunglasses at a pool+A confused panda in calculus class+A cute fluffy panda eating Chinese food in a restaurant+A cute happy Corgi playing in park, sunset+A cute raccoon playing guitar in a boat on the ocean+A happy fuzzy panda playing guitar nearby a campfire, snow mountain in the background+A lightning striking atop of eiffel tower, dark clouds in the sky+A modern art museum, with colorful paintings+A panda cooking in the kitchen+A panda playing on a swing set+A polar bear is playing guitar+A raccoon dressed in suit playing the trumpet, stage background+A robot DJ is playing the turntable, in heavy raining futuristic tokyo rooftop cyberpunk night, sci-fi, fantasy+A shark swimming in clear Caribbean ocean+A super robot protecting city+A teddy bear washing the dishes+An epic tornado attacking above a glowing city at night, the tornado is made of smoke+An oil painting of a couple in formal evening wear going home get caught in a heavy downpour with umbrellas+Clown fish swimming through the coral reef+Hyper-realistic spaceship landing on Mars+The bund Shanghai, vibrant color+Vincent van Gogh is painting in the room+Yellow flowers swing in the wind+alley+amusement park+aquarium+arch+art gallery+bathroom+bakery shop+ballroom+bar+barn+basement+beach+bedroom+bridge+botanical garden+cafeteria+campsite+campus+carrousel+castle+cemetery+classroom+cliff+crosswalk+construction site+corridor+courtyard+desert+downtown+driveway+farm+food court+football field+forest road+fountain+gas station+glacier+golf course+indoor gymnasium+harbor+highway+hospital+house+iceberg+industrial area+jail cell+junkyard+kitchen+indoor library+lighthouse+laboratory+mansion+marsh+mountain+indoor movie theater+indoor museum+music studio+nursery+ocean+office+palace+parking lot+pharmacy+phone booth+raceway+restaurant+river+science museum+shower+ski slope+sky+skyscraper+baseball stadium+staircase+street+supermarket+indoor swimming pool+tower+outdoor track+train railway+train station platform+underwater coral reef+valley+volcano+waterfall+windmill+a bicycle on the left of a car, front view+a car on the right of a motorcycle, front view+a motorcycle on the left of a bus, front view+a bus on the right of a traffic light, front view+a traffic light on the left of a fire hydrant, front view+a fire hydrant on the right of a stop sign, front view+a stop sign on the left of a parking meter, front view+a parking meter on the right of a bench, front view+a bench on the left of a truck, front view+a truck on the right of a bicycle, front view+a bird on the left of a cat, front view+a cat on the right of a dog, front view+a dog on the left of a horse, front view+a horse on the right of a sheep, front view+a sheep on the left of a cow, front view+a cow on the right of an elephant, front view+an elephant on the left of a bear, front view+a bear on the right of a zebra, front view+a zebra on the left of a giraffe, front view+a giraffe on the right of a bird, front view+a bottle on the left of a wine glass, front view+a wine glass on the right of a cup, front view+a cup on the left of a fork, front view+a fork on the right of a knife, front view+a knife on the left of a spoon, front view+a spoon on the right of a bowl, front view+a bowl on the left of a bottle, front view+a potted plant on the left of a remote, front view+a remote on the right of a clock, front view+a clock on the left of a vase, front view+a vase on the right of scissors, front view+scissors on the left of a teddy bear, front view+a teddy bear on the right of a potted plant, front view+a frisbee on the left of a sports ball, front view+a sports ball on the right of a baseball bat, front view+a baseball bat on the left of a baseball glove, front view+a baseball glove on the right of a tennis racket, front view+a tennis racket on the left of a frisbee, front view+a toilet on the left of a hair drier, front view+a hair drier on the right of a toothbrush, front view+a toothbrush on the left of a sink, front view+a sink on the right of a toilet, front view+a chair on the left of a couch, front view+a couch on the right of a bed, front view+a bed on the left of a tv, front view+a tv on the right of a dining table, front view+a dining table on the left of a chair, front view+an airplane on the left of a train, front view+a train on the right of a boat, front view+a boat on the left of an airplane, front view+an oven on the top of a toaster, front view+an oven on the bottom of a toaster, front view+a toaster on the top of a microwave, front view+a toaster on the bottom of a microwave, front view+a microwave on the top of an oven, front view+a microwave on the bottom of an oven, front view+a banana on the top of an apple, front view+a banana on the bottom of an apple, front view+an apple on the top of a sandwich, front view+an apple on the bottom of a sandwich, front view+a sandwich on the top of an orange, front view+a sandwich on the bottom of an orange, front view+an orange on the top of a carrot, front view+an orange on the bottom of a carrot, front view+a carrot on the top of a hot dog, front view+a carrot on the bottom of a hot dog, front view+a hot dog on the top of a pizza, front view+a hot dog on the bottom of a pizza, front view+a pizza on the top of a donut, front view+a pizza on the bottom of a donut, front view+a donut on the top of broccoli, front view+a donut on the bottom of broccoli, front view+broccoli on the top of a banana, front view+broccoli on the bottom of a banana, front view+skis on the top of a snowboard, front view+skis on the bottom of a snowboard, front view+a snowboard on the top of a kite, front view+a snowboard on the bottom of a kite, front view+a kite on the top of a skateboard, front view+a kite on the bottom of a skateboard, front view+a skateboard on the top of a surfboard, front view+a skateboard on the bottom of a surfboard, front view+a surfboard on the top of skis, front view+a surfboard on the bottom of skis, front viewdiff --git a/assets/texts/VBench/prompts_per_category/animal.txt b/assets/texts/VBench/prompts_per_category/animal.txtnew file mode 100755index 0000000..775f4c4--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/animal.txt@@ -0,0 +1,100 @@+a black dog wearing halloween costume+spider making a web+bat eating fruits while hanging+a snake crawling on a wooden flooring+a close up video of a dragonfly+macro shot of ladybug on green leaf plant+chameleon eating ant+a bee feeding on nectars+bird nests on a tree captured with moving camera+a squirrel eating nuts+close up video of snail+top view of a hermit crab crawling on a wooden surface+cat licking another cat+red dragonfly perched on green leaf+close up view of a brown caterpillar crawling on green leaf+ants eating dead spider+an eagle on a tree branch+a frog eating an ant+white rabbit near the fence+a gorilla eating a carrot+close up of wolf+a meerkat looking around+a hyena in a zoo+lemur eating grass leaves+an owl being trained by a man+a lizard on a bamboo+brown chicken hunting for its food+video of parrots perched on bird stand+underwater footage of an octopus in a coral reef+a cute pomeranian dog playing with a soccer ball+white fox on rock+close up footage of a horse figurine+giraffe feeding on a tree in a savannah+curious cat sitting and looking around+hummingbird hawk moth flying near pink flowers+close up of a scorpion on a rock+close up on fish in net+koala eating leaves from a branch+a pod of dolphins swirling in the sea catching forage fish+low angle view of a hawk perched on a tree branch+a lion standing on wild grass+deer grazing in the field+elephant herd in a savanna+close up on lobster under water+hedgehog crossing road in forest+a sheep eating yellow flowers from behind a wire fence+twin sisters and a turtle+a pig wallowing in mud+flock of goose eating on the lake water+cow in a field irritated with flies+a close up shot of a fly+cheetah lying on the grass+close up of a lemur+close up shot of a kangaroo itching in the sand+a tortoise covered with algae+turkey in cage+a great blue heron bird in the lakeside+crab with shell in aquarium+a seagull walking on shore+an american crocodile+a tiger walking inside a cage+alligator in the nature+a raccoon climbing a tree+wild rabbit in a green meadow+group of ring tailed lemurs+a clouded leopard on a tree branch+duck grooming its feathers+an african penguin walking on a beach+a video of a peacock+close up shot of a wild bear+baby rhino plays with mom+porcupine climbs tree branches+close up of a natterjack toad on a rock+a sleeping orangutan+mother whale swimming with babies+a bear wearing red jersey+pink jellyfish swimming underwater in a blue sea+beautiful clown fish swimming+animation of disposable objects shaped as a whale+paper cut out of a pair of hands a whale and a heart+vertical video of camel roaming in the field during daytime+a still video of mosquito biting human+a curious sloth hanging from a tree branch+a plastic flamingo bird stumbles from the wind+a wolf in its natural habitat+a monkey sitting in the stone and scratching his head+bat hanging upside down+a red panda eating leaves+snake on ground+a harbour seal swimming near the shore+shark swimming in the sea+otter on branch while eating+goat standing over a rock+a troop of monkey on top of a mountain+a zebra eating grass on the field+a colorful butterfly perching on a bud+a snail crawling on a leaf+zookeeper showering a baby elephant+a beetle emerging from the sand+a nine banded armadillo searching for fooddiff --git a/assets/texts/VBench/prompts_per_category/architecture.txt b/assets/texts/VBench/prompts_per_category/architecture.txtnew file mode 100755index 0000000..599b76b--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/architecture.txt@@ -0,0 +1,100 @@+an apartment building with balcony+asian garden and medieval castle+illuminated tower in berlin+a wooden house overseeing the lake+a crowd of people in a plaza in front of a government building+a church interior+jewish friends posing with hanukkah menorah in a cabin house+a destroyed building after a missile attack in ukraine+abandoned building in the woods+drone video of an abandoned school building in pripyat ukraine+elegant university building+architecture and designs of buildings in central london+a pancake tower with chocolate syrup and strawberries on top+an ancient white building+friends hanging out at a coffee house+house front door with christmas decorations+city night dark building+a bird house hanging on a tree branch+sacred sculpture in a temple+high angle shot of a clock tower+modern wooden house interior+the interior of an abandoned building+opera house overlooking sea+a concrete structure near the green trees+dome like building in scotland+low angle shot of a building+tower on hill+a miniature house+eiffel tower from the seine river+low angle footage of an apartment building+island with pier and antique building+asian historic architecture+drone footage of a beautiful mansion+mosque in the middle east+building a tent and hammock in the forest camping site+top view of a high rise building+house covered in snow+skyscraper at night+house in village+a casino with people outside the building+silhouette of a building+a woman climbing a tree house+drone view of house near lake during golden hour+an under construction concrete house+a watch tower by the sea+exterior view of arabic style building+video of a hotel building+red paper lantern decorations hanging outside a building+house on seashore+aerial footage of the palace of culture and science building in warsaw poland+aerial video of stuttgart tv tower in germany+aerial view of the highway and building in a city+drone shot of a skyscraper san francisco california usa+waterfall and house+view of the sky through a building+drone footage of a house on top of the mountain+abandoned house in the nature+clouds hovering over a mansion+light house on the ocean+buddhist temple at sunrise+people walking by a graveyard near a mosque at sunset+view of lifeguard tower on the beach+scenic view of a house in the mountains+the landscape in front of a government building+aerial footage of a building and its surrounding landscape in winter+time lapse of a cloudy sky behind a transmission tower+blue ocean near the brown castle+fog over temple+house in countryside top view+building under construction+turkish flag waving on old tower+the georgian building+close up shot of a steel structure+the atrium and interior design of a multi floor building+city view reflected on a glass building+aerial view of a luxurious house with pool+an unpaved road leading to the house+drone footage of a lookout tower in mountain landscape+wind turbines on hill behind building+time lapse footage of the sun light in front of a small house porch+a building built with lots of stairways+overcast over house on seashore+the view of the sydney opera house from the other side of the harbor+candle on a jar and a house figurine on a surface+video of a farm and house+a dilapidated building made of bricks+a view of a unique building from a moving vehicle+aerial footage of a tall building in cambodia+push in shot of a huge house+a beach house built over a seawall protected from the sea waves+exotic house surrounded by trees+drone video of a house surrounded by tropical vegetation+drone footage of a building beside a pond+observation tower on hill in forest+a tree house in the woods+a video of vessel structure during daytime+fire in front of illuminated building at night+a footage of a wooden house on a wheat field+tilt shot of a solar panel below a light tower+water tower on the desertdiff --git a/assets/texts/VBench/prompts_per_category/food.txt b/assets/texts/VBench/prompts_per_category/food.txtnew file mode 100755index 0000000..032aed9--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/food.txt@@ -0,0 +1,100 @@+freshly baked finger looking cookies+video of fake blood in wine glass+halloween food art+a person slicing a vegetable+a serving of pumpkin dish in a plate+close up view of green leafy vegetable+a birthday cake in the plate+video of a slice papaya fruit+a muffin with a burning candle and a love sign by a ceramic mug+a jack o lantern designed cookie+baked bread with chocolate+a broccoli soup on wooden table+a freshly brewed coffee on a pink mug+grabbing sourdough neapolitan style pizza slices+person cooking mushrooms in frying pan+rice grains placed on a reusable cloth bag+slices of kiwi fruit+grilling a steak on a pan grill+close up of bread popping out of a toaster+man eating noodle+preparing a cocktail drink+close up pasta with bacon on plate+milk and cinnamon rolls+boy getting a dumpling using chopsticks+a mother preparing food with her kids+man using his phone while eating+fresh salmon salad on a plate+cutting cucumbers into long thin slices as ingredient for sushi roll+a steaming cup of tea by the window+a glass filled with beer+a kid eating popcorn while watching tv+close up shot of fried fish on the plate+a man eating a donut+person making a vegetarian dish+spreading cheese on bagel+close up view of a man drinking red wine+a couple having breakfast in a restaurant+a student eating her sandwich+girl peeling a banana+red rice in a small bowl+pancake with blueberry on the top+green apple fruit on white wooden table+a man eating a taco by the bar+making of a burrito+squeezing lemon into salad+a chef cutting sushi rolls+video of a delicious dessert+deep frying a crab on a wok in high fire+close up video of a orange juice+video of a cooked chicken breast+woman holding a pineapple+a woman eating a bar of chocolate+decorating christmas cookie+squeezing a slice of fruit+tuna sashimi on a plate+a strawberry fruit mixed in an alcoholic drink+preparing hot dogs in a grill+a woman cutting a tomato+an orange fruit cut in half+a coconut fruit with drinking straw+woman holding a dragon fruit+a woman pouring hot beverage on a cup+waffles with whipped cream and fruit+focus shot of an insect at the bottom of a fruit+preparing a healthy broccoli dish+man eating snack at picnic+close up video of a grilled shrimp skewer+a woman mixing a smoothie drinks+close up video of woman having a bite of jelly+businessman drinking whiskey at the bar counter of a hotel lounge+cutting an onion with a knife over a wooden chopping board+fresh lemonade in bottles+grilling a meat on a charcoal grill+people enjoying asian cuisine+close up footage of a hot dish on a clay pot+pork ribs dish+waffle with strawberry and syrup for breakfast+tofu dish with rose garnish+uncooked pork meat+egg yolk being dumped over gourmet dish+tasty brunch dish close up+little boy pretending to eat the watermelon+slicing roasted beef+close up of a chef adding teriyaki sauce to a dish+flat lay mexican dish+a person placing an octopus dish on a marble surface+close up of tea leaves brewing in a glass kettle+adding fresh herbs to soup dish+a scoop of roasted coffee beans+fresh dim sum set up on a bamboo steam tray for cooking+a girl putting ketchup on food at the kitchen+cooking on electric stove+a woman with a slice of a pie+grapes and wine on a wooden board+man taking picture of his food+hamburger and fries on restaurant table+close up video of japanese food+a cracker sandwich with cheese filling for snack+barista preparing matcha tea+close up of onion rings being deep frieddiff --git a/assets/texts/VBench/prompts_per_category/human.txt b/assets/texts/VBench/prompts_per_category/human.txtnew file mode 100755index 0000000..88de93a--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/human.txt@@ -0,0 +1,100 @@+people carving a pumpkin+people sitting on a sofa+a man with a muertos face painting+man walking in the dark+men in front of their computer editing photos+men loading christmas tree on tow truck+woman washing the dishes+woman adding honey to the cinnamon rolls+two women kissing and smiling+three women looking at watercolor paintings+a family wearing paper bag masks+a family posing for the camera+a boy covering a rose flower with a dome glass+boy sitting on grass petting a dog+a girl in her tennis sportswear+a girl coloring the cardboard+silhouette of the couple during sunset+couple dancing with body paint+a child playing with water+a woman with her child sitting on a couch in the living room+a group of friend place doing hand gestures of agreement+friends having a group selfie+friends talking while on the basketball court+group of people protesting+a group of campers with a cute dog+a group of photographers taking pictures at the north western gardens in llandudno north wales+a group of students laughing and talking+a group of martial artist warming up+a person playing golf+a person walking on a wet wooden bridge+person doing a leg exercise+ice hockey athlete on rink+a young athlete training in swimming+chess player dusting a chessboard+baseball player holding his bat+a bearded man putting a vinyl record on a vinyl player+an orchestra finishes a performance+people applauding the performance of the kids+band performance at the recording studio+father and his children playing jenga game+people playing a board game+man playing a video game+a man video recording the movie in theater+man and a woman eating while watching a movie+movie crew talking together+a director explaining the movie scene+man and woman listening to music on car+man playing music+couple dancing slow dance with sun glare+a ballerina practicing in the dance studio+father and son holding hands+father and daughter talking together+a mother and her kids engaged in a video call+mother and daughter reading a book together+a mother teaching her daughter playing a violin+kid in a halloween costume+a happy kid playing the ukulele+a chef slicing a cucumber+chef wearing his gloves properly+brother and sister using hammock+girl applying sunblock to her brother+a girl pushing the chair while her sister is on the chair+colleagues talking in office building+fighter practice kicking+a woman fighter in her cosplay costume+an engineer holding blueprints while talking with her colleague+a young woman looking at vr controllers with her friend+workmates teasing a colleague in the work+a male police officer talking on the radio+teacher holding a marker while talking+teacher writing on her notebook+a young student attending her online classes+a student showing his classmates his wand+a male vendor selling fruits+a shirtless male climber+a sound engineer listening to music+female talking to a psychiatrist in a therapy session+young female activist posing with flag+a man in a hoodie and woman with a red bandana talking to each other and smiling+a medium close up of women wearing kimonos+a male interviewer listening to a person talking+a social worker having a conversation with the foster parents+a farm worker harvesting onions+worker packing street food+worker and client at barber shop+elderly man lifting kettlebell+mom assisting son in riding a bicycle+dad watching her daughter eat+young guy with vr headset+pregnant woman exercising with trainer+a fortune teller talking to a client+wizard doing a ritual on a woman+a footage of an actor on a movie scene+a man holding a best actor trophy+a singer of a music band+a young singer performing on stage+young dancer practicing at home+seller showing room to a couple+cab driver talking to passenger+a policeman talking to the car driverdiff --git a/assets/texts/VBench/prompts_per_category/lifestyle.txt b/assets/texts/VBench/prompts_per_category/lifestyle.txtnew file mode 100755index 0000000..78c8be0--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/lifestyle.txt@@ -0,0 +1,100 @@+kids celebrating halloween at home+little boy helping mother in kitchen+video of a indoor green plant+a girl arranges a christmas garland hanging by the kitchen cabinet+candle burning in dark room+couple having fun and goofing around the bedroom+girls jumping up and down in the bedroom+woman and man in pajamas working from home+a muslim family sitting and talking in the living room+family enjoying snack time while sitting in the living room+woman holding an animal puppet and a little girl playing together at the living room+kids playing in the indoor tent+young people celebrating new year at the office+a woman writing on the sticky note in the office+a woman exercising at home over a yoga mat+girls preparing easter decorations at home+dog on floor in room+turning on a fluorescent light inside a room+colleagues talking to each other near the office windows+a woman recording herself while exercising at home+music room+different kind of tools kept in a utility room+sofa beds and other furniture+a girl finding her brother reading a book in the bedroom+an elegant ceramic plant pot and hanging plant on indoor+furniture inside a bedroom+interior design of the bar section+living room with party decoration+firewood burning in dark room+a young woman playing the ukulele at home+woman painting at home+a woman in a locker room+video of a bathroom interior+the interior design of a jewish synagogue+a woman in protective suit disinfecting the kitchen+modern minimalist home interior+modern interior design of a coffee shop+person arranging minimalist furniture+aerial shot of interior of the warehouse+a room of a manufacturing facility+interior of catholic+interior design of a restaurant+a female model in a changing room looking herself in mirror+men walking in the office hallway+people sitting in a conference room+the interior design of a shopping mall+chandeliers in room+lucerne railway station interior+a female fencer posing in a foggy room+a toolbox and a paint roller beside a huge package in a room+bedroom in hotel+a woman lying in the operating room+a chef holding and checking kitchen utensils+a couple singing in the shower room together+a woman cleaning mess in the living room+an empty meeting room with natural light+person dancing in a dark room+close up on blood in hospital room+a couple resting on their home floor+a young female staff at courier office+a man entering the gym locker room+a bored man sitting by the tv at home+woman dancing in indoor garden+rubble in the interior of an abandoned house+indoor farm in a greenhouse+man doing handstand in indoor garden+an abandoned indoor swimming pool+home decorations on top of a cabinet+graffiti art on the interior walls of an abandoned mansion+indoor wall climbing activity+sunlight inside a room+teenage girl roller skating at indoor rink+home deco with lighted+baby in the shower room+men enjoying office christmas party+a bedroom with a brick wall+actors prepping in the dressing room+kids playing at an indoor playground+a person sanitizing an office space using smoke machine+mother and daughter choosing clothes at home+a woman sitting by the indoor fire pit+man standing on the corner of the room while looking around+person assembling furniture+a family stacking cardboard boxes in a room+family having fun in the dining room+person disinfecting a room+a woman washing strawberries in the kitchen sink+modern office waiting room+close up view of a person slicing with a kitchen knife+boiling coffee on a stove in the kitchen+modern equipment used in a home studio+interior of a recording studio+people working in a call center office+band performing at a home concert+a group of people watching a concert in a room+people packing their furniture+young employees in office holding a certificate+a criminal inside a dark room handcuffed in a table+couple browsing and looking for furniture in the store+workspace at homediff --git a/assets/texts/VBench/prompts_per_category/plant.txt b/assets/texts/VBench/prompts_per_category/plant.txtnew file mode 100755index 0000000..7d04fc7--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/plant.txt@@ -0,0 +1,100 @@+video of a indoor green plant+close up view of a plant+close up shot of a burning plant+plucking leaves from plant+a plant on gold pot with glass lid+a branch of a tree and a plant+a leafless tree+close up shot of fern leaf+close up video of strawberry plant+plant with blooming flowers+close up video of flower petals+watering yellow plant+beautiful flower decoration+cannabis flower in a jar+a footage of the tree leaves+a red leaf plant+close up view of a white christmas tree+snow pouring on a tree+close up shot of white flowers on the tree+leaves in the trees daytime+a dead tree lying on a grass field+tree branches in a flowing river+purple flowers with leaves+a coconut tree by the house+close up on flower in winter+bamboo leaves backlit by the sun+close up video of a wet flower+a man putting a flower in a box+dropping flower petals on a wooden bowl+a close up shot of gypsophila flower+variety of succulent plants on a garden+variety of trees and plants in a botanical garden+forest of deciduous trees+a stack of dried leaves burning in a forest+tall forest trees on a misty morning+close up view of dewdrops on a leaf+close up view of white petaled flower+removing a pineapple leaf+a dragonfly perched on a leaf+butterfly pollinating flower+person visiting and checking a corn plant+woman picking beans from a plant+woman plucking mint leaves+single tree in the middle of farmland+a plant on a soil+drone footage of a tree on farm field+a tractor harvesting lavender flower+people putting christmas ornaments on a christmas tree+jack o lantern hanging on a tree+tree with halloween decoration+flower field near the waterfall+truck carrying the tree logs+raindrops falling on leaves+shot of a palm tree swaying with the wind+squirrels on a tree branch+person holding a flower+a fallen tree trunk+tree with golden leaves+cherry tree+wind blows through leaves of the tree in autumn+a leaf on a glass+the long trunks of tall trees in the forest+trees in the forest during sunny day+close up video of tree bark+reflection of tree branches+trunks of many trees in the forest+tree leaves providing shades from the sun+leaves swaying in the wind+low angle shot of baobab tree+bare trees in forest+a plant surrounded by fallen leaves+a couple preparing food and pruning a plant+a man cutting a tree bark+oranges on a tree branch+plant connected on the stones+video of a sawmill machine cutting tree log+women drying flower petals+macro view of an agave plant+a video of a person tying a plant on a string+green moss in forest nature+coconut tree near sea under blue sky+the canopy of a coconut tree+a man leaning on a tree at the beach+a full grown plant on a pot+candle wax dripping on flower petals+close up of leaves in autumn+a woman opening a book with a flower inside+a man holding leaves looking at the camera+a shadow of a swaying plant+a tree and concrete structure under a blue and cloudy sky+trimming excess leaves on a potted plant+the changing color of the tree leaves during autumn season+a gooseberry tree swayed by the wind+forest trees and a medieval castle at sunset+woman cut down tree+an old oak tree in a park across the street from a hotel+wild flowers growing in a forest ground+a mossy fountain and green plants in a botanical garden+mansion with beautiful garden+ants on a dragon fruit flowerdiff --git a/assets/texts/VBench/prompts_per_category/scenery.txt b/assets/texts/VBench/prompts_per_category/scenery.txtnew file mode 100755index 0000000..5d77e3f--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/scenery.txt@@ -0,0 +1,100 @@+scenery of desert landscape+landscape agriculture farm tractor+burning slash piles in the forest+graveyard at sunset+view of a jack o lantern with pumpkins in a smoky garden+sun view through a spider web+view of the sea from an abandoned building+close up view of a full moon+close up view of lighted candles+close up view of swaying white flowers and leaves+scenery of a relaxing beach+selective focus video of grass during sunny day+aerial view of brown dry landscape+fireworks display in the sky at night+a bonfire near river+mountain view+waterfalls in between mountain+a picturesque view of nature+exotic view of a riverfront city+tall trees in the forest under the clear sky+snow on branches in forest+stream in the nature+an airplane flying above the sea of clouds+scenic video of sunset+view of houses with bush fence under a blue and cloudy sky+scenic view from wooden pathway+scenic view of a tropical beach+drone footage of waves crashing on beach shore+a scenic view of the golden hour at norway+time lapse video of foggy mountain forest+brown mountain during fall season+video of ocean during daytime+boat sailing in the ocean+top view of yachts+beautiful scenery of flowing waterfalls and river+wild ducks paddling on the lake surface+a relaxing scenery of beach view under cloudy sky+natural rock formations on beach under cloudy sky+a palm tree against blue sky+video of sailboat on a lake during sunset+aerial view of snow piles+time lapse of a sunset sky in the countryside+aerial footage of a statue+time lapse video of a farm during sunset+clouds formation in the sky at sunset+aerial shot of a village+drone shot of a beautiful sunrise at the mountains+time lapse video of foggy morning during sunrise+sun shining between tree leaves at sunrise+video of lake during dawn+vehicles traveling on roadway under cloudy sky+view of golden domed church+a monument under the blue sky+firecrackers in the sky+view of fruit signage in the farm+a dark clouds over shadowing the full moon+view of the amazon river+a big river swamp in a dense forest+a blooming cherry blossom tree under a blue sky with white clouds+a river waterfall cascading down the plunge basin+flooded landscape with palm trees+a blurry waterfall background+waterfall in the mountains+aerial footage of a city at night+pond by small waterfall in forest+aerial view of farmlands at the bay of lake+rice terraces in the countryside+a highway built across an agricultural area in the countryside+gloomy morning in the countryside+drone shot of an abandoned coliseum on a snowy mountain top+boat sailing in the middle of ocean+drone shot of the grass field+natural landscape of mountain and sea with islets developed into a community+aerial view of zaporizhia in ukraine+aerial footage of a herd+an aerial footage of a red sky+grass and plants growing in the remains of an abandoned house+view from hill on city+aerial view on orthodox church+aerial view of bay in croatia+a footage of a frozen river+overlooking view of a city at daylight+view outside the cemetery+clear sky with moon over meadow+clouds over railway+aerial footage of moving vehicles on the road at night+aerial view of town and park+top view of skyscrapers+top view of the empire state building in manhattan+top view of the central park in new york city+sheep running in a grass field+clear sky over factory+smoke and fire in birds eye view+view of a pathway with snow melting on its side+ferry under bridge on river near city in malaysia+mountain slopes covered in green vegetation+panoramic view of a town surrounded by snow covered mountains+aerial view of a palace+top view of vehicles driving on the intersection+a graveyard by a church in a mountain landscapediff --git a/assets/texts/VBench/prompts_per_category/vehicles.txt b/assets/texts/VBench/prompts_per_category/vehicles.txtnew file mode 100755index 0000000..5fd5fca--- /dev/null+++ b/assets/texts/VBench/prompts_per_category/vehicles.txt@@ -0,0 +1,100 @@+a modern railway station in malaysia use for public transportation+drone footage of amsterdam metro station+train arriving at a station+red vehicle driving on field+close up view of flashing emergency vehicle lighting+vehicle with fertilizer on field+a highway built across an agricultural area in the countryside+drone footage of motorcycles driving on country road between agricultural fields+a road in the woods under fog+footage of a car driving through a wheat field+vehicle stops for an ambulance passing through city traffic+emergency vehicle parked outside the casino+zombies attacking a woman and a boy inside a car+woman seating inside the car while chewing+video of passengers riding a double decker bus during night+traffic in london street at night+elderly couple checking engine of automobile+a green vintage automobile with an open hood parked in a parking area+close up of a prototype automobile with exposed engine on the back seat of the car+aerial view of road in forest+train departing from station+aerial view of a train passing by a bridge+video of a train tracks+video footage of a subway+video of blinking traffic lights+couple walking out on the subway+time lapse of a subway tunnel+monitor board inside the subway+metro train at night+zoom in video of a tram passing by city+young man using laptop in the tram+man reading a book at bus stop+close up shot of a moving taxi+night travel in london street on a public bus+red bus in a rainy city+flow of traffic in the city+close up shot of a yellow taxi turning left+two women calling for a taxi+drone view of an illuminated bridge across a river+policeman in police car talking on radio+airplane taking off at night+view through window in airplane+an airplane in the sky+helicopter landing on the street+a pilot getting out of a helicopter+a helicopter flying under blue sky+boat sailing in the middle of the ocean+girl playing with a toy boat+silhouette of a boat on sea during golden hour+a boat travelling around the lake+road on mountain ridge+ship sailing on danube river+slow motion video of a ship water trail in the sea+drone footage of a wreck ship on shore+a white yacht traveling on a river and passing under the bridge+female teenagers drinking champagne in the yacht+video of yacht sailing in the ocean+red combine harvester on road on field+a woman sitting on a bicycle while using a mobile phone+a woman sitting on a motorcycle looking around+three teenagers fixing a bicycle+a woman in a halloween costume posing on a motorcycle+a parked motorcycle on a foggy roadside+cable car near sea shore+a truck travelling in the road+footage of the road without any traffic+a road sign+love padlocks on a bridge+camera moving at highway construction site+vehicles driving on highway+a motorbike on highway at timelapse mode+point of view of a car driving through a tunnel+time lapse of heavy traffic on an avenue+ferry boat on city canal+black vintage car in museum+a zigzag road across a forest+people crossing the road+video of a kayak boat in a river+a person paddling a wooden boat in a lake+a car charging in the parking area+cars parked on the road+footage of the street with people and vehicle passing by in the rain+traffic on busy city street+a woman getting out of the car to walk with their dog+yacht sailing through the ocean+people in queue to military ship+man wearing motorcycle helmet looking at the camera+empty seats in the bus+empty boat on the water+cargo train traveling on the mountainside+cruise ship in harbor+counting down at traffic lights+pressing the car ignition+fire truck driving on the road+a footage of a broken bicycle+drone footage of an ambulance on the road+slow motion footage of a racing car+ship sailing on sea against sunset+big cargo ship passing on the shore+back view of man and woman walking on unpaved roaddiff --git a/assets/texts/VBench/prompts_per_dimension/appearance_style.txt b/assets/texts/VBench/prompts_per_dimension/appearance_style.txtnew file mode 100755index 0000000..4382de9--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/appearance_style.txt@@ -0,0 +1,90 @@+A beautiful coastal beach in spring, waves lapping on sand, Van Gogh style+A beautiful coastal beach in spring, waves lapping on sand, oil painting+A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo+A beautiful coastal beach in spring, waves lapping on sand, black and white+A beautiful coastal beach in spring, waves lapping on sand, pixel art+A beautiful coastal beach in spring, waves lapping on sand, in cyberpunk style+A beautiful coastal beach in spring, waves lapping on sand, animated style+A beautiful coastal beach in spring, waves lapping on sand, watercolor painting+A beautiful coastal beach in spring, waves lapping on sand, surrealism style+The bund Shanghai, Van Gogh style+The bund Shanghai, oil painting+The bund Shanghai by Hokusai, in the style of Ukiyo+The bund Shanghai, black and white+The bund Shanghai, pixel art+The bund Shanghai, in cyberpunk style+The bund Shanghai, animated style+The bund Shanghai, watercolor painting+The bund Shanghai, surrealism style+a shark is swimming in the ocean, Van Gogh style+a shark is swimming in the ocean, oil painting+a shark is swimming in the ocean by Hokusai, in the style of Ukiyo+a shark is swimming in the ocean, black and white+a shark is swimming in the ocean, pixel art+a shark is swimming in the ocean, in cyberpunk style+a shark is swimming in the ocean, animated style+a shark is swimming in the ocean, watercolor painting+a shark is swimming in the ocean, surrealism style+A panda drinking coffee in a cafe in Paris, Van Gogh style+A panda drinking coffee in a cafe in Paris, oil painting+A panda drinking coffee in a cafe in Paris by Hokusai, in the style of Ukiyo+A panda drinking coffee in a cafe in Paris, black and white+A panda drinking coffee in a cafe in Paris, pixel art+A panda drinking coffee in a cafe in Paris, in cyberpunk style+A panda drinking coffee in a cafe in Paris, animated style+A panda drinking coffee in a cafe in Paris, watercolor painting+A panda drinking coffee in a cafe in Paris, surrealism style+A cute happy Corgi playing in park, sunset, Van Gogh style+A cute happy Corgi playing in park, sunset, oil painting+A cute happy Corgi playing in park, sunset by Hokusai, in the style of Ukiyo+A cute happy Corgi playing in park, sunset, black and white+A cute happy Corgi playing in park, sunset, pixel art+A cute happy Corgi playing in park, sunset, in cyberpunk style+A cute happy Corgi playing in park, sunset, animated style+A cute happy Corgi playing in park, sunset, watercolor painting+A cute happy Corgi playing in park, sunset, surrealism style+Gwen Stacy reading a book, Van Gogh style+Gwen Stacy reading a book, oil painting+Gwen Stacy reading a book by Hokusai, in the style of Ukiyo+Gwen Stacy reading a book, black and white+Gwen Stacy reading a book, pixel art+Gwen Stacy reading a book, in cyberpunk style+Gwen Stacy reading a book, animated style+Gwen Stacy reading a book, watercolor painting+Gwen Stacy reading a book, surrealism style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, Van Gogh style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, oil painting+A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Hokusai, in the style of Ukiyo+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, black and white+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pixel art+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, in cyberpunk style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, animated style+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, watercolor painting+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, surrealism style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, Van Gogh style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, oil painting+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas by Hokusai, in the style of Ukiyo+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, black and white+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pixel art+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, in cyberpunk style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, animated style+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, watercolor painting+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, surrealism style+An astronaut flying in space, Van Gogh style+An astronaut flying in space, oil painting+An astronaut flying in space by Hokusai, in the style of Ukiyo+An astronaut flying in space, black and white+An astronaut flying in space, pixel art+An astronaut flying in space, in cyberpunk style+An astronaut flying in space, animated style+An astronaut flying in space, watercolor painting+An astronaut flying in space, surrealism style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, Van Gogh style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, oil painting+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks by Hokusai, in the style of Ukiyo+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, black and white+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pixel art+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, in cyberpunk style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, animated style+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, watercolor painting+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, surrealism stylediff --git a/assets/texts/VBench/prompts_per_dimension/color.txt b/assets/texts/VBench/prompts_per_dimension/color.txtnew file mode 100755index 0000000..42e135a--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/color.txt@@ -0,0 +1,85 @@+a red bicycle+a green bicycle+a blue bicycle+a yellow bicycle+an orange bicycle+a purple bicycle+a pink bicycle+a black bicycle+a white bicycle+a red car+a green car+a blue car+a yellow car+an orange car+a purple car+a pink car+a black car+a white car+a red bird+a green bird+a blue bird+a yellow bird+an orange bird+a purple bird+a pink bird+a black bird+a white bird+a black cat+a white cat+an orange cat+a yellow cat+a red umbrella+a green umbrella+a blue umbrella+a yellow umbrella+an orange umbrella+a purple umbrella+a pink umbrella+a black umbrella+a white umbrella+a red suitcase+a green suitcase+a blue suitcase+a yellow suitcase+an orange suitcase+a purple suitcase+a pink suitcase+a black suitcase+a white suitcase+a red bowl+a green bowl+a blue bowl+a yellow bowl+an orange bowl+a purple bowl+a pink bowl+a black bowl+a white bowl+a red chair+a green chair+a blue chair+a yellow chair+an orange chair+a purple chair+a pink chair+a black chair+a white chair+a red clock+a green clock+a blue clock+a yellow clock+an orange clock+a purple clock+a pink clock+a black clock+a white clock+a red vase+a green vase+a blue vase+a yellow vase+an orange vase+a purple vase+a pink vase+a black vase+a white vasediff --git a/assets/texts/VBench/prompts_per_dimension/human_action.txt b/assets/texts/VBench/prompts_per_dimension/human_action.txtnew file mode 100755index 0000000..2abd9a2--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/human_action.txt@@ -0,0 +1,100 @@+A person is riding a bike+A person is marching+A person is roller skating+A person is tasting beer+A person is clapping+A person is drawing+A person is petting animal (not cat)+A person is eating watermelon+A person is playing harp+A person is wrestling+A person is riding scooter+A person is sweeping floor+A person is skateboarding+A person is dunking basketball+A person is playing flute+A person is stretching leg+A person is tying tie+A person is skydiving+A person is shooting goal (soccer)+A person is playing piano+A person is finger snapping+A person is canoeing or kayaking+A person is laughing+A person is digging+A person is clay pottery making+A person is shooting basketball+A person is bending back+A person is shaking hands+A person is bandaging+A person is push up+A person is catching or throwing frisbee+A person is playing trumpet+A person is flying kite+A person is filling eyebrows+A person is shuffling cards+A person is folding clothes+A person is smoking+A person is tai chi+A person is squat+A person is playing controller+A person is throwing axe+A person is giving or receiving award+A person is air drumming+A person is taking a shower+A person is planting trees+A person is sharpening knives+A person is robot dancing+A person is rock climbing+A person is hula hooping+A person is writing+A person is bungee jumping+A person is pushing cart+A person is cleaning windows+A person is cutting watermelon+A person is cheerleading+A person is washing hands+A person is ironing+A person is cutting nails+A person is hugging+A person is trimming or shaving beard+A person is jogging+A person is making bed+A person is washing dishes+A person is grooming dog+A person is doing laundry+A person is knitting+A person is reading book+A person is baby waking up+A person is massaging legs+A person is brushing teeth+A person is crawling baby+A person is motorcycling+A person is driving car+A person is sticking tongue out+A person is shaking head+A person is sword fighting+A person is doing aerobics+A person is strumming guitar+A person is riding or walking with horse+A person is archery+A person is catching or throwing baseball+A person is playing chess+A person is rock scissors paper+A person is using computer+A person is arranging flowers+A person is bending metal+A person is ice skating+A person is climbing a rope+A person is crying+A person is dancing ballet+A person is getting a haircut+A person is running on treadmill+A person is kissing+A person is counting money+A person is barbequing+A person is peeling apples+A person is milking cow+A person is shining shoes+A person is making snowman+A person is sailingdiff --git a/assets/texts/VBench/prompts_per_dimension/multiple_objects.txt b/assets/texts/VBench/prompts_per_dimension/multiple_objects.txtnew file mode 100755index 0000000..1a1211e--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/multiple_objects.txt@@ -0,0 +1,82 @@+a bird and a cat+a cat and a dog+a dog and a horse+a horse and a sheep+a sheep and a cow+a cow and an elephant+an elephant and a bear+a bear and a zebra+a zebra and a giraffe+a giraffe and a bird+a chair and a couch+a couch and a potted plant+a potted plant and a tv+a tv and a laptop+a laptop and a remote+a remote and a keyboard+a keyboard and a cell phone+a cell phone and a book+a book and a clock+a clock and a backpack+a backpack and an umbrella+an umbrella and a handbag+a handbag and a tie+a tie and a suitcase+a suitcase and a vase+a vase and scissors+scissors and a teddy bear+a teddy bear and a frisbee+a frisbee and skis+skis and a snowboard+a snowboard and a sports ball+a sports ball and a kite+a kite and a baseball bat+a baseball bat and a baseball glove+a baseball glove and a skateboard+a skateboard and a surfboard+a surfboard and a tennis racket+a tennis racket and a bottle+a bottle and a chair+an airplane and a train+a train and a boat+a boat and an airplane+a bicycle and a car+a car and a motorcycle+a motorcycle and a bus+a bus and a traffic light+a traffic light and a fire hydrant+a fire hydrant and a stop sign+a stop sign and a parking meter+a parking meter and a truck+a truck and a bicycle+a toilet and a hair drier+a hair drier and a toothbrush+a toothbrush and a sink+a sink and a toilet+a wine glass and a chair+a cup and a couch+a fork and a potted plant+a knife and a tv+a spoon and a laptop+a bowl and a remote+a banana and a keyboard+an apple and a cell phone+a sandwich and a book+an orange and a clock+broccoli and a backpack+a carrot and an umbrella+a hot dog and a handbag+a pizza and a tie+a donut and a suitcase+a cake and a vase+an oven and scissors+a toaster and a teddy bear+a microwave and a frisbee+a refrigerator and skis+a bicycle and an airplane+a car and a train+a motorcycle and a boat+a person and a toilet+a person and a hair drier+a person and a toothbrush+a person and a sinkdiff --git a/assets/texts/VBench/prompts_per_dimension/object_class.txt b/assets/texts/VBench/prompts_per_dimension/object_class.txtnew file mode 100755index 0000000..11aa72d--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/object_class.txt@@ -0,0 +1,79 @@+a person+a bicycle+a car+a motorcycle+an airplane+a bus+a train+a truck+a boat+a traffic light+a fire hydrant+a stop sign+a parking meter+a bench+a bird+a cat+a dog+a horse+a sheep+a cow+an elephant+a bear+a zebra+a giraffe+a backpack+an umbrella+a handbag+a tie+a suitcase+a frisbee+skis+a snowboard+a sports ball+a kite+a baseball bat+a baseball glove+a skateboard+a surfboard+a tennis racket+a bottle+a wine glass+a cup+a fork+a knife+a spoon+a bowl+a banana+an apple+a sandwich+an orange+broccoli+a carrot+a hot dog+a pizza+a donut+a cake+a chair+a couch+a potted plant+a bed+a dining table+a toilet+a tv+a laptop+a remote+a keyboard+a cell phone+a microwave+an oven+a toaster+a sink+a refrigerator+a book+a clock+a vase+scissors+a teddy bear+a hair drier+a toothbrushdiff --git a/assets/texts/VBench/prompts_per_dimension/overall_consistency.txt b/assets/texts/VBench/prompts_per_dimension/overall_consistency.txtnew file mode 100755index 0000000..360c267--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/overall_consistency.txt@@ -0,0 +1,93 @@+Close up of grapes on a rotating table.+Turtle swimming in ocean.+A storm trooper vacuuming the beach.+A panda standing on a surfboard in the ocean in sunset.+An astronaut feeding ducks on a sunny afternoon, reflection from the water.+Two pandas discussing an academic paper.+Sunset time lapse at the beach with moving clouds and colors in the sky.+A fat rabbit wearing a purple robe walking through a fantasy landscape.+A koala bear playing piano in the forest.+An astronaut flying in space.+Fireworks.+An animated painting of fluffy white clouds moving in sky.+Flying through fantasy landscapes.+A bigfoot walking in the snowstorm.+A squirrel eating a burger.+A cat wearing sunglasses and working as a lifeguard at a pool.+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks.+Splash of turquoise water in extreme slow motion, alpha channel included.+an ice cream is melting on the table.+a drone flying over a snowy forest.+a shark is swimming in the ocean.+Aerial panoramic video from a drone of a fantasy land.+a teddy bear is swimming in the ocean.+time lapse of sunrise on mars.+golden fish swimming in the ocean.+An artist brush painting on a canvas close up.+A drone view of celebration with Christmas tree and fireworks, starry sky - background.+happy dog wearing a yellow turtleneck, studio, portrait, facing camera, dark background+Origami dancers in white paper, 3D render, on white background, studio shot, dancing modern dance.+Campfire at night in a snowy forest with starry sky in the background.+a fantasy landscape+A 3D model of a 1800s victorian house.+this is how I do makeup in the morning.+A raccoon that looks like a turtle, digital art.+Robot dancing in Times Square.+Busy freeway at night.+Balloon full of water exploding in extreme slow motion.+An astronaut is riding a horse in the space in a photorealistic style.+Macro slo-mo. Slow motion cropped closeup of roasted coffee beans falling into an empty bowl.+Sewing machine, old sewing machine working.+Motion colour drop in water, ink swirling in water, colourful ink in water, abstraction fancy dream cloud of ink.+Few big purple plums rotating on the turntable. water drops appear on the skin during rotation. isolated on the white background. close-up. macro.+Vampire makeup face of beautiful girl, red contact lenses.+Ashtray full of butts on table, smoke flowing on black background, close-up+Pacific coast, carmel by the sea ocean and waves.+A teddy bear is playing drum kit in NYC Times Square.+A corgi is playing drum kit.+An Iron man is playing the electronic guitar, high electronic guitar.+A raccoon is playing the electronic guitar.+A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh+A corgi's head depicted as an explosion of a nebula+A fantasy landscape+A future where humans have achieved teleportation technology+A jellyfish floating through the ocean, with bioluminescent tentacles+A Mars rover moving on Mars+A panda drinking coffee in a cafe in Paris+A space shuttle launching into orbit, with flames and smoke billowing out from the engines+A steam train moving on a mountainside+A super cool giant robot in Cyberpunk Beijing+A tropical beach at sunrise, with palm trees and crystal-clear water in the foreground+Cinematic shot of Van Gogh's selfie, Van Gogh style+Gwen Stacy reading a book+Iron Man flying in the sky+The bund Shanghai, oil painting+Yoda playing guitar on the stage+A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo+A beautiful coastal beach in spring, waves lapping on sand by Vincent van Gogh+A boat sailing leisurely along the Seine River with the Eiffel Tower in background+A car moving slowly on an empty street, rainy evening+A cat eating food out of a bowl+A cat wearing sunglasses at a pool+A confused panda in calculus class+A cute fluffy panda eating Chinese food in a restaurant+A cute happy Corgi playing in park, sunset+A cute raccoon playing guitar in a boat on the ocean+A happy fuzzy panda playing guitar nearby a campfire, snow mountain in the background+A lightning striking atop of eiffel tower, dark clouds in the sky+A modern art museum, with colorful paintings+A panda cooking in the kitchen+A panda playing on a swing set+A polar bear is playing guitar+A raccoon dressed in suit playing the trumpet, stage background+A robot DJ is playing the turntable, in heavy raining futuristic tokyo rooftop cyberpunk night, sci-fi, fantasy+A shark swimming in clear Caribbean ocean+A super robot protecting city+A teddy bear washing the dishes+An epic tornado attacking above a glowing city at night, the tornado is made of smoke+An oil painting of a couple in formal evening wear going home get caught in a heavy downpour with umbrellas+Clown fish swimming through the coral reef+Hyper-realistic spaceship landing on Mars+The bund Shanghai, vibrant color+Vincent van Gogh is painting in the room+Yellow flowers swing in the winddiff --git a/assets/texts/VBench/prompts_per_dimension/scene.txt b/assets/texts/VBench/prompts_per_dimension/scene.txtnew file mode 100755index 0000000..1ec0366--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/scene.txt@@ -0,0 +1,86 @@+alley+amusement park+aquarium+arch+art gallery+bathroom+bakery shop+ballroom+bar+barn+basement+beach+bedroom+bridge+botanical garden+cafeteria+campsite+campus+carrousel+castle+cemetery+classroom+cliff+crosswalk+construction site+corridor+courtyard+desert+downtown+driveway+farm+food court+football field+forest road+fountain+gas station+glacier+golf course+indoor gymnasium+harbor+highway+hospital+house+iceberg+industrial area+jail cell+junkyard+kitchen+indoor library+lighthouse+laboratory+mansion+marsh+mountain+indoor movie theater+indoor museum+music studio+nursery+ocean+office+palace+parking lot+pharmacy+phone booth+raceway+restaurant+river+science museum+shower+ski slope+sky+skyscraper+baseball stadium+staircase+street+supermarket+indoor swimming pool+tower+outdoor track+train railway+train station platform+underwater coral reef+valley+volcano+waterfall+windmilldiff --git a/assets/texts/VBench/prompts_per_dimension/spatial_relationship.txt b/assets/texts/VBench/prompts_per_dimension/spatial_relationship.txtnew file mode 100755index 0000000..25fe959--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/spatial_relationship.txt@@ -0,0 +1,84 @@+a bicycle on the left of a car, front view+a car on the right of a motorcycle, front view+a motorcycle on the left of a bus, front view+a bus on the right of a traffic light, front view+a traffic light on the left of a fire hydrant, front view+a fire hydrant on the right of a stop sign, front view+a stop sign on the left of a parking meter, front view+a parking meter on the right of a bench, front view+a bench on the left of a truck, front view+a truck on the right of a bicycle, front view+a bird on the left of a cat, front view+a cat on the right of a dog, front view+a dog on the left of a horse, front view+a horse on the right of a sheep, front view+a sheep on the left of a cow, front view+a cow on the right of an elephant, front view+an elephant on the left of a bear, front view+a bear on the right of a zebra, front view+a zebra on the left of a giraffe, front view+a giraffe on the right of a bird, front view+a bottle on the left of a wine glass, front view+a wine glass on the right of a cup, front view+a cup on the left of a fork, front view+a fork on the right of a knife, front view+a knife on the left of a spoon, front view+a spoon on the right of a bowl, front view+a bowl on the left of a bottle, front view+a potted plant on the left of a remote, front view+a remote on the right of a clock, front view+a clock on the left of a vase, front view+a vase on the right of scissors, front view+scissors on the left of a teddy bear, front view+a teddy bear on the right of a potted plant, front view+a frisbee on the left of a sports ball, front view+a sports ball on the right of a baseball bat, front view+a baseball bat on the left of a baseball glove, front view+a baseball glove on the right of a tennis racket, front view+a tennis racket on the left of a frisbee, front view+a toilet on the left of a hair drier, front view+a hair drier on the right of a toothbrush, front view+a toothbrush on the left of a sink, front view+a sink on the right of a toilet, front view+a chair on the left of a couch, front view+a couch on the right of a bed, front view+a bed on the left of a tv, front view+a tv on the right of a dining table, front view+a dining table on the left of a chair, front view+an airplane on the left of a train, front view+a train on the right of a boat, front view+a boat on the left of an airplane, front view+an oven on the top of a toaster, front view+an oven on the bottom of a toaster, front view+a toaster on the top of a microwave, front view+a toaster on the bottom of a microwave, front view+a microwave on the top of an oven, front view+a microwave on the bottom of an oven, front view+a banana on the top of an apple, front view+a banana on the bottom of an apple, front view+an apple on the top of a sandwich, front view+an apple on the bottom of a sandwich, front view+a sandwich on the top of an orange, front view+a sandwich on the bottom of an orange, front view+an orange on the top of a carrot, front view+an orange on the bottom of a carrot, front view+a carrot on the top of a hot dog, front view+a carrot on the bottom of a hot dog, front view+a hot dog on the top of a pizza, front view+a hot dog on the bottom of a pizza, front view+a pizza on the top of a donut, front view+a pizza on the bottom of a donut, front view+a donut on the top of broccoli, front view+a donut on the bottom of broccoli, front view+broccoli on the top of a banana, front view+broccoli on the bottom of a banana, front view+skis on the top of a snowboard, front view+skis on the bottom of a snowboard, front view+a snowboard on the top of a kite, front view+a snowboard on the bottom of a kite, front view+a kite on the top of a skateboard, front view+a kite on the bottom of a skateboard, front view+a skateboard on the top of a surfboard, front view+a skateboard on the bottom of a surfboard, front view+a surfboard on the top of skis, front view+a surfboard on the bottom of skis, front viewdiff --git a/assets/texts/VBench/prompts_per_dimension/subject_consistency.txt b/assets/texts/VBench/prompts_per_dimension/subject_consistency.txtnew file mode 100755index 0000000..97cb77e--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/subject_consistency.txt@@ -0,0 +1,72 @@+a person swimming in ocean+a person giving a presentation to a room full of colleagues+a person washing the dishes+a person eating a burger+a person walking in the snowstorm+a person drinking coffee in a cafe+a person playing guitar+a bicycle leaning against a tree+a bicycle gliding through a snowy field+a bicycle slowing down to stop+a bicycle accelerating to gain speed+a car stuck in traffic during rush hour+a car turning a corner+a car slowing down to stop+a car accelerating to gain speed+a motorcycle cruising along a coastal highway+a motorcycle turning a corner+a motorcycle slowing down to stop+a motorcycle gliding through a snowy field+a motorcycle accelerating to gain speed+an airplane soaring through a clear blue sky+an airplane taking off+an airplane landing smoothly on a runway+an airplane accelerating to gain speed+a bus turning a corner+a bus stuck in traffic during rush hour+a bus accelerating to gain speed+a train speeding down the tracks+a train crossing over a tall bridge+a train accelerating to gain speed+a truck turning a corner+a truck anchored in a tranquil bay+a truck stuck in traffic during rush hour+a truck slowing down to stop+a truck accelerating to gain speed+a boat sailing smoothly on a calm lake+a boat slowing down to stop+a boat accelerating to gain speed+a bird soaring gracefully in the sky+a bird building a nest from twigs and leaves+a bird flying over a snowy forest+a cat grooming itself meticulously with its tongue+a cat playing in park+a cat drinking water+a cat running happily+a dog enjoying a peaceful walk+a dog playing in park+a dog drinking water+a dog running happily+a horse bending down to drink water from a river+a horse galloping across an open field+a horse taking a peaceful walk+a horse running to join a herd of its kind+a sheep bending down to drink water from a river+a sheep taking a peaceful walk+a sheep running to join a herd of its kind+a cow bending down to drink water from a river+a cow chewing cud while resting in a tranquil barn+a cow running to join a herd of its kind+an elephant spraying itself with water using its trunk to cool down+an elephant taking a peaceful walk+an elephant running to join a herd of its kind+a bear catching a salmon in its powerful jaws+a bear sniffing the air for scents of food+a bear climbing a tree+a bear hunting for prey+a zebra bending down to drink water from a river+a zebra running to join a herd of its kind+a zebra taking a peaceful walk+a giraffe bending down to drink water from a river+a giraffe taking a peaceful walk+a giraffe running to join a herd of its kinddiff --git a/assets/texts/VBench/prompts_per_dimension/temporal_flickering.txt b/assets/texts/VBench/prompts_per_dimension/temporal_flickering.txtnew file mode 100755index 0000000..9fb5cad--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/temporal_flickering.txt@@ -0,0 +1,75 @@+In a still frame, a stop sign+a toilet, frozen in time+a laptop, frozen in time+A tranquil tableau of alley+A tranquil tableau of bar+A tranquil tableau of barn+A tranquil tableau of bathroom+A tranquil tableau of bedroom+A tranquil tableau of cliff+In a still frame, courtyard+In a still frame, gas station+A tranquil tableau of house+indoor gymnasium, frozen in time+A tranquil tableau of indoor library+A tranquil tableau of kitchen+A tranquil tableau of palace+In a still frame, parking lot+In a still frame, phone booth+A tranquil tableau of restaurant+A tranquil tableau of tower+A tranquil tableau of a bowl+A tranquil tableau of an apple+A tranquil tableau of a bench+A tranquil tableau of a bed+A tranquil tableau of a chair+A tranquil tableau of a cup+A tranquil tableau of a dining table+In a still frame, a pear+A tranquil tableau of a bunch of grapes+A tranquil tableau of a bowl on the kitchen counter+A tranquil tableau of a beautiful, handcrafted ceramic bowl+A tranquil tableau of an antique bowl+A tranquil tableau of an exquisite mahogany dining table+A tranquil tableau of a wooden bench in the park+A tranquil tableau of a beautiful wrought-iron bench surrounded by blooming flowers+In a still frame, a park bench with a view of the lake+A tranquil tableau of a vintage rocking chair was placed on the porch+A tranquil tableau of the jail cell was small and dimly lit, with cold, steel bars+A tranquil tableau of the phone booth was tucked away in a quiet alley+a dilapidated phone booth stood as a relic of a bygone era on the sidewalk, frozen in time+A tranquil tableau of the old red barn stood weathered and iconic against the backdrop of the countryside+A tranquil tableau of a picturesque barn was painted a warm shade of red and nestled in a picturesque meadow+In a still frame, within the desolate desert, an oasis unfolded, characterized by the stoic presence of palm trees and a motionless, glassy pool of water+In a still frame, the Parthenon's majestic Doric columns stand in serene solitude atop the Acropolis, framed by the tranquil Athenian landscape+In a still frame, the Temple of Hephaestus, with its timeless Doric grace, stands stoically against the backdrop of a quiet Athens+In a still frame, the ornate Victorian streetlamp stands solemnly, adorned with intricate ironwork and stained glass panels+A tranquil tableau of the Stonehenge presented itself as an enigmatic puzzle, each colossal stone meticulously placed against the backdrop of tranquility+In a still frame, in the vast desert, an oasis nestled among dunes, featuring tall palm trees and an air of serenity+static view on a desert scene with an oasis, palm trees, and a clear, calm pool of water+A tranquil tableau of an ornate Victorian streetlamp standing on a cobblestone street corner, illuminating the empty night+A tranquil tableau of a tranquil lakeside cabin nestled among tall pines, its reflection mirrored perfectly in the calm water+In a still frame, a vintage gas lantern, adorned with intricate details, gracing a historic cobblestone square+In a still frame, a tranquil Japanese tea ceremony room, with tatami mats, a delicate tea set, and a bonsai tree in the corner+A tranquil tableau of the Parthenon stands resolute in its classical elegance, a timeless symbol of Athens' cultural legacy+A tranquil tableau of in the heart of Plaka, the neoclassical architecture of the old city harmonizes with the ancient ruins+A tranquil tableau of in the desolate beauty of the American Southwest, Chaco Canyon's ancient ruins whispered tales of an enigmatic civilization that once thrived amidst the arid landscapes+A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades+In a still frame, amidst the cobblestone streets, an Art Nouveau lamppost stood tall+A tranquil tableau of in the quaint village square, a traditional wrought-iron streetlamp featured delicate filigree patterns and amber-hued glass panels+A tranquil tableau of the lampposts were adorned with Art Deco motifs, their geometric shapes and frosted glass creating a sense of vintage glamour+In a still frame, in the picturesque square, a Gothic-style lamppost adorned with intricate stone carvings added a touch of medieval charm to the setting+In a still frame, in the heart of the old city, a row of ornate lantern-style streetlamps bathed the narrow alleyway in a warm, welcoming light+A tranquil tableau of in the heart of the Utah desert, a massive sandstone arch spanned the horizon+A tranquil tableau of in the Arizona desert, a massive stone bridge arched across a rugged canyon+A tranquil tableau of in the corner of the minimalist tea room, a bonsai tree added a touch of nature's beauty to the otherwise simple and elegant space+In a still frame, amidst the hushed ambiance of the traditional tea room, a meticulously arranged tea set awaited, with porcelain cups, a bamboo whisk+In a still frame, nestled in the Zen garden, a rustic teahouse featured tatami seating and a traditional charcoal brazier+A tranquil tableau of a country estate's library featured elegant wooden shelves+A tranquil tableau of beneath the shade of a solitary oak tree, an old wooden park bench sat patiently+A tranquil tableau of beside a tranquil pond, a weeping willow tree draped its branches gracefully over the water's surface, creating a serene tableau of reflection and calm+A tranquil tableau of in the Zen garden, a perfectly raked gravel path led to a serene rock garden+In a still frame, a tranquil pond was fringed by weeping cherry trees, their blossoms drifting lazily onto the glassy surface+In a still frame, within the historic library's reading room, rows of antique leather chairs and mahogany tables offered a serene haven for literary contemplation+A tranquil tableau of a peaceful orchid garden showcased a variety of delicate blooms+A tranquil tableau of in the serene courtyard, a centuries-old stone well stood as a symbol of a bygone era, its mossy stones bearing witness to the passage of timediff --git a/assets/texts/VBench/prompts_per_dimension/temporal_style.txt b/assets/texts/VBench/prompts_per_dimension/temporal_style.txtnew file mode 100755index 0000000..fea23cb--- /dev/null+++ b/assets/texts/VBench/prompts_per_dimension/temporal_style.txt@@ -0,0 +1,100 @@+A beautiful coastal beach in spring, waves lapping on sand, in super slow motion+A beautiful coastal beach in spring, waves lapping on sand, zoom in+A beautiful coastal beach in spring, waves lapping on sand, zoom out+A beautiful coastal beach in spring, waves lapping on sand, pan left+A beautiful coastal beach in spring, waves lapping on sand, pan right+A beautiful coastal beach in spring, waves lapping on sand, tilt up+A beautiful coastal beach in spring, waves lapping on sand, tilt down+A beautiful coastal beach in spring, waves lapping on sand, with an intense shaking effect+A beautiful coastal beach in spring, waves lapping on sand, featuring a steady and smooth perspective+A beautiful coastal beach in spring, waves lapping on sand, racking focus+The bund Shanghai, in super slow motion+The bund Shanghai, zoom in+The bund Shanghai, zoom out+The bund Shanghai, pan left+The bund Shanghai, pan right+The bund Shanghai, tilt up+The bund Shanghai, tilt down+The bund Shanghai, with an intense shaking effect+The bund Shanghai, featuring a steady and smooth perspective+The bund Shanghai, racking focus+a shark is swimming in the ocean, in super slow motion+a shark is swimming in the ocean, zoom in+a shark is swimming in the ocean, zoom out+a shark is swimming in the ocean, pan left+a shark is swimming in the ocean, pan right+a shark is swimming in the ocean, tilt up+a shark is swimming in the ocean, tilt down+a shark is swimming in the ocean, with an intense shaking effect+a shark is swimming in the ocean, featuring a steady and smooth perspective+a shark is swimming in the ocean, racking focus+A panda drinking coffee in a cafe in Paris, in super slow motion+A panda drinking coffee in a cafe in Paris, zoom in+A panda drinking coffee in a cafe in Paris, zoom out+A panda drinking coffee in a cafe in Paris, pan left+A panda drinking coffee in a cafe in Paris, pan right+A panda drinking coffee in a cafe in Paris, tilt up+A panda drinking coffee in a cafe in Paris, tilt down+A panda drinking coffee in a cafe in Paris, with an intense shaking effect+A panda drinking coffee in a cafe in Paris, featuring a steady and smooth perspective+A panda drinking coffee in a cafe in Paris, racking focus+A cute happy Corgi playing in park, sunset, in super slow motion+A cute happy Corgi playing in park, sunset, zoom in+A cute happy Corgi playing in park, sunset, zoom out+A cute happy Corgi playing in park, sunset, pan left+A cute happy Corgi playing in park, sunset, pan right+A cute happy Corgi playing in park, sunset, tilt up+A cute happy Corgi playing in park, sunset, tilt down+A cute happy Corgi playing in park, sunset, with an intense shaking effect+A cute happy Corgi playing in park, sunset, featuring a steady and smooth perspective+A cute happy Corgi playing in park, sunset, racking focus+Gwen Stacy reading a book, in super slow motion+Gwen Stacy reading a book, zoom in+Gwen Stacy reading a book, zoom out+Gwen Stacy reading a book, pan left+Gwen Stacy reading a book, pan right+Gwen Stacy reading a book, tilt up+Gwen Stacy reading a book, tilt down+Gwen Stacy reading a book, with an intense shaking effect+Gwen Stacy reading a book, featuring a steady and smooth perspective+Gwen Stacy reading a book, racking focus+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, in super slow motion+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, zoom in+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, zoom out+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pan left+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pan right+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, tilt up+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, tilt down+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, with an intense shaking effect+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, featuring a steady and smooth perspective+A boat sailing leisurely along the Seine River with the Eiffel Tower in background, racking focus+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, in super slow motion+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, zoom in+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, zoom out+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pan left+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pan right+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, tilt up+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, tilt down+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, with an intense shaking effect+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, featuring a steady and smooth perspective+A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, racking focus+An astronaut flying in space, in super slow motion+An astronaut flying in space, zoom in+An astronaut flying in space, zoom out+An astronaut flying in space, pan left+An astronaut flying in space, pan right+An astronaut flying in space, tilt up+An astronaut flying in space, tilt down+An astronaut flying in space, with an intense shaking effect+An astronaut flying in space, featuring a steady and smooth perspective+An astronaut flying in space, racking focus+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, in super slow motion+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, zoom in+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, zoom out+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pan left+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pan right+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, tilt up+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, tilt down+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, with an intense shaking effect+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, featuring a steady and smooth perspective+Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, racking focusdiff --git a/configs/opensora/inference-long/16x256x256-sdedit.py b/configs/opensora/inference-long/16x256x256-sdedit.pynew file mode 100644index 0000000..4447455--- /dev/null+++ b/configs/opensora/inference-long/16x256x256-sdedit.py@@ -0,0 +1,67 @@+# scripts/inference_long.py+num_frames = 16+fps = 24 // 3+image_size = (256, 256)++# Define model+model = dict(+ type="STDiT-XL/2",+ space_scale=0.5,+ time_scale=1.0,+ enable_flashattn=True,+ enable_layernorm_kernel=True,+ from_pretrained=None,+)+vae = dict(+ type="VideoAutoencoderKL",+ from_pretrained="stabilityai/sd-vae-ft-ema",+ micro_batch_size=4,+)+text_encoder = dict(+ type="t5",+ from_pretrained="DeepFloyd/t5-v1_1-xxl",+ model_max_length=120,+)+scheduler = dict(+ type="iddpm",+ # type="dpm-solver",+ num_sampling_steps=100,+ cfg_scale=7.0,+)+dtype = "fp16"++# Condition+prompt_path = None+prompt = [+ # "Drone view of waves crashing against the rugged cliffs along Big Sur’s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff’s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff's edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.",+ "In an ornate, historical hall, a massive tidal wave peaks and begins to crash. Two surfers, seizing the moment, skillfully navigate the face of the wave.",+ # "put the video in space with a rainbow road",+ # "make it have dinosaurs",+ # "make it in claymation animation style",+ # "make it go underwater"+]++loop = 1+condition_frame_length = 4+reference_path = [+ # "assets/images/condition/cliff.png",+ "assets/images/condition/wave.png",+ # "/home/zhouyukun/Open-Sora-dev/assets/videos/base.mp4",+ # "/home/zhouyukun/Open-Sora-dev/assets/videos/base.mp4",+ # "/home/zhouyukun/Open-Sora-dev/assets/videos/base.mp4",+ # "/home/zhouyukun/Open-Sora-dev/assets/videos/base.mp4",+]+mask_strategy = [+ # "0,0,0,1,0,0",+ "0,0,0,1,0,0",+ # "0,0,0,12,0,0", # 噪声率+ # "0,0,0,12,0,0", # 噪声率+ # "0,0,0,12,0,0", # 噪声率+ # "0,0,0,12,0,0", # 噪声率+] # valid when reference_path is not None+# (loop id, ref id, ref start, length, target start)++# Others+batch_size = len(prompt)+seed = 42+save_dir = "outputs/SDEdit"diff --git a/eval/README.md b/eval/README.mdnew file mode 100644index 0000000..e69de29diff --git a/eval/VBench_full_info.json b/eval/VBench_full_info.jsonnew file mode 100755index 0000000..e60c40e--- /dev/null+++ b/eval/VBench_full_info.json@@ -0,0 +1,9132 @@+[+ {+ "prompt_en": "In a still frame, a stop sign",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "a toilet, frozen in time",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "a laptop, frozen in time",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of alley",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of bar",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of barn",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of bathroom",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of bedroom",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of cliff",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, courtyard",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, gas station",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of house",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "indoor gymnasium, frozen in time",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of indoor library",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of kitchen",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of palace",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, parking lot",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, phone booth",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of restaurant",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of tower",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a bowl",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of an apple",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a bench",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a bed",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a chair",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a cup",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a dining table",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, a pear",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a bunch of grapes",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a bowl on the kitchen counter",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a beautiful, handcrafted ceramic bowl",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of an antique bowl",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of an exquisite mahogany dining table",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a wooden bench in the park",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a beautiful wrought-iron bench surrounded by blooming flowers",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, a park bench with a view of the lake",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a vintage rocking chair was placed on the porch",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of the jail cell was small and dimly lit, with cold, steel bars",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of the phone booth was tucked away in a quiet alley",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "a dilapidated phone booth stood as a relic of a bygone era on the sidewalk, frozen in time",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of the old red barn stood weathered and iconic against the backdrop of the countryside",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a picturesque barn was painted a warm shade of red and nestled in a picturesque meadow",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, within the desolate desert, an oasis unfolded, characterized by the stoic presence of palm trees and a motionless, glassy pool of water",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, the Parthenon's majestic Doric columns stand in serene solitude atop the Acropolis, framed by the tranquil Athenian landscape",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, the Temple of Hephaestus, with its timeless Doric grace, stands stoically against the backdrop of a quiet Athens",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, the ornate Victorian streetlamp stands solemnly, adorned with intricate ironwork and stained glass panels",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of the Stonehenge presented itself as an enigmatic puzzle, each colossal stone meticulously placed against the backdrop of tranquility",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, in the vast desert, an oasis nestled among dunes, featuring tall palm trees and an air of serenity",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "static view on a desert scene with an oasis, palm trees, and a clear, calm pool of water",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of an ornate Victorian streetlamp standing on a cobblestone street corner, illuminating the empty night",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a tranquil lakeside cabin nestled among tall pines, its reflection mirrored perfectly in the calm water",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, a vintage gas lantern, adorned with intricate details, gracing a historic cobblestone square",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, a tranquil Japanese tea ceremony room, with tatami mats, a delicate tea set, and a bonsai tree in the corner",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of the Parthenon stands resolute in its classical elegance, a timeless symbol of Athens' cultural legacy",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the heart of Plaka, the neoclassical architecture of the old city harmonizes with the ancient ruins",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the desolate beauty of the American Southwest, Chaco Canyon's ancient ruins whispered tales of an enigmatic civilization that once thrived amidst the arid landscapes",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved fa\u00e7ades",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, amidst the cobblestone streets, an Art Nouveau lamppost stood tall",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the quaint village square, a traditional wrought-iron streetlamp featured delicate filigree patterns and amber-hued glass panels",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of the lampposts were adorned with Art Deco motifs, their geometric shapes and frosted glass creating a sense of vintage glamour",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, in the picturesque square, a Gothic-style lamppost adorned with intricate stone carvings added a touch of medieval charm to the setting",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, in the heart of the old city, a row of ornate lantern-style streetlamps bathed the narrow alleyway in a warm, welcoming light",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the heart of the Utah desert, a massive sandstone arch spanned the horizon",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the Arizona desert, a massive stone bridge arched across a rugged canyon",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the corner of the minimalist tea room, a bonsai tree added a touch of nature's beauty to the otherwise simple and elegant space",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, amidst the hushed ambiance of the traditional tea room, a meticulously arranged tea set awaited, with porcelain cups, a bamboo whisk",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, nestled in the Zen garden, a rustic teahouse featured tatami seating and a traditional charcoal brazier",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a country estate's library featured elegant wooden shelves",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of beneath the shade of a solitary oak tree, an old wooden park bench sat patiently",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of beside a tranquil pond, a weeping willow tree draped its branches gracefully over the water's surface, creating a serene tableau of reflection and calm",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the Zen garden, a perfectly raked gravel path led to a serene rock garden",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, a tranquil pond was fringed by weeping cherry trees, their blossoms drifting lazily onto the glassy surface",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "In a still frame, within the historic library's reading room, rows of antique leather chairs and mahogany tables offered a serene haven for literary contemplation",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of a peaceful orchid garden showcased a variety of delicate blooms",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "A tranquil tableau of in the serene courtyard, a centuries-old stone well stood as a symbol of a bygone era, its mossy stones bearing witness to the passage of time",+ "dimension": [+ "temporal_flickering"+ ]+ },+ {+ "prompt_en": "a bird and a cat",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bird and cat"+ }+ }+ },+ {+ "prompt_en": "a cat and a dog",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "cat and dog"+ }+ }+ },+ {+ "prompt_en": "a dog and a horse",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "dog and horse"+ }+ }+ },+ {+ "prompt_en": "a horse and a sheep",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "horse and sheep"+ }+ }+ },+ {+ "prompt_en": "a sheep and a cow",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "sheep and cow"+ }+ }+ },+ {+ "prompt_en": "a cow and an elephant",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "cow and elephant"+ }+ }+ },+ {+ "prompt_en": "an elephant and a bear",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "elephant and bear"+ }+ }+ },+ {+ "prompt_en": "a bear and a zebra",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bear and zebra"+ }+ }+ },+ {+ "prompt_en": "a zebra and a giraffe",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "zebra and giraffe"+ }+ }+ },+ {+ "prompt_en": "a giraffe and a bird",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "giraffe and bird"+ }+ }+ },+ {+ "prompt_en": "a chair and a couch",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "chair and couch"+ }+ }+ },+ {+ "prompt_en": "a couch and a potted plant",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "couch and potted plant"+ }+ }+ },+ {+ "prompt_en": "a potted plant and a tv",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "potted plant and tv"+ }+ }+ },+ {+ "prompt_en": "a tv and a laptop",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "tv and laptop"+ }+ }+ },+ {+ "prompt_en": "a laptop and a remote",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "laptop and remote"+ }+ }+ },+ {+ "prompt_en": "a remote and a keyboard",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "remote and keyboard"+ }+ }+ },+ {+ "prompt_en": "a keyboard and a cell phone",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "keyboard and cell phone"+ }+ }+ },+ {+ "prompt_en": "a cell phone and a book",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "cell phone and book"+ }+ }+ },+ {+ "prompt_en": "a book and a clock",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "book and clock"+ }+ }+ },+ {+ "prompt_en": "a clock and a backpack",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "clock and backpack"+ }+ }+ },+ {+ "prompt_en": "a backpack and an umbrella",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "backpack and umbrella"+ }+ }+ },+ {+ "prompt_en": "an umbrella and a handbag",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "umbrella and handbag"+ }+ }+ },+ {+ "prompt_en": "a handbag and a tie",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "handbag and tie"+ }+ }+ },+ {+ "prompt_en": "a tie and a suitcase",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "tie and suitcase"+ }+ }+ },+ {+ "prompt_en": "a suitcase and a vase",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "suitcase and vase"+ }+ }+ },+ {+ "prompt_en": "a vase and scissors",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "vase and scissors"+ }+ }+ },+ {+ "prompt_en": "scissors and a teddy bear",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "scissors and teddy bear"+ }+ }+ },+ {+ "prompt_en": "a teddy bear and a frisbee",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "teddy bear and frisbee"+ }+ }+ },+ {+ "prompt_en": "a frisbee and skis",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "frisbee and skis"+ }+ }+ },+ {+ "prompt_en": "skis and a snowboard",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "skis and snowboard"+ }+ }+ },+ {+ "prompt_en": "a snowboard and a sports ball",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "snowboard and sports ball"+ }+ }+ },+ {+ "prompt_en": "a sports ball and a kite",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "sports ball and kite"+ }+ }+ },+ {+ "prompt_en": "a kite and a baseball bat",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "kite and baseball bat"+ }+ }+ },+ {+ "prompt_en": "a baseball bat and a baseball glove",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "baseball bat and baseball glove"+ }+ }+ },+ {+ "prompt_en": "a baseball glove and a skateboard",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "baseball glove and skateboard"+ }+ }+ },+ {+ "prompt_en": "a skateboard and a surfboard",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "skateboard and surfboard"+ }+ }+ },+ {+ "prompt_en": "a surfboard and a tennis racket",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "surfboard and tennis racket"+ }+ }+ },+ {+ "prompt_en": "a tennis racket and a bottle",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "tennis racket and bottle"+ }+ }+ },+ {+ "prompt_en": "a bottle and a chair",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bottle and chair"+ }+ }+ },+ {+ "prompt_en": "an airplane and a train",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "airplane and train"+ }+ }+ },+ {+ "prompt_en": "a train and a boat",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "train and boat"+ }+ }+ },+ {+ "prompt_en": "a boat and an airplane",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "boat and airplane"+ }+ }+ },+ {+ "prompt_en": "a bicycle and a car",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bicycle and car"+ }+ }+ },+ {+ "prompt_en": "a car and a motorcycle",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "car and motorcycle"+ }+ }+ },+ {+ "prompt_en": "a motorcycle and a bus",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "motorcycle and bus"+ }+ }+ },+ {+ "prompt_en": "a bus and a traffic light",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bus and traffic light"+ }+ }+ },+ {+ "prompt_en": "a traffic light and a fire hydrant",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "traffic light and fire hydrant"+ }+ }+ },+ {+ "prompt_en": "a fire hydrant and a stop sign",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "fire hydrant and stop sign"+ }+ }+ },+ {+ "prompt_en": "a stop sign and a parking meter",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "stop sign and parking meter"+ }+ }+ },+ {+ "prompt_en": "a parking meter and a truck",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "parking meter and truck"+ }+ }+ },+ {+ "prompt_en": "a truck and a bicycle",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "truck and bicycle"+ }+ }+ },+ {+ "prompt_en": "a toilet and a hair drier",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "toilet and hair drier"+ }+ }+ },+ {+ "prompt_en": "a hair drier and a toothbrush",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "hair drier and toothbrush"+ }+ }+ },+ {+ "prompt_en": "a toothbrush and a sink",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "toothbrush and sink"+ }+ }+ },+ {+ "prompt_en": "a sink and a toilet",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "sink and toilet"+ }+ }+ },+ {+ "prompt_en": "a wine glass and a chair",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "wine glass and chair"+ }+ }+ },+ {+ "prompt_en": "a cup and a couch",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "cup and couch"+ }+ }+ },+ {+ "prompt_en": "a fork and a potted plant",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "fork and potted plant"+ }+ }+ },+ {+ "prompt_en": "a knife and a tv",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "knife and tv"+ }+ }+ },+ {+ "prompt_en": "a spoon and a laptop",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "spoon and laptop"+ }+ }+ },+ {+ "prompt_en": "a bowl and a remote",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bowl and remote"+ }+ }+ },+ {+ "prompt_en": "a banana and a keyboard",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "banana and keyboard"+ }+ }+ },+ {+ "prompt_en": "an apple and a cell phone",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "apple and cell phone"+ }+ }+ },+ {+ "prompt_en": "a sandwich and a book",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "sandwich and book"+ }+ }+ },+ {+ "prompt_en": "an orange and a clock",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "orange and clock"+ }+ }+ },+ {+ "prompt_en": "broccoli and a backpack",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "broccoli and backpack"+ }+ }+ },+ {+ "prompt_en": "a carrot and an umbrella",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "carrot and umbrella"+ }+ }+ },+ {+ "prompt_en": "a hot dog and a handbag",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "hot dog and handbag"+ }+ }+ },+ {+ "prompt_en": "a pizza and a tie",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "pizza and tie"+ }+ }+ },+ {+ "prompt_en": "a donut and a suitcase",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "donut and suitcase"+ }+ }+ },+ {+ "prompt_en": "a cake and a vase",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "cake and vase"+ }+ }+ },+ {+ "prompt_en": "an oven and scissors",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "oven and scissors"+ }+ }+ },+ {+ "prompt_en": "a toaster and a teddy bear",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "toaster and teddy bear"+ }+ }+ },+ {+ "prompt_en": "a microwave and a frisbee",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "microwave and frisbee"+ }+ }+ },+ {+ "prompt_en": "a refrigerator and skis",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "refrigerator and skis"+ }+ }+ },+ {+ "prompt_en": "a bicycle and an airplane",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "bicycle and airplane"+ }+ }+ },+ {+ "prompt_en": "a car and a train",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "car and train"+ }+ }+ },+ {+ "prompt_en": "a motorcycle and a boat",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "motorcycle and boat"+ }+ }+ },+ {+ "prompt_en": "a person and a toilet",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "person and toilet"+ }+ }+ },+ {+ "prompt_en": "a person and a hair drier",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "person and hair drier"+ }+ }+ },+ {+ "prompt_en": "a person and a toothbrush",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "person and toothbrush"+ }+ }+ },+ {+ "prompt_en": "a person and a sink",+ "dimension": [+ "multiple_objects"+ ],+ "auxiliary_info": {+ "multiple_objects": {+ "object": "person and sink"+ }+ }+ },+ {+ "prompt_en": "A person is riding a bike",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is marching",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is roller skating",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is tasting beer",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is clapping",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is drawing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is petting animal (not cat)",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is eating watermelon",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is playing harp",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is wrestling",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is riding scooter",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is sweeping floor",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is skateboarding",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is dunking basketball",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is playing flute",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is stretching leg",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is tying tie",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is skydiving",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is shooting goal (soccer)",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is playing piano",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is finger snapping",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is canoeing or kayaking",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is laughing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is digging",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is clay pottery making",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is shooting basketball",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is bending back",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is shaking hands",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is bandaging",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is push up",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is catching or throwing frisbee",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is playing trumpet",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is flying kite",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is filling eyebrows",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is shuffling cards",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is folding clothes",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is smoking",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is tai chi",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is squat",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is playing controller",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is throwing axe",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is giving or receiving award",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is air drumming",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is taking a shower",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is planting trees",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is sharpening knives",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is robot dancing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is rock climbing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is hula hooping",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is writing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is bungee jumping",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is pushing cart",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is cleaning windows",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is cutting watermelon",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is cheerleading",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is washing hands",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is ironing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is cutting nails",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is hugging",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is trimming or shaving beard",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is jogging",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is making bed",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is washing dishes",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is grooming dog",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is doing laundry",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is knitting",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is reading book",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is baby waking up",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is massaging legs",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is brushing teeth",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is crawling baby",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is motorcycling",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is driving car",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is sticking tongue out",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is shaking head",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is sword fighting",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is doing aerobics",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is strumming guitar",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is riding or walking with horse",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is archery",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is catching or throwing baseball",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is playing chess",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is rock scissors paper",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is using computer",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is arranging flowers",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is bending metal",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is ice skating",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is climbing a rope",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is crying",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is dancing ballet",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is getting a haircut",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is running on treadmill",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is kissing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is counting money",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is barbequing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is peeling apples",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is milking cow",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is shining shoes",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is making snowman",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "A person is sailing",+ "dimension": [+ "human_action"+ ]+ },+ {+ "prompt_en": "a person swimming in ocean",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person giving a presentation to a room full of colleagues",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person washing the dishes",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person eating a burger",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person walking in the snowstorm",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person drinking coffee in a cafe",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person playing guitar",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bicycle leaning against a tree",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bicycle gliding through a snowy field",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bicycle slowing down to stop",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bicycle accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a car stuck in traffic during rush hour",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a car turning a corner",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a car slowing down to stop",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a car accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a motorcycle cruising along a coastal highway",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a motorcycle turning a corner",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a motorcycle slowing down to stop",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a motorcycle gliding through a snowy field",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a motorcycle accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an airplane soaring through a clear blue sky",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an airplane taking off",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an airplane landing smoothly on a runway",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an airplane accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bus turning a corner",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bus stuck in traffic during rush hour",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bus accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a train speeding down the tracks",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a train crossing over a tall bridge",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a train accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a truck turning a corner",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a truck anchored in a tranquil bay",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a truck stuck in traffic during rush hour",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a truck slowing down to stop",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a truck accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a boat sailing smoothly on a calm lake",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a boat slowing down to stop",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a boat accelerating to gain speed",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bird soaring gracefully in the sky",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bird building a nest from twigs and leaves",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bird flying over a snowy forest",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cat grooming itself meticulously with its tongue",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cat playing in park",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cat drinking water",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cat running happily",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a dog enjoying a peaceful walk",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a dog playing in park",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a dog drinking water",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a dog running happily",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a horse bending down to drink water from a river",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a horse galloping across an open field",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a horse taking a peaceful walk",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a horse running to join a herd of its kind",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a sheep bending down to drink water from a river",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a sheep taking a peaceful walk",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a sheep running to join a herd of its kind",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cow bending down to drink water from a river",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cow chewing cud while resting in a tranquil barn",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a cow running to join a herd of its kind",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an elephant spraying itself with water using its trunk to cool down",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an elephant taking a peaceful walk",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "an elephant running to join a herd of its kind",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bear catching a salmon in its powerful jaws",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bear sniffing the air for scents of food",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bear climbing a tree",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a bear hunting for prey",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a zebra bending down to drink water from a river",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a zebra running to join a herd of its kind",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a zebra taking a peaceful walk",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a giraffe bending down to drink water from a river",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a giraffe taking a peaceful walk",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a giraffe running to join a herd of its kind",+ "dimension": [+ "subject_consistency",+ "dynamic_degree",+ "motion_smoothness"+ ]+ },+ {+ "prompt_en": "a person",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "person"+ }+ }+ },+ {+ "prompt_en": "a bicycle",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bicycle"+ }+ }+ },+ {+ "prompt_en": "a car",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "car"+ }+ }+ },+ {+ "prompt_en": "a motorcycle",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "motorcycle"+ }+ }+ },+ {+ "prompt_en": "an airplane",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "airplane"+ }+ }+ },+ {+ "prompt_en": "a bus",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bus"+ }+ }+ },+ {+ "prompt_en": "a train",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "train"+ }+ }+ },+ {+ "prompt_en": "a truck",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "truck"+ }+ }+ },+ {+ "prompt_en": "a boat",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "boat"+ }+ }+ },+ {+ "prompt_en": "a traffic light",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "traffic light"+ }+ }+ },+ {+ "prompt_en": "a fire hydrant",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "fire hydrant"+ }+ }+ },+ {+ "prompt_en": "a stop sign",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "stop sign"+ }+ }+ },+ {+ "prompt_en": "a parking meter",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "parking meter"+ }+ }+ },+ {+ "prompt_en": "a bench",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bench"+ }+ }+ },+ {+ "prompt_en": "a bird",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bird"+ }+ }+ },+ {+ "prompt_en": "a cat",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "cat"+ }+ }+ },+ {+ "prompt_en": "a dog",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "dog"+ }+ }+ },+ {+ "prompt_en": "a horse",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "horse"+ }+ }+ },+ {+ "prompt_en": "a sheep",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "sheep"+ }+ }+ },+ {+ "prompt_en": "a cow",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "cow"+ }+ }+ },+ {+ "prompt_en": "an elephant",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "elephant"+ }+ }+ },+ {+ "prompt_en": "a bear",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bear"+ }+ }+ },+ {+ "prompt_en": "a zebra",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "zebra"+ }+ }+ },+ {+ "prompt_en": "a giraffe",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "giraffe"+ }+ }+ },+ {+ "prompt_en": "a backpack",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "backpack"+ }+ }+ },+ {+ "prompt_en": "an umbrella",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "umbrella"+ }+ }+ },+ {+ "prompt_en": "a handbag",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "handbag"+ }+ }+ },+ {+ "prompt_en": "a tie",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "tie"+ }+ }+ },+ {+ "prompt_en": "a suitcase",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "suitcase"+ }+ }+ },+ {+ "prompt_en": "a frisbee",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "frisbee"+ }+ }+ },+ {+ "prompt_en": "skis",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "skis"+ }+ }+ },+ {+ "prompt_en": "a snowboard",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "snowboard"+ }+ }+ },+ {+ "prompt_en": "a sports ball",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "sports ball"+ }+ }+ },+ {+ "prompt_en": "a kite",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "kite"+ }+ }+ },+ {+ "prompt_en": "a baseball bat",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "baseball bat"+ }+ }+ },+ {+ "prompt_en": "a baseball glove",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "baseball glove"+ }+ }+ },+ {+ "prompt_en": "a skateboard",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "skateboard"+ }+ }+ },+ {+ "prompt_en": "a surfboard",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "surfboard"+ }+ }+ },+ {+ "prompt_en": "a tennis racket",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "tennis racket"+ }+ }+ },+ {+ "prompt_en": "a bottle",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bottle"+ }+ }+ },+ {+ "prompt_en": "a wine glass",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "wine glass"+ }+ }+ },+ {+ "prompt_en": "a cup",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "cup"+ }+ }+ },+ {+ "prompt_en": "a fork",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "fork"+ }+ }+ },+ {+ "prompt_en": "a knife",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "knife"+ }+ }+ },+ {+ "prompt_en": "a spoon",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "spoon"+ }+ }+ },+ {+ "prompt_en": "a bowl",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bowl"+ }+ }+ },+ {+ "prompt_en": "a banana",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "banana"+ }+ }+ },+ {+ "prompt_en": "an apple",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "apple"+ }+ }+ },+ {+ "prompt_en": "a sandwich",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "sandwich"+ }+ }+ },+ {+ "prompt_en": "an orange",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "orange"+ }+ }+ },+ {+ "prompt_en": "broccoli",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "broccoli"+ }+ }+ },+ {+ "prompt_en": "a carrot",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "carrot"+ }+ }+ },+ {+ "prompt_en": "a hot dog",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "hot dog"+ }+ }+ },+ {+ "prompt_en": "a pizza",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "pizza"+ }+ }+ },+ {+ "prompt_en": "a donut",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "donut"+ }+ }+ },+ {+ "prompt_en": "a cake",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "cake"+ }+ }+ },+ {+ "prompt_en": "a chair",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "chair"+ }+ }+ },+ {+ "prompt_en": "a couch",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "couch"+ }+ }+ },+ {+ "prompt_en": "a potted plant",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "potted plant"+ }+ }+ },+ {+ "prompt_en": "a bed",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "bed"+ }+ }+ },+ {+ "prompt_en": "a dining table",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "dining table"+ }+ }+ },+ {+ "prompt_en": "a toilet",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "toilet"+ }+ }+ },+ {+ "prompt_en": "a tv",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "tv"+ }+ }+ },+ {+ "prompt_en": "a laptop",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "laptop"+ }+ }+ },+ {+ "prompt_en": "a remote",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "remote"+ }+ }+ },+ {+ "prompt_en": "a keyboard",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "keyboard"+ }+ }+ },+ {+ "prompt_en": "a cell phone",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "cell phone"+ }+ }+ },+ {+ "prompt_en": "a microwave",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "microwave"+ }+ }+ },+ {+ "prompt_en": "an oven",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "oven"+ }+ }+ },+ {+ "prompt_en": "a toaster",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "toaster"+ }+ }+ },+ {+ "prompt_en": "a sink",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "sink"+ }+ }+ },+ {+ "prompt_en": "a refrigerator",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "refrigerator"+ }+ }+ },+ {+ "prompt_en": "a book",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "book"+ }+ }+ },+ {+ "prompt_en": "a clock",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "clock"+ }+ }+ },+ {+ "prompt_en": "a vase",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "vase"+ }+ }+ },+ {+ "prompt_en": "scissors",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "scissors"+ }+ }+ },+ {+ "prompt_en": "a teddy bear",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "teddy bear"+ }+ }+ },+ {+ "prompt_en": "a hair drier",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "hair drier"+ }+ }+ },+ {+ "prompt_en": "a toothbrush",+ "dimension": [+ "object_class"+ ],+ "auxiliary_info": {+ "object_class": {+ "object": "toothbrush"+ }+ }+ },+ {+ "prompt_en": "a red bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white bicycle",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white car",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white bird",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a black cat",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white cat",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "an orange cat",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a yellow cat",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "a red umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white umbrella",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white suitcase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white bowl",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white chair",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white clock",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "a red vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "red"+ }+ }+ },+ {+ "prompt_en": "a green vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "green"+ }+ }+ },+ {+ "prompt_en": "a blue vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "blue"+ }+ }+ },+ {+ "prompt_en": "a yellow vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "yellow"+ }+ }+ },+ {+ "prompt_en": "an orange vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "orange"+ }+ }+ },+ {+ "prompt_en": "a purple vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "purple"+ }+ }+ },+ {+ "prompt_en": "a pink vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "pink"+ }+ }+ },+ {+ "prompt_en": "a black vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "black"+ }+ }+ },+ {+ "prompt_en": "a white vase",+ "dimension": [+ "color"+ ],+ "auxiliary_info": {+ "color": {+ "color": "white"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "The bund Shanghai, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "a shark is swimming in the ocean, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "Gwen Stacy reading a book, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "An astronaut flying in space, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, Van Gogh style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "Van Gogh style"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, oil painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "oil painting"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks by Hokusai, in the style of Ukiyo",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "by Hokusai, in the style of Ukiyo"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, black and white",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "black and white"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pixel art",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "pixel art"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, in cyberpunk style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "in cyberpunk style"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, animated style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "animated style"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, watercolor painting",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "watercolor painting"+ }+ }+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, surrealism style",+ "dimension": [+ "appearance_style"+ ],+ "auxiliary_info": {+ "appearance_style": {+ "appearance_style": "surrealism style"+ }+ }+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "A couple in formal evening wear going home get caught in a heavy downpour with umbrellas, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, in super slow motion",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, zoom in",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, zoom out",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pan left",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, pan right",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, tilt up",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, tilt down",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, with an intense shaking effect",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, featuring a steady and smooth perspective",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks, racking focus",+ "dimension": [+ "temporal_style"+ ]+ },+ {+ "prompt_en": "Close up of grapes on a rotating table.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Turtle swimming in ocean.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A storm trooper vacuuming the beach.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A panda standing on a surfboard in the ocean in sunset.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An astronaut feeding ducks on a sunny afternoon, reflection from the water.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Two pandas discussing an academic paper.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Sunset time lapse at the beach with moving clouds and colors in the sky.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A fat rabbit wearing a purple robe walking through a fantasy landscape.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A koala bear playing piano in the forest.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An astronaut flying in space.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Fireworks.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An animated painting of fluffy white clouds moving in sky.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Flying through fantasy landscapes.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A bigfoot walking in the snowstorm.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A squirrel eating a burger.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A cat wearing sunglasses and working as a lifeguard at a pool.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. the canyons twist and bend through the high elevated mountain peaks.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Splash of turquoise water in extreme slow motion, alpha channel included.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "an ice cream is melting on the table.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "a drone flying over a snowy forest.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "a shark is swimming in the ocean.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Aerial panoramic video from a drone of a fantasy land.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "a teddy bear is swimming in the ocean.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "time lapse of sunrise on mars.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "golden fish swimming in the ocean.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An artist brush painting on a canvas close up.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A drone view of celebration with Christmas tree and fireworks, starry sky - background.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "happy dog wearing a yellow turtleneck, studio, portrait, facing camera, dark background",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Origami dancers in white paper, 3D render, on white background, studio shot, dancing modern dance.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Campfire at night in a snowy forest with starry sky in the background.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "a fantasy landscape",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A 3D model of a 1800s victorian house.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "this is how I do makeup in the morning.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A raccoon that looks like a turtle, digital art.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Robot dancing in Times Square.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Busy freeway at night.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Balloon full of water exploding in extreme slow motion.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An astronaut is riding a horse in the space in a photorealistic style.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Macro slo-mo. Slow motion cropped closeup of roasted coffee beans falling into an empty bowl.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Sewing machine, old sewing machine working.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Motion colour drop in water, ink swirling in water, colourful ink in water, abstraction fancy dream cloud of ink.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Few big purple plums rotating on the turntable. water drops appear on the skin during rotation. isolated on the white background. close-up. macro.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Vampire makeup face of beautiful girl, red contact lenses.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Ashtray full of butts on table, smoke flowing on black background, close-up",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Pacific coast, carmel by the sea ocean and waves.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A teddy bear is playing drum kit in NYC Times Square.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A corgi is playing drum kit.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An Iron man is playing the electronic guitar, high electronic guitar.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A raccoon is playing the electronic guitar.",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A corgi's head depicted as an explosion of a nebula",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A fantasy landscape",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A future where humans have achieved teleportation technology",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A jellyfish floating through the ocean, with bioluminescent tentacles",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A Mars rover moving on Mars",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A panda drinking coffee in a cafe in Paris",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A space shuttle launching into orbit, with flames and smoke billowing out from the engines",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A steam train moving on a mountainside",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A super cool giant robot in Cyberpunk Beijing",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A tropical beach at sunrise, with palm trees and crystal-clear water in the foreground",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Cinematic shot of Van Gogh's selfie, Van Gogh style",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Gwen Stacy reading a book",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Iron Man flying in the sky",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, oil painting",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Yoda playing guitar on the stage",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A beautiful coastal beach in spring, waves lapping on sand by Vincent van Gogh",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A boat sailing leisurely along the Seine River with the Eiffel Tower in background",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A car moving slowly on an empty street, rainy evening",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A cat eating food out of a bowl",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A cat wearing sunglasses at a pool",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A confused panda in calculus class",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A cute fluffy panda eating Chinese food in a restaurant",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A cute happy Corgi playing in park, sunset",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A cute raccoon playing guitar in a boat on the ocean",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A happy fuzzy panda playing guitar nearby a campfire, snow mountain in the background",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A lightning striking atop of eiffel tower, dark clouds in the sky",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A modern art museum, with colorful paintings",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A panda cooking in the kitchen",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A panda playing on a swing set",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A polar bear is playing guitar",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A raccoon dressed in suit playing the trumpet, stage background",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A robot DJ is playing the turntable, in heavy raining futuristic tokyo rooftop cyberpunk night, sci-fi, fantasy",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A shark swimming in clear Caribbean ocean",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A super robot protecting city",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "A teddy bear washing the dishes",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An epic tornado attacking above a glowing city at night, the tornado is made of smoke",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "An oil painting of a couple in formal evening wear going home get caught in a heavy downpour with umbrellas",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Clown fish swimming through the coral reef",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Hyper-realistic spaceship landing on Mars",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "The bund Shanghai, vibrant color",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Vincent van Gogh is painting in the room",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "Yellow flowers swing in the wind",+ "dimension": [+ "overall_consistency",+ "aesthetic_quality",+ "imaging_quality"+ ]+ },+ {+ "prompt_en": "alley",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "alley"+ }+ }+ }+ },+ {+ "prompt_en": "amusement park",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "amusement park"+ }+ }+ }+ },+ {+ "prompt_en": "aquarium",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "aquarium"+ }+ }+ }+ },+ {+ "prompt_en": "arch",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "arch"+ }+ }+ }+ },+ {+ "prompt_en": "art gallery",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "art gallery"+ }+ }+ }+ },+ {+ "prompt_en": "bathroom",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "bathroom"+ }+ }+ }+ },+ {+ "prompt_en": "bakery shop",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "bakery shop"+ }+ }+ }+ },+ {+ "prompt_en": "ballroom",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "ballroom"+ }+ }+ }+ },+ {+ "prompt_en": "bar",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "bar"+ }+ }+ }+ },+ {+ "prompt_en": "barn",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "barn"+ }+ }+ }+ },+ {+ "prompt_en": "basement",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "basement"+ }+ }+ }+ },+ {+ "prompt_en": "beach",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "beach"+ }+ }+ }+ },+ {+ "prompt_en": "bedroom",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "bedroom"+ }+ }+ }+ },+ {+ "prompt_en": "bridge",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "bridge"+ }+ }+ }+ },+ {+ "prompt_en": "botanical garden",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "botanical garden"+ }+ }+ }+ },+ {+ "prompt_en": "cafeteria",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "cafeteria"+ }+ }+ }+ },+ {+ "prompt_en": "campsite",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "campsite"+ }+ }+ }+ },+ {+ "prompt_en": "campus",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "campus"+ }+ }+ }+ },+ {+ "prompt_en": "carrousel",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "carrousel"+ }+ }+ }+ },+ {+ "prompt_en": "castle",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "castle"+ }+ }+ }+ },+ {+ "prompt_en": "cemetery",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "cemetery"+ }+ }+ }+ },+ {+ "prompt_en": "classroom",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "classroom"+ }+ }+ }+ },+ {+ "prompt_en": "cliff",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "cliff"+ }+ }+ }+ },+ {+ "prompt_en": "crosswalk",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "crosswalk"+ }+ }+ }+ },+ {+ "prompt_en": "construction site",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "construction site"+ }+ }+ }+ },+ {+ "prompt_en": "corridor",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "corridor"+ }+ }+ }+ },+ {+ "prompt_en": "courtyard",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "courtyard"+ }+ }+ }+ },+ {+ "prompt_en": "desert",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "desert"+ }+ }+ }+ },+ {+ "prompt_en": "downtown",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "downtown"+ }+ }+ }+ },+ {+ "prompt_en": "driveway",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "driveway"+ }+ }+ }+ },+ {+ "prompt_en": "farm",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "farm"+ }+ }+ }+ },+ {+ "prompt_en": "food court",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "food court"+ }+ }+ }+ },+ {+ "prompt_en": "football field",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "football field"+ }+ }+ }+ },+ {+ "prompt_en": "forest road",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "forest road"+ }+ }+ }+ },+ {+ "prompt_en": "fountain",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "fountain"+ }+ }+ }+ },+ {+ "prompt_en": "gas station",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "gas station"+ }+ }+ }+ },+ {+ "prompt_en": "glacier",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "glacier"+ }+ }+ }+ },+ {+ "prompt_en": "golf course",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "golf course"+ }+ }+ }+ },+ {+ "prompt_en": "indoor gymnasium",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "indoor gymnasium"+ }+ }+ }+ },+ {+ "prompt_en": "harbor",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "harbor"+ }+ }+ }+ },+ {+ "prompt_en": "highway",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "highway"+ }+ }+ }+ },+ {+ "prompt_en": "hospital",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "hospital"+ }+ }+ }+ },+ {+ "prompt_en": "house",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "house"+ }+ }+ }+ },+ {+ "prompt_en": "iceberg",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "iceberg"+ }+ }+ }+ },+ {+ "prompt_en": "industrial area",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "industrial area"+ }+ }+ }+ },+ {+ "prompt_en": "jail cell",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "jail cell"+ }+ }+ }+ },+ {+ "prompt_en": "junkyard",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "junkyard"+ }+ }+ }+ },+ {+ "prompt_en": "kitchen",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "kitchen"+ }+ }+ }+ },+ {+ "prompt_en": "indoor library",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "indoor library"+ }+ }+ }+ },+ {+ "prompt_en": "lighthouse",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "lighthouse"+ }+ }+ }+ },+ {+ "prompt_en": "laboratory",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "laboratory"+ }+ }+ }+ },+ {+ "prompt_en": "mansion",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "mansion"+ }+ }+ }+ },+ {+ "prompt_en": "marsh",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "marsh"+ }+ }+ }+ },+ {+ "prompt_en": "mountain",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "mountain"+ }+ }+ }+ },+ {+ "prompt_en": "indoor movie theater",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "indoor movie theater"+ }+ }+ }+ },+ {+ "prompt_en": "indoor museum",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "indoor museum"+ }+ }+ }+ },+ {+ "prompt_en": "music studio",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "music studio"+ }+ }+ }+ },+ {+ "prompt_en": "nursery",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "nursery"+ }+ }+ }+ },+ {+ "prompt_en": "ocean",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "ocean"+ }+ }+ }+ },+ {+ "prompt_en": "office",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "office"+ }+ }+ }+ },+ {+ "prompt_en": "palace",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "palace"+ }+ }+ }+ },+ {+ "prompt_en": "parking lot",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "parking lot"+ }+ }+ }+ },+ {+ "prompt_en": "pharmacy",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "pharmacy"+ }+ }+ }+ },+ {+ "prompt_en": "phone booth",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "phone booth"+ }+ }+ }+ },+ {+ "prompt_en": "raceway",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "raceway"+ }+ }+ }+ },+ {+ "prompt_en": "restaurant",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "restaurant"+ }+ }+ }+ },+ {+ "prompt_en": "river",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "river"+ }+ }+ }+ },+ {+ "prompt_en": "science museum",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "science museum"+ }+ }+ }+ },+ {+ "prompt_en": "shower",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "shower"+ }+ }+ }+ },+ {+ "prompt_en": "ski slope",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "ski slope"+ }+ }+ }+ },+ {+ "prompt_en": "sky",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "sky"+ }+ }+ }+ },+ {+ "prompt_en": "skyscraper",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "skyscraper"+ }+ }+ }+ },+ {+ "prompt_en": "baseball stadium",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "baseball stadium"+ }+ }+ }+ },+ {+ "prompt_en": "staircase",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "staircase"+ }+ }+ }+ },+ {+ "prompt_en": "street",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "street"+ }+ }+ }+ },+ {+ "prompt_en": "supermarket",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "supermarket"+ }+ }+ }+ },+ {+ "prompt_en": "indoor swimming pool",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "indoor swimming pool"+ }+ }+ }+ },+ {+ "prompt_en": "tower",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "tower"+ }+ }+ }+ },+ {+ "prompt_en": "outdoor track",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "outdoor track"+ }+ }+ }+ },+ {+ "prompt_en": "train railway",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "train railway"+ }+ }+ }+ },+ {+ "prompt_en": "train station platform",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "train station platform"+ }+ }+ }+ },+ {+ "prompt_en": "underwater coral reef",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "underwater coral reef"+ }+ }+ }+ },+ {+ "prompt_en": "valley",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "valley"+ }+ }+ }+ },+ {+ "prompt_en": "volcano",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "volcano"+ }+ }+ }+ },+ {+ "prompt_en": "waterfall",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "waterfall"+ }+ }+ }+ },+ {+ "prompt_en": "windmill",+ "dimension": [+ "scene",+ "background_consistency"+ ],+ "auxiliary_info": {+ "scene": {+ "scene": {+ "scene": "windmill"+ }+ }+ }+ },+ {+ "prompt_en": "a bicycle on the left of a car, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bicycle",+ "object_b": "car",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a car on the right of a motorcycle, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "car",+ "object_b": "motorcycle",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a motorcycle on the left of a bus, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "motorcycle",+ "object_b": "bus",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a bus on the right of a traffic light, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bus",+ "object_b": "traffic light",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a traffic light on the left of a fire hydrant, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "traffic light",+ "object_b": "fire hydrant",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a fire hydrant on the right of a stop sign, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "fire hydrant",+ "object_b": "stop sign",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a stop sign on the left of a parking meter, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "stop sign",+ "object_b": "parking meter",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a parking meter on the right of a bench, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "parking meter",+ "object_b": "bench",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a bench on the left of a truck, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bench",+ "object_b": "truck",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a truck on the right of a bicycle, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "truck",+ "object_b": "bicycle",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a bird on the left of a cat, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bird",+ "object_b": "cat",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a cat on the right of a dog, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "cat",+ "object_b": "dog",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a dog on the left of a horse, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "dog",+ "object_b": "horse",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a horse on the right of a sheep, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "horse",+ "object_b": "sheep",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a sheep on the left of a cow, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "sheep",+ "object_b": "cow",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a cow on the right of an elephant, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "cow",+ "object_b": "elephant",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "an elephant on the left of a bear, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "elephant",+ "object_b": "bear",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a bear on the right of a zebra, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bear",+ "object_b": "zebra",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a zebra on the left of a giraffe, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "zebra",+ "object_b": "giraffe",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a giraffe on the right of a bird, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "giraffe",+ "object_b": "bird",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a bottle on the left of a wine glass, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bottle",+ "object_b": "wine glass",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a wine glass on the right of a cup, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "wine glass",+ "object_b": "cup",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a cup on the left of a fork, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "cup",+ "object_b": "fork",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a fork on the right of a knife, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "fork",+ "object_b": "knife",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a knife on the left of a spoon, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "knife",+ "object_b": "spoon",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a spoon on the right of a bowl, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "spoon",+ "object_b": "bowl",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a bowl on the left of a bottle, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bowl",+ "object_b": "bottle",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a potted plant on the left of a remote, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "potted plant",+ "object_b": "remote",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a remote on the right of a clock, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "remote",+ "object_b": "clock",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a clock on the left of a vase, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "clock",+ "object_b": "vase",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a vase on the right of scissors, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "vase",+ "object_b": "scissors",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "scissors on the left of a teddy bear, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "scissors",+ "object_b": "teddy bear",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a teddy bear on the right of a potted plant, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "teddy bear",+ "object_b": "potted plant",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a frisbee on the left of a sports ball, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "frisbee",+ "object_b": "sports ball",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a sports ball on the right of a baseball bat, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "sports ball",+ "object_b": "baseball bat",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a baseball bat on the left of a baseball glove, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "baseball bat",+ "object_b": "baseball glove",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a baseball glove on the right of a tennis racket, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "baseball glove",+ "object_b": "tennis racket",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a tennis racket on the left of a frisbee, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "tennis racket",+ "object_b": "frisbee",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a toilet on the left of a hair drier, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "toilet",+ "object_b": "hair drier",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a hair drier on the right of a toothbrush, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "hair drier",+ "object_b": "toothbrush",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a toothbrush on the left of a sink, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "toothbrush",+ "object_b": "sink",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a sink on the right of a toilet, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "sink",+ "object_b": "toilet",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a chair on the left of a couch, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "chair",+ "object_b": "couch",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a couch on the right of a bed, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "couch",+ "object_b": "bed",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a bed on the left of a tv, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "bed",+ "object_b": "tv",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a tv on the right of a dining table, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "tv",+ "object_b": "dining table",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a dining table on the left of a chair, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "dining table",+ "object_b": "chair",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "an airplane on the left of a train, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "airplane",+ "object_b": "train",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "a train on the right of a boat, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "train",+ "object_b": "boat",+ "relationship": "on the right of"+ }+ }+ }+ },+ {+ "prompt_en": "a boat on the left of an airplane, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "boat",+ "object_b": "airplane",+ "relationship": "on the left of"+ }+ }+ }+ },+ {+ "prompt_en": "an oven on the top of a toaster, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "oven",+ "object_b": "toaster",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "an oven on the bottom of a toaster, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "oven",+ "object_b": "toaster",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a toaster on the top of a microwave, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "toaster",+ "object_b": "microwave",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a toaster on the bottom of a microwave, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "toaster",+ "object_b": "microwave",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a microwave on the top of an oven, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "microwave",+ "object_b": "oven",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a microwave on the bottom of an oven, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "microwave",+ "object_b": "oven",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a banana on the top of an apple, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "banana",+ "object_b": "apple",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a banana on the bottom of an apple, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "banana",+ "object_b": "apple",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "an apple on the top of a sandwich, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "apple",+ "object_b": "sandwich",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "an apple on the bottom of a sandwich, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "apple",+ "object_b": "sandwich",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a sandwich on the top of an orange, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "sandwich",+ "object_b": "orange",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a sandwich on the bottom of an orange, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "sandwich",+ "object_b": "orange",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "an orange on the top of a carrot, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "orange",+ "object_b": "carrot",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "an orange on the bottom of a carrot, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "orange",+ "object_b": "carrot",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a carrot on the top of a hot dog, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "carrot",+ "object_b": "hot dog",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a carrot on the bottom of a hot dog, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "carrot",+ "object_b": "hot dog",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a hot dog on the top of a pizza, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "hot dog",+ "object_b": "pizza",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a hot dog on the bottom of a pizza, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "hot dog",+ "object_b": "pizza",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a pizza on the top of a donut, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "pizza",+ "object_b": "donut",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a pizza on the bottom of a donut, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "pizza",+ "object_b": "donut",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a donut on the top of broccoli, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "donut",+ "object_b": "broccoli",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a donut on the bottom of broccoli, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "donut",+ "object_b": "broccoli",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "broccoli on the top of a banana, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "broccoli",+ "object_b": "banana",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "broccoli on the bottom of a banana, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "broccoli",+ "object_b": "banana",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "skis on the top of a snowboard, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "skis",+ "object_b": "snowboard",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "skis on the bottom of a snowboard, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "skis",+ "object_b": "snowboard",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a snowboard on the top of a kite, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "snowboard",+ "object_b": "kite",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a snowboard on the bottom of a kite, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "snowboard",+ "object_b": "kite",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a kite on the top of a skateboard, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "kite",+ "object_b": "skateboard",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a kite on the bottom of a skateboard, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "kite",+ "object_b": "skateboard",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a skateboard on the top of a surfboard, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "skateboard",+ "object_b": "surfboard",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a skateboard on the bottom of a surfboard, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "skateboard",+ "object_b": "surfboard",+ "relationship": "on the bottom of"+ }+ }+ }+ },+ {+ "prompt_en": "a surfboard on the top of skis, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "surfboard",+ "object_b": "skis",+ "relationship": "on the top of"+ }+ }+ }+ },+ {+ "prompt_en": "a surfboard on the bottom of skis, front view",+ "dimension": [+ "spatial_relationship"+ ],+ "auxiliary_info": {+ "spatial_relationship": {+ "spatial_relationship": {+ "object_a": "surfboard",+ "object_b": "skis",+ "relationship": "on the bottom of"+ }+ }+ }+ }+]diff --git a/scripts/misc/sample.sh b/eval/sample.shsimilarity index 69%rename from scripts/misc/sample.shrename to eval/sample.shindex c3a91e8..297205e 100644--- a/scripts/misc/sample.sh+++ b/eval/sample.sh@@ -49,7 +49,7 @@ function run_image() { # 10mineval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 1 --image-size 600 1358 --sample-name 720p_2_1}-function run_video_1() { # 20min+function run_video_a() { # 20min# 2.1.1 16x240x426eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_samples.txt --save-dir $OUTPUT --num-frames 16 --image-size 240 426 --sample-name sample_16x240x426@@ -71,7 +71,7 @@ function run_video_1() { # 20mineval $CMD --ckpt-path $CKPT --prompt \"$PROMPT\" --save-dir $OUTPUT --num-frames 16 --image-size 600 1358 --sample-name 720p_2_1}-function run_video_2() { # 60min+function run_video_b() { # 30min# 2.2.1 16x240x426eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 16 --image-size 240 426 --sample-name short_16x240x426@@ -79,7 +79,7 @@ function run_video_2() { # 60mineval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 64 --image-size 240 426 --sample-name short_64x240x426}-function run_video_3() { # 60min+function run_video_c() { # 30min# 2.3.1 16x240x426eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_sora.txt --save-dir $OUTPUT --num-frames 16 --image-size 240 426 --sample-name sora_16x240x426@@ -87,17 +87,17 @@ function run_video_3() { # 60mineval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 128 --image-size 240 426 --sample-name short_128x240x426}-function run_video_4() { # 120min+function run_video_d() { # 30min# 2.4 16x480x854eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 16 --image-size 480 854 --sample-name short_16x480x854}-function run_video_5() { # 120min+function run_video_e() { # 30min# 2.5 64x480x854eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 64 --image-size 480 854 --sample-name short_64x480x854}-function run_video_6() { # 120min+function run_video_f() { # 30min# 2.6 16x720x1280eval $CMD --ckpt-path $CKPT --prompt-path assets/texts/t2v_short.txt --save-dir $OUTPUT --num-frames 16 --image-size 720 1280 --sample-name short_16x720x1280}@@ -127,41 +127,132 @@ function run_video_edit() { # 23min--mask-strategy "0,0,0,1,0\;0,0,0,1,-1" "0,0,0,1,0\;0,1,0,1,-1"}+# vbench has 950 samples++function run_vbenck_a() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 0 --end-index 120+}++function run_vbenck_b() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 120 --end-index 240+}++function run_vbenck_c() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 240 --end-index 360+}++function run_vbenck_d() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 360 --end-index 480+}++function run_vbenck_e() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 480 --end-index 600+}++function run_vbenck_f() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 600 --end-index 720+}++function run_vbenck_g() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 720 --end-index 840+}++function run_vbenck_h() {+ eval $CMD --ckpt-path $CKPT --save-dir ${OUTPUT}_vbench --prompt-as-path --num-sample 5 \+ --prompt-path assets/texts/VBench/all_dimension.txt \+ --start-index 840+}+### Mainfor arg in "$@"; do+ # imageif [[ "$arg" = -1 ]] || [[ "$arg" = --image ]]; thenecho "Running image samples..."run_imagefi+ # video: sample 16x240p & multi-resolutionif [[ "$arg" = -2a ]] || [[ "$arg" = --video ]]; then- echo "Running video samples 1..."- run_video_1+ echo "Running video samples a..."+ run_video_afi+ # video: short 16x240p & 64x240pif [[ "$arg" = -2b ]] || [[ "$arg" = --video ]]; then- echo "Running video samples 2..."- run_video_2+ echo "Running video samples b..."+ run_video_bfi+ # video: sora 16x240p & short 128x240pif [[ "$arg" = -2c ]] || [[ "$arg" = --video ]]; then- echo "Running video samples 3..."- run_video_3+ echo "Running video samples c..."+ run_video_cfi+ # short 16x480pif [[ "$arg" = -2d ]] || [[ "$arg" = --video ]]; then- echo "Running video samples 4..."- run_video_4+ echo "Running video samples d..."+ run_video_dfi+ # short 64x480pif [[ "$arg" = -2e ]] || [[ "$arg" = --video ]]; then- echo "Running video samples 5..."- run_video_5+ echo "Running video samples e..."+ run_video_efi+ # short 16x720pif [[ "$arg" = -2f ]] || [[ "$arg" = --video ]]; then- echo "Running video samples 6..."- run_video_6+ echo "Running video samples f..."+ run_video_ffi+ # video editif [[ "$arg" = -3 ]] || [[ "$arg" = --video-edit ]]; thenecho "Running video edit samples..."run_video_editfi+ # vbench+ if [[ "$arg" = -4a ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples a..."+ run_vbenck_a+ fi+ if [[ "$arg" = -4b ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples b..."+ run_vbenck_b+ fi+ if [[ "$arg" = -4c ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples c..."+ run_vbenck_c+ fi+ if [[ "$arg" = -4d ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples d..."+ run_vbenck_d+ fi+ if [[ "$arg" = -4e ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples e..."+ run_vbenck_e+ fi+ if [[ "$arg" = -4f ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples f..."+ run_vbenck_f+ fi+ if [[ "$arg" = -4g ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples g..."+ run_vbenck_g+ fi+ if [[ "$arg" = -4h ]] || [[ "$arg" = --vbench ]]; then+ echo "Running vbench samples h..."+ run_vbenck_h+ fidone### Enddiff --git a/eval/vbench.sh b/eval/vbench.shnew file mode 100644index 0000000..864cd9b--- /dev/null+++ b/eval/vbench.sh@@ -0,0 +1,20 @@+#!/bin/bash++# Base path for videos+videos_path='./samples/samples_stage1_41k_2_17k_vbench'+json_path='./eval/VBench_full_info.json'++# Define the dimension list+dimensions=("subject_consistency" "background_consistency" "aesthetic_quality" "imaging_quality" "object_class" "multiple_objects" "color" "spatial_relationship" "scene" "temporal_style" "overall_consistency" "human_action" "temporal_flickering" "motion_smoothness" "dynamic_degree" "appearance_style")++# Loop over each dimension+for i in "${!dimensions[@]}"; do+ # Get the dimension and corresponding folder+ dimension=${dimensions[i]}++ # Construct the video path+ echo "$dimension $videos_path"++ # Run the evaluation script+ vbench evaluate --videos_path $videos_path --dimension $dimension --full_json_dir $json_path+donediff --git a/opensora/acceleration/shardformer/__init__.py b/opensora/acceleration/shardformer/__init__.pyindex 8b13789..e69de29 100644--- a/opensora/acceleration/shardformer/__init__.py+++ b/opensora/acceleration/shardformer/__init__.py@@ -1 +0,0 @@-diff --git a/opensora/models/layers/blocks.py b/opensora/models/layers/blocks.pyindex 0a9ebb1..8d3763b 100644--- a/opensora/models/layers/blocks.py+++ b/opensora/models/layers/blocks.py@@ -23,18 +23,13 @@ import xformers.opsfrom einops import rearrangefrom timm.models.vision_transformer import Mlp-from opensora.acceleration.communications import (- all_to_all,- split_forward_gather_backward,-)+from opensora.acceleration.communications import all_to_all, split_forward_gather_backwardfrom opensora.acceleration.parallel_states import get_sequence_parallel_groupapprox_gelu = lambda: nn.GELU(approximate="tanh")-def get_layernorm(- hidden_size: torch.Tensor, eps: float, affine: bool, use_kernel: bool-):+def get_layernorm(hidden_size: torch.Tensor, eps: float, affine: bool, use_kernel: bool):if use_kernel:try:from apex.normalization import FusedLayerNorm@@ -89,9 +84,7 @@ class PatchEmbed3D(nn.Module):self.in_chans = in_chansself.embed_dim = embed_dim- self.proj = nn.Conv3d(- in_chans, embed_dim, kernel_size=patch_size, stride=patch_size- )+ self.proj = nn.Conv3d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)if norm_layer is not None:self.norm = norm_layer(embed_dim)else:@@ -226,9 +219,7 @@ class SeqParallelAttention(Attention):)def forward(self, x: torch.Tensor) -> torch.Tensor:- B, N, C = (- x.shape- ) # for sequence parallel here, the N is a local sequence length+ B, N, C = x.shape # for sequence parallel here, the N is a local sequence lengthqkv = self.qkv(x)qkv_shape = (B, N, 3, self.num_heads, self.head_dim)@@ -321,9 +312,7 @@ class MultiHeadCrossAttention(nn.Module):attn_bias = Noneif mask is not None:attn_bias = xformers.ops.fmha.BlockDiagonalMask.from_seqlens([N] * B, mask)- x = xformers.ops.memory_efficient_attention(- q, k, v, p=self.attn_drop.p, attn_bias=attn_bias- )+ x = xformers.ops.memory_efficient_attention(q, k, v, p=self.attn_drop.p, attn_bias=attn_bias)x = x.view(B, -1, C)x = self.proj(x)@@ -362,12 +351,8 @@ class SeqParallelMultiHeadCrossAttention(MultiHeadCrossAttention):# apply all_to_all to gather sequence and split attention headsq = all_to_all(q, sp_group, scatter_dim=2, gather_dim=1)- k = split_forward_gather_backward(- k, get_sequence_parallel_group(), dim=2, grad_scale="down"- )- v = split_forward_gather_backward(- v, get_sequence_parallel_group(), dim=2, grad_scale="down"- )+ k = split_forward_gather_backward(k, get_sequence_parallel_group(), dim=2, grad_scale="down")+ v = split_forward_gather_backward(v, get_sequence_parallel_group(), dim=2, grad_scale="down")q = q.view(1, -1, self.num_heads // sp_size, self.head_dim)k = k.view(1, -1, self.num_heads // sp_size, self.head_dim)@@ -377,9 +362,7 @@ class SeqParallelMultiHeadCrossAttention(MultiHeadCrossAttention):attn_bias = Noneif mask is not None:attn_bias = xformers.ops.fmha.BlockDiagonalMask.from_seqlens([N] * B, mask)- x = xformers.ops.memory_efficient_attention(- q, k, v, p=self.attn_drop.p, attn_bias=attn_bias- )+ x = xformers.ops.memory_efficient_attention(q, k, v, p=self.attn_drop.p, attn_bias=attn_bias)# apply all to all to gather back attention heads and scatter sequencex = x.view(B, -1, self.num_heads // sp_size, self.head_dim)@@ -401,9 +384,7 @@ class FinalLayer(nn.Module):super().__init__()self.norm_final = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)self.linear = nn.Linear(hidden_size, num_patch * out_channels, bias=True)- self.adaLN_modulation = nn.Sequential(- nn.SiLU(), nn.Linear(hidden_size, 2 * hidden_size, bias=True)- )+ self.adaLN_modulation = nn.Sequential(nn.SiLU(), nn.Linear(hidden_size, 2 * hidden_size, bias=True))def forward(self, x, c):shift, scale = self.adaLN_modulation(c).chunk(2, dim=1)@@ -421,9 +402,7 @@ class T2IFinalLayer(nn.Module):super().__init__()self.norm_final = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)self.linear = nn.Linear(hidden_size, num_patch * out_channels, bias=True)- self.scale_shift_table = nn.Parameter(- torch.randn(2, hidden_size) / hidden_size**0.5- )+ self.scale_shift_table = nn.Parameter(torch.randn(2, hidden_size) / hidden_size**0.5)self.out_channels = out_channelsself.d_t = d_tself.d_s = d_s@@ -446,9 +425,7 @@ class T2IFinalLayer(nn.Module):shift, scale = (self.scale_shift_table[None] + t[:, None]).chunk(2, dim=1)x = t2i_modulate(self.norm_final(x), shift, scale)if x_mask is not None:- shift_zero, scale_zero = (self.scale_shift_table[None] + t0[:, None]).chunk(- 2, dim=1- )+ shift_zero, scale_zero = (self.scale_shift_table[None] + t0[:, None]).chunk(2, dim=1)x_zero = t2i_modulate(self.norm_final(x), shift_zero, scale_zero)x = self.t_mask_select(x_mask, x, x_zero, T, S)x = self.linear(x)@@ -486,18 +463,12 @@ class TimestepEmbedder(nn.Module):"""# https://github.com/openai/glide-text2im/blob/main/glide_text2im/nn.pyhalf = dim // 2- freqs = torch.exp(- -math.log(max_period)- * torch.arange(start=0, end=half, dtype=torch.float32)- / half- )+ freqs = torch.exp(-math.log(max_period) * torch.arange(start=0, end=half, dtype=torch.float32) / half)freqs = freqs.to(device=t.device)args = t[:, None].float() * freqs[None]embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)if dim % 2:- embedding = torch.cat(- [embedding, torch.zeros_like(embedding[:, :1])], dim=-1- )+ embedding = torch.cat([embedding, torch.zeros_like(embedding[:, :1])], dim=-1)return embeddingdef forward(self, t, dtype):@@ -516,9 +487,7 @@ class LabelEmbedder(nn.Module):def __init__(self, num_classes, hidden_size, dropout_prob):super().__init__()use_cfg_embedding = dropout_prob > 0- self.embedding_table = nn.Embedding(- num_classes + use_cfg_embedding, hidden_size- )+ self.embedding_table = nn.Embedding(num_classes + use_cfg_embedding, hidden_size)self.num_classes = num_classesself.dropout_prob = dropout_prob@@ -546,9 +515,7 @@ class SizeEmbedder(TimestepEmbedder):"""def __init__(self, hidden_size, frequency_embedding_size=256):- super().__init__(- hidden_size=hidden_size, frequency_embedding_size=frequency_embedding_size- )+ super().__init__(hidden_size=hidden_size, frequency_embedding_size=frequency_embedding_size)self.mlp = nn.Sequential(nn.Linear(frequency_embedding_size, hidden_size, bias=True),nn.SiLU(),@@ -566,9 +533,7 @@ class SizeEmbedder(TimestepEmbedder):assert s.shape[0] == bsb, dims = s.shape[0], s.shape[1]s = rearrange(s, "b d -> (b d)")- s_freq = self.timestep_embedding(s, self.frequency_embedding_size).to(- self.dtype- )+ s_freq = self.timestep_embedding(s, self.frequency_embedding_size).to(self.dtype)s_emb = self.mlp(s_freq)s_emb = rearrange(s_emb, "(b d) d2 -> b (d d2)", b=b, d=dims, d2=self.outdim)return s_emb@@ -684,9 +649,7 @@ class PositionEmbedding2D(nn.Module):# https://github.com/facebookresearch/mae/blob/main/util/pos_embed.py-def get_2d_sincos_pos_embed(- embed_dim, grid_size, cls_token=False, extra_tokens=0, scale=1.0, base_size=None-):+def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False, extra_tokens=0, scale=1.0, base_size=None):"""grid_size: int of the grid height and widthreturn:@@ -706,9 +669,7 @@ def get_2d_sincos_pos_embed(grid = grid.reshape([2, 1, grid_size[1], grid_size[0]])pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)if cls_token and extra_tokens > 0:- pos_embed = np.concatenate(- [np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0- )+ pos_embed = np.concatenate([np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0)return pos_embeddiff --git a/opensora/schedulers/iddpm/__init__.py b/opensora/schedulers/iddpm/__init__.pyindex 6989e64..2dfa9f4 100644--- a/opensora/schedulers/iddpm/__init__.py+++ b/opensora/schedulers/iddpm/__init__.py@@ -91,7 +91,8 @@ def forward_with_cfg(model, x, timestep, y, cfg_scale, cfg_channel=None, **kwarghalf = x[: len(x) // 2]combined = torch.cat([half, half], dim=0)if "x_mask" in kwargs and kwargs["x_mask"] is not None:- kwargs["x_mask"] = torch.cat([kwargs["x_mask"], kwargs["x_mask"]], dim=0)+ if len(kwargs["x_mask"]) != len(x):+ kwargs["x_mask"] = torch.cat([kwargs["x_mask"], kwargs["x_mask"]], dim=0)model_out = model.forward(combined, timestep, y, **kwargs)model_out = model_out["x"] if isinstance(model_out, dict) else model_outif cfg_channel is None:diff --git a/opensora/schedulers/iddpm/gaussian_diffusion.py b/opensora/schedulers/iddpm/gaussian_diffusion.pyindex 0ff7a2c..fa734f4 100644--- a/opensora/schedulers/iddpm/gaussian_diffusion.py+++ b/opensora/schedulers/iddpm/gaussian_diffusion.py@@ -405,6 +405,21 @@ class GaussianDiffusion:- 'sample': a random sample from the model.- 'pred_xstart': a prediction of x_0."""+ if mask is not None:+ if mask.shape[0] != x.shape[0]:+ mask = mask.repeat(2, 1) # HACK+ # copy unchanged x values to x0+ x0 = x.clone()+ mask_t = (mask * len(self.betas)).to(torch.int)+ mask_t_equall = (mask_t == t.unsqueeze(1))[:, None, :, None, None]+ mask_t_upper = (mask_t > t.unsqueeze(1))[:, None, :, None, None]+ x_noise = x0 * _extract_into_tensor(self.sqrt_alphas_cumprod, t, x.shape) + torch.randn_like(x) * _extract_into_tensor(+ self.sqrt_one_minus_alphas_cumprod, t, x.shape)+ # active noise addition+ x = torch.where(mask_t_equall, x_noise, x)+ batch_size = x.shape[0]+ model_kwargs["x_mask"] = mask_t_upper.reshape(batch_size, -1).to(torch.bool)+out = self.p_mean_variance(model,x,@@ -418,10 +433,10 @@ class GaussianDiffusion:if cond_fn is not None:out["mean"] = self.condition_mean(cond_fn, out, x, t, model_kwargs=model_kwargs)sample = out["mean"] + nonzero_mask * torch.exp(0.5 * out["log_variance"]) * noise+if mask is not None:- if mask.shape[0] != x.shape[0]:- mask = mask.repeat(2, 1) # HACK- sample = torch.where(mask[:, None, :, None, None], sample, x)+ mask_t_lower = (mask_t < t.unsqueeze(1))[:, None, :, None, None]+ sample = torch.where(mask_t_lower, x0, sample)return {"sample": sample, "pred_xstart": out["pred_xstart"]}diff --git a/opensora/schedulers/iddpm/respace.py b/opensora/schedulers/iddpm/respace.pyindex 42d90e8..b2a130f 100644--- a/opensora/schedulers/iddpm/respace.py+++ b/opensora/schedulers/iddpm/respace.py@@ -42,9 +42,7 @@ def space_timesteps(num_timesteps, section_counts):for i in range(1, num_timesteps):if len(range(0, num_timesteps, i)) == desired_count:return set(range(0, num_timesteps, i))- raise ValueError(- f"cannot create exactly {num_timesteps} steps with an integer stride"- )+ raise ValueError(f"cannot create exactly {num_timesteps} steps with an integer stride")section_counts = [int(x) for x in section_counts.split(",")]size_per = num_timesteps // len(section_counts)extra = num_timesteps % len(section_counts)@@ -53,9 +51,7 @@ def space_timesteps(num_timesteps, section_counts):for i, section_count in enumerate(section_counts):size = size_per + (1 if i < extra else 0)if size < section_count:- raise ValueError(- f"cannot divide section of {size} steps into {section_count}"- )+ raise ValueError(f"cannot divide section of {size} steps into {section_count}")if section_count <= 1:frac_stride = 1else:@@ -95,14 +91,10 @@ class SpacedDiffusion(GaussianDiffusion):super().__init__(**kwargs)self.map_tensor = torch.tensor(self.timestep_map, device=get_current_device())- def p_mean_variance(- self, model, *args, **kwargs- ): # pylint: disable=signature-differs+ def p_mean_variance(self, model, *args, **kwargs): # pylint: disable=signature-differsreturn super().p_mean_variance(self._wrap_model(model), *args, **kwargs)- def training_losses(- self, model, *args, **kwargs- ): # pylint: disable=signature-differs+ def training_losses(self, model, *args, **kwargs): # pylint: disable=signature-differsreturn super().training_losses(self._wrap_model(model), *args, **kwargs)def condition_mean(self, cond_fn, *args, **kwargs):diff --git a/opensora/utils/ckpt_utils.py b/opensora/utils/ckpt_utils.pyindex 75fa700..086742f 100644--- a/opensora/utils/ckpt_utils.py+++ b/opensora/utils/ckpt_utils.py@@ -87,16 +87,23 @@ def find_model(model_name, model=None):return model_ckpt-def download_model(model_name):+def download_model(model_name=None, local_path=None, url=None):"""Downloads a pre-trained DiT model from the web."""- assert model_name in pretrained_models- local_path = f"pretrained_models/{model_name}"+ if model_name is not None:+ assert model_name in pretrained_models+ local_path = f"pretrained_models/{model_name}"+ web_path = pretrained_models[model_name]+ else:+ assert local_path is not None+ assert url is not None+ web_path = urlif not os.path.isfile(local_path):os.makedirs("pretrained_models", exist_ok=True)- web_path = pretrained_models[model_name]- download_url(web_path, "pretrained_models", model_name)+ dir_name = os.path.dirname(local_path)+ file_name = os.path.basename(local_path)+ download_url(web_path, dir_name, file_name)model = torch.load(local_path, map_location=lambda storage, loc: storage)return modeldiff --git a/opensora/utils/config_utils.py b/opensora/utils/config_utils.pyindex 51e8c8c..78ee638 100644--- a/opensora/utils/config_utils.py+++ b/opensora/utils/config_utils.py@@ -35,6 +35,8 @@ def parse_args(training=False):parser.add_argument("--sample-name", default=None, type=str, help="sample name, default is sample_idx")parser.add_argument("--start-index", default=None, type=int, help="start index for sample name")parser.add_argument("--end-index", default=None, type=int, help="end index for sample name")+ parser.add_argument("--num-sample", default=None, type=int, help="number of samples to generate for one prompt")+ parser.add_argument("--prompt-as-path", action="store_true", help="use prompt as path to save samples")# promptparser.add_argument("--prompt-path", default=None, type=str, help="path to prompt txt file")@@ -88,6 +90,10 @@ def merge_args(cfg, args, training=False):cfg["frame_interval"] = 1if "sample_name" not in cfg:cfg["sample_name"] = None+ if "num_sample" not in cfg:+ cfg["num_sample"] = 1+ if "prompt_as_path" not in cfg:+ cfg["prompt_as_path"] = False# - Prompt handlingif "prompt" not in cfg or cfg["prompt"] is None:assert cfg["prompt_path"] is not None, "prompt or prompt_path must be provided"diff --git a/scripts/inference-long.py b/scripts/inference-long.pyindex 2860dee..1ab74d3 100644--- a/scripts/inference-long.py+++ b/scripts/inference-long.py@@ -7,7 +7,7 @@ from colossalai.cluster import DistCoordinatorfrom mmengine.runner import set_random_seedfrom opensora.acceleration.parallel_states import set_sequence_parallel_group-from opensora.datasets import save_sample+from opensora.datasets import IMG_FPS, save_samplefrom opensora.datasets.utils import read_from_pathfrom opensora.models.text_encoder.t5 import text_preprocessingfrom opensora.registry import MODELS, SCHEDULERS, build_module@@ -168,7 +168,12 @@ def main():# 4. inference# ======================================================sample_idx = 0- sample_name = cfg.sample_name if cfg.sample_name is not None else "sample"+ if cfg.sample_name is not None:+ sample_name = cfg.sample_name+ elif cfg.prompt_as_path:+ sample_name = ""+ else:+ sample_name = "sample"save_dir = cfg.save_diros.makedirs(save_dir, exist_ok=True)@@ -202,30 +207,41 @@ def main():model_args["x_mask"] = masks# 4.6. diffusion sampling- samples = scheduler.sample(- model,- text_encoder,- z=z,- prompts=batch_prompts,- device=device,- additional_args=model_args,- mask=masks, # scheduler must support mask- )- samples = vae.decode(samples.to(dtype))- video_clips.append(samples)-- # 4.7. save video- if loop_i == cfg.loop - 1:- if not use_dist or coordinator.is_master():- for idx in range(len(video_clips[0])):- video_clips_i = [video_clips[0][idx]] + [- video_clips[i][idx][:, cfg.condition_frame_length :] for i in range(1, cfg.loop)- ]- video = torch.cat(video_clips_i, dim=1)- print(f"Prompt: {prompts[i + idx]}")- save_path = os.path.join(save_dir, f"{sample_name}_{sample_idx}")- save_sample(video, fps=cfg.fps // cfg.frame_interval, save_path=save_path)- sample_idx += 1+ old_sample_idx = sample_idx+ # generate multiple samples for each prompt+ for k in range(cfg.num_sample):+ sample_idx = old_sample_idx++ samples = scheduler.sample(+ model,+ text_encoder,+ z=z,+ prompts=batch_prompts,+ device=device,+ additional_args=model_args,+ mask=masks, # scheduler must support mask+ )+ samples = vae.decode(samples.to(dtype))+ video_clips.append(samples)++ # 4.7. save video+ if loop_i == cfg.loop - 1:+ if not use_dist or coordinator.is_master():+ for idx in range(len(video_clips[0])):+ video_clips_i = [video_clips[0][idx]] + [+ video_clips[i][idx][:, cfg.condition_frame_length :] for i in range(1, cfg.loop)+ ]+ video = torch.cat(video_clips_i, dim=1)+ print(f"Prompt: {prompts[i + idx]}")+ if cfg.prompt_as_path:+ sample_name_suffix = batch_prompts[idx]+ else:+ sample_name_suffix = f"_{sample_idx}"+ save_path = os.path.join(save_dir, f"{sample_name}{sample_name_suffix}")+ if cfg.num_sample != 1:+ save_path = f"{save_path}-{k}"+ save_sample(video, fps=cfg.fps // cfg.frame_interval, save_path=save_path)+ sample_idx += 1if __name__ == "__main__":diff --git a/scripts/inference-sdedit.py b/scripts/inference-sdedit.pynew file mode 100644index 0000000..9adafcf--- /dev/null+++ b/scripts/inference-sdedit.py@@ -0,0 +1,220 @@+import os++import colossalai+import torch+import torch.distributed as dist+from colossalai.cluster import DistCoordinator+from mmengine.runner import set_random_seed++from opensora.acceleration.parallel_states import set_sequence_parallel_group+from opensora.datasets import save_sample+from opensora.datasets.utils import read_from_path+from opensora.registry import MODELS, SCHEDULERS, build_module+from opensora.utils.config_utils import parse_configs+from opensora.utils.misc import to_torch_dtype+++def collect_references_batch(reference_paths, vae, image_size):+ refs_x = []+ for reference_path in reference_paths:+ ref_path = reference_path.split(";")+ ref = []+ for r_path in ref_path:+ r = read_from_path(r_path, image_size)+ r_x = vae.encode(r.unsqueeze(0).to(vae.device, vae.dtype))+ r_x = r_x.squeeze(0)+ ref.append(r_x)+ refs_x.append(ref)+ # refs_x: [batch, ref_num, C, T, H, W]+ return refs_x+++def apply_mask_strategy(z, refs_x, mask_strategys, loop_i):+ masks = []+ for i, mask_strategy in enumerate(mask_strategys):+ mask_strategy = mask_strategy.split(";")+ mask = torch.ones(z.shape[2], dtype=torch.float, device=z.device)+ for mst in mask_strategy:+ mask_batch = mst.split(",")+ loop_id, m_id, m_ref_start, m_length, m_target_start = mask_batch[:5]+ edit_ratio = mask_batch[5] if len(mask_batch) == 6 else 0.0+ loop_id = int(loop_id)+ if loop_id != loop_i:+ continue+ m_id = int(m_id)+ m_ref_start = int(m_ref_start)+ m_length = int(m_length)+ m_target_start = int(m_target_start)+ edit_ratio = float(edit_ratio)+ ref = refs_x[i][m_id] # [C, T, H, W]+ if m_ref_start < 0:+ m_ref_start = ref.shape[1] + m_ref_start+ if m_target_start < 0:+ # z: [B, C, T, H, W]+ m_target_start = z.shape[2] + m_target_start+ z[i, :, m_target_start : m_target_start + m_length] = ref[:, m_ref_start : m_ref_start + m_length]+ mask[m_target_start : m_target_start + m_length] = edit_ratio+ masks.append(mask)+ masks = torch.stack(masks)+ return masks, z+++def process_prompts(prompts, num_loop):+ ret_prompts = []+ for prompt in prompts:+ if prompt.startswith("|0|"):+ prompt_list = prompt.split("|")[1:]+ text_list = []+ for i in range(0, len(prompt_list), 2):+ start_loop = int(prompt_list[i])+ text = prompt_list[i + 1]+ end_loop = int(prompt_list[i + 2]) if i + 2 < len(prompt_list) else num_loop+ text_list.extend([text] * (end_loop - start_loop))+ assert len(text_list) == num_loop+ ret_prompts.append(text_list)+ else:+ ret_prompts.append([prompt] * num_loop)+ return ret_prompts+++def main():+ # ======================================================+ # 1. cfg and init distributed env+ # ======================================================+ cfg = parse_configs(training=False)+ print(cfg)++ # init distributed+ colossalai.launch_from_torch({})+ coordinator = DistCoordinator()++ if coordinator.world_size > 1:+ set_sequence_parallel_group(dist.group.WORLD)+ enable_sequence_parallelism = True+ else:+ enable_sequence_parallelism = False++ # ======================================================+ # 2. runtime variables+ # ======================================================+ torch.set_grad_enabled(False)+ torch.backends.cuda.matmul.allow_tf32 = True+ torch.backends.cudnn.allow_tf32 = True+ device = "cuda" if torch.cuda.is_available() else "cpu"+ dtype = to_torch_dtype(cfg.dtype)+ set_random_seed(seed=cfg.seed)+ prompts = cfg.prompt++ # ======================================================+ # 3. build model & load weights+ # ======================================================+ # 3.1. build model+ input_size = (cfg.num_frames, *cfg.image_size)+ vae = build_module(cfg.vae, MODELS)+ latent_size = vae.get_latent_size(input_size)+ text_encoder = build_module(cfg.text_encoder, MODELS, device=device) # T5 must be fp32+ model = build_module(+ cfg.model,+ MODELS,+ input_size=latent_size,+ in_channels=vae.out_channels,+ caption_channels=text_encoder.output_dim,+ model_max_length=text_encoder.model_max_length,+ dtype=dtype,+ enable_sequence_parallelism=enable_sequence_parallelism,+ )+ text_encoder.y_embedder = model.y_embedder # hack for classifier-free guidance++ # 3.2. move to device & eval+ vae = vae.to(device, dtype).eval()+ model = model.to(device, dtype).eval()++ # 3.3. build scheduler+ scheduler = build_module(cfg.scheduler, SCHEDULERS)++ # 3.4. support for multi-resolution+ model_args = dict()+ if cfg.multi_resolution == "PixArtMS":+ image_size = cfg.image_size+ hw = torch.tensor([image_size], device=device, dtype=dtype).repeat(cfg.batch_size, 1)+ ar = torch.tensor([[image_size[0] / image_size[1]]], device=device, dtype=dtype).repeat(cfg.batch_size, 1)+ model_args["data_info"] = dict(ar=ar, hw=hw)+ elif cfg.multi_resolution == "STDiT2":+ image_size = cfg.image_size+ height = torch.tensor([image_size[0]], device=device, dtype=dtype).repeat(cfg.batch_size)+ width = torch.tensor([image_size[1]], device=device, dtype=dtype).repeat(cfg.batch_size)+ num_frames = torch.tensor([cfg.num_frames], device=device, dtype=dtype).repeat(cfg.batch_size)+ ar = torch.tensor([image_size[0] / image_size[1]], device=device, dtype=dtype).repeat(cfg.batch_size)+ model_args["height"] = height+ model_args["width"] = width+ model_args["num_frames"] = num_frames+ model_args["ar"] = ar++ # 3.5 reference+ if cfg.reference_path is not None:+ assert len(cfg.reference_path) == len(prompts)+ assert len(cfg.reference_path) == len(cfg.mask_strategy)++ # ======================================================+ # 4. inference+ # ======================================================+ sample_idx = 0+ save_dir = cfg.save_dir+ os.makedirs(save_dir, exist_ok=True)++ # 4.1. batch generation+ for i in range(0, len(prompts), cfg.batch_size):+ batch_prompts_loops = process_prompts(prompts[i : i + cfg.batch_size], cfg.loop)+ video_clips = []++ # 4.2. load reference videos & images+ if cfg.reference_path is not None:+ refs_x = collect_references_batch(cfg.reference_path[i : i + cfg.batch_size], vae, cfg.image_size)+ mask_strategy = cfg.mask_strategy[i : i + cfg.batch_size]++ # 4.3. long video generation+ for loop_i in range(cfg.loop):+ # 4.4 sample in hidden space+ batch_prompts = [prompt[loop_i] for prompt in batch_prompts_loops]+ z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)++ # 4.5. apply mask strategy+ masks = None+ if cfg.reference_path is not None:+ if loop_i > 0:+ ref_x = vae.encode(video_clips[-1])+ for j, refs in enumerate(refs_x):+ refs.append(ref_x[j])+ mask_strategy[+ j+ ] += f";{loop_i},{len(refs)-1},-{cfg.condition_frame_length},{cfg.condition_frame_length},0,0"+ masks, z = apply_mask_strategy(z, refs_x, mask_strategy, loop_i)+ # 4.6. diffusion sampling+ samples = scheduler.sample(+ model,+ text_encoder,+ z=z,+ prompts=batch_prompts,+ device=device,+ additional_args=model_args,+ mask=masks, # scheduler must support mask+ )+ samples = vae.decode(samples.to(dtype))+ video_clips.append(samples)++ # 4.7. save video+ if loop_i == cfg.loop - 1:+ if coordinator.is_master():+ for idx in range(len(video_clips[0])):+ video_clips_i = [video_clips[0][idx]] + [+ video_clips[i][idx][:, cfg.condition_frame_length :] for i in range(1, cfg.loop)+ ]+ video = torch.cat(video_clips_i, dim=1)+ print(f"Prompt: {prompts[i + idx]}")+ save_path = os.path.join(save_dir, f"sample_{prompts[i + idx]}")+ save_sample(video, fps=cfg.fps, save_path=save_path)+ sample_idx += 1+++if __name__ == "__main__":+ main()diff --git a/scripts/inference.py b/scripts/inference.pyindex c59e560..733118f 100644--- a/scripts/inference.py+++ b/scripts/inference.py@@ -100,7 +100,12 @@ def main():# 4. inference# ======================================================sample_idx = 0- sample_name = cfg.sample_name if cfg.sample_name is not None else "sample"+ if cfg.sample_name is not None:+ sample_name = cfg.sample_name+ elif cfg.prompt_as_path:+ sample_name = ""+ else:+ sample_name = "sample"save_dir = cfg.save_diros.makedirs(save_dir, exist_ok=True)@@ -112,22 +117,33 @@ def main():z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)# 4.3. diffusion sampling- samples = scheduler.sample(- model,- text_encoder,- z=z,- prompts=batch_prompts,- device=device,- additional_args=model_args,- )- samples = vae.decode(samples.to(dtype))-- if not use_dist or coordinator.is_master():- for idx, sample in enumerate(samples):- print(f"Prompt: {batch_prompts[idx]}")- save_path = os.path.join(save_dir, f"{sample_name}_{sample_idx}")- save_sample(sample, fps=cfg.fps // cfg.frame_interval, save_path=save_path)- sample_idx += 1+ old_sample_idx = sample_idx+ # generate multiple samples for each prompt+ for k in range(cfg.num_sample):+ sample_idx = old_sample_idx++ samples = scheduler.sample(+ model,+ text_encoder,+ z=z,+ prompts=batch_prompts,+ device=device,+ additional_args=model_args,+ )+ samples = vae.decode(samples.to(dtype))++ if not use_dist or coordinator.is_master():+ for idx, sample in enumerate(samples):+ print(f"Prompt: {batch_prompts[idx]}")+ if cfg.prompt_as_path:+ sample_name_suffix = batch_prompts[idx]+ else:+ sample_name_suffix = f"_{sample_idx}"+ save_path = os.path.join(save_dir, f"{sample_name}{sample_name_suffix}")+ if cfg.num_sample != 1:+ save_path = f"{save_path}-{k}"+ save_sample(sample, fps=cfg.fps // cfg.frame_interval, save_path=save_path)+ sample_idx += 1if __name__ == "__main__":diff --git a/tests/test_attn.py b/tests/test_attn.pyindex 22444a1..48f8c43 100644--- a/tests/test_attn.py+++ b/tests/test_attn.py@@ -21,9 +21,7 @@ def run_attn(enable_flashattn: bool):rope=rope.rotate_queries_or_keys,enable_flashattn=enable_flashattn,).to(device=get_current_device(), dtype=torch.bfloat16)- x = torch.randn(- B, S, H, device=get_current_device(), dtype=torch.bfloat16- ).requires_grad_()+ x = torch.randn(B, S, H, device=get_current_device(), dtype=torch.bfloat16).requires_grad_()y = attn(x)y.mean().backward()print(f"Peak memory: {get_accelerator().max_memory_allocated() / 1024**2:.2f} MB")diff --git a/tools/datasets/datautil.py b/tools/datasets/datautil.pyindex cfe5147..ef4ece7 100644--- a/tools/datasets/datautil.py+++ b/tools/datasets/datautil.py@@ -20,14 +20,13 @@ tqdm.pandas()try:from pandarallel import pandarallel- pandarallel.initialize(progress_bar=True)- pandas_has_parallel = True+ PANDA_USE_PARALLEL = Trueexcept ImportError:- pandas_has_parallel = False+ PANDA_USE_PARALLEL = Falsedef apply(df, func, **kwargs):- if pandas_has_parallel:+ if PANDA_USE_PARALLEL:return df.parallel_apply(func, **kwargs)return df.progress_apply(func, **kwargs)@@ -554,6 +553,7 @@ def parse_args():parser.add_argument("--output", type=str, default=None, help="output path")parser.add_argument("--format", type=str, default="csv", help="output format", choices=["csv", "parquet"])parser.add_argument("--disable-parallel", action="store_true", help="disable parallel processing")+ parser.add_argument("--num-workers", type=int, default=None, help="number of workers")parser.add_argument("--seed", type=int, default=None, help="random seed")# special case@@ -684,7 +684,12 @@ def get_output_path(args, input_name):if __name__ == "__main__":args = parse_args()if args.disable_parallel:- pandas_has_parallel = False+ PANDA_USE_PARALLEL = False+ if PANDA_USE_PARALLEL:+ if args.num_workers is not None:+ pandarallel.initialize(nb_workers=args.num_workers, progress_bar=True)+ else:+ pandarallel.initialize(progress_bar=True)if args.seed is not None:random.seed(args.seed)np.random.seed(args.seed)diff --git a/tools/datasets/split.py b/tools/datasets/split.pyindex 611183d..4e312b2 100644--- a/tools/datasets/split.py+++ b/tools/datasets/split.py@@ -37,9 +37,7 @@ def split_by_bucket(continueif bucket_cnt[bucket_id] < limit:bucket_cnt[bucket_id] += 1- output_df = pd.concat(- [output_df, pd.DataFrame([row])], ignore_index=True- )+ output_df = pd.concat([output_df, pd.DataFrame([row])], ignore_index=True)if len(output_df) >= total_limit:breakif len(output_df) >= total_limit:@@ -71,6 +69,4 @@ if __name__ == "__main__":p = 1.0d[frames] = (p, bs)bucket = Bucket(bucket_config)- split_by_bucket(- bucket, args.input, args.output, args.limit, cfg.dataset.frame_interval- )+ split_by_bucket(bucket, args.input, args.output, args.limit, cfg.dataset.frame_interval)diff --git a/tools/frame_interpolation/README.md b/tools/frame_interpolation/README.mdnew file mode 100644index 0000000..8418e66--- /dev/null+++ b/tools/frame_interpolation/README.md@@ -0,0 +1,42 @@+# Frame Interpolation++For current version, we sample 1 frame out of 3 frames in the video. Although we are going to use VAE to avoid frame loss, we provide a frame interpolation tool to interpolate the video now. The frame interpolation tool is based on [AMT](https://github.com/MCG-NKU/AMT).++Interpolation can be useful for scenery videos, but it may not be suitable for videos with fast motion.++## Requirement++```bash+conda install -c conda-forge opencv+pip install imageio+```++## Model++We use **AMT** as our frame interpolation model. After sampling, you can use frame interpolation model to interpolate your video smoothly.++## Usage++The ckpt file will be automatically downloaded in user's `.cache` directory. You can use frame interpolation to your video file or a video folder.++1. Process a video file++```python+python -m tools.frame_interpolation.interpolation your_video.mp4+```++2. Process all video file in target directory++```python+python -m tools.frame_interpolation.interpolation your_video_dir --output_path samples/interpolation+```++The output video will be stored at `output_path` and its duration time is equal `the total number of frames after frame interpolation / the frame rate`++### Command Line Arguments++* `input`: Path of the input video. **Video path** or **Folder path(with --folder)**+* `--ckpt`: Pretrained model of [AMT](https://github.com/MCG-NKU/AMT). Default path: `~/.cache/amt-g.pth`.+* `--niter`: Iterations of interpolation. With $m$ input frames, `[N_ITER]` $=n$ corresponds to $2^n\times (m-1)+1$ output frames.+* `--fps`: Frame rate of the input video. (Default: 8)+* `--output_path`: **Folder Path** of the output video.diff --git a/tools/frame_interpolation/__init__.py b/tools/frame_interpolation/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/tools/frame_interpolation/interpolation.py b/tools/frame_interpolation/interpolation.pynew file mode 100644index 0000000..808bf8b--- /dev/null+++ b/tools/frame_interpolation/interpolation.py@@ -0,0 +1,219 @@+# this script is modified from https://github.com/MCG-NKU/AMT/blob/main/demos/demo_2x.py+import argparse+import os+import os.path as osp++import cv2+import numpy as np+import torch++from opensora.utils.ckpt_utils import download_model++from .networks.amt_g import Model+from .utils.utils import InputPadder, img2tensor, tensor2img++hf_endpoint = os.environ.get("HF_ENDPOINT")+if hf_endpoint is None:+ hf_endpoint = "https://huggingface.co"+VID_EXT = [".mp4", ".avi", ".mov", ".mkv", ".flv", ".wmv", ".webm"]+network_cfg = {+ "params": {+ "corr_radius": 3,+ "corr_lvls": 4,+ "num_flows": 5,+ },+}+device = "cuda" if torch.cuda.is_available() else "cpu"+++def init():+ """+ initialize the device and the anchor resolution.+ """++ if device == "cuda":+ anchor_resolution = 1024 * 512+ anchor_memory = 1500 * 1024**2+ anchor_memory_bias = 2500 * 1024**2+ vram_avail = torch.cuda.get_device_properties(device).total_memory+ print("VRAM available: {:.1f} MB".format(vram_avail / 1024**2))+ else:+ # Do not resize in cpu mode+ anchor_resolution = 8192 * 8192+ anchor_memory = 1+ anchor_memory_bias = 0+ vram_avail = 1++ return anchor_resolution, anchor_memory, anchor_memory_bias, vram_avail+++def get_input_video_from_path(input_path):+ """+ Get the input video from the input_path.++ params:+ input_path: str, the path of the input video.+ devices: str, the device to run the model.+ returns:+ inputs: list, the list of the input frames.+ scale: float, the scale of the input frames.+ padder: InputPadder, the padder to pad the input frames.+ """++ anchor_resolution, anchor_memory, anchor_memory_bias, vram_avail = init()++ if osp.splitext(input_path)[-1].lower() in VID_EXT:+ vcap = cv2.VideoCapture(input_path)++ inputs = []+ w = int(vcap.get(cv2.CAP_PROP_FRAME_WIDTH))+ h = int(vcap.get(cv2.CAP_PROP_FRAME_HEIGHT))+ scale = anchor_resolution / (h * w) * np.sqrt((vram_avail - anchor_memory_bias) / anchor_memory)+ scale = 1 if scale > 1 else scale+ scale = 1 / np.floor(1 / np.sqrt(scale) * 16) * 16+ if scale < 1:+ print(f"Due to the limited VRAM, the video will be scaled by {scale:.2f}")+ padding = int(16 / scale)+ padder = InputPadder((h, w), padding)+ while True:+ ret, frame = vcap.read()+ if ret is False:+ break+ frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)+ frame_t = img2tensor(frame).to(device)+ frame_t = padder.pad(frame_t)+ inputs.append(frame_t)+ print(f"Loading the [video] from {input_path}, the number of frames [{len(inputs)}]")+ else:+ raise TypeError("Input should be a video.")++ return inputs, scale, padder+++def load_model(ckpt):+ """+ load the frame interpolation model.+ """+ params = network_cfg.get("params", {})+ model = Model(**params)+ model.load_state_dict(ckpt["state_dict"])+ model = model.to(device)+ model.eval()+ return model+++def interpolater(model, inputs, scale, padder, iters=1):+ """+ interpolating with the interpolation model.++ params:+ model: nn.Module, the frame interpolation model.+ inputs: list, the list of the input frames.+ scale: float, the scale of the input frames.+ iters: int, the number of iterations of interpolation. The final frames model generating is 2 ** iters * (m - 1) + 1 and m is input frames.+ returns:+ outputs: list, the list of the output frames.+ """++ print("Start frame interpolation:")+ embt = torch.tensor(1 / 2).float().view(1, 1, 1, 1).to(device)++ for i in range(iters):+ print(f"Iter {i+1}. input_frames={len(inputs)} output_frames={2*len(inputs)-1}")+ outputs = [inputs[0]]+ for in_0, in_1 in zip(inputs[:-1], inputs[1:]):+ in_0 = in_0.to(device)+ in_1 = in_1.to(device)+ with torch.no_grad():+ imgt_pred = model(in_0, in_1, embt, scale_factor=scale, eval=True)["imgt_pred"]+ outputs += [imgt_pred.cpu(), in_1.cpu()]+ inputs = outputs++ outputs = padder.unpad(*outputs)+ return outputs+++def write(outputs, input_path, output_path, fps=30):+ """+ write results to the output_path.+ """++ if osp.exists(output_path) is False:+ os.makedirs(output_path)++ size = outputs[0].shape[2:][::-1]++ _, file_name_with_extension = os.path.split(input_path)+ file_name, _ = os.path.splitext(file_name_with_extension)++ save_video_path = f"{output_path}/fps{fps}_{file_name}.mp4"+ fourcc = cv2.VideoWriter_fourcc(*"avc1")+ writer = cv2.VideoWriter(save_video_path, fourcc, fps, size)++ for i, imgt_pred in enumerate(outputs):+ imgt_pred = tensor2img(imgt_pred)+ imgt_pred = cv2.cvtColor(imgt_pred, cv2.COLOR_RGB2BGR)+ writer.write(imgt_pred)+ print(f"Demo video is saved to [{save_video_path}]")++ writer.release()+++def process(+ model,+ image_path,+ output_path,+ fps,+ iters,+):+ inputs, scale, padder = get_input_video_from_path(image_path)+ outputs = interpolater(model, inputs, scale, padder, iters)+ write(outputs, image_path, output_path, fps)+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("input", help="Input video.")+ parser.add_argument("--ckpt", type=str, default="./pretrained_models/amt-g.pth", help="The pretrained model.")+ parser.add_argument(+ "--niters",+ type=int,+ default=1,+ help="Iter of Interpolation. The number of frames will be double after per iter.",+ )+ parser.add_argument("--output_path", type=str, default="samples", help="Output path.")+ parser.add_argument("--fps", type=int, default=8, help="Frames rate of the output video.")+ parser.add_argument("--folder", action="store_true", help="If the input is a folder, set this flag.")+ args = parser.parse_args()++ times_frame = 2**args.niters+ old_fps = args.fps+ args.fps = args.fps * times_frame+ print(f"Interpolation will turn {old_fps}fps video to {args.fps}fps video.")+ args.input = os.path.expanduser(args.input)+ args.ckpt = os.path.expanduser(args.ckpt)+ args.folder = osp.splitext(args.input)[-1].lower() not in VID_EXT+ args.ckpt = download_model(local_path=args.ckpt, url=hf_endpoint + "/lalala125/AMT/resolve/main/amt-g.pth")+ return args+++if __name__ == "__main__":+ args = parse_args()+ ckpt_path = args.ckpt+ input_path = args.input+ output_path = args.output_path+ iters = int(args.niters)+ fps = int(args.fps)++ model = load_model(ckpt_path)++ if args.folder:+ for file in os.listdir(input_path):+ if osp.splitext(input_path)[-1].lower() in VID_EXT:+ vid_path = os.path.join(input_path, file)+ process(model, vid_path, output_path, fps, iters)+ else:+ process(model, input_path, output_path, fps, iters)++ print("Interpolation is done.")+ print(f"Output path: {output_path}")diff --git a/tools/frame_interpolation/networks/__init__.py b/tools/frame_interpolation/networks/__init__.pynew file mode 100644index 0000000..4db0516--- /dev/null+++ b/tools/frame_interpolation/networks/__init__.py@@ -0,0 +1 @@+from .amt_g import Modeldiff --git a/tools/frame_interpolation/networks/amt_g.py b/tools/frame_interpolation/networks/amt_g.pynew file mode 100644index 0000000..84b28cb--- /dev/null+++ b/tools/frame_interpolation/networks/amt_g.py@@ -0,0 +1,156 @@+import torch+import torch.nn as nn++from .blocks.feat_enc import LargeEncoder+from .blocks.ifrnet import Encoder, InitDecoder, IntermediateDecoder, resize+from .blocks.multi_flow import MultiFlowDecoder, multi_flow_combine+from .blocks.raft import BasicUpdateBlock, BidirCorrBlock, coords_grid+++class Model(nn.Module):+ def __init__(self, corr_radius=3, corr_lvls=4, num_flows=5, channels=[84, 96, 112, 128], skip_channels=84):+ super(Model, self).__init__()+ self.radius = corr_radius+ self.corr_levels = corr_lvls+ self.num_flows = num_flows++ self.feat_encoder = LargeEncoder(output_dim=128, norm_fn="instance", dropout=0.0)+ self.encoder = Encoder(channels, large=True)+ self.decoder4 = InitDecoder(channels[3], channels[2], skip_channels)+ self.decoder3 = IntermediateDecoder(channels[2], channels[1], skip_channels)+ self.decoder2 = IntermediateDecoder(channels[1], channels[0], skip_channels)+ self.decoder1 = MultiFlowDecoder(channels[0], skip_channels, num_flows)++ self.update4 = self._get_updateblock(112, None)+ self.update3_low = self._get_updateblock(96, 2.0)+ self.update2_low = self._get_updateblock(84, 4.0)++ self.update3_high = self._get_updateblock(96, None)+ self.update2_high = self._get_updateblock(84, None)++ self.comb_block = nn.Sequential(+ nn.Conv2d(3 * self.num_flows, 6 * self.num_flows, 7, 1, 3),+ nn.PReLU(6 * self.num_flows),+ nn.Conv2d(6 * self.num_flows, 3, 7, 1, 3),+ )++ def _get_updateblock(self, cdim, scale_factor=None):+ return BasicUpdateBlock(+ cdim=cdim,+ hidden_dim=192,+ flow_dim=64,+ corr_dim=256,+ corr_dim2=192,+ fc_dim=188,+ scale_factor=scale_factor,+ corr_levels=self.corr_levels,+ radius=self.radius,+ )++ def _corr_scale_lookup(self, corr_fn, coord, flow0, flow1, embt, downsample=1):+ # convert t -> 0 to 0 -> 1 | convert t -> 1 to 1 -> 0+ # based on linear assumption+ t1_scale = 1.0 / embt+ t0_scale = 1.0 / (1.0 - embt)+ if downsample != 1:+ inv = 1 / downsample+ flow0 = inv * resize(flow0, scale_factor=inv)+ flow1 = inv * resize(flow1, scale_factor=inv)++ corr0, corr1 = corr_fn(coord + flow1 * t1_scale, coord + flow0 * t0_scale)+ corr = torch.cat([corr0, corr1], dim=1)+ flow = torch.cat([flow0, flow1], dim=1)+ return corr, flow++ def forward(self, img0, img1, embt, scale_factor=1.0, eval=False, **kwargs):+ mean_ = torch.cat([img0, img1], 2).mean(1, keepdim=True).mean(2, keepdim=True).mean(3, keepdim=True)+ img0 = img0 - mean_+ img1 = img1 - mean_+ img0_ = resize(img0, scale_factor) if scale_factor != 1.0 else img0+ img1_ = resize(img1, scale_factor) if scale_factor != 1.0 else img1+ b, _, h, w = img0_.shape+ coord = coords_grid(b, h // 8, w // 8, img0.device)++ fmap0, fmap1 = self.feat_encoder([img0_, img1_]) # [1, 128, H//8, W//8]+ corr_fn = BidirCorrBlock(fmap0, fmap1, radius=self.radius, num_levels=self.corr_levels)++ # f0_1: [1, c0, H//2, W//2] | f0_2: [1, c1, H//4, W//4]+ # f0_3: [1, c2, H//8, W//8] | f0_4: [1, c3, H//16, W//16]+ f0_1, f0_2, f0_3, f0_4 = self.encoder(img0_)+ f1_1, f1_2, f1_3, f1_4 = self.encoder(img1_)++ ######################################### the 4th decoder #########################################+ up_flow0_4, up_flow1_4, ft_3_ = self.decoder4(f0_4, f1_4, embt)+ corr_4, flow_4 = self._corr_scale_lookup(corr_fn, coord, up_flow0_4, up_flow1_4, embt, downsample=1)++ # residue update with lookup corr+ delta_ft_3_, delta_flow_4 = self.update4(ft_3_, flow_4, corr_4)+ delta_flow0_4, delta_flow1_4 = torch.chunk(delta_flow_4, 2, 1)+ up_flow0_4 = up_flow0_4 + delta_flow0_4+ up_flow1_4 = up_flow1_4 + delta_flow1_4+ ft_3_ = ft_3_ + delta_ft_3_++ ######################################### the 3rd decoder #########################################+ up_flow0_3, up_flow1_3, ft_2_ = self.decoder3(ft_3_, f0_3, f1_3, up_flow0_4, up_flow1_4)+ corr_3, flow_3 = self._corr_scale_lookup(corr_fn, coord, up_flow0_3, up_flow1_3, embt, downsample=2)++ # residue update with lookup corr+ delta_ft_2_, delta_flow_3 = self.update3_low(ft_2_, flow_3, corr_3)+ delta_flow0_3, delta_flow1_3 = torch.chunk(delta_flow_3, 2, 1)+ up_flow0_3 = up_flow0_3 + delta_flow0_3+ up_flow1_3 = up_flow1_3 + delta_flow1_3+ ft_2_ = ft_2_ + delta_ft_2_++ # residue update with lookup corr (hr)+ corr_3 = resize(corr_3, scale_factor=2.0)+ up_flow_3 = torch.cat([up_flow0_3, up_flow1_3], dim=1)+ delta_ft_2_, delta_up_flow_3 = self.update3_high(ft_2_, up_flow_3, corr_3)+ ft_2_ += delta_ft_2_+ up_flow0_3 += delta_up_flow_3[:, 0:2]+ up_flow1_3 += delta_up_flow_3[:, 2:4]++ ######################################### the 2nd decoder #########################################+ up_flow0_2, up_flow1_2, ft_1_ = self.decoder2(ft_2_, f0_2, f1_2, up_flow0_3, up_flow1_3)+ corr_2, flow_2 = self._corr_scale_lookup(corr_fn, coord, up_flow0_2, up_flow1_2, embt, downsample=4)++ # residue update with lookup corr+ delta_ft_1_, delta_flow_2 = self.update2_low(ft_1_, flow_2, corr_2)+ delta_flow0_2, delta_flow1_2 = torch.chunk(delta_flow_2, 2, 1)+ up_flow0_2 = up_flow0_2 + delta_flow0_2+ up_flow1_2 = up_flow1_2 + delta_flow1_2+ ft_1_ = ft_1_ + delta_ft_1_++ # residue update with lookup corr (hr)+ corr_2 = resize(corr_2, scale_factor=4.0)+ up_flow_2 = torch.cat([up_flow0_2, up_flow1_2], dim=1)+ delta_ft_1_, delta_up_flow_2 = self.update2_high(ft_1_, up_flow_2, corr_2)+ ft_1_ += delta_ft_1_+ up_flow0_2 += delta_up_flow_2[:, 0:2]+ up_flow1_2 += delta_up_flow_2[:, 2:4]++ ######################################### the 1st decoder #########################################+ up_flow0_1, up_flow1_1, mask, img_res = self.decoder1(ft_1_, f0_1, f1_1, up_flow0_2, up_flow1_2)++ if scale_factor != 1.0:+ up_flow0_1 = resize(up_flow0_1, scale_factor=(1.0 / scale_factor)) * (1.0 / scale_factor)+ up_flow1_1 = resize(up_flow1_1, scale_factor=(1.0 / scale_factor)) * (1.0 / scale_factor)+ mask = resize(mask, scale_factor=(1.0 / scale_factor))+ img_res = resize(img_res, scale_factor=(1.0 / scale_factor))++ # Merge multiple predictions+ imgt_pred = multi_flow_combine(self.comb_block, img0, img1, up_flow0_1, up_flow1_1, mask, img_res, mean_)+ imgt_pred = torch.clamp(imgt_pred, 0, 1)++ if eval:+ return {+ "imgt_pred": imgt_pred,+ }+ else:+ up_flow0_1 = up_flow0_1.reshape(b, self.num_flows, 2, h, w)+ up_flow1_1 = up_flow1_1.reshape(b, self.num_flows, 2, h, w)+ return {+ "imgt_pred": imgt_pred,+ "flow0_pred": [up_flow0_1, up_flow0_2, up_flow0_3, up_flow0_4],+ "flow1_pred": [up_flow1_1, up_flow1_2, up_flow1_3, up_flow1_4],+ "ft_pred": [ft_1_, ft_2_, ft_3_],+ }diff --git a/tools/frame_interpolation/networks/blocks/__init__.py b/tools/frame_interpolation/networks/blocks/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/tools/frame_interpolation/networks/blocks/feat_enc.py b/tools/frame_interpolation/networks/blocks/feat_enc.pynew file mode 100644index 0000000..4798338--- /dev/null+++ b/tools/frame_interpolation/networks/blocks/feat_enc.py@@ -0,0 +1,335 @@+import torch+import torch.nn as nn+++class BottleneckBlock(nn.Module):+ def __init__(self, in_planes, planes, norm_fn="group", stride=1):+ super(BottleneckBlock, self).__init__()++ self.conv1 = nn.Conv2d(in_planes, planes // 4, kernel_size=1, padding=0)+ self.conv2 = nn.Conv2d(planes // 4, planes // 4, kernel_size=3, padding=1, stride=stride)+ self.conv3 = nn.Conv2d(planes // 4, planes, kernel_size=1, padding=0)+ self.relu = nn.ReLU(inplace=True)++ num_groups = planes // 8++ if norm_fn == "group":+ self.norm1 = nn.GroupNorm(num_groups=num_groups, num_channels=planes // 4)+ self.norm2 = nn.GroupNorm(num_groups=num_groups, num_channels=planes // 4)+ self.norm3 = nn.GroupNorm(num_groups=num_groups, num_channels=planes)+ if not stride == 1:+ self.norm4 = nn.GroupNorm(num_groups=num_groups, num_channels=planes)++ elif norm_fn == "batch":+ self.norm1 = nn.BatchNorm2d(planes // 4)+ self.norm2 = nn.BatchNorm2d(planes // 4)+ self.norm3 = nn.BatchNorm2d(planes)+ if not stride == 1:+ self.norm4 = nn.BatchNorm2d(planes)++ elif norm_fn == "instance":+ self.norm1 = nn.InstanceNorm2d(planes // 4)+ self.norm2 = nn.InstanceNorm2d(planes // 4)+ self.norm3 = nn.InstanceNorm2d(planes)+ if not stride == 1:+ self.norm4 = nn.InstanceNorm2d(planes)++ elif norm_fn == "none":+ self.norm1 = nn.Sequential()+ self.norm2 = nn.Sequential()+ self.norm3 = nn.Sequential()+ if not stride == 1:+ self.norm4 = nn.Sequential()++ if stride == 1:+ self.downsample = None++ else:+ self.downsample = nn.Sequential(nn.Conv2d(in_planes, planes, kernel_size=1, stride=stride), self.norm4)++ def forward(self, x):+ y = x+ y = self.relu(self.norm1(self.conv1(y)))+ y = self.relu(self.norm2(self.conv2(y)))+ y = self.relu(self.norm3(self.conv3(y)))++ if self.downsample is not None:+ x = self.downsample(x)++ return self.relu(x + y)+++class ResidualBlock(nn.Module):+ def __init__(self, in_planes, planes, norm_fn="group", stride=1):+ super(ResidualBlock, self).__init__()++ self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, padding=1, stride=stride)+ self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, padding=1)+ self.relu = nn.ReLU(inplace=True)++ num_groups = planes // 8++ if norm_fn == "group":+ self.norm1 = nn.GroupNorm(num_groups=num_groups, num_channels=planes)+ self.norm2 = nn.GroupNorm(num_groups=num_groups, num_channels=planes)+ if not stride == 1:+ self.norm3 = nn.GroupNorm(num_groups=num_groups, num_channels=planes)++ elif norm_fn == "batch":+ self.norm1 = nn.BatchNorm2d(planes)+ self.norm2 = nn.BatchNorm2d(planes)+ if not stride == 1:+ self.norm3 = nn.BatchNorm2d(planes)++ elif norm_fn == "instance":+ self.norm1 = nn.InstanceNorm2d(planes)+ self.norm2 = nn.InstanceNorm2d(planes)+ if not stride == 1:+ self.norm3 = nn.InstanceNorm2d(planes)++ elif norm_fn == "none":+ self.norm1 = nn.Sequential()+ self.norm2 = nn.Sequential()+ if not stride == 1:+ self.norm3 = nn.Sequential()++ if stride == 1:+ self.downsample = None++ else:+ self.downsample = nn.Sequential(nn.Conv2d(in_planes, planes, kernel_size=1, stride=stride), self.norm3)++ def forward(self, x):+ y = x+ y = self.relu(self.norm1(self.conv1(y)))+ y = self.relu(self.norm2(self.conv2(y)))++ if self.downsample is not None:+ x = self.downsample(x)++ return self.relu(x + y)+++class SmallEncoder(nn.Module):+ def __init__(self, output_dim=128, norm_fn="batch", dropout=0.0):+ super(SmallEncoder, self).__init__()+ self.norm_fn = norm_fn++ if self.norm_fn == "group":+ self.norm1 = nn.GroupNorm(num_groups=8, num_channels=32)++ elif self.norm_fn == "batch":+ self.norm1 = nn.BatchNorm2d(32)++ elif self.norm_fn == "instance":+ self.norm1 = nn.InstanceNorm2d(32)++ elif self.norm_fn == "none":+ self.norm1 = nn.Sequential()++ self.conv1 = nn.Conv2d(3, 32, kernel_size=7, stride=2, padding=3)+ self.relu1 = nn.ReLU(inplace=True)++ self.in_planes = 32+ self.layer1 = self._make_layer(32, stride=1)+ self.layer2 = self._make_layer(64, stride=2)+ self.layer3 = self._make_layer(96, stride=2)++ self.dropout = None+ if dropout > 0:+ self.dropout = nn.Dropout2d(p=dropout)++ self.conv2 = nn.Conv2d(96, output_dim, kernel_size=1)++ for m in self.modules():+ if isinstance(m, nn.Conv2d):+ nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu")+ elif isinstance(m, (nn.BatchNorm2d, nn.InstanceNorm2d, nn.GroupNorm)):+ if m.weight is not None:+ nn.init.constant_(m.weight, 1)+ if m.bias is not None:+ nn.init.constant_(m.bias, 0)++ def _make_layer(self, dim, stride=1):+ layer1 = BottleneckBlock(self.in_planes, dim, self.norm_fn, stride=stride)+ layer2 = BottleneckBlock(dim, dim, self.norm_fn, stride=1)+ layers = (layer1, layer2)++ self.in_planes = dim+ return nn.Sequential(*layers)++ def forward(self, x):+ # if input is list, combine batch dimension+ is_list = isinstance(x, tuple) or isinstance(x, list)+ if is_list:+ batch_dim = x[0].shape[0]+ x = torch.cat(x, dim=0)++ x = self.conv1(x)+ x = self.norm1(x)+ x = self.relu1(x)++ x = self.layer1(x)+ x = self.layer2(x)+ x = self.layer3(x)+ x = self.conv2(x)++ if self.training and self.dropout is not None:+ x = self.dropout(x)++ if is_list:+ x = torch.split(x, [batch_dim, batch_dim], dim=0)++ return x+++class BasicEncoder(nn.Module):+ def __init__(self, output_dim=128, norm_fn="batch", dropout=0.0):+ super(BasicEncoder, self).__init__()+ self.norm_fn = norm_fn++ if self.norm_fn == "group":+ self.norm1 = nn.GroupNorm(num_groups=8, num_channels=64)++ elif self.norm_fn == "batch":+ self.norm1 = nn.BatchNorm2d(64)++ elif self.norm_fn == "instance":+ self.norm1 = nn.InstanceNorm2d(64)++ elif self.norm_fn == "none":+ self.norm1 = nn.Sequential()++ self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)+ self.relu1 = nn.ReLU(inplace=True)++ self.in_planes = 64+ self.layer1 = self._make_layer(64, stride=1)+ self.layer2 = self._make_layer(72, stride=2)+ self.layer3 = self._make_layer(128, stride=2)++ # output convolution+ self.conv2 = nn.Conv2d(128, output_dim, kernel_size=1)++ self.dropout = None+ if dropout > 0:+ self.dropout = nn.Dropout2d(p=dropout)++ for m in self.modules():+ if isinstance(m, nn.Conv2d):+ nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu")+ elif isinstance(m, (nn.BatchNorm2d, nn.InstanceNorm2d, nn.GroupNorm)):+ if m.weight is not None:+ nn.init.constant_(m.weight, 1)+ if m.bias is not None:+ nn.init.constant_(m.bias, 0)++ def _make_layer(self, dim, stride=1):+ layer1 = ResidualBlock(self.in_planes, dim, self.norm_fn, stride=stride)+ layer2 = ResidualBlock(dim, dim, self.norm_fn, stride=1)+ layers = (layer1, layer2)++ self.in_planes = dim+ return nn.Sequential(*layers)++ def forward(self, x):+ # if input is list, combine batch dimension+ is_list = isinstance(x, tuple) or isinstance(x, list)+ if is_list:+ batch_dim = x[0].shape[0]+ x = torch.cat(x, dim=0)++ x = self.conv1(x)+ x = self.norm1(x)+ x = self.relu1(x)++ x = self.layer1(x)+ x = self.layer2(x)+ x = self.layer3(x)++ x = self.conv2(x)++ if self.training and self.dropout is not None:+ x = self.dropout(x)++ if is_list:+ x = torch.split(x, [batch_dim, batch_dim], dim=0)++ return x+++class LargeEncoder(nn.Module):+ def __init__(self, output_dim=128, norm_fn="batch", dropout=0.0):+ super(LargeEncoder, self).__init__()+ self.norm_fn = norm_fn++ if self.norm_fn == "group":+ self.norm1 = nn.GroupNorm(num_groups=8, num_channels=64)++ elif self.norm_fn == "batch":+ self.norm1 = nn.BatchNorm2d(64)++ elif self.norm_fn == "instance":+ self.norm1 = nn.InstanceNorm2d(64)++ elif self.norm_fn == "none":+ self.norm1 = nn.Sequential()++ self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)+ self.relu1 = nn.ReLU(inplace=True)++ self.in_planes = 64+ self.layer1 = self._make_layer(64, stride=1)+ self.layer2 = self._make_layer(112, stride=2)+ self.layer3 = self._make_layer(160, stride=2)+ self.layer3_2 = self._make_layer(160, stride=1)++ # output convolution+ self.conv2 = nn.Conv2d(self.in_planes, output_dim, kernel_size=1)++ self.dropout = None+ if dropout > 0:+ self.dropout = nn.Dropout2d(p=dropout)++ for m in self.modules():+ if isinstance(m, nn.Conv2d):+ nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu")+ elif isinstance(m, (nn.BatchNorm2d, nn.InstanceNorm2d, nn.GroupNorm)):+ if m.weight is not None:+ nn.init.constant_(m.weight, 1)+ if m.bias is not None:+ nn.init.constant_(m.bias, 0)++ def _make_layer(self, dim, stride=1):+ layer1 = ResidualBlock(self.in_planes, dim, self.norm_fn, stride=stride)+ layer2 = ResidualBlock(dim, dim, self.norm_fn, stride=1)+ layers = (layer1, layer2)++ self.in_planes = dim+ return nn.Sequential(*layers)++ def forward(self, x):+ # if input is list, combine batch dimension+ is_list = isinstance(x, tuple) or isinstance(x, list)+ if is_list:+ batch_dim = x[0].shape[0]+ x = torch.cat(x, dim=0)++ x = self.conv1(x)+ x = self.norm1(x)+ x = self.relu1(x)++ x = self.layer1(x)+ x = self.layer2(x)+ x = self.layer3(x)+ x = self.layer3_2(x)++ x = self.conv2(x)++ if self.training and self.dropout is not None:+ x = self.dropout(x)++ if is_list:+ x = torch.split(x, [batch_dim, batch_dim], dim=0)++ return xdiff --git a/tools/frame_interpolation/networks/blocks/ifrnet.py b/tools/frame_interpolation/networks/blocks/ifrnet.pynew file mode 100644index 0000000..5719a04--- /dev/null+++ b/tools/frame_interpolation/networks/blocks/ifrnet.py@@ -0,0 +1,115 @@+import torch+import torch.nn as nn+import torch.nn.functional as F++from tools.frame_interpolation.utils.flow_utils import warp+++def resize(x, scale_factor):+ return F.interpolate(x, scale_factor=scale_factor, mode="bilinear", align_corners=False)+++def convrelu(in_channels, out_channels, kernel_size=3, stride=1, padding=1, dilation=1, groups=1, bias=True):+ return nn.Sequential(+ nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias=bias),+ nn.PReLU(out_channels),+ )+++class ResBlock(nn.Module):+ def __init__(self, in_channels, side_channels, bias=True):+ super(ResBlock, self).__init__()+ self.side_channels = side_channels+ self.conv1 = nn.Sequential(+ nn.Conv2d(in_channels, in_channels, kernel_size=3, stride=1, padding=1, bias=bias), nn.PReLU(in_channels)+ )+ self.conv2 = nn.Sequential(+ nn.Conv2d(side_channels, side_channels, kernel_size=3, stride=1, padding=1, bias=bias),+ nn.PReLU(side_channels),+ )+ self.conv3 = nn.Sequential(+ nn.Conv2d(in_channels, in_channels, kernel_size=3, stride=1, padding=1, bias=bias), nn.PReLU(in_channels)+ )+ self.conv4 = nn.Sequential(+ nn.Conv2d(side_channels, side_channels, kernel_size=3, stride=1, padding=1, bias=bias),+ nn.PReLU(side_channels),+ )+ self.conv5 = nn.Conv2d(in_channels, in_channels, kernel_size=3, stride=1, padding=1, bias=bias)+ self.prelu = nn.PReLU(in_channels)++ def forward(self, x):+ out = self.conv1(x)++ res_feat = out[:, : -self.side_channels, ...]+ side_feat = out[:, -self.side_channels :, :, :]+ side_feat = self.conv2(side_feat)+ out = self.conv3(torch.cat([res_feat, side_feat], 1))++ res_feat = out[:, : -self.side_channels, ...]+ side_feat = out[:, -self.side_channels :, :, :]+ side_feat = self.conv4(side_feat)+ out = self.conv5(torch.cat([res_feat, side_feat], 1))++ out = self.prelu(x + out)+ return out+++class Encoder(nn.Module):+ def __init__(self, channels, large=False):+ super(Encoder, self).__init__()+ self.channels = channels+ prev_ch = 3+ for idx, ch in enumerate(channels, 1):+ k = 7 if large and idx == 1 else 3+ p = 3 if k == 7 else 1+ self.register_module(+ f"pyramid{idx}", nn.Sequential(convrelu(prev_ch, ch, k, 2, p), convrelu(ch, ch, 3, 1, 1))+ )+ prev_ch = ch++ def forward(self, in_x):+ fs = []+ for idx in range(len(self.channels)):+ out_x = getattr(self, f"pyramid{idx+1}")(in_x)+ fs.append(out_x)+ in_x = out_x+ return fs+++class InitDecoder(nn.Module):+ def __init__(self, in_ch, out_ch, skip_ch) -> None:+ super().__init__()+ self.convblock = nn.Sequential(+ convrelu(in_ch * 2 + 1, in_ch * 2),+ ResBlock(in_ch * 2, skip_ch),+ nn.ConvTranspose2d(in_ch * 2, out_ch + 4, 4, 2, 1, bias=True),+ )++ def forward(self, f0, f1, embt):+ h, w = f0.shape[2:]+ embt = embt.repeat(1, 1, h, w)+ out = self.convblock(torch.cat([f0, f1, embt], 1))+ flow0, flow1 = torch.chunk(out[:, :4, ...], 2, 1)+ ft_ = out[:, 4:, ...]+ return flow0, flow1, ft_+++class IntermediateDecoder(nn.Module):+ def __init__(self, in_ch, out_ch, skip_ch) -> None:+ super().__init__()+ self.convblock = nn.Sequential(+ convrelu(in_ch * 3 + 4, in_ch * 3),+ ResBlock(in_ch * 3, skip_ch),+ nn.ConvTranspose2d(in_ch * 3, out_ch + 4, 4, 2, 1, bias=True),+ )++ def forward(self, ft_, f0, f1, flow0_in, flow1_in):+ f0_warp = warp(f0, flow0_in)+ f1_warp = warp(f1, flow1_in)+ f_in = torch.cat([ft_, f0_warp, f1_warp, flow0_in, flow1_in], 1)+ out = self.convblock(f_in)+ flow0, flow1 = torch.chunk(out[:, :4, ...], 2, 1)+ ft_ = out[:, 4:, ...]+ flow0 = flow0 + 2.0 * resize(flow0_in, scale_factor=2.0)+ flow1 = flow1 + 2.0 * resize(flow1_in, scale_factor=2.0)+ return flow0, flow1, ft_diff --git a/tools/frame_interpolation/networks/blocks/multi_flow.py b/tools/frame_interpolation/networks/blocks/multi_flow.pynew file mode 100644index 0000000..cbb96a9--- /dev/null+++ b/tools/frame_interpolation/networks/blocks/multi_flow.py@@ -0,0 +1,62 @@+import torch+import torch.nn as nn++from tools.frame_interpolation.utils.flow_utils import warp++from .ifrnet import ResBlock, convrelu, resize+++def multi_flow_combine(comb_block, img0, img1, flow0, flow1, mask=None, img_res=None, mean=None):+ """+ A parallel implementation of multiple flow field warping+ comb_block: An nn.Seqential object.+ img shape: [b, c, h, w]+ flow shape: [b, 2*num_flows, h, w]+ mask (opt):+ If 'mask' is None, the function conduct a simple average.+ img_res (opt):+ If 'img_res' is None, the function adds zero instead.+ mean (opt):+ If 'mean' is None, the function adds zero instead.+ """+ b, c, h, w = flow0.shape+ num_flows = c // 2+ flow0 = flow0.reshape(b, num_flows, 2, h, w).reshape(-1, 2, h, w)+ flow1 = flow1.reshape(b, num_flows, 2, h, w).reshape(-1, 2, h, w)++ mask = mask.reshape(b, num_flows, 1, h, w).reshape(-1, 1, h, w) if mask is not None else None+ img_res = img_res.reshape(b, num_flows, 3, h, w).reshape(-1, 3, h, w) if img_res is not None else 0+ img0 = torch.stack([img0] * num_flows, 1).reshape(-1, 3, h, w)+ img1 = torch.stack([img1] * num_flows, 1).reshape(-1, 3, h, w)+ mean = torch.stack([mean] * num_flows, 1).reshape(-1, 1, 1, 1) if mean is not None else 0++ img0_warp = warp(img0, flow0)+ img1_warp = warp(img1, flow1)+ img_warps = mask * img0_warp + (1 - mask) * img1_warp + mean + img_res+ img_warps = img_warps.reshape(b, num_flows, 3, h, w)+ imgt_pred = img_warps.mean(1) + comb_block(img_warps.view(b, -1, h, w))+ return imgt_pred+++class MultiFlowDecoder(nn.Module):+ def __init__(self, in_ch, skip_ch, num_flows=3):+ super(MultiFlowDecoder, self).__init__()+ self.num_flows = num_flows+ self.convblock = nn.Sequential(+ convrelu(in_ch * 3 + 4, in_ch * 3),+ ResBlock(in_ch * 3, skip_ch),+ nn.ConvTranspose2d(in_ch * 3, 8 * num_flows, 4, 2, 1, bias=True),+ )++ def forward(self, ft_, f0, f1, flow0, flow1):+ n = self.num_flows+ f0_warp = warp(f0, flow0)+ f1_warp = warp(f1, flow1)+ out = self.convblock(torch.cat([ft_, f0_warp, f1_warp, flow0, flow1], 1))+ delta_flow0, delta_flow1, mask, img_res = torch.split(out, [2 * n, 2 * n, n, 3 * n], 1)+ mask = torch.sigmoid(mask)++ flow0 = delta_flow0 + 2.0 * resize(flow0, scale_factor=2.0).repeat(1, self.num_flows, 1, 1)+ flow1 = delta_flow1 + 2.0 * resize(flow1, scale_factor=2.0).repeat(1, self.num_flows, 1, 1)++ return flow0, flow1, mask, img_resdiff --git a/tools/frame_interpolation/networks/blocks/raft.py b/tools/frame_interpolation/networks/blocks/raft.pynew file mode 100644index 0000000..1576889--- /dev/null+++ b/tools/frame_interpolation/networks/blocks/raft.py@@ -0,0 +1,213 @@+import torch+import torch.nn as nn+import torch.nn.functional as F+++def resize(x, scale_factor):+ return F.interpolate(x, scale_factor=scale_factor, mode="bilinear", align_corners=False)+++def bilinear_sampler(img, coords, mask=False):+ """Wrapper for grid_sample, uses pixel coordinates"""+ H, W = img.shape[-2:]+ xgrid, ygrid = coords.split([1, 1], dim=-1)+ xgrid = 2 * xgrid / (W - 1) - 1+ ygrid = 2 * ygrid / (H - 1) - 1++ grid = torch.cat([xgrid, ygrid], dim=-1)+ img = F.grid_sample(img, grid, align_corners=True)++ if mask:+ mask = (xgrid > -1) & (ygrid > -1) & (xgrid < 1) & (ygrid < 1)+ return img, mask.float()++ return img+++def coords_grid(batch, ht, wd, device):+ coords = torch.meshgrid(torch.arange(ht, device=device), torch.arange(wd, device=device), indexing="ij")+ coords = torch.stack(coords[::-1], dim=0).float()+ return coords[None].repeat(batch, 1, 1, 1)+++class SmallUpdateBlock(nn.Module):+ def __init__(self, cdim, hidden_dim, flow_dim, corr_dim, fc_dim, corr_levels=4, radius=3, scale_factor=None):+ super(SmallUpdateBlock, self).__init__()+ cor_planes = corr_levels * (2 * radius + 1) ** 2+ self.scale_factor = scale_factor++ self.convc1 = nn.Conv2d(2 * cor_planes, corr_dim, 1, padding=0)+ self.convf1 = nn.Conv2d(4, flow_dim * 2, 7, padding=3)+ self.convf2 = nn.Conv2d(flow_dim * 2, flow_dim, 3, padding=1)+ self.conv = nn.Conv2d(corr_dim + flow_dim, fc_dim, 3, padding=1)++ self.gru = nn.Sequential(+ nn.Conv2d(fc_dim + 4 + cdim, hidden_dim, 3, padding=1),+ nn.LeakyReLU(negative_slope=0.1, inplace=True),+ nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),+ )++ self.feat_head = nn.Sequential(+ nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),+ nn.LeakyReLU(negative_slope=0.1, inplace=True),+ nn.Conv2d(hidden_dim, cdim, 3, padding=1),+ )++ self.flow_head = nn.Sequential(+ nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),+ nn.LeakyReLU(negative_slope=0.1, inplace=True),+ nn.Conv2d(hidden_dim, 4, 3, padding=1),+ )++ self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)++ def forward(self, net, flow, corr):+ net = resize(net, 1 / self.scale_factor) if self.scale_factor is not None else net+ cor = self.lrelu(self.convc1(corr))+ flo = self.lrelu(self.convf1(flow))+ flo = self.lrelu(self.convf2(flo))+ cor_flo = torch.cat([cor, flo], dim=1)+ inp = self.lrelu(self.conv(cor_flo))+ inp = torch.cat([inp, flow, net], dim=1)++ out = self.gru(inp)+ delta_net = self.feat_head(out)+ delta_flow = self.flow_head(out)++ if self.scale_factor is not None:+ delta_net = resize(delta_net, scale_factor=self.scale_factor)+ delta_flow = self.scale_factor * resize(delta_flow, scale_factor=self.scale_factor)++ return delta_net, delta_flow+++class BasicUpdateBlock(nn.Module):+ def __init__(+ self,+ cdim,+ hidden_dim,+ flow_dim,+ corr_dim,+ corr_dim2,+ fc_dim,+ corr_levels=4,+ radius=3,+ scale_factor=None,+ out_num=1,+ ):+ super(BasicUpdateBlock, self).__init__()+ cor_planes = corr_levels * (2 * radius + 1) ** 2++ self.scale_factor = scale_factor+ self.convc1 = nn.Conv2d(2 * cor_planes, corr_dim, 1, padding=0)+ self.convc2 = nn.Conv2d(corr_dim, corr_dim2, 3, padding=1)+ self.convf1 = nn.Conv2d(4, flow_dim * 2, 7, padding=3)+ self.convf2 = nn.Conv2d(flow_dim * 2, flow_dim, 3, padding=1)+ self.conv = nn.Conv2d(flow_dim + corr_dim2, fc_dim, 3, padding=1)++ self.gru = nn.Sequential(+ nn.Conv2d(fc_dim + 4 + cdim, hidden_dim, 3, padding=1),+ nn.LeakyReLU(negative_slope=0.1, inplace=True),+ nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),+ )++ self.feat_head = nn.Sequential(+ nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),+ nn.LeakyReLU(negative_slope=0.1, inplace=True),+ nn.Conv2d(hidden_dim, cdim, 3, padding=1),+ )++ self.flow_head = nn.Sequential(+ nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),+ nn.LeakyReLU(negative_slope=0.1, inplace=True),+ nn.Conv2d(hidden_dim, 4 * out_num, 3, padding=1),+ )++ self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)++ def forward(self, net, flow, corr):+ net = resize(net, 1 / self.scale_factor) if self.scale_factor is not None else net+ cor = self.lrelu(self.convc1(corr))+ cor = self.lrelu(self.convc2(cor))+ flo = self.lrelu(self.convf1(flow))+ flo = self.lrelu(self.convf2(flo))+ cor_flo = torch.cat([cor, flo], dim=1)+ inp = self.lrelu(self.conv(cor_flo))+ inp = torch.cat([inp, flow, net], dim=1)++ out = self.gru(inp)+ delta_net = self.feat_head(out)+ delta_flow = self.flow_head(out)++ if self.scale_factor is not None:+ delta_net = resize(delta_net, scale_factor=self.scale_factor)+ delta_flow = self.scale_factor * resize(delta_flow, scale_factor=self.scale_factor)+ return delta_net, delta_flow+++class BidirCorrBlock:+ def __init__(self, fmap1, fmap2, num_levels=4, radius=4):+ self.num_levels = num_levels+ self.radius = radius+ self.corr_pyramid = []+ self.corr_pyramid_T = []++ corr = BidirCorrBlock.corr(fmap1, fmap2)+ batch, h1, w1, dim, h2, w2 = corr.shape+ corr_T = corr.clone().permute(0, 4, 5, 3, 1, 2)++ corr = corr.reshape(batch * h1 * w1, dim, h2, w2)+ corr_T = corr_T.reshape(batch * h2 * w2, dim, h1, w1)++ self.corr_pyramid.append(corr)+ self.corr_pyramid_T.append(corr_T)++ for _ in range(self.num_levels - 1):+ corr = F.avg_pool2d(corr, 2, stride=2)+ corr_T = F.avg_pool2d(corr_T, 2, stride=2)+ self.corr_pyramid.append(corr)+ self.corr_pyramid_T.append(corr_T)++ def __call__(self, coords0, coords1):+ r = self.radius+ coords0 = coords0.permute(0, 2, 3, 1)+ coords1 = coords1.permute(0, 2, 3, 1)+ assert coords0.shape == coords1.shape, f"coords0 shape: [{coords0.shape}] is not equal to [{coords1.shape}]"+ batch, h1, w1, _ = coords0.shape++ out_pyramid = []+ out_pyramid_T = []+ for i in range(self.num_levels):+ corr = self.corr_pyramid[i]+ corr_T = self.corr_pyramid_T[i]++ dx = torch.linspace(-r, r, 2 * r + 1, device=coords0.device)+ dy = torch.linspace(-r, r, 2 * r + 1, device=coords0.device)+ delta = torch.stack(torch.meshgrid(dy, dx, indexing="ij"), axis=-1)+ delta_lvl = delta.view(1, 2 * r + 1, 2 * r + 1, 2)++ centroid_lvl_0 = coords0.reshape(batch * h1 * w1, 1, 1, 2) / 2**i+ centroid_lvl_1 = coords1.reshape(batch * h1 * w1, 1, 1, 2) / 2**i+ coords_lvl_0 = centroid_lvl_0 + delta_lvl+ coords_lvl_1 = centroid_lvl_1 + delta_lvl++ corr = bilinear_sampler(corr, coords_lvl_0)+ corr_T = bilinear_sampler(corr_T, coords_lvl_1)+ corr = corr.view(batch, h1, w1, -1)+ corr_T = corr_T.view(batch, h1, w1, -1)+ out_pyramid.append(corr)+ out_pyramid_T.append(corr_T)++ out = torch.cat(out_pyramid, dim=-1)+ out_T = torch.cat(out_pyramid_T, dim=-1)+ return out.permute(0, 3, 1, 2).contiguous().float(), out_T.permute(0, 3, 1, 2).contiguous().float()++ @staticmethod+ def corr(fmap1, fmap2):+ batch, dim, ht, wd = fmap1.shape+ fmap1 = fmap1.view(batch, dim, ht * wd)+ fmap2 = fmap2.view(batch, dim, ht * wd)++ corr = torch.matmul(fmap1.transpose(1, 2), fmap2)+ corr = corr.view(batch, ht, wd, 1, ht, wd)+ return corr / torch.sqrt(torch.tensor(dim).float())diff --git a/tools/frame_interpolation/utils/__init__.py b/tools/frame_interpolation/utils/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/tools/frame_interpolation/utils/dist_utils.py b/tools/frame_interpolation/utils/dist_utils.pynew file mode 100644index 0000000..d754d4f--- /dev/null+++ b/tools/frame_interpolation/utils/dist_utils.py@@ -0,0 +1,48 @@+import os++import torch+++def get_world_size():+ """Find OMPI world size without calling mpi functions+ :rtype: int+ """+ if os.environ.get("PMI_SIZE") is not None:+ return int(os.environ.get("PMI_SIZE") or 1)+ elif os.environ.get("OMPI_COMM_WORLD_SIZE") is not None:+ return int(os.environ.get("OMPI_COMM_WORLD_SIZE") or 1)+ else:+ return torch.cuda.device_count()+++def get_global_rank():+ """Find OMPI world rank without calling mpi functions+ :rtype: int+ """+ if os.environ.get("PMI_RANK") is not None:+ return int(os.environ.get("PMI_RANK") or 0)+ elif os.environ.get("OMPI_COMM_WORLD_RANK") is not None:+ return int(os.environ.get("OMPI_COMM_WORLD_RANK") or 0)+ else:+ return 0+++def get_local_rank():+ """Find OMPI local rank without calling mpi functions+ :rtype: int+ """+ if os.environ.get("MPI_LOCALRANKID") is not None:+ return int(os.environ.get("MPI_LOCALRANKID") or 0)+ elif os.environ.get("OMPI_COMM_WORLD_LOCAL_RANK") is not None:+ return int(os.environ.get("OMPI_COMM_WORLD_LOCAL_RANK") or 0)+ else:+ return 0+++def get_master_ip():+ if os.environ.get("AZ_BATCH_MASTER_NODE") is not None:+ return os.environ.get("AZ_BATCH_MASTER_NODE").split(":")[0]+ elif os.environ.get("AZ_BATCHAI_MPI_MASTER_NODE") is not None:+ return os.environ.get("AZ_BATCHAI_MPI_MASTER_NODE")+ else:+ return "127.0.0.1"diff --git a/tools/frame_interpolation/utils/flow_utils.py b/tools/frame_interpolation/utils/flow_utils.pynew file mode 100644index 0000000..4edee46--- /dev/null+++ b/tools/frame_interpolation/utils/flow_utils.py@@ -0,0 +1,125 @@+import numpy as np+import torch+import torch.nn.functional as F+from PIL import ImageFile++ImageFile.LOAD_TRUNCATED_IMAGES = True+++def warp(img, flow):+ B, _, H, W = flow.shape+ xx = torch.linspace(-1.0, 1.0, W).view(1, 1, 1, W).expand(B, -1, H, -1)+ yy = torch.linspace(-1.0, 1.0, H).view(1, 1, H, 1).expand(B, -1, -1, W)+ grid = torch.cat([xx, yy], 1).to(img)+ flow_ = torch.cat([flow[:, 0:1, :, :] / ((W - 1.0) / 2.0), flow[:, 1:2, :, :] / ((H - 1.0) / 2.0)], 1)+ grid_ = (grid + flow_).permute(0, 2, 3, 1)+ output = F.grid_sample(input=img, grid=grid_, mode="bilinear", padding_mode="border", align_corners=True)+ return output+++def make_colorwheel():+ """+ Generates a color wheel for optical flow visualization as presented in:+ Baker et al. "A Database and Evaluation Methodology for Optical Flow" (ICCV, 2007)+ URL: http://vision.middlebury.edu/flow/flowEval-iccv07.pdf+ Code follows the original C++ source code of Daniel Scharstein.+ Code follows the the Matlab source code of Deqing Sun.+ Returns:+ np.ndarray: Color wheel+ """++ RY = 15+ YG = 6+ GC = 4+ CB = 11+ BM = 13+ MR = 6++ ncols = RY + YG + GC + CB + BM + MR+ colorwheel = np.zeros((ncols, 3))+ col = 0++ # RY+ colorwheel[0:RY, 0] = 255+ colorwheel[0:RY, 1] = np.floor(255 * np.arange(0, RY) / RY)+ col = col + RY+ # YG+ colorwheel[col : col + YG, 0] = 255 - np.floor(255 * np.arange(0, YG) / YG)+ colorwheel[col : col + YG, 1] = 255+ col = col + YG+ # GC+ colorwheel[col : col + GC, 1] = 255+ colorwheel[col : col + GC, 2] = np.floor(255 * np.arange(0, GC) / GC)+ col = col + GC+ # CB+ colorwheel[col : col + CB, 1] = 255 - np.floor(255 * np.arange(CB) / CB)+ colorwheel[col : col + CB, 2] = 255+ col = col + CB+ # BM+ colorwheel[col : col + BM, 2] = 255+ colorwheel[col : col + BM, 0] = np.floor(255 * np.arange(0, BM) / BM)+ col = col + BM+ # MR+ colorwheel[col : col + MR, 2] = 255 - np.floor(255 * np.arange(MR) / MR)+ colorwheel[col : col + MR, 0] = 255+ return colorwheel+++def flow_uv_to_colors(u, v, convert_to_bgr=False):+ """+ Applies the flow color wheel to (possibly clipped) flow components u and v.+ According to the C++ source code of Daniel Scharstein+ According to the Matlab source code of Deqing Sun+ Args:+ u (np.ndarray): Input horizontal flow of shape [H,W]+ v (np.ndarray): Input vertical flow of shape [H,W]+ convert_to_bgr (bool, optional): Convert output image to BGR. Defaults to False.+ Returns:+ np.ndarray: Flow visualization image of shape [H,W,3]+ """+ flow_image = np.zeros((u.shape[0], u.shape[1], 3), np.uint8)+ colorwheel = make_colorwheel() # shape [55x3]+ ncols = colorwheel.shape[0]+ rad = np.sqrt(np.square(u) + np.square(v))+ a = np.arctan2(-v, -u) / np.pi+ fk = (a + 1) / 2 * (ncols - 1)+ k0 = np.floor(fk).astype(np.int32)+ k1 = k0 + 1+ k1[k1 == ncols] = 0+ f = fk - k0+ for i in range(colorwheel.shape[1]):+ tmp = colorwheel[:, i]+ col0 = tmp[k0] / 255.0+ col1 = tmp[k1] / 255.0+ col = (1 - f) * col0 + f * col1+ idx = rad <= 1+ col[idx] = 1 - rad[idx] * (1 - col[idx])+ col[~idx] = col[~idx] * 0.75 # out of range+ # Note the 2-i => BGR instead of RGB+ ch_idx = 2 - i if convert_to_bgr else i+ flow_image[:, :, ch_idx] = np.floor(255 * col)+ return flow_image+++def flow_to_image(flow_uv, clip_flow=None, convert_to_bgr=False):+ """+ Expects a two dimensional flow image of shape.+ Args:+ flow_uv (np.ndarray): Flow UV image of shape [H,W,2]+ clip_flow (float, optional): Clip maximum of flow values. Defaults to None.+ convert_to_bgr (bool, optional): Convert output image to BGR. Defaults to False.+ Returns:+ np.ndarray: Flow visualization image of shape [H,W,3]+ """+ assert flow_uv.ndim == 3, "input flow must have three dimensions"+ assert flow_uv.shape[2] == 2, "input flow must have shape [H,W,2]"+ if clip_flow is not None:+ flow_uv = np.clip(flow_uv, 0, clip_flow)+ u = flow_uv[:, :, 0]+ v = flow_uv[:, :, 1]+ rad = np.sqrt(np.square(u) + np.square(v))+ rad_max = np.max(rad)+ epsilon = 1e-5+ u = u / (rad_max + epsilon)+ v = v / (rad_max + epsilon)+ return flow_uv_to_colors(u, v, convert_to_bgr)diff --git a/tools/frame_interpolation/utils/utils.py b/tools/frame_interpolation/utils/utils.pynew file mode 100644index 0000000..285a65f--- /dev/null+++ b/tools/frame_interpolation/utils/utils.py@@ -0,0 +1,314 @@+import random+import re+import sys++import numpy as np+import torch+import torch.nn.functional as F+from imageio import imread, imwrite+from PIL import ImageFile++ImageFile.LOAD_TRUNCATED_IMAGES = True+++class AverageMeter:+ def __init__(self):+ self.reset()++ def reset(self):+ self.val = 0.0+ self.avg = 0.0+ self.sum = 0.0+ self.count = 0++ def update(self, val, n=1):+ self.val = val+ self.sum += val * n+ self.count += n+ self.avg = self.sum / self.count+++class AverageMeterGroups:+ def __init__(self) -> None:+ self.meter_dict = dict()++ def update(self, dict, n=1):+ for name, val in dict.items():+ if self.meter_dict.get(name) is None:+ self.meter_dict[name] = AverageMeter()+ self.meter_dict[name].update(val, n)++ def reset(self, name=None):+ if name is None:+ for v in self.meter_dict.values():+ v.reset()+ else:+ meter = self.meter_dict.get(name)+ if meter is not None:+ meter.reset()++ def avg(self, name):+ meter = self.meter_dict.get(name)+ if meter is not None:+ return meter.avg+++class InputPadder:+ """Pads images such that dimensions are divisible by divisor"""++ def __init__(self, dims, divisor=16):+ self.ht, self.wd = dims[-2:]+ pad_ht = (((self.ht // divisor) + 1) * divisor - self.ht) % divisor+ pad_wd = (((self.wd // divisor) + 1) * divisor - self.wd) % divisor+ self._pad = [pad_wd // 2, pad_wd - pad_wd // 2, pad_ht // 2, pad_ht - pad_ht // 2]++ def pad(self, *inputs):+ if len(inputs) == 1:+ return F.pad(inputs[0], self._pad, mode="replicate")+ else:+ return [F.pad(x, self._pad, mode="replicate") for x in inputs]++ def unpad(self, *inputs):+ if len(inputs) == 1:+ return self._unpad(inputs[0])+ else:+ return [self._unpad(x) for x in inputs]++ def _unpad(self, x):+ ht, wd = x.shape[-2:]+ c = [self._pad[2], ht - self._pad[3], self._pad[0], wd - self._pad[1]]+ return x[..., c[0] : c[1], c[2] : c[3]]+++def img2tensor(img):+ if img.shape[-1] > 3:+ img = img[:, :, :3]+ return torch.tensor(img).permute(2, 0, 1).unsqueeze(0) / 255.0+++def tensor2img(img_t):+ return (img_t * 255.0).detach().squeeze(0).permute(1, 2, 0).cpu().numpy().clip(0, 255).astype(np.uint8)+++def seed_all(seed):+ random.seed(seed)+ np.random.seed(seed)+ torch.manual_seed(seed)+ torch.cuda.manual_seed_all(seed)+++def read(file):+ if file.endswith(".float3"):+ return readFloat(file)+ elif file.endswith(".flo"):+ return readFlow(file)+ elif file.endswith(".ppm"):+ return readImage(file)+ elif file.endswith(".pgm"):+ return readImage(file)+ elif file.endswith(".png"):+ return readImage(file)+ elif file.endswith(".jpg"):+ return readImage(file)+ elif file.endswith(".pfm"):+ return readPFM(file)[0]+ else:+ raise Exception("don't know how to read %s" % file)+++def write(file, data):+ if file.endswith(".float3"):+ return writeFloat(file, data)+ elif file.endswith(".flo"):+ return writeFlow(file, data)+ elif file.endswith(".ppm"):+ return writeImage(file, data)+ elif file.endswith(".pgm"):+ return writeImage(file, data)+ elif file.endswith(".png"):+ return writeImage(file, data)+ elif file.endswith(".jpg"):+ return writeImage(file, data)+ elif file.endswith(".pfm"):+ return writePFM(file, data)+ else:+ raise Exception("don't know how to write %s" % file)+++def readPFM(file):+ file = open(file, "rb")++ color = None+ width = None+ height = None+ scale = None+ endian = None++ header = file.readline().rstrip()+ if header.decode("ascii") == "PF":+ color = True+ elif header.decode("ascii") == "Pf":+ color = False+ else:+ raise Exception("Not a PFM file.")++ dim_match = re.match(r"^(\d+)\s(\d+)\s$", file.readline().decode("ascii"))+ if dim_match:+ width, height = list(map(int, dim_match.groups()))+ else:+ raise Exception("Malformed PFM header.")++ scale = float(file.readline().decode("ascii").rstrip())+ if scale < 0:+ endian = "<"+ scale = -scale+ else:+ endian = ">"++ data = np.fromfile(file, endian + "f")+ shape = (height, width, 3) if color else (height, width)++ data = np.reshape(data, shape)+ data = np.flipud(data)+ return data, scale+++def writePFM(file, image, scale=1):+ file = open(file, "wb")++ color = None++ if image.dtype.name != "float32":+ raise Exception("Image dtype must be float32.")++ image = np.flipud(image)++ if len(image.shape) == 3 and image.shape[2] == 3:+ color = True+ elif len(image.shape) == 2 or len(image.shape) == 3 and image.shape[2] == 1:+ color = False+ else:+ raise Exception("Image must have H x W x 3, H x W x 1 or H x W dimensions.")++ file.write("PF\n" if color else "Pf\n".encode())+ file.write("%d %d\n".encode() % (image.shape[1], image.shape[0]))++ endian = image.dtype.byteorder++ if endian == "<" or endian == "=" and sys.byteorder == "little":+ scale = -scale++ file.write("%f\n".encode() % scale)++ image.tofile(file)+++def readFlow(name):+ if name.endswith(".pfm") or name.endswith(".PFM"):+ return readPFM(name)[0][:, :, 0:2]++ f = open(name, "rb")++ header = f.read(4)+ if header.decode("utf-8") != "PIEH":+ raise Exception("Flow file header does not contain PIEH")++ width = np.fromfile(f, np.int32, 1).squeeze()+ height = np.fromfile(f, np.int32, 1).squeeze()++ flow = np.fromfile(f, np.float32, width * height * 2).reshape((height, width, 2))++ return flow.astype(np.float32)+++def readImage(name):+ if name.endswith(".pfm") or name.endswith(".PFM"):+ data = readPFM(name)[0]+ if len(data.shape) == 3:+ return data[:, :, 0:3]+ else:+ return data+ return imread(name)+++def writeImage(name, data):+ if name.endswith(".pfm") or name.endswith(".PFM"):+ return writePFM(name, data, 1)+ return imwrite(name, data)+++def writeFlow(name, flow):+ f = open(name, "wb")+ f.write("PIEH".encode("utf-8"))+ np.array([flow.shape[1], flow.shape[0]], dtype=np.int32).tofile(f)+ flow = flow.astype(np.float32)+ flow.tofile(f)+++def readFloat(name):+ f = open(name, "rb")++ if (f.readline().decode("utf-8")) != "float\n":+ raise Exception("float file %s did not contain <float> keyword" % name)++ dim = int(f.readline())++ dims = []+ count = 1+ for i in range(0, dim):+ d = int(f.readline())+ dims.append(d)+ count *= d++ dims = list(reversed(dims))++ data = np.fromfile(f, np.float32, count).reshape(dims)+ if dim > 2:+ data = np.transpose(data, (2, 1, 0))+ data = np.transpose(data, (1, 0, 2))++ return data+++def writeFloat(name, data):+ f = open(name, "wb")++ dim = len(data.shape)+ if dim > 3:+ raise Exception("bad float file dimension: %d" % dim)++ f.write(("float\n").encode("ascii"))+ f.write(("%d\n" % dim).encode("ascii"))++ if dim == 1:+ f.write(("%d\n" % data.shape[0]).encode("ascii"))+ else:+ f.write(("%d\n" % data.shape[1]).encode("ascii"))+ f.write(("%d\n" % data.shape[0]).encode("ascii"))+ for i in range(2, dim):+ f.write(("%d\n" % data.shape[i]).encode("ascii"))++ data = data.astype(np.float32)+ if dim == 2:+ data.tofile(f)++ else:+ np.transpose(data, (2, 0, 1)).tofile(f)+++def check_dim_and_resize(tensor_list):+ shape_list = []+ for t in tensor_list:+ shape_list.append(t.shape[2:])++ if len(set(shape_list)) > 1:+ desired_shape = shape_list[0]+ print(f"Inconsistent size of input video frames. All frames will be resized to {desired_shape}")++ resize_tensor_list = []+ for t in tensor_list:+ resize_tensor_list.append(torch.nn.functional.interpolate(t, size=tuple(desired_shape), mode="bilinear"))++ tensor_list = resize_tensor_list++ return tensor_listdiff --git a/tools/scene_cut/main_cut_multi_thread.py b/tools/scene_cut/main_cut_multi_thread.pyindex 45fd2aa..cab91f5 100644--- a/tools/scene_cut/main_cut_multi_thread.py+++ b/tools/scene_cut/main_cut_multi_thread.py@@ -1,168 +1,162 @@-import os-import argparse-import time-import subprocess-from tqdm import tqdm--import pandas as pd-from scenedetect import FrameTimecode-from imageio_ffmpeg import get_ffmpeg_exe-from concurrent.futures import ThreadPoolExecutor, as_completed--from mmengine.logging import MMLogger, print_log-from utils_video import is_intact_video, iterate_files, clone_folder_structure---def single_process(row, save_dir, logger=None):- # video_id = row['videoID']- # video_path = os.path.join(root_src, f'{video_id}.mp4')- video_path = row['path']-- # check mp4 integrity- # if not is_intact_video(video_path, logger=logger):- # return False-- timestamp = row['timestamp']- if not (timestamp.startswith('[') and timestamp.endswith(']')):- return False- scene_list = eval(timestamp)- scene_list = [- (FrameTimecode(s, fps=1), FrameTimecode(t, fps=1))- for s, t in scene_list- ]- split_video(video_path, scene_list, save_dir=save_dir, logger=logger)- return True---def split_video(- video_path,- scene_list,- save_dir,- min_seconds=None,- max_seconds=None,- target_fps=30,- shorter_size=512,- verbose=False,- logger=None,-):- """- scenes shorter than min_seconds will be ignored;- scenes longer than max_seconds will be cut to save the beginning max_seconds.- Currently, the saved file name pattern is f'{fname}_scene-{idx}'.mp4-- Args:- scene_list (List[Tuple[FrameTimecode, FrameTimecode]]): each element is (s, t): start and end of a scene.- min_seconds (float | None)- max_seconds (float | None)- target_fps (int | None)- shorter_size (int | None)- """- FFMPEG_PATH = get_ffmpeg_exe()-- save_path_list = []- for idx, scene in enumerate(scene_list):- s, t = scene # FrameTimecode- if min_seconds is not None:- if (t - s).get_seconds() < min_seconds:- continue-- duration = t - s- if max_seconds is not None:- fps = s.framerate- max_duration = FrameTimecode(timecode="00:00:00", fps=fps)- max_duration.frame_num = round(fps * max_seconds)- duration = min(max_duration, duration)-- # save path- fname = os.path.basename(video_path)- fname_wo_ext = os.path.splitext(fname)[0]- # TODO: fname pattern- save_path = os.path.join(save_dir, f"{fname_wo_ext}_scene-{idx}.mp4")-- # ffmpeg cmd- cmd = [FFMPEG_PATH]-- # Only show ffmpeg output for the first call, which will display any- # errors if it fails, and then break the loop. We only show error messages- # for the remaining calls.- # cmd += ['-v', 'error']-- # input path- # cmd += ["-i", video_path]-- # clip to cut- cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-i", video_path, "-t", str(duration.get_seconds())]- # cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())]-- # target fps- # cmd += ['-vf', 'select=mod(n\,2)']- if target_fps is not None:- cmd += ["-r", f"{target_fps}"]-- # aspect ratio- if shorter_size is not None:- cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]- # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]-- cmd += ["-map", "0", save_path]-- proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)- stdout, stderr = proc.communicate()- if verbose:- stdout = stdout.decode("utf-8")- print_log(stdout, logger=logger)-- save_path_list.append(video_path)- print_log(f"Video clip saved to '{save_path}'", logger=logger)-- return save_path_list---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument('--root', default='F:/Panda-70M/')- parser.add_argument('--split', default='test')- parser.add_argument('--num_workers', default=5, type=int)-- args = parser.parse_args()- return args---def main():- # args = parse_args()- # root = args.root- # split = args.split-- root = 'F:/Panda-70M/'- root, split = 'F:/pexels_new/', 'popular_2'- meta_path = os.path.join(root, f'raw/meta/{split}_format_timestamp.csv')- root_dst = os.path.join(root, f'scene_cut/data/{split}')-- folder_dst = root_dst- # folder_src = os.path.join(root_src, f'data/{split}')- # folder_dst = os.path.join(root_dst, os.path.relpath(folder_src, root_src))- os.makedirs(folder_dst, exist_ok=True)-- meta = pd.read_csv(meta_path)-- # create logger- # folder_path_log = os.path.dirname(root_dst)- # log_name = os.path.basename(root_dst)- # timestamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(time.time()))- # log_path = os.path.join(folder_path_log, f"{log_name}_{timestamp}.log")- # logger = MMLogger.get_instance(log_name, log_file=log_path)- logger = None-- tasks = []- pool = ThreadPoolExecutor(max_workers=1)- for idx, row in meta.iterrows():- task = pool.submit(single_process, row, folder_dst, logger)- tasks.append(task)-- for task in tqdm(as_completed(tasks), total=len(meta)):- task.result()- pool.shutdown()---if __name__ == '__main__':- main()+import argparse+import os+import subprocess+from concurrent.futures import ThreadPoolExecutor, as_completed++import pandas as pd+from imageio_ffmpeg import get_ffmpeg_exe+from mmengine.logging import print_log+from scenedetect import FrameTimecode+from tqdm import tqdm+++def single_process(row, save_dir, logger=None):+ # video_id = row['videoID']+ # video_path = os.path.join(root_src, f'{video_id}.mp4')+ video_path = row["path"]++ # check mp4 integrity+ # if not is_intact_video(video_path, logger=logger):+ # return False++ timestamp = row["timestamp"]+ if not (timestamp.startswith("[") and timestamp.endswith("]")):+ return False+ scene_list = eval(timestamp)+ scene_list = [(FrameTimecode(s, fps=1), FrameTimecode(t, fps=1)) for s, t in scene_list]+ split_video(video_path, scene_list, save_dir=save_dir, logger=logger)+ return True+++def split_video(+ video_path,+ scene_list,+ save_dir,+ min_seconds=None,+ max_seconds=None,+ target_fps=30,+ shorter_size=512,+ verbose=False,+ logger=None,+):+ """+ scenes shorter than min_seconds will be ignored;+ scenes longer than max_seconds will be cut to save the beginning max_seconds.+ Currently, the saved file name pattern is f'{fname}_scene-{idx}'.mp4++ Args:+ scene_list (List[Tuple[FrameTimecode, FrameTimecode]]): each element is (s, t): start and end of a scene.+ min_seconds (float | None)+ max_seconds (float | None)+ target_fps (int | None)+ shorter_size (int | None)+ """+ FFMPEG_PATH = get_ffmpeg_exe()++ save_path_list = []+ for idx, scene in enumerate(scene_list):+ s, t = scene # FrameTimecode+ if min_seconds is not None:+ if (t - s).get_seconds() < min_seconds:+ continue++ duration = t - s+ if max_seconds is not None:+ fps = s.framerate+ max_duration = FrameTimecode(timecode="00:00:00", fps=fps)+ max_duration.frame_num = round(fps * max_seconds)+ duration = min(max_duration, duration)++ # save path+ fname = os.path.basename(video_path)+ fname_wo_ext = os.path.splitext(fname)[0]+ # TODO: fname pattern+ save_path = os.path.join(save_dir, f"{fname_wo_ext}_scene-{idx}.mp4")++ # ffmpeg cmd+ cmd = [FFMPEG_PATH]++ # Only show ffmpeg output for the first call, which will display any+ # errors if it fails, and then break the loop. We only show error messages+ # for the remaining calls.+ # cmd += ['-v', 'error']++ # input path+ # cmd += ["-i", video_path]++ # clip to cut+ cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-i", video_path, "-t", str(duration.get_seconds())]+ # cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())]++ # target fps+ # cmd += ['-vf', 'select=mod(n\,2)']+ if target_fps is not None:+ cmd += ["-r", f"{target_fps}"]++ # aspect ratio+ if shorter_size is not None:+ cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]+ # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]++ cmd += ["-map", "0", save_path]++ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)+ stdout, stderr = proc.communicate()+ if verbose:+ stdout = stdout.decode("utf-8")+ print_log(stdout, logger=logger)++ save_path_list.append(video_path)+ print_log(f"Video clip saved to '{save_path}'", logger=logger)++ return save_path_list+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("--root", default="F:/Panda-70M/")+ parser.add_argument("--split", default="test")+ parser.add_argument("--num_workers", default=5, type=int)++ args = parser.parse_args()+ return args+++def main():+ # args = parse_args()+ # root = args.root+ # split = args.split++ root = "F:/Panda-70M/"+ root, split = "F:/pexels_new/", "popular_2"+ meta_path = os.path.join(root, f"raw/meta/{split}_format_timestamp.csv")+ root_dst = os.path.join(root, f"scene_cut/data/{split}")++ folder_dst = root_dst+ # folder_src = os.path.join(root_src, f'data/{split}')+ # folder_dst = os.path.join(root_dst, os.path.relpath(folder_src, root_src))+ os.makedirs(folder_dst, exist_ok=True)++ meta = pd.read_csv(meta_path)++ # create logger+ # folder_path_log = os.path.dirname(root_dst)+ # log_name = os.path.basename(root_dst)+ # timestamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(time.time()))+ # log_path = os.path.join(folder_path_log, f"{log_name}_{timestamp}.log")+ # logger = MMLogger.get_instance(log_name, log_file=log_path)+ logger = None++ tasks = []+ pool = ThreadPoolExecutor(max_workers=1)+ for idx, row in meta.iterrows():+ task = pool.submit(single_process, row, folder_dst, logger)+ tasks.append(task)++ for task in tqdm(as_completed(tasks), total=len(meta)):+ task.result()+ pool.shutdown()+++if __name__ == "__main__":+ main()diff --git a/tools/scene_cut/main_cut_pandarallel.py b/tools/scene_cut/main_cut_pandarallel.pyindex e011254..d300eda 100644--- a/tools/scene_cut/main_cut_pandarallel.py+++ b/tools/scene_cut/main_cut_pandarallel.py@@ -1,163 +1,157 @@-import os-import argparse-import time-import subprocess-from tqdm import tqdm--import pandas as pd-from scenedetect import FrameTimecode-from functools import partial-from pandarallel import pandarallel-from imageio_ffmpeg import get_ffmpeg_exe--from mmengine.logging import MMLogger, print_log-from utils_video import is_intact_video, iterate_files, clone_folder_structure---def process_single_row(row, save_dir, log_name=None):- video_path = row['path']-- logger = None- if log_name is not None:- logger = MMLogger.get_instance(log_name)-- # check mp4 integrity- # if not is_intact_video(video_path, logger=logger):- # return False-- timestamp = row['timestamp']- if not (timestamp.startswith('[') and timestamp.endswith(']')):- return False- scene_list = eval(timestamp)- scene_list = [- (FrameTimecode(s, fps=1), FrameTimecode(t, fps=1))- for s, t in scene_list- ]- split_video(video_path, scene_list, save_dir=save_dir,- min_seconds=2, max_seconds=15, shorter_size=720,- logger=logger)---def split_video(- video_path,- scene_list,- save_dir,- min_seconds=None,- max_seconds=None,- target_fps=30,- shorter_size=512,- verbose=False,- logger=None,-):- """- scenes shorter than min_seconds will be ignored;- scenes longer than max_seconds will be cut to save the beginning max_seconds.- Currently, the saved file name pattern is f'{fname}_scene-{idx}'.mp4-- Args:- scene_list (List[Tuple[FrameTimecode, FrameTimecode]]): each element is (s, t): start and end of a scene.- min_seconds (float | None)- max_seconds (float | None)- target_fps (int | None)- shorter_size (int | None)- """- FFMPEG_PATH = get_ffmpeg_exe()-- save_path_list = []- for idx, scene in enumerate(scene_list):- s, t = scene # FrameTimecode- if min_seconds is not None:- if (t - s).get_seconds() < min_seconds:- continue-- duration = t - s- if max_seconds is not None:- fps = s.framerate- max_duration = FrameTimecode(timecode="00:00:00", fps=fps)- max_duration.frame_num = round(fps * max_seconds)- duration = min(max_duration, duration)-- # save path- fname = os.path.basename(video_path)- fname_wo_ext = os.path.splitext(fname)[0]- # TODO: fname pattern- save_path = os.path.join(save_dir, f"{fname_wo_ext}_scene-{idx}.mp4")-- # ffmpeg cmd- cmd = [FFMPEG_PATH]-- # Only show ffmpeg output for the first call, which will display any- # errors if it fails, and then break the loop. We only show error messages- # for the remaining calls.- # cmd += ['-v', 'error']-- # -ss after -i is very slow; put -ss before -i- # input path- # cmd += ["-i", video_path]-- # clip to cut- # cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())]-- # clip to cut- cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-i", video_path, "-t", str(duration.get_seconds())]-- # target fps- # cmd += ['-vf', 'select=mod(n\,2)']- if target_fps is not None:- cmd += ["-r", f"{target_fps}"]-- # aspect ratio- if shorter_size is not None:- cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]- # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]-- cmd += ["-map", "0", save_path]-- proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)- stdout, stderr = proc.communicate()- # stdout = stdout.decode("utf-8")- # print_log(stdout, logger=logger)-- save_path_list.append(video_path)- if verbose:- print_log(f"Video clip saved to '{save_path}'", logger=logger)-- return save_path_list---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument('--meta_path', default='./data/pexels_new/raw/meta/popular_5_format_timestamp.csv')- parser.add_argument('--out_dir', default='./data/pexels_new/scene_cut/data/popular_5')- parser.add_argument('--num_workers', default=5, type=int)-- args = parser.parse_args()- return args---def main():- args = parse_args()- meta_path = args.meta_path- out_dir = args.out_dir-- assert os.path.basename(os.path.dirname(out_dir)) == 'data'-- os.makedirs(out_dir, exist_ok=True)-- meta = pd.read_csv(meta_path)-- # create logger- log_dir = os.path.dirname(out_dir)- log_name = os.path.basename(out_dir)- timestamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(time.time()))- log_path = os.path.join(log_dir, f"{log_name}_{timestamp}.log")- logger = MMLogger.get_instance(log_name, log_file=log_path)- # logger = None-- pandarallel.initialize(progress_bar=True)- process_single_row_partial = partial(process_single_row, save_dir=out_dir, log_name=log_name)- meta.parallel_apply(process_single_row_partial, axis=1)---if __name__ == '__main__':- main()+import argparse+import os+import subprocess+import time+from functools import partial++import pandas as pd+from imageio_ffmpeg import get_ffmpeg_exe+from mmengine.logging import MMLogger, print_log+from pandarallel import pandarallel+from scenedetect import FrameTimecode+++def process_single_row(row, save_dir, log_name=None):+ video_path = row["path"]++ logger = None+ if log_name is not None:+ logger = MMLogger.get_instance(log_name)++ # check mp4 integrity+ # if not is_intact_video(video_path, logger=logger):+ # return False++ timestamp = row["timestamp"]+ if not (timestamp.startswith("[") and timestamp.endswith("]")):+ return False+ scene_list = eval(timestamp)+ scene_list = [(FrameTimecode(s, fps=1), FrameTimecode(t, fps=1)) for s, t in scene_list]+ split_video(+ video_path, scene_list, save_dir=save_dir, min_seconds=2, max_seconds=15, shorter_size=720, logger=logger+ )+++def split_video(+ video_path,+ scene_list,+ save_dir,+ min_seconds=None,+ max_seconds=None,+ target_fps=30,+ shorter_size=512,+ verbose=False,+ logger=None,+):+ """+ scenes shorter than min_seconds will be ignored;+ scenes longer than max_seconds will be cut to save the beginning max_seconds.+ Currently, the saved file name pattern is f'{fname}_scene-{idx}'.mp4++ Args:+ scene_list (List[Tuple[FrameTimecode, FrameTimecode]]): each element is (s, t): start and end of a scene.+ min_seconds (float | None)+ max_seconds (float | None)+ target_fps (int | None)+ shorter_size (int | None)+ """+ FFMPEG_PATH = get_ffmpeg_exe()++ save_path_list = []+ for idx, scene in enumerate(scene_list):+ s, t = scene # FrameTimecode+ if min_seconds is not None:+ if (t - s).get_seconds() < min_seconds:+ continue++ duration = t - s+ if max_seconds is not None:+ fps = s.framerate+ max_duration = FrameTimecode(timecode="00:00:00", fps=fps)+ max_duration.frame_num = round(fps * max_seconds)+ duration = min(max_duration, duration)++ # save path+ fname = os.path.basename(video_path)+ fname_wo_ext = os.path.splitext(fname)[0]+ # TODO: fname pattern+ save_path = os.path.join(save_dir, f"{fname_wo_ext}_scene-{idx}.mp4")++ # ffmpeg cmd+ cmd = [FFMPEG_PATH]++ # Only show ffmpeg output for the first call, which will display any+ # errors if it fails, and then break the loop. We only show error messages+ # for the remaining calls.+ # cmd += ['-v', 'error']++ # -ss after -i is very slow; put -ss before -i+ # input path+ # cmd += ["-i", video_path]++ # clip to cut+ # cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-t", str(duration.get_seconds())]++ # clip to cut+ cmd += ["-nostdin", "-y", "-ss", str(s.get_seconds()), "-i", video_path, "-t", str(duration.get_seconds())]++ # target fps+ # cmd += ['-vf', 'select=mod(n\,2)']+ if target_fps is not None:+ cmd += ["-r", f"{target_fps}"]++ # aspect ratio+ if shorter_size is not None:+ cmd += ["-vf", f"scale='if(gt(iw,ih),-2,{shorter_size})':'if(gt(iw,ih),{shorter_size},-2)'"]+ # cmd += ['-vf', f"scale='if(gt(iw,ih),{shorter_size},trunc(ow/a/2)*2)':-2"]++ cmd += ["-map", "0", save_path]++ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)+ stdout, stderr = proc.communicate()+ # stdout = stdout.decode("utf-8")+ # print_log(stdout, logger=logger)++ save_path_list.append(video_path)+ if verbose:+ print_log(f"Video clip saved to '{save_path}'", logger=logger)++ return save_path_list+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("--meta_path", default="./data/pexels_new/raw/meta/popular_5_format_timestamp.csv")+ parser.add_argument("--out_dir", default="./data/pexels_new/scene_cut/data/popular_5")+ parser.add_argument("--num_workers", default=5, type=int)++ args = parser.parse_args()+ return args+++def main():+ args = parse_args()+ meta_path = args.meta_path+ out_dir = args.out_dir++ assert os.path.basename(os.path.dirname(out_dir)) == "data"++ os.makedirs(out_dir, exist_ok=True)++ meta = pd.read_csv(meta_path)++ # create logger+ log_dir = os.path.dirname(out_dir)+ log_name = os.path.basename(out_dir)+ timestamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(time.time()))+ log_path = os.path.join(log_dir, f"{log_name}_{timestamp}.log")+ logger = MMLogger.get_instance(log_name, log_file=log_path)+ # logger = None++ pandarallel.initialize(progress_bar=True)+ process_single_row_partial = partial(process_single_row, save_dir=out_dir, log_name=log_name)+ meta.parallel_apply(process_single_row_partial, axis=1)+++if __name__ == "__main__":+ main()diff --git a/tools/scene_cut/process_meta.py b/tools/scene_cut/process_meta.pyindex 0feffe9..7a664b0 100644--- a/tools/scene_cut/process_meta.py+++ b/tools/scene_cut/process_meta.py@@ -1,266 +1,263 @@-"""-1. format_raw_meta()- - only keep intact videos- - add 'path' column (abs path)-2. create_meta_for_folder()-"""--import os-# os.chdir('../..')-print(f'Current working directory: {os.getcwd()}')--import argparse-import json-import subprocess-import pandas as pd-from tqdm import tqdm-import pickle as pkl-from pandarallel import pandarallel-from functools import partial-import numpy as np--from utils_video import is_intact_video---def has_downloaded_success(json_path):- if not os.path.exists(json_path):- return False-- try:- with open(json_path, 'r') as f:- data = json.load(f)- if 'success' not in data or isinstance(data['success'], bool) is False or data['success'] is False:- return False- except Exception as e:- return False-- return True---def split_meta_csv(chunk_size=60000):- """- Split csv into multiple small csv in order- """- root = './data/Panda-70M'- # meta_name = 'meta/panda70m_training_full.csv'- meta_name = 'meta/panda70m_training_10m.csv'- # meta_name = 'meta/training_10m/train_0.csv'- meta_path = os.path.join(root, meta_name)-- df = pd.read_csv(meta_path)- num_rows = len(df)-- # Split the DataFrame into smaller DataFrames- for idx, i in enumerate(range(0, num_rows, chunk_size)):- df_i = df.iloc[i:i + chunk_size]- out_path = os.path.join(root, f'meta/train_{idx}.csv')- df_i.to_csv(out_path, index=False)-- # If there are remaining rows- if num_rows > chunk_size and num_rows % chunk_size != 0:- df_last = df.iloc[-(num_rows % chunk_size):]- out_path = os.path.join(root, f'meta/train_{idx + 1}.csv')- df_last.to_csv(out_path, index=False)---def remove_index():- df = pd.read_csv('your_file.csv', index_col=0)- df.to_csv('your_file_without_index.csv', index=False)---def append_format(meta_path, mode='.mp4'):- """- Append _format to csv file:- - filter broken videos; only intact videos are kept- - add column 'path'-- input csv should satisfy:- - name should be: {split}.csv- - contain column 'videoID'/'videoId'- """- # meta_path = os.path.join(root, f'raw/meta/{split}.csv')- meta_dirname = os.path.dirname(meta_path)- assert meta_dirname.endswith('raw/meta')- root_raw = os.path.dirname(meta_dirname)-- meta_fname = os.path.basename(meta_path)- split, ext = os.path.splitext(meta_fname)-- meta = pd.read_csv(meta_path)-- path_list = []- new_meta = []- for idx, row in tqdm(meta.iterrows(), total=len(meta)):- # video_id = row['videoID'] # panda- video_id = row['videoId'] # pexels_new- video_path = os.path.join(root_raw, f'data/{split}/{video_id}.mp4')- if mode == '.mp4':- if not is_intact_video(video_path):- continue- elif mode == '.json':- json_path = os.path.join(root_raw, f'data/{split}/{video_id}.json')- if not has_downloaded_success(json_path):- continue- else:- raise ValueError-- new_meta.append(row)- path_list.append(video_path)-- new_meta = pd.DataFrame(new_meta)- new_meta['path'] = path_list-- out_path = os.path.join(root_raw, f'meta/{split}_format.csv')- new_meta.to_csv(out_path, index=False)- print(f'New meta (shape={new_meta.shape}) saved to \'{out_path}\'')---def append_format_pandarallel(meta_path, split, mode='.mp4'):- """- Append _format to csv file:- - filter broken videos; only intact videos are kept- - add column 'path'-- input csv should satisfy:- - name should be: {split}.csv- - contain column 'videoID'/'videoId'- """- # meta_path = os.path.join(root, f'raw/meta/{split}.csv')- meta_dirname = os.path.dirname(meta_path)- assert meta_dirname.endswith('raw/meta')- root_raw = os.path.dirname(meta_dirname)-- meta_fname = os.path.basename(meta_path)- wo_ext, ext = os.path.splitext(meta_fname)-- meta = pd.read_csv(meta_path)-- def is_intact(row, mode='.json'):- video_id = row['videoId'] # pexels_new- video_path = os.path.join(root_raw, f'data/{split}/{video_id}.mp4')- row['path'] = video_path- if mode == '.mp4':- if is_intact_video(video_path):- return True, video_path- return False, video_path- elif mode == '.json':- json_path = os.path.join(root_raw, f'data/{split}/{video_id}.json')- if has_downloaded_success(json_path):- return True, video_path- return False, video_path- else:- raise ValueError-- pandarallel.initialize(progress_bar=True)- is_intact_partial = partial(is_intact, mode=mode)- ret = meta.parallel_apply(is_intact_partial, axis=1)-- intact, paths = list(zip(*ret))-- meta['intact'] = intact- meta['path'] = paths- out_path = os.path.join(root_raw, f'meta/{wo_ext}_intact.csv')- meta.to_csv(out_path, index=False)- print(f'New meta (shape={meta.shape}) with intact info saved to \'{out_path}\'')-- # meta_format = meta[meta['intact']]- meta_format = meta[np.array(intact)]- meta_format.drop('intact', axis=1, inplace=True)- out_path = os.path.join(root_raw, f'meta/{wo_ext}_format.csv')- meta_format.to_csv(out_path, index=False)- print(f'New meta (shape={meta_format.shape}) with format info saved to \'{out_path}\'')---def create_subset(meta_path):- meta = pd.read_csv(meta_path)- meta_subset = meta.iloc[:100]-- wo_ext, ext = os.path.splitext(meta_path)- out_path = f'{wo_ext}_head-100{ext}'- meta_subset.to_csv(out_path, index=False)- print(f'New meta (shape={meta_subset.shape}) saved to \'{out_path}\'')---def append_cut(root='./data/Panda-70M'):- """- Append _cut to csv file- input csv should satisfy:- - name_should be {split}_intact.csv- - contain column 'timestamp': list of timestamp- """- split = 'test'- meta_path = os.path.join(root, f'processed/meta/{split}_intact.csv')-- wo_ext, ext = os.path.splitext(meta_path)- suffix = 'cut'- out_path = f'{wo_ext}_{suffix}{ext}'-- meta = pd.read_csv(meta_path)-- new_meta = []- for idx, row in tqdm(meta.iterrows(), total=len(meta)):- video_id = row['videoID']- timestamps = eval(row['timestamp'])- captions = eval(row['caption'])- scores = eval(row['matching_score'])-- num_clips = len(timestamps)- for idx_c in range(num_clips):- path_i = os.path.join(root, f'processed/{split}/{video_id}_scene-{idx_c}.mp4')- # if not is_intact_video(path_i):- # continue-- row_i = [f'{video_id}_scene-{idx_c}', path_i, timestamps[idx_c], captions[idx_c], scores[idx_c]]-- new_meta.append(row_i)-- columns = ['videoID', 'path', 'timestamp', 'text', 'match_official']- new_meta = pd.DataFrame(new_meta, columns=columns)-- new_meta.to_csv(out_path, index=False)- print(f'New meta (shape={new_meta.shape}) saved to \'{out_path}\'')---def debug_meta_topk():- meta_path = 'F:/Panda-70M/meta/test_intact_cut_flow.csv'- meta = pd.read_csv(meta_path)-- score_column = 'flow_score'- topk = meta.nlargest(10, columns=score_column)- topk_s = meta.nsmallest(200, columns=score_column)-- topk_list = [(row['path'], row['caption'], row[score_column]) for idx, row in topk.iterrows()]- topk_s_list = [(row['path'], row['caption'], row[score_column]) for idx, row in topk_s.iterrows()]-- x = 0---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument('--task', default='append_format')- parser.add_argument('--meta_path', default='./data/pexels_new/raw/meta/popular_1.csv')- parser.add_argument('--split', default='popular_5')- parser.add_argument('--num_workers', default=5, type=int)-- args = parser.parse_args()- return args---if __name__ == '__main__':- # split_meta_csv()-- args = parse_args()- meta_path = args.meta_path- task = args.task-- if task == 'append_format':- # append_format(meta_path=meta_path, mode='.mp4')- append_format_pandarallel(meta_path=meta_path, split=args.split, mode='.json')- elif task == 'create_subset':- create_subset(meta_path=meta_path)- else:- raise ValueError-- # append_cut(root=root)- # append_score(root=root)- # debug_meta_topk()+"""+1. format_raw_meta()+ - only keep intact videos+ - add 'path' column (abs path)+2. create_meta_for_folder()+"""++import os++# os.chdir('../..')+print(f"Current working directory: {os.getcwd()}")++import argparse+import json+from functools import partial++import numpy as np+import pandas as pd+from pandarallel import pandarallel+from tqdm import tqdm+from utils_video import is_intact_video+++def has_downloaded_success(json_path):+ if not os.path.exists(json_path):+ return False++ try:+ with open(json_path, "r") as f:+ data = json.load(f)+ if "success" not in data or isinstance(data["success"], bool) is False or data["success"] is False:+ return False+ except Exception:+ return False++ return True+++def split_meta_csv(chunk_size=60000):+ """+ Split csv into multiple small csv in order+ """+ root = "./data/Panda-70M"+ # meta_name = 'meta/panda70m_training_full.csv'+ meta_name = "meta/panda70m_training_10m.csv"+ # meta_name = 'meta/training_10m/train_0.csv'+ meta_path = os.path.join(root, meta_name)++ df = pd.read_csv(meta_path)+ num_rows = len(df)++ # Split the DataFrame into smaller DataFrames+ for idx, i in enumerate(range(0, num_rows, chunk_size)):+ df_i = df.iloc[i : i + chunk_size]+ out_path = os.path.join(root, f"meta/train_{idx}.csv")+ df_i.to_csv(out_path, index=False)++ # If there are remaining rows+ if num_rows > chunk_size and num_rows % chunk_size != 0:+ df_last = df.iloc[-(num_rows % chunk_size) :]+ out_path = os.path.join(root, f"meta/train_{idx + 1}.csv")+ df_last.to_csv(out_path, index=False)+++def remove_index():+ df = pd.read_csv("your_file.csv", index_col=0)+ df.to_csv("your_file_without_index.csv", index=False)+++def append_format(meta_path, mode=".mp4"):+ """+ Append _format to csv file:+ - filter broken videos; only intact videos are kept+ - add column 'path'++ input csv should satisfy:+ - name should be: {split}.csv+ - contain column 'videoID'/'videoId'+ """+ # meta_path = os.path.join(root, f'raw/meta/{split}.csv')+ meta_dirname = os.path.dirname(meta_path)+ assert meta_dirname.endswith("raw/meta")+ root_raw = os.path.dirname(meta_dirname)++ meta_fname = os.path.basename(meta_path)+ split, ext = os.path.splitext(meta_fname)++ meta = pd.read_csv(meta_path)++ path_list = []+ new_meta = []+ for idx, row in tqdm(meta.iterrows(), total=len(meta)):+ # video_id = row['videoID'] # panda+ video_id = row["videoId"] # pexels_new+ video_path = os.path.join(root_raw, f"data/{split}/{video_id}.mp4")+ if mode == ".mp4":+ if not is_intact_video(video_path):+ continue+ elif mode == ".json":+ json_path = os.path.join(root_raw, f"data/{split}/{video_id}.json")+ if not has_downloaded_success(json_path):+ continue+ else:+ raise ValueError++ new_meta.append(row)+ path_list.append(video_path)++ new_meta = pd.DataFrame(new_meta)+ new_meta["path"] = path_list++ out_path = os.path.join(root_raw, f"meta/{split}_format.csv")+ new_meta.to_csv(out_path, index=False)+ print(f"New meta (shape={new_meta.shape}) saved to '{out_path}'")+++def append_format_pandarallel(meta_path, split, mode=".mp4"):+ """+ Append _format to csv file:+ - filter broken videos; only intact videos are kept+ - add column 'path'++ input csv should satisfy:+ - name should be: {split}.csv+ - contain column 'videoID'/'videoId'+ """+ # meta_path = os.path.join(root, f'raw/meta/{split}.csv')+ meta_dirname = os.path.dirname(meta_path)+ assert meta_dirname.endswith("raw/meta")+ root_raw = os.path.dirname(meta_dirname)++ meta_fname = os.path.basename(meta_path)+ wo_ext, ext = os.path.splitext(meta_fname)++ meta = pd.read_csv(meta_path)++ def is_intact(row, mode=".json"):+ video_id = row["videoId"] # pexels_new+ video_path = os.path.join(root_raw, f"data/{split}/{video_id}.mp4")+ row["path"] = video_path+ if mode == ".mp4":+ if is_intact_video(video_path):+ return True, video_path+ return False, video_path+ elif mode == ".json":+ json_path = os.path.join(root_raw, f"data/{split}/{video_id}.json")+ if has_downloaded_success(json_path):+ return True, video_path+ return False, video_path+ else:+ raise ValueError++ pandarallel.initialize(progress_bar=True)+ is_intact_partial = partial(is_intact, mode=mode)+ ret = meta.parallel_apply(is_intact_partial, axis=1)++ intact, paths = list(zip(*ret))++ meta["intact"] = intact+ meta["path"] = paths+ out_path = os.path.join(root_raw, f"meta/{wo_ext}_intact.csv")+ meta.to_csv(out_path, index=False)+ print(f"New meta (shape={meta.shape}) with intact info saved to '{out_path}'")++ # meta_format = meta[meta['intact']]+ meta_format = meta[np.array(intact)]+ meta_format.drop("intact", axis=1, inplace=True)+ out_path = os.path.join(root_raw, f"meta/{wo_ext}_format.csv")+ meta_format.to_csv(out_path, index=False)+ print(f"New meta (shape={meta_format.shape}) with format info saved to '{out_path}'")+++def create_subset(meta_path):+ meta = pd.read_csv(meta_path)+ meta_subset = meta.iloc[:100]++ wo_ext, ext = os.path.splitext(meta_path)+ out_path = f"{wo_ext}_head-100{ext}"+ meta_subset.to_csv(out_path, index=False)+ print(f"New meta (shape={meta_subset.shape}) saved to '{out_path}'")+++def append_cut(root="./data/Panda-70M"):+ """+ Append _cut to csv file+ input csv should satisfy:+ - name_should be {split}_intact.csv+ - contain column 'timestamp': list of timestamp+ """+ split = "test"+ meta_path = os.path.join(root, f"processed/meta/{split}_intact.csv")++ wo_ext, ext = os.path.splitext(meta_path)+ suffix = "cut"+ out_path = f"{wo_ext}_{suffix}{ext}"++ meta = pd.read_csv(meta_path)++ new_meta = []+ for idx, row in tqdm(meta.iterrows(), total=len(meta)):+ video_id = row["videoID"]+ timestamps = eval(row["timestamp"])+ captions = eval(row["caption"])+ scores = eval(row["matching_score"])++ num_clips = len(timestamps)+ for idx_c in range(num_clips):+ path_i = os.path.join(root, f"processed/{split}/{video_id}_scene-{idx_c}.mp4")+ # if not is_intact_video(path_i):+ # continue++ row_i = [f"{video_id}_scene-{idx_c}", path_i, timestamps[idx_c], captions[idx_c], scores[idx_c]]++ new_meta.append(row_i)++ columns = ["videoID", "path", "timestamp", "text", "match_official"]+ new_meta = pd.DataFrame(new_meta, columns=columns)++ new_meta.to_csv(out_path, index=False)+ print(f"New meta (shape={new_meta.shape}) saved to '{out_path}'")+++def debug_meta_topk():+ meta_path = "F:/Panda-70M/meta/test_intact_cut_flow.csv"+ meta = pd.read_csv(meta_path)++ score_column = "flow_score"+ topk = meta.nlargest(10, columns=score_column)+ topk_s = meta.nsmallest(200, columns=score_column)++ [(row["path"], row["caption"], row[score_column]) for idx, row in topk.iterrows()]+ [(row["path"], row["caption"], row[score_column]) for idx, row in topk_s.iterrows()]+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("--task", default="append_format")+ parser.add_argument("--meta_path", default="./data/pexels_new/raw/meta/popular_1.csv")+ parser.add_argument("--split", default="popular_5")+ parser.add_argument("--num_workers", default=5, type=int)++ args = parser.parse_args()+ return args+++if __name__ == "__main__":+ # split_meta_csv()++ args = parse_args()+ meta_path = args.meta_path+ task = args.task++ if task == "append_format":+ # append_format(meta_path=meta_path, mode='.mp4')+ append_format_pandarallel(meta_path=meta_path, split=args.split, mode=".json")+ elif task == "create_subset":+ create_subset(meta_path=meta_path)+ else:+ raise ValueError++ # append_cut(root=root)+ # append_score(root=root)+ # debug_meta_topk()diff --git a/tools/scene_cut/scene_detect.py b/tools/scene_cut/scene_detect.pyindex 7a952cb..c0f45cc 100644--- a/tools/scene_cut/scene_detect.py+++ b/tools/scene_cut/scene_detect.py@@ -1,89 +1,89 @@-import argparse-import os-import numpy as np-from tqdm import tqdm-import pandas as pd-from functools import partial-from pandarallel import pandarallel-from scenedetect import detect, ContentDetector, AdaptiveDetector, FrameTimecode---def process_single_row(row):- # windows- # from scenedetect import detect, ContentDetector, AdaptiveDetector-- video_path = row['path']-- detector = AdaptiveDetector(- adaptive_threshold=3.0,- # luma_only=True,- )- # detector = ContentDetector()- # TODO: catch error here- try:- scene_list = detect(video_path, detector, start_in_scene=True)- timestamp = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]- return True, str(timestamp)- except Exception as e:- print(f'Video \'{video_path}\' with error {e}')- return False, ''---def main():- meta_path = 'F:/pexels_new/raw/meta/popular_1_format.csv'- meta = pd.read_csv(meta_path)-- timestamp_list = []- for idx, row in tqdm(meta.iterrows()):- video_path = row['path']-- detector = AdaptiveDetector(- adaptive_threshold=1.5,- luma_only=True,- )- # detector = ContentDetector()- scene_list = detect(video_path, detector, start_in_scene=True)-- timestamp = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]- timestamp_list.append(timestamp)-- meta['timestamp'] = timestamp_list-- wo_ext, ext = os.path.splitext(meta_path)- out_path = f"{wo_ext}_timestamp{ext}"- meta.to_csv(out_path, index=False)- print(f"New meta with timestamp saved to '{out_path}'.")---def parse_args():- parser = argparse.ArgumentParser()- parser.add_argument('--meta_path', default='F:/pexels_new/raw/meta/popular_1_format.csv')- parser.add_argument('--num_workers', default=5, type=int)-- args = parser.parse_args()- return args---def main_pandarallel():- args = parse_args()- meta_path = args.meta_path-- # meta_path = 'F:/pexels_new/raw/meta/popular_1_format.csv'- meta = pd.read_csv(meta_path)-- pandarallel.initialize(progress_bar=True)- ret = meta.parallel_apply(process_single_row, axis=1)-- succ, timestamps = list(zip(*ret))-- meta['timestamp'] = timestamps- meta = meta[np.array(succ)]-- wo_ext, ext = os.path.splitext(meta_path)- out_path = f"{wo_ext}_timestamp{ext}"- meta.to_csv(out_path, index=False)- print(f"New meta (shape={meta.shape}) with timestamp saved to '{out_path}'.")---if __name__ == '__main__':- main_pandarallel()+import argparse+import os++import numpy as np+import pandas as pd+from pandarallel import pandarallel+from scenedetect import AdaptiveDetector, detect+from tqdm import tqdm+++def process_single_row(row):+ # windows+ # from scenedetect import detect, ContentDetector, AdaptiveDetector++ video_path = row["path"]++ detector = AdaptiveDetector(+ adaptive_threshold=3.0,+ # luma_only=True,+ )+ # detector = ContentDetector()+ # TODO: catch error here+ try:+ scene_list = detect(video_path, detector, start_in_scene=True)+ timestamp = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]+ return True, str(timestamp)+ except Exception as e:+ print(f"Video '{video_path}' with error {e}")+ return False, ""+++def main():+ meta_path = "F:/pexels_new/raw/meta/popular_1_format.csv"+ meta = pd.read_csv(meta_path)++ timestamp_list = []+ for idx, row in tqdm(meta.iterrows()):+ video_path = row["path"]++ detector = AdaptiveDetector(+ adaptive_threshold=1.5,+ luma_only=True,+ )+ # detector = ContentDetector()+ scene_list = detect(video_path, detector, start_in_scene=True)++ timestamp = [(s.get_timecode(), t.get_timecode()) for s, t in scene_list]+ timestamp_list.append(timestamp)++ meta["timestamp"] = timestamp_list++ wo_ext, ext = os.path.splitext(meta_path)+ out_path = f"{wo_ext}_timestamp{ext}"+ meta.to_csv(out_path, index=False)+ print(f"New meta with timestamp saved to '{out_path}'.")+++def parse_args():+ parser = argparse.ArgumentParser()+ parser.add_argument("--meta_path", default="F:/pexels_new/raw/meta/popular_1_format.csv")+ parser.add_argument("--num_workers", default=5, type=int)++ args = parser.parse_args()+ return args+++def main_pandarallel():+ args = parse_args()+ meta_path = args.meta_path++ # meta_path = 'F:/pexels_new/raw/meta/popular_1_format.csv'+ meta = pd.read_csv(meta_path)++ pandarallel.initialize(progress_bar=True)+ ret = meta.parallel_apply(process_single_row, axis=1)++ succ, timestamps = list(zip(*ret))++ meta["timestamp"] = timestamps+ meta = meta[np.array(succ)]++ wo_ext, ext = os.path.splitext(meta_path)+ out_path = f"{wo_ext}_timestamp{ext}"+ meta.to_csv(out_path, index=False)+ print(f"New meta (shape={meta.shape}) with timestamp saved to '{out_path}'.")+++if __name__ == "__main__":+ main_pandarallel()diff --git a/tools/scene_cut/utils_video.py b/tools/scene_cut/utils_video.pyindex 2f02e8f..4847180 100644--- a/tools/scene_cut/utils_video.py+++ b/tools/scene_cut/utils_video.py@@ -1,97 +1,97 @@-import os-import cv2-from mmengine.logging import print_log-from moviepy.editor import VideoFileClip---def iterate_files(folder_path):- for root, dirs, files in os.walk(folder_path):- # root contains the current directory path- # dirs contains the list of subdirectories in the current directory- # files contains the list of files in the current directory-- # Process files in the current directory- for file in files:- file_path = os.path.join(root, file)- # print("File:", file_path)- yield file_path-- # Process subdirectories and recursively call the function- for subdir in dirs:- subdir_path = os.path.join(root, subdir)- # print("Subdirectory:", subdir_path)- iterate_files(subdir_path)---def iterate_folders(folder_path):- for root, dirs, files in os.walk(folder_path):- for subdir in dirs:- subdir_path = os.path.join(root, subdir)- yield subdir_path- # print("Subdirectory:", subdir_path)- iterate_folders(subdir_path)---def clone_folder_structure(root_src, root_dst, verbose=False):- src_path_list = iterate_folders(root_src)- src_relpath_list = [os.path.relpath(x, root_src) for x in src_path_list]-- os.makedirs(root_dst, exist_ok=True)- dst_path_list = [os.path.join(root_dst, x) for x in src_relpath_list]- for folder_path in dst_path_list:- os.makedirs(folder_path, exist_ok=True)- if verbose:- print(f"Create folder: '{folder_path}'")---def is_intact_video(video_path, mode='moviepy', verbose=False, logger=None):- if not os.path.exists(video_path):- if verbose:- print_log(f"Could not find '{video_path}'", logger=logger)- return False-- if mode == 'moviepy':- try:- VideoFileClip(video_path)- if verbose:- print_log(f"The video file '{video_path}' is intact.", logger=logger)- return True- except Exception as e:- if verbose:- print_log(f"Error: {e}", logger=logger)- print_log(f"The video file '{video_path}' is not intact.", logger=logger)- return False- elif mode == 'cv2':- try:- cap = cv2.VideoCapture(video_path)- if cap.isOpened():- if verbose:- print_log(f"The video file '{video_path}' is intact.", logger=logger)- return True- except Exception as e:- if verbose:- print_log(f"Error: {e}", logger=logger)- print_log(f"The video file '{video_path}' is not intact.", logger=logger)- return False- else:- raise ValueError---def count_frames(video_path, logger=None):- cap = cv2.VideoCapture(video_path)-- if not cap.isOpened():- print_log(f"Error: Could not open video file '{video_path}'", logger=logger)- return-- total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))- print_log(f"Total frames in the video '{video_path}': {total_frames}", logger=logger)-- cap.release()---def count_files(root, suffix=".mp4"):- files_list = iterate_files(root)- cnt = len([x for x in files_list if x.endswith(suffix)])- return cnt-+import os++import cv2+from mmengine.logging import print_log+from moviepy.editor import VideoFileClip+++def iterate_files(folder_path):+ for root, dirs, files in os.walk(folder_path):+ # root contains the current directory path+ # dirs contains the list of subdirectories in the current directory+ # files contains the list of files in the current directory++ # Process files in the current directory+ for file in files:+ file_path = os.path.join(root, file)+ # print("File:", file_path)+ yield file_path++ # Process subdirectories and recursively call the function+ for subdir in dirs:+ subdir_path = os.path.join(root, subdir)+ # print("Subdirectory:", subdir_path)+ iterate_files(subdir_path)+++def iterate_folders(folder_path):+ for root, dirs, files in os.walk(folder_path):+ for subdir in dirs:+ subdir_path = os.path.join(root, subdir)+ yield subdir_path+ # print("Subdirectory:", subdir_path)+ iterate_folders(subdir_path)+++def clone_folder_structure(root_src, root_dst, verbose=False):+ src_path_list = iterate_folders(root_src)+ src_relpath_list = [os.path.relpath(x, root_src) for x in src_path_list]++ os.makedirs(root_dst, exist_ok=True)+ dst_path_list = [os.path.join(root_dst, x) for x in src_relpath_list]+ for folder_path in dst_path_list:+ os.makedirs(folder_path, exist_ok=True)+ if verbose:+ print(f"Create folder: '{folder_path}'")+++def is_intact_video(video_path, mode="moviepy", verbose=False, logger=None):+ if not os.path.exists(video_path):+ if verbose:+ print_log(f"Could not find '{video_path}'", logger=logger)+ return False++ if mode == "moviepy":+ try:+ VideoFileClip(video_path)+ if verbose:+ print_log(f"The video file '{video_path}' is intact.", logger=logger)+ return True+ except Exception as e:+ if verbose:+ print_log(f"Error: {e}", logger=logger)+ print_log(f"The video file '{video_path}' is not intact.", logger=logger)+ return False+ elif mode == "cv2":+ try:+ cap = cv2.VideoCapture(video_path)+ if cap.isOpened():+ if verbose:+ print_log(f"The video file '{video_path}' is intact.", logger=logger)+ return True+ except Exception as e:+ if verbose:+ print_log(f"Error: {e}", logger=logger)+ print_log(f"The video file '{video_path}' is not intact.", logger=logger)+ return False+ else:+ raise ValueError+++def count_frames(video_path, logger=None):+ cap = cv2.VideoCapture(video_path)++ if not cap.isOpened():+ print_log(f"Error: Could not open video file '{video_path}'", logger=logger)+ return++ total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))+ print_log(f"Total frames in the video '{video_path}': {total_frames}", logger=logger)++ cap.release()+++def count_files(root, suffix=".mp4"):+ files_list = iterate_files(root)+ cnt = len([x for x in files_list if x.endswith(suffix)])+ return cntdiff --git a/tools/scoring/optical_flow/unimatch/__init__.py b/tools/scoring/optical_flow/unimatch/__init__.pyindex 3590101..c1f4eb2 100644--- a/tools/scoring/optical_flow/unimatch/__init__.py+++ b/tools/scoring/optical_flow/unimatch/__init__.py@@ -1 +1 @@-from .unimatch import UniMatch+from .unimatch import UniMatchdiff --git a/tools/scoring/optical_flow/unimatch/attention.py b/tools/scoring/optical_flow/unimatch/attention.pyindex a10f758..23fb904 100644--- a/tools/scoring/optical_flow/unimatch/attention.py+++ b/tools/scoring/optical_flow/unimatch/attention.py@@ -1,253 +1,280 @@-import torch-import torch.nn as nn-import torch.nn.functional as F--from .utils import split_feature, merge_splits, split_feature_1d, merge_splits_1d---def single_head_full_attention(q, k, v):- # q, k, v: [B, L, C]- assert q.dim() == k.dim() == v.dim() == 3-- scores = torch.matmul(q, k.permute(0, 2, 1)) / (q.size(2) ** .5) # [B, L, L]- attn = torch.softmax(scores, dim=2) # [B, L, L]- out = torch.matmul(attn, v) # [B, L, C]-- return out---def single_head_full_attention_1d(q, k, v,- h=None,- w=None,- ):- # q, k, v: [B, L, C]-- assert h is not None and w is not None- assert q.size(1) == h * w-- b, _, c = q.size()-- q = q.view(b, h, w, c) # [B, H, W, C]- k = k.view(b, h, w, c)- v = v.view(b, h, w, c)-- scale_factor = c ** 0.5-- scores = torch.matmul(q, k.permute(0, 1, 3, 2)) / scale_factor # [B, H, W, W]-- attn = torch.softmax(scores, dim=-1)-- out = torch.matmul(attn, v).view(b, -1, c) # [B, H*W, C]-- return out---def single_head_split_window_attention(q, k, v,- num_splits=1,- with_shift=False,- h=None,- w=None,- attn_mask=None,- ):- # ref: https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py- # q, k, v: [B, L, C]- assert q.dim() == k.dim() == v.dim() == 3-- assert h is not None and w is not None- assert q.size(1) == h * w-- b, _, c = q.size()-- b_new = b * num_splits * num_splits-- window_size_h = h // num_splits- window_size_w = w // num_splits-- q = q.view(b, h, w, c) # [B, H, W, C]- k = k.view(b, h, w, c)- v = v.view(b, h, w, c)-- scale_factor = c ** 0.5-- if with_shift:- assert attn_mask is not None # compute once- shift_size_h = window_size_h // 2- shift_size_w = window_size_w // 2-- q = torch.roll(q, shifts=(-shift_size_h, -shift_size_w), dims=(1, 2))- k = torch.roll(k, shifts=(-shift_size_h, -shift_size_w), dims=(1, 2))- v = torch.roll(v, shifts=(-shift_size_h, -shift_size_w), dims=(1, 2))-- q = split_feature(q, num_splits=num_splits, channel_last=True) # [B*K*K, H/K, W/K, C]- k = split_feature(k, num_splits=num_splits, channel_last=True)- v = split_feature(v, num_splits=num_splits, channel_last=True)-- scores = torch.matmul(q.view(b_new, -1, c), k.view(b_new, -1, c).permute(0, 2, 1)- ) / scale_factor # [B*K*K, H/K*W/K, H/K*W/K]-- if with_shift:- scores += attn_mask.repeat(b, 1, 1)-- attn = torch.softmax(scores, dim=-1)-- out = torch.matmul(attn, v.view(b_new, -1, c)) # [B*K*K, H/K*W/K, C]-- out = merge_splits(out.view(b_new, h // num_splits, w // num_splits, c),- num_splits=num_splits, channel_last=True) # [B, H, W, C]-- # shift back- if with_shift:- out = torch.roll(out, shifts=(shift_size_h, shift_size_w), dims=(1, 2))-- out = out.view(b, -1, c)-- return out---def single_head_split_window_attention_1d(q, k, v,- relative_position_bias=None,- num_splits=1,- with_shift=False,- h=None,- w=None,- attn_mask=None,- ):- # q, k, v: [B, L, C]-- assert h is not None and w is not None- assert q.size(1) == h * w-- b, _, c = q.size()-- b_new = b * num_splits * h-- window_size_w = w // num_splits-- q = q.view(b * h, w, c) # [B*H, W, C]- k = k.view(b * h, w, c)- v = v.view(b * h, w, c)-- scale_factor = c ** 0.5-- if with_shift:- assert attn_mask is not None # compute once- shift_size_w = window_size_w // 2-- q = torch.roll(q, shifts=-shift_size_w, dims=1)- k = torch.roll(k, shifts=-shift_size_w, dims=1)- v = torch.roll(v, shifts=-shift_size_w, dims=1)-- q = split_feature_1d(q, num_splits=num_splits) # [B*H*K, W/K, C]- k = split_feature_1d(k, num_splits=num_splits)- v = split_feature_1d(v, num_splits=num_splits)-- scores = torch.matmul(q.view(b_new, -1, c), k.view(b_new, -1, c).permute(0, 2, 1)- ) / scale_factor # [B*H*K, W/K, W/K]-- if with_shift:- # attn_mask: [K, W/K, W/K]- scores += attn_mask.repeat(b * h, 1, 1) # [B*H*K, W/K, W/K]-- attn = torch.softmax(scores, dim=-1)-- out = torch.matmul(attn, v.view(b_new, -1, c)) # [B*H*K, W/K, C]-- out = merge_splits_1d(out, h, num_splits=num_splits) # [B, H, W, C]-- # shift back- if with_shift:- out = torch.roll(out, shifts=shift_size_w, dims=2)-- out = out.view(b, -1, c)-- return out---class SelfAttnPropagation(nn.Module):- """- flow propagation with self-attention on feature- query: feature0, key: feature0, value: flow- """-- def __init__(self, in_channels,- **kwargs,- ):- super(SelfAttnPropagation, self).__init__()-- self.q_proj = nn.Linear(in_channels, in_channels)- self.k_proj = nn.Linear(in_channels, in_channels)-- for p in self.parameters():- if p.dim() > 1:- nn.init.xavier_uniform_(p)-- def forward(self, feature0, flow,- local_window_attn=False,- local_window_radius=1,- **kwargs,- ):- # q, k: feature [B, C, H, W], v: flow [B, 2, H, W]- if local_window_attn:- return self.forward_local_window_attn(feature0, flow,- local_window_radius=local_window_radius)-- b, c, h, w = feature0.size()-- query = feature0.view(b, c, h * w).permute(0, 2, 1) # [B, H*W, C]-- # a note: the ``correct'' implementation should be:- # ``query = self.q_proj(query), key = self.k_proj(query)''- # this problem is observed while cleaning up the code- # however, this doesn't affect the performance since the projection is a linear operation,- # thus the two projection matrices for key can be merged- # so I just leave it as is in order to not re-train all models :)- query = self.q_proj(query) # [B, H*W, C]- key = self.k_proj(query) # [B, H*W, C]-- value = flow.view(b, flow.size(1), h * w).permute(0, 2, 1) # [B, H*W, 2]-- scores = torch.matmul(query, key.permute(0, 2, 1)) / (c ** 0.5) # [B, H*W, H*W]- prob = torch.softmax(scores, dim=-1)-- out = torch.matmul(prob, value) # [B, H*W, 2]- out = out.view(b, h, w, value.size(-1)).permute(0, 3, 1, 2) # [B, 2, H, W]-- return out-- def forward_local_window_attn(self, feature0, flow,- local_window_radius=1,- ):- assert flow.size(1) == 2 or flow.size(1) == 1 # flow or disparity or depth- assert local_window_radius > 0-- b, c, h, w = feature0.size()-- value_channel = flow.size(1)-- feature0_reshape = self.q_proj(feature0.view(b, c, -1).permute(0, 2, 1)- ).reshape(b * h * w, 1, c) # [B*H*W, 1, C]-- kernel_size = 2 * local_window_radius + 1-- feature0_proj = self.k_proj(feature0.view(b, c, -1).permute(0, 2, 1)).permute(0, 2, 1).reshape(b, c, h, w)-- feature0_window = F.unfold(feature0_proj, kernel_size=kernel_size,- padding=local_window_radius) # [B, C*(2R+1)^2), H*W]-- feature0_window = feature0_window.view(b, c, kernel_size ** 2, h, w).permute(- 0, 3, 4, 1, 2).reshape(b * h * w, c, kernel_size ** 2) # [B*H*W, C, (2R+1)^2]-- flow_window = F.unfold(flow, kernel_size=kernel_size,- padding=local_window_radius) # [B, 2*(2R+1)^2), H*W]-- flow_window = flow_window.view(b, value_channel, kernel_size ** 2, h, w).permute(- 0, 3, 4, 2, 1).reshape(b * h * w, kernel_size ** 2, value_channel) # [B*H*W, (2R+1)^2, 2]-- scores = torch.matmul(feature0_reshape, feature0_window) / (c ** 0.5) # [B*H*W, 1, (2R+1)^2]-- prob = torch.softmax(scores, dim=-1)-- out = torch.matmul(prob, flow_window).view(b, h, w, value_channel- ).permute(0, 3, 1, 2).contiguous() # [B, 2, H, W]-- return out+import torch+import torch.nn as nn+import torch.nn.functional as F++from .utils import merge_splits, merge_splits_1d, split_feature, split_feature_1d+++def single_head_full_attention(q, k, v):+ # q, k, v: [B, L, C]+ assert q.dim() == k.dim() == v.dim() == 3++ scores = torch.matmul(q, k.permute(0, 2, 1)) / (q.size(2) ** 0.5) # [B, L, L]+ attn = torch.softmax(scores, dim=2) # [B, L, L]+ out = torch.matmul(attn, v) # [B, L, C]++ return out+++def single_head_full_attention_1d(+ q,+ k,+ v,+ h=None,+ w=None,+):+ # q, k, v: [B, L, C]++ assert h is not None and w is not None+ assert q.size(1) == h * w++ b, _, c = q.size()++ q = q.view(b, h, w, c) # [B, H, W, C]+ k = k.view(b, h, w, c)+ v = v.view(b, h, w, c)++ scale_factor = c**0.5++ scores = torch.matmul(q, k.permute(0, 1, 3, 2)) / scale_factor # [B, H, W, W]++ attn = torch.softmax(scores, dim=-1)++ out = torch.matmul(attn, v).view(b, -1, c) # [B, H*W, C]++ return out+++def single_head_split_window_attention(+ q,+ k,+ v,+ num_splits=1,+ with_shift=False,+ h=None,+ w=None,+ attn_mask=None,+):+ # ref: https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py+ # q, k, v: [B, L, C]+ assert q.dim() == k.dim() == v.dim() == 3++ assert h is not None and w is not None+ assert q.size(1) == h * w++ b, _, c = q.size()++ b_new = b * num_splits * num_splits++ window_size_h = h // num_splits+ window_size_w = w // num_splits++ q = q.view(b, h, w, c) # [B, H, W, C]+ k = k.view(b, h, w, c)+ v = v.view(b, h, w, c)++ scale_factor = c**0.5++ if with_shift:+ assert attn_mask is not None # compute once+ shift_size_h = window_size_h // 2+ shift_size_w = window_size_w // 2++ q = torch.roll(q, shifts=(-shift_size_h, -shift_size_w), dims=(1, 2))+ k = torch.roll(k, shifts=(-shift_size_h, -shift_size_w), dims=(1, 2))+ v = torch.roll(v, shifts=(-shift_size_h, -shift_size_w), dims=(1, 2))++ q = split_feature(q, num_splits=num_splits, channel_last=True) # [B*K*K, H/K, W/K, C]+ k = split_feature(k, num_splits=num_splits, channel_last=True)+ v = split_feature(v, num_splits=num_splits, channel_last=True)++ scores = (+ torch.matmul(q.view(b_new, -1, c), k.view(b_new, -1, c).permute(0, 2, 1)) / scale_factor+ ) # [B*K*K, H/K*W/K, H/K*W/K]++ if with_shift:+ scores += attn_mask.repeat(b, 1, 1)++ attn = torch.softmax(scores, dim=-1)++ out = torch.matmul(attn, v.view(b_new, -1, c)) # [B*K*K, H/K*W/K, C]++ out = merge_splits(+ out.view(b_new, h // num_splits, w // num_splits, c), num_splits=num_splits, channel_last=True+ ) # [B, H, W, C]++ # shift back+ if with_shift:+ out = torch.roll(out, shifts=(shift_size_h, shift_size_w), dims=(1, 2))++ out = out.view(b, -1, c)++ return out+++def single_head_split_window_attention_1d(+ q,+ k,+ v,+ relative_position_bias=None,+ num_splits=1,+ with_shift=False,+ h=None,+ w=None,+ attn_mask=None,+):+ # q, k, v: [B, L, C]++ assert h is not None and w is not None+ assert q.size(1) == h * w++ b, _, c = q.size()++ b_new = b * num_splits * h++ window_size_w = w // num_splits++ q = q.view(b * h, w, c) # [B*H, W, C]+ k = k.view(b * h, w, c)+ v = v.view(b * h, w, c)++ scale_factor = c**0.5++ if with_shift:+ assert attn_mask is not None # compute once+ shift_size_w = window_size_w // 2++ q = torch.roll(q, shifts=-shift_size_w, dims=1)+ k = torch.roll(k, shifts=-shift_size_w, dims=1)+ v = torch.roll(v, shifts=-shift_size_w, dims=1)++ q = split_feature_1d(q, num_splits=num_splits) # [B*H*K, W/K, C]+ k = split_feature_1d(k, num_splits=num_splits)+ v = split_feature_1d(v, num_splits=num_splits)++ scores = (+ torch.matmul(q.view(b_new, -1, c), k.view(b_new, -1, c).permute(0, 2, 1)) / scale_factor+ ) # [B*H*K, W/K, W/K]++ if with_shift:+ # attn_mask: [K, W/K, W/K]+ scores += attn_mask.repeat(b * h, 1, 1) # [B*H*K, W/K, W/K]++ attn = torch.softmax(scores, dim=-1)++ out = torch.matmul(attn, v.view(b_new, -1, c)) # [B*H*K, W/K, C]++ out = merge_splits_1d(out, h, num_splits=num_splits) # [B, H, W, C]++ # shift back+ if with_shift:+ out = torch.roll(out, shifts=shift_size_w, dims=2)++ out = out.view(b, -1, c)++ return out+++class SelfAttnPropagation(nn.Module):+ """+ flow propagation with self-attention on feature+ query: feature0, key: feature0, value: flow+ """++ def __init__(+ self,+ in_channels,+ **kwargs,+ ):+ super(SelfAttnPropagation, self).__init__()++ self.q_proj = nn.Linear(in_channels, in_channels)+ self.k_proj = nn.Linear(in_channels, in_channels)++ for p in self.parameters():+ if p.dim() > 1:+ nn.init.xavier_uniform_(p)++ def forward(+ self,+ feature0,+ flow,+ local_window_attn=False,+ local_window_radius=1,+ **kwargs,+ ):+ # q, k: feature [B, C, H, W], v: flow [B, 2, H, W]+ if local_window_attn:+ return self.forward_local_window_attn(feature0, flow, local_window_radius=local_window_radius)++ b, c, h, w = feature0.size()++ query = feature0.view(b, c, h * w).permute(0, 2, 1) # [B, H*W, C]++ # a note: the ``correct'' implementation should be:+ # ``query = self.q_proj(query), key = self.k_proj(query)''+ # this problem is observed while cleaning up the code+ # however, this doesn't affect the performance since the projection is a linear operation,+ # thus the two projection matrices for key can be merged+ # so I just leave it as is in order to not re-train all models :)+ query = self.q_proj(query) # [B, H*W, C]+ key = self.k_proj(query) # [B, H*W, C]++ value = flow.view(b, flow.size(1), h * w).permute(0, 2, 1) # [B, H*W, 2]++ scores = torch.matmul(query, key.permute(0, 2, 1)) / (c**0.5) # [B, H*W, H*W]+ prob = torch.softmax(scores, dim=-1)++ out = torch.matmul(prob, value) # [B, H*W, 2]+ out = out.view(b, h, w, value.size(-1)).permute(0, 3, 1, 2) # [B, 2, H, W]++ return out++ def forward_local_window_attn(+ self,+ feature0,+ flow,+ local_window_radius=1,+ ):+ assert flow.size(1) == 2 or flow.size(1) == 1 # flow or disparity or depth+ assert local_window_radius > 0++ b, c, h, w = feature0.size()++ value_channel = flow.size(1)++ feature0_reshape = self.q_proj(feature0.view(b, c, -1).permute(0, 2, 1)).reshape(+ b * h * w, 1, c+ ) # [B*H*W, 1, C]++ kernel_size = 2 * local_window_radius + 1++ feature0_proj = self.k_proj(feature0.view(b, c, -1).permute(0, 2, 1)).permute(0, 2, 1).reshape(b, c, h, w)++ feature0_window = F.unfold(+ feature0_proj, kernel_size=kernel_size, padding=local_window_radius+ ) # [B, C*(2R+1)^2), H*W]++ feature0_window = (+ feature0_window.view(b, c, kernel_size**2, h, w)+ .permute(0, 3, 4, 1, 2)+ .reshape(b * h * w, c, kernel_size**2)+ ) # [B*H*W, C, (2R+1)^2]++ flow_window = F.unfold(flow, kernel_size=kernel_size, padding=local_window_radius) # [B, 2*(2R+1)^2), H*W]++ flow_window = (+ flow_window.view(b, value_channel, kernel_size**2, h, w)+ .permute(0, 3, 4, 2, 1)+ .reshape(b * h * w, kernel_size**2, value_channel)+ ) # [B*H*W, (2R+1)^2, 2]++ scores = torch.matmul(feature0_reshape, feature0_window) / (c**0.5) # [B*H*W, 1, (2R+1)^2]++ prob = torch.softmax(scores, dim=-1)++ out = (+ torch.matmul(prob, flow_window).view(b, h, w, value_channel).permute(0, 3, 1, 2).contiguous()+ ) # [B, 2, H, W]++ return outdiff --git a/tools/scoring/optical_flow/unimatch/backbone.py b/tools/scoring/optical_flow/unimatch/backbone.pyindex d5c92b7..5c2cc19 100644--- a/tools/scoring/optical_flow/unimatch/backbone.py+++ b/tools/scoring/optical_flow/unimatch/backbone.py@@ -1,117 +1,128 @@-import torch.nn as nn--from .trident_conv import MultiScaleTridentConv---class ResidualBlock(nn.Module):- def __init__(self, in_planes, planes, norm_layer=nn.InstanceNorm2d, stride=1, dilation=1,- ):- super(ResidualBlock, self).__init__()-- self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3,- dilation=dilation, padding=dilation, stride=stride, bias=False)- self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,- dilation=dilation, padding=dilation, bias=False)- self.relu = nn.ReLU(inplace=True)-- self.norm1 = norm_layer(planes)- self.norm2 = norm_layer(planes)- if not stride == 1 or in_planes != planes:- self.norm3 = norm_layer(planes)-- if stride == 1 and in_planes == planes:- self.downsample = None- else:- self.downsample = nn.Sequential(- nn.Conv2d(in_planes, planes, kernel_size=1, stride=stride), self.norm3)-- def forward(self, x):- y = x- y = self.relu(self.norm1(self.conv1(y)))- y = self.relu(self.norm2(self.conv2(y)))-- if self.downsample is not None:- x = self.downsample(x)-- return self.relu(x + y)---class CNNEncoder(nn.Module):- def __init__(self, output_dim=128,- norm_layer=nn.InstanceNorm2d,- num_output_scales=1,- **kwargs,- ):- super(CNNEncoder, self).__init__()- self.num_branch = num_output_scales-- feature_dims = [64, 96, 128]-- self.conv1 = nn.Conv2d(3, feature_dims[0], kernel_size=7, stride=2, padding=3, bias=False) # 1/2- self.norm1 = norm_layer(feature_dims[0])- self.relu1 = nn.ReLU(inplace=True)-- self.in_planes = feature_dims[0]- self.layer1 = self._make_layer(feature_dims[0], stride=1, norm_layer=norm_layer) # 1/2- self.layer2 = self._make_layer(feature_dims[1], stride=2, norm_layer=norm_layer) # 1/4-- # highest resolution 1/4 or 1/8- stride = 2 if num_output_scales == 1 else 1- self.layer3 = self._make_layer(feature_dims[2], stride=stride,- norm_layer=norm_layer,- ) # 1/4 or 1/8-- self.conv2 = nn.Conv2d(feature_dims[2], output_dim, 1, 1, 0)-- if self.num_branch > 1:- if self.num_branch == 4:- strides = (1, 2, 4, 8)- elif self.num_branch == 3:- strides = (1, 2, 4)- elif self.num_branch == 2:- strides = (1, 2)- else:- raise ValueError-- self.trident_conv = MultiScaleTridentConv(output_dim, output_dim,- kernel_size=3,- strides=strides,- paddings=1,- num_branch=self.num_branch,- )-- for m in self.modules():- if isinstance(m, nn.Conv2d):- nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')- elif isinstance(m, (nn.BatchNorm2d, nn.InstanceNorm2d, nn.GroupNorm)):- if m.weight is not None:- nn.init.constant_(m.weight, 1)- if m.bias is not None:- nn.init.constant_(m.bias, 0)-- def _make_layer(self, dim, stride=1, dilation=1, norm_layer=nn.InstanceNorm2d):- layer1 = ResidualBlock(self.in_planes, dim, norm_layer=norm_layer, stride=stride, dilation=dilation)- layer2 = ResidualBlock(dim, dim, norm_layer=norm_layer, stride=1, dilation=dilation)-- layers = (layer1, layer2)-- self.in_planes = dim- return nn.Sequential(*layers)-- def forward(self, x):- x = self.conv1(x)- x = self.norm1(x)- x = self.relu1(x)-- x = self.layer1(x) # 1/2- x = self.layer2(x) # 1/4- x = self.layer3(x) # 1/8 or 1/4-- x = self.conv2(x)-- if self.num_branch > 1:- out = self.trident_conv([x] * self.num_branch) # high to low res- else:- out = [x]-- return out+import torch.nn as nn++from .trident_conv import MultiScaleTridentConv+++class ResidualBlock(nn.Module):+ def __init__(+ self,+ in_planes,+ planes,+ norm_layer=nn.InstanceNorm2d,+ stride=1,+ dilation=1,+ ):+ super(ResidualBlock, self).__init__()++ self.conv1 = nn.Conv2d(+ in_planes, planes, kernel_size=3, dilation=dilation, padding=dilation, stride=stride, bias=False+ )+ self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, dilation=dilation, padding=dilation, bias=False)+ self.relu = nn.ReLU(inplace=True)++ self.norm1 = norm_layer(planes)+ self.norm2 = norm_layer(planes)+ if not stride == 1 or in_planes != planes:+ self.norm3 = norm_layer(planes)++ if stride == 1 and in_planes == planes:+ self.downsample = None+ else:+ self.downsample = nn.Sequential(nn.Conv2d(in_planes, planes, kernel_size=1, stride=stride), self.norm3)++ def forward(self, x):+ y = x+ y = self.relu(self.norm1(self.conv1(y)))+ y = self.relu(self.norm2(self.conv2(y)))++ if self.downsample is not None:+ x = self.downsample(x)++ return self.relu(x + y)+++class CNNEncoder(nn.Module):+ def __init__(+ self,+ output_dim=128,+ norm_layer=nn.InstanceNorm2d,+ num_output_scales=1,+ **kwargs,+ ):+ super(CNNEncoder, self).__init__()+ self.num_branch = num_output_scales++ feature_dims = [64, 96, 128]++ self.conv1 = nn.Conv2d(3, feature_dims[0], kernel_size=7, stride=2, padding=3, bias=False) # 1/2+ self.norm1 = norm_layer(feature_dims[0])+ self.relu1 = nn.ReLU(inplace=True)++ self.in_planes = feature_dims[0]+ self.layer1 = self._make_layer(feature_dims[0], stride=1, norm_layer=norm_layer) # 1/2+ self.layer2 = self._make_layer(feature_dims[1], stride=2, norm_layer=norm_layer) # 1/4++ # highest resolution 1/4 or 1/8+ stride = 2 if num_output_scales == 1 else 1+ self.layer3 = self._make_layer(+ feature_dims[2],+ stride=stride,+ norm_layer=norm_layer,+ ) # 1/4 or 1/8++ self.conv2 = nn.Conv2d(feature_dims[2], output_dim, 1, 1, 0)++ if self.num_branch > 1:+ if self.num_branch == 4:+ strides = (1, 2, 4, 8)+ elif self.num_branch == 3:+ strides = (1, 2, 4)+ elif self.num_branch == 2:+ strides = (1, 2)+ else:+ raise ValueError++ self.trident_conv = MultiScaleTridentConv(+ output_dim,+ output_dim,+ kernel_size=3,+ strides=strides,+ paddings=1,+ num_branch=self.num_branch,+ )++ for m in self.modules():+ if isinstance(m, nn.Conv2d):+ nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu")+ elif isinstance(m, (nn.BatchNorm2d, nn.InstanceNorm2d, nn.GroupNorm)):+ if m.weight is not None:+ nn.init.constant_(m.weight, 1)+ if m.bias is not None:+ nn.init.constant_(m.bias, 0)++ def _make_layer(self, dim, stride=1, dilation=1, norm_layer=nn.InstanceNorm2d):+ layer1 = ResidualBlock(self.in_planes, dim, norm_layer=norm_layer, stride=stride, dilation=dilation)+ layer2 = ResidualBlock(dim, dim, norm_layer=norm_layer, stride=1, dilation=dilation)++ layers = (layer1, layer2)++ self.in_planes = dim+ return nn.Sequential(*layers)++ def forward(self, x):+ x = self.conv1(x)+ x = self.norm1(x)+ x = self.relu1(x)++ x = self.layer1(x) # 1/2+ x = self.layer2(x) # 1/4+ x = self.layer3(x) # 1/8 or 1/4++ x = self.conv2(x)++ if self.num_branch > 1:+ out = self.trident_conv([x] * self.num_branch) # high to low res+ else:+ out = [x]++ return outdiff --git a/tools/scoring/optical_flow/unimatch/geometry.py b/tools/scoring/optical_flow/unimatch/geometry.pyindex 775a957..df4d8e3 100644--- a/tools/scoring/optical_flow/unimatch/geometry.py+++ b/tools/scoring/optical_flow/unimatch/geometry.py@@ -24,9 +24,9 @@ def coords_grid(b, h, w, homogeneous=False, device=None):def generate_window_grid(h_min, h_max, w_min, w_max, len_h, len_w, device=None):assert device is not None- x, y = torch.meshgrid([torch.linspace(w_min, w_max, len_w, device=device),- torch.linspace(h_min, h_max, len_h, device=device)],- )+ x, y = torch.meshgrid(+ [torch.linspace(w_min, w_max, len_w, device=device), torch.linspace(h_min, h_max, len_h, device=device)],+ )grid = torch.stack((x, y), -1).transpose(0, 1).float() # [H, W, 2]return grid@@ -34,11 +34,11 @@ def generate_window_grid(h_min, h_max, w_min, w_max, len_h, len_w, device=None):def normalize_coords(coords, h, w):# coords: [B, H, W, 2]- c = torch.Tensor([(w - 1) / 2., (h - 1) / 2.]).float().to(coords.device)+ c = torch.Tensor([(w - 1) / 2.0, (h - 1) / 2.0]).float().to(coords.device)return (coords - c) / c # [-1, 1]-def bilinear_sample(img, sample_coords, mode='bilinear', padding_mode='zeros', return_mask=False):+def bilinear_sample(img, sample_coords, mode="bilinear", padding_mode="zeros", return_mask=False):# img: [B, C, H, W]# sample_coords: [B, 2, H, W] in image scaleif sample_coords.size(1) != 2: # [B, H, W, 2]@@ -62,20 +62,16 @@ def bilinear_sample(img, sample_coords, mode='bilinear', padding_mode='zeros', rreturn img-def flow_warp(feature, flow, mask=False, padding_mode='zeros'):+def flow_warp(feature, flow, mask=False, padding_mode="zeros"):b, c, h, w = feature.size()assert flow.size(1) == 2grid = coords_grid(b, h, w).to(flow.device) + flow # [B, 2, H, W]- return bilinear_sample(feature, grid, padding_mode=padding_mode,- return_mask=mask)+ return bilinear_sample(feature, grid, padding_mode=padding_mode, return_mask=mask)-def forward_backward_consistency_check(fwd_flow, bwd_flow,- alpha=0.01,- beta=0.5- ):+def forward_backward_consistency_check(fwd_flow, bwd_flow, alpha=0.01, beta=0.5):# fwd_flow, bwd_flow: [B, 2, H, W]# alpha and beta values are following UnFlow (https://arxiv.org/abs/1711.07837)assert fwd_flow.dim() == 4 and bwd_flow.dim() == 4@@ -121,8 +117,9 @@ def camera_transform(points_ref, extrinsics_ref=None, extrinsics_tgt=None, extriif extrinsics_rel is None:extrinsics_rel = torch.bmm(extrinsics_tgt, torch.inverse(extrinsics_ref)) # [B, 4, 4]- points_tgt = torch.bmm(extrinsics_rel[:, :3, :3],- points_ref.view(b, 3, -1)) + extrinsics_rel[:, :3, -1:] # [B, 3, H*W]+ points_tgt = (+ torch.bmm(extrinsics_rel[:, :3, :3], points_ref.view(b, 3, -1)) + extrinsics_rel[:, :3, -1:]+ ) # [B, 3, H*W]points_tgt = points_tgt.view(b, 3, h, w) # [B, 3, H, W]@@ -146,49 +143,57 @@ def reproject(points_tgt, intrinsics, return_mask=False):if return_mask:# valid mask in pixel space- mask = (pixel_coords[:, 0] >= 0) & (pixel_coords[:, 0] <= (w - 1)) & (- pixel_coords[:, 1] >= 0) & (pixel_coords[:, 1] <= (h - 1)) # [B, H, W]+ mask = (+ (pixel_coords[:, 0] >= 0)+ & (pixel_coords[:, 0] <= (w - 1))+ & (pixel_coords[:, 1] >= 0)+ & (pixel_coords[:, 1] <= (h - 1))+ ) # [B, H, W]return pixel_coords, maskreturn pixel_coords-def reproject_coords(depth_ref, intrinsics, extrinsics_ref=None, extrinsics_tgt=None, extrinsics_rel=None,- return_mask=False):+def reproject_coords(+ depth_ref, intrinsics, extrinsics_ref=None, extrinsics_tgt=None, extrinsics_rel=None, return_mask=False+):# Compute reprojection sample coordspoints_ref = back_project(depth_ref, intrinsics) # [B, 3, H, W]points_tgt = camera_transform(points_ref, extrinsics_ref, extrinsics_tgt, extrinsics_rel=extrinsics_rel)if return_mask:- reproj_coords, mask = reproject(points_tgt, intrinsics,- return_mask=return_mask) # [B, 2, H, W] in image scale+ reproj_coords, mask = reproject(points_tgt, intrinsics, return_mask=return_mask) # [B, 2, H, W] in image scalereturn reproj_coords, mask- reproj_coords = reproject(points_tgt, intrinsics,- return_mask=return_mask) # [B, 2, H, W] in image scale+ reproj_coords = reproject(points_tgt, intrinsics, return_mask=return_mask) # [B, 2, H, W] in image scalereturn reproj_coords-def compute_flow_with_depth_pose(depth_ref, intrinsics,- extrinsics_ref=None, extrinsics_tgt=None, extrinsics_rel=None,- return_mask=False):+def compute_flow_with_depth_pose(+ depth_ref, intrinsics, extrinsics_ref=None, extrinsics_tgt=None, extrinsics_rel=None, return_mask=False+):b, h, w = depth_ref.shapecoords_init = coords_grid(b, h, w, device=depth_ref.device) # [B, 2, H, W]if return_mask:- reproj_coords, mask = reproject_coords(depth_ref, intrinsics, extrinsics_ref, extrinsics_tgt,- extrinsics_rel=extrinsics_rel,- return_mask=return_mask) # [B, 2, H, W]+ reproj_coords, mask = reproject_coords(+ depth_ref,+ intrinsics,+ extrinsics_ref,+ extrinsics_tgt,+ extrinsics_rel=extrinsics_rel,+ return_mask=return_mask,+ ) # [B, 2, H, W]rigid_flow = reproj_coords - coords_initreturn rigid_flow, mask- reproj_coords = reproject_coords(depth_ref, intrinsics, extrinsics_ref, extrinsics_tgt,- extrinsics_rel=extrinsics_rel,- return_mask=return_mask) # [B, 2, H, W]+ reproj_coords = reproject_coords(+ depth_ref, intrinsics, extrinsics_ref, extrinsics_tgt, extrinsics_rel=extrinsics_rel, return_mask=return_mask+ ) # [B, 2, H, W]rigid_flow = reproj_coords - coords_initdiff --git a/tools/scoring/optical_flow/unimatch/matching.py b/tools/scoring/optical_flow/unimatch/matching.pyindex 6471025..fe5e103 100644--- a/tools/scoring/optical_flow/unimatch/matching.py+++ b/tools/scoring/optical_flow/unimatch/matching.py@@ -1,279 +1,307 @@-import torch-import torch.nn.functional as F--from .geometry import coords_grid, generate_window_grid, normalize_coords---def global_correlation_softmax(feature0, feature1,- pred_bidir_flow=False,- ):- # global correlation- b, c, h, w = feature0.shape- feature0 = feature0.view(b, c, -1).permute(0, 2, 1) # [B, H*W, C]- feature1 = feature1.view(b, c, -1) # [B, C, H*W]-- correlation = torch.matmul(feature0, feature1).view(b, h, w, h, w) / (c ** 0.5) # [B, H, W, H, W]-- # flow from softmax- init_grid = coords_grid(b, h, w).to(correlation.device) # [B, 2, H, W]- grid = init_grid.view(b, 2, -1).permute(0, 2, 1) # [B, H*W, 2]-- correlation = correlation.view(b, h * w, h * w) # [B, H*W, H*W]-- if pred_bidir_flow:- correlation = torch.cat((correlation, correlation.permute(0, 2, 1)), dim=0) # [2*B, H*W, H*W]- init_grid = init_grid.repeat(2, 1, 1, 1) # [2*B, 2, H, W]- grid = grid.repeat(2, 1, 1) # [2*B, H*W, 2]- b = b * 2-- prob = F.softmax(correlation, dim=-1) # [B, H*W, H*W]-- correspondence = torch.matmul(prob, grid).view(b, h, w, 2).permute(0, 3, 1, 2) # [B, 2, H, W]-- # when predicting bidirectional flow, flow is the concatenation of forward flow and backward flow- flow = correspondence - init_grid-- return flow, prob---def local_correlation_softmax(feature0, feature1, local_radius,- padding_mode='zeros',- ):- b, c, h, w = feature0.size()- coords_init = coords_grid(b, h, w).to(feature0.device) # [B, 2, H, W]- coords = coords_init.view(b, 2, -1).permute(0, 2, 1) # [B, H*W, 2]-- local_h = 2 * local_radius + 1- local_w = 2 * local_radius + 1-- window_grid = generate_window_grid(-local_radius, local_radius,- -local_radius, local_radius,- local_h, local_w, device=feature0.device) # [2R+1, 2R+1, 2]- window_grid = window_grid.reshape(-1, 2).repeat(b, 1, 1, 1) # [B, 1, (2R+1)^2, 2]- sample_coords = coords.unsqueeze(-2) + window_grid # [B, H*W, (2R+1)^2, 2]-- sample_coords_softmax = sample_coords-- # exclude coords that are out of image space- valid_x = (sample_coords[:, :, :, 0] >= 0) & (sample_coords[:, :, :, 0] < w) # [B, H*W, (2R+1)^2]- valid_y = (sample_coords[:, :, :, 1] >= 0) & (sample_coords[:, :, :, 1] < h) # [B, H*W, (2R+1)^2]-- valid = valid_x & valid_y # [B, H*W, (2R+1)^2], used to mask out invalid values when softmax-- # normalize coordinates to [-1, 1]- sample_coords_norm = normalize_coords(sample_coords, h, w) # [-1, 1]- window_feature = F.grid_sample(feature1, sample_coords_norm,- padding_mode=padding_mode, align_corners=True- ).permute(0, 2, 1, 3) # [B, H*W, C, (2R+1)^2]- feature0_view = feature0.permute(0, 2, 3, 1).view(b, h * w, 1, c) # [B, H*W, 1, C]-- corr = torch.matmul(feature0_view, window_feature).view(b, h * w, -1) / (c ** 0.5) # [B, H*W, (2R+1)^2]-- # mask invalid locations- corr[~valid] = -1e9-- prob = F.softmax(corr, -1) # [B, H*W, (2R+1)^2]-- correspondence = torch.matmul(prob.unsqueeze(-2), sample_coords_softmax).squeeze(-2).view(- b, h, w, 2).permute(0, 3, 1, 2) # [B, 2, H, W]-- flow = correspondence - coords_init- match_prob = prob-- return flow, match_prob---def local_correlation_with_flow(feature0, feature1,- flow,- local_radius,- padding_mode='zeros',- dilation=1,- ):- b, c, h, w = feature0.size()- coords_init = coords_grid(b, h, w).to(feature0.device) # [B, 2, H, W]- coords = coords_init.view(b, 2, -1).permute(0, 2, 1) # [B, H*W, 2]-- local_h = 2 * local_radius + 1- local_w = 2 * local_radius + 1-- window_grid = generate_window_grid(-local_radius, local_radius,- -local_radius, local_radius,- local_h, local_w, device=feature0.device) # [2R+1, 2R+1, 2]- window_grid = window_grid.reshape(-1, 2).repeat(b, 1, 1, 1) # [B, 1, (2R+1)^2, 2]- sample_coords = coords.unsqueeze(-2) + window_grid * dilation # [B, H*W, (2R+1)^2, 2]-- # flow can be zero when using features after transformer- if not isinstance(flow, float):- sample_coords = sample_coords + flow.view(- b, 2, -1).permute(0, 2, 1).unsqueeze(-2) # [B, H*W, (2R+1)^2, 2]- else:- assert flow == 0.-- # normalize coordinates to [-1, 1]- sample_coords_norm = normalize_coords(sample_coords, h, w) # [-1, 1]- window_feature = F.grid_sample(feature1, sample_coords_norm,- padding_mode=padding_mode, align_corners=True- ).permute(0, 2, 1, 3) # [B, H*W, C, (2R+1)^2]- feature0_view = feature0.permute(0, 2, 3, 1).view(b, h * w, 1, c) # [B, H*W, 1, C]-- corr = torch.matmul(feature0_view, window_feature).view(b, h * w, -1) / (c ** 0.5) # [B, H*W, (2R+1)^2]-- corr = corr.view(b, h, w, -1).permute(0, 3, 1, 2).contiguous() # [B, (2R+1)^2, H, W]-- return corr---def global_correlation_softmax_stereo(feature0, feature1,- ):- # global correlation on horizontal direction- b, c, h, w = feature0.shape-- x_grid = torch.linspace(0, w - 1, w, device=feature0.device) # [W]-- feature0 = feature0.permute(0, 2, 3, 1) # [B, H, W, C]- feature1 = feature1.permute(0, 2, 1, 3) # [B, H, C, W]-- correlation = torch.matmul(feature0, feature1) / (c ** 0.5) # [B, H, W, W]-- # mask subsequent positions to make disparity positive- mask = torch.triu(torch.ones((w, w)), diagonal=1).type_as(feature0) # [W, W]- valid_mask = (mask == 0).unsqueeze(0).unsqueeze(0).repeat(b, h, 1, 1) # [B, H, W, W]-- correlation[~valid_mask] = -1e9-- prob = F.softmax(correlation, dim=-1) # [B, H, W, W]-- correspondence = (x_grid.view(1, 1, 1, w) * prob).sum(-1) # [B, H, W]-- # NOTE: unlike flow, disparity is typically positive- disparity = x_grid.view(1, 1, w).repeat(b, h, 1) - correspondence # [B, H, W]-- return disparity.unsqueeze(1), prob # feature resolution---def local_correlation_softmax_stereo(feature0, feature1, local_radius,- ):- b, c, h, w = feature0.size()- coords_init = coords_grid(b, h, w).to(feature0.device) # [B, 2, H, W]- coords = coords_init.view(b, 2, -1).permute(0, 2, 1).contiguous() # [B, H*W, 2]-- local_h = 1- local_w = 2 * local_radius + 1-- window_grid = generate_window_grid(0, 0,- -local_radius, local_radius,- local_h, local_w, device=feature0.device) # [1, 2R+1, 2]- window_grid = window_grid.reshape(-1, 2).repeat(b, 1, 1, 1) # [B, 1, (2R+1), 2]- sample_coords = coords.unsqueeze(-2) + window_grid # [B, H*W, (2R+1), 2]-- sample_coords_softmax = sample_coords-- # exclude coords that are out of image space- valid_x = (sample_coords[:, :, :, 0] >= 0) & (sample_coords[:, :, :, 0] < w) # [B, H*W, (2R+1)^2]- valid_y = (sample_coords[:, :, :, 1] >= 0) & (sample_coords[:, :, :, 1] < h) # [B, H*W, (2R+1)^2]-- valid = valid_x & valid_y # [B, H*W, (2R+1)^2], used to mask out invalid values when softmax-- # normalize coordinates to [-1, 1]- sample_coords_norm = normalize_coords(sample_coords, h, w) # [-1, 1]- window_feature = F.grid_sample(feature1, sample_coords_norm,- padding_mode='zeros', align_corners=True- ).permute(0, 2, 1, 3) # [B, H*W, C, (2R+1)]- feature0_view = feature0.permute(0, 2, 3, 1).contiguous().view(b, h * w, 1, c) # [B, H*W, 1, C]-- corr = torch.matmul(feature0_view, window_feature).view(b, h * w, -1) / (c ** 0.5) # [B, H*W, (2R+1)]-- # mask invalid locations- corr[~valid] = -1e9-- prob = F.softmax(corr, -1) # [B, H*W, (2R+1)]-- correspondence = torch.matmul(prob.unsqueeze(-2),- sample_coords_softmax).squeeze(-2).view(- b, h, w, 2).permute(0, 3, 1, 2).contiguous() # [B, 2, H, W]-- flow = correspondence - coords_init # flow at feature resolution- match_prob = prob-- flow_x = -flow[:, :1] # [B, 1, H, W]-- return flow_x, match_prob---def correlation_softmax_depth(feature0, feature1,- intrinsics,- pose,- depth_candidates,- depth_from_argmax=False,- pred_bidir_depth=False,- ):- b, c, h, w = feature0.size()- assert depth_candidates.dim() == 4 # [B, D, H, W]- scale_factor = c ** 0.5-- if pred_bidir_depth:- feature0, feature1 = torch.cat((feature0, feature1), dim=0), torch.cat((feature1, feature0), dim=0)- intrinsics = intrinsics.repeat(2, 1, 1)- pose = torch.cat((pose, torch.inverse(pose)), dim=0)- depth_candidates = depth_candidates.repeat(2, 1, 1, 1)-- # depth candidates are actually inverse depth- warped_feature1 = warp_with_pose_depth_candidates(feature1, intrinsics, pose,- 1. / depth_candidates,- ) # [B, C, D, H, W]-- correlation = (feature0.unsqueeze(2) * warped_feature1).sum(1) / scale_factor # [B, D, H, W]-- match_prob = F.softmax(correlation, dim=1) # [B, D, H, W]-- # for cross-task transfer (flow -> depth), extract depth with argmax at test time- if depth_from_argmax:- index = torch.argmax(match_prob, dim=1, keepdim=True)- depth = torch.gather(depth_candidates, dim=1, index=index)- else:- depth = (match_prob * depth_candidates).sum(dim=1, keepdim=True) # [B, 1, H, W]-- return depth, match_prob---def warp_with_pose_depth_candidates(feature1, intrinsics, pose, depth,- clamp_min_depth=1e-3,- ):- """- feature1: [B, C, H, W]- intrinsics: [B, 3, 3]- pose: [B, 4, 4]- depth: [B, D, H, W]- """-- assert intrinsics.size(1) == intrinsics.size(2) == 3- assert pose.size(1) == pose.size(2) == 4- assert depth.dim() == 4-- b, d, h, w = depth.size()- c = feature1.size(1)-- with torch.no_grad():- # pixel coordinates- grid = coords_grid(b, h, w, homogeneous=True, device=depth.device) # [B, 3, H, W]- # back project to 3D and transform viewpoint- points = torch.inverse(intrinsics).bmm(grid.view(b, 3, -1)) # [B, 3, H*W]- points = torch.bmm(pose[:, :3, :3], points).unsqueeze(2).repeat(- 1, 1, d, 1) * depth.view(b, 1, d, h * w) # [B, 3, D, H*W]- points = points + pose[:, :3, -1:].unsqueeze(-1) # [B, 3, D, H*W]- # reproject to 2D image plane- points = torch.bmm(intrinsics, points.view(b, 3, -1)).view(b, 3, d, h * w) # [B, 3, D, H*W]- pixel_coords = points[:, :2] / points[:, -1:].clamp(min=clamp_min_depth) # [B, 2, D, H*W]-- # normalize to [-1, 1]- x_grid = 2 * pixel_coords[:, 0] / (w - 1) - 1- y_grid = 2 * pixel_coords[:, 1] / (h - 1) - 1-- grid = torch.stack([x_grid, y_grid], dim=-1) # [B, D, H*W, 2]-- # sample features- warped_feature = F.grid_sample(feature1, grid.view(b, d * h, w, 2), mode='bilinear',- padding_mode='zeros',- align_corners=True).view(b, c, d, h, w) # [B, C, D, H, W]-- return warped_feature+import torch+import torch.nn.functional as F++from .geometry import coords_grid, generate_window_grid, normalize_coords+++def global_correlation_softmax(+ feature0,+ feature1,+ pred_bidir_flow=False,+):+ # global correlation+ b, c, h, w = feature0.shape+ feature0 = feature0.view(b, c, -1).permute(0, 2, 1) # [B, H*W, C]+ feature1 = feature1.view(b, c, -1) # [B, C, H*W]++ correlation = torch.matmul(feature0, feature1).view(b, h, w, h, w) / (c**0.5) # [B, H, W, H, W]++ # flow from softmax+ init_grid = coords_grid(b, h, w).to(correlation.device) # [B, 2, H, W]+ grid = init_grid.view(b, 2, -1).permute(0, 2, 1) # [B, H*W, 2]++ correlation = correlation.view(b, h * w, h * w) # [B, H*W, H*W]++ if pred_bidir_flow:+ correlation = torch.cat((correlation, correlation.permute(0, 2, 1)), dim=0) # [2*B, H*W, H*W]+ init_grid = init_grid.repeat(2, 1, 1, 1) # [2*B, 2, H, W]+ grid = grid.repeat(2, 1, 1) # [2*B, H*W, 2]+ b = b * 2++ prob = F.softmax(correlation, dim=-1) # [B, H*W, H*W]++ correspondence = torch.matmul(prob, grid).view(b, h, w, 2).permute(0, 3, 1, 2) # [B, 2, H, W]++ # when predicting bidirectional flow, flow is the concatenation of forward flow and backward flow+ flow = correspondence - init_grid++ return flow, prob+++def local_correlation_softmax(+ feature0,+ feature1,+ local_radius,+ padding_mode="zeros",+):+ b, c, h, w = feature0.size()+ coords_init = coords_grid(b, h, w).to(feature0.device) # [B, 2, H, W]+ coords = coords_init.view(b, 2, -1).permute(0, 2, 1) # [B, H*W, 2]++ local_h = 2 * local_radius + 1+ local_w = 2 * local_radius + 1++ window_grid = generate_window_grid(+ -local_radius, local_radius, -local_radius, local_radius, local_h, local_w, device=feature0.device+ ) # [2R+1, 2R+1, 2]+ window_grid = window_grid.reshape(-1, 2).repeat(b, 1, 1, 1) # [B, 1, (2R+1)^2, 2]+ sample_coords = coords.unsqueeze(-2) + window_grid # [B, H*W, (2R+1)^2, 2]++ sample_coords_softmax = sample_coords++ # exclude coords that are out of image space+ valid_x = (sample_coords[:, :, :, 0] >= 0) & (sample_coords[:, :, :, 0] < w) # [B, H*W, (2R+1)^2]+ valid_y = (sample_coords[:, :, :, 1] >= 0) & (sample_coords[:, :, :, 1] < h) # [B, H*W, (2R+1)^2]++ valid = valid_x & valid_y # [B, H*W, (2R+1)^2], used to mask out invalid values when softmax++ # normalize coordinates to [-1, 1]+ sample_coords_norm = normalize_coords(sample_coords, h, w) # [-1, 1]+ window_feature = F.grid_sample(feature1, sample_coords_norm, padding_mode=padding_mode, align_corners=True).permute(+ 0, 2, 1, 3+ ) # [B, H*W, C, (2R+1)^2]+ feature0_view = feature0.permute(0, 2, 3, 1).view(b, h * w, 1, c) # [B, H*W, 1, C]++ corr = torch.matmul(feature0_view, window_feature).view(b, h * w, -1) / (c**0.5) # [B, H*W, (2R+1)^2]++ # mask invalid locations+ corr[~valid] = -1e9++ prob = F.softmax(corr, -1) # [B, H*W, (2R+1)^2]++ correspondence = (+ torch.matmul(prob.unsqueeze(-2), sample_coords_softmax).squeeze(-2).view(b, h, w, 2).permute(0, 3, 1, 2)+ ) # [B, 2, H, W]++ flow = correspondence - coords_init+ match_prob = prob++ return flow, match_prob+++def local_correlation_with_flow(+ feature0,+ feature1,+ flow,+ local_radius,+ padding_mode="zeros",+ dilation=1,+):+ b, c, h, w = feature0.size()+ coords_init = coords_grid(b, h, w).to(feature0.device) # [B, 2, H, W]+ coords = coords_init.view(b, 2, -1).permute(0, 2, 1) # [B, H*W, 2]++ local_h = 2 * local_radius + 1+ local_w = 2 * local_radius + 1++ window_grid = generate_window_grid(+ -local_radius, local_radius, -local_radius, local_radius, local_h, local_w, device=feature0.device+ ) # [2R+1, 2R+1, 2]+ window_grid = window_grid.reshape(-1, 2).repeat(b, 1, 1, 1) # [B, 1, (2R+1)^2, 2]+ sample_coords = coords.unsqueeze(-2) + window_grid * dilation # [B, H*W, (2R+1)^2, 2]++ # flow can be zero when using features after transformer+ if not isinstance(flow, float):+ sample_coords = sample_coords + flow.view(b, 2, -1).permute(0, 2, 1).unsqueeze(-2) # [B, H*W, (2R+1)^2, 2]+ else:+ assert flow == 0.0++ # normalize coordinates to [-1, 1]+ sample_coords_norm = normalize_coords(sample_coords, h, w) # [-1, 1]+ window_feature = F.grid_sample(feature1, sample_coords_norm, padding_mode=padding_mode, align_corners=True).permute(+ 0, 2, 1, 3+ ) # [B, H*W, C, (2R+1)^2]+ feature0_view = feature0.permute(0, 2, 3, 1).view(b, h * w, 1, c) # [B, H*W, 1, C]++ corr = torch.matmul(feature0_view, window_feature).view(b, h * w, -1) / (c**0.5) # [B, H*W, (2R+1)^2]++ corr = corr.view(b, h, w, -1).permute(0, 3, 1, 2).contiguous() # [B, (2R+1)^2, H, W]++ return corr+++def global_correlation_softmax_stereo(+ feature0,+ feature1,+):+ # global correlation on horizontal direction+ b, c, h, w = feature0.shape++ x_grid = torch.linspace(0, w - 1, w, device=feature0.device) # [W]++ feature0 = feature0.permute(0, 2, 3, 1) # [B, H, W, C]+ feature1 = feature1.permute(0, 2, 1, 3) # [B, H, C, W]++ correlation = torch.matmul(feature0, feature1) / (c**0.5) # [B, H, W, W]++ # mask subsequent positions to make disparity positive+ mask = torch.triu(torch.ones((w, w)), diagonal=1).type_as(feature0) # [W, W]+ valid_mask = (mask == 0).unsqueeze(0).unsqueeze(0).repeat(b, h, 1, 1) # [B, H, W, W]++ correlation[~valid_mask] = -1e9++ prob = F.softmax(correlation, dim=-1) # [B, H, W, W]++ correspondence = (x_grid.view(1, 1, 1, w) * prob).sum(-1) # [B, H, W]++ # NOTE: unlike flow, disparity is typically positive+ disparity = x_grid.view(1, 1, w).repeat(b, h, 1) - correspondence # [B, H, W]++ return disparity.unsqueeze(1), prob # feature resolution+++def local_correlation_softmax_stereo(+ feature0,+ feature1,+ local_radius,+):+ b, c, h, w = feature0.size()+ coords_init = coords_grid(b, h, w).to(feature0.device) # [B, 2, H, W]+ coords = coords_init.view(b, 2, -1).permute(0, 2, 1).contiguous() # [B, H*W, 2]++ local_h = 1+ local_w = 2 * local_radius + 1++ window_grid = generate_window_grid(+ 0, 0, -local_radius, local_radius, local_h, local_w, device=feature0.device+ ) # [1, 2R+1, 2]+ window_grid = window_grid.reshape(-1, 2).repeat(b, 1, 1, 1) # [B, 1, (2R+1), 2]+ sample_coords = coords.unsqueeze(-2) + window_grid # [B, H*W, (2R+1), 2]++ sample_coords_softmax = sample_coords++ # exclude coords that are out of image space+ valid_x = (sample_coords[:, :, :, 0] >= 0) & (sample_coords[:, :, :, 0] < w) # [B, H*W, (2R+1)^2]+ valid_y = (sample_coords[:, :, :, 1] >= 0) & (sample_coords[:, :, :, 1] < h) # [B, H*W, (2R+1)^2]++ valid = valid_x & valid_y # [B, H*W, (2R+1)^2], used to mask out invalid values when softmax++ # normalize coordinates to [-1, 1]+ sample_coords_norm = normalize_coords(sample_coords, h, w) # [-1, 1]+ window_feature = F.grid_sample(feature1, sample_coords_norm, padding_mode="zeros", align_corners=True).permute(+ 0, 2, 1, 3+ ) # [B, H*W, C, (2R+1)]+ feature0_view = feature0.permute(0, 2, 3, 1).contiguous().view(b, h * w, 1, c) # [B, H*W, 1, C]++ corr = torch.matmul(feature0_view, window_feature).view(b, h * w, -1) / (c**0.5) # [B, H*W, (2R+1)]++ # mask invalid locations+ corr[~valid] = -1e9++ prob = F.softmax(corr, -1) # [B, H*W, (2R+1)]++ correspondence = (+ torch.matmul(prob.unsqueeze(-2), sample_coords_softmax)+ .squeeze(-2)+ .view(b, h, w, 2)+ .permute(0, 3, 1, 2)+ .contiguous()+ ) # [B, 2, H, W]++ flow = correspondence - coords_init # flow at feature resolution+ match_prob = prob++ flow_x = -flow[:, :1] # [B, 1, H, W]++ return flow_x, match_prob+++def correlation_softmax_depth(+ feature0,+ feature1,+ intrinsics,+ pose,+ depth_candidates,+ depth_from_argmax=False,+ pred_bidir_depth=False,+):+ b, c, h, w = feature0.size()+ assert depth_candidates.dim() == 4 # [B, D, H, W]+ scale_factor = c**0.5++ if pred_bidir_depth:+ feature0, feature1 = torch.cat((feature0, feature1), dim=0), torch.cat((feature1, feature0), dim=0)+ intrinsics = intrinsics.repeat(2, 1, 1)+ pose = torch.cat((pose, torch.inverse(pose)), dim=0)+ depth_candidates = depth_candidates.repeat(2, 1, 1, 1)++ # depth candidates are actually inverse depth+ warped_feature1 = warp_with_pose_depth_candidates(+ feature1,+ intrinsics,+ pose,+ 1.0 / depth_candidates,+ ) # [B, C, D, H, W]++ correlation = (feature0.unsqueeze(2) * warped_feature1).sum(1) / scale_factor # [B, D, H, W]++ match_prob = F.softmax(correlation, dim=1) # [B, D, H, W]++ # for cross-task transfer (flow -> depth), extract depth with argmax at test time+ if depth_from_argmax:+ index = torch.argmax(match_prob, dim=1, keepdim=True)+ depth = torch.gather(depth_candidates, dim=1, index=index)+ else:+ depth = (match_prob * depth_candidates).sum(dim=1, keepdim=True) # [B, 1, H, W]++ return depth, match_prob+++def warp_with_pose_depth_candidates(+ feature1,+ intrinsics,+ pose,+ depth,+ clamp_min_depth=1e-3,+):+ """+ feature1: [B, C, H, W]+ intrinsics: [B, 3, 3]+ pose: [B, 4, 4]+ depth: [B, D, H, W]+ """++ assert intrinsics.size(1) == intrinsics.size(2) == 3+ assert pose.size(1) == pose.size(2) == 4+ assert depth.dim() == 4++ b, d, h, w = depth.size()+ c = feature1.size(1)++ with torch.no_grad():+ # pixel coordinates+ grid = coords_grid(b, h, w, homogeneous=True, device=depth.device) # [B, 3, H, W]+ # back project to 3D and transform viewpoint+ points = torch.inverse(intrinsics).bmm(grid.view(b, 3, -1)) # [B, 3, H*W]+ points = torch.bmm(pose[:, :3, :3], points).unsqueeze(2).repeat(1, 1, d, 1) * depth.view(+ b, 1, d, h * w+ ) # [B, 3, D, H*W]+ points = points + pose[:, :3, -1:].unsqueeze(-1) # [B, 3, D, H*W]+ # reproject to 2D image plane+ points = torch.bmm(intrinsics, points.view(b, 3, -1)).view(b, 3, d, h * w) # [B, 3, D, H*W]+ pixel_coords = points[:, :2] / points[:, -1:].clamp(min=clamp_min_depth) # [B, 2, D, H*W]++ # normalize to [-1, 1]+ x_grid = 2 * pixel_coords[:, 0] / (w - 1) - 1+ y_grid = 2 * pixel_coords[:, 1] / (h - 1) - 1++ grid = torch.stack([x_grid, y_grid], dim=-1) # [B, D, H*W, 2]++ # sample features+ warped_feature = F.grid_sample(+ feature1, grid.view(b, d * h, w, 2), mode="bilinear", padding_mode="zeros", align_corners=True+ ).view(+ b, c, d, h, w+ ) # [B, C, D, H, W]++ return warped_featurediff --git a/tools/scoring/optical_flow/unimatch/position.py b/tools/scoring/optical_flow/unimatch/position.pyindex 42435d0..619f356 100644--- a/tools/scoring/optical_flow/unimatch/position.py+++ b/tools/scoring/optical_flow/unimatch/position.py@@ -1,46 +1,47 @@-# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved-# https://github.com/facebookresearch/detr/blob/main/models/position_encoding.py--import torch-import torch.nn as nn-import math---class PositionEmbeddingSine(nn.Module):- """- This is a more standard version of the position embedding, very similar to the one- used by the Attention is all you need paper, generalized to work on images.- """-- def __init__(self, num_pos_feats=64, temperature=10000, normalize=True, scale=None):- super().__init__()- self.num_pos_feats = num_pos_feats- self.temperature = temperature- self.normalize = normalize- if scale is not None and normalize is False:- raise ValueError("normalize should be True if scale is passed")- if scale is None:- scale = 2 * math.pi- self.scale = scale-- def forward(self, x):- # x = tensor_list.tensors # [B, C, H, W]- # mask = tensor_list.mask # [B, H, W], input with padding, valid as 0- b, c, h, w = x.size()- mask = torch.ones((b, h, w), device=x.device) # [B, H, W]- y_embed = mask.cumsum(1, dtype=torch.float32)- x_embed = mask.cumsum(2, dtype=torch.float32)- if self.normalize:- eps = 1e-6- y_embed = y_embed / (y_embed[:, -1:, :] + eps) * self.scale- x_embed = x_embed / (x_embed[:, :, -1:] + eps) * self.scale-- dim_t = torch.arange(self.num_pos_feats, dtype=torch.float32, device=x.device)- dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats)-- pos_x = x_embed[:, :, :, None] / dim_t- pos_y = y_embed[:, :, :, None] / dim_t- pos_x = torch.stack((pos_x[:, :, :, 0::2].sin(), pos_x[:, :, :, 1::2].cos()), dim=4).flatten(3)- pos_y = torch.stack((pos_y[:, :, :, 0::2].sin(), pos_y[:, :, :, 1::2].cos()), dim=4).flatten(3)- pos = torch.cat((pos_y, pos_x), dim=3).permute(0, 3, 1, 2)- return pos+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved+# https://github.com/facebookresearch/detr/blob/main/models/position_encoding.py++import math++import torch+import torch.nn as nn+++class PositionEmbeddingSine(nn.Module):+ """+ This is a more standard version of the position embedding, very similar to the one+ used by the Attention is all you need paper, generalized to work on images.+ """++ def __init__(self, num_pos_feats=64, temperature=10000, normalize=True, scale=None):+ super().__init__()+ self.num_pos_feats = num_pos_feats+ self.temperature = temperature+ self.normalize = normalize+ if scale is not None and normalize is False:+ raise ValueError("normalize should be True if scale is passed")+ if scale is None:+ scale = 2 * math.pi+ self.scale = scale++ def forward(self, x):+ # x = tensor_list.tensors # [B, C, H, W]+ # mask = tensor_list.mask # [B, H, W], input with padding, valid as 0+ b, c, h, w = x.size()+ mask = torch.ones((b, h, w), device=x.device) # [B, H, W]+ y_embed = mask.cumsum(1, dtype=torch.float32)+ x_embed = mask.cumsum(2, dtype=torch.float32)+ if self.normalize:+ eps = 1e-6+ y_embed = y_embed / (y_embed[:, -1:, :] + eps) * self.scale+ x_embed = x_embed / (x_embed[:, :, -1:] + eps) * self.scale++ dim_t = torch.arange(self.num_pos_feats, dtype=torch.float32, device=x.device)+ dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats)++ pos_x = x_embed[:, :, :, None] / dim_t+ pos_y = y_embed[:, :, :, None] / dim_t+ pos_x = torch.stack((pos_x[:, :, :, 0::2].sin(), pos_x[:, :, :, 1::2].cos()), dim=4).flatten(3)+ pos_y = torch.stack((pos_y[:, :, :, 0::2].sin(), pos_y[:, :, :, 1::2].cos()), dim=4).flatten(3)+ pos = torch.cat((pos_y, pos_x), dim=3).permute(0, 3, 1, 2)+ return posdiff --git a/tools/scoring/optical_flow/unimatch/reg_refine.py b/tools/scoring/optical_flow/unimatch/reg_refine.pyindex 310807b..965f4ca 100644--- a/tools/scoring/optical_flow/unimatch/reg_refine.py+++ b/tools/scoring/optical_flow/unimatch/reg_refine.py@@ -1,119 +1,133 @@-import torch-import torch.nn as nn-import torch.nn.functional as F---class FlowHead(nn.Module):- def __init__(self, input_dim=128, hidden_dim=256,- out_dim=2,- ):- super(FlowHead, self).__init__()-- self.conv1 = nn.Conv2d(input_dim, hidden_dim, 3, padding=1)- self.conv2 = nn.Conv2d(hidden_dim, out_dim, 3, padding=1)- self.relu = nn.ReLU(inplace=True)-- def forward(self, x):- out = self.conv2(self.relu(self.conv1(x)))-- return out---class SepConvGRU(nn.Module):- def __init__(self, hidden_dim=128, input_dim=192 + 128,- kernel_size=5,- ):- padding = (kernel_size - 1) // 2-- super(SepConvGRU, self).__init__()- self.convz1 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (1, kernel_size), padding=(0, padding))- self.convr1 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (1, kernel_size), padding=(0, padding))- self.convq1 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (1, kernel_size), padding=(0, padding))-- self.convz2 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (kernel_size, 1), padding=(padding, 0))- self.convr2 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (kernel_size, 1), padding=(padding, 0))- self.convq2 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (kernel_size, 1), padding=(padding, 0))-- def forward(self, h, x):- # horizontal- hx = torch.cat([h, x], dim=1)- z = torch.sigmoid(self.convz1(hx))- r = torch.sigmoid(self.convr1(hx))- q = torch.tanh(self.convq1(torch.cat([r * h, x], dim=1)))- h = (1 - z) * h + z * q-- # vertical- hx = torch.cat([h, x], dim=1)- z = torch.sigmoid(self.convz2(hx))- r = torch.sigmoid(self.convr2(hx))- q = torch.tanh(self.convq2(torch.cat([r * h, x], dim=1)))- h = (1 - z) * h + z * q-- return h---class BasicMotionEncoder(nn.Module):- def __init__(self, corr_channels=324,- flow_channels=2,- ):- super(BasicMotionEncoder, self).__init__()-- self.convc1 = nn.Conv2d(corr_channels, 256, 1, padding=0)- self.convc2 = nn.Conv2d(256, 192, 3, padding=1)- self.convf1 = nn.Conv2d(flow_channels, 128, 7, padding=3)- self.convf2 = nn.Conv2d(128, 64, 3, padding=1)- self.conv = nn.Conv2d(64 + 192, 128 - flow_channels, 3, padding=1)-- def forward(self, flow, corr):- cor = F.relu(self.convc1(corr))- cor = F.relu(self.convc2(cor))- flo = F.relu(self.convf1(flow))- flo = F.relu(self.convf2(flo))-- cor_flo = torch.cat([cor, flo], dim=1)- out = F.relu(self.conv(cor_flo))- return torch.cat([out, flow], dim=1)---class BasicUpdateBlock(nn.Module):- def __init__(self, corr_channels=324,- hidden_dim=128,- context_dim=128,- downsample_factor=8,- flow_dim=2,- bilinear_up=False,- ):- super(BasicUpdateBlock, self).__init__()-- self.encoder = BasicMotionEncoder(corr_channels=corr_channels,- flow_channels=flow_dim,- )-- self.gru = SepConvGRU(hidden_dim=hidden_dim, input_dim=context_dim + hidden_dim)-- self.flow_head = FlowHead(hidden_dim, hidden_dim=256,- out_dim=flow_dim,- )-- if bilinear_up:- self.mask = None- else:- self.mask = nn.Sequential(- nn.Conv2d(hidden_dim, 256, 3, padding=1),- nn.ReLU(inplace=True),- nn.Conv2d(256, downsample_factor ** 2 * 9, 1, padding=0))-- def forward(self, net, inp, corr, flow):- motion_features = self.encoder(flow, corr)-- inp = torch.cat([inp, motion_features], dim=1)-- net = self.gru(net, inp)- delta_flow = self.flow_head(net)-- if self.mask is not None:- mask = self.mask(net)- else:- mask = None-- return net, mask, delta_flow+import torch+import torch.nn as nn+import torch.nn.functional as F+++class FlowHead(nn.Module):+ def __init__(+ self,+ input_dim=128,+ hidden_dim=256,+ out_dim=2,+ ):+ super(FlowHead, self).__init__()++ self.conv1 = nn.Conv2d(input_dim, hidden_dim, 3, padding=1)+ self.conv2 = nn.Conv2d(hidden_dim, out_dim, 3, padding=1)+ self.relu = nn.ReLU(inplace=True)++ def forward(self, x):+ out = self.conv2(self.relu(self.conv1(x)))++ return out+++class SepConvGRU(nn.Module):+ def __init__(+ self,+ hidden_dim=128,+ input_dim=192 + 128,+ kernel_size=5,+ ):+ padding = (kernel_size - 1) // 2++ super(SepConvGRU, self).__init__()+ self.convz1 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (1, kernel_size), padding=(0, padding))+ self.convr1 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (1, kernel_size), padding=(0, padding))+ self.convq1 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (1, kernel_size), padding=(0, padding))++ self.convz2 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (kernel_size, 1), padding=(padding, 0))+ self.convr2 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (kernel_size, 1), padding=(padding, 0))+ self.convq2 = nn.Conv2d(hidden_dim + input_dim, hidden_dim, (kernel_size, 1), padding=(padding, 0))++ def forward(self, h, x):+ # horizontal+ hx = torch.cat([h, x], dim=1)+ z = torch.sigmoid(self.convz1(hx))+ r = torch.sigmoid(self.convr1(hx))+ q = torch.tanh(self.convq1(torch.cat([r * h, x], dim=1)))+ h = (1 - z) * h + z * q++ # vertical+ hx = torch.cat([h, x], dim=1)+ z = torch.sigmoid(self.convz2(hx))+ r = torch.sigmoid(self.convr2(hx))+ q = torch.tanh(self.convq2(torch.cat([r * h, x], dim=1)))+ h = (1 - z) * h + z * q++ return h+++class BasicMotionEncoder(nn.Module):+ def __init__(+ self,+ corr_channels=324,+ flow_channels=2,+ ):+ super(BasicMotionEncoder, self).__init__()++ self.convc1 = nn.Conv2d(corr_channels, 256, 1, padding=0)+ self.convc2 = nn.Conv2d(256, 192, 3, padding=1)+ self.convf1 = nn.Conv2d(flow_channels, 128, 7, padding=3)+ self.convf2 = nn.Conv2d(128, 64, 3, padding=1)+ self.conv = nn.Conv2d(64 + 192, 128 - flow_channels, 3, padding=1)++ def forward(self, flow, corr):+ cor = F.relu(self.convc1(corr))+ cor = F.relu(self.convc2(cor))+ flo = F.relu(self.convf1(flow))+ flo = F.relu(self.convf2(flo))++ cor_flo = torch.cat([cor, flo], dim=1)+ out = F.relu(self.conv(cor_flo))+ return torch.cat([out, flow], dim=1)+++class BasicUpdateBlock(nn.Module):+ def __init__(+ self,+ corr_channels=324,+ hidden_dim=128,+ context_dim=128,+ downsample_factor=8,+ flow_dim=2,+ bilinear_up=False,+ ):+ super(BasicUpdateBlock, self).__init__()++ self.encoder = BasicMotionEncoder(+ corr_channels=corr_channels,+ flow_channels=flow_dim,+ )++ self.gru = SepConvGRU(hidden_dim=hidden_dim, input_dim=context_dim + hidden_dim)++ self.flow_head = FlowHead(+ hidden_dim,+ hidden_dim=256,+ out_dim=flow_dim,+ )++ if bilinear_up:+ self.mask = None+ else:+ self.mask = nn.Sequential(+ nn.Conv2d(hidden_dim, 256, 3, padding=1),+ nn.ReLU(inplace=True),+ nn.Conv2d(256, downsample_factor**2 * 9, 1, padding=0),+ )++ def forward(self, net, inp, corr, flow):+ motion_features = self.encoder(flow, corr)++ inp = torch.cat([inp, motion_features], dim=1)++ net = self.gru(net, inp)+ delta_flow = self.flow_head(net)++ if self.mask is not None:+ mask = self.mask(net)+ else:+ mask = None++ return net, mask, delta_flowdiff --git a/tools/scoring/optical_flow/unimatch/transformer.py b/tools/scoring/optical_flow/unimatch/transformer.pyindex a93660c..7fdffd1 100644--- a/tools/scoring/optical_flow/unimatch/transformer.py+++ b/tools/scoring/optical_flow/unimatch/transformer.py@@ -1,294 +1,339 @@-import torch-import torch.nn as nn--from .attention import (single_head_full_attention, single_head_split_window_attention,- single_head_full_attention_1d, single_head_split_window_attention_1d)-from .utils import generate_shift_window_attn_mask, generate_shift_window_attn_mask_1d---class TransformerLayer(nn.Module):- def __init__(self,- d_model=128,- nhead=1,- no_ffn=False,- ffn_dim_expansion=4,- ):- super(TransformerLayer, self).__init__()-- self.dim = d_model- self.nhead = nhead- self.no_ffn = no_ffn-- # multi-head attention- self.q_proj = nn.Linear(d_model, d_model, bias=False)- self.k_proj = nn.Linear(d_model, d_model, bias=False)- self.v_proj = nn.Linear(d_model, d_model, bias=False)-- self.merge = nn.Linear(d_model, d_model, bias=False)-- self.norm1 = nn.LayerNorm(d_model)-- # no ffn after self-attn, with ffn after cross-attn- if not self.no_ffn:- in_channels = d_model * 2- self.mlp = nn.Sequential(- nn.Linear(in_channels, in_channels * ffn_dim_expansion, bias=False),- nn.GELU(),- nn.Linear(in_channels * ffn_dim_expansion, d_model, bias=False),- )-- self.norm2 = nn.LayerNorm(d_model)-- def forward(self, source, target,- height=None,- width=None,- shifted_window_attn_mask=None,- shifted_window_attn_mask_1d=None,- attn_type='swin',- with_shift=False,- attn_num_splits=None,- ):- # source, target: [B, L, C]- query, key, value = source, target, target-- # for stereo: 2d attn in self-attn, 1d attn in cross-attn- is_self_attn = (query - key).abs().max() < 1e-6-- # single-head attention- query = self.q_proj(query) # [B, L, C]- key = self.k_proj(key) # [B, L, C]- value = self.v_proj(value) # [B, L, C]-- if attn_type == 'swin' and attn_num_splits > 1: # self, cross-attn: both swin 2d- if self.nhead > 1:- # we observe that multihead attention slows down the speed and increases the memory consumption- # without bringing obvious performance gains and thus the implementation is removed- raise NotImplementedError- else:- message = single_head_split_window_attention(query, key, value,- num_splits=attn_num_splits,- with_shift=with_shift,- h=height,- w=width,- attn_mask=shifted_window_attn_mask,- )-- elif attn_type == 'self_swin2d_cross_1d': # self-attn: swin 2d, cross-attn: full 1d- if self.nhead > 1:- raise NotImplementedError- else:- if is_self_attn:- if attn_num_splits > 1:- message = single_head_split_window_attention(query, key, value,- num_splits=attn_num_splits,- with_shift=with_shift,- h=height,- w=width,- attn_mask=shifted_window_attn_mask,- )- else:- # full 2d attn- message = single_head_full_attention(query, key, value) # [N, L, C]-- else:- # cross attn 1d- message = single_head_full_attention_1d(query, key, value,- h=height,- w=width,- )-- elif attn_type == 'self_swin2d_cross_swin1d': # self-attn: swin 2d, cross-attn: swin 1d- if self.nhead > 1:- raise NotImplementedError- else:- if is_self_attn:- if attn_num_splits > 1:- # self attn shift window- message = single_head_split_window_attention(query, key, value,- num_splits=attn_num_splits,- with_shift=with_shift,- h=height,- w=width,- attn_mask=shifted_window_attn_mask,- )- else:- # full 2d attn- message = single_head_full_attention(query, key, value) # [N, L, C]- else:- if attn_num_splits > 1:- assert shifted_window_attn_mask_1d is not None- # cross attn 1d shift- message = single_head_split_window_attention_1d(query, key, value,- num_splits=attn_num_splits,- with_shift=with_shift,- h=height,- w=width,- attn_mask=shifted_window_attn_mask_1d,- )- else:- message = single_head_full_attention_1d(query, key, value,- h=height,- w=width,- )-- else:- message = single_head_full_attention(query, key, value) # [B, L, C]-- message = self.merge(message) # [B, L, C]- message = self.norm1(message)-- if not self.no_ffn:- message = self.mlp(torch.cat([source, message], dim=-1))- message = self.norm2(message)-- return source + message---class TransformerBlock(nn.Module):- """self attention + cross attention + FFN"""-- def __init__(self,- d_model=128,- nhead=1,- ffn_dim_expansion=4,- ):- super(TransformerBlock, self).__init__()-- self.self_attn = TransformerLayer(d_model=d_model,- nhead=nhead,- no_ffn=True,- ffn_dim_expansion=ffn_dim_expansion,- )-- self.cross_attn_ffn = TransformerLayer(d_model=d_model,- nhead=nhead,- ffn_dim_expansion=ffn_dim_expansion,- )-- def forward(self, source, target,- height=None,- width=None,- shifted_window_attn_mask=None,- shifted_window_attn_mask_1d=None,- attn_type='swin',- with_shift=False,- attn_num_splits=None,- ):- # source, target: [B, L, C]-- # self attention- source = self.self_attn(source, source,- height=height,- width=width,- shifted_window_attn_mask=shifted_window_attn_mask,- attn_type=attn_type,- with_shift=with_shift,- attn_num_splits=attn_num_splits,- )-- # cross attention and ffn- source = self.cross_attn_ffn(source, target,- height=height,- width=width,- shifted_window_attn_mask=shifted_window_attn_mask,- shifted_window_attn_mask_1d=shifted_window_attn_mask_1d,- attn_type=attn_type,- with_shift=with_shift,- attn_num_splits=attn_num_splits,- )-- return source---class FeatureTransformer(nn.Module):- def __init__(self,- num_layers=6,- d_model=128,- nhead=1,- ffn_dim_expansion=4,- ):- super(FeatureTransformer, self).__init__()-- self.d_model = d_model- self.nhead = nhead-- self.layers = nn.ModuleList([- TransformerBlock(d_model=d_model,- nhead=nhead,- ffn_dim_expansion=ffn_dim_expansion,- )- for i in range(num_layers)])-- for p in self.parameters():- if p.dim() > 1:- nn.init.xavier_uniform_(p)-- def forward(self, feature0, feature1,- attn_type='swin',- attn_num_splits=None,- **kwargs,- ):-- b, c, h, w = feature0.shape- assert self.d_model == c-- feature0 = feature0.flatten(-2).permute(0, 2, 1) # [B, H*W, C]- feature1 = feature1.flatten(-2).permute(0, 2, 1) # [B, H*W, C]-- # 2d attention- if 'swin' in attn_type and attn_num_splits > 1:- # global and refine use different number of splits- window_size_h = h // attn_num_splits- window_size_w = w // attn_num_splits-- # compute attn mask once- shifted_window_attn_mask = generate_shift_window_attn_mask(- input_resolution=(h, w),- window_size_h=window_size_h,- window_size_w=window_size_w,- shift_size_h=window_size_h // 2,- shift_size_w=window_size_w // 2,- device=feature0.device,- ) # [K*K, H/K*W/K, H/K*W/K]- else:- shifted_window_attn_mask = None-- # 1d attention- if 'swin1d' in attn_type and attn_num_splits > 1:- window_size_w = w // attn_num_splits-- # compute attn mask once- shifted_window_attn_mask_1d = generate_shift_window_attn_mask_1d(- input_w=w,- window_size_w=window_size_w,- shift_size_w=window_size_w // 2,- device=feature0.device,- ) # [K, W/K, W/K]- else:- shifted_window_attn_mask_1d = None-- # concat feature0 and feature1 in batch dimension to compute in parallel- concat0 = torch.cat((feature0, feature1), dim=0) # [2B, H*W, C]- concat1 = torch.cat((feature1, feature0), dim=0) # [2B, H*W, C]-- for i, layer in enumerate(self.layers):- concat0 = layer(concat0, concat1,- height=h,- width=w,- attn_type=attn_type,- with_shift='swin' in attn_type and attn_num_splits > 1 and i % 2 == 1,- attn_num_splits=attn_num_splits,- shifted_window_attn_mask=shifted_window_attn_mask,- shifted_window_attn_mask_1d=shifted_window_attn_mask_1d,- )-- # update feature1- concat1 = torch.cat(concat0.chunk(chunks=2, dim=0)[::-1], dim=0)-- feature0, feature1 = concat0.chunk(chunks=2, dim=0) # [B, H*W, C]-- # reshape back- feature0 = feature0.view(b, h, w, c).permute(0, 3, 1, 2).contiguous() # [B, C, H, W]- feature1 = feature1.view(b, h, w, c).permute(0, 3, 1, 2).contiguous() # [B, C, H, W]-- return feature0, feature1+import torch+import torch.nn as nn++from .attention import (+ single_head_full_attention,+ single_head_full_attention_1d,+ single_head_split_window_attention,+ single_head_split_window_attention_1d,+)+from .utils import generate_shift_window_attn_mask, generate_shift_window_attn_mask_1d+++class TransformerLayer(nn.Module):+ def __init__(+ self,+ d_model=128,+ nhead=1,+ no_ffn=False,+ ffn_dim_expansion=4,+ ):+ super(TransformerLayer, self).__init__()++ self.dim = d_model+ self.nhead = nhead+ self.no_ffn = no_ffn++ # multi-head attention+ self.q_proj = nn.Linear(d_model, d_model, bias=False)+ self.k_proj = nn.Linear(d_model, d_model, bias=False)+ self.v_proj = nn.Linear(d_model, d_model, bias=False)++ self.merge = nn.Linear(d_model, d_model, bias=False)++ self.norm1 = nn.LayerNorm(d_model)++ # no ffn after self-attn, with ffn after cross-attn+ if not self.no_ffn:+ in_channels = d_model * 2+ self.mlp = nn.Sequential(+ nn.Linear(in_channels, in_channels * ffn_dim_expansion, bias=False),+ nn.GELU(),+ nn.Linear(in_channels * ffn_dim_expansion, d_model, bias=False),+ )++ self.norm2 = nn.LayerNorm(d_model)++ def forward(+ self,+ source,+ target,+ height=None,+ width=None,+ shifted_window_attn_mask=None,+ shifted_window_attn_mask_1d=None,+ attn_type="swin",+ with_shift=False,+ attn_num_splits=None,+ ):+ # source, target: [B, L, C]+ query, key, value = source, target, target++ # for stereo: 2d attn in self-attn, 1d attn in cross-attn+ is_self_attn = (query - key).abs().max() < 1e-6++ # single-head attention+ query = self.q_proj(query) # [B, L, C]+ key = self.k_proj(key) # [B, L, C]+ value = self.v_proj(value) # [B, L, C]++ if attn_type == "swin" and attn_num_splits > 1: # self, cross-attn: both swin 2d+ if self.nhead > 1:+ # we observe that multihead attention slows down the speed and increases the memory consumption+ # without bringing obvious performance gains and thus the implementation is removed+ raise NotImplementedError+ else:+ message = single_head_split_window_attention(+ query,+ key,+ value,+ num_splits=attn_num_splits,+ with_shift=with_shift,+ h=height,+ w=width,+ attn_mask=shifted_window_attn_mask,+ )++ elif attn_type == "self_swin2d_cross_1d": # self-attn: swin 2d, cross-attn: full 1d+ if self.nhead > 1:+ raise NotImplementedError+ else:+ if is_self_attn:+ if attn_num_splits > 1:+ message = single_head_split_window_attention(+ query,+ key,+ value,+ num_splits=attn_num_splits,+ with_shift=with_shift,+ h=height,+ w=width,+ attn_mask=shifted_window_attn_mask,+ )+ else:+ # full 2d attn+ message = single_head_full_attention(query, key, value) # [N, L, C]++ else:+ # cross attn 1d+ message = single_head_full_attention_1d(+ query,+ key,+ value,+ h=height,+ w=width,+ )++ elif attn_type == "self_swin2d_cross_swin1d": # self-attn: swin 2d, cross-attn: swin 1d+ if self.nhead > 1:+ raise NotImplementedError+ else:+ if is_self_attn:+ if attn_num_splits > 1:+ # self attn shift window+ message = single_head_split_window_attention(+ query,+ key,+ value,+ num_splits=attn_num_splits,+ with_shift=with_shift,+ h=height,+ w=width,+ attn_mask=shifted_window_attn_mask,+ )+ else:+ # full 2d attn+ message = single_head_full_attention(query, key, value) # [N, L, C]+ else:+ if attn_num_splits > 1:+ assert shifted_window_attn_mask_1d is not None+ # cross attn 1d shift+ message = single_head_split_window_attention_1d(+ query,+ key,+ value,+ num_splits=attn_num_splits,+ with_shift=with_shift,+ h=height,+ w=width,+ attn_mask=shifted_window_attn_mask_1d,+ )+ else:+ message = single_head_full_attention_1d(+ query,+ key,+ value,+ h=height,+ w=width,+ )++ else:+ message = single_head_full_attention(query, key, value) # [B, L, C]++ message = self.merge(message) # [B, L, C]+ message = self.norm1(message)++ if not self.no_ffn:+ message = self.mlp(torch.cat([source, message], dim=-1))+ message = self.norm2(message)++ return source + message+++class TransformerBlock(nn.Module):+ """self attention + cross attention + FFN"""++ def __init__(+ self,+ d_model=128,+ nhead=1,+ ffn_dim_expansion=4,+ ):+ super(TransformerBlock, self).__init__()++ self.self_attn = TransformerLayer(+ d_model=d_model,+ nhead=nhead,+ no_ffn=True,+ ffn_dim_expansion=ffn_dim_expansion,+ )++ self.cross_attn_ffn = TransformerLayer(+ d_model=d_model,+ nhead=nhead,+ ffn_dim_expansion=ffn_dim_expansion,+ )++ def forward(+ self,+ source,+ target,+ height=None,+ width=None,+ shifted_window_attn_mask=None,+ shifted_window_attn_mask_1d=None,+ attn_type="swin",+ with_shift=False,+ attn_num_splits=None,+ ):+ # source, target: [B, L, C]++ # self attention+ source = self.self_attn(+ source,+ source,+ height=height,+ width=width,+ shifted_window_attn_mask=shifted_window_attn_mask,+ attn_type=attn_type,+ with_shift=with_shift,+ attn_num_splits=attn_num_splits,+ )++ # cross attention and ffn+ source = self.cross_attn_ffn(+ source,+ target,+ height=height,+ width=width,+ shifted_window_attn_mask=shifted_window_attn_mask,+ shifted_window_attn_mask_1d=shifted_window_attn_mask_1d,+ attn_type=attn_type,+ with_shift=with_shift,+ attn_num_splits=attn_num_splits,+ )++ return source+++class FeatureTransformer(nn.Module):+ def __init__(+ self,+ num_layers=6,+ d_model=128,+ nhead=1,+ ffn_dim_expansion=4,+ ):+ super(FeatureTransformer, self).__init__()++ self.d_model = d_model+ self.nhead = nhead++ self.layers = nn.ModuleList(+ [+ TransformerBlock(+ d_model=d_model,+ nhead=nhead,+ ffn_dim_expansion=ffn_dim_expansion,+ )+ for i in range(num_layers)+ ]+ )++ for p in self.parameters():+ if p.dim() > 1:+ nn.init.xavier_uniform_(p)++ def forward(+ self,+ feature0,+ feature1,+ attn_type="swin",+ attn_num_splits=None,+ **kwargs,+ ):+ b, c, h, w = feature0.shape+ assert self.d_model == c++ feature0 = feature0.flatten(-2).permute(0, 2, 1) # [B, H*W, C]+ feature1 = feature1.flatten(-2).permute(0, 2, 1) # [B, H*W, C]++ # 2d attention+ if "swin" in attn_type and attn_num_splits > 1:+ # global and refine use different number of splits+ window_size_h = h // attn_num_splits+ window_size_w = w // attn_num_splits++ # compute attn mask once+ shifted_window_attn_mask = generate_shift_window_attn_mask(+ input_resolution=(h, w),+ window_size_h=window_size_h,+ window_size_w=window_size_w,+ shift_size_h=window_size_h // 2,+ shift_size_w=window_size_w // 2,+ device=feature0.device,+ ) # [K*K, H/K*W/K, H/K*W/K]+ else:+ shifted_window_attn_mask = None++ # 1d attention+ if "swin1d" in attn_type and attn_num_splits > 1:+ window_size_w = w // attn_num_splits++ # compute attn mask once+ shifted_window_attn_mask_1d = generate_shift_window_attn_mask_1d(+ input_w=w,+ window_size_w=window_size_w,+ shift_size_w=window_size_w // 2,+ device=feature0.device,+ ) # [K, W/K, W/K]+ else:+ shifted_window_attn_mask_1d = None++ # concat feature0 and feature1 in batch dimension to compute in parallel+ concat0 = torch.cat((feature0, feature1), dim=0) # [2B, H*W, C]+ concat1 = torch.cat((feature1, feature0), dim=0) # [2B, H*W, C]++ for i, layer in enumerate(self.layers):+ concat0 = layer(+ concat0,+ concat1,+ height=h,+ width=w,+ attn_type=attn_type,+ with_shift="swin" in attn_type and attn_num_splits > 1 and i % 2 == 1,+ attn_num_splits=attn_num_splits,+ shifted_window_attn_mask=shifted_window_attn_mask,+ shifted_window_attn_mask_1d=shifted_window_attn_mask_1d,+ )++ # update feature1+ concat1 = torch.cat(concat0.chunk(chunks=2, dim=0)[::-1], dim=0)++ feature0, feature1 = concat0.chunk(chunks=2, dim=0) # [B, H*W, C]++ # reshape back+ feature0 = feature0.view(b, h, w, c).permute(0, 3, 1, 2).contiguous() # [B, C, H, W]+ feature1 = feature1.view(b, h, w, c).permute(0, 3, 1, 2).contiguous() # [B, C, H, W]++ return feature0, feature1diff --git a/tools/scoring/optical_flow/unimatch/trident_conv.py b/tools/scoring/optical_flow/unimatch/trident_conv.pyindex 445663c..d87579b 100644--- a/tools/scoring/optical_flow/unimatch/trident_conv.py+++ b/tools/scoring/optical_flow/unimatch/trident_conv.py@@ -1,90 +1,88 @@-# Copyright (c) Facebook, Inc. and its affiliates.-# https://github.com/facebookresearch/detectron2/blob/main/projects/TridentNet/tridentnet/trident_conv.py--import torch-from torch import nn-from torch.nn import functional as F-from torch.nn.modules.utils import _pair---class MultiScaleTridentConv(nn.Module):- def __init__(- self,- in_channels,- out_channels,- kernel_size,- stride=1,- strides=1,- paddings=0,- dilations=1,- dilation=1,- groups=1,- num_branch=1,- test_branch_idx=-1,- bias=False,- norm=None,- activation=None,- ):- super(MultiScaleTridentConv, self).__init__()- self.in_channels = in_channels- self.out_channels = out_channels- self.kernel_size = _pair(kernel_size)- self.num_branch = num_branch- self.stride = _pair(stride)- self.groups = groups- self.with_bias = bias- self.dilation = dilation- if isinstance(paddings, int):- paddings = [paddings] * self.num_branch- if isinstance(dilations, int):- dilations = [dilations] * self.num_branch- if isinstance(strides, int):- strides = [strides] * self.num_branch- self.paddings = [_pair(padding) for padding in paddings]- self.dilations = [_pair(dilation) for dilation in dilations]- self.strides = [_pair(stride) for stride in strides]- self.test_branch_idx = test_branch_idx- self.norm = norm- self.activation = activation-- assert len({self.num_branch, len(self.paddings), len(self.strides)}) == 1-- self.weight = nn.Parameter(- torch.Tensor(out_channels, in_channels // groups, *self.kernel_size)- )- if bias:- self.bias = nn.Parameter(torch.Tensor(out_channels))- else:- self.bias = None-- nn.init.kaiming_uniform_(self.weight, nonlinearity="relu")- if self.bias is not None:- nn.init.constant_(self.bias, 0)-- def forward(self, inputs):- num_branch = self.num_branch if self.training or self.test_branch_idx == -1 else 1- assert len(inputs) == num_branch-- if self.training or self.test_branch_idx == -1:- outputs = [- F.conv2d(input, self.weight, self.bias, stride, padding, self.dilation, self.groups)- for input, stride, padding in zip(inputs, self.strides, self.paddings)- ]- else:- outputs = [- F.conv2d(- inputs[0],- self.weight,- self.bias,- self.strides[self.test_branch_idx] if self.test_branch_idx == -1 else self.strides[-1],- self.paddings[self.test_branch_idx] if self.test_branch_idx == -1 else self.paddings[-1],- self.dilation,- self.groups,- )- ]-- if self.norm is not None:- outputs = [self.norm(x) for x in outputs]- if self.activation is not None:- outputs = [self.activation(x) for x in outputs]- return outputs+# Copyright (c) Facebook, Inc. and its affiliates.+# https://github.com/facebookresearch/detectron2/blob/main/projects/TridentNet/tridentnet/trident_conv.py++import torch+from torch import nn+from torch.nn import functional as F+from torch.nn.modules.utils import _pair+++class MultiScaleTridentConv(nn.Module):+ def __init__(+ self,+ in_channels,+ out_channels,+ kernel_size,+ stride=1,+ strides=1,+ paddings=0,+ dilations=1,+ dilation=1,+ groups=1,+ num_branch=1,+ test_branch_idx=-1,+ bias=False,+ norm=None,+ activation=None,+ ):+ super(MultiScaleTridentConv, self).__init__()+ self.in_channels = in_channels+ self.out_channels = out_channels+ self.kernel_size = _pair(kernel_size)+ self.num_branch = num_branch+ self.stride = _pair(stride)+ self.groups = groups+ self.with_bias = bias+ self.dilation = dilation+ if isinstance(paddings, int):+ paddings = [paddings] * self.num_branch+ if isinstance(dilations, int):+ dilations = [dilations] * self.num_branch+ if isinstance(strides, int):+ strides = [strides] * self.num_branch+ self.paddings = [_pair(padding) for padding in paddings]+ self.dilations = [_pair(dilation) for dilation in dilations]+ self.strides = [_pair(stride) for stride in strides]+ self.test_branch_idx = test_branch_idx+ self.norm = norm+ self.activation = activation++ assert len({self.num_branch, len(self.paddings), len(self.strides)}) == 1++ self.weight = nn.Parameter(torch.Tensor(out_channels, in_channels // groups, *self.kernel_size))+ if bias:+ self.bias = nn.Parameter(torch.Tensor(out_channels))+ else:+ self.bias = None++ nn.init.kaiming_uniform_(self.weight, nonlinearity="relu")+ if self.bias is not None:+ nn.init.constant_(self.bias, 0)++ def forward(self, inputs):+ num_branch = self.num_branch if self.training or self.test_branch_idx == -1 else 1+ assert len(inputs) == num_branch++ if self.training or self.test_branch_idx == -1:+ outputs = [+ F.conv2d(input, self.weight, self.bias, stride, padding, self.dilation, self.groups)+ for input, stride, padding in zip(inputs, self.strides, self.paddings)+ ]+ else:+ outputs = [+ F.conv2d(+ inputs[0],+ self.weight,+ self.bias,+ self.strides[self.test_branch_idx] if self.test_branch_idx == -1 else self.strides[-1],+ self.paddings[self.test_branch_idx] if self.test_branch_idx == -1 else self.paddings[-1],+ self.dilation,+ self.groups,+ )+ ]++ if self.norm is not None:+ outputs = [self.norm(x) for x in outputs]+ if self.activation is not None:+ outputs = [self.activation(x) for x in outputs]+ return outputsdiff --git a/tools/scoring/optical_flow/unimatch/unimatch.py b/tools/scoring/optical_flow/unimatch/unimatch.pyindex 96db16e..c625b99 100644--- a/tools/scoring/optical_flow/unimatch/unimatch.py+++ b/tools/scoring/optical_flow/unimatch/unimatch.py@@ -1,367 +1,393 @@-import torch-import torch.nn as nn-import torch.nn.functional as F--from .backbone import CNNEncoder-from .transformer import FeatureTransformer-from .matching import (global_correlation_softmax, local_correlation_softmax, local_correlation_with_flow,- global_correlation_softmax_stereo, local_correlation_softmax_stereo,- correlation_softmax_depth)-from .attention import SelfAttnPropagation-from .geometry import flow_warp, compute_flow_with_depth_pose-from .reg_refine import BasicUpdateBlock-from .utils import normalize_img, feature_add_position, upsample_flow_with_mask---class UniMatch(nn.Module):- def __init__(self,- num_scales=1,- feature_channels=128,- upsample_factor=8,- num_head=1,- ffn_dim_expansion=4,- num_transformer_layers=6,- reg_refine=False, # optional local regression refinement- task='flow',- ):- super(UniMatch, self).__init__()-- self.feature_channels = feature_channels- self.num_scales = num_scales- self.upsample_factor = upsample_factor- self.reg_refine = reg_refine-- # CNN- self.backbone = CNNEncoder(output_dim=feature_channels, num_output_scales=num_scales)-- # Transformer- self.transformer = FeatureTransformer(num_layers=num_transformer_layers,- d_model=feature_channels,- nhead=num_head,- ffn_dim_expansion=ffn_dim_expansion,- )-- # propagation with self-attn- self.feature_flow_attn = SelfAttnPropagation(in_channels=feature_channels)-- if not self.reg_refine or task == 'depth':- # convex upsampling simiar to RAFT- # concat feature0 and low res flow as input- self.upsampler = nn.Sequential(nn.Conv2d(2 + feature_channels, 256, 3, 1, 1),- nn.ReLU(inplace=True),- nn.Conv2d(256, upsample_factor ** 2 * 9, 1, 1, 0))- # thus far, all the learnable parameters are task-agnostic-- if reg_refine:- # optional task-specific local regression refinement- self.refine_proj = nn.Conv2d(128, 256, 1)- self.refine = BasicUpdateBlock(corr_channels=(2 * 4 + 1) ** 2,- downsample_factor=upsample_factor,- flow_dim=2 if task == 'flow' else 1,- bilinear_up=task == 'depth',- )-- def extract_feature(self, img0, img1):- concat = torch.cat((img0, img1), dim=0) # [2B, C, H, W]- features = self.backbone(concat) # list of [2B, C, H, W], resolution from high to low-- # reverse: resolution from low to high- features = features[::-1]-- feature0, feature1 = [], []-- for i in range(len(features)):- feature = features[i]- chunks = torch.chunk(feature, 2, 0) # tuple- feature0.append(chunks[0])- feature1.append(chunks[1])-- return feature0, feature1-- def upsample_flow(self, flow, feature, bilinear=False, upsample_factor=8,- is_depth=False):- if bilinear:- multiplier = 1 if is_depth else upsample_factor- up_flow = F.interpolate(flow, scale_factor=upsample_factor,- mode='bilinear', align_corners=True) * multiplier- else:- concat = torch.cat((flow, feature), dim=1)- mask = self.upsampler(concat)- up_flow = upsample_flow_with_mask(flow, mask, upsample_factor=self.upsample_factor,- is_depth=is_depth)-- return up_flow-- def forward(self, img0, img1,- attn_type=None,- attn_splits_list=None,- corr_radius_list=None,- prop_radius_list=None,- num_reg_refine=1,- pred_bidir_flow=False,- task='flow',- intrinsics=None,- pose=None, # relative pose transform- min_depth=1. / 0.5, # inverse depth range- max_depth=1. / 10,- num_depth_candidates=64,- depth_from_argmax=False,- pred_bidir_depth=False,- **kwargs,- ):-- if pred_bidir_flow:- assert task == 'flow'-- if task == 'depth':- assert self.num_scales == 1 # multi-scale depth model is not supported yet-- results_dict = {}- flow_preds = []-- if task == 'flow':- # stereo and depth tasks have normalized img in dataloader- img0, img1 = normalize_img(img0, img1) # [B, 3, H, W]-- # list of features, resolution low to high- feature0_list, feature1_list = self.extract_feature(img0, img1) # list of features-- flow = None-- if task != 'depth':- assert len(attn_splits_list) == len(corr_radius_list) == len(prop_radius_list) == self.num_scales- else:- assert len(attn_splits_list) == len(prop_radius_list) == self.num_scales == 1-- for scale_idx in range(self.num_scales):- feature0, feature1 = feature0_list[scale_idx], feature1_list[scale_idx]-- if pred_bidir_flow and scale_idx > 0:- # predicting bidirectional flow with refinement- feature0, feature1 = torch.cat((feature0, feature1), dim=0), torch.cat((feature1, feature0), dim=0)-- feature0_ori, feature1_ori = feature0, feature1-- upsample_factor = self.upsample_factor * (2 ** (self.num_scales - 1 - scale_idx))-- if task == 'depth':- # scale intrinsics- intrinsics_curr = intrinsics.clone()- intrinsics_curr[:, :2] = intrinsics_curr[:, :2] / upsample_factor-- if scale_idx > 0:- assert task != 'depth' # not supported for multi-scale depth model- flow = F.interpolate(flow, scale_factor=2, mode='bilinear', align_corners=True) * 2-- if flow is not None:- assert task != 'depth'- flow = flow.detach()-- if task == 'stereo':- # construct flow vector for disparity- # flow here is actually disparity- zeros = torch.zeros_like(flow) # [B, 1, H, W]- # NOTE: reverse disp, disparity is positive- displace = torch.cat((-flow, zeros), dim=1) # [B, 2, H, W]- feature1 = flow_warp(feature1, displace) # [B, C, H, W]- elif task == 'flow':- feature1 = flow_warp(feature1, flow) # [B, C, H, W]- else:- raise NotImplementedError-- attn_splits = attn_splits_list[scale_idx]- if task != 'depth':- corr_radius = corr_radius_list[scale_idx]- prop_radius = prop_radius_list[scale_idx]-- # add position to features- feature0, feature1 = feature_add_position(feature0, feature1, attn_splits, self.feature_channels)-- # Transformer- feature0, feature1 = self.transformer(feature0, feature1,- attn_type=attn_type,- attn_num_splits=attn_splits,- )-- # correlation and softmax- if task == 'depth':- # first generate depth candidates- b, _, h, w = feature0.size()- depth_candidates = torch.linspace(min_depth, max_depth, num_depth_candidates).type_as(feature0)- depth_candidates = depth_candidates.view(1, num_depth_candidates, 1, 1).repeat(b, 1, h,- w) # [B, D, H, W]-- flow_pred = correlation_softmax_depth(feature0, feature1,- intrinsics_curr,- pose,- depth_candidates=depth_candidates,- depth_from_argmax=depth_from_argmax,- pred_bidir_depth=pred_bidir_depth,- )[0]-- else:- if corr_radius == -1: # global matching- if task == 'flow':- flow_pred = global_correlation_softmax(feature0, feature1, pred_bidir_flow)[0]- elif task == 'stereo':- flow_pred = global_correlation_softmax_stereo(feature0, feature1)[0]- else:- raise NotImplementedError- else: # local matching- if task == 'flow':- flow_pred = local_correlation_softmax(feature0, feature1, corr_radius)[0]- elif task == 'stereo':- flow_pred = local_correlation_softmax_stereo(feature0, feature1, corr_radius)[0]- else:- raise NotImplementedError-- # flow or residual flow- flow = flow + flow_pred if flow is not None else flow_pred-- if task == 'stereo':- flow = flow.clamp(min=0) # positive disparity-- # upsample to the original resolution for supervison at training time only- if self.training:- flow_bilinear = self.upsample_flow(flow, None, bilinear=True, upsample_factor=upsample_factor,- is_depth=task == 'depth')- flow_preds.append(flow_bilinear)-- # flow propagation with self-attn- if (pred_bidir_flow or pred_bidir_depth) and scale_idx == 0:- feature0 = torch.cat((feature0, feature1), dim=0) # [2*B, C, H, W] for propagation-- flow = self.feature_flow_attn(feature0, flow.detach(),- local_window_attn=prop_radius > 0,- local_window_radius=prop_radius,- )-- # bilinear exclude the last one- if self.training and scale_idx < self.num_scales - 1:- flow_up = self.upsample_flow(flow, feature0, bilinear=True,- upsample_factor=upsample_factor,- is_depth=task == 'depth')- flow_preds.append(flow_up)-- if scale_idx == self.num_scales - 1:- if not self.reg_refine:- # upsample to the original image resolution-- if task == 'stereo':- flow_pad = torch.cat((-flow, torch.zeros_like(flow)), dim=1) # [B, 2, H, W]- flow_up_pad = self.upsample_flow(flow_pad, feature0)- flow_up = -flow_up_pad[:, :1] # [B, 1, H, W]- elif task == 'depth':- depth_pad = torch.cat((flow, torch.zeros_like(flow)), dim=1) # [B, 2, H, W]- depth_up_pad = self.upsample_flow(depth_pad, feature0,- is_depth=True).clamp(min=min_depth, max=max_depth)- flow_up = depth_up_pad[:, :1] # [B, 1, H, W]- else:- flow_up = self.upsample_flow(flow, feature0)-- flow_preds.append(flow_up)- else:- # task-specific local regression refinement- # supervise current flow- if self.training:- flow_up = self.upsample_flow(flow, feature0, bilinear=True,- upsample_factor=upsample_factor,- is_depth=task == 'depth')- flow_preds.append(flow_up)-- assert num_reg_refine > 0- for refine_iter_idx in range(num_reg_refine):- flow = flow.detach()-- if task == 'stereo':- zeros = torch.zeros_like(flow) # [B, 1, H, W]- # NOTE: reverse disp, disparity is positive- displace = torch.cat((-flow, zeros), dim=1) # [B, 2, H, W]- correlation = local_correlation_with_flow(- feature0_ori,- feature1_ori,- flow=displace,- local_radius=4,- ) # [B, (2R+1)^2, H, W]- elif task == 'depth':- if pred_bidir_depth and refine_iter_idx == 0:- intrinsics_curr = intrinsics_curr.repeat(2, 1, 1)- pose = torch.cat((pose, torch.inverse(pose)), dim=0)-- feature0_ori, feature1_ori = torch.cat((feature0_ori, feature1_ori),- dim=0), torch.cat((feature1_ori,- feature0_ori), dim=0)-- flow_from_depth = compute_flow_with_depth_pose(1. / flow.squeeze(1),- intrinsics_curr,- extrinsics_rel=pose,- )-- correlation = local_correlation_with_flow(- feature0_ori,- feature1_ori,- flow=flow_from_depth,- local_radius=4,- ) # [B, (2R+1)^2, H, W]-- else:- correlation = local_correlation_with_flow(- feature0_ori,- feature1_ori,- flow=flow,- local_radius=4,- ) # [B, (2R+1)^2, H, W]-- proj = self.refine_proj(feature0)-- net, inp = torch.chunk(proj, chunks=2, dim=1)-- net = torch.tanh(net)- inp = torch.relu(inp)-- net, up_mask, residual_flow = self.refine(net, inp, correlation, flow.clone(),- )-- if task == 'depth':- flow = (flow - residual_flow).clamp(min=min_depth, max=max_depth)- else:- flow = flow + residual_flow-- if task == 'stereo':- flow = flow.clamp(min=0) # positive-- if self.training or refine_iter_idx == num_reg_refine - 1:- if task == 'depth':- if refine_iter_idx < num_reg_refine - 1:- # bilinear upsampling- flow_up = self.upsample_flow(flow, feature0, bilinear=True,- upsample_factor=upsample_factor,- is_depth=True)- else:- # last one convex upsampling- # NOTE: clamp depth due to the zero padding in the unfold in the convex upsampling- # pad depth to 2 channels as flow- depth_pad = torch.cat((flow, torch.zeros_like(flow)), dim=1) # [B, 2, H, W]- depth_up_pad = self.upsample_flow(depth_pad, feature0,- is_depth=True).clamp(min=min_depth,- max=max_depth)- flow_up = depth_up_pad[:, :1] # [B, 1, H, W]-- else:- flow_up = upsample_flow_with_mask(flow, up_mask, upsample_factor=self.upsample_factor,- is_depth=task == 'depth')-- flow_preds.append(flow_up)-- if task == 'stereo':- for i in range(len(flow_preds)):- flow_preds[i] = flow_preds[i].squeeze(1) # [B, H, W]-- # convert inverse depth to depth- if task == 'depth':- for i in range(len(flow_preds)):- flow_preds[i] = 1. / flow_preds[i].squeeze(1) # [B, H, W]-- results_dict.update({'flow_preds': flow_preds})-- return results_dict+import torch+import torch.nn as nn+import torch.nn.functional as F++from .attention import SelfAttnPropagation+from .backbone import CNNEncoder+from .geometry import compute_flow_with_depth_pose, flow_warp+from .matching import (+ correlation_softmax_depth,+ global_correlation_softmax,+ global_correlation_softmax_stereo,+ local_correlation_softmax,+ local_correlation_softmax_stereo,+ local_correlation_with_flow,+)+from .reg_refine import BasicUpdateBlock+from .transformer import FeatureTransformer+from .utils import feature_add_position, normalize_img, upsample_flow_with_mask+++class UniMatch(nn.Module):+ def __init__(+ self,+ num_scales=1,+ feature_channels=128,+ upsample_factor=8,+ num_head=1,+ ffn_dim_expansion=4,+ num_transformer_layers=6,+ reg_refine=False, # optional local regression refinement+ task="flow",+ ):+ super(UniMatch, self).__init__()++ self.feature_channels = feature_channels+ self.num_scales = num_scales+ self.upsample_factor = upsample_factor+ self.reg_refine = reg_refine++ # CNN+ self.backbone = CNNEncoder(output_dim=feature_channels, num_output_scales=num_scales)++ # Transformer+ self.transformer = FeatureTransformer(+ num_layers=num_transformer_layers,+ d_model=feature_channels,+ nhead=num_head,+ ffn_dim_expansion=ffn_dim_expansion,+ )++ # propagation with self-attn+ self.feature_flow_attn = SelfAttnPropagation(in_channels=feature_channels)++ if not self.reg_refine or task == "depth":+ # convex upsampling simiar to RAFT+ # concat feature0 and low res flow as input+ self.upsampler = nn.Sequential(+ nn.Conv2d(2 + feature_channels, 256, 3, 1, 1),+ nn.ReLU(inplace=True),+ nn.Conv2d(256, upsample_factor**2 * 9, 1, 1, 0),+ )+ # thus far, all the learnable parameters are task-agnostic++ if reg_refine:+ # optional task-specific local regression refinement+ self.refine_proj = nn.Conv2d(128, 256, 1)+ self.refine = BasicUpdateBlock(+ corr_channels=(2 * 4 + 1) ** 2,+ downsample_factor=upsample_factor,+ flow_dim=2 if task == "flow" else 1,+ bilinear_up=task == "depth",+ )++ def extract_feature(self, img0, img1):+ concat = torch.cat((img0, img1), dim=0) # [2B, C, H, W]+ features = self.backbone(concat) # list of [2B, C, H, W], resolution from high to low++ # reverse: resolution from low to high+ features = features[::-1]++ feature0, feature1 = [], []++ for i in range(len(features)):+ feature = features[i]+ chunks = torch.chunk(feature, 2, 0) # tuple+ feature0.append(chunks[0])+ feature1.append(chunks[1])++ return feature0, feature1++ def upsample_flow(self, flow, feature, bilinear=False, upsample_factor=8, is_depth=False):+ if bilinear:+ multiplier = 1 if is_depth else upsample_factor+ up_flow = (+ F.interpolate(flow, scale_factor=upsample_factor, mode="bilinear", align_corners=True) * multiplier+ )+ else:+ concat = torch.cat((flow, feature), dim=1)+ mask = self.upsampler(concat)+ up_flow = upsample_flow_with_mask(flow, mask, upsample_factor=self.upsample_factor, is_depth=is_depth)++ return up_flow++ def forward(+ self,+ img0,+ img1,+ attn_type=None,+ attn_splits_list=None,+ corr_radius_list=None,+ prop_radius_list=None,+ num_reg_refine=1,+ pred_bidir_flow=False,+ task="flow",+ intrinsics=None,+ pose=None, # relative pose transform+ min_depth=1.0 / 0.5, # inverse depth range+ max_depth=1.0 / 10,+ num_depth_candidates=64,+ depth_from_argmax=False,+ pred_bidir_depth=False,+ **kwargs,+ ):+ if pred_bidir_flow:+ assert task == "flow"++ if task == "depth":+ assert self.num_scales == 1 # multi-scale depth model is not supported yet++ results_dict = {}+ flow_preds = []++ if task == "flow":+ # stereo and depth tasks have normalized img in dataloader+ img0, img1 = normalize_img(img0, img1) # [B, 3, H, W]++ # list of features, resolution low to high+ feature0_list, feature1_list = self.extract_feature(img0, img1) # list of features++ flow = None++ if task != "depth":+ assert len(attn_splits_list) == len(corr_radius_list) == len(prop_radius_list) == self.num_scales+ else:+ assert len(attn_splits_list) == len(prop_radius_list) == self.num_scales == 1++ for scale_idx in range(self.num_scales):+ feature0, feature1 = feature0_list[scale_idx], feature1_list[scale_idx]++ if pred_bidir_flow and scale_idx > 0:+ # predicting bidirectional flow with refinement+ feature0, feature1 = torch.cat((feature0, feature1), dim=0), torch.cat((feature1, feature0), dim=0)++ feature0_ori, feature1_ori = feature0, feature1++ upsample_factor = self.upsample_factor * (2 ** (self.num_scales - 1 - scale_idx))++ if task == "depth":+ # scale intrinsics+ intrinsics_curr = intrinsics.clone()+ intrinsics_curr[:, :2] = intrinsics_curr[:, :2] / upsample_factor++ if scale_idx > 0:+ assert task != "depth" # not supported for multi-scale depth model+ flow = F.interpolate(flow, scale_factor=2, mode="bilinear", align_corners=True) * 2++ if flow is not None:+ assert task != "depth"+ flow = flow.detach()++ if task == "stereo":+ # construct flow vector for disparity+ # flow here is actually disparity+ zeros = torch.zeros_like(flow) # [B, 1, H, W]+ # NOTE: reverse disp, disparity is positive+ displace = torch.cat((-flow, zeros), dim=1) # [B, 2, H, W]+ feature1 = flow_warp(feature1, displace) # [B, C, H, W]+ elif task == "flow":+ feature1 = flow_warp(feature1, flow) # [B, C, H, W]+ else:+ raise NotImplementedError++ attn_splits = attn_splits_list[scale_idx]+ if task != "depth":+ corr_radius = corr_radius_list[scale_idx]+ prop_radius = prop_radius_list[scale_idx]++ # add position to features+ feature0, feature1 = feature_add_position(feature0, feature1, attn_splits, self.feature_channels)++ # Transformer+ feature0, feature1 = self.transformer(+ feature0,+ feature1,+ attn_type=attn_type,+ attn_num_splits=attn_splits,+ )++ # correlation and softmax+ if task == "depth":+ # first generate depth candidates+ b, _, h, w = feature0.size()+ depth_candidates = torch.linspace(min_depth, max_depth, num_depth_candidates).type_as(feature0)+ depth_candidates = depth_candidates.view(1, num_depth_candidates, 1, 1).repeat(+ b, 1, h, w+ ) # [B, D, H, W]++ flow_pred = correlation_softmax_depth(+ feature0,+ feature1,+ intrinsics_curr,+ pose,+ depth_candidates=depth_candidates,+ depth_from_argmax=depth_from_argmax,+ pred_bidir_depth=pred_bidir_depth,+ )[0]++ else:+ if corr_radius == -1: # global matching+ if task == "flow":+ flow_pred = global_correlation_softmax(feature0, feature1, pred_bidir_flow)[0]+ elif task == "stereo":+ flow_pred = global_correlation_softmax_stereo(feature0, feature1)[0]+ else:+ raise NotImplementedError+ else: # local matching+ if task == "flow":+ flow_pred = local_correlation_softmax(feature0, feature1, corr_radius)[0]+ elif task == "stereo":+ flow_pred = local_correlation_softmax_stereo(feature0, feature1, corr_radius)[0]+ else:+ raise NotImplementedError++ # flow or residual flow+ flow = flow + flow_pred if flow is not None else flow_pred++ if task == "stereo":+ flow = flow.clamp(min=0) # positive disparity++ # upsample to the original resolution for supervison at training time only+ if self.training:+ flow_bilinear = self.upsample_flow(+ flow, None, bilinear=True, upsample_factor=upsample_factor, is_depth=task == "depth"+ )+ flow_preds.append(flow_bilinear)++ # flow propagation with self-attn+ if (pred_bidir_flow or pred_bidir_depth) and scale_idx == 0:+ feature0 = torch.cat((feature0, feature1), dim=0) # [2*B, C, H, W] for propagation++ flow = self.feature_flow_attn(+ feature0,+ flow.detach(),+ local_window_attn=prop_radius > 0,+ local_window_radius=prop_radius,+ )++ # bilinear exclude the last one+ if self.training and scale_idx < self.num_scales - 1:+ flow_up = self.upsample_flow(+ flow, feature0, bilinear=True, upsample_factor=upsample_factor, is_depth=task == "depth"+ )+ flow_preds.append(flow_up)++ if scale_idx == self.num_scales - 1:+ if not self.reg_refine:+ # upsample to the original image resolution++ if task == "stereo":+ flow_pad = torch.cat((-flow, torch.zeros_like(flow)), dim=1) # [B, 2, H, W]+ flow_up_pad = self.upsample_flow(flow_pad, feature0)+ flow_up = -flow_up_pad[:, :1] # [B, 1, H, W]+ elif task == "depth":+ depth_pad = torch.cat((flow, torch.zeros_like(flow)), dim=1) # [B, 2, H, W]+ depth_up_pad = self.upsample_flow(depth_pad, feature0, is_depth=True).clamp(+ min=min_depth, max=max_depth+ )+ flow_up = depth_up_pad[:, :1] # [B, 1, H, W]+ else:+ flow_up = self.upsample_flow(flow, feature0)++ flow_preds.append(flow_up)+ else:+ # task-specific local regression refinement+ # supervise current flow+ if self.training:+ flow_up = self.upsample_flow(+ flow, feature0, bilinear=True, upsample_factor=upsample_factor, is_depth=task == "depth"+ )+ flow_preds.append(flow_up)++ assert num_reg_refine > 0+ for refine_iter_idx in range(num_reg_refine):+ flow = flow.detach()++ if task == "stereo":+ zeros = torch.zeros_like(flow) # [B, 1, H, W]+ # NOTE: reverse disp, disparity is positive+ displace = torch.cat((-flow, zeros), dim=1) # [B, 2, H, W]+ correlation = local_correlation_with_flow(+ feature0_ori,+ feature1_ori,+ flow=displace,+ local_radius=4,+ ) # [B, (2R+1)^2, H, W]+ elif task == "depth":+ if pred_bidir_depth and refine_iter_idx == 0:+ intrinsics_curr = intrinsics_curr.repeat(2, 1, 1)+ pose = torch.cat((pose, torch.inverse(pose)), dim=0)++ feature0_ori, feature1_ori = torch.cat((feature0_ori, feature1_ori), dim=0), torch.cat(+ (feature1_ori, feature0_ori), dim=0+ )++ flow_from_depth = compute_flow_with_depth_pose(+ 1.0 / flow.squeeze(1),+ intrinsics_curr,+ extrinsics_rel=pose,+ )++ correlation = local_correlation_with_flow(+ feature0_ori,+ feature1_ori,+ flow=flow_from_depth,+ local_radius=4,+ ) # [B, (2R+1)^2, H, W]++ else:+ correlation = local_correlation_with_flow(+ feature0_ori,+ feature1_ori,+ flow=flow,+ local_radius=4,+ ) # [B, (2R+1)^2, H, W]++ proj = self.refine_proj(feature0)++ net, inp = torch.chunk(proj, chunks=2, dim=1)++ net = torch.tanh(net)+ inp = torch.relu(inp)++ net, up_mask, residual_flow = self.refine(+ net,+ inp,+ correlation,+ flow.clone(),+ )++ if task == "depth":+ flow = (flow - residual_flow).clamp(min=min_depth, max=max_depth)+ else:+ flow = flow + residual_flow++ if task == "stereo":+ flow = flow.clamp(min=0) # positive++ if self.training or refine_iter_idx == num_reg_refine - 1:+ if task == "depth":+ if refine_iter_idx < num_reg_refine - 1:+ # bilinear upsampling+ flow_up = self.upsample_flow(+ flow, feature0, bilinear=True, upsample_factor=upsample_factor, is_depth=True+ )+ else:+ # last one convex upsampling+ # NOTE: clamp depth due to the zero padding in the unfold in the convex upsampling+ # pad depth to 2 channels as flow+ depth_pad = torch.cat((flow, torch.zeros_like(flow)), dim=1) # [B, 2, H, W]+ depth_up_pad = self.upsample_flow(depth_pad, feature0, is_depth=True).clamp(+ min=min_depth, max=max_depth+ )+ flow_up = depth_up_pad[:, :1] # [B, 1, H, W]++ else:+ flow_up = upsample_flow_with_mask(+ flow, up_mask, upsample_factor=self.upsample_factor, is_depth=task == "depth"+ )++ flow_preds.append(flow_up)++ if task == "stereo":+ for i in range(len(flow_preds)):+ flow_preds[i] = flow_preds[i].squeeze(1) # [B, H, W]++ # convert inverse depth to depth+ if task == "depth":+ for i in range(len(flow_preds)):+ flow_preds[i] = 1.0 / flow_preds[i].squeeze(1) # [B, H, W]++ results_dict.update({"flow_preds": flow_preds})++ return results_dictdiff --git a/tools/scoring/optical_flow/unimatch/utils.py b/tools/scoring/optical_flow/unimatch/utils.pyindex 0c3dbea..60f40be 100644--- a/tools/scoring/optical_flow/unimatch/utils.py+++ b/tools/scoring/optical_flow/unimatch/utils.py@@ -1,216 +1,219 @@-import torch-import torch.nn.functional as F-from .position import PositionEmbeddingSine---def generate_window_grid(h_min, h_max, w_min, w_max, len_h, len_w, device=None):- assert device is not None-- x, y = torch.meshgrid([torch.linspace(w_min, w_max, len_w, device=device),- torch.linspace(h_min, h_max, len_h, device=device)],- )- grid = torch.stack((x, y), -1).transpose(0, 1).float() # [H, W, 2]-- return grid---def normalize_coords(coords, h, w):- # coords: [B, H, W, 2]- c = torch.Tensor([(w - 1) / 2., (h - 1) / 2.]).float().to(coords.device)- return (coords - c) / c # [-1, 1]---def normalize_img(img0, img1):- # loaded images are in [0, 255]- # normalize by ImageNet mean and std- mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1).to(img1.device)- std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1).to(img1.device)- img0 = (img0 / 255. - mean) / std- img1 = (img1 / 255. - mean) / std-- return img0, img1---def split_feature(feature,- num_splits=2,- channel_last=False,- ):- if channel_last: # [B, H, W, C]- b, h, w, c = feature.size()- assert h % num_splits == 0 and w % num_splits == 0-- b_new = b * num_splits * num_splits- h_new = h // num_splits- w_new = w // num_splits-- feature = feature.view(b, num_splits, h // num_splits, num_splits, w // num_splits, c- ).permute(0, 1, 3, 2, 4, 5).reshape(b_new, h_new, w_new, c) # [B*K*K, H/K, W/K, C]- else: # [B, C, H, W]- b, c, h, w = feature.size()- assert h % num_splits == 0 and w % num_splits == 0-- b_new = b * num_splits * num_splits- h_new = h // num_splits- w_new = w // num_splits-- feature = feature.view(b, c, num_splits, h // num_splits, num_splits, w // num_splits- ).permute(0, 2, 4, 1, 3, 5).reshape(b_new, c, h_new, w_new) # [B*K*K, C, H/K, W/K]-- return feature---def merge_splits(splits,- num_splits=2,- channel_last=False,- ):- if channel_last: # [B*K*K, H/K, W/K, C]- b, h, w, c = splits.size()- new_b = b // num_splits // num_splits-- splits = splits.view(new_b, num_splits, num_splits, h, w, c)- merge = splits.permute(0, 1, 3, 2, 4, 5).contiguous().view(- new_b, num_splits * h, num_splits * w, c) # [B, H, W, C]- else: # [B*K*K, C, H/K, W/K]- b, c, h, w = splits.size()- new_b = b // num_splits // num_splits-- splits = splits.view(new_b, num_splits, num_splits, c, h, w)- merge = splits.permute(0, 3, 1, 4, 2, 5).contiguous().view(- new_b, c, num_splits * h, num_splits * w) # [B, C, H, W]-- return merge---def generate_shift_window_attn_mask(input_resolution, window_size_h, window_size_w,- shift_size_h, shift_size_w, device=torch.device('cuda')):- # ref: https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py- # calculate attention mask for SW-MSA- h, w = input_resolution- img_mask = torch.zeros((1, h, w, 1)).to(device) # 1 H W 1- h_slices = (slice(0, -window_size_h),- slice(-window_size_h, -shift_size_h),- slice(-shift_size_h, None))- w_slices = (slice(0, -window_size_w),- slice(-window_size_w, -shift_size_w),- slice(-shift_size_w, None))- cnt = 0- for h in h_slices:- for w in w_slices:- img_mask[:, h, w, :] = cnt- cnt += 1-- mask_windows = split_feature(img_mask, num_splits=input_resolution[-1] // window_size_w, channel_last=True)-- mask_windows = mask_windows.view(-1, window_size_h * window_size_w)- attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)- attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))-- return attn_mask---def feature_add_position(feature0, feature1, attn_splits, feature_channels):- pos_enc = PositionEmbeddingSine(num_pos_feats=feature_channels // 2)-- if attn_splits > 1: # add position in splited window- feature0_splits = split_feature(feature0, num_splits=attn_splits)- feature1_splits = split_feature(feature1, num_splits=attn_splits)-- position = pos_enc(feature0_splits)-- feature0_splits = feature0_splits + position- feature1_splits = feature1_splits + position-- feature0 = merge_splits(feature0_splits, num_splits=attn_splits)- feature1 = merge_splits(feature1_splits, num_splits=attn_splits)- else:- position = pos_enc(feature0)-- feature0 = feature0 + position- feature1 = feature1 + position-- return feature0, feature1---def upsample_flow_with_mask(flow, up_mask, upsample_factor,- is_depth=False):- # convex upsampling following raft-- mask = up_mask- b, flow_channel, h, w = flow.shape- mask = mask.view(b, 1, 9, upsample_factor, upsample_factor, h, w) # [B, 1, 9, K, K, H, W]- mask = torch.softmax(mask, dim=2)-- multiplier = 1 if is_depth else upsample_factor- up_flow = F.unfold(multiplier * flow, [3, 3], padding=1)- up_flow = up_flow.view(b, flow_channel, 9, 1, 1, h, w) # [B, 2, 9, 1, 1, H, W]-- up_flow = torch.sum(mask * up_flow, dim=2) # [B, 2, K, K, H, W]- up_flow = up_flow.permute(0, 1, 4, 2, 5, 3) # [B, 2, K, H, K, W]- up_flow = up_flow.reshape(b, flow_channel, upsample_factor * h,- upsample_factor * w) # [B, 2, K*H, K*W]-- return up_flow---def split_feature_1d(feature,- num_splits=2,- ):- # feature: [B, W, C]- b, w, c = feature.size()- assert w % num_splits == 0-- b_new = b * num_splits- w_new = w // num_splits-- feature = feature.view(b, num_splits, w // num_splits, c- ).view(b_new, w_new, c) # [B*K, W/K, C]-- return feature---def merge_splits_1d(splits,- h,- num_splits=2,- ):- b, w, c = splits.size()- new_b = b // num_splits // h-- splits = splits.view(new_b, h, num_splits, w, c)- merge = splits.view(- new_b, h, num_splits * w, c) # [B, H, W, C]-- return merge---def window_partition_1d(x, window_size_w):- """- Args:- x: (B, W, C)- window_size (int): window size-- Returns:- windows: (num_windows*B, window_size, C)- """- B, W, C = x.shape- x = x.view(B, W // window_size_w, window_size_w, C).view(-1, window_size_w, C)- return x---def generate_shift_window_attn_mask_1d(input_w, window_size_w,- shift_size_w, device=torch.device('cuda')):- # calculate attention mask for SW-MSA- img_mask = torch.zeros((1, input_w, 1)).to(device) # 1 W 1- w_slices = (slice(0, -window_size_w),- slice(-window_size_w, -shift_size_w),- slice(-shift_size_w, None))- cnt = 0- for w in w_slices:- img_mask[:, w, :] = cnt- cnt += 1-- mask_windows = window_partition_1d(img_mask, window_size_w) # nW, window_size, 1- mask_windows = mask_windows.view(-1, window_size_w)- attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW, window_size, window_size- attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))-- return attn_mask+import torch+import torch.nn.functional as F++from .position import PositionEmbeddingSine+++def generate_window_grid(h_min, h_max, w_min, w_max, len_h, len_w, device=None):+ assert device is not None++ x, y = torch.meshgrid(+ [torch.linspace(w_min, w_max, len_w, device=device), torch.linspace(h_min, h_max, len_h, device=device)],+ )+ grid = torch.stack((x, y), -1).transpose(0, 1).float() # [H, W, 2]++ return grid+++def normalize_coords(coords, h, w):+ # coords: [B, H, W, 2]+ c = torch.Tensor([(w - 1) / 2.0, (h - 1) / 2.0]).float().to(coords.device)+ return (coords - c) / c # [-1, 1]+++def normalize_img(img0, img1):+ # loaded images are in [0, 255]+ # normalize by ImageNet mean and std+ mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1).to(img1.device)+ std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1).to(img1.device)+ img0 = (img0 / 255.0 - mean) / std+ img1 = (img1 / 255.0 - mean) / std++ return img0, img1+++def split_feature(+ feature,+ num_splits=2,+ channel_last=False,+):+ if channel_last: # [B, H, W, C]+ b, h, w, c = feature.size()+ assert h % num_splits == 0 and w % num_splits == 0++ b_new = b * num_splits * num_splits+ h_new = h // num_splits+ w_new = w // num_splits++ feature = (+ feature.view(b, num_splits, h // num_splits, num_splits, w // num_splits, c)+ .permute(0, 1, 3, 2, 4, 5)+ .reshape(b_new, h_new, w_new, c)+ ) # [B*K*K, H/K, W/K, C]+ else: # [B, C, H, W]+ b, c, h, w = feature.size()+ assert h % num_splits == 0 and w % num_splits == 0++ b_new = b * num_splits * num_splits+ h_new = h // num_splits+ w_new = w // num_splits++ feature = (+ feature.view(b, c, num_splits, h // num_splits, num_splits, w // num_splits)+ .permute(0, 2, 4, 1, 3, 5)+ .reshape(b_new, c, h_new, w_new)+ ) # [B*K*K, C, H/K, W/K]++ return feature+++def merge_splits(+ splits,+ num_splits=2,+ channel_last=False,+):+ if channel_last: # [B*K*K, H/K, W/K, C]+ b, h, w, c = splits.size()+ new_b = b // num_splits // num_splits++ splits = splits.view(new_b, num_splits, num_splits, h, w, c)+ merge = (+ splits.permute(0, 1, 3, 2, 4, 5).contiguous().view(new_b, num_splits * h, num_splits * w, c)+ ) # [B, H, W, C]+ else: # [B*K*K, C, H/K, W/K]+ b, c, h, w = splits.size()+ new_b = b // num_splits // num_splits++ splits = splits.view(new_b, num_splits, num_splits, c, h, w)+ merge = (+ splits.permute(0, 3, 1, 4, 2, 5).contiguous().view(new_b, c, num_splits * h, num_splits * w)+ ) # [B, C, H, W]++ return merge+++def generate_shift_window_attn_mask(+ input_resolution, window_size_h, window_size_w, shift_size_h, shift_size_w, device=torch.device("cuda")+):+ # ref: https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py+ # calculate attention mask for SW-MSA+ h, w = input_resolution+ img_mask = torch.zeros((1, h, w, 1)).to(device) # 1 H W 1+ h_slices = (slice(0, -window_size_h), slice(-window_size_h, -shift_size_h), slice(-shift_size_h, None))+ w_slices = (slice(0, -window_size_w), slice(-window_size_w, -shift_size_w), slice(-shift_size_w, None))+ cnt = 0+ for h in h_slices:+ for w in w_slices:+ img_mask[:, h, w, :] = cnt+ cnt += 1++ mask_windows = split_feature(img_mask, num_splits=input_resolution[-1] // window_size_w, channel_last=True)++ mask_windows = mask_windows.view(-1, window_size_h * window_size_w)+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))++ return attn_mask+++def feature_add_position(feature0, feature1, attn_splits, feature_channels):+ pos_enc = PositionEmbeddingSine(num_pos_feats=feature_channels // 2)++ if attn_splits > 1: # add position in splited window+ feature0_splits = split_feature(feature0, num_splits=attn_splits)+ feature1_splits = split_feature(feature1, num_splits=attn_splits)++ position = pos_enc(feature0_splits)++ feature0_splits = feature0_splits + position+ feature1_splits = feature1_splits + position++ feature0 = merge_splits(feature0_splits, num_splits=attn_splits)+ feature1 = merge_splits(feature1_splits, num_splits=attn_splits)+ else:+ position = pos_enc(feature0)++ feature0 = feature0 + position+ feature1 = feature1 + position++ return feature0, feature1+++def upsample_flow_with_mask(flow, up_mask, upsample_factor, is_depth=False):+ # convex upsampling following raft++ mask = up_mask+ b, flow_channel, h, w = flow.shape+ mask = mask.view(b, 1, 9, upsample_factor, upsample_factor, h, w) # [B, 1, 9, K, K, H, W]+ mask = torch.softmax(mask, dim=2)++ multiplier = 1 if is_depth else upsample_factor+ up_flow = F.unfold(multiplier * flow, [3, 3], padding=1)+ up_flow = up_flow.view(b, flow_channel, 9, 1, 1, h, w) # [B, 2, 9, 1, 1, H, W]++ up_flow = torch.sum(mask * up_flow, dim=2) # [B, 2, K, K, H, W]+ up_flow = up_flow.permute(0, 1, 4, 2, 5, 3) # [B, 2, K, H, K, W]+ up_flow = up_flow.reshape(b, flow_channel, upsample_factor * h, upsample_factor * w) # [B, 2, K*H, K*W]++ return up_flow+++def split_feature_1d(+ feature,+ num_splits=2,+):+ # feature: [B, W, C]+ b, w, c = feature.size()+ assert w % num_splits == 0++ b_new = b * num_splits+ w_new = w // num_splits++ feature = feature.view(b, num_splits, w // num_splits, c).view(b_new, w_new, c) # [B*K, W/K, C]++ return feature+++def merge_splits_1d(+ splits,+ h,+ num_splits=2,+):+ b, w, c = splits.size()+ new_b = b // num_splits // h++ splits = splits.view(new_b, h, num_splits, w, c)+ merge = splits.view(new_b, h, num_splits * w, c) # [B, H, W, C]++ return merge+++def window_partition_1d(x, window_size_w):+ """+ Args:+ x: (B, W, C)+ window_size (int): window size++ Returns:+ windows: (num_windows*B, window_size, C)+ """+ B, W, C = x.shape+ x = x.view(B, W // window_size_w, window_size_w, C).view(-1, window_size_w, C)+ return x+++def generate_shift_window_attn_mask_1d(input_w, window_size_w, shift_size_w, device=torch.device("cuda")):+ # calculate attention mask for SW-MSA+ img_mask = torch.zeros((1, input_w, 1)).to(device) # 1 W 1+ w_slices = (slice(0, -window_size_w), slice(-window_size_w, -shift_size_w), slice(-shift_size_w, None))+ cnt = 0+ for w in w_slices:+ img_mask[:, w, :] = cnt+ cnt += 1++ mask_windows = window_partition_1d(img_mask, window_size_w) # nW, window_size, 1+ mask_windows = mask_windows.view(-1, window_size_w)+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW, window_size, window_size+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))++ return attn_mask