Skip to content

GitLab

Commit 1d5a34cf authored by wanglch's avatar wanglch
Browse files

Initial commit

parents
Hide whitespace changes
Inline Side-by-side
Showing with 5917 additions and 0 deletions
internvl_chat/eval/mme/Your_Results/posters.txt 0 → 100644
View file @ 1d5a34cf
tt0048028.jpg Is this movie directed by elia kazan? Please answer yes or no. Yes
tt0048028.jpg Is this movie directed by maneesh sharma? Please answer yes or no. No
tt0053221.jpg Is this movie titled rio bravo (1959)? Please answer yes or no. Yes
tt0053221.jpg Is this movie titled the matrix (1999)? Please answer yes or no. No
tt0062622.jpg Is this movie originated from the country or region of uk? Please answer yes or no. Yes
tt0062622.jpg Is this movie originated from the country or region of china? Please answer yes or no. No
tt0063442.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0063442.jpg Is this movie originated from the country or region of japan? Please answer yes or no. No
tt0065724.jpg Is this movie directed by bob rafelson? Please answer yes or no. Yes
tt0065724.jpg Is this movie directed by anthony minghella? Please answer yes or no. No
tt0067116.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0067116.jpg Is this movie originated from the country or region of uk? Please answer yes or no. No
tt0067140.jpg Is this movie titled duck, you sucker (1971)? Please answer yes or no. Yes
tt0067140.jpg Is this movie titled crimson tide (1995)? Please answer yes or no. No
tt0067185.jpg Is this movie directed by hal ashby? Please answer yes or no. Yes
tt0067185.jpg Is this movie directed by john hillcoat? Please answer yes or no. No
tt0068646.jpg Is this movie titled the godfather (1972)? Please answer yes or no. Yes
tt0068646.jpg Is this movie titled bring on the melody (2017)? Please answer yes or no. No
tt0070291.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0070291.jpg Is this movie originated from the country or region of spain? Please answer yes or no. No
tt0071315.jpg Is this movie titled chinatown (1974)? Please answer yes or no. Yes
tt0071315.jpg Is this movie titled planet of the apes (1968)? Please answer yes or no. No
tt0074119.jpg Is this movie directed by alan j. pakula? Please answer yes or no. Yes
tt0074119.jpg Is this movie directed by stanley kubrick? Please answer yes or no. No
tt0074749.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0074749.jpg Is this movie originated from the country or region of venezuela? Please answer yes or no. No
tt0082186.jpg Is this movie directed by desmond davis? Please answer yes or no. Yes
tt0082186.jpg Is this movie directed by edward zwick? Please answer yes or no. No
tt0083907.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0083907.jpg Is this movie originated from the country or region of bulgaria? Please answer yes or no. No
tt0083946.jpg Is this movie originated from the country or region of west germany? Please answer yes or no. Yes
tt0083946.jpg Is this movie originated from the country or region of denmark? Please answer yes or no. No
tt0084787.jpg Is this movie titled the thing (1982)? Please answer yes or no. Yes
tt0084787.jpg Is this movie titled jay and silent bob strike back (2001)? Please answer yes or no. No
tt0086250.jpg Is this movie directed by brian de palma? Please answer yes or no. Yes
tt0086250.jpg Is this movie directed by patrice leconte? Please answer yes or no. No
tt0093565.jpg Is this movie titled moonstruck (1987)? Please answer yes or no. Yes
tt0093565.jpg Is this movie titled now you see me (2013)? Please answer yes or no. No
tt0093773.jpg Is this movie titled predator (1987)? Please answer yes or no. Yes
tt0093773.jpg Is this movie titled the cell (2000)? Please answer yes or no. No
tt0094291.jpg Is this movie directed by oliver stone? Please answer yes or no. Yes
tt0094291.jpg Is this movie directed by raja gosnell? Please answer yes or no. No
tt0096446.jpg Is this movie directed by ron howard? Please answer yes or no. Yes
tt0096446.jpg Is this movie directed by peter jackson? Please answer yes or no. No
tt0096463.jpg Is this movie directed by mike nichols? Please answer yes or no. Yes
tt0096463.jpg Is this movie directed by cary joji fukunaga? Please answer yes or no. No
tt0096895.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0096895.jpg Is this movie originated from the country or region of china? Please answer yes or no. No
tt0097576.jpg Is this movie directed by steven spielberg? Please answer yes or no. Yes
tt0097576.jpg Is this movie directed by francis ford coppola? Please answer yes or no. No
tt0098635.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0098635.jpg Is this movie originated from the country or region of spain? Please answer yes or no. No
tt0098724.jpg Is this movie titled sex, lies, and videotape (1989)? Please answer yes or no. Yes
tt0098724.jpg Is this movie titled final destination (2000)? Please answer yes or no. No
tt0099653.jpg Is this movie directed by jerry zucker? Please answer yes or no. Yes
tt0099653.jpg Is this movie directed by felix chong,alan mak? Please answer yes or no. No
tt0100112.jpg Is this movie originated from the country or region of france? Please answer yes or no. Yes
tt0100112.jpg Is this movie originated from the country or region of new zealand? Please answer yes or no. No
tt0100150.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0100150.jpg Is this movie originated from the country or region of hong kong? Please answer yes or no. No
tt0100935.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0100935.jpg Is this movie originated from the country or region of canada? Please answer yes or no. No
tt0101889.jpg Is this movie titled the fisher king (1991)? Please answer yes or no. Yes
tt0101889.jpg Is this movie titled a good day to die hard (2013)? Please answer yes or no. No
tt0101921.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0101921.jpg Is this movie originated from the country or region of australia? Please answer yes or no. No
tt0107653.jpg Is this movie titled naked (1993)? Please answer yes or no. Yes
tt0107653.jpg Is this movie titled captain america: civil war (2016)? Please answer yes or no. No
tt0107808.jpg Is this movie titled a perfect world (1993)? Please answer yes or no. Yes
tt0107808.jpg Is this movie titled harold and maude (1971)? Please answer yes or no. No
tt0110201.jpg Is this movie titled hail the judge (1994)? Please answer yes or no. Yes
tt0110201.jpg Is this movie titled batman (1989)? Please answer yes or no. No
tt0112740.jpg Is this movie titled crimson tide (1995)? Please answer yes or no. Yes
tt0112740.jpg Is this movie titled insidious: the last key (2018)? Please answer yes or no. No
tt0113497.jpg Is this movie titled jumanji (1995)? Please answer yes or no. Yes
tt0113497.jpg Is this movie titled morgan (2016)? Please answer yes or no. No
tt0114369.jpg Is this movie titled se7en (1995)? Please answer yes or no. Yes
tt0114369.jpg Is this movie titled a perfect world (1993)? Please answer yes or no. No
tt0114388.jpg Is this movie directed by ang lee? Please answer yes or no. Yes
tt0114388.jpg Is this movie directed by david lowery? Please answer yes or no. No
tt0116209.jpg Is this movie directed by anthony minghella? Please answer yes or no. Yes
tt0116209.jpg Is this movie directed by john cassavetes? Please answer yes or no. No
tt0116629.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0116629.jpg Is this movie originated from the country or region of australia? Please answer yes or no. No
tt0118636.jpg Is this movie titled apt pupil (1998)? Please answer yes or no. Yes
tt0118636.jpg Is this movie titled a good day to die hard (2013)? Please answer yes or no. No
tt0118655.jpg Is this movie directed by jay roach? Please answer yes or no. Yes
tt0118655.jpg Is this movie directed by peter jackson? Please answer yes or no. No
tt0118715.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0118715.jpg Is this movie originated from the country or region of venezuela? Please answer yes or no. No
tt0119303.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0119303.jpg Is this movie originated from the country or region of spain? Please answer yes or no. No
tt0120738.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0120738.jpg Is this movie originated from the country or region of germany? Please answer yes or no. No
tt0129387.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0129387.jpg Is this movie originated from the country or region of germany? Please answer yes or no. No
tt0133093.jpg Is this movie directed by lana wachowski,lilly wachowski? Please answer yes or no. Yes
tt0133093.jpg Is this movie directed by siu-tung ching? Please answer yes or no. No
tt0163025.jpg Is this movie directed by joe johnston? Please answer yes or no. Yes
tt0163025.jpg Is this movie directed by adam robitel? Please answer yes or no. No
tt0166924.jpg Is this movie titled mulholland drive (2001)? Please answer yes or no. Yes
tt0166924.jpg Is this movie titled lucky number slevin (2006)? Please answer yes or no. No
tt0167261.jpg Is this movie directed by peter jackson? Please answer yes or no. Yes
tt0167261.jpg Is this movie directed by gabe ibáñez? Please answer yes or no. No
tt0182789.jpg Is this movie titled bicentennial man (1999)? Please answer yes or no. Yes
tt0182789.jpg Is this movie titled ocean's eleven (2001)? Please answer yes or no. No
tt0187078.jpg Is this movie titled gone in 60 seconds (2000)? Please answer yes or no. Yes
tt0187078.jpg Is this movie titled avatar (2009)? Please answer yes or no. No
tt0195714.jpg Is this movie directed by james wong? Please answer yes or no. Yes
tt0195714.jpg Is this movie directed by bob rafelson? Please answer yes or no. No
tt0209958.jpg Is this movie titled the cell (2000)? Please answer yes or no. Yes
tt0209958.jpg Is this movie titled the matrix revolutions (2003)? Please answer yes or no. No
tt0232500.jpg Is this movie titled the fast and the furious (2001)? Please answer yes or no. Yes
tt0232500.jpg Is this movie titled east of eden (1955)? Please answer yes or no. No
tt0240772.jpg Is this movie directed by steven soderbergh? Please answer yes or no. Yes
tt0240772.jpg Is this movie directed by paul mcguigan? Please answer yes or no. No
tt0242653.jpg Is this movie titled the matrix revolutions (2003)? Please answer yes or no. Yes
tt0242653.jpg Is this movie titled sense and sensibility (1995)? Please answer yes or no. No
tt0244244.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0244244.jpg Is this movie originated from the country or region of argentina? Please answer yes or no. No
tt0258463.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0258463.jpg Is this movie originated from the country or region of italy? Please answer yes or no. No
tt0261392.jpg Is this movie titled jay and silent bob strike back (2001)? Please answer yes or no. Yes
tt0261392.jpg Is this movie titled avatar (2009)? Please answer yes or no. No
tt0264395.jpg Is this movie originated from the country or region of germany? Please answer yes or no. Yes
tt0264395.jpg Is this movie originated from the country or region of australia? Please answer yes or no. No
tt0266697.jpg Is this movie directed by quentin tarantino? Please answer yes or no. Yes
tt0266697.jpg Is this movie directed by peyton reed? Please answer yes or no. No
tt0268978.jpg Is this movie titled a beautiful mind (2001)? Please answer yes or no. Yes
tt0268978.jpg Is this movie titled blue jasmine (2013)? Please answer yes or no. No
tt0318627.jpg Is this movie titled resident evil: apocalypse (2004)? Please answer yes or no. Yes
tt0318627.jpg Is this movie titled the meddler (2015)? Please answer yes or no. No
tt0328107.jpg Is this movie titled man on fire (2004)? Please answer yes or no. Yes
tt0328107.jpg Is this movie titled real steel (2011)? Please answer yes or no. No
tt0338751.jpg Is this movie directed by martin scorsese? Please answer yes or no. Yes
tt0338751.jpg Is this movie directed by chris columbus? Please answer yes or no. No
tt0341495.jpg Is this movie directed by siu-tung ching? Please answer yes or no. Yes
tt0341495.jpg Is this movie directed by lorenzo vigas? Please answer yes or no. No
tt0351977.jpg Is this movie directed by kevin bray? Please answer yes or no. Yes
tt0351977.jpg Is this movie directed by joe johnston? Please answer yes or no. No
tt0369702.jpg Is this movie originated from the country or region of spain? Please answer yes or no. Yes
tt0369702.jpg Is this movie originated from the country or region of france? Please answer yes or no. No
tt0373469.jpg Is this movie directed by shane black? Please answer yes or no. Yes
tt0373469.jpg Is this movie directed by dennis dugan? Please answer yes or no. No
tt0378194.jpg Is this movie directed by quentin tarantino? Please answer yes or no. Yes
tt0378194.jpg Is this movie directed by todd phillips? Please answer yes or no. No
tt0379786.jpg Is this movie titled serenity (2005)? Please answer yes or no. Yes
tt0379786.jpg Is this movie titled bad genius (2017)? Please answer yes or no. No
tt0393109.jpg Is this movie titled brick (2005)? Please answer yes or no. Yes
tt0393109.jpg Is this movie titled sense and sensibility (1995)? Please answer yes or no. No
tt0399201.jpg Is this movie directed by michael bay? Please answer yes or no. Yes
tt0399201.jpg Is this movie directed by angelina jolie? Please answer yes or no. No
tt0425210.jpg Is this movie titled lucky number slevin (2006)? Please answer yes or no. Yes
tt0425210.jpg Is this movie titled the godfather (1972)? Please answer yes or no. No
tt0433035.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0433035.jpg Is this movie originated from the country or region of new zealand? Please answer yes or no. No
tt0443680.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0443680.jpg Is this movie originated from the country or region of germany? Please answer yes or no. No
tt0449088.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0449088.jpg Is this movie originated from the country or region of japan? Please answer yes or no. No
tt0450259.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt0450259.jpg Is this movie originated from the country or region of china? Please answer yes or no. No
tt0462499.jpg Is this movie titled rambo (2008)? Please answer yes or no. Yes
tt0462499.jpg Is this movie titled walking tall (2004)? Please answer yes or no. No
tt0499549.jpg Is this movie directed by james cameron? Please answer yes or no. Yes
tt0499549.jpg Is this movie directed by jerry zucker? Please answer yes or no. No
tt0780653.jpg Is this movie titled the wolfman (2010)? Please answer yes or no. Yes
tt0780653.jpg Is this movie titled wild at heart (1990)? Please answer yes or no. No
tt0829482.jpg Is this movie directed by greg mottola? Please answer yes or no. Yes
tt0829482.jpg Is this movie directed by jing wong? Please answer yes or no. No
tt0848228.jpg Is this movie titled the avengers (2012)? Please answer yes or no. Yes
tt0848228.jpg Is this movie titled wolf warrior 2 (2017)? Please answer yes or no. No
tt0898367.jpg Is this movie directed by john hillcoat? Please answer yes or no. Yes
tt0898367.jpg Is this movie directed by franklin j. schaffner? Please answer yes or no. No
tt0942385.jpg Is this movie titled tropic thunder (2008)? Please answer yes or no. Yes
tt0942385.jpg Is this movie titled bad teacher (2011)? Please answer yes or no. No
tt0993846.jpg Is this movie titled the wolf of wall street (2013)? Please answer yes or no. Yes
tt0993846.jpg Is this movie titled lucky number slevin (2006)? Please answer yes or no. No
tt1017460.jpg Is this movie directed by vincenzo natali? Please answer yes or no. Yes
tt1017460.jpg Is this movie directed by raja gosnell? Please answer yes or no. No
tt1057500.jpg Is this movie directed by clint eastwood? Please answer yes or no. Yes
tt1057500.jpg Is this movie directed by john cassavetes? Please answer yes or no. No
tt1068680.jpg Is this movie titled yes man (2008)? Please answer yes or no. Yes
tt1068680.jpg Is this movie titled rio bravo (1959)? Please answer yes or no. No
tt1099212.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt1099212.jpg Is this movie originated from the country or region of japan? Please answer yes or no. No
tt1119646.jpg Is this movie directed by todd phillips? Please answer yes or no. Yes
tt1119646.jpg Is this movie directed by franklin j. schaffner? Please answer yes or no. No
tt1124037.jpg Is this movie directed by gary ross? Please answer yes or no. Yes
tt1124037.jpg Is this movie directed by francis ford coppola? Please answer yes or no. No
tt1229822.jpg Is this movie titled jane eyre (2011)? Please answer yes or no. Yes
tt1229822.jpg Is this movie titled kiss kiss bang bang (2005)? Please answer yes or no. No
tt1284575.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt1284575.jpg Is this movie originated from the country or region of bulgaria? Please answer yes or no. No
tt1291150.jpg Is this movie directed by jonathan liebesman? Please answer yes or no. Yes
tt1291150.jpg Is this movie directed by masahide ichii? Please answer yes or no. No
tt1300851.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt1300851.jpg Is this movie originated from the country or region of germany? Please answer yes or no. No
tt1305806.jpg Is this movie titled the secret in their eyes (2009)? Please answer yes or no. Yes
tt1305806.jpg Is this movie titled ocean's eleven (2001)? Please answer yes or no. No
tt1343092.jpg Is this movie directed by baz luhrmann? Please answer yes or no. Yes
tt1343092.jpg Is this movie directed by lorenzo vigas? Please answer yes or no. No
tt1385826.jpg Is this movie titled the adjustment bureau (2011)? Please answer yes or no. Yes
tt1385826.jpg Is this movie titled the last of sheila (1973)? Please answer yes or no. No
tt1403865.jpg Is this movie directed by ethan coen,joel coen? Please answer yes or no. Yes
tt1403865.jpg Is this movie directed by john hillcoat? Please answer yes or no. No
tt1438176.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt1438176.jpg Is this movie originated from the country or region of west germany? Please answer yes or no. No
tt1454029.jpg Is this movie titled the help (2011)? Please answer yes or no. Yes
tt1454029.jpg Is this movie titled avatar (2009)? Please answer yes or no. No
tt1560747.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt1560747.jpg Is this movie originated from the country or region of canada? Please answer yes or no. No
tt1564367.jpg Is this movie titled just go with it (2011)? Please answer yes or no. Yes
tt1564367.jpg Is this movie titled journey to the west (2013)? Please answer yes or no. No
tt1606378.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt1606378.jpg Is this movie originated from the country or region of uk? Please answer yes or no. No
tt1637725.jpg Is this movie titled ted (2012)? Please answer yes or no. Yes
tt1637725.jpg Is this movie titled lost in space (1998)? Please answer yes or no. No
tt1670345.jpg Is this movie titled now you see me (2013)? Please answer yes or no. Yes
tt1670345.jpg Is this movie titled bicentennial man (1999)? Please answer yes or no. No
tt1703957.jpg Is this movie titled genius (2016)? Please answer yes or no. Yes
tt1703957.jpg Is this movie titled fan (2016)? Please answer yes or no. No
tt1722484.jpg Is this movie directed by hermine huntgeburth? Please answer yes or no. Yes
tt1722484.jpg Is this movie directed by todd phillips? Please answer yes or no. No
tt1800241.jpg Is this movie directed by david o. russell? Please answer yes or no. Yes
tt1800241.jpg Is this movie directed by raja gosnell? Please answer yes or no. No
tt1809398.jpg Is this movie titled unbroken (2014)? Please answer yes or no. Yes
tt1809398.jpg Is this movie titled home alone 3 (1997)? Please answer yes or no. No
tt1971325.jpg Is this movie originated from the country or region of bulgaria? Please answer yes or no. Yes
tt1971325.jpg Is this movie originated from the country or region of spain? Please answer yes or no. No
tt1974419.jpg Is this movie titled the neon demon (2016)? Please answer yes or no. Yes
tt1974419.jpg Is this movie titled man on fire (2004)? Please answer yes or no. No
tt2017561.jpg Is this movie directed by stephen chow,chi-kin kwok? Please answer yes or no. Yes
tt2017561.jpg Is this movie directed by craig gillespie? Please answer yes or no. No
tt2078768.jpg Is this movie originated from the country or region of hong kong? Please answer yes or no. Yes
tt2078768.jpg Is this movie originated from the country or region of venezuela? Please answer yes or no. No
tt2132285.jpg Is this movie titled the bling ring (2013)? Please answer yes or no. Yes
tt2132285.jpg Is this movie titled tropic thunder (2008)? Please answer yes or no. No
tt2238032.jpg Is this movie titled skiptrace (2016)? Please answer yes or no. Yes
tt2238032.jpg Is this movie titled the bourne identity (2002)? Please answer yes or no. No
tt2334873.jpg Is this movie titled blue jasmine (2013)? Please answer yes or no. Yes
tt2334873.jpg Is this movie titled a good day to die hard (2013)? Please answer yes or no. No
tt2788732.jpg Is this movie titled pete's dragon (2016)? Please answer yes or no. Yes
tt2788732.jpg Is this movie titled willow (1988)? Please answer yes or no. No
tt2802144.jpg Is this movie directed by matthew vaughn? Please answer yes or no. Yes
tt2802144.jpg Is this movie directed by dietrich brüggemann? Please answer yes or no. No
tt2908856.jpg Is this movie directed by israel horovitz? Please answer yes or no. Yes
tt2908856.jpg Is this movie directed by mike nichols? Please answer yes or no. No
tt2980516.jpg Is this movie titled the theory of everything (2014)? Please answer yes or no. Yes
tt2980516.jpg Is this movie titled stations of the cross (2014)? Please answer yes or no. No
tt3077214.jpg Is this movie titled suffragette (2015)? Please answer yes or no. Yes
tt3077214.jpg Is this movie titled east of eden (1955)? Please answer yes or no. No
tt3316960.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt3316960.jpg Is this movie originated from the country or region of west germany? Please answer yes or no. No
tt3465916.jpg Is this movie originated from the country or region of germany? Please answer yes or no. Yes
tt3465916.jpg Is this movie originated from the country or region of spain? Please answer yes or no. No
tt3495026.jpg Is this movie titled fan (2016)? Please answer yes or no. Yes
tt3495026.jpg Is this movie titled from afar (2015)? Please answer yes or no. No
tt3498820.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt3498820.jpg Is this movie originated from the country or region of germany? Please answer yes or no. No
tt3700804.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt3700804.jpg Is this movie originated from the country or region of uk? Please answer yes or no. No
tt3824458.jpg Is this movie titled tangerine (2015)? Please answer yes or no. Yes
tt3824458.jpg Is this movie titled miller's crossing (1990)? Please answer yes or no. No
tt3860916.jpg Is this movie directed by ben howling? Please answer yes or no. Yes
tt3860916.jpg Is this movie directed by quentin tarantino? Please answer yes or no. No
tt4046784.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt4046784.jpg Is this movie originated from the country or region of uk? Please answer yes or no. No
tt4242158.jpg Is this movie titled lost in hong kong (2015)? Please answer yes or no. Yes
tt4242158.jpg Is this movie titled harold and maude (1971)? Please answer yes or no. No
tt4273292.jpg Is this movie directed by babak anvari? Please answer yes or no. Yes
tt4273292.jpg Is this movie directed by ethan coen,joel coen? Please answer yes or no. No
tt4501454.jpg Is this movie directed by lorene scafaria? Please answer yes or no. Yes
tt4501454.jpg Is this movie directed by herbert ross? Please answer yes or no. No
tt4520364.jpg Is this movie directed by luke scott? Please answer yes or no. Yes
tt4520364.jpg Is this movie directed by dominic sena? Please answer yes or no. No
tt4651520.jpg Is this movie originated from the country or region of usa? Please answer yes or no. Yes
tt4651520.jpg Is this movie originated from the country or region of china? Please answer yes or no. No
tt4721400.jpg Is this movie directed by lorenzo vigas? Please answer yes or no. Yes
tt4721400.jpg Is this movie directed by jonathan liebesman? Please answer yes or no. No
tt4972062.jpg Is this movie directed by mike birbiglia? Please answer yes or no. Yes
tt4972062.jpg Is this movie directed by david fincher? Please answer yes or no. No
tt5564148.jpg Is this movie directed by masahide ichii? Please answer yes or no. Yes
tt5564148.jpg Is this movie directed by dietrich brüggemann? Please answer yes or no. No
tt5688868.jpg Is this movie titled primal rage: the legend of konga (2018)? Please answer yes or no. Yes
tt5688868.jpg Is this movie titled the neon demon (2016)? Please answer yes or no. No
tt5726086.jpg Is this movie directed by adam robitel? Please answer yes or no. Yes
tt5726086.jpg Is this movie directed by gore verbinski? Please answer yes or no. No
tt6788942.jpg Is this movie originated from the country or region of thailand? Please answer yes or no. Yes
tt6788942.jpg Is this movie originated from the country or region of argentina? Please answer yes or no. No
tt7055592.jpg Is this movie titled brotherhood of blades ii: the infernal battlefield (2017)? Please answer yes or no. Yes
tt7055592.jpg Is this movie titled stations of the cross (2014)? Please answer yes or no. No
tt7131870.jpg Is this movie directed by jing wu? Please answer yes or no. Yes
tt7131870.jpg Is this movie directed by david lynch? Please answer yes or no. No
internvl_chat/eval/mme/Your_Results/scene.txt 0 → 100644
View file @ 1d5a34cf
Places365_val_00000001.jpg Is this picture captured in a place of greenhouse indoor? Please answer yes or no. Yes
Places365_val_00000001.jpg Is this picture captured in a place of waiting room? Please answer yes or no. No
Places365_val_00000002.jpg Is this photo taken in a place of wet bar? Please answer yes or no. Yes
Places365_val_00000002.jpg Is this photo taken in a place of stage indoor? Please answer yes or no. No
Places365_val_00000003.jpg Is this picture captured in a place of clean room? Please answer yes or no. Yes
Places365_val_00000003.jpg Is this picture captured in a place of bookstore? Please answer yes or no. No
Places365_val_00000004.jpg Is this picture captured in a place of golf course? Please answer yes or no. Yes
Places365_val_00000004.jpg Is this picture captured in a place of computer room? Please answer yes or no. No
Places365_val_00000005.jpg Is this picture captured in a place of rock arch? Please answer yes or no. Yes
Places365_val_00000005.jpg Is this picture captured in a place of desert sand? Please answer yes or no. No
Places365_val_00000006.jpg Is this photo taken in a place of corridor? Please answer yes or no. Yes
Places365_val_00000006.jpg Is this photo taken in a place of jail cell? Please answer yes or no. No
Places365_val_00000007.jpg Does this image describe a place of canyon? Please answer yes or no. Yes
Places365_val_00000007.jpg Does this image describe a place of basement? Please answer yes or no. No
Places365_val_00000008.jpg Is this picture captured in a place of dining room? Please answer yes or no. Yes
Places365_val_00000008.jpg Is this picture captured in a place of ball pit? Please answer yes or no. No
Places365_val_00000009.jpg Is this picture captured in a place of forest broadleaf? Please answer yes or no. Yes
Places365_val_00000009.jpg Is this picture captured in a place of swamp? Please answer yes or no. No
Places365_val_00000010.jpg Does this image describe a place of shopping mall indoor? Please answer yes or no. Yes
Places365_val_00000010.jpg Does this image describe a place of mosque outdoor? Please answer yes or no. No
Places365_val_00000011.jpg Is this photo taken in a place of baseball field? Please answer yes or no. Yes
Places365_val_00000011.jpg Is this photo taken in a place of bus station indoor? Please answer yes or no. No
Places365_val_00000012.jpg Does this image describe a place of campus? Please answer yes or no. Yes
Places365_val_00000012.jpg Does this image describe a place of sandbox? Please answer yes or no. No
Places365_val_00000013.jpg Is this picture captured in a place of beach house? Please answer yes or no. Yes
Places365_val_00000013.jpg Is this picture captured in a place of alcove? Please answer yes or no. No
Places365_val_00000014.jpg Does this image describe a place of art gallery? Please answer yes or no. Yes
Places365_val_00000014.jpg Does this image describe a place of volleyball court outdoor? Please answer yes or no. No
Places365_val_00000015.jpg Does this image describe a place of bus interior? Please answer yes or no. Yes
Places365_val_00000015.jpg Does this image describe a place of butchers shop? Please answer yes or no. No
Places365_val_00000016.jpg Does this image describe a place of gymnasium indoor? Please answer yes or no. Yes
Places365_val_00000016.jpg Does this image describe a place of crevasse? Please answer yes or no. No
Places365_val_00000017.jpg Is this picture captured in a place of glacier? Please answer yes or no. Yes
Places365_val_00000017.jpg Is this picture captured in a place of fishpond? Please answer yes or no. No
Places365_val_00000018.jpg Is this photo taken in a place of nursing home? Please answer yes or no. Yes
Places365_val_00000018.jpg Is this photo taken in a place of server room? Please answer yes or no. No
Places365_val_00000019.jpg Does this image describe a place of storage room? Please answer yes or no. Yes
Places365_val_00000019.jpg Does this image describe a place of aquarium? Please answer yes or no. No
Places365_val_00000020.jpg Is this picture captured in a place of florist shop indoor? Please answer yes or no. Yes
Places365_val_00000020.jpg Is this picture captured in a place of office? Please answer yes or no. No
Places365_val_00000021.jpg Is this picture captured in a place of restaurant kitchen? Please answer yes or no. Yes
Places365_val_00000021.jpg Is this picture captured in a place of garage outdoor? Please answer yes or no. No
Places365_val_00000022.jpg Is this photo taken in a place of recreation room? Please answer yes or no. Yes
Places365_val_00000022.jpg Is this photo taken in a place of runway? Please answer yes or no. No
Places365_val_00000023.jpg Does this image describe a place of schoolhouse? Please answer yes or no. Yes
Places365_val_00000023.jpg Does this image describe a place of raft? Please answer yes or no. No
Places365_val_00000024.jpg Does this image describe a place of amusement arcade? Please answer yes or no. Yes
Places365_val_00000024.jpg Does this image describe a place of medina? Please answer yes or no. No
Places365_val_00000025.jpg Is this photo taken in a place of ocean? Please answer yes or no. Yes
Places365_val_00000025.jpg Is this photo taken in a place of sauna? Please answer yes or no. No
Places365_val_00000026.jpg Does this image describe a place of river? Please answer yes or no. Yes
Places365_val_00000026.jpg Does this image describe a place of corral? Please answer yes or no. No
Places365_val_00000027.jpg Is this photo taken in a place of pub indoor? Please answer yes or no. Yes
Places365_val_00000027.jpg Is this photo taken in a place of basement? Please answer yes or no. No
Places365_val_00000028.jpg Is this picture captured in a place of lecture room? Please answer yes or no. Yes
Places365_val_00000028.jpg Is this picture captured in a place of aqueduct? Please answer yes or no. No
Places365_val_00000029.jpg Is this photo taken in a place of japanese garden? Please answer yes or no. Yes
Places365_val_00000029.jpg Is this photo taken in a place of pharmacy? Please answer yes or no. No
Places365_val_00000030.jpg Does this image describe a place of motel? Please answer yes or no. Yes
Places365_val_00000030.jpg Does this image describe a place of toyshop? Please answer yes or no. No
Places365_val_00000031.jpg Does this image describe a place of assembly line? Please answer yes or no. Yes
Places365_val_00000031.jpg Does this image describe a place of beer garden? Please answer yes or no. No
Places365_val_00000032.jpg Does this image describe a place of campsite? Please answer yes or no. Yes
Places365_val_00000032.jpg Does this image describe a place of elevator shaft? Please answer yes or no. No
Places365_val_00000033.jpg Is this photo taken in a place of florist shop indoor? Please answer yes or no. Yes
Places365_val_00000033.jpg Is this photo taken in a place of sushi bar? Please answer yes or no. No
Places365_val_00000034.jpg Does this image describe a place of racecourse? Please answer yes or no. Yes
Places365_val_00000034.jpg Does this image describe a place of galley? Please answer yes or no. No
Places365_val_00000035.jpg Is this photo taken in a place of kennel outdoor? Please answer yes or no. Yes
Places365_val_00000035.jpg Is this photo taken in a place of bathroom? Please answer yes or no. No
Places365_val_00000036.jpg Is this picture captured in a place of bus interior? Please answer yes or no. Yes
Places365_val_00000036.jpg Is this picture captured in a place of mansion? Please answer yes or no. No
Places365_val_00000037.jpg Does this image describe a place of dressing room? Please answer yes or no. Yes
Places365_val_00000037.jpg Does this image describe a place of chemistry lab? Please answer yes or no. No
Places365_val_00000038.jpg Does this image describe a place of wind farm? Please answer yes or no. Yes
Places365_val_00000038.jpg Does this image describe a place of volleyball court outdoor? Please answer yes or no. No
Places365_val_00000039.jpg Is this picture captured in a place of auto factory? Please answer yes or no. Yes
Places365_val_00000039.jpg Is this picture captured in a place of tree farm? Please answer yes or no. No
Places365_val_00000040.jpg Is this picture captured in a place of beach? Please answer yes or no. Yes
Places365_val_00000040.jpg Is this picture captured in a place of library outdoor? Please answer yes or no. No
Places365_val_00000041.jpg Is this photo taken in a place of mountain? Please answer yes or no. Yes
Places365_val_00000041.jpg Is this photo taken in a place of formal garden? Please answer yes or no. No
Places365_val_00000042.jpg Does this image describe a place of pavilion? Please answer yes or no. Yes
Places365_val_00000042.jpg Does this image describe a place of stable? Please answer yes or no. No
Places365_val_00000043.jpg Does this image describe a place of tree house? Please answer yes or no. Yes
Places365_val_00000043.jpg Does this image describe a place of patio? Please answer yes or no. No
Places365_val_00000044.jpg Is this photo taken in a place of coffee shop? Please answer yes or no. Yes
Places365_val_00000044.jpg Is this photo taken in a place of elevator lobby? Please answer yes or no. No
Places365_val_00000045.jpg Is this photo taken in a place of skyscraper? Please answer yes or no. Yes
Places365_val_00000045.jpg Is this photo taken in a place of forest broadleaf? Please answer yes or no. No
Places365_val_00000046.jpg Does this image describe a place of street? Please answer yes or no. Yes
Places365_val_00000046.jpg Does this image describe a place of golf course? Please answer yes or no. No
Places365_val_00000047.jpg Is this picture captured in a place of alley? Please answer yes or no. Yes
Places365_val_00000047.jpg Is this picture captured in a place of boardwalk? Please answer yes or no. No
Places365_val_00000048.jpg Is this photo taken in a place of supermarket? Please answer yes or no. Yes
Places365_val_00000048.jpg Is this photo taken in a place of creek? Please answer yes or no. No
Places365_val_00000049.jpg Is this picture captured in a place of arena hockey? Please answer yes or no. Yes
Places365_val_00000049.jpg Is this picture captured in a place of ski slope? Please answer yes or no. No
Places365_val_00000050.jpg Is this picture captured in a place of zen garden? Please answer yes or no. Yes
Places365_val_00000050.jpg Is this picture captured in a place of bazaar indoor? Please answer yes or no. No
Places365_val_00000051.jpg Is this picture captured in a place of library indoor? Please answer yes or no. Yes
Places365_val_00000051.jpg Is this picture captured in a place of orchard? Please answer yes or no. No
Places365_val_00000052.jpg Is this photo taken in a place of ice skating rink indoor? Please answer yes or no. Yes
Places365_val_00000052.jpg Is this photo taken in a place of wind farm? Please answer yes or no. No
Places365_val_00000053.jpg Is this picture captured in a place of arena hockey? Please answer yes or no. Yes
Places365_val_00000053.jpg Is this picture captured in a place of loading dock? Please answer yes or no. No
Places365_val_00000054.jpg Does this image describe a place of chalet? Please answer yes or no. Yes
Places365_val_00000054.jpg Does this image describe a place of tower? Please answer yes or no. No
Places365_val_00000055.jpg Does this image describe a place of bedchamber? Please answer yes or no. Yes
Places365_val_00000055.jpg Does this image describe a place of music studio? Please answer yes or no. No
Places365_val_00000056.jpg Is this photo taken in a place of botanical garden? Please answer yes or no. Yes
Places365_val_00000056.jpg Is this photo taken in a place of legislative chamber? Please answer yes or no. No
Places365_val_00000057.jpg Is this picture captured in a place of galley? Please answer yes or no. Yes
Places365_val_00000057.jpg Is this picture captured in a place of physics laboratory? Please answer yes or no. No
Places365_val_00000058.jpg Does this image describe a place of department store? Please answer yes or no. Yes
Places365_val_00000058.jpg Does this image describe a place of rope bridge? Please answer yes or no. No
Places365_val_00000059.jpg Is this photo taken in a place of schoolhouse? Please answer yes or no. Yes
Places365_val_00000059.jpg Is this photo taken in a place of hangar outdoor? Please answer yes or no. No
Places365_val_00000060.jpg Is this photo taken in a place of atrium public? Please answer yes or no. Yes
Places365_val_00000060.jpg Is this photo taken in a place of subway station platform? Please answer yes or no. No
Places365_val_00000061.jpg Is this photo taken in a place of dam? Please answer yes or no. Yes
Places365_val_00000061.jpg Is this photo taken in a place of ice skating rink indoor? Please answer yes or no. No
Places365_val_00000062.jpg Does this image describe a place of army base? Please answer yes or no. Yes
Places365_val_00000062.jpg Does this image describe a place of train station platform? Please answer yes or no. No
Places365_val_00000063.jpg Does this image describe a place of gas station? Please answer yes or no. Yes
Places365_val_00000063.jpg Does this image describe a place of doorway outdoor? Please answer yes or no. No
Places365_val_00000064.jpg Does this image describe a place of ballroom? Please answer yes or no. Yes
Places365_val_00000064.jpg Does this image describe a place of corn field? Please answer yes or no. No
Places365_val_00000065.jpg Does this image describe a place of biology laboratory? Please answer yes or no. Yes
Places365_val_00000065.jpg Does this image describe a place of candy store? Please answer yes or no. No
Places365_val_00000066.jpg Does this image describe a place of pantry? Please answer yes or no. Yes
Places365_val_00000066.jpg Does this image describe a place of lighthouse? Please answer yes or no. No
Places365_val_00000067.jpg Is this picture captured in a place of greenhouse indoor? Please answer yes or no. Yes
Places365_val_00000067.jpg Is this picture captured in a place of pasture? Please answer yes or no. No
Places365_val_00000068.jpg Is this photo taken in a place of kasbah? Please answer yes or no. Yes
Places365_val_00000068.jpg Is this photo taken in a place of swimming hole? Please answer yes or no. No
Places365_val_00000069.jpg Is this picture captured in a place of mausoleum? Please answer yes or no. Yes
Places365_val_00000069.jpg Is this picture captured in a place of alcove? Please answer yes or no. No
Places365_val_00000070.jpg Is this picture captured in a place of booth indoor? Please answer yes or no. Yes
Places365_val_00000070.jpg Is this picture captured in a place of bazaar indoor? Please answer yes or no. No
Places365_val_00000071.jpg Is this picture captured in a place of youth hostel? Please answer yes or no. Yes
Places365_val_00000071.jpg Is this picture captured in a place of landfill? Please answer yes or no. No
Places365_val_00000072.jpg Does this image describe a place of fire escape? Please answer yes or no. Yes
Places365_val_00000072.jpg Does this image describe a place of playroom? Please answer yes or no. No
Places365_val_00000073.jpg Is this photo taken in a place of rice paddy? Please answer yes or no. Yes
Places365_val_00000073.jpg Is this photo taken in a place of bus station indoor? Please answer yes or no. No
Places365_val_00000074.jpg Is this photo taken in a place of mausoleum? Please answer yes or no. Yes
Places365_val_00000074.jpg Is this photo taken in a place of train station platform? Please answer yes or no. No
Places365_val_00000075.jpg Is this photo taken in a place of entrance hall? Please answer yes or no. Yes
Places365_val_00000075.jpg Is this photo taken in a place of residential neighborhood? Please answer yes or no. No
Places365_val_00000076.jpg Is this photo taken in a place of bakery shop? Please answer yes or no. Yes
Places365_val_00000076.jpg Is this photo taken in a place of doorway outdoor? Please answer yes or no. No
Places365_val_00000077.jpg Does this image describe a place of escalator indoor? Please answer yes or no. Yes
Places365_val_00000077.jpg Does this image describe a place of ice shelf? Please answer yes or no. No
Places365_val_00000078.jpg Is this picture captured in a place of elevator shaft? Please answer yes or no. Yes
Places365_val_00000078.jpg Is this picture captured in a place of fire escape? Please answer yes or no. No
Places365_val_00000079.jpg Is this picture captured in a place of industrial area? Please answer yes or no. Yes
Places365_val_00000079.jpg Is this picture captured in a place of botanical garden? Please answer yes or no. No
Places365_val_00000080.jpg Is this photo taken in a place of escalator indoor? Please answer yes or no. Yes
Places365_val_00000080.jpg Is this photo taken in a place of coffee shop? Please answer yes or no. No
Places365_val_00000081.jpg Does this image describe a place of operating room? Please answer yes or no. Yes
Places365_val_00000081.jpg Does this image describe a place of house? Please answer yes or no. No
Places365_val_00000082.jpg Is this photo taken in a place of mosque outdoor? Please answer yes or no. Yes
Places365_val_00000082.jpg Is this photo taken in a place of recreation room? Please answer yes or no. No
Places365_val_00000083.jpg Is this photo taken in a place of cottage? Please answer yes or no. Yes
Places365_val_00000083.jpg Is this photo taken in a place of assembly line? Please answer yes or no. No
Places365_val_00000084.jpg Does this image describe a place of courtyard? Please answer yes or no. Yes
Places365_val_00000084.jpg Does this image describe a place of greenhouse outdoor? Please answer yes or no. No
Places365_val_00000085.jpg Is this photo taken in a place of flea market indoor? Please answer yes or no. Yes
Places365_val_00000085.jpg Is this photo taken in a place of pavilion? Please answer yes or no. No
Places365_val_00000086.jpg Does this image describe a place of banquet hall? Please answer yes or no. Yes
Places365_val_00000086.jpg Does this image describe a place of storage room? Please answer yes or no. No
Places365_val_00000087.jpg Is this photo taken in a place of auto factory? Please answer yes or no. Yes
Places365_val_00000087.jpg Is this photo taken in a place of motel? Please answer yes or no. No
Places365_val_00000088.jpg Is this photo taken in a place of cockpit? Please answer yes or no. Yes
Places365_val_00000088.jpg Is this photo taken in a place of gift shop? Please answer yes or no. No
Places365_val_00000089.jpg Is this photo taken in a place of martial arts gym? Please answer yes or no. Yes
Places365_val_00000089.jpg Is this photo taken in a place of atrium public? Please answer yes or no. No
Places365_val_00000090.jpg Is this picture captured in a place of general store outdoor? Please answer yes or no. Yes
Places365_val_00000090.jpg Is this picture captured in a place of bazaar outdoor? Please answer yes or no. No
Places365_val_00000091.jpg Is this photo taken in a place of motel? Please answer yes or no. Yes
Places365_val_00000091.jpg Is this photo taken in a place of chemistry lab? Please answer yes or no. No
Places365_val_00000092.jpg Is this photo taken in a place of playground? Please answer yes or no. Yes
Places365_val_00000092.jpg Is this photo taken in a place of snowfield? Please answer yes or no. No
Places365_val_00000093.jpg Is this picture captured in a place of basement? Please answer yes or no. Yes
Places365_val_00000093.jpg Is this picture captured in a place of jewelry shop? Please answer yes or no. No
Places365_val_00000094.jpg Is this picture captured in a place of playroom? Please answer yes or no. Yes
Places365_val_00000094.jpg Is this picture captured in a place of pier? Please answer yes or no. No
Places365_val_00000095.jpg Is this photo taken in a place of fabric store? Please answer yes or no. Yes
Places365_val_00000095.jpg Is this photo taken in a place of beauty salon? Please answer yes or no. No
Places365_val_00000096.jpg Is this picture captured in a place of forest path? Please answer yes or no. Yes
Places365_val_00000096.jpg Is this picture captured in a place of balcony interior? Please answer yes or no. No
Places365_val_00000097.jpg Does this image describe a place of hangar indoor? Please answer yes or no. Yes
Places365_val_00000097.jpg Does this image describe a place of vegetable garden? Please answer yes or no. No
Places365_val_00000098.jpg Does this image describe a place of crosswalk? Please answer yes or no. Yes
Places365_val_00000098.jpg Does this image describe a place of snowfield? Please answer yes or no. No
Places365_val_00000099.jpg Does this image describe a place of physics laboratory? Please answer yes or no. Yes
Places365_val_00000099.jpg Does this image describe a place of stage indoor? Please answer yes or no. No
Places365_val_00000100.jpg Is this picture captured in a place of pub indoor? Please answer yes or no. Yes
Places365_val_00000100.jpg Is this picture captured in a place of pier? Please answer yes or no. No
Places365_val_00000101.jpg Is this photo taken in a place of schoolhouse? Please answer yes or no. Yes
Places365_val_00000101.jpg Is this photo taken in a place of vineyard? Please answer yes or no. No
Places365_val_00000102.jpg Is this photo taken in a place of industrial area? Please answer yes or no. Yes
Places365_val_00000102.jpg Is this photo taken in a place of street? Please answer yes or no. No
Places365_val_00000103.jpg Is this picture captured in a place of baseball field? Please answer yes or no. Yes
Places365_val_00000103.jpg Is this picture captured in a place of airfield? Please answer yes or no. No
Places365_val_00000104.jpg Is this picture captured in a place of ice floe? Please answer yes or no. Yes
Places365_val_00000104.jpg Is this picture captured in a place of kindergarden classroom? Please answer yes or no. No
Places365_val_00000105.jpg Is this picture captured in a place of train interior? Please answer yes or no. Yes
Places365_val_00000105.jpg Is this picture captured in a place of movie theater indoor? Please answer yes or no. No
Places365_val_00000106.jpg Is this picture captured in a place of desert sand? Please answer yes or no. Yes
Places365_val_00000106.jpg Is this picture captured in a place of burial chamber? Please answer yes or no. No
Places365_val_00000107.jpg Is this photo taken in a place of castle? Please answer yes or no. Yes
Places365_val_00000107.jpg Is this photo taken in a place of computer room? Please answer yes or no. No
Places365_val_00000108.jpg Does this image describe a place of burial chamber? Please answer yes or no. Yes
Places365_val_00000108.jpg Does this image describe a place of oilrig? Please answer yes or no. No
Places365_val_00000109.jpg Is this photo taken in a place of nursing home? Please answer yes or no. Yes
Places365_val_00000109.jpg Is this photo taken in a place of amusement park? Please answer yes or no. No
Places365_val_00000110.jpg Does this image describe a place of church indoor? Please answer yes or no. Yes
Places365_val_00000110.jpg Does this image describe a place of general store outdoor? Please answer yes or no. No
Places365_val_00000111.jpg Is this picture captured in a place of palace? Please answer yes or no. Yes
Places365_val_00000111.jpg Is this picture captured in a place of tree house? Please answer yes or no. No
Places365_val_00000112.jpg Does this image describe a place of bus station indoor? Please answer yes or no. Yes
Places365_val_00000112.jpg Does this image describe a place of hardware store? Please answer yes or no. No
Places365_val_00000113.jpg Is this picture captured in a place of atrium public? Please answer yes or no. Yes
Places365_val_00000113.jpg Is this picture captured in a place of music studio? Please answer yes or no. No
Places365_val_00000114.jpg Is this photo taken in a place of butte? Please answer yes or no. Yes
Places365_val_00000114.jpg Is this photo taken in a place of valley? Please answer yes or no. No
Places365_val_00000115.jpg Is this picture captured in a place of zen garden? Please answer yes or no. Yes
Places365_val_00000115.jpg Is this picture captured in a place of banquet hall? Please answer yes or no. No
Places365_val_00000116.jpg Is this picture captured in a place of racecourse? Please answer yes or no. Yes
Places365_val_00000116.jpg Is this picture captured in a place of basketball court indoor? Please answer yes or no. No
Places365_val_00000117.jpg Is this picture captured in a place of cliff? Please answer yes or no. Yes
Places365_val_00000117.jpg Is this picture captured in a place of sauna? Please answer yes or no. No
Places365_val_00000118.jpg Is this photo taken in a place of laundromat? Please answer yes or no. Yes
Places365_val_00000118.jpg Is this photo taken in a place of archive? Please answer yes or no. No
Places365_val_00000119.jpg Does this image describe a place of wave? Please answer yes or no. Yes
Places365_val_00000119.jpg Does this image describe a place of shed? Please answer yes or no. No
Places365_val_00000120.jpg Is this photo taken in a place of legislative chamber? Please answer yes or no. Yes
Places365_val_00000120.jpg Is this photo taken in a place of nursery? Please answer yes or no. No
Places365_val_00000121.jpg Is this photo taken in a place of elevator shaft? Please answer yes or no. Yes
Places365_val_00000121.jpg Is this photo taken in a place of berth? Please answer yes or no. No
Places365_val_00000122.jpg Is this photo taken in a place of hayfield? Please answer yes or no. Yes
Places365_val_00000122.jpg Is this photo taken in a place of skyscraper? Please answer yes or no. No
Places365_val_00000123.jpg Is this photo taken in a place of hospital room? Please answer yes or no. Yes
Places365_val_00000123.jpg Is this photo taken in a place of mosque outdoor? Please answer yes or no. No
Places365_val_00000124.jpg Does this image describe a place of swimming hole? Please answer yes or no. Yes
Places365_val_00000124.jpg Does this image describe a place of television room? Please answer yes or no. No
Places365_val_00000125.jpg Is this picture captured in a place of biology laboratory? Please answer yes or no. Yes
Places365_val_00000125.jpg Is this picture captured in a place of lake natural? Please answer yes or no. No
Places365_val_00000126.jpg Is this picture captured in a place of church indoor? Please answer yes or no. Yes
Places365_val_00000126.jpg Is this picture captured in a place of server room? Please answer yes or no. No
Places365_val_00000127.jpg Is this picture captured in a place of temple asia? Please answer yes or no. Yes
Places365_val_00000127.jpg Is this picture captured in a place of ball pit? Please answer yes or no. No
Places365_val_00000128.jpg Is this photo taken in a place of landfill? Please answer yes or no. Yes
Places365_val_00000128.jpg Is this photo taken in a place of viaduct? Please answer yes or no. No
Places365_val_00000129.jpg Does this image describe a place of arena hockey? Please answer yes or no. Yes
Places365_val_00000129.jpg Does this image describe a place of clothing store? Please answer yes or no. No
Places365_val_00000130.jpg Is this photo taken in a place of baseball field? Please answer yes or no. Yes
Places365_val_00000130.jpg Is this photo taken in a place of art gallery? Please answer yes or no. No
Places365_val_00000131.jpg Is this picture captured in a place of gazebo exterior? Please answer yes or no. Yes
Places365_val_00000131.jpg Is this picture captured in a place of driveway? Please answer yes or no. No
Places365_val_00000132.jpg Does this image describe a place of greenhouse outdoor? Please answer yes or no. Yes
Places365_val_00000132.jpg Does this image describe a place of stage indoor? Please answer yes or no. No
Places365_val_00000133.jpg Is this picture captured in a place of beauty salon? Please answer yes or no. Yes
Places365_val_00000133.jpg Is this picture captured in a place of junkyard? Please answer yes or no. No
Places365_val_00000134.jpg Does this image describe a place of ice shelf? Please answer yes or no. Yes
Places365_val_00000134.jpg Does this image describe a place of vegetable garden? Please answer yes or no. No
Places365_val_00000135.jpg Is this picture captured in a place of islet? Please answer yes or no. Yes
Places365_val_00000135.jpg Is this picture captured in a place of recreation room? Please answer yes or no. No
Places365_val_00000136.jpg Is this picture captured in a place of railroad track? Please answer yes or no. Yes
Places365_val_00000136.jpg Is this picture captured in a place of department store? Please answer yes or no. No
Places365_val_00000137.jpg Is this photo taken in a place of bazaar outdoor? Please answer yes or no. Yes
Places365_val_00000137.jpg Is this photo taken in a place of office? Please answer yes or no. No
Places365_val_00000138.jpg Is this photo taken in a place of basement? Please answer yes or no. Yes
Places365_val_00000138.jpg Is this photo taken in a place of banquet hall? Please answer yes or no. No
Places365_val_00000139.jpg Is this picture captured in a place of television studio? Please answer yes or no. Yes
Places365_val_00000139.jpg Is this picture captured in a place of general store indoor? Please answer yes or no. No
Places365_val_00000140.jpg Is this picture captured in a place of mezzanine? Please answer yes or no. Yes
Places365_val_00000140.jpg Is this picture captured in a place of bedroom? Please answer yes or no. No
Places365_val_00000141.jpg Does this image describe a place of roof garden? Please answer yes or no. Yes
Places365_val_00000141.jpg Does this image describe a place of television room? Please answer yes or no. No
Places365_val_00000142.jpg Does this image describe a place of restaurant kitchen? Please answer yes or no. Yes
Places365_val_00000142.jpg Does this image describe a place of fabric store? Please answer yes or no. No
Places365_val_00000143.jpg Is this photo taken in a place of house? Please answer yes or no. Yes
Places365_val_00000143.jpg Is this photo taken in a place of swimming hole? Please answer yes or no. No
Places365_val_00000144.jpg Is this picture captured in a place of vineyard? Please answer yes or no. Yes
Places365_val_00000144.jpg Is this picture captured in a place of castle? Please answer yes or no. No
Places365_val_00000145.jpg Is this picture captured in a place of driveway? Please answer yes or no. Yes
Places365_val_00000145.jpg Is this picture captured in a place of badlands? Please answer yes or no. No
Places365_val_00000146.jpg Does this image describe a place of chemistry lab? Please answer yes or no. Yes
Places365_val_00000146.jpg Does this image describe a place of industrial area? Please answer yes or no. No
Places365_val_00000147.jpg Does this image describe a place of cabin outdoor? Please answer yes or no. Yes
Places365_val_00000147.jpg Does this image describe a place of creek? Please answer yes or no. No
Places365_val_00000148.jpg Is this photo taken in a place of restaurant? Please answer yes or no. Yes
Places365_val_00000148.jpg Is this photo taken in a place of apartment building outdoor? Please answer yes or no. No
Places365_val_00000149.jpg Is this picture captured in a place of auditorium? Please answer yes or no. Yes
Places365_val_00000149.jpg Is this picture captured in a place of rock arch? Please answer yes or no. No
Places365_val_00000150.jpg Is this photo taken in a place of street? Please answer yes or no. Yes
Places365_val_00000150.jpg Is this photo taken in a place of toyshop? Please answer yes or no. No
Places365_val_00000151.jpg Does this image describe a place of lagoon? Please answer yes or no. Yes
Places365_val_00000151.jpg Does this image describe a place of aquarium? Please answer yes or no. No
Places365_val_00000152.jpg Is this picture captured in a place of hangar indoor? Please answer yes or no. Yes
Places365_val_00000152.jpg Is this picture captured in a place of street? Please answer yes or no. No
Places365_val_00000153.jpg Does this image describe a place of bar? Please answer yes or no. Yes
Places365_val_00000153.jpg Does this image describe a place of tree house? Please answer yes or no. No
Places365_val_00000154.jpg Is this picture captured in a place of classroom? Please answer yes or no. Yes
Places365_val_00000154.jpg Is this picture captured in a place of wheat field? Please answer yes or no. No
Places365_val_00000155.jpg Does this image describe a place of creek? Please answer yes or no. Yes
Places365_val_00000155.jpg Does this image describe a place of youth hostel? Please answer yes or no. No
Places365_val_00000156.jpg Does this image describe a place of stable? Please answer yes or no. Yes
Places365_val_00000156.jpg Does this image describe a place of junkyard? Please answer yes or no. No
Places365_val_00000157.jpg Is this picture captured in a place of swamp? Please answer yes or no. Yes
Places365_val_00000157.jpg Is this picture captured in a place of village? Please answer yes or no. No
Places365_val_00000158.jpg Is this picture captured in a place of ice shelf? Please answer yes or no. Yes
Places365_val_00000158.jpg Is this picture captured in a place of motel? Please answer yes or no. No
Places365_val_00000159.jpg Is this picture captured in a place of train interior? Please answer yes or no. Yes
Places365_val_00000159.jpg Is this picture captured in a place of jewelry shop? Please answer yes or no. No
Places365_val_00000160.jpg Does this image describe a place of windmill? Please answer yes or no. Yes
Places365_val_00000160.jpg Does this image describe a place of lake natural? Please answer yes or no. No
Places365_val_00000161.jpg Is this photo taken in a place of archive? Please answer yes or no. Yes
Places365_val_00000161.jpg Is this photo taken in a place of village? Please answer yes or no. No
Places365_val_00000162.jpg Does this image describe a place of train station platform? Please answer yes or no. Yes
Places365_val_00000162.jpg Does this image describe a place of manufactured home? Please answer yes or no. No
Places365_val_00000163.jpg Is this photo taken in a place of chalet? Please answer yes or no. Yes
Places365_val_00000163.jpg Is this photo taken in a place of throne room? Please answer yes or no. No
Places365_val_00000164.jpg Does this image describe a place of lake natural? Please answer yes or no. Yes
Places365_val_00000164.jpg Does this image describe a place of athletic field outdoor? Please answer yes or no. No
Places365_val_00000165.jpg Is this picture captured in a place of playground? Please answer yes or no. Yes
Places365_val_00000165.jpg Is this picture captured in a place of viaduct? Please answer yes or no. No
Places365_val_00000166.jpg Is this photo taken in a place of dorm room? Please answer yes or no. Yes
Places365_val_00000166.jpg Is this photo taken in a place of ski resort? Please answer yes or no. No
Places365_val_00000167.jpg Is this picture captured in a place of flea market indoor? Please answer yes or no. Yes
Places365_val_00000167.jpg Is this picture captured in a place of cliff? Please answer yes or no. No
Places365_val_00000168.jpg Does this image describe a place of airplane cabin? Please answer yes or no. Yes
Places365_val_00000168.jpg Does this image describe a place of rock arch? Please answer yes or no. No
Places365_val_00000169.jpg Is this picture captured in a place of drugstore? Please answer yes or no. Yes
Places365_val_00000169.jpg Is this picture captured in a place of canyon? Please answer yes or no. No
Places365_val_00000170.jpg Is this picture captured in a place of pharmacy? Please answer yes or no. Yes
Places365_val_00000170.jpg Is this picture captured in a place of orchestra pit? Please answer yes or no. No
Places365_val_00000171.jpg Is this picture captured in a place of greenhouse outdoor? Please answer yes or no. Yes
Places365_val_00000171.jpg Is this picture captured in a place of archaelogical excavation? Please answer yes or no. No
Places365_val_00000172.jpg Is this photo taken in a place of bathroom? Please answer yes or no. Yes
Places365_val_00000172.jpg Is this photo taken in a place of auto factory? Please answer yes or no. No
Places365_val_00000173.jpg Is this photo taken in a place of shoe shop? Please answer yes or no. Yes
Places365_val_00000173.jpg Is this photo taken in a place of car interior? Please answer yes or no. No
Places365_val_00000174.jpg Does this image describe a place of moat water? Please answer yes or no. Yes
Places365_val_00000174.jpg Does this image describe a place of marsh? Please answer yes or no. No
Places365_val_00000175.jpg Does this image describe a place of vegetable garden? Please answer yes or no. Yes
Places365_val_00000175.jpg Does this image describe a place of parking garage indoor? Please answer yes or no. No
Places365_val_00000176.jpg Is this picture captured in a place of bowling alley? Please answer yes or no. Yes
Places365_val_00000176.jpg Is this picture captured in a place of hotel outdoor? Please answer yes or no. No
Places365_val_00000177.jpg Does this image describe a place of embassy? Please answer yes or no. Yes
Places365_val_00000177.jpg Does this image describe a place of chemistry lab? Please answer yes or no. No
Places365_val_00000178.jpg Is this photo taken in a place of youth hostel? Please answer yes or no. Yes
Places365_val_00000178.jpg Is this photo taken in a place of hangar indoor? Please answer yes or no. No
Places365_val_00000179.jpg Does this image describe a place of hot spring? Please answer yes or no. Yes
Places365_val_00000179.jpg Does this image describe a place of mausoleum? Please answer yes or no. No
Places365_val_00000180.jpg Does this image describe a place of cottage? Please answer yes or no. Yes
Places365_val_00000180.jpg Does this image describe a place of parking garage indoor? Please answer yes or no. No
Places365_val_00000181.jpg Does this image describe a place of general store outdoor? Please answer yes or no. Yes
Places365_val_00000181.jpg Does this image describe a place of hotel room? Please answer yes or no. No
Places365_val_00000182.jpg Is this photo taken in a place of chalet? Please answer yes or no. Yes
Places365_val_00000182.jpg Is this photo taken in a place of dorm room? Please answer yes or no. No
Places365_val_00000183.jpg Is this photo taken in a place of flea market indoor? Please answer yes or no. Yes
Places365_val_00000183.jpg Is this photo taken in a place of tree house? Please answer yes or no. No
Places365_val_00000184.jpg Is this photo taken in a place of shower? Please answer yes or no. Yes
Places365_val_00000184.jpg Is this photo taken in a place of living room? Please answer yes or no. No
Places365_val_00000185.jpg Is this picture captured in a place of supermarket? Please answer yes or no. Yes
Places365_val_00000185.jpg Is this picture captured in a place of orchard? Please answer yes or no. No
Places365_val_00000186.jpg Is this picture captured in a place of bowling alley? Please answer yes or no. Yes
Places365_val_00000186.jpg Is this picture captured in a place of ice skating rink outdoor? Please answer yes or no. No
Places365_val_00000187.jpg Is this picture captured in a place of barn? Please answer yes or no. Yes
Places365_val_00000187.jpg Is this picture captured in a place of lobby? Please answer yes or no. No
Places365_val_00000188.jpg Is this photo taken in a place of japanese garden? Please answer yes or no. Yes
Places365_val_00000188.jpg Is this photo taken in a place of hangar indoor? Please answer yes or no. No
Places365_val_00000189.jpg Is this picture captured in a place of swimming hole? Please answer yes or no. Yes
Places365_val_00000189.jpg Is this picture captured in a place of bedroom? Please answer yes or no. No
Places365_val_00000190.jpg Is this picture captured in a place of pizzeria? Please answer yes or no. Yes
Places365_val_00000190.jpg Is this picture captured in a place of computer room? Please answer yes or no. No
Places365_val_00000191.jpg Does this image describe a place of volleyball court outdoor? Please answer yes or no. Yes
Places365_val_00000191.jpg Does this image describe a place of church indoor? Please answer yes or no. No
Places365_val_00000192.jpg Is this photo taken in a place of market indoor? Please answer yes or no. Yes
Places365_val_00000192.jpg Is this photo taken in a place of home office? Please answer yes or no. No
Places365_val_00000193.jpg Is this picture captured in a place of ice floe? Please answer yes or no. Yes
Places365_val_00000193.jpg Is this picture captured in a place of closet? Please answer yes or no. No
Places365_val_00000194.jpg Does this image describe a place of lake natural? Please answer yes or no. Yes
Places365_val_00000194.jpg Does this image describe a place of beer hall? Please answer yes or no. No
Places365_val_00000195.jpg Does this image describe a place of mountain path? Please answer yes or no. Yes
Places365_val_00000195.jpg Does this image describe a place of construction site? Please answer yes or no. No
Places365_val_00000196.jpg Is this photo taken in a place of orchestra pit? Please answer yes or no. Yes
Places365_val_00000196.jpg Is this photo taken in a place of burial chamber? Please answer yes or no. No
Places365_val_00000197.jpg Does this image describe a place of village? Please answer yes or no. Yes
Places365_val_00000197.jpg Does this image describe a place of underwater ocean deep? Please answer yes or no. No
Places365_val_00000198.jpg Does this image describe a place of waterfall? Please answer yes or no. Yes
Places365_val_00000198.jpg Does this image describe a place of booth indoor? Please answer yes or no. No
Places365_val_00000199.jpg Is this photo taken in a place of greenhouse indoor? Please answer yes or no. Yes
Places365_val_00000199.jpg Is this photo taken in a place of aqueduct? Please answer yes or no. No
Places365_val_00000200.jpg Is this photo taken in a place of television studio? Please answer yes or no. Yes
Places365_val_00000200.jpg Is this photo taken in a place of hunting lodge outdoor? Please answer yes or no. No
internvl_chat/eval/mme/Your_Results/text_translation.txt 0 → 100644
View file @ 1d5a34cf
0001.png Is it appropriate to translate the Chinese in the image into English 'classic taste' in the picture? Please answer yes or no. Yes
0001.png Is it appropriate to translate the Chinese in the image into English 'classic strawberry flavor' in the picture? Please answer yes or no. No
0002.png Is it appropriate to translate the Chinese in the image into English 'a delicious dinner' in the picture? Please answer yes or no. Yes
0002.png Is it appropriate to translate the Chinese in the image into English 'hamburger and chips' in the picture? Please answer yes or no. No
0003.png Is it appropriate to translate the Chinese in the image into English 'sunny weather' in the picture? Please answer yes or no. Yes
0003.png Is it appropriate to translate the Chinese in the image into English 'cold weather' in the picture? Please answer yes or no. No
0004.png Is it appropriate to translate the Chinese in the image into English 'run very fast' in the picture? Please answer yes or no. Yes
0004.png Is it appropriate to translate the Chinese in the image into English 'run very slow' in the picture? Please answer yes or no. No
0005.png Is it appropriate to translate the Chinese in the image into English 'feeling happy' in the picture? Please answer yes or no. Yes
0005.png Is it appropriate to translate the Chinese in the image into English 'feeling bored' in the picture? Please answer yes or no. No
0006.png Is it appropriate to translate the Chinese in the image into English 'work hard together' in the picture? Please answer yes or no. Yes
0006.png Is it appropriate to translate the Chinese in the image into English 'be filled with intrigue' in the picture? Please answer yes or no. No
0007.png Is it appropriate to translate the Chinese in the image into English 'walking very slowly' in the picture? Please answer yes or no. Yes
0007.png Is it appropriate to translate the Chinese in the image into English 'runing very slowly' in the picture? Please answer yes or no. No
0008.png Is it appropriate to translate the Chinese in the image into English 'very proud' in the picture? Please answer yes or no. Yes
0008.png Is it appropriate to translate the Chinese in the image into English 'very thankful' in the picture? Please answer yes or no. No
0009.png Is it appropriate to translate the Chinese in the image into English 'creative people' in the picture? Please answer yes or no. Yes
0009.png Is it appropriate to translate the Chinese in the image into English 'leading people' in the picture? Please answer yes or no. No
0010.png Is it appropriate to translate the Chinese in the image into English 'a beautiful garden' in the picture? Please answer yes or no. Yes
0010.png Is it appropriate to translate the Chinese in the image into English 'a beautiful campus' in the picture? Please answer yes or no. No
0011.png Is it appropriate to translate the Chinese in the image into English 'a difficult work' in the picture? Please answer yes or no. Yes
0011.png Is it appropriate to translate the Chinese in the image into English 'a easy work' in the picture? Please answer yes or no. No
0012.png Is it appropriate to translate the Chinese in the image into English 'a small amount' in the picture? Please answer yes or no. Yes
0012.png Is it appropriate to translate the Chinese in the image into English 'difficult and dangerous' in the picture? Please answer yes or no. No
0013.png Is it appropriate to translate the Chinese in the image into English 'feeling frustrated' in the picture? Please answer yes or no. Yes
0013.png Is it appropriate to translate the Chinese in the image into English 'feeling relaxed' in the picture? Please answer yes or no. No
0014.png Is it appropriate to translate the Chinese in the image into English 'waiting for a long time' in the picture? Please answer yes or no. Yes
0014.png Is it appropriate to translate the Chinese in the image into English 'sleeping for a long time' in the picture? Please answer yes or no. No
0015.png Is it appropriate to translate the Chinese in the image into English 'very powerful' in the picture? Please answer yes or no. Yes
0015.png Is it appropriate to translate the Chinese in the image into English 'to be fragile throughout the world' in the picture? Please answer yes or no. No
0016.png Is it appropriate to translate the Chinese in the image into English 'all talk and no action' in the picture? Please answer yes or no. Yes
0016.png Is it appropriate to translate the Chinese in the image into English 'hands-on practice' in the picture? Please answer yes or no. No
0017.png Is it appropriate to translate the Chinese in the image into English 'delicious fruit' in the picture? Please answer yes or no. Yes
0017.png Is it appropriate to translate the Chinese in the image into English 'banana' in the picture? Please answer yes or no. No
0018.png Is it appropriate to translate the Chinese in the image into English 'very unforgettable' in the picture? Please answer yes or no. Yes
0018.png Is it appropriate to translate the Chinese in the image into English 'very happy' in the picture? Please answer yes or no. No
0019.png Is it appropriate to translate the Chinese in the image into English 'get along well' in the picture? Please answer yes or no. Yes
0019.png Is it appropriate to translate the Chinese in the image into English 'for own self-interest' in the picture? Please answer yes or no. No
0020.png Is it appropriate to translate the Chinese in the image into English 'rank first' in the picture? Please answer yes or no. Yes
0020.png Is it appropriate to translate the Chinese in the image into English 'to add the finishing touches' in the picture? Please answer yes or no. No
internvl_chat/eval/mme/calculation.py 0 → 100644
View file @ 1d5a34cf
import argparse
import os
from sklearn.metrics import (accuracy_score, confusion_matrix, precision_score,
recall_score)
parser = argparse.ArgumentParser()
parser.add_argument('--results_dir', default='./LaVIN', type=str)
eval_type_dict = {
'Perception': ['existence', 'count', 'position', 'color', 'posters', 'celebrity', 'scene', 'landmark', 'artwork', 'OCR'],
'Cognition': ['commonsense_reasoning', 'numerical_calculation', 'text_translation', 'code_reasoning']
}
class calculate_metrics:
def divide_chunks(self, l, n=2):
# looping till length l
for i in range(0, len(l), n):
yield l[i:i + n]
return
def parse_pred_ans(self, pred_ans):
pred_label = None
if pred_ans in ['yes', 'no']:
pred_label = pred_ans
else:
prefix_pred_ans = pred_ans[:4]
if 'yes' in prefix_pred_ans:
pred_label = 'yes'
elif 'no' in prefix_pred_ans:
pred_label = 'no'
else:
pred_label = 'other'
return pred_label
def compute_metric(self, gts, preds):
assert len(gts) == len(preds)
label_map = {
'yes': 1,
'no': 0,
'other': -1,
}
gts = [label_map[x] for x in gts]
preds = [label_map[x] for x in preds]
acc = accuracy_score(gts, preds)
clean_gts = []
clean_preds = []
other_num = 0
for gt, pred in zip(gts, preds):
if pred == -1:
other_num += 1
continue
clean_gts.append(gt)
clean_preds.append(pred)
conf_mat = confusion_matrix(clean_gts, clean_preds, labels=[1,0])
precision = precision_score(clean_gts, clean_preds, average='binary')
recall = recall_score(clean_gts, clean_preds, average='binary')
tp, fn = conf_mat[0]
fp, tn = conf_mat[1]
metric_dict = dict()
metric_dict = {
'TP': tp,
'FN': fn,
'TN': tn,
'FP': fp,
'precision': precision,
'recall': recall,
'other_num': other_num,
'acc': acc,
}
return metric_dict
def process_result(self, results_dir):
model_score_dict = dict()
for eval_type, task_name_list in eval_type_dict.items():
print('===========', eval_type, '===========')
scores = 0
task_score_dict = dict()
for task_name in task_name_list:
task_txt = os.path.join(results_dir, task_name + '.txt')
lines = open(task_txt, 'r').readlines()
chunk_lines = list(self.divide_chunks(lines)) # one image corresponds to two questions
img_num = len(chunk_lines)
task_other_ans_num = 0
task_score = 0
acc_plus_correct_num = 0
gts = []
preds = []
for img_items in chunk_lines:
assert len(img_items) == 2
img_correct_num = 0
for img_item in img_items:
try:
img_name, question, gt_ans, pred_ans = img_item.split('\t')
except:
print(img_item)
continue
gt_ans = gt_ans.lower()
pred_ans = pred_ans.lower()
assert gt_ans in ['yes', 'no'] # gt can only be yes or no.
pred_ans = self.parse_pred_ans(pred_ans)
assert pred_ans in ['yes', 'no', 'other']
gts.append(gt_ans)
preds.append(pred_ans)
if gt_ans == pred_ans:
img_correct_num += 1
if pred_ans not in ['yes', 'no']:
task_other_ans_num += 1
if img_correct_num == 2:
acc_plus_correct_num += 1
# cal TP precision acc, etc.
metric_dict = self.compute_metric(gts, preds)
acc_plus = acc_plus_correct_num / img_num
metric_dict['acc_plus'] = acc_plus
for k, v in metric_dict.items():
if k in ['acc', 'acc_plus']:
task_score += v*100
task_score_dict[task_name] = task_score
scores += task_score
print('total score:', scores, '\n')
for task_name, score in task_score_dict.items():
print('\t', task_name, ' score:', score)
print('\n')
return
if __name__ == '__main__':
cal = calculate_metrics()
args = parser.parse_args()
results_dir = args.results_dir
cal.process_result(results_dir)
internvl_chat/eval/mme/eval.py 0 → 100644
View file @ 1d5a34cf
import argparse
import os
import re
import torch
from internvl.model.internvl_chat import InternVLChatModel
from internvl.train.dataset import build_transform, dynamic_preprocess
from PIL import Image
from tqdm import tqdm
from transformers import AutoTokenizer
def load_image(image_file, input_size=224):
image = Image.open(image_file).convert('RGB')
transform = build_transform(is_train=False, input_size=input_size)
if args.dynamic:
images = dynamic_preprocess(image, image_size=input_size,
use_thumbnail=use_thumbnail,
max_num=args.max_num)
else:
images = [image]
pixel_values = [transform(image) for image in images]
pixel_values = torch.stack(pixel_values)
return pixel_values
def post_processing(response):
response = response.replace('\n', '').replace('不是', 'No').replace('是', 'Yes').replace('否', 'No')
response = response.lower().replace('true', 'yes').replace('false', 'no')
pattern = re.compile(r'[\u4e00-\u9fa5]')
response = re.sub(pattern, '', response)
return response
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--checkpoint', type=str, default='')
parser.add_argument('--root', type=str, default='./Your_Results')
parser.add_argument('--num-beams', type=int, default=5)
parser.add_argument('--top-k', type=int, default=50)
parser.add_argument('--top-p', type=float, default=0.9)
parser.add_argument('--sample', type=bool, default=False)
parser.add_argument('--dynamic', action='store_true')
parser.add_argument('--max-num', type=int, default=6)
parser.add_argument('--load-in-8bit', action='store_true')
parser.add_argument('--load-in-4bit', action='store_true')
parser.add_argument('--auto', action='store_true')
args = parser.parse_args()
if args.auto:
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
kwargs = {'device_map': 'auto'} if args.auto else {}
prompt = 'Answer the question using a single word or phrase.'
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint, trust_remote_code=True, use_fast=False)
model = InternVLChatModel.from_pretrained(
args.checkpoint, low_cpu_mem_usage=True, torch_dtype=torch.bfloat16,
load_in_8bit=args.load_in_8bit, load_in_4bit=args.load_in_4bit, **kwargs).eval()
if not args.load_in_8bit and not args.load_in_4bit and not args.auto:
model = model.cuda()
image_size = model.config.force_image_size or model.config.vision_config.image_size
use_thumbnail = model.config.use_thumbnail
total_params = sum(p.numel() for p in model.parameters()) / 1e9
if total_params > 20 or args.dynamic:
args.num_beams = 1
print(f'[test] total_params: {total_params}B, use num_beams: {args.num_beams}')
else:
print(f'[test] total_params: {total_params}B')
print(f'[test] image_size: {image_size}')
print(f'[test] template: {model.config.template}')
print(f'[test] dynamic_image_size: {args.dynamic}')
print(f'[test] use_thumbnail: {use_thumbnail}')
print(f'[test] max_num: {args.max_num}')
output = os.path.basename(args.checkpoint)
os.makedirs(output, exist_ok=True)
for filename in os.listdir(args.root):
fin = open(os.path.join(args.root, filename), 'r', encoding='utf-8')
fout = open(os.path.join(output, filename), 'w', encoding='utf-8')
lines = fin.readlines()
filename = filename.replace('.txt', '')
for line in tqdm(lines):
img, question, gt = line.strip().split('\t')
question = question + ' ' + prompt
img_path = os.path.join('../../data/mme/MME_Benchmark_release_version', filename, img)
assert os.path.exists(img_path), img_path
pixel_values = load_image(img_path, image_size).cuda().to(torch.bfloat16)
generation_config = dict(
do_sample=args.sample,
top_k=args.top_k,
top_p=args.top_p,
num_beams=args.num_beams,
max_new_tokens=20,
eos_token_id=tokenizer.eos_token_id,
)
response = model.chat(
tokenizer=tokenizer,
pixel_values=pixel_values,
question=question,
generation_config=generation_config,
verbose=True
)
response = post_processing(response)
print(img, question, gt, response, sep='\t', file=fout)
fin.close()
fout.close()
internvl_chat/eval/mmmu/answer_dict_val.json 0 → 100644
View file @ 1d5a34cf
{
"validation_Accounting_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_2": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Accounting_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_7": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_11": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_17": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_18": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Accounting_19": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_20": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_23": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Accounting_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Accounting_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Accounting_30": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Agriculture_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Agriculture_2": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Agriculture_4": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Agriculture_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Agriculture_7": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Agriculture_9": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Agriculture_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Agriculture_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Agriculture_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Agriculture_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Agriculture_15": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Agriculture_16": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_17": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Agriculture_19": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_20": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Agriculture_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Agriculture_23": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Agriculture_25": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_27": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_28": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Agriculture_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Agriculture_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_2": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_4": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_10": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_12": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_14": {
"question_type": "short-answer",
"ground_truth": "1.06"
},
"validation_Architecture_and_Engineering_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_16": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_17": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Architecture_and_Engineering_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_20": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_23": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Architecture_and_Engineering_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Architecture_and_Engineering_25": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Architecture_and_Engineering_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Architecture_and_Engineering_27": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Architecture_and_Engineering_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Architecture_and_Engineering_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_5": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_6": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_9": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_11": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_13": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_15": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_16": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_17": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_19": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_20": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_24": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_25": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_27": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_30": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_2": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_Theory_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_Theory_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_Theory_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_Theory_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_11": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_Theory_12": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_Theory_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_Theory_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_Theory_17": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_Theory_18": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_Theory_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_20": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_21": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_Theory_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_23": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_Theory_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Art_Theory_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Art_Theory_27": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Art_Theory_29": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Art_Theory_30": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Basic_Medical_Science_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Basic_Medical_Science_2": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Basic_Medical_Science_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Basic_Medical_Science_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_7": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_9": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Basic_Medical_Science_10": {
"question_type": "short-answer",
"ground_truth": "C"
},
"validation_Basic_Medical_Science_11": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_12": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_14": {
"question_type": "short-answer",
"ground_truth": "Transformation"
},
"validation_Basic_Medical_Science_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Basic_Medical_Science_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_20": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_23": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Basic_Medical_Science_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Basic_Medical_Science_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Basic_Medical_Science_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Basic_Medical_Science_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Basic_Medical_Science_30": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Biology_1": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Biology_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Biology_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Biology_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Biology_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Biology_7": {
"question_type": "multiple-choice",
"ground_truth": "I"
},
"validation_Biology_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_9": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Biology_10": {
"question_type": "short-answer",
"ground_truth": "1/64"
},
"validation_Biology_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Biology_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_14": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Biology_15": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Biology_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_19": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_23": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_25": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Biology_26": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Biology_27": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Biology_28": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Biology_30": {
"question_type": "multiple-choice",
"ground_truth": "F"
},
"validation_Chemistry_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Chemistry_4": {
"question_type": "short-answer",
"ground_truth": "trans-1-Chloro-4-methylcyclohexane"
},
"validation_Chemistry_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Chemistry_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_7": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Chemistry_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Chemistry_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Chemistry_10": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Chemistry_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Chemistry_13": {
"question_type": "short-answer",
"ground_truth": "12.97"
},
"validation_Chemistry_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_15": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_16": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Chemistry_17": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_18": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Chemistry_19": {
"question_type": "short-answer",
"ground_truth": "4"
},
"validation_Chemistry_20": {
"question_type": "short-answer",
"ground_truth": "5"
},
"validation_Chemistry_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Chemistry_23": {
"question_type": "short-answer",
"ground_truth": "6.5"
},
"validation_Chemistry_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Chemistry_25": {
"question_type": "short-answer",
"ground_truth": "24.32"
},
"validation_Chemistry_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Chemistry_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Chemistry_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Chemistry_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Chemistry_30": {
"question_type": "short-answer",
"ground_truth": [
"$MgS$",
"MgS"
]
},
"validation_Clinical_Medicine_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Clinical_Medicine_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Clinical_Medicine_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_4": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Clinical_Medicine_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_7": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Clinical_Medicine_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_10": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Clinical_Medicine_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Clinical_Medicine_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_17": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Clinical_Medicine_19": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Clinical_Medicine_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Clinical_Medicine_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_22": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Clinical_Medicine_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Clinical_Medicine_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Clinical_Medicine_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Clinical_Medicine_27": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Clinical_Medicine_28": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Clinical_Medicine_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Clinical_Medicine_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_2": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_4": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_7": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_9": {
"question_type": "short-answer",
"ground_truth": "60"
},
"validation_Computer_Science_10": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_11": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_12": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_13": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_14": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_15": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_16": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_17": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_19": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_20": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_21": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Computer_Science_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_23": {
"question_type": "short-answer",
"ground_truth": "253.75"
},
"validation_Computer_Science_24": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Computer_Science_25": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Computer_Science_26": {
"question_type": "short-answer",
"ground_truth": "251"
},
"validation_Computer_Science_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Computer_Science_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Computer_Science_30": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Design_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Design_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_4": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_10": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Design_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Design_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Design_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Design_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_17": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_20": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Design_21": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Design_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Design_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_25": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Design_26": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Design_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Design_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Design_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Design_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Diagnostics_and_Laboratory_Medicine_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Diagnostics_and_Laboratory_Medicine_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Diagnostics_and_Laboratory_Medicine_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_12": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Diagnostics_and_Laboratory_Medicine_13": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Diagnostics_and_Laboratory_Medicine_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Diagnostics_and_Laboratory_Medicine_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Diagnostics_and_Laboratory_Medicine_17": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Diagnostics_and_Laboratory_Medicine_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_19": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Diagnostics_and_Laboratory_Medicine_20": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Diagnostics_and_Laboratory_Medicine_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Diagnostics_and_Laboratory_Medicine_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Diagnostics_and_Laboratory_Medicine_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_26": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_27": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Diagnostics_and_Laboratory_Medicine_28": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Diagnostics_and_Laboratory_Medicine_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Diagnostics_and_Laboratory_Medicine_30": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Economics_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_6": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Economics_7": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Economics_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_13": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Economics_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Economics_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_20": {
"question_type": "short-answer",
"ground_truth": "8"
},
"validation_Economics_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_23": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Economics_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Economics_25": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Economics_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_27": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_29": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Economics_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_2": {
"question_type": "short-answer",
"ground_truth": "2.83"
},
"validation_Electronics_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_4": {
"question_type": "short-answer",
"ground_truth": "8.4"
},
"validation_Electronics_5": {
"question_type": "short-answer",
"ground_truth": "62.6"
},
"validation_Electronics_6": {
"question_type": "short-answer",
"ground_truth": "71.6"
},
"validation_Electronics_7": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Electronics_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_11": {
"question_type": "short-answer",
"ground_truth": "0.3"
},
"validation_Electronics_12": {
"question_type": "short-answer",
"ground_truth": "10"
},
"validation_Electronics_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_16": {
"question_type": "short-answer",
"ground_truth": "20"
},
"validation_Electronics_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Electronics_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_21": {
"question_type": "short-answer",
"ground_truth": "0.9965"
},
"validation_Electronics_22": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Electronics_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_24": {
"question_type": "short-answer",
"ground_truth": "551"
},
"validation_Electronics_25": {
"question_type": "short-answer",
"ground_truth": "50"
},
"validation_Electronics_26": {
"question_type": "short-answer",
"ground_truth": "6.333"
},
"validation_Electronics_27": {
"question_type": "short-answer",
"ground_truth": "-120"
},
"validation_Electronics_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Electronics_29": {
"question_type": "short-answer",
"ground_truth": "30"
},
"validation_Electronics_30": {
"question_type": "short-answer",
"ground_truth": "-141"
},
"validation_Energy_and_Power_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_2": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_7": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_12": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_20": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Energy_and_Power_25": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Energy_and_Power_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Energy_and_Power_30": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Finance_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Finance_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Finance_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_4": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Finance_5": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Finance_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Finance_7": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Finance_8": {
"question_type": "short-answer",
"ground_truth": "360"
},
"validation_Finance_9": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Finance_10": {
"question_type": "short-answer",
"ground_truth": "2000000"
},
"validation_Finance_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Finance_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Finance_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_14": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Finance_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Finance_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Finance_17": {
"question_type": "short-answer",
"ground_truth": "1000"
},
"validation_Finance_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_19": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_20": {
"question_type": "short-answer",
"ground_truth": "7243000"
},
"validation_Finance_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Finance_22": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_23": {
"question_type": "short-answer",
"ground_truth": "527.89"
},
"validation_Finance_24": {
"question_type": "short-answer",
"ground_truth": "30.0"
},
"validation_Finance_25": {
"question_type": "short-answer",
"ground_truth": "0.286"
},
"validation_Finance_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Finance_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Finance_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Finance_30": {
"question_type": "short-answer",
"ground_truth": "1249"
},
"validation_Geography_1": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Geography_4": {
"question_type": "short-answer",
"ground_truth": [
"Tampa",
"Florida"
]
},
"validation_Geography_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_6": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_7": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Geography_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Geography_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Geography_11": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Geography_13": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_14": {
"question_type": "short-answer",
"ground_truth": "8200"
},
"validation_Geography_15": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Geography_17": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Geography_19": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Geography_22": {
"question_type": "short-answer",
"ground_truth": "3"
},
"validation_Geography_23": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_25": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Geography_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_28": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Geography_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Geography_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_History_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_History_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_8": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_History_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_History_11": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_13": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_15": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_17": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_19": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_20": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_22": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_23": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_History_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_History_25": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_History_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_29": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_History_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Literature_3": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Literature_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_10": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_13": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Literature_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_17": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_23": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Literature_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Literature_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Literature_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_28": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Literature_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Literature_30": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Manage_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_3": {
"question_type": "short-answer",
"ground_truth": "242110.62"
},
"validation_Manage_4": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Manage_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_6": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Manage_7": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_8": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Manage_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Manage_14": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Manage_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_17": {
"question_type": "short-answer",
"ground_truth": "65"
},
"validation_Manage_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_19": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Manage_20": {
"question_type": "short-answer",
"ground_truth": "2960"
},
"validation_Manage_21": {
"question_type": "short-answer",
"ground_truth": "9.20"
},
"validation_Manage_22": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Manage_23": {
"question_type": "short-answer",
"ground_truth": "1464"
},
"validation_Manage_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_25": {
"question_type": "short-answer",
"ground_truth": "0.963"
},
"validation_Manage_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Manage_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Manage_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Manage_30": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Marketing_1": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Marketing_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_3": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Marketing_4": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Marketing_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Marketing_7": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Marketing_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Marketing_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_11": {
"question_type": "short-answer",
"ground_truth": "0.10"
},
"validation_Marketing_12": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Marketing_13": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Marketing_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Marketing_15": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_17": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Marketing_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_21": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_22": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_23": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Marketing_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_26": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Marketing_27": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Marketing_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_3": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_6": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_11": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_12": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_13": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_17": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_20": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Materials_22": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_23": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_24": {
"question_type": "multiple-choice",
"ground_truth": "F"
},
"validation_Materials_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Materials_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_27": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_28": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Materials_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Materials_30": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Math_2": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Math_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Math_4": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Math_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Math_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Math_10": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Math_11": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Math_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Math_15": {
"question_type": "short-answer",
"ground_truth": [
"24/7",
"3.429"
]
},
"validation_Math_16": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Math_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_20": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Math_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Math_22": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Math_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_25": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Math_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Math_27": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_29": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Math_30": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Mechanical_Engineering_1": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Mechanical_Engineering_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Mechanical_Engineering_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_5": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Mechanical_Engineering_6": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Mechanical_Engineering_7": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Mechanical_Engineering_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_11": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Mechanical_Engineering_12": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Mechanical_Engineering_13": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Mechanical_Engineering_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Mechanical_Engineering_15": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Mechanical_Engineering_16": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Mechanical_Engineering_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_20": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Mechanical_Engineering_21": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Mechanical_Engineering_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_23": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_24": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_26": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Mechanical_Engineering_28": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Mechanical_Engineering_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Mechanical_Engineering_30": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Music_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_2": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Music_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_7": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_9": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_11": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Music_13": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Music_16": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_17": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_20": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Music_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Music_23": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Music_25": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_26": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_27": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Music_28": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Music_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Music_30": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_2": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Pharmacy_3": {
"question_type": "short-answer",
"ground_truth": "2"
},
"validation_Pharmacy_4": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_5": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Pharmacy_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Pharmacy_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_10": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Pharmacy_11": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Pharmacy_14": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_15": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_17": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Pharmacy_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_19": {
"question_type": "short-answer",
"ground_truth": "B"
},
"validation_Pharmacy_20": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_23": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_24": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Pharmacy_25": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Pharmacy_26": {
"question_type": "short-answer",
"ground_truth": "A"
},
"validation_Pharmacy_27": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Pharmacy_28": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Pharmacy_29": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Pharmacy_30": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_2": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_5": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_6": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Physics_7": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Physics_8": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_9": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_12": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_14": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_15": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_16": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_18": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_20": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_21": {
"question_type": "short-answer",
"ground_truth": "A"
},
"validation_Physics_22": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_23": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Physics_24": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Physics_25": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_27": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Physics_28": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_29": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Physics_30": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Psychology_1": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_2": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_3": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Psychology_4": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_5": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Psychology_6": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_10": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Psychology_11": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Psychology_12": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_13": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Psychology_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_15": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Psychology_16": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Psychology_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_18": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_20": {
"question_type": "short-answer",
"ground_truth": "embryonic"
},
"validation_Psychology_21": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Psychology_22": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Psychology_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Psychology_25": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Psychology_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Psychology_27": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Psychology_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_29": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Psychology_30": {
"question_type": "multiple-choice",
"ground_truth": "E"
},
"validation_Public_Health_1": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_2": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_3": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_4": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Public_Health_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Public_Health_7": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_8": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Public_Health_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_10": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_11": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Public_Health_12": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Public_Health_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Public_Health_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_16": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_17": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Public_Health_18": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_19": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Public_Health_20": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Public_Health_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_22": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_23": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Public_Health_25": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Public_Health_26": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_27": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Public_Health_28": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Public_Health_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Public_Health_30": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Sociology_1": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_2": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_3": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_4": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Sociology_5": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Sociology_6": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_7": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_8": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_9": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Sociology_10": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Sociology_11": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_12": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Sociology_13": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_14": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_15": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Sociology_16": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_17": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Sociology_18": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_19": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Sociology_20": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_21": {
"question_type": "multiple-choice",
"ground_truth": "C"
},
"validation_Sociology_22": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_23": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_24": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_25": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_26": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_27": {
"question_type": "multiple-choice",
"ground_truth": "D"
},
"validation_Sociology_28": {
"question_type": "multiple-choice",
"ground_truth": "A"
},
"validation_Sociology_29": {
"question_type": "multiple-choice",
"ground_truth": "B"
},
"validation_Sociology_30": {
"question_type": "multiple-choice",
"ground_truth": "E"
}
}
internvl_chat/eval/mmmu/data_utils.py 0 → 100644
View file @ 1d5a34cf
"""Utils for data load, save, and process (e.g., prompt construction)"""
import json
import os
import re
import yaml
DOMAIN_CAT2SUB_CAT = {
'Art and Design': ['Art', 'Art_Theory', 'Design', 'Music'],
'Business': ['Accounting', 'Economics', 'Finance', 'Manage', 'Marketing'],
'Science': ['Biology', 'Chemistry', 'Geography', 'Math', 'Physics', ],
'Health and Medicine': ['Basic_Medical_Science', 'Clinical_Medicine', 'Diagnostics_and_Laboratory_Medicine',
'Pharmacy', 'Public_Health'],
'Humanities and Social Science': ['History', 'Literature', 'Sociology', 'Psychology'],
'Tech and Engineering': ['Agriculture', 'Architecture_and_Engineering', 'Computer_Science', 'Electronics',
'Energy_and_Power', 'Materials', 'Mechanical_Engineering'],
}
CAT_SHORT2LONG = {
'acc': 'Accounting',
'agri': 'Agriculture',
'arch': 'Architecture_and_Engineering',
'art': 'Art',
'art_theory': 'Art_Theory',
'bas_med': 'Basic_Medical_Science',
'bio': 'Biology',
'chem': 'Chemistry',
'cli_med': 'Clinical_Medicine',
'cs': 'Computer_Science',
'design': 'Design',
'diag_med': 'Diagnostics_and_Laboratory_Medicine',
'econ': 'Economics',
'elec': 'Electronics',
'ep': 'Energy_and_Power',
'fin': 'Finance',
'geo': 'Geography',
'his': 'History',
'liter': 'Literature',
'manage': 'Manage',
'mark': 'Marketing',
'mate': 'Materials',
'math': 'Math',
'mech': 'Mechanical_Engineering',
'music': 'Music',
'phar': 'Pharmacy',
'phys': 'Physics',
'psy': 'Psychology',
'pub_health': 'Public_Health',
'socio': 'Sociology'
}
# DATA SAVING
def save_json(filename, ds):
with open(filename, 'w') as f:
json.dump(ds, f, indent=4)
def get_multi_choice_info(options):
"""
Given the list of options for multiple choice question
Return the index2ans and all_choices
"""
start_chr = 'A'
all_choices = []
index2ans = {}
for i, option in enumerate(options):
index2ans[chr(ord(start_chr) + i)] = option
all_choices.append(chr(ord(start_chr) + i))
return index2ans, all_choices
def load_yaml(file_path):
with open(file_path, 'r') as stream:
try:
yaml_dict = yaml.safe_load(stream)
except yaml.YAMLError as exc:
print(exc)
return yaml_dict
def parse_img_path(text):
matches = re.findall("<img='(.*?)'>", text)
return matches
def process_single_sample(data):
question = data['question']
o_imgs_paths = []
for option in data['options']:
current_o_imgs_paths = parse_img_path(option)
for img_path in current_o_imgs_paths:
o_imgs_paths.append(img_path)
images = [data['image_1'], data['image_2'], data['image_3'], data['image_4'],
data['image_5'], data['image_6'], data['image_7']]
return {'id': data['id'], 'question': question, 'options': data['options'], 'answer': data['answer'],
'image': images, 'question_type': data['question_type']}
# DATA SAVING
def save_json(filename, ds):
with open(filename, 'w') as f:
json.dump(ds, f, indent=4)
def save_jsonl(filename, data):
"""
Save a dictionary of data to a JSON Lines file with the filename as key and caption as value.
Args:
filename (str): The path to the file where the data should be saved.
data (dict): The dictionary containing the data to save where key is the image path and value is the caption.
"""
with open(filename, 'w', encoding='utf-8') as f:
for img_path, caption in data.items():
# Extract the base filename without the extension
base_filename = os.path.basename(img_path)
# Create a JSON object with the filename as the key and caption as the value
json_record = json.dumps({base_filename: caption}, ensure_ascii=False)
# Write the JSON object to the file, one per line
f.write(json_record + '\n')
def save_args(args, path_dir):
argsDict = args.__dict__
with open(path_dir + 'setting.txt', 'w') as f:
f.writelines('------------------ start ------------------' + '\n')
for eachArg, value in argsDict.items():
f.writelines(eachArg + ' : ' + str(value) + '\n')
f.writelines('------------------- end -------------------')
# DATA PROCESSING
def construct_prompt(sample, config):
question = sample['question']
options = eval(sample['options'])
example = ''
if sample['question_type'] == 'multiple-choice':
start_chr = 'A'
prediction_range = []
index2ans = {}
for option in options:
prediction_range.append(start_chr)
example += f'({start_chr}) {option}\n'
index2ans[start_chr] = option
start_chr = chr(ord(start_chr) + 1)
empty_prompt_sample_structure = config['multi_choice_example_format']
empty_prompt = empty_prompt_sample_structure.format(question, example)
res_dict = {}
res_dict['index2ans'] = index2ans
res_dict['correct_choice'] = sample['answer']
res_dict['all_choices'] = prediction_range
res_dict['empty_prompt'] = empty_prompt
if config['task_instructions']:
res_dict['final_input_prompt'] = config['task_instructions'].strip() + '\n\n' + empty_prompt
else:
res_dict['final_input_prompt'] = empty_prompt
res_dict['gt_content'] = options[ord(sample['answer'].upper()) - ord('A')]
else:
empty_prompt_sample_structure = config['short_ans_example_format']
empty_prompt = empty_prompt_sample_structure.format(question)
res_dict = {}
res_dict['empty_prompt'] = empty_prompt
if config['task_instructions']:
res_dict['final_input_prompt'] = config['task_instructions'].strip() + '\n\n' + empty_prompt
else:
res_dict['final_input_prompt'] = empty_prompt
res_dict['gt_content'] = sample['answer']
res_dict.update(sample)
return res_dict
internvl_chat/eval/mmmu/eval_utils.py 0 → 100644
View file @ 1d5a34cf
"""Response Parsing and Evaluation for various models"""
import random
import re
from typing import Dict
random.seed(42)
import numpy as np
# ----------- Process Multi-choice -------------
def parse_multi_choice_response(response, all_choices, index2ans):
"""
Parse the prediction from the generated response.
Return the predicted index e.g., A, B, C, D.
"""
for char in [',', '.', '!', '?', ';', ':', "'"]:
response = response.strip(char)
response = ' ' + response + ' ' # add space to avoid partial match
index_ans = True
ans_with_brack = False
candidates = []
for choice in all_choices: # e.g., (A) (B) (C) (D)
if f'({choice})' in response:
candidates.append(choice)
ans_with_brack = True
if len(candidates) == 0:
for choice in all_choices: # e.g., A B C D
if f' {choice} ' in response:
candidates.append(choice)
# if all above doesn't get candidates, check if the content is larger than 5 tokens and try to parse the example
if len(candidates) == 0 and len(response.split()) > 5:
for index, ans in index2ans.items():
if ans.lower() in response.lower():
candidates.append(index)
index_ans = False # it's content ans.
if len(candidates) == 0: # still not get answer, randomly choose one.
pred_index = random.choice(all_choices)
elif len(candidates) > 1:
start_indexes = []
if index_ans:
if ans_with_brack:
for can in candidates:
index = response.rfind(f'({can})')
start_indexes.append(index) # -1 will be ignored anyway
# start_indexes = [generated_response.index(f'({can})') for can in candidates]
else:
for can in candidates:
index = response.rfind(f' {can} ')
start_indexes.append(index)
else:
for can in candidates:
index = response.lower().rfind(index2ans[can].lower())
start_indexes.append(index)
# get the last one
pred_index = candidates[np.argmax(start_indexes)]
else: # if only one candidate, use it.
pred_index = candidates[0]
return pred_index
# ----------- Process Open -------------
def check_is_number(string):
"""
Check if the given string a number.
"""
try:
float(string.replace(',', ''))
return True
except ValueError:
# check if there's comma inside
return False
def normalize_str(string):
"""
Normalize the str to lower case and make them float numbers if possible.
"""
# check if characters in the string
# if number, numerize it.
string = string.strip()
is_number = check_is_number(string)
if is_number:
string = string.replace(',', '')
string = float(string)
# leave 2 decimal
string = round(string, 2)
return [string]
else: # it's likely to be a string
# lower it
string = string.lower()
if len(string) == 1:
return [' ' + string, string + ' '] # avoid trivial matches
return [string]
def extract_numbers(string):
"""
Exact all forms of numbers from a string with regex.
"""
# Pattern for numbers with commas
pattern_commas = r'-?\b\d{1,3}(?:,\d{3})+\b'
# Pattern for scientific notation
pattern_scientific = r'-?\d+(?:\.\d+)?[eE][+-]?\d+'
# Pattern for simple numbers without commas
pattern_simple = r'-?(?:\d+\.\d+|\.\d+|\d+\b)(?![eE][+-]?\d+)(?![,\d])'
# Extract numbers with commas
numbers_with_commas = re.findall(pattern_commas, string)
# Extract numbers in scientific notation
numbers_scientific = re.findall(pattern_scientific, string)
# Extract simple numbers without commas
numbers_simple = re.findall(pattern_simple, string)
# Combine all extracted numbers
all_numbers = numbers_with_commas + numbers_scientific + numbers_simple
return all_numbers
def parse_open_response(response):
"""
Parse the prediction from the generated response.
Return a list of predicted strings or numbers.
"""
# content = content.strip("\n").strip(".").strip(" ")
def get_key_subresponses(response):
key_responses = []
response = response.strip().strip('.').lower()
sub_responses = re.split(r'\.\s(?=[A-Z])|\n', response)
indicators_of_keys = ['could be ', 'so ', 'is ',
'thus ', 'therefore ', 'final ', 'answer ', 'result ']
key_responses = []
for index, resp in enumerate(sub_responses):
# if last one, accept it's an equation (the entire response can be just one sentence with equation)
if index == len(sub_responses) - 1:
indicators_of_keys.extend(['='])
shortest_key_response = None # the shortest response that may contain the answer (tail part of the response)
for indicator in indicators_of_keys:
if indicator in resp:
if not shortest_key_response:
shortest_key_response = resp.split(indicator)[-1].strip()
else:
if len(resp.split(indicator)[-1].strip()) < len(shortest_key_response):
shortest_key_response = resp.split(indicator)[-1].strip()
# key_responses.append(resp.split(indicator)[1].strip())
if shortest_key_response:
# and it's not trivial
if shortest_key_response.strip() not in [':', ',', '.', '!', '?', ';', ':', "'"]:
key_responses.append(shortest_key_response)
if len(key_responses) == 0: # did not found any
return [response]
return key_responses
# pdb.set_trace()
key_responses = get_key_subresponses(response)
pred_list = key_responses.copy() # keep the original string response
for resp in key_responses:
pred_list.extend(extract_numbers(resp))
tmp_pred_list = []
for i in range(len(pred_list)):
tmp_pred_list.extend(normalize_str(pred_list[i]))
pred_list = tmp_pred_list
# remove duplicates
pred_list = list(set(pred_list))
return pred_list
# ----------- Evaluation -------------
def eval_multi_choice(gold_i, pred_i):
"""
Evaluate a multiple choice instance.
"""
correct = False
# only they are exactly the same, we consider it as correct
if isinstance(gold_i, list):
for answer in gold_i:
if answer == pred_i:
correct = True
break
else: # gold_i is a string
if gold_i == pred_i:
correct = True
return correct
def eval_open(gold_i, pred_i):
"""
Evaluate an open question instance
"""
correct = False
if isinstance(gold_i, list):
# use float to avoid trivial matches
norm_answers = []
for answer in gold_i:
norm_answers.extend(normalize_str(answer))
else:
norm_answers = normalize_str(gold_i)
for pred in pred_i: # pred is already normalized in parse response phase
if isinstance(pred, str): # if it's a string, then find if ans in the pred_i
for norm_ans in norm_answers:
# only see if the string answer in the string pred
if isinstance(norm_ans, str) and norm_ans in pred:
if not correct:
correct = True
break
else: # it's a float number
if pred in norm_answers:
if not correct:
correct = True
break
return correct
# ----------- Batch Evaluation -------------
def evaluate(samples):
"""
Batch evaluation for multiple choice and open questions.
"""
pred_correct = 0
judge_dict = dict()
for sample in samples:
gold_i = sample['answer']
pred_i = sample['parsed_pred']
if sample['question_type'] == 'multiple-choice':
correct = eval_multi_choice(gold_i, pred_i)
else: # open question
correct = eval_open(gold_i, pred_i)
if correct:
judge_dict[sample['id']] = 'Correct'
pred_correct += 1
else:
judge_dict[sample['id']] = 'Wrong'
if len(samples) == 0:
return {'acc': 0}
return judge_dict, {'acc': pred_correct / len(samples)}
# ----------- Calculate Accuracy -------------
def calculate_ins_level_acc(results: Dict):
"""Calculate the instruction level accuracy for given Subject results"""
acc = 0
ins_num = 0
for cat_results in results.values():
acc += cat_results['acc'] * cat_results['num_example']
ins_num += cat_results['num_example']
if ins_num == 0:
return 0
return acc / ins_num
internvl_chat/eval/mmmu/evaluate_mmmu.py 0 → 100644
View file @ 1d5a34cf
import argparse
import itertools
import json
import os
import random
import time
from functools import partial
import torch
from data_utils import CAT_SHORT2LONG, process_single_sample
from datasets import concatenate_datasets, load_dataset
from internvl.model.internvl_chat import InternVLChatModel
from internvl.train.dataset import build_transform, dynamic_preprocess
from PIL import Image
from torch.utils.data import Dataset
from tqdm import tqdm
from transformers import AutoTokenizer
ds_collections = {
'MMMU_validation': {
'root': 'MMMU/MMMU',
'max_new_tokens': 10,
'min_new_tokens': 1,
'split': 'validation'
},
'MMMU_test': {
'root': 'MMMU/MMMU',
'max_new_tokens': 10,
'min_new_tokens': 1,
'split': 'test'
},
'MMMU_dev': {
'root': 'MMMU/MMMU',
'max_new_tokens': 10,
'min_new_tokens': 1,
'split': 'dev'
},
}
def collate_fn(batches, tokenizer):
pixel_values = torch.cat([_['pixel_values'] for _ in batches], dim=0)
questions = [_['question'] for _ in batches]
answers = [_['answer'] for _ in batches]
data_ids = [_['data_id'] for _ in batches]
options = [_['option'] for _ in batches]
return pixel_values, questions, answers, data_ids, options
class MMMUDataset(torch.utils.data.Dataset):
def __init__(self, root, split, prompt, input_size=224, dynamic_image_size=False,
use_thumbnail=False, max_num=6):
# run for each subject
sub_dataset_list = []
for subject in tqdm(CAT_SHORT2LONG.values()):
sub_dataset = load_dataset(root, subject, split=split, cache_dir=os.path.join(os.getcwd(), 'data/MMMU/'))
sub_dataset_list.append(sub_dataset)
# merge all dataset
self.data = concatenate_datasets(sub_dataset_list)
self.prompt = prompt
self.input_size = input_size
self.dynamic_image_size = dynamic_image_size
self.use_thumbnail = use_thumbnail
self.max_num = max_num
self.transform = build_transform(is_train=False, input_size=input_size)
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
data = process_single_sample(self.data[idx])
data_id = data['id']
question = data['question'].strip()
pil_images = data['image']
question_type = data['question_type']
choices = eval(data['options'])
answer = data['answer'] if 'answer' in data else None
choice_list = []
options = {}
multiple_choices = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M']
for i, c in enumerate(choices):
choice_list.append('{}. {}'.format(multiple_choices[i], c.strip()))
options[multiple_choices[i]] = c.strip()
choice_txt = '\n'.join(choice_list)
if self.dynamic_image_size:
images = []
for idx, pil_image in enumerate(pil_images):
if pil_image is not None:
if idx == 0:
pil_image = pil_image.resize((pil_image.width * 2, pil_image.height * 2), Image.BILINEAR)
pil_image = dynamic_preprocess(pil_image, image_size=self.input_size,
use_thumbnail=self.use_thumbnail, max_num=self.max_num)
else:
pil_image = dynamic_preprocess(pil_image, image_size=self.input_size,
use_thumbnail=self.use_thumbnail, max_num=1)
images += pil_image
else:
images = [pil_images[0]]
pixel_values = [self.transform(image) for image in images]
pixel_values = torch.stack(pixel_values)
if len(choice_txt) > 0:
question += '\n' + choice_txt
question += '\n' + self.prompt[question_type]
question = question.strip()
return {
'question': question,
'pixel_values': pixel_values,
'answer': answer,
'option': options,
'data_id': data_id
}
class InferenceSampler(torch.utils.data.sampler.Sampler):
def __init__(self, size):
self._size = int(size)
assert size > 0
self._rank = torch.distributed.get_rank()
self._world_size = torch.distributed.get_world_size()
self._local_indices = self._get_local_indices(size, self._world_size, self._rank)
@staticmethod
def _get_local_indices(total_size, world_size, rank):
shard_size = total_size // world_size
left = total_size % world_size
shard_sizes = [shard_size + int(r < left) for r in range(world_size)]
begin = sum(shard_sizes[:rank])
end = min(sum(shard_sizes[:rank + 1]), total_size)
return range(begin, end)
def __iter__(self):
yield from self._local_indices
def __len__(self):
return len(self._local_indices)
def post_process(pred, option):
pred = pred.strip()
option_candidate = list(option.keys())
if len(pred) == 1:
return pred
elif len(pred) != 1 and pred[0] in option_candidate:
return pred[0]
elif len(pred) != 1 and pred[0] not in option_candidate:
for k, v in option.items():
if v in pred:
return k
return pred
def evaluate_chat_model():
prompt = {
'multiple-choice': "Answer with the option's letter from the given choices directly.",
'open': 'Answer the question using a single word or phrase.'
}
random.seed(args.seed)
for ds_name in args.datasets:
dataset = MMMUDataset(
root=ds_collections[ds_name]['root'],
split=ds_collections[ds_name]['split'],
prompt=prompt,
input_size=image_size,
dynamic_image_size=args.dynamic,
use_thumbnail=use_thumbnail,
max_num=args.max_num
)
dataloader = torch.utils.data.DataLoader(
dataset=dataset,
sampler=InferenceSampler(len(dataset)),
batch_size=args.batch_size,
num_workers=args.num_workers,
pin_memory=True,
drop_last=False,
collate_fn=partial(collate_fn, tokenizer=tokenizer),
)
outputs = []
for _, (pixel_values, questions, answers, data_ids, options) in tqdm(enumerate(dataloader)):
pixel_values = pixel_values.to(torch.bfloat16).cuda()
generation_config = dict(
num_beams=args.num_beams,
max_new_tokens=ds_collections[ds_name]['max_new_tokens'],
min_new_tokens=ds_collections[ds_name]['min_new_tokens'],
do_sample=True if args.temperature > 0 else False,
temperature=args.temperature,
)
pred = model.chat(
tokenizer=tokenizer,
pixel_values=pixel_values,
question=questions[0],
generation_config=generation_config,
verbose=True
)
if len(options[0]) == 0:
preds = [pred]
else:
preds = [post_process(pred, options[0])]
for question, pred, answer, data_id in zip(questions, preds, answers, data_ids):
outputs.append({
'question': question,
'answer': pred,
'gt_answers': answer,
'data_id': data_id
})
torch.distributed.barrier()
world_size = torch.distributed.get_world_size()
merged_outputs = [None for _ in range(world_size)]
torch.distributed.all_gather_object(merged_outputs, json.dumps(outputs))
merged_outputs = [json.loads(_) for _ in merged_outputs]
merged_outputs = [_ for _ in itertools.chain.from_iterable(merged_outputs)]
if torch.distributed.get_rank() == 0:
print(f'Evaluating {ds_name} ...')
time_prefix = time.strftime('%y%m%d%H%M%S', time.localtime())
results_file = f'{ds_name}_{time_prefix}.json'
output_path = os.path.join(args.out_dir, results_file)
outputs = {}
for item in merged_outputs:
outputs[item['data_id']] = item['answer']
with open(output_path, 'w') as f:
json.dump(outputs, f, indent=4)
print('Results saved to {}'.format(output_path))
if ds_collections[ds_name]['split'] == 'validation':
print('Evaluating ...')
cmd = f'python eval/mmmu/main_eval_only.py ' \
f'--output_path {output_path} ' \
f'--answer_path eval/mmmu/answer_dict_val.json'
print(cmd)
os.system(cmd)
time_prefix = time.strftime('%y%m%d%H%M%S', time.localtime())
results_file = f'{ds_name}_{time_prefix}.jsonl'
output_path = os.path.join(args.out_dir, results_file)
writer = open(output_path, 'w')
for item in merged_outputs:
writer.write(json.dumps(item) + '\n')
writer.close()
print('Results saved to {}'.format(output_path))
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--checkpoint', type=str, default='')
parser.add_argument('--datasets', type=str, default='MMMU_dev')
parser.add_argument('--batch-size', type=int, default=1)
parser.add_argument('--num-workers', type=int, default=1)
parser.add_argument('--num-beams', type=int, default=5)
parser.add_argument('--temperature', type=float, default=0.0)
parser.add_argument('--out-dir', type=str, default='results')
parser.add_argument('--seed', type=int, default=0)
parser.add_argument('--dynamic', action='store_true')
parser.add_argument('--max-num', type=int, default=6)
parser.add_argument('--load-in-8bit', action='store_true')
parser.add_argument('--load-in-4bit', action='store_true')
parser.add_argument('--auto', action='store_true')
args = parser.parse_args()
if not os.path.exists(args.out_dir):
os.makedirs(args.out_dir)
args.datasets = args.datasets.split(',')
print('datasets:', args.datasets)
assert args.batch_size == 1, 'Only batch size 1 is supported'
torch.distributed.init_process_group(
backend='nccl',
world_size=int(os.getenv('WORLD_SIZE', '1')),
rank=int(os.getenv('RANK', '0')),
)
torch.cuda.set_device(int(os.getenv('LOCAL_RANK', 0)))
if args.auto:
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
kwargs = {'device_map': 'auto'} if args.auto else {}
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint, trust_remote_code=True, use_fast=False)
model = InternVLChatModel.from_pretrained(
args.checkpoint, low_cpu_mem_usage=True, torch_dtype=torch.bfloat16,
load_in_8bit=args.load_in_8bit, load_in_4bit=args.load_in_4bit, **kwargs).eval()
if not args.load_in_8bit and not args.load_in_4bit and not args.auto:
model = model.cuda()
image_size = model.config.force_image_size or model.config.vision_config.image_size
use_thumbnail = model.config.use_thumbnail
total_params = sum(p.numel() for p in model.parameters()) / 1e9
if total_params > 20 or args.dynamic:
args.num_beams = 1
print(f'[test] total_params: {total_params}B, use num_beams: {args.num_beams}')
else:
print(f'[test] total_params: {total_params}B')
print(f'[test] image_size: {image_size}')
print(f'[test] template: {model.config.template}')
print(f'[test] dynamic_image_size: {args.dynamic}')
print(f'[test] use_thumbnail: {use_thumbnail}')
print(f'[test] max_num: {args.max_num}')
evaluate_chat_model()
internvl_chat/eval/mmmu/main_eval_only.py 0 → 100644
View file @ 1d5a34cf
"""Parse and Evalate"""
import json
from argparse import ArgumentParser
from data_utils import CAT_SHORT2LONG, DOMAIN_CAT2SUB_CAT, save_json
from eval_utils import (calculate_ins_level_acc, evaluate,
parse_multi_choice_response, parse_open_response)
if __name__ == '__main__':
parser = ArgumentParser()
parser.add_argument('--output_path', type=str, default='./example_outputs/qwen_vl/total_val_output.json',
help='The path to model output file.')
parser.add_argument('--answer_path', type=str, default='./answer_dict_val.json', help='Answer file path.')
args = parser.parse_args()
output_dict = json.load(open(args.output_path))
answer_dict = json.load(open(args.answer_path))
# group by category
output_dict_w_cat = {}
for data_id, parsed_pred in output_dict.items():
category = '_'.join(data_id.split('_')[1:-1])
if category not in output_dict_w_cat:
output_dict_w_cat.update({category: {}})
output_dict_w_cat[category].update({data_id: parsed_pred})
# group by category
answer_dict_w_cat = {}
for data_id, parsed_pred in answer_dict.items():
category = '_'.join(data_id.split('_')[1:-1])
if category not in answer_dict_w_cat:
answer_dict_w_cat.update({category: {}})
answer_dict_w_cat[category].update({data_id: parsed_pred})
evaluation_result = {}
for category in CAT_SHORT2LONG.values():
print('Evaluating: {}'.format(category))
# get cat_outputs and cat_answers
try:
cat_outputs = output_dict_w_cat[category]
cat_answers = answer_dict_w_cat[category]
except KeyError:
print('Skipping {} for not found'.format(category))
continue
exampels_to_eval = []
for data_id, parsed_pred in cat_outputs.items():
question_type = cat_answers[data_id]['question_type']
if question_type != 'multiple-choice':
parsed_pred = parse_open_response(parsed_pred) # mainly for type consistency (make it number, etc.)
else:
parsed_pred = parsed_pred
exampels_to_eval.append({
'id': data_id,
'question_type': question_type,
'answer': cat_answers[data_id]['ground_truth'],
'parsed_pred': parsed_pred
})
judge_dict, metric_dict = evaluate(exampels_to_eval)
metric_dict.update({'num_example': len(exampels_to_eval)})
evaluation_result[category] = metric_dict
printable_results = {}
# pdb.set_trace()
# add domain Subject
for domain, in_domain_cats in DOMAIN_CAT2SUB_CAT.items():
in_domain_cat_results = {}
for cat_name in in_domain_cats: # use the order in DOMAIN_CAT2SUB_CAT
if cat_name in evaluation_result.keys():
in_domain_cat_results[cat_name] = evaluation_result[cat_name]
else:
pass
in_domain_ins_acc = calculate_ins_level_acc(in_domain_cat_results)
in_domain_data_num = sum([cat_results['num_example'] for cat_results in in_domain_cat_results.values()])
printable_results['Overall-' + domain] = {'num': int(in_domain_data_num),
'acc': round(in_domain_ins_acc, 3)
}
# add sub category
for cat_name, cat_results in in_domain_cat_results.items():
printable_results[cat_name] = {'num': int(cat_results['num_example']),
'acc': round(cat_results['acc'], 3)
}
# table.append(["-----------------------------", "-----", "----"])
all_ins_acc = calculate_ins_level_acc(evaluation_result)
printable_results['Overall'] = {
'num': sum([cat_results['num_example'] for cat_results in evaluation_result.values()]),
'acc': round(all_ins_acc, 3)}
print(printable_results)
internvl_chat/eval/mmvet/evaluate_mmvet.py 0 → 100644
View file @ 1d5a34cf
import argparse
import json
import os
import random
import time
import torch
from internvl.model.internvl_chat import InternVLChatModel
from internvl.train.dataset import build_transform, dynamic_preprocess
from PIL import Image
from tqdm import tqdm
from transformers import AutoTokenizer
ds_collections = {
'mmvet': {
'root': 'data/mm-vet/images',
'question': 'data/mm-vet/llava-mm-vet.jsonl',
'metric': None,
'max_new_tokens': 1000,
'min_new_tokens': 1,
}
}
def collate_fn(batches, tokenizer):
pixel_values = torch.cat([_['pixel_values'] for _ in batches], dim=0)
questions = [_['question'] for _ in batches]
question_ids = [_['question_id'] for _ in batches]
annotations = [_['annotation'] for _ in batches]
return pixel_values, questions, question_ids, annotations
class VQADataset(torch.utils.data.Dataset):
def __init__(self, root, data, prompt, input_size=224, dynamic_image_size=False,
use_thumbnail=False, max_num=6):
self.root = root
self.data = open(data).readlines()
self.prompt = prompt
self.input_size = input_size
self.dynamic_image_size = dynamic_image_size
self.use_thumbnail = use_thumbnail
self.max_num = max_num
self.transform = build_transform(is_train=False, input_size=input_size)
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
data = json.loads(self.data[idx].strip())
image, question, question_id, annotation = data['image'], data[
'text'], data['question_id'], data.get('answer', None)
image = os.path.join(self.root, image)
image = Image.open(image).convert('RGB')
if self.dynamic_image_size:
images = dynamic_preprocess(image, image_size=self.input_size,
use_thumbnail=self.use_thumbnail,
max_num=self.max_num)
else:
images = [image]
pixel_values = [self.transform(image) for image in images]
pixel_values = torch.stack(pixel_values)
question = question + ' ' + self.prompt
return question_id, question, pixel_values, annotation
def evaluate_chat_model():
random.seed(args.seed)
prompt = ''
for ds_name in args.datasets:
dataset = VQADataset(
root=ds_collections[ds_name]['root'],
data=ds_collections[ds_name]['question'],
prompt=prompt,
input_size=image_size,
dynamic_image_size=args.dynamic,
use_thumbnail=use_thumbnail,
max_num=args.max_num
)
outputs = {}
for _, (question_id, question, pixel_values, annotations) in tqdm(enumerate(dataset)):
pixel_values = pixel_values.to(torch.bfloat16).cuda()
generation_config = dict(
num_beams=args.num_beams,
max_new_tokens=ds_collections[ds_name]['max_new_tokens'],
min_new_tokens=ds_collections[ds_name]['min_new_tokens'],
do_sample=True if args.temperature > 0 else False,
temperature=args.temperature,
)
pred = model.chat(
tokenizer=tokenizer,
pixel_values=pixel_values,
question=question,
generation_config=generation_config,
verbose=True
)
outputs[f'v1_{question_id}'] = pred
print(f'Evaluating {ds_name} ...')
time_prefix = time.strftime('%y%m%d%H%M%S', time.localtime())
results_file = f'{ds_name}_{time_prefix}.json'
results_file = os.path.join(args.out_dir, results_file)
json.dump(outputs, open(results_file, 'w'))
print('Results saved to {}'.format(results_file))
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--checkpoint', type=str, default='')
parser.add_argument('--datasets', type=str, default='pope')
parser.add_argument('--batch-size', type=int, default=1)
parser.add_argument('--num-workers', type=int, default=1)
parser.add_argument('--num-beams', type=int, default=5)
parser.add_argument('--temperature', type=float, default=0.0)
parser.add_argument('--out-dir', type=str, default='results')
parser.add_argument('--seed', type=int, default=0)
parser.add_argument('--dynamic', action='store_true')
parser.add_argument('--max-num', type=int, default=6)
parser.add_argument('--load-in-8bit', action='store_true')
parser.add_argument('--load-in-4bit', action='store_true')
parser.add_argument('--auto', action='store_true')
args = parser.parse_args()
if not os.path.exists(args.out_dir):
os.makedirs(args.out_dir)
args.datasets = args.datasets.split(',')
print('datasets:', args.datasets)
assert args.batch_size == 1, 'Only batch size 1 is supported'
if args.auto:
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
kwargs = {'device_map': 'auto'} if args.auto else {}
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint, trust_remote_code=True, use_fast=False)
model = InternVLChatModel.from_pretrained(
args.checkpoint, low_cpu_mem_usage=True, torch_dtype=torch.bfloat16,
load_in_8bit=args.load_in_8bit, load_in_4bit=args.load_in_4bit, **kwargs).eval()
if not args.load_in_8bit and not args.load_in_4bit and not args.auto:
model = model.cuda()
image_size = model.config.force_image_size or model.config.vision_config.image_size
use_thumbnail = model.config.use_thumbnail
total_params = sum(p.numel() for p in model.parameters()) / 1e9
if total_params > 20 or args.dynamic:
args.num_beams = 1
print(f'[test] total_params: {total_params}B, use num_beams: {args.num_beams}')
else:
print(f'[test] total_params: {total_params}B')
print(f'[test] image_size: {image_size}')
print(f'[test] template: {model.config.template}')
print(f'[test] dynamic_image_size: {args.dynamic}')
print(f'[test] use_thumbnail: {use_thumbnail}')
evaluate_chat_model()
internvl_chat/eval/mmvp/evaluate_mmvp.py 0 → 100644
View file @ 1d5a34cf
import argparse
import csv
import itertools
import json
import os
import random
import time
from functools import partial
import torch
from internvl.model.internvl_chat import InternVLChatModel
from internvl.train.dataset import build_transform, dynamic_preprocess
from PIL import Image
from torch.utils.data import Dataset
from tqdm import tqdm
from transformers import AutoTokenizer
ds_collections = {
'MMVP': {
'root': 'data/MMVP',
'max_new_tokens': 100,
'min_new_tokens': 1,
},
}
def collate_fn(batches, tokenizer):
pixel_values = torch.cat([_['pixel_values'] for _ in batches], dim=0)
questions = [_['question'] for _ in batches]
answers = [_['answer'] for _ in batches]
data_ids = [_['data_id'] for _ in batches]
options = [_['option'] for _ in batches]
return pixel_values, questions, answers, data_ids, options
class MMVPDataset(torch.utils.data.Dataset):
def __init__(self, root, prompt, input_size=224, dynamic_image_size=False,
use_thumbnail=False, max_num=6):
# run for each subject
meta_path = os.path.join(root, 'Questions.csv')
with open(meta_path, 'r') as file:
csv_reader = csv.DictReader(file)
data = [row for row in csv_reader]
self.data = data
self.root = root
self.prompt = prompt
self.input_size = input_size
self.dynamic_image_size = dynamic_image_size
self.use_thumbnail = use_thumbnail
self.max_num = max_num
self.transform = build_transform(is_train=False, input_size=input_size)
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
data = self.data[idx]
# {'lndex': '1', 'Question': "Are the butterfly's wings closer to being open or closed?",
# 'Options': '(a) Open (b) Closed', 'Correct Answer': '(a)'}
data_id = data['lndex']
question = data['Question']
# question = '<image>\n' + question
image = os.path.join(self.root + '/MMVP Images', data_id + '.jpg')
image = Image.open(image).convert('RGB')
options = data['Options'].split('(b)')
options[0] = options[0].replace('(a)', '').strip()
options[1] = options[1].replace('(b)', '').strip()
answer = data['Correct Answer'] if 'Correct Answer' in data else None
answer = answer.replace('(a)', 'A').replace('(b)', 'B').replace('(c)', 'C').replace('(d)', 'D')
choice_list = []
new_options = {}
multiple_choices = ['A', 'B', 'C', 'D']
for i, c in enumerate(options):
choice_list.append('{}. {}'.format(multiple_choices[i], c.strip()))
new_options[multiple_choices[i]] = c.strip()
choice_txt = '\n'.join(choice_list)
if self.dynamic_image_size:
images = dynamic_preprocess(image, image_size=self.input_size,
use_thumbnail=self.use_thumbnail,
max_num=self.max_num)
else:
images = [image]
pixel_values = [self.transform(image) for image in images]
pixel_values = torch.stack(pixel_values)
if len(choice_txt) > 0:
question += '\n' + choice_txt
question += '\n' + self.prompt
print(question, answer)
return {
'question': question,
'pixel_values': pixel_values,
'answer': answer,
'option': new_options,
'data_id': data_id
}
class InferenceSampler(torch.utils.data.sampler.Sampler):
def __init__(self, size):
self._size = int(size)
assert size > 0
self._rank = torch.distributed.get_rank()
self._world_size = torch.distributed.get_world_size()
self._local_indices = self._get_local_indices(size, self._world_size, self._rank)
@staticmethod
def _get_local_indices(total_size, world_size, rank):
shard_size = total_size // world_size
left = total_size % world_size
shard_sizes = [shard_size + int(r < left) for r in range(world_size)]
begin = sum(shard_sizes[:rank])
end = min(sum(shard_sizes[:rank + 1]), total_size)
return range(begin, end)
def __iter__(self):
yield from self._local_indices
def __len__(self):
return len(self._local_indices)
def post_process(pred, option):
pred = pred.strip()
option_candidate = list(option.keys())
if len(pred) == 1:
return pred
elif len(pred) != 1 and pred[0] in option_candidate:
return pred[0]
elif len(pred) != 1 and pred[0] not in option_candidate:
for k, v in option.items():
if v in pred:
return k
return pred
def evaluate_chat_model():
prompt = "Answer with the option's letter from the given choices directly."
random.seed(args.seed)
for ds_name in args.datasets:
dataset = MMVPDataset(
root=ds_collections[ds_name]['root'], # hf dataset path.
prompt=prompt,
input_size=image_size,
dynamic_image_size=args.dynamic,
use_thumbnail=use_thumbnail,
max_num=args.max_num
)
dataloader = torch.utils.data.DataLoader(
dataset=dataset,
sampler=InferenceSampler(len(dataset)),
batch_size=args.batch_size,
num_workers=args.num_workers,
pin_memory=True,
drop_last=False,
collate_fn=partial(collate_fn, tokenizer=tokenizer),
)
outputs = []
for _, (pixel_values, questions, answers, data_ids, options) in tqdm(enumerate(dataloader)):
pixel_values = pixel_values.to(torch.bfloat16).cuda()
generation_config = dict(
num_beams=args.num_beams,
max_new_tokens=ds_collections[ds_name]['max_new_tokens'],
min_new_tokens=ds_collections[ds_name]['min_new_tokens'],
do_sample=True if args.temperature > 0 else False,
temperature=args.temperature,
)
pred = model.chat(
tokenizer=tokenizer,
pixel_values=pixel_values,
question=questions[0],
generation_config=generation_config,
verbose=True
)
if len(options[0]) == 0:
preds = [pred]
else:
preds = [post_process(pred, options[0])]
for question, pred, answer, data_id in zip(questions, preds, answers, data_ids):
outputs.append({
'question': question,
'answer': pred,
'gt_answers': answer,
'data_id': data_id
})
torch.distributed.barrier()
world_size = torch.distributed.get_world_size()
merged_outputs = [None for _ in range(world_size)]
torch.distributed.all_gather_object(merged_outputs, json.dumps(outputs))
merged_outputs = [json.loads(_) for _ in merged_outputs]
merged_outputs = [_ for _ in itertools.chain.from_iterable(merged_outputs)]
if torch.distributed.get_rank() == 0:
print(f'Evaluating {ds_name} ...')
time_prefix = time.strftime('%y%m%d%H%M%S', time.localtime())
results_file = f'{ds_name}_{time_prefix}.jsonl'
output_path = os.path.join(args.out_dir, results_file)
writer = open(output_path, 'w')
num_correct, num_total = 0, 0
index, round_correct = 0, 0
for item in merged_outputs:
writer.write(json.dumps(item) + '\n')
answer = item['answer']
gt_answer = item['gt_answers']
index += 1
if answer == gt_answer:
round_correct += 1
if index == 2:
index = 0
if round_correct == 2:
num_correct += 1
round_correct = 0
num_total += 1
writer.close()
print('Results saved to {}'.format(output_path))
print(f'The accuracy is {num_correct/num_total}')
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--checkpoint', type=str, default='')
parser.add_argument('--datasets', type=str, default='MMVP')
parser.add_argument('--batch-size', type=int, default=1)
parser.add_argument('--num-workers', type=int, default=1)
parser.add_argument('--num-beams', type=int, default=5)
parser.add_argument('--temperature', type=float, default=0.0)
parser.add_argument('--out-dir', type=str, default='results')
parser.add_argument('--seed', type=int, default=0)
parser.add_argument('--dynamic', action='store_true')
parser.add_argument('--max-num', type=int, default=6)
parser.add_argument('--load-in-8bit', action='store_true')
parser.add_argument('--load-in-4bit', action='store_true')
parser.add_argument('--auto', action='store_true')
args = parser.parse_args()
if not os.path.exists(args.out_dir):
os.makedirs(args.out_dir)
args.datasets = args.datasets.split(',')
print('datasets:', args.datasets)
assert args.batch_size == 1, 'Only batch size 1 is supported'
torch.distributed.init_process_group(
backend='nccl',
world_size=int(os.getenv('WORLD_SIZE', '1')),
rank=int(os.getenv('RANK', '0')),
)
torch.cuda.set_device(int(os.getenv('LOCAL_RANK', 0)))
if args.auto:
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
kwargs = {'device_map': 'auto'} if args.auto else {}
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint, trust_remote_code=True, use_fast=False)
model = InternVLChatModel.from_pretrained(
args.checkpoint, low_cpu_mem_usage=True, torch_dtype=torch.bfloat16,
load_in_8bit=args.load_in_8bit, load_in_4bit=args.load_in_4bit, **kwargs).eval()
if not args.load_in_8bit and not args.load_in_4bit and not args.auto:
model = model.cuda()
image_size = model.config.force_image_size or model.config.vision_config.image_size
use_thumbnail = model.config.use_thumbnail
total_params = sum(p.numel() for p in model.parameters()) / 1e9
if total_params > 20 or args.dynamic:
args.num_beams = 1
print(f'[test] total_params: {total_params}B, use num_beams: {args.num_beams}')
else:
print(f'[test] total_params: {total_params}B')
print(f'[test] image_size: {image_size}')
print(f'[test] template: {model.config.template}')
print(f'[test] dynamic_image_size: {args.dynamic}')
print(f'[test] use_thumbnail: {use_thumbnail}')
print(f'[test] max_num: {args.max_num}')
evaluate_chat_model()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment