Skip to content

GitLab

"include/ck/config.hpp" did not exist on "6014185ac65e75f2a84cb67ef6ba83b48ae0fcb3"
Commit 1d5a34cf authored by wanglch's avatar wanglch
Browse files

Initial commit

parents
Pipeline #1446 canceled with stages
Hide whitespace changes
Inline Side-by-side
Showing with 4716 additions and 0 deletions
clip_benchmark/benchmark/README.md 0 → 100644
View file @ 1d5a34cf
# Benchmark
the benchmark results are available in [benchmark.csv](benchmark.csv).
You can visualize the results in the [notebook](results.ipynb)
# How to reproduce the CLIP benchmark results
## Webdataset evaluation: VTAB+ and retrieval datasets (MSCOCO, Flickr8k, Flickr30k)
```bash
clip_benchmark eval --pretrained_model openai openclip_base \
--dataset "webdatasets.txt" \
--dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" \
--output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Once the evaluation finishes, you can construct a CSV with all the results:
```bash
clip_benchmark build benchmark_*.json --output benchmark.csv
```
*Notes:* Pascal VOC 2007 multilabel is not yet included in the webdataset test suite. Multilingual support with webdataset is in progress.
## Alternative: Local download
```bash
clip_benchmark eval --pretrained_model openai openclip_base --dataset vtab+ retrieval \
--dataset_root "clip_benchmark_datasets/{dataset}" \
--output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
(Change `--dataset_root` accordingly)
## Multilingual ImageNet benchmark
To run the multilingual ImageNet benchmark, use:
```bash
clip_benchmark eval --pretrained_model openclip_multilingual openclip_base openai --dataset imagenet1k --language cn it jp en ar\
--dataset_root "clip_benchmark_datasets/{dataset}" \
--output "multilingual_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
(Change `--dataset_root` accordingly)
## Multilingual MS-COCO benchmark
To run the multilingual MS-COCO benchmark, use:
```bash
clip_benchmark eval --pretrained_model openclip_multilingual openclip_base openai --dataset multilingual_mscoco_captions --language es it ko pl ru tr zh en \
--dataset_root "clip_benchmark_datasets/{dataset}" \
--output "multilingual_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
(Change `--dataset_root` accordingly)
clip_benchmark/benchmark/benchmark.csv 0 → 100644
View file @ 1d5a34cf
acc1,acc5,mean_per_class_recall,dataset,model,pretrained,task,mean_average_precision,image_retrieval_recall@5,text_retrieval_recall@5,model_fullname
0.0232340494791666,0.1152615017361111,0.0242046402834269,vtab/dsprites_label_orientation,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.4460852605182502,0.9469211479520758,0.3940612716631316,fer2013,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.7738666666666667,0.9362666666666668,0.7345750490593081,imagenet-a,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4457446808510638,0.7585106382978724,0.449468085106383,vtab/dtd,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7521562425290105,0.9608106368501388,0.7512820500659019,sun397,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,voc2007_multilabel,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,0.796263575553894,,,ViT-B-32 laion2b_s34b_b79k
0.7557,0.9386,0.7554000000000001,vtab/cifar100,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.665,0.89844,0.66506,imagenet1k,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1889597536837475,0.819793270288102,0.1681683759314615,vtab/dmlab,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.4955,0.8497,0.5259367109048434,mnist,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5175536361103371,0.9651713569239344,0.5055924943171644,fer2013,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6246666789267521,0.91313422954558,0.6346910684418603,sun397,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.9070591441809758,0.995366584900518,0.9070894162634328,vtab/pets,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.5609659540775931,0.8069675376088677,0.5169427663206733,gtsrb,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.0313720703125,0.1469997829861111,0.0320098582855241,vtab/dsprites_label_x_position,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.3628226797787339,0.6565765212046711,0.4033487053539641,vtab/svhn,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.4362628661916072,0.7091844813935075,0.370358797280688,gtsrb,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5535494775660725,0.9136447449293178,0.5591738787984386,vtab/svhn,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4417465274038979,0.7008721869279638,0.4272611771980346,objectnet,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7174074074074074,0.9561111111111112,0.7202760268647987,vtab/eurosat,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.501821060965954,0.7586698337292161,0.4394378810250903,gtsrb,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.9321340964840557,0.9978195693649496,0.9313974108097984,vtab/pets,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.4934283452098179,0.7430720506730008,0.4353727539920664,gtsrb,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8522353714661407,0.963346482577252,0.944284654839904,vtab/caltech101,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
,,,flickr30k,ViT-B-32-quickgelu,laion400m_e32,zeroshot_retrieval,,0.8546000123023987,0.9409999847412108,ViT-B-32-quickgelu laion400m_e32
0.1178600823045267,0.5817283950617284,0.1208963734895024,vtab/smallnorb_label_elevation,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.76664,0.9485,0.76656,imagenet1k,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.7679,0.9386,0.7581398074696393,mnist,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.2883263009845288,,0.3645688070267072,vtab/kitti_closest_vehicle_distance,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.1854185418541854,0.4452445244524452,0.1875846702317291,fgvc_aircraft,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0197618272569444,0.1133355034722222,0.0176459217575104,vtab/dsprites_label_orientation,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
,,,flickr8k,ViT-B-16-plus-240,laion400m_e32,zeroshot_retrieval,,0.873199999332428,0.9549999833106995,ViT-B-16-plus-240 laion400m_e32
0.551063829787234,0.8356382978723405,0.5499999999999999,vtab/dtd,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8391,0.9729,0.8388,vtab/cifar100,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.482037037037037,0.935,0.493913656654034,vtab/eurosat,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.6828752642706131,0.8578630671653927,0.6628139602370955,vtab/flowers,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5844813935075218,0.820744259699129,0.5442606899522975,gtsrb,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.50628662109375,,0.5062329426609509,vtab/pcam,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.6932,0.8887,0.6785851251044699,imagenet-r,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.2304230423042304,0.5295529552955296,0.2319696969696969,fgvc_aircraft,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.605010986328125,,0.605165824864527,vtab/pcam,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr8k,ViT-B-16,laion400m_e32,zeroshot_retrieval,,0.8575999736785889,0.9409999847412109,ViT-B-16 laion400m_e32
0.7835420393559929,0.9242153195641568,0.7862863094545812,vtab/flowers,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.9467,0.999,0.9466,vtab/cifar10,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.6961,0.9086,0.6957000000000001,imagenetv2,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.4345209817893903,0.707680126682502,0.400638686800972,gtsrb,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.0317789713541666,0.147705078125,0.0324921668671336,vtab/dsprites_label_x_position,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.7871,0.9505,0.7775130720394369,mnist,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.63334,0.88778,0.63284,imagenet1k,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6031914893617021,0.8867021276595745,0.6042553191489363,vtab/dtd,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.985875,0.99975,0.9864999999999998,stl10,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2636018957345971,0.5149289099526067,0.2629857819905213,country211,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5132978723404256,0.7787234042553192,0.5095744680851063,vtab/dtd,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.4342160768689946,1.0,0.2167811161586062,vtab/diabetic_retinopathy,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5512,0.8156,0.5509,imagenetv2,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7105220361034315,0.8590014636526264,0.6857234783638442,vtab/flowers,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6867793368519779,0.9372344925244128,0.6845985471139586,sun397,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.5016666666666667,0.959074074074074,0.511474800858698,vtab/eurosat,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.8468,0.9733,0.8471000000000001,vtab/cifar100,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,voc2007_multilabel,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,0.801436722278595,,,ViT-H-14 laion2b_s32b_b79k
0.9172,0.9975,0.9172,vtab/cifar10,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6981352410026298,0.9398183055335896,0.6849982147927691,sun397,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.77972,0.95216,0.77952,imagenet1k,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.029541015625,0.1558973524305555,0.0292117961574083,vtab/dsprites_label_x_position,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5,0.7502771179730799,0.4499584478173962,gtsrb,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7590333333333333,0.9128666666666668,0.7444515544684158,imagenet-r,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.4690721649484536,0.931178601281694,0.4334946917742944,fer2013,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.485626220703125,,0.4856418903784925,vtab/pcam,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.9438539111474517,0.9986372308530936,0.9434557685576204,vtab/pets,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,flickr8k,ViT-L-14,laion400m_e32,zeroshot_retrieval,,0.8984000086784363,0.9649999737739563,ViT-L-14 laion400m_e32
0.0972839506172839,0.5397530864197531,0.0973286727349915,vtab/smallnorb_label_elevation,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5159912109375,,0.5157975788193991,vtab/pcam,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.7824519230769231,0.9688835470085472,0.8629106820310023,voc2007,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0293918185763888,0.150390625,0.0306791200330755,vtab/dsprites_label_x_position,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5710666666666666,0.8348,0.5639196371233688,imagenet-a,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6408566721581549,,0.6410094956864107,renderedsst2,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1134156378600823,0.5579423868312757,0.1146512139135114,vtab/smallnorb_label_elevation,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1590499230261711,0.8403782713877281,0.1701282125753662,vtab/dmlab,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
,,,voc2007_multilabel,ViT-B-32,laion2b_e16,zeroshot_classification,0.7927550077438354,,,ViT-B-32 laion2b_e16
,,,mscoco_captions,ViT-B-16,laion400m_e32,zeroshot_retrieval,,0.6364254355430603,0.7961999773979187,ViT-B-16 laion400m_e32
0.758985200845666,0.8894129126687266,0.7455642498757189,vtab/flowers,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-32,openai,zeroshot_classification,0.7601363658905029,,,ViT-B-32 openai
0.6968,0.9081,0.6974,imagenetv2,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8893431452711911,0.9940038157536112,0.884512216368383,vtab/pets,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7021,0.9244,0.703,vtab/cifar100,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5072222222222222,0.9255555555555556,0.489609055911575,vtab/eurosat,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7067545304777595,,0.7068315384169996,renderedsst2,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.665743087897188,0.8816640138340309,0.6655023529411764,imagenet_sketch,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.5155555555555555,0.9201851851851852,0.526225901185735,vtab/eurosat,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
,,,flickr30k,ViT-B-32,openai,zeroshot_retrieval,,0.8338000178337097,0.9490000009536744,ViT-B-32 openai
0.1593333333333333,0.9299333333333332,0.1673057808855792,vtab/clevr_closest_object_distance,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7464,0.9285,0.7471,vtab/cifar100,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
,,,mscoco_captions,ViT-L-14,laion400m_e32,zeroshot_retrieval,,0.6805678009986877,0.8216000199317932,ViT-L-14 laion400m_e32
0.9307713273371492,0.9980921231943308,0.9330900923082088,vtab/pets,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6188,0.8745,0.6202000000000001,imagenetv2,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.4230383776454636,0.7030792509186661,0.4230898039215686,imagenet_sketch,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7910657051282052,0.9600026709401708,0.8052125971178338,voc2007,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1674950516824279,0.8040906091928745,0.1782001238774596,vtab/dmlab,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8310322156476002,0.9529914529914528,0.903296243135675,vtab/caltech101,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1094650205761317,0.5571193415637861,0.1098932195729559,vtab/smallnorb_label_elevation,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5716,0.8386,0.5721,imagenetv2,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1080658436213991,0.5204115226337449,0.108510287776451,vtab/smallnorb_label_elevation,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.8982,0.9963,0.8995000000000001,vtab/cifar10,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.9381302807304442,0.9986372308530936,0.9372081115891412,vtab/pets,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5819,0.8386,0.5815,imagenetv2,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.6666666666666666,0.941904761904762,0.6759805529181834,vtab/resisc45,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,0.7846916913986206,,,ViT-B-16-plus-240 laion400m_e32
0.5363831083338246,0.7926467409459804,0.53684,imagenet_sketch,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7930605646063923,0.986693197363512,0.79277195221544,cars,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.6457990115321252,,0.6459400874297956,renderedsst2,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6445715707001617,0.9434149981345604,0.6469166001999892,cars,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.503753662109375,,0.5035515136049098,vtab/pcam,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.5706822372464659,0.9138752304855562,0.5886888789913961,vtab/svhn,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.7761084401709402,0.9418402777777778,0.8508423918048074,voc2007,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1884360189573459,0.4163507109004739,0.1883412322274882,country211,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.97075,0.999375,0.971875,stl10,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1508247195953375,0.8088849791071036,0.1720986035113953,vtab/dmlab,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.2629716428404031,1.0,0.2194476579174435,vtab/diabetic_retinopathy,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.4630454824830977,0.817340196681008,0.4869931863892911,vtab/svhn,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,mscoco_captions,ViT-B-32,laion2b_e16,zeroshot_retrieval,,0.6467413306236267,0.7950000166893005,ViT-B-32 laion2b_e16
0.270042194092827,,0.3517916468296155,vtab/kitti_closest_vehicle_distance,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7082,0.9169,0.709,imagenetv2,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6352380952380953,0.9225396825396824,0.6419889996880412,vtab/resisc45,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5404255319148936,0.8398936170212766,0.5367021276595745,vtab/dtd,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.2236286919831223,,0.3717030717825018,vtab/kitti_closest_vehicle_distance,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5850793650793651,0.910952380952381,0.5919202546199539,vtab/resisc45,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7567441239316239,0.9461805555555556,0.7914514618991711,voc2007,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.3319333333333333,0.9534,0.3193231509666999,vtab/clevr_count_all,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5661724327292696,,0.5658672775172254,renderedsst2,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2686357243319268,,0.3735376764204429,vtab/kitti_closest_vehicle_distance,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.1856168902573125,0.82608313173521,0.1925396140565279,vtab/dmlab,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.8504273504273504,0.9681130834976988,0.9394300669046936,vtab/caltech101,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7768934212162666,0.983957219251337,0.777448930592225,cars,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.3048171481253841,0.7613706207744315,0.3503741918499782,vtab/svhn,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.96575,0.999375,0.966625,stl10,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.6219482421875,,0.6220625731388706,vtab/pcam,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
,,,flickr30k,ViT-L-14,laion400m_e32,zeroshot_retrieval,,0.9082000255584716,0.977999985218048,ViT-L-14 laion400m_e32
0.72734,0.9293,0.72694,imagenet1k,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7766666666666666,0.9304333333333332,0.7605432098970494,imagenet-r,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.6137,0.8644,0.6146999999999999,imagenetv2,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.2877333333333333,0.9075333333333332,0.2821869879006831,vtab/clevr_count_all,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6190740740740741,0.962037037037037,0.6309344676440423,vtab/eurosat,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0313313802083333,0.1560872395833333,0.031335128135504,vtab/dsprites_label_x_position,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1149794238683127,0.5761316872427984,0.115904554765943,vtab/smallnorb_label_elevation,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7760611481541714,0.9046999512115792,0.7813275676765017,vtab/flowers,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.8865,0.9695666666666668,0.8748158307459671,imagenet-r,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
,,,mscoco_captions,ViT-H-14,laion2b_s32b_b79k,zeroshot_retrieval,,0.734306275844574,0.8604000210762024,ViT-H-14 laion2b_s32b_b79k
0.7524,0.9418,0.7528999999999999,vtab/cifar100,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.2618666666666667,0.5721333333333334,0.2839019082932269,imagenet-a,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.9345852505907224,0.9988807362268376,0.9351484667320789,cars,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.4286709389802173,0.898300362217888,0.392124222029496,fer2013,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.68352,0.91864,0.68396,imagenet1k,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.1123456790123456,0.5467489711934156,0.1133381228564946,vtab/smallnorb_label_elevation,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4932893159621922,0.7566861207726621,0.4940521568627451,imagenet_sketch,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5009752020061299,0.932153803287824,0.449919123283142,fer2013,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7549,0.946,0.7545,imagenet1k,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.0346001519097222,0.1698269314236111,0.0339338982697889,vtab/dsprites_label_x_position,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.71777753849467,0.9007214385700442,0.7007187288769688,objectnet,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.7294,0.9415,0.7332910901261492,mnist,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0342746310763888,0.1416965060763889,0.034365141983162,vtab/dsprites_label_orientation,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.210639793766112,1.0,0.2335698910327324,vtab/diabetic_retinopathy,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8392504930966469,0.9510190664036818,0.9090841082001052,vtab/caltech101,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2841068917018284,,0.4076831334000694,vtab/kitti_closest_vehicle_distance,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6813829787234043,0.925531914893617,0.6829787234042553,vtab/dtd,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6909,0.91432,0.69156,imagenet1k,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.2424666666666666,0.7876,0.2312503165591237,vtab/clevr_count_all,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.67002,0.90424,0.67026,imagenet1k,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-B-16,laion400m_e32,zeroshot_retrieval,,0.881600022315979,0.9679999947547911,ViT-B-16 laion400m_e32
0.8787,0.9709333333333332,0.8651131734542029,imagenet-r,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.62918,0.87652,0.6289,imagenet1k,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5469522240527183,,0.5464163192635053,renderedsst2,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.4425531914893617,0.7638297872340426,0.4430851063829787,vtab/dtd,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.5149133196941962,0.7512652094325402,0.5017005288059357,objectnet,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.4855,0.8418,0.4575381785680641,mnist,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6550016151609777,0.8503284160654678,0.6433184742666794,objectnet,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7744,0.9471,0.7737999999999998,vtab/cifar100,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7920656634746922,0.9787339883099117,0.7926165075935756,cars,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.2081575246132208,,0.1791674645508319,vtab/kitti_closest_vehicle_distance,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0260687934027777,0.1259223090277778,0.0268337475785376,vtab/dsprites_label_orientation,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6922,0.9403,0.6883700135057857,mnist,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1143209876543209,0.5373662551440329,0.1138628810502107,vtab/smallnorb_label_elevation,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6769,0.9012,0.6781,imagenetv2,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7683627136752137,0.952590811965812,0.8035754023986389,voc2007,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.8504273504273504,0.953155818540434,0.9440706929933655,vtab/caltech101,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6434889751181566,0.9243614027989776,0.6527406670624641,sun397,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.3311163895486936,0.7250989707046714,0.3196447660118034,gtsrb,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6521841655367565,0.8748649020416986,0.6524090196078433,imagenet_sketch,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1469194312796208,0.3500473933649289,0.1470142180094787,country211,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.96875,0.99975,0.9695,stl10,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6908043501669,0.8805319263486594,0.6736647184602601,objectnet,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.984375,0.999875,0.9849999999999998,stl10,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.5544309249488533,0.7926671691611931,0.5363732822578842,objectnet,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7555422008547008,0.9489182692307692,0.830992972154603,voc2007,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5293285385839769,0.78944369117098,0.5286741176470588,imagenet_sketch,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.7889957264957265,0.9573317307692308,0.8054447759101763,voc2007,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.675285506739982,0.9369310554094562,0.6824513086557495,sun397,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.111275720164609,0.5548971193415638,0.1102182875784799,vtab/smallnorb_label_elevation,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,flickr8k,ViT-B-32,laion2b_e16,zeroshot_retrieval,,0.8574000000953674,0.9319999814033508,ViT-B-32 laion2b_e16
0.5989555292344136,0.8103262625174976,0.586320244179702,objectnet,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
,,,mscoco_captions,ViT-g-14,laion2b_s12b_b42k,zeroshot_retrieval,,0.7239903807640076,0.853600025177002,ViT-g-14 laion2b_s12b_b42k
0.7101,0.9209,0.7106999999999999,vtab/cifar100,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.189873417721519,,0.2568338943834677,vtab/kitti_closest_vehicle_distance,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,mscoco_captions,ViT-B-32,laion2b_s34b_b79k,zeroshot_retrieval,,0.654218316078186,0.7982000112533569,ViT-B-32 laion2b_s34b_b79k
0.1186008230452674,0.560082304526749,0.1176434679315522,vtab/smallnorb_label_elevation,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5634266886326195,,0.5635892536622082,renderedsst2,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
,,,flickr8k,ViT-L-14,laion2b_s32b_b82k,zeroshot_retrieval,,0.9147999882698059,0.9670000076293945,ViT-L-14 laion2b_s32b_b82k
0.0237358940972222,0.1186116536458333,0.0218059467808842,vtab/dsprites_label_orientation,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1316147176001874,1.0,0.230780955782727,vtab/diabetic_retinopathy,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.8599676657132197,0.9912946150976246,0.8615494787047302,cars,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.5569148936170213,0.8361702127659575,0.5563829787234043,vtab/dtd,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
,,,flickr30k,ViT-B-32,laion2b_s34b_b79k,zeroshot_retrieval,,0.8835999965667725,0.9629999995231628,ViT-B-32 laion2b_s34b_b79k
0.0715256620576517,1.0,0.2196341124982623,vtab/diabetic_retinopathy,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5699,0.8279,0.5675693520579019,mnist,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.5940803382663847,0.917174480785972,0.5967107342155968,cars,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.98275,1.0,0.982875,stl10,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.4199524940617577,0.6838479809976247,0.393417229364651,gtsrb,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.0633231778767283,1.0,0.2107285863733837,vtab/diabetic_retinopathy,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.9497,0.9963,0.9497,vtab/cifar10,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.76548,0.95168,0.7656000000000001,imagenet1k,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.2742616033755274,,0.405770386325608,vtab/kitti_closest_vehicle_distance,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.2380126552613077,1.0,0.233320813717688,vtab/diabetic_retinopathy,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.2307582938388625,0.4708056872037914,0.2308530805687204,country211,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.2633333333333333,0.574,0.2790514196577626,imagenet-a,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,voc2007_multilabel,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,0.7621888518333435,,,ViT-B-32-quickgelu laion400m_e32
0.199,0.7285333333333334,0.1948019376127608,vtab/clevr_count_all,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5376190476190477,0.8671428571428571,0.5417275836816031,vtab/resisc45,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.0302191840277777,0.1466335720486111,0.0300817149220147,vtab/dsprites_label_x_position,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,flickr8k,ViT-g-14,laion2b_s12b_b42k,zeroshot_retrieval,,0.9175999760627747,0.9739999771118164,ViT-g-14 laion2b_s12b_b42k
0.157994281944139,0.8222564328128437,0.1661260800820199,vtab/dmlab,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1812322274881516,0.4011374407582938,0.181563981042654,country211,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.9272,0.9988,0.9272000000000002,vtab/cifar10,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.9401,0.9992,0.9405,vtab/cifar10,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.7331380360909304,1.0,0.206266837915639,vtab/diabetic_retinopathy,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.2587904360056259,,0.3397102822066769,vtab/kitti_closest_vehicle_distance,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2166666666666666,0.5310666666666667,0.2348332752565157,imagenet-a,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.99375,1.0,0.993625,stl10,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.087368174361378,1.0,0.2520359622114083,vtab/diabetic_retinopathy,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.1236939151813153,0.6480485556238476,0.1332639799678708,vtab/svhn,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.5782537067545305,,0.5785203520352036,renderedsst2,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0588477366255144,0.268395061728395,0.0601925439678357,vtab/smallnorb_label_azimuth,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6367,0.9218,0.6276012948452819,mnist,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.0313856336805555,0.1553955078125,0.0307666564982354,vtab/dsprites_label_x_position,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.9569,0.9963,0.9572,vtab/cifar10,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.4973871733966746,0.7704671417260491,0.4655936453259506,gtsrb,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5853968253968254,0.896984126984127,0.5932699170625959,vtab/resisc45,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.1419837255333186,0.8292060699362217,0.16573619863836,vtab/dmlab,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.612762451171875,,0.6128028129314551,vtab/pcam,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.1945898394545854,0.8513305476138113,0.1635130198328585,vtab/dmlab,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7259778950659286,0.9500616069294004,0.7127784763377728,sun397,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.4661333333333333,0.7688,0.4728184257301568,imagenet-a,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.6371428571428571,0.9277777777777778,0.6463008933552572,vtab/resisc45,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.1111111111111111,,0.2722929936305732,vtab/kitti_closest_vehicle_distance,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6105248678496336,0.8503998899565721,0.6103113725490197,imagenet_sketch,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.8732624693376942,0.9926410466067048,0.8695864391489154,vtab/pets,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1486,0.9095333333333332,0.1443124682607226,vtab/clevr_closest_object_distance,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0243462456597222,0.1128879123263889,0.0249444752915047,vtab/dsprites_label_orientation,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.596238872840889,0.8439741397944546,0.5964254901960785,imagenet_sketch,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6276595744680851,0.9069148936170212,0.6324468085106383,vtab/dtd,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-16,openai,zeroshot_classification,0.788827121257782,,,ViT-B-16 openai
,,,flickr30k,ViT-L-14,openai,zeroshot_retrieval,,0.8715999722480774,0.9739999771118164,ViT-L-14 openai
0.2505250525052505,0.6012601260126013,0.2483244206773618,fgvc_aircraft,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5559259259259259,0.8868518518518519,0.5469811732579133,vtab/eurosat,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.9427636958299264,0.9983646770237122,0.9434000102313552,vtab/pets,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1746174617461746,0.45004500450045,0.1753386809269162,fgvc_aircraft,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-B-16,openai,zeroshot_retrieval,,0.855400025844574,0.9629999995231628,ViT-B-16 openai
0.8934314527119106,0.9956391387298992,0.8906060208128682,vtab/pets,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.6266666666666667,0.9611111111111112,0.6380077170682305,vtab/eurosat,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5596,0.8341,0.5602,imagenetv2,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.481439996855902,0.763229774607479,0.4822572549019607,imagenet_sketch,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.2450666666666666,0.8254,0.1666666666666666,vtab/clevr_closest_object_distance,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-H-14,laion2b_s32b_b79k,zeroshot_retrieval,,0.9409999847412108,0.9929999709129332,ViT-H-14 laion2b_s32b_b79k
,,,voc2007_multilabel,ViT-L-14,openai,zeroshot_classification,0.7903817892074585,,,ViT-L-14 openai
0.4267205349679576,0.9361939258846476,0.3989364402674789,fer2013,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.6960663515824705,0.9390183349577946,0.6804128851625355,sun397,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.3318666666666666,0.6664,0.3409181030776702,imagenet-a,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.9083,0.9944,0.9082,vtab/cifar10,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
,,,voc2007_multilabel,ViT-L-14-336,openai,zeroshot_classification,0.8035513162612915,,,ViT-L-14-336 openai
0.5922666666666667,0.8565333333333334,0.5810468077571583,imagenet-a,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6730158730158731,0.938095238095238,0.6781338184038964,vtab/resisc45,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7354112959925005,1.0,0.1999235670840696,vtab/diabetic_retinopathy,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,flickr30k,ViT-B-16-plus-240,laion400m_e32,zeroshot_retrieval,,0.8894000053405762,0.9710000157356262,ViT-B-16-plus-240 laion400m_e32
,,,flickr8k,ViT-B-32,openai,zeroshot_retrieval,,0.805400013923645,0.9139999747276306,ViT-B-32 openai
0.5897858319604613,,0.5895015488390944,renderedsst2,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.5388,0.8212,0.5362716338708938,imagenet-a,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,mscoco_captions,ViT-B-16,openai,zeroshot_retrieval,,0.5836865305900574,0.7681999802589417,ViT-B-16 openai
,,,flickr8k,ViT-B-32,laion2b_s34b_b79k,zeroshot_retrieval,,0.8629999756813049,0.9409999847412109,ViT-B-32 laion2b_s34b_b79k
0.99425,0.999875,0.9945,stl10,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5612707437000615,0.8849492931776275,0.5565312569182044,vtab/svhn,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1097119341563786,0.5409053497942387,0.1081320376350696,vtab/smallnorb_label_elevation,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.805221688034188,0.9551282051282052,0.8491537874687232,voc2007,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,flickr8k,ViT-L-14-336,openai,zeroshot_retrieval,,0.8795999884605408,0.9390000104904175,ViT-L-14-336 openai
0.1716,0.9095333333333332,0.1619443104834767,vtab/clevr_closest_object_distance,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1573333333333333,0.6964666666666667,0.1495026928557915,vtab/clevr_count_all,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,mscoco_captions,ViT-B-16-plus-240,laion400m_e32,zeroshot_retrieval,,0.6620951890945435,0.8101999759674072,ViT-B-16-plus-240 laion400m_e32
0.8336,0.9666,0.8325000000000001,vtab/cifar100,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.0984362139917695,0.5293827160493827,0.0977694426163014,vtab/smallnorb_label_elevation,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,voc2007_multilabel,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,0.8066232800483704,,,ViT-g-14 laion2b_s12b_b42k
0.6087301587301587,0.9147619047619048,0.6152225643499313,vtab/resisc45,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.0550617283950617,0.2645267489711934,0.055379474051494,vtab/smallnorb_label_azimuth,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1676666666666666,0.8819333333333333,0.1954414937508546,vtab/clevr_closest_object_distance,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0200330946180555,0.1053195529513889,0.0222572574909185,vtab/dsprites_label_orientation,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5526123046875,,0.5526478181769814,vtab/pcam,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,mscoco_captions,ViT-L-14,openai,zeroshot_retrieval,,0.6108356714248657,0.7918000221252441,ViT-L-14 openai
,,,mscoco_captions,ViT-B-32-quickgelu,laion400m_e32,zeroshot_retrieval,,0.6084766387939453,0.7675999999046326,ViT-B-32-quickgelu laion400m_e32
0.7537811026183119,0.8923402179216132,0.7256696912813558,vtab/flowers,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.1659634317862166,,0.3247233185334074,vtab/kitti_closest_vehicle_distance,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.0193684895833333,0.1180826822916666,0.0197744129948227,vtab/dsprites_label_orientation,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.9037884982284,0.9970019078768056,0.9039815014388496,vtab/pets,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.3742,0.7294,0.3706020613065869,mnist,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.6746527403897922,0.8668030580381177,0.6650572756513145,objectnet,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.765625,0.959869123931624,0.8071517771314477,voc2007,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.546031746031746,0.902063492063492,0.5542849348347576,vtab/resisc45,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.0427924262152777,0.1601019965277778,0.0430371102717507,vtab/dsprites_label_x_position,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6957142857142857,0.9571428571428572,0.706242238089474,vtab/resisc45,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6514814814814814,0.9551851851851852,0.6638062361650154,vtab/eurosat,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8475333333333334,0.9550666666666666,0.8331685531673508,imagenet-r,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.1902,0.9085333333333332,0.1387289271305892,vtab/clevr_closest_object_distance,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.3852181929932391,0.7822295636140135,0.379296565517112,vtab/svhn,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.3182464454976303,0.5937914691943128,0.3175829383886256,country211,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6659,0.9224,0.6665327286676481,mnist,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6327693607655879,0.8639588123170037,0.6325447058823529,imagenet_sketch,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.2279620853080568,0.486303317535545,0.2282938388625592,country211,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.764,0.9231,0.7589335620721703,mnist,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0268283420138888,0.1092258029513889,0.025387675404758,vtab/dsprites_label_orientation,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.3693369336933693,0.744974497449745,0.3649286987522281,fgvc_aircraft,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.4102258758451137,0.8022818070067609,0.4216815643098484,vtab/svhn,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.743319785938908,0.9583279695459478,0.7348385903018446,sun397,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,flickr8k,ViT-H-14,laion2b_s32b_b79k,zeroshot_retrieval,,0.9277999997138977,0.9729999899864197,ViT-H-14 laion2b_s32b_b79k
0.5487,0.8411,0.5430718178404617,mnist,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7171428571428572,0.958095238095238,0.7258469461953507,vtab/resisc45,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.3454976303317535,0.6221800947867299,0.3445971563981042,country211,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.9645,0.999375,0.965125,stl10,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.779,0.9289333333333334,0.7643246651538985,imagenet-r,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.1633333333333333,0.7125333333333334,0.1575975877364315,vtab/clevr_count_all,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.8683565004088307,0.9934587080948488,0.8661667839491306,vtab/pets,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.647962962962963,0.9805555555555556,0.6445566610022502,vtab/eurosat,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6929582045861116,0.8668076109936576,0.6668176645957112,vtab/flowers,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6630346397788258,0.8573751829565783,0.6645264657992297,vtab/flowers,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1772666666666666,0.7803333333333333,0.2270014558851883,vtab/clevr_closest_object_distance,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2784,0.8929333333333334,0.2563239722391655,vtab/clevr_count_all,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.8328402366863905,0.94543063773833,0.9085289082247568,vtab/caltech101,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.2786954517516902,0.7308312845728334,0.2795999671407248,vtab/svhn,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.035400390625,0.1624348958333333,0.0364155761076967,vtab/dsprites_label_x_position,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2699554722287321,1.0,0.2211405088000341,vtab/diabetic_retinopathy,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.0259874131944444,0.1214463975694444,0.0262399607702056,vtab/dsprites_label_orientation,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.819197896120973,0.9465811965811964,0.8786521640800292,vtab/caltech101,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.4896907216494845,0.9721370855391472,0.4887152232577444,fer2013,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
,,,voc2007_multilabel,ViT-L-14,laion400m_e32,zeroshot_classification,0.7847012877464294,,,ViT-L-14 laion400m_e32
0.8020816392909416,0.9253537160513904,0.7985485908748879,vtab/flowers,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.5163895486935867,0.7605700712589074,0.4472134621454642,gtsrb,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5255354200988468,,0.5258471570841294,renderedsst2,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.2242797448867385,0.8738508906971629,0.1816019549723685,vtab/dmlab,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5442040519562185,0.8052231327005837,0.5451807843137254,imagenet_sketch,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.545867919921875,,0.5459467330999297,vtab/pcam,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.697049639280715,0.877516959190266,0.6846202591899297,objectnet,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0558847736625514,0.2832098765432099,0.0522661133926831,vtab/smallnorb_label_azimuth,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.2352,0.7871333333333334,0.2199570643185355,vtab/clevr_count_all,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.8039,0.9392333333333334,0.7907448605291261,imagenet-r,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.2292545710267229,,0.3081761474014508,vtab/kitti_closest_vehicle_distance,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.04559670781893,0.274320987654321,0.0456436809466583,vtab/smallnorb_label_azimuth,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.585723876953125,,0.5857677281000415,vtab/pcam,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8932,0.9735333333333334,0.8804663010091829,imagenet-r,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.3677333333333333,0.7024,0.3814206048810433,imagenet-a,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.378037803780378,0.8028802880288028,0.3781639928698752,fgvc_aircraft,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5430851063829787,0.8356382978723405,0.5473404255319149,vtab/dtd,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.9261285909712722,0.9976371098122124,0.9261518670531818,cars,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.6904,0.9518,0.6833555055021581,mnist,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6174603174603175,0.921904761904762,0.624279367877562,vtab/resisc45,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.537057676232934,0.9492894956812482,0.5338791359180816,fer2013,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-16,laion400m_e32,zeroshot_classification,0.7843208312988281,,,ViT-B-16 laion400m_e32
0.4582056283087211,0.9531903037057676,0.4167779537443768,fer2013,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.1594666666666666,0.8705333333333334,0.1702024394180443,vtab/clevr_closest_object_distance,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.955,0.9995,0.955375,stl10,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7640666666666667,0.9159,0.7521727338627011,imagenet-r,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8328402366863905,0.952827087442472,0.9177765781998192,vtab/caltech101,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.53631591796875,,0.53614377703907,vtab/pcam,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0517695473251028,0.2697942386831276,0.0537673960190341,vtab/smallnorb_label_azimuth,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.6453,0.888,0.6451,vtab/cifar100,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
,,,flickr30k,ViT-B-32,laion2b_e16,zeroshot_retrieval,,0.8812000155448914,0.9639999866485596,ViT-B-32 laion2b_e16
0.4393776246365888,0.684612899752342,0.4268760394558317,objectnet,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
,,,flickr30k,ViT-g-14,laion2b_s12b_b42k,zeroshot_retrieval,,0.9348000288009644,0.99099999666214,ViT-g-14 laion2b_s12b_b42k
0.381760909649662,0.7632913337430854,0.4057750407393451,vtab/svhn,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5006666666666667,0.8033333333333333,0.4832835476168289,imagenet-a,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.9705,0.9994,0.9711,vtab/cifar10,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.4800780161604904,0.9711618835330176,0.4909158276198425,fer2013,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
,,,mscoco_captions,ViT-L-14-336,openai,zeroshot_retrieval,,0.615513801574707,0.8101999759674072,ViT-L-14-336 openai
0.7068,0.9062666666666668,0.6753602288814418,imagenet-a,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.9078,0.9977,0.9083,vtab/cifar10,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.3860633066994468,0.8061232329440688,0.3685311302620919,vtab/svhn,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7160513904699951,0.8744511302650837,0.6998995164388323,vtab/flowers,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1971197119711971,0.5022502250225023,0.197344028520499,fgvc_aircraft,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6033343577135832,0.8973570989551322,0.5683458085959752,vtab/svhn,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.4899860019381932,0.7341983417680629,0.4820376550831692,objectnet,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7626201923076923,0.9526575854700856,0.815718996924191,voc2007,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1462728551336146,,0.1818997858588953,vtab/kitti_closest_vehicle_distance,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6787234042553192,0.923936170212766,0.6813829787234043,vtab/dtd,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1721574664614031,0.8585001099626127,0.1577427840925754,vtab/dmlab,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7916734428362335,0.9193364774760124,0.7931691849985836,vtab/flowers,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.0549794238683127,0.2670781893004115,0.0559609415916413,vtab/smallnorb_label_azimuth,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.7843883547008547,0.9570646367521368,0.835061321772101,voc2007,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.6177777777777778,0.957037037037037,0.6299267597724122,vtab/eurosat,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.543701171875,,0.5435553179465629,vtab/pcam,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0315348307291666,0.1596544053819444,0.0323590170008084,vtab/dsprites_label_x_position,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5796296296296296,0.9525925925925924,0.5888803943202612,vtab/eurosat,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.9185064050149904,0.9956391387298992,0.9161666963118203,vtab/pets,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5381177990739744,0.7715085603531818,0.5274232792623416,objectnet,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
,,,flickr30k,ViT-L-14,laion2b_s32b_b82k,zeroshot_retrieval,,0.929199993610382,0.9869999885559082,ViT-L-14 laion2b_s32b_b82k
0.8368362144011939,0.987439373212287,0.8375394945435981,cars,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr8k,ViT-B-32-quickgelu,laion400m_e32,zeroshot_retrieval,,0.8303999900817871,0.9169999957084656,ViT-B-32-quickgelu laion400m_e32
0.9742,0.9994,0.9742999999999998,vtab/cifar10,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.613015873015873,0.91,0.6150226494017114,vtab/resisc45,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1901913349461183,0.8472839234660215,0.1733383864929211,vtab/dmlab,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1979333333333333,0.8076,0.1820160989597282,vtab/clevr_count_all,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
,,,voc2007_multilabel,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,0.8199208974838257,,,ViT-L-14 laion2b_s32b_b82k
0.1674,0.8695333333333334,0.1827530663763387,vtab/clevr_closest_object_distance,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1715165876777251,0.4024170616113744,0.1707582938388625,country211,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.75202,0.94252,0.7526400000000001,imagenet1k,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8923412373943854,0.9967293540474244,0.8919759072052595,vtab/pets,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.487670937870141,0.7249919241951115,0.4750858280106649,objectnet,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.8959084690958836,0.9960203954732,0.8961635505798664,cars,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7832532051282052,0.9692174145299144,0.864352156405908,voc2007,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5031481481481481,0.92,0.5110567650187513,vtab/eurosat,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.161,0.9128666666666668,0.1739135544326629,vtab/clevr_closest_object_distance,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8441736102474816,0.9912946150976246,0.8456794155511405,cars,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.7372,0.9338,0.7372000000000001,vtab/cifar100,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.7136119694259229,0.8591640917222313,0.691284904068223,vtab/flowers,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.1485814822960193,0.8129316032548933,0.1482313831309264,vtab/dmlab,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6112026359143328,,0.6113264286955011,renderedsst2,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5370675453047776,,0.5373060332349024,renderedsst2,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
,,,mscoco_captions,ViT-B-32,openai,zeroshot_retrieval,,0.5584565997123718,0.748199999332428,ViT-B-32 openai
,,,flickr8k,ViT-B-16,openai,zeroshot_retrieval,,0.8285999894142151,0.9139999747276306,ViT-B-16 openai
0.8379355687047995,0.9518408941485864,0.9328934325841473,vtab/caltech101,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0307752821180555,0.1437852647569444,0.0304426618193977,vtab/dsprites_label_orientation,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.841715976331361,0.940828402366864,0.9341112975275198,vtab/caltech101,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0311008029513888,0.1568060980902778,0.0321637232587243,vtab/dsprites_label_x_position,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-L-14-336,openai,zeroshot_retrieval,,0.8889999985694885,0.9810000061988832,ViT-L-14-336 openai
0.5961406197803062,0.8388060288077973,0.5956784313725489,imagenet_sketch,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.4657286152131513,0.9445528002229032,0.4812190866355833,fer2013,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.0562962962962962,0.274156378600823,0.0567608380809354,vtab/smallnorb_label_azimuth,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.1583333333333333,0.8006,0.1676244908434004,vtab/clevr_closest_object_distance,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7326224513709867,1.0,0.2068672210509669,vtab/diabetic_retinopathy,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.8911666666666667,0.9757333333333332,0.8776131879598171,imagenet-r,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.988625,0.999875,0.988625,stl10,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.6865218750574692,0.9403608143148756,0.6921311150610732,sun397,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5509033203125,,0.5508520491113902,vtab/pcam,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1606666666666666,0.9079333333333334,0.1772310296867651,vtab/clevr_closest_object_distance,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5574468085106383,0.8648936170212767,0.5622340425531915,vtab/dtd,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2315333333333333,0.805,0.2332901934437162,vtab/clevr_count_all,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0298394097222222,0.1414794921875,0.0308127750442299,vtab/dsprites_label_x_position,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0623868312757201,0.2708641975308642,0.0631906837062103,vtab/smallnorb_label_azimuth,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.4774310392867094,0.9402340484814712,0.4649079545536754,fer2013,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.5230010414824422,0.7913694511584036,0.5233670588235293,imagenet_sketch,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7341666666666666,0.9035,0.7214778957504825,imagenet-r,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.8353057199211046,0.9546351084812624,0.9005042720791782,vtab/caltech101,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.3109333333333333,0.8006,0.3066391557930237,vtab/clevr_count_all,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.6985172981878089,,0.6986603265589717,renderedsst2,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.1087242798353909,0.540164609053498,0.1086253991970707,vtab/smallnorb_label_elevation,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.9351,0.998,0.936,vtab/cifar10,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,flickr8k,ViT-L-14,openai,zeroshot_retrieval,,0.8633999824523926,0.9409999847412109,ViT-L-14 openai
0.65528,0.894,0.6563199999999999,imagenet1k,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.5930807248764415,,0.5925413265010712,renderedsst2,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.1650710900473933,0.3788625592417061,0.164218009478673,country211,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.168016801680168,0.4119411941194119,0.1658110516934046,fgvc_aircraft,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.0303819444444444,0.1311170789930555,0.0332405586865836,vtab/dsprites_label_orientation,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.3171317131713171,0.7827782778277828,0.3170053475935828,fgvc_aircraft,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6691,0.8925,0.6685000000000001,vtab/cifar100,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.0516872427983539,0.2725925925925926,0.0526600212836742,vtab/smallnorb_label_azimuth,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.6559,0.8854,0.6541,imagenetv2,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5569148936170213,0.8558510638297873,0.5542553191489361,vtab/dtd,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1663981042654028,0.3825118483412322,0.1670142180094786,country211,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7074,0.9179,0.7075,imagenetv2,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.3137333333333333,0.6410666666666667,0.3236449813371542,imagenet-a,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.0454320987654321,0.2672427983539094,0.0450240211431134,vtab/smallnorb_label_azimuth,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.8741,0.9655666666666668,0.8599962924086103,imagenet-r,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.4275427542754275,0.8361836183618362,0.4260962566844919,fgvc_aircraft,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,mscoco_captions,ViT-L-14,laion2b_s32b_b82k,zeroshot_retrieval,,0.7107957005500793,0.8399999737739563,ViT-L-14 laion2b_s32b_b82k
0.5958,0.8547,0.5955999999999999,imagenetv2,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7540228405391985,0.9617025580668296,0.7523924485563404,sun397,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2418241824182418,0.6054605460546054,0.2405525846702317,fgvc_aircraft,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.0590123456790123,0.279835390946502,0.0603763349553147,vtab/smallnorb_label_azimuth,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7613,0.9289,0.7611,vtab/cifar100,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.9277453053102848,0.9983832856609874,0.9288577913034778,cars,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.0257297092013888,0.1208224826388889,0.0259727501505128,vtab/dsprites_label_orientation,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.8254437869822485,0.954963839579224,0.908884143116176,vtab/caltech101,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.980125,0.999875,0.980875,stl10,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.2463246324632463,0.5724572457245725,0.2460249554367201,fgvc_aircraft,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8102964743589743,0.9655448717948718,0.8579252085748035,voc2007,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2044,0.7866666666666666,0.2151124081246672,vtab/clevr_count_all,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.2872511848341232,0.542085308056872,0.2880094786729857,country211,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.06,0.288312757201646,0.0630119225348046,vtab/smallnorb_label_azimuth,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.9664,0.9987,0.9665,vtab/cifar10,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.1895333333333333,0.7248666666666667,0.1870254885776544,vtab/clevr_count_all,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8415619947767691,0.9900509886829996,0.8435641961196442,cars,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.337965783923131,1.0,0.2591163084325771,vtab/diabetic_retinopathy,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.1582,0.8814666666666666,0.1812267071156741,vtab/clevr_closest_object_distance,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0475720164609053,0.2691358024691358,0.0457953163680852,vtab/smallnorb_label_azimuth,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.3651623119556611,0.7007917656373713,0.3512103003164681,gtsrb,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1592258632065097,0.8009676709918627,0.1713416703583154,vtab/dmlab,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4948535233570863,0.7578780680918448,0.4319824074083427,gtsrb,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.3294329432943294,0.7836783678367837,0.3317290552584671,fgvc_aircraft,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.684701252367729,0.9328576420177648,0.6783238400960281,sun397,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.6885672467067816,0.8466417303626605,0.6732279033655394,vtab/flowers,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.8385930309007232,0.9539776462853384,0.9334530557615972,vtab/caltech101,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6696489324530592,0.9273222134358275,0.6609448269493161,sun397,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.3000947867298578,0.556872037914692,0.2994312796208531,country211,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.409445528002229,0.9413485650599052,0.3587300457745208,fer2013,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.96975,0.999875,0.96975,stl10,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
clip_benchmark/benchmark/dataset_type.csv 0 → 100644
View file @ 1d5a34cf
dataset,type
imagenet1k,natural
imagenetv2,natural
imagenet-r,natural
imagenet_sketch,specialized
objectnet,natural
imagenet-a,natural
imagenet-o,natural
vtab/cifar10,natural
vtab/cifar100,natural
mnist,specialized
vtab/flowers,natural
cars,natural
vtab/svhn,natural
fer2013,natural
renderedsst2,specialized
vtab/pets,natural
vtab/caltech101,natural
voc2007_multilabel,natural
voc2007,natural
sun397,natural
fgvc_aircraft,natural
country211,natural
vtab/dtd,natural
gtsrb,natural
stl10,natural
vtab/diabetic_retinopathy,specialized
vtab/eurosat,specialized
vtab/resisc45,specialized
vtab/pcam,specialized
vtab/clevr_count_all,structured
vtab/clevr_closest_object_distance,structured
vtab/dsprites_label_orientation,structured
vtab/dsprites_label_x_position,structured
vtab/dsprites_label_y_position,structured
vtab/smallnorb_label_elevation,structured
vtab/smallnorb_label_azimuth,structured
vtab/dmlab,structured
vtab/kitti_closest_vehicle_distance,structured
mscoco_captions,retrieval
flickr8k,retrieval
flickr30k,retrieval
clip_benchmark/benchmark/datasets.txt 0 → 100644
View file @ 1d5a34cf
mscoco_captions
flickr8k
flickr30k
imagenet1k
imagenetv2
imagenet_sketch
imagenet-a
imagenet-r
objectnet
fer2013
voc2007
voc2007_multilabel
sun397
cars
fgvc_aircraft
mnist
stl10
gtsrb
country211
renderedsst2
vtab/caltech101
vtab/cifar10
vtab/cifar100
vtab/clevr_count_all
vtab/clevr_closest_object_distance
vtab/diabetic_retinopathy
vtab/dmlab
vtab/dsprites_label_orientation
vtab/dsprites_label_x_position
vtab/dtd
vtab/eurosat
vtab/kitti_closest_vehicle_distance
vtab/flowers
vtab/pets
vtab/pcam
vtab/resisc45
vtab/smallnorb_label_azimuth
vtab/smallnorb_label_elevation
vtab/svhn
clip_benchmark/benchmark/datasets_multilingual.txt 0 → 100644
View file @ 1d5a34cf
multilingual_mscoco_captions,es
multilingual_mscoco_captions,it
multilingual_mscoco_captions,ko
multilingual_mscoco_captions,pl
multilingual_mscoco_captions,ru
multilingual_mscoco_captions,tr
multilingual_mscoco_captions,zh
multilingual_mscoco_captions,en
imagenet1k,zh
imagenet1k,it
imagenet1k,jp
imagenet1k,en
imagenet1k,ar
clip_benchmark/benchmark/models.txt 0 → 100644
View file @ 1d5a34cf
ViT-B-32,openai
ViT-B-16,openai
ViT-L-14,openai
ViT-L-14-336,openai
ViT-B-32-quickgelu,laion400m_e32
ViT-B-32,laion2b_e16
ViT-B-32,laion2b_s34b_b79k
ViT-B-16,laion400m_e32
ViT-B-16-plus-240,laion400m_e32
ViT-L-14,laion400m_e32
ViT-L-14,laion2b_s32b_b82k
ViT-H-14,laion2b_s32b_b79k
ViT-g-14,laion2b_s12b_b42k
clip_benchmark/benchmark/results.ipynb 0 → 100644
View file @ 1d5a34cf
This source diff could not be displayed because it is too large. You can view the blob instead.
clip_benchmark/benchmark/webdatasets.txt 0 → 100644
View file @ 1d5a34cf
wds/mscoco_captions
wds/flickr8k
wds/flickr30k
wds/imagenet1k
wds/imagenetv2
wds/imagenet_sketch
wds/imagenet-a
wds/imagenet-r
wds/imagenet-o
wds/objectnet
wds/fer2013
wds/voc2007
wds/voc2007_multilabel
wds/sun397
wds/cars
wds/fgvc_aircraft
wds/mnist
wds/stl10
wds/gtsrb
wds/country211
wds/renderedsst2
wds/vtab/caltech101
wds/vtab/cifar10
wds/vtab/cifar100
wds/vtab/clevr_count_all
wds/vtab/clevr_closest_object_distance
wds/vtab/diabetic_retinopathy
wds/vtab/dmlab
wds/vtab/dsprites_label_orientation
wds/vtab/dsprites_label_x_position
wds/vtab/dsprites_label_y_position
wds/vtab/dtd
wds/vtab/eurosat
wds/vtab/kitti_closest_vehicle_distance
wds/vtab/flowers
wds/vtab/pets
wds/vtab/pcam
wds/vtab/resisc45
wds/vtab/smallnorb_label_azimuth
wds/vtab/smallnorb_label_elevation
wds/vtab/svhn
clip_benchmark/clip_benchmark/__init__.py 0 → 100644
View file @ 1d5a34cf
"""Top-level package for CLIP Benchmark."""
__author__ = """Mehdi Cherti"""
__email__ = 'mehdicherti@gmail.com'
__version__ = '0.1.0'
clip_benchmark/clip_benchmark/cli.py 0 → 100644
View file @ 1d5a34cf
"""Console script for clip_benchmark."""
import argparse
import csv
import json
import os
import sys
from copy import copy
import torch
from clip_benchmark.datasets.builder import (build_dataset, dataset_collection,
get_dataset_collate_fn,
get_dataset_collection_from_file,
get_dataset_default_task)
from clip_benchmark.metrics import (linear_probe, mscoco_generative,
zeroshot_classification,
zeroshot_retrieval)
from clip_benchmark.model_collection import (get_model_collection_from_file,
model_collection)
from clip_benchmark.models import MODEL_TYPES, load_clip
def get_parser_args():
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_eval = subparsers.add_parser('eval', help='Evaluate')
parser_eval.add_argument('--dataset', type=str, default='cifar10', nargs='+',
help="Dataset(s) to use for the benchmark. Can be the name of a dataset, or a collection name ('vtab', 'vtab+', 'imagenet_robustness', 'retrieval') or path of a text file where each line is a dataset name")
parser_eval.add_argument('--dataset_root', default='root', type=str,
help="dataset root folder where the datasets are downloaded. Can be in the form of a template depending on dataset name, e.g., --dataset_root='datasets/{dataset}'. This is useful if you evaluate on multiple datasets.")
parser_eval.add_argument('--split', type=str, default='test', help='Dataset split to use')
parser_eval.add_argument('--model', type=str, default='ViT-B-32-quickgelu',
help='Model architecture to use from OpenCLIP')
parser_eval.add_argument('--pretrained', type=str, default='laion400m_e32',
help='Model checkpoint name to use from OpenCLIP')
parser_eval.add_argument('--pretrained_model', type=str, default='', nargs='+',
help="Pre-trained model(s) to use. Can be the full model name where `model` and `pretrained` are comma separated (e.g., --pretrained_model='ViT-B-32-quickgelu,laion400m_e32'), a model collection name ('openai' or 'openclip_base' or 'openclip_multilingual' or 'openclip_all'), or path of a text file where each line is a model fullname where model and pretrained are comma separated (e.g., ViT-B-32-quickgelu,laion400m_e32). --model and --pretrained are ignored if --pretrained_model is used.")
parser_eval.add_argument('--task', type=str, default='auto',
choices=['zeroshot_classification', 'zeroshot_retrieval', 'zeroshot_retrieval_with_itm',
'linear_probe', 'mscoco_generative', 'auto'],
help='Task to evaluate on. With --task=auto, the task is automatically inferred from the dataset.')
parser_eval.add_argument('--no_amp', action='store_false', dest='amp', default=True,
help='whether to use mixed precision')
parser_eval.add_argument('--num_workers', default=4, type=int)
parser_eval.add_argument('--recall_k', default=[1, 5, 10], type=int,
help='for retrieval, select the k for Recall@K metric. ', nargs='+', )
parser_eval.add_argument('--fewshot_k', default=-1, type=int,
help='for linear probe, how many shots. -1 = whole dataset.')
parser_eval.add_argument('--fewshot_epochs', default=10, type=int, help='for linear probe, how many epochs.')
parser_eval.add_argument('--fewshot_lr', default=0.1, type=float,
help='for linear probe, what is the learning rate.')
parser_eval.add_argument('--skip_load', action='store_true',
help='for linear probes, when everything is cached, no need to load model.')
parser_eval.add_argument('--seed', default=0, type=int, help='random seed.')
parser_eval.add_argument('--batch_size', default=64, type=int)
parser_eval.add_argument('--model_cache_dir', default=None, type=str,
help='directory to where downloaded models are cached')
parser_eval.add_argument('--feature_root', default='features', type=str,
help='feature root folder where the features are stored.')
parser_eval.add_argument('--annotation_file', default='', type=str,
help='text annotation file for retrieval datasets. Only needed for when `--task` is `zeroshot_retrieval`.')
parser_eval.add_argument('--language', default='en', type=str, nargs='+',
help='language(s) of classname and prompts to use for zeroshot classification.')
parser_eval.add_argument('--output', default='result.json', type=str,
help="output file where to dump the metrics. Can be in form of a template, e.g., --output='{dataset}_{pretrained}_{model}_{language}_{task}.json'")
parser_eval.add_argument('--quiet', dest='verbose', action='store_false', help='suppress verbose messages')
parser_eval.add_argument('--cupl', default=False, action='store_true',
help='Use natural language prompt from CuPL paper')
parser_eval.add_argument('--save_clf', default=None, type=str,
help='optionally save the classification layer output by the text tower')
parser_eval.add_argument('--load_clfs', nargs='+', default=[], type=str,
help='optionally load and average mutliple layers output by text towers.')
parser_eval.add_argument('--skip_existing', default=False, action='store_true',
help='whether to skip an evaluation if the output file exists.')
parser_eval.add_argument('--model_type', default='open_clip', type=str, choices=MODEL_TYPES, help='clip model type')
parser_eval.add_argument('--wds_cache_dir', default=None, type=str,
help='optional cache directory for webdataset only')
parser_eval.set_defaults(which='eval')
parser_build = subparsers.add_parser('build', help='Build CSV from evaluations')
parser_build.add_argument('files', type=str, nargs='+', help='path(s) of JSON result files')
parser_build.add_argument('--output', type=str, default='benchmark.csv', help='CSV output file')
parser_build.set_defaults(which='build')
args = parser.parse_args()
return args
def main():
base = get_parser_args()
if base.which == 'eval':
main_eval(base)
elif base.which == 'build':
main_build(base)
def main_build(base):
# Build a benchmark single CSV file from a set of evaluations (JSON files)
rows = []
fieldnames = set()
for path in base.files:
data = json.load(open(path))
row = {}
row.update(data['metrics'])
row.update(data)
del row['metrics']
row['model_fullname'] = row['model'] + ' ' + row['pretrained']
for field in row.keys():
fieldnames.add(field)
rows.append(row)
with open(base.output, 'a') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
for row in rows:
writer.writerow(row)
def main_eval(base):
# Get list of pre-trained models to evaluate
pretrained_model = _as_list(base.pretrained_model)
if pretrained_model:
models = []
for name in pretrained_model:
if os.path.isfile(name):
# if path, read file, each line is a pre-trained model
models.extend(get_model_collection_from_file(name))
elif name in model_collection:
# if part of `model_collection`, retrieve from it
models.extend(model_collection[name])
else:
# if not, assume it is in the form of `model,pretrained`
model, pretrained = name.split(',')
models.append((model, pretrained))
else:
models = [(base.model, base.pretrained)]
# Ge list of datasets to evaluate on
datasets = []
for name in _as_list(base.dataset):
if os.path.isfile(name):
# If path, read file, each line is a dataset name
datasets.extend(get_dataset_collection_from_file(name))
elif name in dataset_collection:
# if part of `dataset_collection`, retrieve from it
datasets.extend(dataset_collection[name])
else:
# if not, assume it is simply the name of the dataset
datasets.append(name)
# Get list of languages to evaluate on
languages = _as_list(base.language)
if base.verbose:
print(f'Models: {models}')
print(f'Datasets: {datasets}')
print(f'Languages: {languages}')
for model, pretrained in models:
for dataset in datasets:
for language in languages:
# We iterative over all possible model/dataset/languages
# TODO: possibility to parallelize evaluation here
args = copy(base)
args.model = model
args.pretrained = pretrained
args.dataset = dataset
args.language = language
run(args)
def _as_list(l):
if not l:
return []
return [l] if type(l) != list else l
def run(args):
"""Console script for clip_benchmark."""
args.device = 'cuda' if torch.cuda.is_available() else 'cpu'
# set seed.
torch.manual_seed(args.seed)
task = args.task
if args.dataset.startswith('wds/'):
dataset_name = args.dataset.replace('wds/', '', 1)
else:
dataset_name = args.dataset
if task == 'auto':
task = get_dataset_default_task(dataset_name)
pretrained_slug = os.path.basename(args.pretrained) if os.path.isfile(args.pretrained) else args.pretrained
pretrained_slug_full_path = args.pretrained.replace('/', '_') if os.path.isfile(
args.pretrained) else args.pretrained
dataset_slug = dataset_name.replace('/', '_')
output = args.output.format(
model=args.model,
pretrained=pretrained_slug,
pretrained_full_path=pretrained_slug_full_path,
task=task,
dataset=dataset_slug,
language=args.language
)
if os.path.exists(output) and args.skip_existing:
if args.verbose:
print(f'Skip {output}, exists already.')
return
if args.verbose:
print(f"Running '{task}' on '{dataset_name}' with the model '{args.pretrained}' on language '{args.language}'")
dataset_root = args.dataset_root.format(dataset=dataset_name, dataset_cleaned=dataset_name.replace('/', '-'))
if args.skip_load:
model, transform, collate_fn, dataloader = None, None, None, None
else:
model, transform, tokenizer = load_clip(
model_type=args.model_type,
model_name=args.model,
pretrained=args.pretrained,
cache_dir=args.model_cache_dir,
device=args.device
)
print(transform)
model.eval()
dataset = build_dataset(
dataset_name=args.dataset,
root=dataset_root,
transform=transform,
split=args.split,
annotation_file=args.annotation_file,
download=True,
language=args.language,
task=task,
cupl=args.cupl,
wds_cache_dir=args.wds_cache_dir,
)
collate_fn = get_dataset_collate_fn(args.dataset)
if args.verbose:
try:
print(f'Dataset size: {len(dataset)}')
except TypeError:
print('IterableDataset has no len()')
print(f'Dataset split: {args.split}')
try:
print(f'Dataset classes: {dataset.classes}')
print(f'Dataset number of classes: {len(dataset.classes)}')
except:
print('Dataset has no classes.')
if args.dataset.startswith('wds/'):
dataloader = torch.utils.data.DataLoader(
dataset.batched(args.batch_size), batch_size=None,
shuffle=False, num_workers=args.num_workers,
)
else:
dataloader = torch.utils.data.DataLoader(
dataset, batch_size=args.batch_size,
shuffle=False, num_workers=args.num_workers,
collate_fn=collate_fn
)
if task == 'zeroshot_classification':
zeroshot_templates = dataset.templates if hasattr(dataset, 'templates') else None
if args.cupl:
assert (zeroshot_templates is not None), 'Dataset does not support CuPL prompts'
# if args.verbose:
# print(f"Zero-shot templates: {zeroshot_templates}")
classnames = dataset.classes if hasattr(dataset, 'classes') else None
assert (zeroshot_templates is not None and classnames is not None), 'Dataset does not support classification'
metrics = zeroshot_classification.evaluate(
model,
dataloader,
tokenizer,
classnames, zeroshot_templates,
device=args.device,
amp=args.amp,
verbose=args.verbose,
cupl=args.cupl,
save_clf=args.save_clf,
load_clfs=args.load_clfs,
)
elif task == 'zeroshot_retrieval':
metrics = zeroshot_retrieval.evaluate(
model,
dataloader,
tokenizer,
recall_k_list=args.recall_k,
device=args.device,
amp=args.amp
)
elif task == 'zeroshot_retrieval_with_itm':
metrics = zeroshot_retrieval_with_itm.evaluate(
model,
dataloader,
tokenizer,
recall_k_list=args.recall_k,
device=args.device,
amp=args.amp
)
elif task == 'linear_probe':
# we also need the train split for linear probing.
train_dataset = build_dataset(
dataset_name=args.dataset,
root=dataset_root,
transform=transform,
split='train',
annotation_file=args.annotation_file,
download=True,
)
train_dataloader = torch.utils.data.DataLoader(
train_dataset, batch_size=args.batch_size,
shuffle=False, num_workers=args.num_workers,
collate_fn=collate_fn, pin_memory=True,
)
metrics = linear_probe.evaluate(
model,
train_dataloader,
dataloader,
args.fewshot_k,
args.batch_size,
args.num_workers,
args.fewshot_lr,
args.fewshot_epochs,
(args.model + '-' + args.pretrained + '-' + args.dataset).replace('/', '_'),
args.seed,
args.feature_root,
device=args.device,
amp=args.amp,
verbose=args.verbose,
)
elif task == 'mscoco_generative':
metrics = mscoco_generative.evaluate(
model=model,
dataloader=dataloader,
batch_size=args.batch_size,
num_workers=args.num_workers,
device=args.device,
amp=args.amp,
verbose=args.verbose,
transform=transform
)
else:
raise ValueError(
'Unsupported task: {}. task should `zeroshot_classification` or `zeroshot_retrieval`'.format(task))
dump = {
'dataset': args.dataset,
'model': args.model,
'pretrained': args.pretrained,
'task': task,
'metrics': metrics,
'language': args.language,
}
if args.verbose:
print(f'Dump results to: {output}')
with open(output, 'a') as f:
f.write(json.dumps(dump) + '\n')
# json.dump(dump, f)
return 0
if __name__ == '__main__':
sys.exit(main()) # pragma: no cover
clip_benchmark/clip_benchmark/datasets/ar_classnames.json 0 → 100644
View file @ 1d5a34cf
{
"imagenet1k": [
"\u0633\u0645\u0643 \u0627\u0644\u062a\u0646\u0634",
"\u0627\u0644\u0633\u0645\u0643\u0629 \u0627\u0644\u0630\u0647\u0628\u064a\u0629",
"\u0627\u0644\u0642\u0631\u0634 \u0627\u0644\u0623\u0628\u064a\u0636 \u0627\u0644\u0643\u0628\u064a\u0631",
"\u0627\u0644\u0642\u0631\u0634 \u0627\u0644\u0628\u0628\u0631\u064a",
"\u0627\u0644\u0642\u0631\u0634 \u0627\u0644\u0645\u0637\u0631\u0642\u0629",
"\u0633\u0645\u0643 \u0627\u0644\u0631\u0639\u0627\u062f",
"\u0633\u0645\u0643 \u0627\u0644\u0631\u0642\u064a\u0637\u0629",
"\u062f\u064a\u0643",
"\u062f\u062c\u0627\u062c\u0629",
"\u0646\u0639\u0627\u0645\u0629",
"\u0627\u0644\u0634\u0631\u0634\u0648\u0631 \u0627\u0644\u062c\u0628\u0644\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u0633\u0648\u0646",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062a\u0641\u0627\u062d\u064a \u0627\u0644\u0627\u0648\u0631\u0648\u0628\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062c\u0646\u0643 \u062f\u0627\u0643\u0646 \u0627\u0644\u0639\u064a\u0648\u0646",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062f\u0631\u0633\u0629 \u0627\u0644\u0633\u0645\u0627\u0648\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0628\u0648 \u0627\u0644\u062d\u0646\u0627\u0621",
"\u0628\u0644\u0628\u0644",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0642\u064a\u0642",
"\u0639\u0642\u0639\u0642 \u0637\u0627\u0626\u0631 \u0627\u0644\u0630\u064a\u0644 \u0627\u0644\u0637\u0648\u064a\u0644",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0642\u0631\u0642\u0641",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u063a\u0637\u0627\u0633",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u062f\u0623\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0639\u0642\u0627\u0628 \u0627\u0644\u0631\u062e\u0645\u0629",
"\u0646\u0633\u0631",
"\u0627\u0644\u0628\u0648\u0645\u0629 \u0627\u0644\u0631\u0645\u0627\u062f\u064a\u0629",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0646\u0627\u0631\u064a",
"\u0644\u064a\u0633\u0648\u062a\u0631\u064a\u062a\u0648\u0646 \u0641\u0648\u0644\u062c\u0627\u0631\u064a\u0633",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0645\u0627\u0626\u064a",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0645\u0631\u0642\u0637",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0645\u0643\u0633\u064a\u0643\u064a",
"\u0636\u0641\u062f\u0639 \u0627\u0644\u062b\u0648\u0631 \u0627\u0644\u0627\u0645\u0631\u064a\u0643\u064a",
"\u0636\u0641\u062f\u0639 \u0627\u0644\u0634\u062c\u0631",
"\u0627\u0644\u0636\u0641\u0627\u062f\u0639 \u0630\u0627\u062a \u0627\u0644\u0630\u064a\u0644",
"\u0627\u0644\u0633\u0644\u062d\u0641\u0627\u0629 \u0627\u0644\u0628\u062d\u0631\u064a\u0629 \u0636\u062e\u0645\u0629 \u0627\u0644\u0631\u0623\u0633",
"\u0633\u0644\u062d\u0641\u0627\u0629 \u0627\u0644\u0645\u062d\u064a\u0637 \u062c\u0644\u062f\u064a\u0629 \u0627\u0644\u0638\u0647\u0631",
"\u0633\u0644\u062d\u0641\u0627\u0629 \u0627\u0644\u0637\u064a\u0646",
"\u0627\u0644\u0633\u0644\u062d\u0641\u0627\u0629 \u0630\u0627\u062a \u0638\u0647\u0631 \u0627\u0644\u0645\u0639\u064a\u0646",
"\u0633\u0644\u062d\u0641\u0627\u0629 \u0635\u0646\u062f\u0648\u0642\u064a\u0629",
"\u0627\u0644\u0648\u0632\u063a",
"\u0627\u0644\u0625\u063a\u0648\u0627\u0646\u0629",
"\u0627\u0644\u062d\u0631\u0628\u0627\u0621 \u0627\u0644\u062e\u0636\u0631\u0627\u0621",
"\u0627\u0644\u0633\u062d\u0627\u0644\u064a \u0627\u0644\u0635\u062d\u0631\u0627\u0648\u064a\u0629",
"\u0627\u0644\u0639\u064e\u0636\u0652\u0631\u064e\u0641\u064f\u0648\u0637",
" \u0633\u062d\u0644\u064a\u0629 \u0647\u062f\u0628 \u0627\u0644\u0639\u0646\u0642",
"\u0633\u062d\u0644\u064a\u0629 \u0627\u0644\u062a\u0645\u0633\u0627\u062d",
"\u0648\u062d\u0634 \u062c\u064a\u0644\u0627",
"\u0633\u062d\u0644\u064a\u0629 \u062e\u0636\u0631\u0627\u0621",
"\u062d\u0631\u0628\u0627\u0621 \u0627\u0641\u0631\u064a\u0642\u064a\u0629",
"\u062a\u0646\u064a\u0646 \u0643\u0648\u0645\u0648\u062f\u0648",
"\u0627\u0644\u062a\u0645\u0633\u0627\u062d \u0627\u0644\u0623\u0641\u0631\u064a\u0642\u064a",
"\u062a\u0645\u0633\u0627\u062d \u0627\u0644\u0642\u0627\u0637\u0648\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0627\u0644\u062f\u064a\u0646\u0627\u0635\u0648\u0631 \u062b\u064f\u0644\u0627\u062b\u064a\u064f\u0651 \u0627\u0644\u0642\u064f\u0631\u0648\u0646",
"\u0627\u0644\u062b\u0639\u0627\u0628\u064a\u0646 \u0627\u0644\u062f\u0648\u062f\u064a\u0629",
"\u0623\u0641\u0639\u0649 \u0627\u0644\u0637\u0648\u0642",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u0646\u0641\u0627\u062b\u0629 ",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u062e\u0636\u0631\u0627\u0621",
"\u0627\u0644\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0645\u0644\u0643",
"\u0623\u0641\u0639\u0649 \u0627\u0644\u0631\u0628\u0627\u0637",
"\u0627\u0641\u0639\u0649 \u0627\u0644\u0645\u0627\u0621",
"\u062b\u0639\u0628\u0627\u0646 \u0646\u0628\u0627\u062a \u0643\u0631\u0645\u0629",
"\u0627\u0644\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0644\u064a\u0644\u064a ",
"\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0623\u0635\u0644\u0629 \u0627\u0644\u0639\u0627\u0635\u0631\u0629",
"\u0627\u0644\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0635\u062e\u0631\u064a",
"\u0643\u0648\u0628\u0631\u0627 \u0647\u0646\u062f\u064a\u0629",
"\u0645\u0627\u0645\u0628\u0627 \u062e\u0636\u0631\u0627\u0621",
"\u0627\u0644\u063a\u064a\u062f\u0642\u0627\u0648\u0627\u062a",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u0645\u0642\u0631\u0646\u0629",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u062c\u0631\u0633\u064a\u0629 \u0630\u0627\u062a \u0627\u0644\u0635\u062f\u0631 \u0627\u0644\u0645\u0627\u0633\u064a",
"\u0627\u0641\u0639\u0649 \u0644\u0627\u0641\u0651\u0629 \u0627\u0644\u062c\u0646\u0628",
"\u0645\u0641\u0635\u0644\u064a\u0627\u062a \u062b\u0644\u0627\u062b\u064a\u0629 \u0627\u0644\u0641\u0635\u0648\u0635",
"\u062d\u0634\u0631\u0629 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a \u0630\u0627\u062a \u0627\u0644\u0642\u0648\u0627\u0626\u0645 \u0627\u0644\u0637\u0648\u064a\u0644\u0629",
"\u0639\u0642\u0631\u0628",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u062d\u062f\u064a\u0642\u0629 \u0630\u0627 \u0627\u0644\u0644\u0648\u0646 \u0627\u0644\u0623\u0633\u0648\u062f \u0648\u0627\u0644\u0623\u0635\u0641\u0631 ",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u062d\u0638\u064a\u0631\u0629",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u062d\u062f\u064a\u0642\u0629 \u0627\u0644\u0623\u0648\u0631\u0628\u064a",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u0623\u0631\u0645\u0644\u0629 \u0627\u0644\u0633\u0648\u062f\u0627\u0621",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0631\u062a\u064a\u0644\u0627\u0621 \u0630\u0627\u062a \u0623\u0631\u062c\u0644 \u062d\u0645\u0631\u0627\u0621",
"\u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u0630\u0626\u0628\u064a",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u0642\u064f\u0631\u064e\u0627\u062f",
"\u062d\u0634\u0631\u0629 \u0623\u0645 \u0623\u0631\u0628\u0639\u0629 \u0648\u0623\u0631\u0628\u0639\u064a\u0646",
"\u062f\u062c\u0627\u062c\u0629 \u0627\u0644\u0637\u0647\u064a\u0648\u062c \u0627\u0644\u0623\u0633\u0648\u062f",
"\u062f\u062c\u0627\u062c\u0629 \u062a\u0631\u0645\u062c\u0627\u0646",
"\u062f\u062c\u0627\u062c\u0629 \u0637\u064a\u0647\u0648\u062c \u0645\u0637\u0648\u0642",
"\u062f\u062c\u0627\u062c\u0629 \u0627\u0644\u0637\u0647\u0628\u0648\u062c",
"\u0627\u0644\u0637\u0627\u0648\u0648\u0633",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0633\u0645\u0627\u0646",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u062c\u0644",
"\u0627\u0644\u0628\u0628\u063a\u0627\u0621 \u0627\u0644\u0631\u0645\u0627\u062f\u064a",
"\u0628\u0628\u063a\u0627\u0621 \u0645\u0643\u0627\u0648",
"\u0628\u0628\u063a\u0627\u0621 \u0643\u0648\u0643\u0627\u062a\u0648 \u0643\u0628\u0631\u064a\u062a\u064a \u0627\u0644\u0639\u0631\u0641",
"\u0628\u0628\u063a\u0627\u0621 \u0642\u0648\u0633 \u0642\u0632\u062d",
"\u0637\u064a\u0631 \u0627\u0644\u0648\u0642\u0648\u0627\u0642 \u0643\u0648\u0643\u0627\u0644",
"\u0637\u064a\u0631 \u0648\u0631\u0648\u0627\u0631 ",
"\u0637\u064a\u0631 \u0623\u0628\u0648 \u0642\u0631\u0646",
"\u0627\u0644\u0637\u0627\u0626\u0631 \u0627\u0644\u0637\u0646\u0627\u0646",
"\u0637\u064a\u0648\u0631 \u0627\u0644\u064a\u0642\u0645\u064e\u0631",
"\u0637\u0627\u0626\u0631 \u0645\u0637\u0648\u0642",
"\u0630\u0643\u0631 \u0627\u0644\u0628\u0637",
"\u0627\u0644\u0628\u0637\u0629 \u0627\u0644\u063a\u0648\u0627\u0635\u0629 \u062d\u0645\u0631\u0627\u0621 \u0627\u0644\u0635\u062f\u0631",
"\u0625\u0648\u0632\u0629",
"\u0627\u0644\u0625\u0648\u0632\u0629 \u0633\u0648\u062f\u0627\u0621",
"\u0627\u0644\u0641\u064a\u0644",
" \u0622\u0643\u0644 \u0627\u0644\u0646\u0645\u0644 \u0627\u0644\u0634\u0648\u0643\u064a",
"\u062e\u0644\u062f \u0627\u0644\u0645\u0627\u0621",
"\u0648\u0644\u0628",
"\u0643\u0648\u0627\u0644\u0627",
"\u0627\u0644\u0648\u0645\u0628\u062a\u064a\u0627\u062a",
"\u0642\u0646\u062f\u064a\u0644 \u0628\u062d\u0631",
"\u0634\u0642\u0627\u0626\u0642 \u0646\u0639\u0645\u0627\u0646 \u0627\u0644\u0628\u062d\u0631",
"\u0645\u0631\u062c\u0627\u0646 \u0627\u0644\u062f\u0645\u0627\u063a",
"\u062f\u064a\u062f\u0627\u0646 \u0645\u0633\u0637\u062d\u0629",
"\u062f\u064a\u062f\u0627\u0646 \u0623\u0633\u0637\u0648\u0627\u0646\u064a\u0629",
"\u0645\u062d\u0627\u0631\u0629",
"\u062d\u0644\u0632\u0648\u0646",
"\u0628\u0632\u0627\u0642",
"\u062d\u0644\u0632\u0648\u0646 \u0645\u0627\u0626\u064a",
"\u0631\u062e\u0648\u064a\u0627\u062a \u0628\u062d\u0631\u064a\u0647 \u062a\u0633\u0645\u0649 \u0627\u0644\u062e\u064a\u062a\u0648\u0646",
"\u0646\u0648\u062a\u064a\u0644\u0648\u0633 \u0627\u0644\u062d\u062c\u0631\u064a",
"\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0628\u062d\u0631",
"\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0628\u062d\u0631 \u0627\u0644\u0623\u0637\u0644\u0633\u064a",
"\u0627\u0644\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0639\u0627\u0632\u0641",
"\u0645\u0644\u0643 \u0627\u0644\u0633\u0644\u0637\u0639\u0648\u0646",
"\u062c\u0631\u0627\u062f \u0627\u0644\u0628\u062d\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0643\u0631\u0643\u0646\u062f \u0634\u0627\u0626\u0643",
"\u062c\u0631\u0627\u062f \u0627\u0644\u0645\u064a\u0627\u0647 \u0627\u0644\u0639\u0630\u0628\u0629",
"\u0627\u0644\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0646\u0627\u0633\u0643",
"\u0645\u062a\u0633\u0627\u0648\u064a\u0627\u062a \u0627\u0644\u0623\u0642\u062f\u0627\u0645",
"\u0644\u0642\u0644\u0642 \u0623\u0628\u064a\u0636",
"\u0644\u0642\u0644\u0642 \u0623\u0633\u0648\u062f",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0628\u062c\u0639 \u0627\u0644\u0645\u0633\u0645\u0649 \u0623\u0628\u0648 \u0645\u0644\u0639\u0642\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0641\u0644\u0627\u0645\u064a\u0646\u063a\u0648",
"\u0637\u0627\u0626\u0631 \u0628\u0644\u0634\u0648\u0646 \u0627\u0644\u0623\u0632\u0631\u0642 \u0627\u0644\u0635\u063a\u064a\u0631",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0628\u0644\u0634\u0648\u0646 \u0627\u0644\u0623\u0628\u064a\u0636 \u0627\u0644\u0643\u0628\u064a\u0631",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0639\u062c\u0627\u062c",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0643\u0631\u0643\u064a\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0631\u0637\u0627\u0633",
"\u062f\u062c\u0627\u062c\u0629 \u0627\u0644\u0645\u0627\u0621 \u0627\u0644\u0623\u0631\u062c\u0648\u0627\u0646\u064a\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u063a\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u0628\u0627\u0631",
"\u0637\u0627\u0626\u0631 \u0642\u0646\u0628\u0631\u0629 \u0627\u0644\u0645\u0627\u0621 \u0627\u0644\u0645\u062a\u0648\u0631\u062f",
"\u0637\u0627\u0626\u0631 \u062f\u0631\u064a\u062c\u0629 \u0623\u0644\u0628\u064a\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0637\u064a\u0637\u0648\u064a \u0623\u062d\u0645\u0631 \u0627\u0644\u0633\u0627\u0642",
"\u0639\u064e\u062f\u0651\u0627\u0621 \u0627\u0644\u0645\u0633\u062a\u0646\u0642\u0639\u0627\u062a \u0623\u0648 \u0627\u0644\u062f\u0651\u062a\u0634\u0631",
"\u0635\u0627\u0626\u062f \u0627\u0644\u0645\u062d\u0627\u0631",
"\u0628\u062c\u0639\u0629",
"\u0627\u0644\u0628\u0637\u0631\u064a\u0642 \u0627\u0644\u0645\u0644\u0643",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0642\u0637\u0631\u0633",
"\u0627\u0644\u062d\u0648\u062a \u0627\u0644\u0631\u0645\u0627\u062f\u064a \u0627\u0644\u0635\u0644\u0628",
"\u0627\u0644\u062d\u064f\u0648\u062a\u064f \u0627\u0644\u0642\u0627\u062a\u0650\u0644\u064f",
"\u0628\u0642\u0631\u0629 \u0627\u0644\u0628\u062d\u0631",
"\u0623\u0633\u062f \u0627\u0644\u0628\u062d\u0631",
"\u0643\u0644\u0628 \u0634\u064a\u0648\u0627\u0648\u0627",
"\u0643\u0644\u0628 \u0627\u0644\u0630\u0642\u0646 \u0627\u0644\u064a\u0627\u0628\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0645\u0627\u0644\u0637\u064a",
"\u0643\u0644\u0628 \u0628\u0643\u064a\u0646\u064a",
"\u0643\u0644\u0628 \u062a\u0634\u064a\u0647 \u062a\u0632\u0648 \u0627\u0644\u0635\u064a\u0646\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0644\u0643 \u062a\u0634\u0627\u0631\u0644\u0632",
"\u0643\u0644\u0628 \u0628\u0627\u0628\u064a\u0644\u0648\u0646",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0631\u064a\u0631 \u0627\u0644\u0639\u0631\u0636",
"\u0643\u0644\u0627\u0628 \u0631\u064a\u062f\u062c \u0628\u0627\u0643",
"\u0643\u0644\u0628 \u0627\u0644\u0635\u064a\u062f \u0627\u0644\u0623\u0641\u063a\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0627\u0633\u0637",
"\u0643\u0644\u0628 \u0628\u064a\u063a\u0644",
"\u0643\u0644\u0628 \u0627\u0644\u062f\u0645\u0648\u0645",
"\u0643\u0644\u0628 \u0628\u0644\u0648\u064a\u062a\u064a\u0643 \u0643\u0648\u0646\u0647\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0643\u0648\u0646\u0647\u0648\u0646\u062f \u0627\u0644\u0628\u0646\u064a \u0648\u0627\u0644\u0623\u0633\u0648\u062f",
"\u0643\u0644\u0628 \u0627\u0644\u0648\u0648\u0643\u0631",
"\u0643\u0644\u0628 \u0635\u064a\u062f \u0627\u0644\u062b\u0639\u0627\u0644\u0628 \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a",
"\u0643\u0644\u0628 \u0631\u064a\u062f\u0628\u0648\u0646 \u0643\u0648\u0646\u0647\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0628\u0648\u0631\u0632\u0648\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0627\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0633\u0644\u0648\u0642\u064a \u0627\u0644\u0627\u064a\u0637\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0648\u064a\u0628\u062a",
"\u0643\u0644\u0628 \u0627\u064a\u0628\u064a\u0632\u0627\u0646 \u0647\u0648\u0646\u062f",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0646\u0631\u0648\u064a\u062c\u064a",
"\u0643\u0644\u0628 \u0623\u0648\u062a\u064a\u0631 \u0647\u0627\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0633\u0644\u0648\u0642\u064a",
"\u0643\u0644\u0628 \u062f\u064a\u0631 \u0647\u0627\u0648\u0646\u062f \u0627\u0644\u0627\u0633\u0643\u062a\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0648\u0627\u064a\u0645\u0631\u064a",
"\u0643\u0644\u0628 \u0633\u062a\u0627\u0641\u0648\u0631\u062f\u0634\u0627\u064a\u0631 \u0628\u0648\u0644 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0633\u062a\u0627\u0641\u0648\u0631\u062f\u0634\u0627\u064a\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0643\u0644\u0628 \u0628\u064a\u062f\u0644\u064a\u0646\u062c\u062a\u0648\u0646 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u0631\u062f\u0631 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0643\u064a\u0631\u064a \u0628\u0644\u0648 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0631\u064a\u0631 \u0627\u0644\u0625\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0646\u0648\u0631\u0641\u0648\u0644\u0643 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0646\u0648\u0631\u064a\u062a\u0634 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u064a\u0648\u0631\u0643 \u0634\u0627\u064a\u0631",
"\u0643\u0644\u0628 \u0648\u064a\u0631 \u0641\u0648\u0643\u0633 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0644\u064a\u0643\u0644\u0627\u0646\u062f \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0633\u064a\u0627\u0644\u064a\u0647\u0627\u0645 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0627\u0628 \u0627\u0644\u0623\u0631\u062f\u064a\u0644",
"\u0643\u0644\u0628 \u0643\u064a\u0631\u0646 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u062a\u0631\u064a\u0631 \u0627\u0633\u062a\u0631\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u062f\u0627\u0646\u062f\u064a \u062f\u064a\u0646\u0645\u0648\u0646\u062a",
"\u0643\u0644\u0628 \u0628\u0648\u0633\u0637\u0646 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0634\u0646\u0627\u0648\u062a\u0633\u0631 \u0645\u0646\u0645\u0646\u0645",
"\u0643\u0644\u0628 \u0634\u0646\u0627\u0648\u062a\u0633\u0631 \u0627\u0644\u0639\u0645\u0644\u0627\u0642",
"\u0643\u0644\u0628 \u0634\u0646\u0627\u0648\u062a\u0633\u0631 \u0627\u0644\u0639\u0627\u062f\u064a",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0627\u0633\u0643\u062a\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u062a\u0631\u064a\u0631 \u0627\u0644\u062a\u0628\u062a",
"\u0643\u0644\u0628 \u0633\u064a\u0644\u0643\u064a \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u064a\u0631\u064a\u0631 \u0627\u0644\u0642\u0645\u062d\u064a \u0627\u0644\u0646\u0627\u0639\u0645",
"\u0643\u0644\u0628 \u0648\u064a\u0633\u062a\u064a",
"\u0643\u0644\u0628 \u0644\u0627\u0633\u0627 \u0623\u0628\u0633\u0648",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0633\u062a\u0631\u062f \u0627\u0644\u0630\u0647\u0628\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0633\u062a\u0631\u062f \u0645\u062c\u0639\u062f \u0627\u0644\u0634\u0639\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0633\u062a\u0631\u062f \u0627\u0644\u0630\u0647\u0628\u064a",
"\u0643\u0644\u0628 \u0644\u0627\u0628\u0631\u0627\u062f\u0648\u0631 \u0631\u064a\u062a\u0631\u064a\u0641\u0631",
"\u0643\u0644\u0628 \u0634\u064a\u0633\u0628\u064a\u0643\u0627",
"\u0643\u0644\u0628 \u0628\u0648\u064a\u0646\u062a\u0631 \u0627\u0644\u0623\u0644\u0645\u0627\u0646\u064a \u0642\u0635\u064a\u0631 \u0627\u0644\u0634\u0639\u0631",
"\u0643\u0644\u0628 \u0641\u064a\u0632\u0644\u0627",
"\u0643\u0644\u0628 \u0633\u064a\u062a\u0631 \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a",
"\u0643\u0644\u0628 \u0633\u064a\u062a\u0631 \u0627\u0644\u0625\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0633\u064a\u062a\u0631 \u0627\u0644\u062c\u0648\u0631\u062f\u0648\u0646\u064a",
"\u0643\u0644\u0628 \u0628\u0631\u064a\u062a\u0627\u0646\u064a \u0633\u0628\u0627\u0646\u064a\u0644",
"\u0643\u0644\u0628 \u0642\u0644\u0645\u0628\u0631",
"\u0643\u0644\u0628 \u0633\u0628\u0631\u064a\u0646\u063a\u0631 \u0633\u0628\u0627\u0646\u064a\u0644 \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0633\u0628\u0631\u064a\u0646\u063a\u0631 \u0627\u0644\u0648\u064a\u0644\u0632\u064a",
"\u0643\u0644\u0628 \u062f\u0644\u0644 \u0627\u0644\u0630\u0644\u064a\u0644",
"\u0643\u0644\u0628 \u0627\u0644\u0633\u0627\u0643\u0633 \u0627\u0644\u0625\u0633\u0628\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0633\u0628\u0627\u064a\u0646\u0644 \u0627\u0644\u0645\u0627\u0621 \u0627\u0644\u0625\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0643\u0648\u0641\u0627\u0632",
"\u0643\u0644\u0628 \u0634\u064a\u0628\u0631\u0643",
"\u0643\u0644\u0628 \u062c\u0631\u0648\u0646\u064a\u0646\u062f\u064a\u0644",
"\u0643\u0644\u0628 \u0645\u0627\u0644\u064a\u0646\u0648",
"\u0643\u0644\u0628 \u0628\u0631\u064a\u0627\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u0644\u064a\u0628\u064a \u0627\u0644\u0625\u0633\u062a\u0631\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u0643\u0648\u0645\u0648\u0646\u062f\u0648\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0631\u0627\u0639\u064a \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a \u0627\u0644\u0642\u062f\u064a\u0645",
"\u0643\u0644\u0628 \u0627\u0644\u0631\u0627\u0639\u064a \u0627\u0644\u0634\u062a\u0644\u0646\u062f\u0649",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u0648\u0644\u064a",
"\u0643\u0644\u0628 \u0628\u0648\u0631\u062f\u0631 \u0643\u0648\u0644\u064a",
"\u0643\u0644\u0628 \u0628\u0648\u0641\u064a \u062f\u064a \u0641\u0644\u0627\u0646\u062f\u0631",
"\u0643\u0644\u0628 \u0631\u0648\u062a \u0648\u0627\u064a\u0644\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0631\u0627\u0639\u064a \u0627\u0644\u0623\u0644\u0645\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u062f\u0648\u0628\u064a\u0631\u0645\u0627\u0646",
"\u0643\u0644\u0628 \u0628\u064a\u0646\u0634\u0631 \u0627\u0644\u0645\u0635\u063a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u062c\u0628\u0644 \u0627\u0644\u0633\u0648\u064a\u0633\u0631\u064a",
"\u0643\u0644\u0628 \u062c\u0628\u0644 \u0627\u0644\u0628\u0631\u0646\u064a\u0632",
"\u0643\u0644\u0628 \u0627\u067e\u064a\u0646\u0632\u064a\u0644\u064a\u0631 \u0633\u064a\u0646\u064a\u0646\u0647\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0627\u0646\u062a\u0644\u0628\u062a\u0634\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u0643\u0633\u0631",
"\u0643\u0644\u0628 \u0628\u0648\u0644 \u0645\u0627\u0633\u062a\u064a\u0641",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0627\u0633\u062a\u064a\u0641 \u0627\u0644\u062a\u064a\u0628\u062a\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u0644\u062f\u0648\u063a \u0627\u0644\u0641\u0631\u0646\u0633\u064a",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u062f\u0627\u0646\u0645\u0627\u0631\u0643\u064a \u0627\u0644\u0636\u062e\u0645",
"\u0643\u0644\u0628 \u0633\u0627\u0646\u062a \u0628\u0631\u0646\u0627\u0631\u062f",
"\u0643\u0644\u0628 \u0627\u0644\u0647\u0627\u0633\u0643\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0644\u0645\u0648\u062a \u0627\u0644\u0623\u0644\u0627\u0633\u0643\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0647\u0633\u0643\u064a \u0627\u0644\u0633\u064a\u0628\u064a\u0631\u064a",
"\u0643\u0644\u0628 \u062f\u0644\u0645\u0627\u0633\u064a \u0627\u0644\u0645\u0631\u0642\u0637",
"\u0643\u0644\u0628 \u0623\u0641\u064a\u0646\u0628\u064a\u0646\u0634\u0631",
"\u0643\u0644\u0628 \u0628\u0627\u0633\u0646\u062c\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u062c",
"\u0643\u0644\u0628 \u0644\u064a\u0648\u0646 \u0628\u064a\u0631\u062c\u0631",
"\u0643\u0644\u0628 \u0646\u064a\u0648\u0641\u0627\u0648\u0646\u062f\u0644\u0627\u0646\u062f",
"\u0643\u0644\u0640\u0628 \u062c\u0628\u0627\u0644 \u0627\u0644\u0628\u0631\u0627\u0646\u0633",
"\u0643\u0644\u0628 \u0633\u0627\u0645\u0648\u062f\u064a",
"\u0643\u0644\u0628 \u0628\u0648\u0645\u064a\u0631\u0627\u0646\u064a\u0627\u0646",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0634\u0627\u0648 \u062a\u0634\u0627\u0648",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u064a\u0634\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0627\u0644\u062c\u0631\u064a\u0641\u0648\u0646",
"\u0643\u0644\u0628\u0628 \u0628\u064a\u0645\u0628\u0631\u0648\u0643 \u0648\u064a\u0644\u0634 \u0643\u0648\u0631\u062c\u0649",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u0627\u0631\u062f\u064a\u062c\u0627\u0646",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0648\u064a \u0627\u0644\u0628\u0648\u062f\u0644",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u062f\u0644 \u0627\u0644\u0635\u063a\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u062f\u0644 \u0627\u0644\u0642\u064a\u0627\u0633\u064a",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0645\u0643\u0633\u064a\u0643\u064a \u0628\u0644\u0627 \u0634\u0639\u0631",
"\u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0631\u0645\u0627\u062f\u064a",
"\u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0623\u0628\u064a\u0636",
"\u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0627\u0644\u0642\u064a\u0648\u0637",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0625\u0633\u062a\u0631\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u062f\u0648\u0644",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0628\u0631\u064a \u0627\u0644\u0625\u0641\u0631\u064a\u0642\u064a",
"\u0627\u0644\u0636\u0628\u0639",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0642\u0632\u0645",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0642\u0637\u0628\u064a",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0631\u0645\u0627\u062f\u064a",
"\u0627\u0644\u0642\u0637\u0637 \u0627\u0644\u0646\u0645\u0631\u064a\u0629",
"\u0627\u0644\u0633\u0646\u0648\u0631 \u0627\u0644\u0645\u064f\u0631\u0642\u0637 \u0627\u0644\u0635\u063a\u064a\u0631",
"\u0642\u0637 \u0634\u064a\u0631\u0627\u0632\u064a",
"\u0627\u0644\u0642\u0637 \u0627\u0644\u0633\u064a\u0627\u0645\u064a",
"\u0642\u0637 \u0644\u064a\u0628\u064a",
"\u0623\u0633\u062f \u0627\u0644\u062c\u0628\u0627\u0644",
"\u0627\u0644\u0648\u064e\u0634\u064e\u0642",
"\u0646\u0645\u0631 ",
"\u0646\u0645\u0631 \u0627\u0644\u062b\u0644\u0648\u062c",
"\u064a\u063a\u0648\u0631",
"\u0623\u0633\u062f",
"\u0627\u0644\u0628\u0628\u0631",
"\u0627\u0644\u0641\u0647\u062f",
"\u062f\u0628 \u0628\u0646\u064a",
"\u062f\u0628 \u0623\u0633\u0648\u062f \u0623\u0645\u0631\u064a\u0643\u064a",
"\u062f\u0628 \u0642\u0637\u0628\u064a",
"\u0627\u0644\u062f\u0628 \u0627\u0644\u0643\u0633\u0644\u0627\u0646 ",
"\u0627\u0644\u0633\u0645\u0648\u0631\u064a\u0627\u062a",
"\u0633\u0631\u0642\u0627\u0637",
"\u062e\u0646\u0627\u0641\u0633 \u0646\u0645\u0631\u064a\u0629",
"\u062f\u0639\u0633\u0648\u0642\u0629",
"\u062e\u0646\u0641\u0633\u0627\u0621 \u0623\u0631\u0636\u064a\u0629",
"\u062e\u0646\u0627\u0641\u0633 \u0637\u0648\u064a\u0644\u0629 \u0627\u0644\u0642\u0631\u0648\u0646 ",
"\u062e\u0646\u0641\u0633\u0629 \u0627\u0644\u0623\u0648\u0631\u0627\u0642",
"\u062e\u0646\u0627\u0641\u0633 \u0627\u0644\u0631\u0648\u062b",
"\u062e\u0646\u0641\u0633\u0627\u0621 \u0648\u062d\u064a\u062f \u0627\u0644\u0642\u0631\u0646",
"\u0627\u0644\u0633\u064f\u0648\u0633\u064a\u0646\u0627\u062a",
"\u062d\u0634\u0631\u0629 \u0630\u0648\u0627\u062a \u0627\u0644\u062c\u0646\u0627\u062d\u064a\u0646",
"\u0646\u062d\u0644",
"\u0627\u0644\u0646\u0645\u0644",
"\u062c\u0646\u062f\u0628",
"\u062d\u0634\u0631\u0629 \u0627\u0644\u0643\u0631\u064a\u0643\u064a\u062a",
"\u0627\u0644\u062d\u0634\u0631\u0629 \u0627\u0644\u0639\u0635\u0648\u064a\u0629",
"\u0635\u0631\u0635\u0648\u0631",
"\u0641\u0631\u0633 \u0627\u0644\u0646\u0628\u064a",
"\u062d\u0634\u0631\u0629 \u0632\u064a\u0632\u064a\u0627\u062a",
"\u0642\u0627\u0641\u0632\u0627\u062a \u0627\u0644\u0623\u0648\u0631\u0627\u0642",
"\u0639\u0631\u0642\u064a\u0627\u062a \u0627\u0644\u0623\u062c\u0646\u062d\u0629",
"\u0627\u0644\u064a\u0639\u0633\u0648\u0628",
"\u0645\u0642\u062a\u0631\u0646\u0627\u062a \u0627\u0644\u0623\u062c\u0646\u062d\u0629",
"\u0641\u0631\u0627\u0634\u0629 \u0628\u0634\u0648\u0631\u0629 \u0627\u0644\u0635\u064a\u0641",
"\u0641\u0631\u0627\u0634\u0629 \u062d\u0644\u064a\u0642\u0629",
"\u0641\u064e\u0631\u064e\u0627\u0634\u0629 \u0627\u0644\u0645\u064e\u0644\u0643",
"\u0641\u0631\u0627\u0634\u0629 \u0627\u0644\u0628\u064a\u0636\u0627\u0621 \u0627\u0644\u0635\u063a\u064a\u0631\u0629",
"\u0641\u0631\u0627\u0634\u0629 \u0627\u0644\u0643\u0628\u0631\u064a\u062a",
"\u0627\u0644\u0641\u0631\u0627\u0634\u0629 \u0627\u0644\u0646\u062d\u0627\u0633\u064a\u0629",
"\u0646\u062c\u0645 \u0627\u0644\u0628\u062d\u0631",
"\u0642\u0646\u0641\u0630 \u0627\u0644\u0628\u062d\u0631",
"\u062e\u064a\u0627\u0631 \u0627\u0644\u0628\u062d\u0631",
"\u0623\u0631\u0627\u0646\u0628 \u0642\u0637\u0646\u064a\u0629 \u0627\u0644\u0630\u064a\u0644",
"\u0623\u0631\u0646\u0628 \u0628\u0631\u064a",
"\u0627\u0644\u0623\u0646\u062c\u0648\u0631\u0627",
"\u0623\u0642\u062f\u0627\u062f",
"\u0627\u0644\u0646\u064a\u0635",
"\u0633\u0646\u062c\u0627\u0628 \u062b\u0639\u0644\u0628\u064a",
"\u0627\u0644\u0645\u0631\u0645\u0648\u0637",
"\u0627\u0644\u0642\u0646\u062f\u0633",
"\u0643\u0627\u0628\u064a\u0627\u0621 \u062e\u0646\u0632\u064a\u0631\u064a\u0629",
"\u0627\u0644\u062d\u0635\u0627\u0646 \u0627\u0644\u062d\u0645\u064a\u0636",
"\u0627\u0644\u062d\u0645\u0627\u0631 \u0627\u0644\u0645\u062e\u0637\u0637",
"\u0627\u0644\u062e\u0646\u0632\u064a\u0631 \u0627\u0644\u0623\u0644\u064a\u0641 \u0623\u0648 \u0627\u0644\u062e\u0646\u0632\u064a\u0631 \u0627\u0644\u0645\u0633\u062a\u0623\u0646\u0633",
"\u0627\u0644\u062e\u0646\u0632\u064a\u0631 \u0627\u0644\u0628\u0631\u064a ",
"\u062e\u0646\u0627\u0632\u064a\u0631 \u062a\u0624\u0644\u0648\u0644\u064a\u0629",
"\u0641\u0631\u0633 \u0627\u0644\u0646\u0647\u0631",
"\u0627\u0644\u062b\u0648\u0631",
"\u062c\u0627\u0645\u0648\u0633 \u0627\u0644\u0645\u0627\u0621",
"\u0627\u0644\u0628\u064a\u0633\u0648\u0646",
"\u0630\u0643\u0631 \u0627\u0644\u062e\u0631\u0648\u0641",
"\u0643\u0628\u0634 \u0627\u0644\u062c\u0628\u0627\u0644 \u0627\u0644\u0635\u062e\u0631\u064a\u0629",
"\u0627\u0644\u0648\u0639\u0644",
"\u062b\u064a\u062a\u0644 \u0627\u0644\u0647\u0631\u062a\u0628\u064a\u0633",
"\u0625\u0645\u0628\u0627\u0644\u0629",
"\u063a\u0632\u0627\u0644",
"\u0627\u0644\u062c\u0645\u0644 \u0627\u0644\u0639\u0631\u0628\u064a",
"\u0627\u0644\u0644\u0627\u0651\u0645\u0629",
"\u0627\u0628\u0646 \u0639\u0631\u0633",
"\u0627\u0644\u0645\u0646\u0643",
"\u0627\u0644\u0633\u0641\u0634\u0629",
"\u0627\u0628\u0646 \u0645\u0642\u0631\u0636 \u0623\u0633\u0648\u062f \u0627\u0644\u0623\u0642\u062f\u0627\u0645",
"\u0642\u0636\u0627\u0639\u0629",
"\u0638\u0631\u0628\u0627\u0646",
"\u0627\u0644\u063a\u0631\u064a\u0631",
"\u0627\u0644\u0645\u064f\u062f\u064e\u0631\u064e\u0651\u0639 \u0623\u0648 \u0627\u0644\u0623\u0631\u0645\u0627\u062f\u064a\u0644\u0644\u0648",
"\u0643\u0633\u0644\u0627\u0646 \u062b\u0644\u0627\u062b\u064a \u0627\u0644\u0623\u0635\u0627\u0628\u0639",
"\u0642\u0631\u062f \u0627\u0644\u0627\u0648\u0631\u0627\u0646\u063a\u0648\u062a\u0627\u0646",
"\u0627\u0644\u063a\u0648\u0631\u064a\u0644\u0627 \u0623\u0648 \u0627\u0644\u0642\u064f\u0631\u062f\u0648\u062d",
"\u0627\u0644\u0634\u0645\u0628\u0627\u0646\u0632\u064a \u0627\u0644\u0634\u0627\u0626\u0639 \u0623\u0648 \u0627\u0644\u0628\u064e\u0639\u0627\u0645",
"\u0642\u0631\u062f \u0627\u0644\u062c\u0628\u0648\u0646",
"\u0642\u0631\u062f \u0627\u0644\u0633\u064a\u0627\u0645\u0646\u062c",
"\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u063a\u064a\u0646\u0648\u0646",
"\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u0628\u0627\u062a\u0627\u0633",
"\u0627\u0644\u0631\u064f\u0628\u064e\u0651\u0627\u062d",
"\u0642\u0631\u062f \u0627\u0644\u0645\u0643\u0627\u0643",
"\u0642\u0631\u062f \u0627\u0644\u0643\u0648\u0644\u0628\u0633\u0627\u0648\u0627\u062a",
"\u0642\u0631\u062f \u0627\u0644\u0643\u0648\u0644\u0628\u0633",
"\u0642\u0631\u062f \u0627\u0644\u0645\u0644\u0645\u0644\u0629",
"\u0642\u0631\u0648\u062f \u0627\u0644\u0642\u0634\u0629",
"\u0642\u0631\u062f \u0627\u0644\u0643\u0628\u0648\u0634\u0629 \u0623\u0628\u064a\u0636 \u0627\u0644\u0648\u062c\u0647",
"\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u0639\u0648\u0627\u0621",
"\u0642\u0631\u062f \u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u062a\u064a\u062a\u064a",
"\u0627\u0644\u0633\u064e\u0651\u0639\u062f\u0627\u0646 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a\u064a",
"\u0627\u0644\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u0633\u0646\u062c\u0627\u0628\u064a",
"\u0644\u064a\u0645\u0648\u0631 \u062d\u0644\u0642\u064a \u0627\u0644\u0630\u064a\u0644",
"\u062d\u064a\u0648\u0627\u0646 \u0627\u0644\u0627\u0646\u062f\u0631\u064a",
"\u0641\u064a\u0644 \u0647\u0646\u062f\u064a",
"\u0641\u064a\u0644 \u0623\u0641\u0631\u064a\u0642\u064a",
"\u0627\u0644\u0628\u0627\u0646\u062f\u0627 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0627\u0644\u0628\u0627\u0646\u062f\u0627 \u0627\u0644\u0639\u0645\u0644\u0627\u0642\u0629",
"\u062b\u064a\u0631\u0633\u064a\u062a\u064a\u0627\u062a",
"\u0633\u0645\u0643 \u0627\u0644\u0627\u0646\u0642\u0644\u064a\u0633",
"\u0633\u0645\u0643 \u0627\u0644\u0643\u0648\u0647\u0648 \u0627\u0644\u0633\u064a\u0644\u0645\u0648\u0646",
"\u0633\u0645\u0643 \u0627\u0644\u062c\u0645\u0627\u0644 \u0627\u0644\u0635\u062e\u0631\u064a",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u0645\u0647\u0631\u062c",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u062d\u0641\u0634\u064a\u0629",
"\u0633\u0645\u0643 \u0627\u0644\u0631\u0645\u062d",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u062a\u0646\u064a\u0646",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u064a\u0646\u0641\u0648\u062e\u064a\u0629",
"\u0627\u0644\u0645\u0650\u0639\u0652\u062f\u064e\u0627\u062f",
"\u0627\u0644\u0639\u0628\u0627\u0621\u0629",
"\u0644\u0628\u0627\u0633 \u062a\u062e\u0631\u062c",
"\u0623\u0643\u0648\u0631\u062f\u064a\u0648\u0646",
"\u0627\u0644\u0642\u064a\u062b\u0627\u0631\u0629 \u0627\u0644\u0635\u0648\u062a\u064a\u0629",
"\u062d\u0627\u0645\u0644\u0629 \u0637\u0627\u0626\u0631\u0627\u062a",
"\u0637\u0627\u0626\u0631\u0629 \u0631\u062d\u0644\u0627\u062a",
"\u0633\u0641\u064a\u0646\u0629 \u0647\u0648\u0627\u0626\u064a\u0629",
"\u0645\u0630\u0628\u062d",
"\u0633\u064a\u0627\u0631\u0629 \u0625\u0633\u0639\u0627\u0641",
"\u0627\u0644\u0645\u0631\u0643\u0628\u0629 \u0627\u0644\u0628\u0631\u0645\u0627\u0626\u064a\u0629",
"\u0627\u0644\u0633\u0627\u0639\u0629 \u0627\u0644\u0645\u062a\u0646\u0627\u0638\u0631\u0629",
"\u0627\u0644\u0645\u0646\u062d\u0644 \u0623\u0648 \u0627\u0644\u0645\u064e\u0646\u062d\u064e\u0644\u064e\u0629",
"\u0645\u0626\u0632\u0631",
"\u062d\u0627\u0648\u064a\u0629 \u0627\u0644\u0646\u0641\u0627\u064a\u0627\u062a",
"\u0628\u0646\u062f\u0642\u064a\u0629 \u0627\u0642\u062a\u062d\u0627\u0645",
"\u062d\u0642\u064a\u0628\u0629 \u0638\u0647\u0631",
"\u0627\u0644\u0645\u062e\u0628\u0632",
"\u0639\u0627\u0631\u0636\u0629 \u0627\u0644\u062a\u0648\u0627\u0632\u0646",
"\u0627\u0644\u0628\u0627\u0644\u0648\u0646",
"\u0642\u0644\u0645 \u062d\u0628\u0631 \u062c\u0627\u0641",
"\u0636\u0645\u0627\u062f\u0629 \u0637\u0628\u064a\u0629 \u0644\u0627\u0635\u0642\u0629",
"\u0627\u0644\u0628\u0627\u0646\u062c\u0648",
"\u062f\u0631\u0627\u0628\u0632\u064a\u0646",
"\u062d\u062f\u064a\u062f\u0629 (\u0631\u0641\u0639 \u0623\u062b\u0642\u0627\u0644)",
"\u0643\u0631\u0633\u064a \u0627\u0644\u062d\u0644\u0627\u0642\u0629",
"\u0645\u062d\u0644 \u0635\u0627\u0644\u0648\u0646 \u0627\u0644\u062d\u0644\u0627\u0642\u0629",
"\u062d\u0638\u064a\u0631\u0629",
"\u0627\u0644\u0628\u0627\u0631\u0648\u0645\u062a\u0631",
"\u0627\u0644\u0628\u0631\u0645\u064a\u0644",
"\u0639\u062c\u0644\u0629 \u0627\u0644\u064a\u062f",
"\u0643\u0631\u0629 \u0627\u0644\u0642\u0627\u0639\u062f\u0629 \u0623\u0648 \u0627\u0644\u0628\u064a\u0633\u0628\u0648\u0644",
"\u0643\u0631\u0629 \u0633\u0644\u0629",
"\u0633\u0631\u064a\u0631 \u0627\u0644\u0623\u0637\u0641\u0627\u0644",
"\u0645\u0632\u0645\u0627\u0631",
"\u0642\u0628\u0639\u0629 \u0633\u0628\u0627\u062d\u0629",
"\u0645\u0646\u0634\u0641\u0629",
"\u062d\u0648\u0636 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u0633\u064a\u0627\u0629 \u0648\u0627\u063a\u0646",
"\u0627\u0644\u0645\u0646\u0627\u0631\u0629 \u0623\u0648 \u0627\u0644\u0641\u0646\u0627\u0631",
"\u0643\u0648\u0628 \u0632\u062c\u0627\u062c\u064a",
"\u0642\u0628\u0639\u0629 \u0627\u0644\u062f\u0628",
"\u0632\u062c\u0627\u062c\u0629 \u0627\u0644\u0628\u064a\u0631\u0629",
"\u0643\u0623\u0633 \u062c\u0639\u0629",
"\u0628\u0631\u062c \u0627\u0644\u0646\u0627\u0642\u0648\u0633",
"\u0645\u0631\u0648\u0644\u0629",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u062a\u0631\u0627\u062f\u0641\u064a\u0629",
"\u0628\u0643\u064a\u0646\u064a",
"\u0627\u0644\u0645\u062c\u0644\u062f\u0627\u062a \u0627\u0644\u062d\u0644\u0642\u064a\u0629",
"\u0627\u0644\u0645\u0650\u0646\u0652\u0638\u0627\u0631",
"\u0635\u0646\u062f\u0648\u0642 \u0627\u0644\u0639\u0634",
"\u0627\u0644\u0645\u0631\u0641\u0623",
"\u0627\u0644\u0632\u0644\u0627\u062c\u0629 \u0627\u0644\u062c\u0645\u0627\u0639\u064a\u0629",
"\u0631\u0628\u0637\u0629 \u0639\u0646\u0642 \u0628\u0648\u0644\u0648",
"\u0627\u0644\u0643\u0632\u0629",
"\u0631\u0641 \u0627\u0644\u0643\u062a\u0628",
"\u0645\u0643\u062a\u0628\u0629",
"\u063a\u0637\u0627\u0621 \u0642\u0627\u0631\u0648\u0631\u0629",
"\u0627\u0644\u0642\u0648\u0633",
"\u0623\u0631\u0628\u0629 \u0641\u0631\u0627\u0634\u064a\u0629",
"\u0644\u0627\u0641\u062a\u0629 \u062a\u0627\u0631\u064a\u062e\u064a\u0629 \u0645\u0646 \u0627\u0644\u0646\u062d\u0627\u0633",
"\u062d\u0645\u0627\u0644\u0629 \u0627\u0644\u0635\u062f\u0631",
"\u062d\u0627\u062c\u0632 \u0627\u0644\u0623\u0645\u0648\u0627\u062c",
"\u062f\u0631\u0639 \u0627\u0644\u0635\u062f\u0631",
"\u0627\u0644\u0645\u0643\u0646\u0633\u0629",
"\u0627\u0644\u062f\u0644\u0648",
"\u0645\u0631\u0628\u0637 \u0627\u0644\u062d\u0632\u0627\u0645",
"\u0627\u0644\u0633\u062a\u0631\u0629 \u0627\u0644\u0648\u0627\u0642\u064a\u0629 \u0645\u0646 \u0627\u0644\u0631\u0635\u0627\u0635",
"\u0642\u0637\u0627\u0631 \u0627\u0644\u0637\u0644\u0642\u0629",
"\u0627\u0644\u0645\u062c\u0632\u0631\u0629",
"\u0633\u064a\u0627\u0631\u0629 \u0623\u062c\u0631\u0629",
"\u062d\u0644\u0629 (\u0622\u0646\u064a\u0629)",
"\u0634\u0645\u0639\u0629",
"\u0645\u062f\u0641\u0639",
"\u0642\u0627\u0631\u0628 \u0627\u0644\u0643\u0627\u0646\u0648",
"\u0641\u0627\u062a\u062d\u0629 \u0639\u0644\u0628",
"\u0633\u062a\u0631\u0629 \u0645\u062d\u0628\u0648\u0643\u0629",
"\u0627\u0644\u0645\u0631\u0622\u0629 \u0627\u0644\u062c\u0627\u0646\u0628\u064a\u0629",
"\u062f\u0648\u0627\u0645\u0629 \u0627\u0644\u062e\u064a\u0644",
" \u0623\u062f\u0648\u0627\u062a \u0627\u0644\u0635\u064a\u0627\u0646\u0629",
"\u0635\u0646\u062f\u0648\u0642 \u0643\u0631\u062a\u0648\u0646",
"\u0627\u0644\u0625\u0637\u0627\u0631 \u0627\u0644\u0645\u0637\u0627\u0637",
"\u0627\u0644\u0635\u0631\u0627\u0641 \u0627\u0644\u0622\u0644\u064a",
"\u0627\u0644\u0634\u0631\u064a\u0637 \u0627\u0644\u0645\u062f\u0645\u062c",
"\u0627\u0644\u0645\u0633\u062c\u0644",
"\u0627\u0644\u0642\u064e\u0644\u0652\u0639\u064e\u0629",
"\u0627\u0644\u0642\u0637\u0645\u0631\u0627\u0646",
"\u062c\u0647\u0627\u0632 \u0627\u0644\u0642\u0631\u0635 \u0627\u0644\u0645\u0636\u063a\u0648\u0637",
"\u062a\u0634\u064a\u0644\u0648",
"\u0647\u0627\u062a\u0641 \u0645\u062d\u0645\u0648\u0644",
"\u0633\u0644\u0633\u0644\u0629",
"\u0633\u064a\u0627\u062c \u0645\u0634\u0628\u0643",
"\u0627\u0644\u0632\u0631\u062f",
"\u0645\u0646\u0634\u0627\u0631 \u062c\u0646\u0632\u064a\u0631\u064a",
" \u0635\u0646\u062f\u0648\u0642 \u0627\u0644\u062a\u062e\u0632\u064a\u0646",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0623\u062b\u0627\u062b",
"\u0627\u0644\u0622\u0644\u0629 \u0627\u0644\u0625\u064a\u0642\u0627\u0639\u064a\u0629 ",
"\u0627\u0644\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0635\u064a\u0646\u064a\u0629",
"\u062c\u0648\u0631\u0628 \u0639\u064a\u062f \u0627\u0644\u0645\u064a\u0644\u0627\u062f",
"\u0643\u0646\u064a\u0633\u0629",
"\u0645\u0633\u0631\u062d \u0623\u0641\u0644\u0627\u0645",
"\u0627\u0644\u0633\u0627\u0637\u0648\u0631",
"\u0645\u0633\u0627\u0643\u0646 \u0627\u0644\u062c\u0631\u0641",
"\u0627\u0644\u0645\u0639\u0637\u0641 \u0627\u0644\u0641\u0636\u0641\u0627\u0636",
"\u0627\u0644\u0642\u0628\u0642\u0627\u0628",
"\u062e\u0627\u0644\u0637 \u0627\u0644\u0645\u0634\u0631\u0648\u0628\u0627\u062a \u0627\u0644\u0643\u062d\u0648\u0644\u064a\u0629",
"\u0643\u0648\u0632 (\u0622\u0646\u064a\u0629)",
"\u0622\u0644\u0629 \u062a\u062d\u0636\u064a\u0631 \u0627\u0644\u0642\u0647\u0648\u0629",
"\u0627\u0644\u0634\u0643\u0644 \u0627\u0644\u062d\u0644\u0632\u0648\u0646\u064a",
"\u0627\u0644\u0642\u0641\u0644 \u0627\u0644\u0631\u0645\u0632\u064a",
"\u0644\u0648\u062d\u0629 \u0627\u0644\u0645\u0641\u0627\u062a\u064a\u062d",
"\u0645\u062a\u062c\u0631 \u0627\u0644\u062d\u0644\u0648\u064a\u0627\u062a",
"\u0633\u0641\u064a\u0646\u0629 \u062d\u0627\u0648\u064a\u0627\u062a",
"\u0633\u064a\u0627\u0631\u0629 \u0645\u0643\u0634\u0648\u0641\u0629",
"\u0628\u0631\u0627\u0645\u0629",
"\u0627\u0644\u0634\u064a\u0627\u0639",
"\u062c\u0632\u0645\u0629 \u0631\u0627\u0639\u064a \u0627\u0644\u0628\u0642\u0631",
"\u0642\u0628\u0639\u0629 \u0631\u0627\u0639\u064a \u0627\u0644\u0628\u0642\u0631",
"\u0627\u0644\u0645\u0647\u062f",
"\u0631\u0627\u0641\u0639\u0629",
"\u0627\u0644\u062e\u0648\u0630\u0629",
"\u062d\u0627\u0648\u064a\u0629 \u0634\u062d\u0646 \u0643\u0628\u064a\u0631\u0629",
"\u0633\u0631\u064a\u0631 \u0627\u0644\u0631\u0636\u064a\u0639",
"\u0642\u062f\u0631 \u0627\u0644\u0637\u0628\u062e \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0643\u0631\u0648\u0643\u064a\u062a",
"\u0627\u0644\u0639\u0643\u0627\u0632",
"\u0643\u0648\u064a\u0631\u0633",
"\u0627\u0644\u0633\u062f",
"\u0627\u0644\u0645\u0643\u062a\u0628",
"\u062d\u0627\u0633\u0648\u0628 \u0645\u0643\u062a\u0628\u064a",
"\u0627\u0644\u0647\u0627\u062a\u0641 \u0627\u0644\u062f\u0648\u0627\u0631",
"\u0627\u0644\u062d\u0641\u0627\u0638\u0629",
"\u0627\u0644\u0633\u0627\u0639\u0629 \u0627\u0644\u0631\u0642\u0645\u064a\u0629",
"\u0633\u0627\u0639\u0627\u062a \u0627\u0644\u064a\u062f \u0627\u0644\u0631\u0642\u0645\u064a\u0629",
"\u0627\u0644\u0645\u0646\u0636\u062f\u0629",
"\u0642\u0645\u0627\u0634 \u0627\u0644\u0623\u0637\u0628\u0627\u0642",
"\u063a\u0633\u0627\u0644\u0629 \u0635\u062d\u0648\u0646",
"\u0645\u0643\u0628\u062d \u0642\u0631\u0635\u064a",
"\u0645\u064a\u0646\u0627\u0621",
"\u0627\u0644\u0632\u0644\u0627\u062c\u0629 \u0627\u0644\u062a\u064a \u062a\u062c\u0631\u0647\u0627 \u0627\u0644\u0643\u0644\u0627\u0628",
"\u0642\u0628\u0629",
"\u0627\u0644\u062d\u0635\u064a\u0631\u0629",
"\u0645\u0646\u0635\u0629 \u062d\u0641\u0631",
"\u0627\u0644\u0637\u0628\u0644",
"\u0639\u0635\u0627 \u0627\u0644\u0637\u0628\u0644",
"\u062b\u0642\u0627\u0644\u0627\u062a \u062d\u062f\u064a\u062f",
"\u0641\u0631\u0646 \u0647\u0648\u0644\u0646\u062f\u064a",
"\u0645\u0631\u0648\u062d\u0629",
"\u0627\u0644\u062c\u064a\u062a\u0627\u0631 \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0627\u0644\u0642\u0627\u0637\u0631\u0629 \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a\u0629",
"\u0645\u0631\u0643\u0632 \u0627\u0644\u062a\u0631\u0641\u064a\u0647",
"\u0638\u0631\u0641 \u0628\u0631\u064a\u062f\u064a",
"\u0622\u0644\u0629 \u0627\u0644\u0625\u0633\u0628\u0631\u064a\u0633\u0648",
"\u0628\u0648\u062f\u0631\u0629 \u0627\u0644\u0648\u062c\u0647",
"\u0623\u0635\u0644\u0629 \u0631\u064a\u0634\u064a\u0629",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0645\u0644\u0641\u0627\u062a",
"\u0632\u0648\u0631\u0642 \u0627\u0644\u0625\u0637\u0641\u0627\u0621",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0625\u0637\u0641\u0627\u0621",
"\u0648\u0627\u0642\u064a \u0627\u0644\u0646\u0627\u0631",
"\u0633\u0627\u0631\u064a\u0629 \u0627\u0644\u0639\u0644\u0645",
"\u0627\u0644\u0641\u0644\u0648\u062a",
"\u0643\u0631\u0633\u064a \u0642\u0627\u0628\u0644 \u0644\u0644\u0637\u064a",
"\u062e\u0648\u0630\u0629 \u0643\u0631\u0629 \u0627\u0644\u0642\u062f\u0645",
"\u0631\u0627\u0641\u0639\u0629 \u0627\u0644\u062d\u0645\u0648\u0644\u0629",
"\u0627\u0644\u0646\u0627\u0641\u0648\u0631\u0629",
"\u0642\u0644\u0645 \u062d\u0628\u0631 \u0633\u0627\u0626\u0644",
"\u0627\u0644\u0633\u0631\u064a\u0631 \u0628\u0623\u0631\u0628\u0639\u0629 \u0623\u0639\u0645\u062f\u0629",
"\u0633\u064a\u0627\u0631\u0629 \u0634\u062d\u0646",
"\u0627\u0644\u0628\u0648\u0642 \u0627\u0644\u0641\u0631\u0646\u0633\u064a",
"\u0645\u0642\u0644\u0627\u0629",
"\u0627\u0644\u0645\u0644\u0627\u0628\u0633 \u0627\u0644\u0645\u0635\u0646\u0648\u0639\u0629 \u0645\u0646 \u0627\u0644\u0641\u0631\u0648\u0629",
"\u0634\u0627\u062d\u0646\u0629 \u0642\u0645\u0627\u0645\u0629",
"\u0642\u0646\u0627\u0639 \u0627\u0644\u063a\u0627\u0632",
"\u0645\u0636\u062e\u0629 \u0627\u0644\u0648\u0642\u0648\u062f",
"\u0643\u0623\u0633 \u0627\u0644\u0646\u0628\u064a\u0630",
"\u0633\u064a\u0627\u0631\u0629 \u062c\u0648 \u0643\u0627\u0631\u062a",
"\u0643\u0631\u0629 \u0627\u0644\u062c\u0648\u0644\u0641",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062c\u0648\u0644\u0641",
"\u0627\u0644\u0642\u0627\u0631\u0628 \u0627\u0644\u0637\u0648\u064a\u0644 \u0623\u0648 \u0627\u0644\u063a\u0646\u062f\u0648\u0644",
"\u0627\u0644\u0635\u0646\u062c\u0629",
"\u0627\u0644\u062b\u0648\u0628 \u0627\u0644\u0646\u0633\u0627\u0626\u064a \u0623\u0648 \u0627\u0644\u0641\u0633\u062a\u0627\u0646",
"\u0627\u0644\u0628\u064a\u0627\u0646\u0648 \u0627\u0644\u0643\u0628\u064a\u0631",
"\u0627\u0644\u062f\u0641\u064a\u0626\u0629 \u0627\u0644\u0632\u0631\u0627\u0639\u064a\u0629",
"\u0634\u0628\u0643 \u0627\u0644\u0633\u064a\u0627\u0631\u0629",
"\u0627\u0644\u0628\u0642\u0627\u0644\u0629",
"\u0627\u0644\u0645\u0642\u0635\u0644\u0629",
"\u0645\u0634\u0628\u0643 \u0644\u0644\u0634\u0639\u0631",
"\u0628\u062e\u0627\u062e \u0645\u062b\u0628\u062a \u0627\u0644\u0634\u0639\u0631",
"\u0627\u0644\u0639\u0631\u0628\u0629 \u0646\u0635\u0641 \u0627\u0644\u0645\u062c\u0646\u0632\u0631\u0629",
"\u0645\u0637\u0631\u0642\u0629",
"\u0627\u0644\u0633\u0644\u0629",
"\u0645\u062c\u0641\u0641 \u0627\u0644\u0634\u0639\u0631",
"\u062c\u0647\u0627\u0632 \u0645\u062d\u0645\u0648\u0644 \u0628\u0627\u0644\u064a\u062f",
"\u0627\u0644\u0645\u0646\u062f\u064a\u0644 \u0623\u0648 \u0627\u0644\u0645\u062d\u0631\u0645\u0629",
"\u0642\u0631\u0635 \u0635\u0644\u0628",
"\u0627\u0644\u0634\u064e\u0651\u0641\u064e\u0648\u0650\u064a\u064e\u0651\u0629",
"\u0627\u0644\u0642\u064a\u062b\u0627\u0631\u0629",
"\u0627\u0644\u062d\u0635\u0651\u0627\u062f\u0629",
"\u062e\u0635\u064a\u0646",
"\u062d\u0627\u0641\u0638\u0629 \u0627\u0644\u0645\u0633\u062f\u0633",
"\u0627\u0644\u0645\u0633\u0631\u062d \u0627\u0644\u0645\u0646\u0632\u0644\u064a",
"\u0642\u0631\u0635 \u0639\u0633\u0644",
"\u0627\u0644\u062e\u0637\u0627\u0641",
"\u0627\u0644\u062a\u0646\u0648\u0631\u0629 \u0627\u0644\u0645\u064f\u0637\u064e\u0648\u064e\u0651\u0642\u0629",
"\u0639\u0642\u0644\u0629 (\u062c\u0645\u0628\u0627\u0632)",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062e\u064a\u0648\u0644",
"\u0633\u0627\u0639\u0629 \u0631\u0645\u0644\u064a\u0629",
"\u0622\u064a \u0628\u0648\u062f",
"\u0627\u0644\u0645\u0643\u0648\u0627\u0629",
"\u0627\u0644\u0642\u0631\u0639\u0629 \u0627\u0644\u0645\u0636\u064a\u0626\u0629",
"\u0627\u0644\u062c\u064a\u0646\u0632",
"\u062c\u064a\u0628 (\u0633\u064a\u0627\u0631\u0629)",
"\u0642\u0645\u064a\u0635 \u0642\u0635\u064a\u0631 \u0627\u0644\u0643\u0645\u064a\u0646",
"\u0623\u062d\u062c\u064a\u0629 \u0627\u0644\u0635\u0648\u0631 \u0627\u0644\u0645\u0642\u0637\u0648\u0639\u0629",
"\u0631\u064a\u0643\u0634\u0627",
"\u0630\u0631\u0627\u0639 \u0627\u0644\u062a\u0648\u062c\u064a\u0647",
"\u0627\u0644\u0643\u064a\u0645\u0648\u0646\u0648",
"\u0648\u0633\u0627\u062f\u0627\u062a \u0627\u0644\u0631\u0643\u0628\u0629",
"\u0627\u0644\u0639\u0642\u062f\u0629",
"\u0645\u0639\u0637\u0641 \u0627\u0644\u0645\u062e\u062a\u0628\u0631",
"\u0627\u0644\u0645\u063a\u0631\u0641\u0629",
"\u0639\u0627\u0643\u0633 \u0627\u0644\u0636\u0648\u0621",
"\u062d\u0627\u0633\u0648\u0628 \u0645\u062d\u0645\u0648\u0644",
"\u062c\u0632\u0627\u0632\u0629 \u0627\u0644\u0639\u0634\u0628",
"\u063a\u0637\u0627\u0621 \u0627\u0644\u0639\u062f\u0633\u0629",
"\u0641\u062a\u0627\u062d\u0629 \u0627\u0644\u0631\u0633\u0627\u0626\u0644",
"\u0645\u0643\u062a\u0628\u0629",
"\u0642\u0627\u0631\u0628 \u0627\u0644\u0646\u062c\u0627\u0629",
"\u0627\u0644\u0642\u064e\u062f\u064e\u0651\u0627\u062d\u064e\u0629",
"\u0627\u0644\u0644\u064a\u0645\u0648\u0632\u064a\u0646",
"\u0639\u0627\u0628\u0631\u0629 \u0645\u062d\u064a\u0637 \u0645\u0646\u062a\u0638\u0645\u0629",
"\u0623\u062d\u0645\u0631 \u0634\u0641\u0627\u0647",
"\u0627\u0644\u062d\u0630\u0627\u0621 \u0633\u0647\u0644 \u0627\u0644\u0627\u0631\u062a\u062f\u0627\u0621",
"\u063a\u0633\u0648\u0644",
"\u0645\u0643\u0628\u0631 \u0627\u0644\u0635\u0648\u062a",
"\u0639\u062f\u0633\u0629",
"\u0627\u0644\u0645\u0646\u0634\u0631\u0629",
"\u0627\u0644\u0628\u0648\u0635\u0644\u0629",
"\u062d\u0642\u064a\u0628\u0629 \u0633\u0627\u0639\u064a \u0627\u0644\u0628\u0631\u064a\u062f",
"\u0635\u0646\u062f\u0648\u0642 \u0628\u0631\u064a\u062f",
"\u0627\u0644\u062c\u0648\u0627\u0631\u0628 \u0627\u0644\u0637\u0648\u064a\u0644\u0629",
"\u0627\u0644\u0645\u0627\u064a\u0648\u0647",
"\u063a\u0637\u0627\u0621 \u0627\u0644\u0645\u0637\u0628\u0642",
"\u0622\u0644\u0629 \u0645\u0627\u0631\u0627\u0643\u0633",
"\u0627\u0644\u0645\u0627\u0631\u064a\u0645\u0628\u0627",
"\u0642\u0646\u0627\u0639",
"\u0623\u0639\u0648\u0627\u062f \u0627\u0644\u062b\u0642\u0627\u0628",
"\u0633\u0627\u0631\u064a\u0629 \u0645\u0627\u064a\u0648",
"\u0627\u0644\u0645\u062a\u0627\u0647\u0629",
"\u0643\u0648\u0628 \u0627\u0644\u0642\u064a\u0627\u0633",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0623\u062f\u0648\u064a\u0629",
"\u0627\u0644\u0622\u062b\u0627\u0631 \u0627\u0644\u0635\u062e\u0631\u064a\u0629",
"\u0627\u0644\u0644\u0627\u0642\u0637 \u0627\u0644\u0635\u0648\u062a\u064a",
"\u0641\u0631\u0646 \u0627\u0644\u0645\u064a\u0643\u0631\u0648\u064a\u0641",
"\u0627\u0644\u0632\u064a \u0627\u0644\u0639\u0633\u0643\u0631\u064a",
"\u0645\u062f\u0644\u062c\u0629",
"\u0627\u0644\u062d\u0627\u0641\u0644\u0629 \u0627\u0644\u0635\u063a\u064a\u0631\u0629",
"\u0627\u0644\u062a\u0646\u0648\u0631\u0629 \u0627\u0644\u0642\u0635\u064a\u0631\u0629",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0645\u064a\u0646\u064a \u0641\u0627\u0646 \u0627\u0644\u0639\u0627\u0626\u0644\u064a\u0629",
"\u0627\u0644\u0642\u0630\u064a\u0641\u0629 \u0627\u0644\u0645\u0648\u062c\u0647\u0629",
"\u0627\u0644\u0642\u0641\u0627\u0632 \u0645\u0644\u062a\u0635\u0642 \u0627\u0644\u0623\u0635\u0627\u0628\u0639",
"\u0637\u0628\u0642 \u062e\u0644\u0637",
"\u0627\u0644\u0645\u0646\u0632\u0644 \u0627\u0644\u0645\u062a\u0646\u0642\u0644",
"\u0641\u0648\u0631\u062f \u0645\u0648\u062f\u064a\u0644 \u062a\u064a",
"\u0627\u0644\u0645\u0648\u062f\u0645",
"\u0627\u0644\u062f\u064a\u0631",
"\u0634\u0627\u0634\u0629 \u062d\u0627\u0633\u0648\u0628",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u0646\u0627\u0631\u064a\u0629 \u0627\u0644\u0635\u063a\u064a\u0631\u0629",
"\u0627\u0644\u0647\u0627\u0648\u0646 \u0648\u0627\u0644\u0645\u062f\u0642\u0629",
"\u0627\u0644\u0642\u0628\u0639\u0629 \u0627\u0644\u062c\u0627\u0645\u0639\u064a\u0629 \u0627\u0644\u0645\u0631\u0628\u0639\u0629",
"\u0645\u0633\u062c\u062f",
"\u0627\u0644\u0646\u0627\u0645\u0648\u0633\u064a\u0651\u0629",
"\u0627\u0644\u062f\u0639\u0631\u0648\u0645\u0629",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u0647\u0648\u0627\u0626\u064a\u0629 \u0627\u0644\u062c\u0628\u0644\u064a\u0629",
"\u062e\u064a\u0645\u0629",
"\u0627\u0644\u0641\u0623\u0631\u0629",
"\u0645\u0635\u064a\u062f\u0629 \u0627\u0644\u0641\u0626\u0631\u0627\u0646",
"\u0633\u064a\u0627\u0631\u0627\u062a \u0634\u0631\u0643\u0629 \u0627\u0644\u0646\u0642\u0644",
"\u0643\u0645\u0627\u0645 \u0627\u0644\u0641\u0645",
"\u0645\u0633\u0645\u0627\u0631",
"\u0637\u0648\u0642 \u0627\u0644\u0639\u0646\u0642",
"\u0627\u0644\u0642\u0644\u0627\u062f\u0629",
"\u0627\u0644\u0631\u0636\u0627\u0639\u0629",
"\u062d\u0627\u0633\u0628 \u0627\u0644\u0645\u0641\u0643\u0631\u0629",
"\u0627\u0644\u0645\u0633\u0644\u0629",
"\u0622\u0644\u0629 \u0627\u0644\u0623\u0648\u0628\u0648\u0627",
"\u0623\u0643\u0631\u064a\u0646\u0629",
"\u0639\u062f\u0627\u062f \u0627\u0644\u0645\u0633\u0627\u0641\u0627\u062a",
"\u0641\u0644\u062a\u0631 \u0627\u0644\u0632\u064a\u062a",
"\u0627\u0644\u0623\u0631\u063a\u0646 \u0630\u0648 \u0627\u0644\u0623\u0646\u0627\u0628\u064a\u0628",
"\u062c\u0647\u0627\u0632 \u0631\u0627\u0633\u0645 \u0627\u0644\u0625\u0634\u0627\u0631\u0629",
"\u0627\u0644\u062a\u0646\u0648\u0631\u0629 \u0627\u0644\u062e\u0627\u0631\u062c\u064a\u0629",
"\u0639\u0631\u0628\u0629 \u064a\u062c\u0631\u0647\u0627 \u0627\u0644\u062b\u0648\u0631",
"\u0642\u0646\u0627\u0639 \u0623\u0643\u0633\u062c\u064a\u0646",
"\u0627\u0644\u062a\u063a\u0644\u064a\u0641",
"\u0627\u0644\u0645\u0650\u063a\u0652\u062f\u0627\u0641",
"\u0639\u064e\u062c\u064e\u0644\u0629 \u0627\u0644\u062a\u064e\u063a\u062f\u064a\u0641",
"\u0642\u0641\u0644 \u062d\u0644\u0642\u064a",
"\u0641\u0631\u0634\u0627\u0629 \u0627\u0644\u0631\u0633\u0645",
"\u0627\u0644\u0645\u0650\u0646\u064e\u0627\u0645\u064e\u0629\u064f",
"\u0642\u0635\u0631",
"\u0627\u0644\u0645\u0650\u0635\u0641\u0627\u0631",
"\u0645\u0646\u0634\u0641\u0629 \u0648\u0631\u0642\u064a\u0629",
"\u0645\u0650\u0638\u064e\u0644\u064e\u0651\u0629 \u0627\u0644\u0647\u0628\u064f\u0648\u0637",
"\u062c\u0647\u0627\u0632 \u0627\u0644\u0639\u0642\u0644\u0629",
"\u0645\u0642\u0639\u062f \u0639\u0627\u0645",
"\u0639\u062f\u0627\u062f \u0627\u0646\u062a\u0638\u0627\u0631 \u0627\u0644\u0633\u064a\u0627\u0631\u0627\u062a",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u0642\u0637\u0627\u0631",
"\u0627\u0644\u0641\u0646\u0627\u0621",
"\u0627\u0644\u0647\u0627\u062a\u0641 \u0627\u0644\u0639\u0645\u0648\u0645\u064a",
"\u0627\u0644\u0631\u0643\u064a\u0632\u0629",
"\u0627\u0644\u0645\u064e\u0642\u0652\u0644\u064e\u0645\u064e\u0629",
"\u0627\u0644\u0645\u0628\u0631\u0627\u0629",
"\u0627\u0644\u0639\u0637\u0631",
"\u0637\u0628\u0642 \u0628\u062a\u0631\u064a",
"\u0627\u0644\u0622\u0644\u0629 \u0627\u0644\u0646\u0627\u0633\u062e\u0629",
"\u0627\u0644\u0631\u064a\u0634\u0629 \u0627\u0644\u0645\u0648\u0633\u064a\u0642\u064a\u0629",
"\u062e\u0648\u0630\u0629 \u0628\u064a\u0643\u0644\u0647\u0627\u0648\u0628\u0647",
"\u0627\u0644\u0633\u064a\u0627\u062c \u0627\u0644\u0648\u062a\u062f\u064a",
"\u0627\u0644\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0646\u0635\u0641-\u0646\u0642\u0644",
"\u0627\u0644\u0631\u0635\u064a\u0641 \u0627\u0644\u0628\u062d\u0631\u064a",
"\u062d\u0635\u0627\u0644\u0629",
"\u0627\u0644\u062a\u063a\u0644\u064a\u0641 \u0627\u0644\u0635\u064a\u062f\u0644\u0627\u0646\u064a",
"\u0648\u0633\u0627\u062f\u0629",
"\u0643\u0631\u0629 \u0627\u0644\u0637\u0627\u0648\u0644\u0629",
"\u0644\u0639\u0628\u0629 \u0637\u0627\u062d\u0648\u0646\u0629 \u0647\u0648\u0627\u0621",
"\u0633\u0641\u064a\u0646\u0629 \u0627\u0644\u0642\u0631\u0627\u0635\u0646\u0629",
"\u0627\u0644\u0625\u0628\u0631\u064a\u0642",
"\u0627\u0644\u0645\u0650\u0633\u0652\u062d\u064e\u062c",
"\u0627\u0644\u0642\u0628\u0629 \u0627\u0644\u0641\u0644\u0643\u064a\u0629",
"\u0643\u064a\u0633 \u0646\u0627\u064a\u0644\u0648\u0646",
"\u0631\u0641 \u062a\u0646\u0634\u064a\u0641 \u0627\u0644\u0623\u0637\u0628\u0627\u0642",
"\u0627\u0644\u0645\u062d\u0627\u0631\u064a\u062b \u0627\u0644\u062d\u0641\u0627\u0631\u0629",
"\u0627\u0644\u0645\u0643\u0628\u0633 \u0627\u0644\u063a\u0637\u064e\u0651\u0627\u0633",
"\u0627\u0644\u0643\u0627\u0645\u064a\u0631\u0627 \u0627\u0644\u0641\u0648\u0631\u064a\u0629",
"\u0627\u0644\u0639\u0645\u0648\u062f",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u0634\u0631\u0637\u0629",
"\u0644\u0628\u0627\u0633 \u0627\u0644\u0628\u0646\u0634",
"\u0637\u0627\u0648\u0644\u0629 \u0627\u0644\u0628\u0644\u064a\u0627\u0631\u062f\u0648",
"\u0639\u0628\u0648\u0629 \u0627\u0644\u0645\u0634\u0631\u0648\u0628\u0627\u062a",
"\u0627\u0644\u0623\u0635\u064a\u0635",
"\u0639\u062c\u0644\u0629 \u0641\u062e\u0627\u0631",
"\u0627\u0644\u0645\u0650\u062b\u0652\u0642\u064e\u0628",
"\u0633\u062c\u0627\u062f\u0629 \u0627\u0644\u0635\u0644\u0627\u0629",
"\u0627\u0644\u0637\u0627\u0628\u0639\u0629 \u0627\u0644\u062d\u0627\u0633\u0648\u0628\u064a\u0629",
"\u0627\u0644\u0633\u062c\u0646",
"\u0627\u0644\u0642\u0630\u064a\u0641\u0629",
"\u0627\u0644\u0628\u0631\u0648\u062c\u0643\u062a\u0631",
"\u0642\u0631\u0635 \u0627\u0644\u0647\u0648\u0643\u064a",
"\u0627\u0644\u0645\u0644\u0643\u0645\u0629",
"\u062d\u0642\u064a\u0628\u0629 \u064a\u062f",
"\u0627\u0644\u0631\u064a\u0634\u0629 ",
"\u0627\u0644\u0644\u0650\u062d\u064e\u0627\u0641",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0633\u0628\u0627\u0642",
"\u0645\u0636\u0631\u0628 \u0627\u0644\u062a\u0646\u0633",
"\u0645\u0634\u0639\u0627\u0639",
"\u0627\u0644\u0645\u0630\u064a\u0627\u0639",
"\u0627\u0644\u0645\u0642\u0631\u0627\u0628 \u0627\u0644\u0627\u0630\u0627\u0639\u064a",
"\u062e\u0632\u0627\u0646 \u0645\u064a\u0627\u0647 \u0627\u0644\u0623\u0645\u0637\u0627\u0631",
"\u0627\u0644\u0645\u0631\u0643\u0628\u0627\u062a \u0627\u0644\u062a\u0631\u0641\u064a\u0647\u064a\u0629",
"\u0628\u0643\u0631\u0629 \u0635\u064a\u062f",
"\u0627\u0644\u0643\u0627\u0645\u064a\u0631\u0627 \u0627\u0644\u0627\u0646\u0639\u0643\u0627\u0633\u064a\u0629",
"\u0627\u0644\u062b\u0644\u0627\u064e\u0651\u062c\u0629",
"\u062c\u0647\u0627\u0632 \u062a\u062d\u0643\u0645 \u0639\u0646 \u0628\u0639\u062f",
"\u0645\u0637\u0639\u0645",
"\u0627\u0644\u0645\u0633\u062f\u0633 \u0627\u0644\u062f\u0648\u0627\u0631",
"\u0628\u0646\u062f\u0642\u064a\u0629",
"\u0627\u0644\u0643\u0631\u0633\u064a \u0627\u0644\u0647\u0632\u0627\u0632",
"\u0627\u0644\u0645\u0634\u0648\u0627\u0629",
"\u0627\u0644\u0645\u0645\u062d\u0627\u0629",
"\u0643\u0631\u0629 \u0627\u0644\u0631\u063a\u0628\u064a",
"\u0627\u0644\u0645\u0633\u0637\u0631\u0629",
"\u0627\u0644\u062d\u0630\u0627\u0621 \u0627\u0644\u0631\u064a\u0627\u0636\u064a",
"\u0627\u0644\u062e\u0632\u0627\u0646\u0629",
"\u062f\u0628\u0648\u0633 \u0645\u0634\u0628\u0643",
"\u0639\u0644\u0628 \u0627\u0644\u0645\u0644\u062d \u0648\u0627\u0644\u0641\u0644\u0641\u0644",
"\u0627\u0644\u0635\u0646\u062f\u0644",
"\u0627\u0644\u0633\u0627\u0631\u0648\u0646\u062c",
"\u0633\u0627\u0643\u0633\u0641\u0648\u0646 \u0627\u0644\u0629 \u0646\u0641\u062e \u0645\u0648\u0633\u064a\u0642\u064a\u0629",
"\u0627\u0644\u063a\u0650\u0645\u0652\u062f \u0623\u0648 \u063a\u0650\u0645\u0652\u062f \u0627\u0644\u0633\u064a\u0641 \u0623\u0648 \u063a\u0650\u0645\u0652\u062f \u0627\u0644\u062e\u0646\u062c\u0631",
"\u0627\u0644\u0645\u064a\u0632\u0627\u0646",
"\u062d\u0627\u0641\u0644\u0629 \u0645\u062f\u0631\u0633\u064a\u0629",
"\u0627\u0644\u0645\u0631\u0643\u0628 \u0627\u0644\u0634\u0631\u0627\u0639\u064a ",
"\u0644\u0648\u062d\u0629 \u0627\u0644\u0646\u062a\u0627\u0626\u062c",
"\u0634\u0627\u0634\u0629 \u0627\u0644\u0633\u064a \u0623\u0631 \u062a\u064a",
"\u0628\u0631\u063a\u064a",
"\u0645\u0641\u0643 \u0627\u0644\u0628\u0631\u0627\u063a\u064a",
"\u062d\u0632\u0627\u0645 \u0627\u0644\u0623\u0645\u0627\u0646",
"\u0622\u0644\u0629 \u0627\u0644\u062e\u064a\u0627\u0637\u0629",
"\u0627\u0644\u062f\u0631\u0639",
"\u0645\u062a\u062c\u0631 \u0627\u0644\u0623\u062d\u0630\u064a\u0629",
"\u0645\u0642\u0633\u0645 \u0627\u0644\u063a\u0631\u0641\u0629",
"\u0633\u0644\u0629 \u0627\u0644\u062a\u0633\u0648\u0642",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062a\u0633\u0648\u0642",
"\u0627\u0644\u0645\u0650\u062c\u0652\u0631\u064e\u0641\u064e\u0629",
"\u0642\u0628\u0639\u0629 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u0633\u062a\u0627\u0626\u0631 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u062a\u0632\u062d\u0644\u0642 \u0639\u0644\u0649 \u0627\u0644\u062b\u0644\u062c",
"\u0642\u0646\u0627\u0639 \u0627\u0644\u062a\u0632\u0644\u062c",
"\u0643\u064a\u0633 \u0627\u0644\u0646\u0648\u0645",
"\u0627\u0644\u0645\u0633\u0637\u0631\u0629 \u0627\u0644\u062d\u0627\u0633\u0628\u0629",
"\u0627\u0644\u0628\u0627\u0628 \u0627\u0644\u0645\u0646\u0632\u0644\u0642",
"\u0645\u0627\u0643\u064a\u0646\u0629 \u0627\u0644\u062d\u0638",
"\u0627\u0644\u063a\u0637\u0633 \u062a\u062d\u062a \u0627\u0644\u0645\u0627\u0621",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062c\u0644\u064a\u062f \u0627\u0644\u0622\u0644\u064a\u0629",
"\u0643\u0627\u0633\u062d\u0629 \u062b\u0644\u0648\u062c",
"\u0645\u0648\u0632\u0639 \u0627\u0644\u0635\u0627\u0628\u0648\u0646",
"\u0643\u0631\u0629 (\u0643\u0631\u0629 \u0627\u0644\u0642\u062f\u0645)",
"\u062c\u0648\u0631\u0628",
"\u0645\u062c\u0645\u0639 \u0627\u0644\u0637\u0627\u0642\u0629 \u0627\u0644\u0634\u0645\u0633\u064a\u0629 \u0627\u0644\u062d\u0631\u0627\u0631\u064a\u0629",
"\u0627\u0644\u0635\u064e\u0645\u0652\u0628\u0631\u0650\u064a\u0631\u0629",
"\u0648\u0639\u0627\u0621 \u0627\u0644\u0634\u0648\u0631\u0628\u0629",
"\u0645\u0641\u062a\u0627\u062d \u0627\u0644\u0645\u0633\u0627\u0641\u0629",
"\u0627\u0644\u0645\u062f\u0641\u0623\u0629",
"\u0645\u0643\u0648\u0643 \u0627\u0644\u0641\u0636\u0627\u0621",
"\u0627\u0644\u0645\u0650\u0644\u0648\u064e\u0642",
"\u0627\u0644\u0642\u0627\u0631\u0628 \u0627\u0644\u0633\u0631\u064a\u0639",
"\u0634\u0628\u0643\u0629 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a",
"\u062e\u0634\u0628\u0629 \u0627\u0644\u0645\u063a\u0632\u0644",
"\u0627\u0644\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0631\u064a\u0627\u0636\u064a\u0629",
"\u0628\u0642\u0639\u0629 \u0636\u0648\u0621",
"\u0633\u0637\u062d \u0627\u0644\u0645\u0633\u0631\u062d",
"\u0627\u0644\u0642\u0627\u0637\u0631\u0629 \u0627\u0644\u0628\u062e\u0627\u0631\u064a\u0629",
"\u062c\u0633\u0631 \u0645\u0642\u0648\u0633 \u0646\u0641\u0642\u064a",
"\u0627\u0644\u0637\u0628\u0644 \u0627\u0644\u0646\u062d\u0627\u0633\u064a",
"\u0627\u0644\u0633\u0645\u0627\u0639\u0629 \u0627\u0644\u0637\u0628\u064a\u0629",
"\u0627\u0644\u0644\u0650\u0641\u0627\u0639",
"\u0627\u0644\u062c\u062f\u0627\u0631 \u0627\u0644\u062c\u0627\u0641",
"\u0645\u0624\u0642\u062a",
"\u0627\u0644\u0645\u0648\u0642\u062f",
"\u0627\u0644\u063a\u0631\u0628\u0627\u0644",
"\u0627\u0644\u062a\u0631\u0627\u0645",
"\u0627\u0644\u0646\u0642\u0627\u0644\u0629",
"\u0627\u0644\u0623\u0631\u064a\u0643\u0629",
"\u0645\u0628\u0646\u0649 \u0633\u062a\u0648\u064a\u0627",
"\u0627\u0644\u063a\u0648\u0627\u0635\u0629",
"\u0627\u0644\u0628\u0630\u0644\u0629",
"\u0627\u0644\u0645\u0632\u0648\u0644\u0629",
"\u0627\u0644\u0646\u0638\u0627\u0631\u0629 \u0627\u0644\u0634\u0645\u0633\u064a\u0629",
"\u0627\u0644\u0646\u0638\u0627\u0631\u0629 \u0627\u0644\u0634\u0645\u0633\u064a\u0629",
"\u0627\u0644\u0648\u0627\u0642\u064a \u0627\u0644\u0634\u0645\u0633\u064a",
"\u0627\u0644\u062c\u0633\u0631 \u0627\u0644\u0645\u0639\u0644\u0642",
"\u0627\u0644\u0645\u0645\u0633\u062d\u0629",
"\u0627\u0644\u0642\u0645\u064a\u0635 \u0627\u0644\u062b\u0642\u064a\u0644",
"\u0627\u0644\u062a\u064f\u0628\u0651\u0627\u0646 \u0623\u0648 \u0627\u0644\u0628\u0646\u0637\u0627\u0644 \u0627\u0644\u0642\u0635\u064a\u0631\\",
"\u0627\u0644\u0623\u0631\u062c\u0648\u062d\u0629",
"\u0627\u0644\u0645\u0641\u062a\u0627\u062d \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0645\u062d\u0642\u0646\u0629 \u0623\u0648 \u0627\u0644\u0625\u0628\u0631\u0629",
"\u0627\u0644\u0623\u0628\u0627\u062c\u0648\u0631\u0629",
"\u0627\u0644\u062f\u0628\u0627\u0628\u0629",
"\u0645\u0633\u062c\u0644 \u0627\u0644\u0634\u0631\u064a\u0637 \u0627\u0644\u0635\u0648\u062a\u064a",
"\u0625\u0628\u0631\u064a\u0642 \u0627\u0644\u0634\u0627\u064a",
"\u0627\u0644\u062f\u0628\u062f\u0648\u0628",
"\u0627\u0644\u0631\u0627\u0626\u064a",
"\u0643\u0631\u0629 \u062a\u0646\u0633",
"\u0627\u0644\u062a\u0633\u0642\u064a\u0641 \u0628\u0627\u0644\u0642\u0634",
"\u0627\u0644\u0633\u062a\u0627\u0631\u0629 \u0627\u0644\u0645\u0633\u0631\u062d\u064a\u0629",
"\u0627\u0644\u0643\u064f\u0634\u0652\u062a\u0650\u0628\u064e\u0627\u0646",
"\u0627\u0644\u062f\u0631\u064e\u0651\u0627\u0633\u0629",
"\u0639\u0631\u0634",
"\u0628\u0644\u0627\u0637 \u0627\u0644\u0633\u0642\u0641",
"\u0622\u0644\u0629 \u062a\u062d\u0645\u064a\u0635 \u0627\u0644\u062e\u0628\u0632",
"\u0645\u062d\u0644\u0627\u062a \u0628\u064a\u0639 \u0644\u0648\u0627\u0632\u0645 \u0627\u0644\u062a\u062f\u062e\u064a\u0646",
"\u0645\u0642\u0639\u062f \u0627\u0644\u0645\u0631\u062d\u0627\u0636",
"\u0645\u0634\u0639\u0644\u0629",
"\u0627\u0644\u0623\u064e\u0639\u0652\u0645\u0650\u062f\u064e\u0629\u064f \u0627\u0644\u0637\u064e\u0651\u0648\u0652\u0637\u064e\u0645\u0650\u064a\u064e\u0651\u0629\u0650",
"\u0634\u0627\u062d\u0646\u0629 \u0627\u0644\u0642\u0637\u0631",
"\u0645\u062a\u062c\u0631 \u0627\u0644\u0623\u0644\u0639\u0627\u0628",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u062c\u0631\u0627\u0631",
"\u0634\u0627\u062d\u0646\u0629 \u0646\u0635\u0641 \u0645\u0642\u0637\u0648\u0631\u0629",
"\u0627\u0644\u0635\u064a\u0646\u064a\u0629",
"\u0645\u0639\u0637\u0641 \u0627\u0644\u062e\u0646\u062f\u0642",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u062b\u0644\u0627\u062b\u064a\u0629 \u0627\u0644\u0639\u062c\u0644\u0627\u062a",
"\u0642\u0627\u0631\u0628 \u0627\u0644\u062f\u0639\u0627\u0645\u0629 \u0627\u0644\u0645\u0632\u062f\u0648\u062c\u0629",
"\u062d\u0627\u0645\u0644 \u062b\u0644\u0627\u062b\u064a",
"\u0642\u0648\u0633 \u0627\u0644\u0646\u0635\u0631",
"\u0627\u0644\u062d\u0627\u0641\u0644\u0629 \u0633\u0637\u062d\u064a\u0629 \u0627\u0644\u062a\u0645\u062f\u064a\u062f \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0627\u0644\u062a\u0631\u0648\u0645\u0628\u0648\u0646",
"\u062d\u0648\u0636 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u0627\u0644\u0628\u0648\u0627\u0628\u0629 \u0627\u0644\u062f\u0648\u0627\u0631\u0629",
"\u0627\u0644\u0622\u0644\u0629 \u0627\u0644\u0643\u0627\u062a\u0628\u0629",
"\u0627\u0644\u0645\u0638\u0644\u0629",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u0623\u062d\u0627\u062f\u064a\u0629",
"\u0627\u0644\u0628\u064a\u0627\u0646\u0648 \u0627\u0644\u0642\u0627\u0626\u0645",
"\u0627\u0644\u0645\u0643\u0646\u0633\u0629 \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a\u0629",
"\u0627\u0644\u0645\u0632\u0647\u0631\u064a\u0629",
"\u0627\u0644\u0642\u0646\u0637\u0631\u0629",
"\u0627\u0644\u0642\u0645\u0627\u0634 \u0627\u0644\u0645\u062e\u0645\u0644\u064a",
"\u0622\u0644\u0629 \u0627\u0644\u0628\u064a\u0639 \u0627\u0644\u0630\u0627\u062a\u064a",
"\u0627\u0644\u0635\u062f\u0627\u0631\u064a",
"\u0642\u0646\u0637\u0631\u0629 \u0645\u062a\u0639\u062f\u062f\u0629 \u0627\u0644\u0631\u0643\u0627\u0626\u0632",
"\u0627\u0644\u0643\u0645\u0627\u0646",
"\u0627\u0644\u0643\u0631\u0629 \u0627\u0644\u0637\u0627\u0626\u0631\u0629",
"\u0635\u0627\u0646\u0639\u0629 \u0627\u0644\u0648\u0627\u0641\u0644",
"\u0633\u0627\u0639\u0629 \u0627\u0644\u062d\u0627\u0626\u0637",
"\u0645\u062d\u0641\u0638\u0629",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0635\u0648\u0627\u0646",
"\u0637\u0627\u0626\u0631\u0629 \u0639\u0633\u0643\u0631\u064a\u0629",
"\u0627\u0644\u0645\u062c\u0644\u0649",
"\u0627\u0644\u063a\u0633\u0627\u0644\u0629",
"\u0642\u0627\u0631\u0648\u0631\u0629 \u0645\u0627\u0621",
"\u0625\u0628\u0631\u064a\u0642 \u0627\u0644\u0645\u0627\u0621",
"\u0628\u0631\u062c \u0627\u0644\u0645\u064a\u0627\u0647",
"\u0625\u0628\u0631\u064a\u0642 \u0627\u0644\u0643\u062d\u0648\u0644\u064a\u0627\u062a",
"\u0627\u0644\u0635\u0627\u0641\u0631\u0629",
"\u0634\u0639\u0631 \u0645\u0633\u062a\u0639\u0627\u0631",
"\u0627\u0644\u0646\u0627\u0641\u0630\u0629 \u0627\u0644\u0648\u0627\u0642\u064a\u0629",
"\u0627\u0644\u0633\u062a\u0627\u0631 \u0627\u0644\u0644\u0641\u064e\u0651\u0627\u0641",
"\u0631\u0628\u0637\u0629 \u0639\u0646\u0642 \u0648\u0646\u062f\u0633\u0648\u0631 ",
"\u0632\u062c\u0627\u062c\u0629 \u0627\u0644\u0646\u0628\u064a\u0630",
"\u062c\u0646\u0627\u062d \u0627\u0644\u0637\u0627\u0626\u0631\u0629",
"\u0645\u0642\u0644\u0627\u0629 \u0635\u064a\u0646\u064a\u0629",
"\u0627\u0644\u0645\u0644\u0639\u0642\u0629 \u0627\u0644\u062e\u0634\u0628\u064a\u0629",
"\u0627\u0644\u0635\u0648\u0641",
"\u0627\u0644\u0633\u064a\u0627\u062c \u0627\u0644\u0645\u0646\u0642\u0633\u0645",
"\u062d\u0637\u0627\u0645 \u0627\u0644\u0633\u0641\u064a\u0646\u0629",
"\u0627\u0644\u0632\u0648\u0631\u0642 \u0627\u0644\u0634\u0631\u0627\u0639\u064a",
"\u0645\u0646\u0632\u0644 \u0627\u0644\u064a\u0648\u0631\u062a",
"\u0645\u0648\u0627\u0642\u0639 \u0627\u0644\u0648\u064a\u0628",
"\u0643\u062a\u0627\u0628 \u0631\u0633\u0648\u0645 \u0647\u0632\u0644\u064a\u0629",
"\u0627\u0644\u0643\u0644\u0645\u0627\u062a \u0627\u0644\u0645\u062a\u0642\u0627\u0637\u0639\u0629",
"\u0644\u0627\u0641\u062a\u0629 \u0645\u0631\u0648\u0631\u064a\u0629",
"\u0625\u0634\u0627\u0631\u0629 \u0627\u0644\u0645\u0631\u0648\u0631 \u0627\u0644\u0636\u0648\u0626\u064a\u0629",
"\u063a\u0644\u0627\u0641 \u0627\u0644\u062d\u0645\u0627\u064a\u0629 \u0644\u0644\u0643\u062a\u0627\u0628",
"\u0642\u0627\u0626\u0645\u0629 \u0637\u0639\u0627\u0645",
"\u0635\u062d\u0646",
"\u0642\u0646\u0628\u064a\u0637 \u0623\u062e\u0636\u0631",
"\u062d\u0633\u0627\u0621 \u0627\u0644\u0643\u0648\u0646\u0633\u0648\u0645\u064a\u0629",
"\u0627\u0644\u0648\u0639\u0627\u0621 \u0627\u0644\u0633\u0627\u062e\u0646",
"\u062a\u0631\u0627\u064a\u0641\u0644",
"\u0627\u0644\u0645\u062b\u0644\u062c\u0627\u062a",
"\u0627\u0644\u0645\u0635\u0627\u0635\u0629",
"\u0627\u0644\u062e\u0628\u0632 \u0627\u0644\u0641\u0631\u0646\u0633\u064a",
"\u062e\u0628\u0632 \u0627\u0644\u0628\u064a\u063a\u0644",
"\u0627\u0644\u0645\u062e\u0628\u0648\u0632\u0627\u062a \u0627\u0644\u0639\u064f\u0642\u0652\u062f\u0650\u064a\u064e\u0651\u0629",
"\u062a\u0634\u064a\u0632 \u0628\u0631\u062c\u0631",
"\u0627\u0644\u0646\u0642\u0627\u0646\u0642",
"\u0627\u0644\u0628\u0637\u0627\u0637\u0627 \u0627\u0644\u0645\u0647\u0631\u0648\u0633\u0629",
"\u0645\u0644\u0641\u0648\u0641",
"\u0627\u0644\u0642\u0631\u0646\u0628\u064a\u0637 \u0627\u0644\u0623\u062e\u0636\u0631",
"\u0627\u0644\u0642\u0631\u0646\u0628\u064a\u0637",
"\u0627\u0644\u0643\u0648\u0633\u0627",
"\u0645\u0639\u0643\u0631\u0648\u0646\u0629 \u0627\u0644\u0627\u0633\u0643\u0648\u0627\u0634",
"\u0642\u0631\u0639 \u0627\u0644\u0628\u0644\u0648\u0637",
"\u0642\u0631\u0639 \u0627\u0644\u062c\u0648\u0632",
"\u062e\u064a\u0627\u0631",
"\u0627\u0644\u062e\u0631\u0634\u0648\u0641 \u0627\u0644\u0634\u0648\u0643\u064a",
"\u0641\u0644\u0641\u0644 \u062d\u0644\u0648",
"\u0627\u0644\u062e\u0631\u0634\u0648\u0641 \u0627\u0644\u0633\u0643\u0648\u0644\u064a\u0645\u064a",
"\u0639\u064a\u0634 \u0627\u0644\u063a\u0631\u0627\u0628",
"\u062a\u0641\u0627\u062d \u0623\u062e\u0636\u0631",
"\u0627\u0644\u0641\u0631\u0627\u0648\u0644\u0629",
"\u0627\u0644\u0628\u0631\u062a\u0642\u0627\u0644",
"\u0627\u0644\u0644\u064a\u0645\u0648\u0646",
"\u0627\u0644\u062a\u064a\u0646",
"\u0627\u0644\u0623\u0646\u0627\u0646\u0627\u0633",
"\u0627\u0644\u0645\u0648\u0632",
"\u062c\u0627\u0643 \u0641\u0631\u0648\u062a",
"\u0642\u0634\u0637\u0629 \u0634\u0631\u064a\u0645\u0648\u0644\u064a\u0627",
"\u0627\u0644\u0631\u0645\u0627\u0646",
"\u0627\u0644\u062f\u0631\u064a\u0633",
"\u0643\u0631\u0628\u0646\u0627\u0631\u0629",
"\u0635\u0644\u0635\u0629 \u0627\u0644\u0634\u0648\u0643\u0648\u0644\u0627",
"\u0639\u062c\u064a\u0646\u0629 \u0627\u0644\u062e\u0628\u0632",
"\u0631\u063a\u064a\u0641 \u0627\u0644\u0644\u062d\u0645",
"\u0628\u064a\u062a\u0632\u0627",
"\u0641\u0637\u064a\u0631\u0629 \u0627\u0644\u0642\u062f\u0631",
"\u0628\u0648\u0631\u064a\u062a\u0648",
"\u0627\u0644\u0646\u0628\u064a\u0630 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0642\u0647\u0648\u0629 \u0625\u0633\u0628\u0631\u064a\u0633\u0648",
"\u0641\u0646\u062c\u0627\u0646 \u0627\u0644\u0634\u0627\u064a",
"\u062d\u0644\u064a\u0628 \u0627\u0644\u0628\u064a\u0636",
"\u0627\u0644\u062c\u0628\u0644",
"\u0641\u0642\u0627\u0639\u0629",
"\u0627\u0644\u062c\u0631\u0641",
"\u0627\u0644\u0634\u0639\u0627\u0628 \u0627\u0644\u0645\u0631\u062c\u0627\u0646\u064a\u0629",
"\u0627\u0644\u0641\u0648\u0627\u0631\u0629 \u0627\u0644\u062d\u0627\u0631\u0629",
"\u0627\u0644\u0636\u0641\u0629",
"\u0627\u0644\u0634\u0646\u062e\u0629",
"\u0627\u0644\u0645\u064a\u0627\u0647 \u0627\u0644\u0636\u062d\u0644\u0629",
"\u0627\u0644\u0634\u0627\u0637\u0626",
"\u0627\u0644\u0648\u0627\u062f\u064a",
"\u0628\u0631\u0643\u0627\u0646",
"\u0644\u0627\u0639\u0628 \u0643\u0631\u0629 \u0627\u0644\u0642\u0627\u0639\u062f\u0629",
"\u0627\u0644\u0639\u0631\u064a\u0633",
"\u0627\u0644\u063a\u0648\u0635 \u0628\u062c\u0647\u0627\u0632 \u0627\u0644\u062a\u0646\u0641\u0633 ",
"\u0627\u0644\u0633\u0644\u062c\u0645",
"\u0632\u0647\u0631\u0629 \u0627\u0644\u0644\u0624\u0644\u0624 ",
"\u062e\u0641 \u0627\u0644\u0633\u064a\u062f\u0629 \u0627\u0644\u0623\u0635\u0641\u0631",
"\u0627\u0644\u0630\u0631\u0629",
"\u0634\u062c\u0631\u0629 \u062b\u0645\u0631\u0629 \u0627\u0644\u0628\u0644\u0648\u0637",
"\u062b\u0645\u0631 \u0627\u0644\u0648\u0631\u062f \u0627\u0644\u0628\u0631\u064a",
"\u0628\u0630\u0648\u0631 \u0643\u0633\u062a\u0646\u0627\u0621 \u0627\u0644\u062d\u0635\u0627\u0646",
"\u0627\u0644\u0641\u0637\u0631\u064a\u0627\u062a \u0627\u0644\u0645\u0631\u062c\u0627\u0646\u064a\u0629",
"\u0641\u0637\u0631 \u063a\u0627\u0631\u064a\u0642\u0648\u0646",
"\u0641\u0637\u0631 \u062c\u0627\u0631\u0648\u0645\u064a\u062a\u0631\u0627 \u0627\u064a\u0633\u0643\u0644\u0646\u062a\u0627",
"\u0627\u0644\u0642\u0631\u0646 \u0627\u0644\u0646\u062a\u0646",
"\u0641\u0637\u0631 \u0646\u062c\u0645 \u0627\u0644\u0623\u0631\u0636",
"\u0641\u0637\u0631 \u0631\u0641 \u0627\u0644\u0643\u0628\u0631\u064a\u062a",
"\u0641\u0637\u0631 \u0627\u0644\u0628\u0648\u0644\u064a\u0637",
"\u0627\u0644\u0639\u0631\u0646\u0627\u0633",
"\u0648\u0631\u0642 \u0627\u0644\u0645\u0631\u062d\u0627\u0636"
]
}
clip_benchmark/clip_benchmark/datasets/ar_zeroshot_classification_templates.json 0 → 100644
View file @ 1d5a34cf
{
"imagenet1k": [
"{c}",
"\u0635\u0648\u0631\u0629 \u0633\u064a\u0626\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0633\u064a\u0626\u0629 \u062a\u062d\u062a\u0648\u064a \u0639\u0644\u0649 {c}",
"\u0646\u062d\u062a \u0644\u0634\u0643\u0644 {c}",
"\u0646\u062d\u062a \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0630\u0627\u062a \u062c\u0648\u0648\u062f\u0629 \u0645\u0646\u062e\u0641\u0636\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0630\u0627\u062a \u062c\u0648\u0648\u062f\u0629 \u0645\u0646\u062e\u0641\u0636\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0631\u0633\u0648\u0645\u0627\u062a \u062c\u062f\u0627\u0631\u064a\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0631\u0633\u0648\u0645\u0627\u062a \u062c\u062f\u0627\u0631\u064a\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0642\u062a\u0637\u0639\u0629 \u062a\u062d\u062a\u0648\u064a \u0639\u0644\u0649 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0642\u062a\u0637\u0639\u0629 \u0644\u0640 {c}",
"\u062a\u0637\u0631\u064a\u0632 {c} ",
" \u0635\u0648\u0631\u0629 \u064a\u0635\u0639\u0628 \u0641\u064a\u0647\u0627 \u0631\u0624\u064a\u0629 {c} ",
"\u0635\u0648\u0631\u0629 \u0633\u0627\u0637\u0639\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0648\u0627\u0636\u062d\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u062a\u0633\u062e\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0638\u0644\u0645\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0623\u0628\u064a\u0636 \u0648\u0623\u0633\u0648\u062f {c}",
"{c} \u0641\u064a \u0644\u0642\u0637\u0629 \u0642\u0631\u064a\u0628\u0629",
"\u0635\u0648\u0631\u0629 \u0631\u0627\u0626\u0639\u0629 \u0644\u0640 {c}",
"\u0644\u0642\u0637\u0629 \u0642\u0631\u064a\u0628\u0629 \u0644\u0640 {c}",
"\u0631\u0633\u0645 \u062d\u0627\u0633\u0648\u0628\u064a \u064a\u062d\u062a\u0648\u064a {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0631\u0633\u0648\u0645\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0631\u0633\u0645\u0629 \u0644\u0640 {c}",
"\u0631\u0633\u0645\u0629 {c}",
"\u0631\u0633\u0645 \u064a\u062d\u062a\u0648\u064a {c} ",
"\u0635\u0648\u0631\u0629 \u0628\u0646\u0645\u0637 \u0627\u0644\u0628\u0643\u0633\u0644 \u0644\u0640 {c}",
" \u0635\u0648\u0631\u0629 \u0633\u0627\u0637\u0639\u0629 {c}",
"\u0648\u0634\u0645 {c}",
"{c} \u0641\u064a \u0627\u0644\u0635\u0648\u0631\u0629",
"\u0635\u0648\u0631\u0629 \u0645\u062a\u0633\u062e\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0635\u0648\u0631\u0629 \u062a\u0627\u0644\u0641\u0629 {c}",
"\u0635\u0648\u0631\u0629 \u0636\u0628\u0627\u0628\u064a\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 {c}",
"\u0635\u0648\u0631\u0629 \u062c\u064a\u062f\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0644\u0640 {c}",
"\u062a\u0635\u064a\u064a\u0631 \u0644\u0640 {c}",
"{c} \u0639\u0644\u0649 \u0634\u0643\u0644 \u0631\u0633\u0645 \u062d\u0627\u0633\u0648\u0628\u064a \u062b\u0646\u0627\u0626\u064a \u0623\u0648 \u062b\u0644\u0627\u062b\u064a \u0627\u0644\u0623\u0628\u0639\u0627\u062f",
"\u064a\u0648\u062c\u062f {c} \u0648\u0627\u062d\u062f \u0641\u064a \u0627\u0644\u0635\u0648\u0631\u0629",
"\u0631\u0633\u0645 \u062d\u0627\u0633\u0648\u0628\u064a \u0644\u0640 {c}",
"\u0627\u0648\u0631\u064a\u063a\u0627\u0645\u064a \u0644\u0640 {c}",
"{c} \u0645\u0635\u0646\u0648\u0639 \u0639\u0646 \u0637\u0631\u064a\u0642 \u0641\u0646 \u0637\u064a \u0627\u0644\u0648\u0631\u0642",
"{c} \u0641\u064a \u0644\u0639\u0628\u0629 \u0641\u064a\u062f\u064a\u0648",
"{c} \u0645\u0648\u062c\u0648\u062f \u0641\u064a \u0644\u0639\u0628\u0629 \u0627\u0644\u0641\u064a\u062f\u064a\u0648",
"\u0631\u0633\u0645 \u062a\u0642\u0631\u064a\u0628\u064a \u0644\u0640 {c}",
"{c} \u0645\u0631\u0633\u0648\u0645 \u0628\u0627\u0644\u062e\u0631\u0627\u0628\u064a\u0634",
"\u0635\u0648\u0631\u0629 \u0628\u0641\u0646 \u0627\u0644\u062e\u0631\u0627\u0628\u064a\u0634 \u0644\u0640 {c}",
"\u0644\u0639\u0628\u0629 {c}",
"\u0635\u0648\u0631\u0629 \u064a\u0648\u062c\u062f \u0641\u064a\u0647\u0627 {c}",
"\u0631\u0633\u0648\u0645 \u0645\u062a\u062d\u0631\u0643\u0629 \u0644\u0640 {c} ",
"\u0635\u0648\u0631\u0629 \u0644\u0639\u062f\u062f \u0645\u0646 {c}",
"\u0635\u0648\u0631\u0629 \u064a\u0638\u0647\u0631 \u0641\u064a\u0647\u0627 {c}",
"\u0635\u0648\u0631\u0629 {c} \u0635\u063a\u064a\u0631 ",
"\u0635\u0648\u0631\u0629 {c} \u0643\u0628\u064a\u0631",
"{c} \u064a\u0638\u0647\u0631 \u0641\u064a \u0627\u0644\u0635\u0648\u0631\u0629"
]
}
clip_benchmark/clip_benchmark/datasets/birdsnap.py 0 → 100644
View file @ 1d5a34cf
import concurrent.futures
import csv
import hashlib
import os
from pathlib import Path
import torch
from PIL import Image as Image
from torchvision.datasets.utils import download_and_extract_archive
Image.LOAD_TRUNCATED_IMAGES = True
class Birdsnap(torch.utils.data.Dataset):
"""This is the BirdSnap dataset presented in
- Berg et al., "Birdsnap: Large-scale Fine-grained Visual Categorization of Birds"
It contains a lot of classes of birds and can be used as a replacement for ImageNet validation images
with similar image fidelity but less of the baggage, given that all subjects are in fact birds.
This is too small to train on though and hence not even partitioned into train/test.
Several images are missing from flickr (in 2021), these will be discarded automatically.
"""
METADATA_URL = 'http://thomasberg.org/datasets/birdsnap/1.1/birdsnap.tgz'
METADATA_ARCHIVE = 'birdsnap.tgz'
META_MD5 = '1788158175f6ae794aebf27bcd7a3f5d'
BASE_FOLDER = 'birdsnap'
def __init__(self, root, split='train', transform=None, target_transform=None, download=True, crop_to_bbx=False):
"""Init with split, transform, target_transform."""
self.root = os.path.expanduser(root)
self.transform = transform
self.target_transform = target_transform
self.crop_to_bbx = crop_to_bbx # Crop to dataset default bounding boxes
if download:
self.download()
if not self.check_integrity():
raise ValueError('Dataset Birdsnap not downloaded completely or possibly corrupted.')
# self._purge_missing_data()
def _check_integrity_of_metadata(self, chunk_size=8192):
"""This only checks if all files are there."""
try:
with open(os.path.join(self.root, self.METADATA_ARCHIVE), 'rb') as f:
archive_hash = hashlib.md5()
while chunk := f.read(chunk_size):
archive_hash.update(chunk)
return self.META_MD5 == archive_hash.hexdigest()
except FileNotFoundError:
return False
def check_integrity(self):
"""Full integrity check."""
if not self._check_integrity_of_metadata():
return False
else:
self._parse_metadata()
missing_images = 0
for idx, file in enumerate(self.meta):
if not self._verify_image(idx):
missing_images += 1
if missing_images > 0:
print(f'{missing_images} images could not be downloaded.')
return True
def download(self):
# Metadata:
if self._check_integrity_of_metadata():
print('Metadata already downloaded and verified')
else:
download_and_extract_archive(self.METADATA_URL, self.root, filename=self.METADATA_ARCHIVE)
# Actual files:
self._parse_metadata()
missing_ids = []
for idx, file in enumerate(self.meta):
if not self._verify_image(idx):
missing_ids.append(idx)
if len(missing_ids) > 0:
print(f'Downloading {len(missing_ids)} missing files now...')
self.scrape_images(missing_ids)
def __len__(self):
"""Return length via metainfo."""
return len(self.meta)
def __getitem__(self, index):
"""Return image, label."""
img = Image.open(self.paths[index])
if self.crop_to_bbx:
img = img.crop(
(
self.meta[index]['bb_x1'],
self.meta[index]['bb_y1'],
self.meta[index]['bb_x2'],
self.meta[index]['bb_y2'],
)
)
img = img.convert('RGB')
label = self.labels[index]
img = self.transform(img) if self.transform else img
label = self.target_transform(label) if self.target_transform else label
return img, label
def _parse_metadata(self):
"""Metadata keys are
dict_keys(['url', 'md5', 'path', 'species_id', 'bb_x1', 'bb_y1', 'bb_x2', 'bb_y2', 'back_x', 'back_y', 'beak_x',
'beak_y', 'belly_x', 'belly_y', 'breast_x', 'breast_y', 'crown_x', 'crown_y', 'forehead_x', 'forehead_y',
'left_cheek_x', 'left_cheek_y', 'left_eye_x', 'left_eye_y', 'left_leg_x', 'left_leg_y', 'left_wing_x',
'left_wing_y', 'nape_x', 'nape_y', 'right_cheek_x', 'right_cheek_y', 'right_eye_x', 'right_eye_y',
'right_leg_x', 'right_leg_y', 'right_wing_x', 'right_wing_y', 'tail_x', 'tail_y', 'throat_x', 'throat_y']
"""
with open(os.path.join(self.root, self.BASE_FOLDER, 'images.txt'), 'r') as f:
reader = csv.DictReader(f, delimiter='\t')
self.meta = list(reader) # List of dictionaries.
self.labels = [int(entry['species_id']) for entry in self.meta]
self.paths = [os.path.join(self.root, self.BASE_FOLDER, entry['path']) for entry in self.meta]
with open(os.path.join(self.root, self.BASE_FOLDER, 'species.txt'), 'r') as f:
reader = csv.DictReader(f, delimiter='\t')
self.classes_metadata = list(reader)
self.classes = [str(entry['common']) for entry in self.classes_metadata]
def _verify_image(self, idx):
try:
# Do this if you want to check in detail:
with open(os.path.join(self.root, self.BASE_FOLDER, self.meta[idx]['path']), 'rb') as fin:
return (hashlib.md5(fin.read()).hexdigest() == self.meta[idx]['md5'])
# In the mean time, just check if everything is there:
# return os.path.exists(os.path.join(self.root, self.BASE_FOLDER, self.meta[idx]["path"]))
except FileNotFoundError:
return False
def scrape_images(self, missing_ids, chunk_size=8196):
"""Scrape images using the python default ThreadPool example."""
import requests
def _load_url_and_save_image(idx, timeout):
full_path = os.path.join(self.root, self.BASE_FOLDER, self.meta[idx]['path'])
os.makedirs(os.path.split(full_path)[0], exist_ok=True)
r = requests.get(self.meta[idx]['url'], stream=True)
with open(full_path, 'wb') as write_file:
for chunk in r.iter_content(chunk_size=chunk_size):
write_file.write(chunk)
return True
# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=None) as executor: # Choose max_workers dynamically
# Start the load operations and mark each future with its URL
future_to_url = {
executor.submit(_load_url_and_save_image, idx, 600): self.meta[idx]['url'] for idx in missing_ids
}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
data = future.result()
except Exception as exc:
print(f'{url} generated exception: {exc}')
else:
print(f'{url} downloaded successfully.')
def _purge_missing_data(self):
"""Iterate over all data and throw out missing images."""
JPG = b'\xff\xd8\xff'
clean_meta = []
invalid_files = 0
for entry in self.meta:
full_path = os.path.join(self.root, self.BASE_FOLDER, entry['path'])
with open(full_path, 'rb') as file_handle:
if file_handle.read(3) == JPG:
clean_meta.append(entry)
else:
invalid_files += 1
print(f'Discarded {invalid_files} invalid files.')
self.meta = clean_meta
self.labels = [int(entry['species_id']) for entry in self.meta]
self.paths = [os.path.join(self.root, self.BASE_FOLDER, entry['path']) for entry in self.meta]
class BirdsnapV2(torch.utils.data.Dataset):
def __init__(self, root, split='test', transform=None, target_transform=None):
assert split == 'test', print('Only test split is supported for now.')
self.root_dir = Path(root)
self.image_dir = self.root_dir
self.test_elements = self.root_dir / 'test_images_valid.txt'
self.classes_file = self.root_dir / 'species.txt'
self.target_transform = target_transform
self.transform = transform
with open(self.test_elements, 'r') as f:
self.imgs = [line.rstrip('\n') for line in f][1:]
with open(self.classes_file, 'r') as f:
self.classes = [line.rstrip('\n').split('\t')[1] for line in f][1:]
with open(self.classes_file, 'r') as f:
self.dirs = [line.rstrip('\n').split('\t')[-1] for line in f][1:]
dir2id = {dir: idx for idx, dir in enumerate(self.dirs)}
self.labels = []
for image in self.imgs:
dir = image.split('/')[0].lower()
label = dir2id[dir]
self.labels.append(label)
self.imgs = [os.path.join(self.root_dir, item) for item in self.imgs]
def __len__(self):
"""Return length via metainfo."""
return len(self.imgs)
def __getitem__(self, index):
"""Return image, label."""
img = Image.open(self.imgs[index])
img = img.convert('RGB')
label = self.labels[index]
img = self.transform(img) if self.transform else img
label = self.target_transform(label) if self.target_transform else label
return img, label
if __name__ == '__main__':
from tqdm import tqdm
dataset = BirdsnapV2(root='/mnt/petrelfs/wangwenhai/workspace/InternVL2/benchmark/imagenet/birdsnap', split='test')
print(len(dataset.imgs))
for i in tqdm(range(len(dataset))):
try:
dataset.__getitem__(i)
print(dataset.imgs[i])
except:
print(dataset.imgs[i])
print(dataset.classes)
clip_benchmark/clip_benchmark/datasets/builder.py 0 → 100644
View file @ 1d5a34cf
import json
import os
import sys
import warnings
from subprocess import call
import torch
from torch.utils.data import default_collate
from torchvision.datasets import (CIFAR10, CIFAR100, DTD, GTSRB, MNIST, PCAM,
STL10, SUN397, CocoCaptions, Country211,
EuroSAT, FGVCAircraft, Flowers102, Food101,
ImageFolder, ImageNet, OxfordIIITPet,
RenderedSST2, StanfordCars)
from . import caltech101, flickr, imagenetv2, objectnet, voc2007
from .birdsnap import BirdsnapV2
from .tools import pre_caption
def _load_classnames_and_classification_templates(dataset_name, current_folder, language):
with open(os.path.join(current_folder, language + '_classnames.json'), 'r') as f:
classnames = json.load(f)
# Zero-shot classification templates, collected from a bunch of sources
# - CLIP paper (https://github.com/openai/CLIP/blob/main/data/prompts.md)
# - Lit Paper (https://arxiv.org/pdf/2111.07991.pdf)
# - SLIP paper (https://github.com/facebookresearch/SLIP/blob/main/templates.json)
# Some are fixed mnaually
with open(os.path.join(current_folder, language + '_zeroshot_classification_templates.json'), 'r') as f:
zeroshot_classification_templates = json.load(f)
# default template to use when the dataset name does not belong to `zeroshot_classification_templates`
DEFAULT_ZEROSHOT_CLASSIFICATION_TEMPLATES = zeroshot_classification_templates['imagenet1k']
if dataset_name.startswith('tfds/') or dataset_name.startswith('vtab/') or dataset_name.startswith('wds/'):
name = dataset_name.split('/')[-1]
else:
name = dataset_name
templates = zeroshot_classification_templates.get(name, DEFAULT_ZEROSHOT_CLASSIFICATION_TEMPLATES)
return classnames, templates
def build_dataset(dataset_name, root='root', transform=None, split='test', download=True, annotation_file=None,
language='en', task='zeroshot_classification', cupl=False, wds_cache_dir=None, **kwargs):
"""
Main function to use in order to build a dataset instance,
dataset_name: str
name of the dataset
root: str
root folder where the dataset is downloaded and stored. can be shared among datasets.
transform: torchvision transform applied to images
split: str
split to use, depending on the dataset can have different options.
In general, `train` and `test` are available.
For specific splits, please look at the corresponding dataset.
annotation_file: str or None
only for datasets with captions (used for retrieval) such as COCO
and Flickr.
"""
current_folder = os.path.dirname(__file__)
if task in ('zeroshot_classification', 'linear_probe'): # Only load templates and classnames if we have to
classnames, templates = _load_classnames_and_classification_templates(dataset_name, current_folder, language)
else:
classnames, templates = None, None
with open(os.path.join(current_folder, 'cupl_prompts.json'), 'r') as f:
cupl_prompts = json.load(f)
templates_cupl = None
train = (split == 'train')
if dataset_name == 'cifar10':
ds = CIFAR10(root=root, train=train, transform=transform, download=download, **kwargs)
elif dataset_name == 'cifar100':
ds = CIFAR100(root=root, train=train, transform=transform, download=download, **kwargs)
elif dataset_name == 'imagenet1k':
if not os.path.exists(root):
os.makedirs(root, exist_ok=True)
call(
f'wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_devkit_t12.tar.gz --output-document={root}/ILSVRC2012_devkit_t12.tar.gz',
shell=True)
call(
f'wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar --output-document={root}/ILSVRC2012_img_val.tar',
shell=True)
ds = ImageNet(root=root, split='train' if train else 'val', transform=transform, **kwargs)
# use classnames from OpenAI
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenet1k-unverified':
split = 'train' if train else 'val'
ds = ImageFolder(root=os.path.join(root, split), transform=transform, **kwargs)
# use classnames from OpenAI
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenetv2':
assert split == 'test', f'Only test split available for {dataset_name}'
os.makedirs(root, exist_ok=True)
ds = imagenetv2.ImageNetV2Dataset(variant='matched-frequency', transform=transform, location=root)
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenet_sketch':
assert split == 'test', f'Only test split available for {dataset_name}'
# Downloadable from https://drive.google.com/open?id=1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA
if not os.path.exists(root):
# Automatic download
print('Downloading imagenet_sketch...')
if not has_gdown():
print('GDown is needed to download the dataset. Please install it via `pip install gdown`')
sys.exit(1)
# Download ImageNet-Sketch.zip
call('gdown --id 1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA', shell=True)
assert os.path.exists('ImageNet-Sketch.zip')
# Unzip and move to `root`
call('unzip ImageNet-Sketch.zip', shell=True)
call(f'mv sketch {root}', shell=True)
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenet-a':
assert split == 'test', f'Only test split available for {dataset_name}'
# Downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar
if not os.path.exists(root):
print('Downloading imagenet-a...')
call('wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar', shell=True)
# Untar and move to `root`
call('tar xvf imagenet-a.tar', shell=True)
call(f'mv imagenet-a {root}', shell=True)
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
imagenet_a_wnids = ['n01498041', 'n01531178', 'n01534433', 'n01558993', 'n01580077', 'n01614925', 'n01616318',
'n01631663', 'n01641577', 'n01669191', 'n01677366', 'n01687978', 'n01694178', 'n01698640',
'n01735189', 'n01770081', 'n01770393', 'n01774750', 'n01784675', 'n01819313', 'n01820546',
'n01833805', 'n01843383', 'n01847000', 'n01855672', 'n01882714', 'n01910747', 'n01914609',
'n01924916', 'n01944390', 'n01985128', 'n01986214', 'n02007558', 'n02009912', 'n02037110',
'n02051845', 'n02077923', 'n02085620', 'n02099601', 'n02106550', 'n02106662', 'n02110958',
'n02119022', 'n02123394', 'n02127052', 'n02129165', 'n02133161', 'n02137549', 'n02165456',
'n02174001', 'n02177972', 'n02190166', 'n02206856', 'n02219486', 'n02226429', 'n02231487',
'n02233338', 'n02236044', 'n02259212', 'n02268443', 'n02279972', 'n02280649', 'n02281787',
'n02317335', 'n02325366', 'n02346627', 'n02356798', 'n02361337', 'n02410509', 'n02445715',
'n02454379', 'n02486410', 'n02492035', 'n02504458', 'n02655020', 'n02669723', 'n02672831',
'n02676566', 'n02690373', 'n02701002', 'n02730930', 'n02777292', 'n02782093', 'n02787622',
'n02793495', 'n02797295', 'n02802426', 'n02814860', 'n02815834', 'n02837789', 'n02879718',
'n02883205', 'n02895154', 'n02906734', 'n02948072', 'n02951358', 'n02980441', 'n02992211',
'n02999410', 'n03014705', 'n03026506', 'n03124043', 'n03125729', 'n03187595', 'n03196217',
'n03223299', 'n03250847', 'n03255030', 'n03291819', 'n03325584', 'n03355925', 'n03384352',
'n03388043', 'n03417042', 'n03443371', 'n03444034', 'n03445924', 'n03452741', 'n03483316',
'n03584829', 'n03590841', 'n03594945', 'n03617480', 'n03666591', 'n03670208', 'n03717622',
'n03720891', 'n03721384', 'n03724870', 'n03775071', 'n03788195', 'n03804744', 'n03837869',
'n03840681', 'n03854065', 'n03888257', 'n03891332', 'n03935335', 'n03982430', 'n04019541',
'n04033901', 'n04039381', 'n04067472', 'n04086273', 'n04099969', 'n04118538', 'n04131690',
'n04133789', 'n04141076', 'n04146614', 'n04147183', 'n04179913', 'n04208210', 'n04235860',
'n04252077', 'n04252225', 'n04254120', 'n04270147', 'n04275548', 'n04310018', 'n04317175',
'n04344873', 'n04347754', 'n04355338', 'n04366367', 'n04376876', 'n04389033', 'n04399382',
'n04442312', 'n04456115', 'n04482393', 'n04507155', 'n04509417', 'n04532670', 'n04540053',
'n04554684', 'n04562935', 'n04591713', 'n04606251', 'n07583066', 'n07695742', 'n07697313',
'n07697537', 'n07714990', 'n07718472', 'n07720875', 'n07734744', 'n07749582', 'n07753592',
'n07760859', 'n07768694', 'n07831146', 'n09229709', 'n09246464', 'n09472597', 'n09835506',
'n11879895', 'n12057211', 'n12144580', 'n12267677']
imagenet_a_mask = [wnid in set(imagenet_a_wnids) for wnid in all_imagenet_wordnet_ids]
ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_a_mask) if mask]
elif dataset_name == 'imagenet-r':
assert split == 'test', f'Only test split available for {dataset_name}'
# downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar
if not os.path.exists(root):
print('Downloading imagenet-r...')
call('wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar', shell=True)
# Untar and move to `root`
call('tar xvf imagenet-r.tar', shell=True)
call(f'mv imagenet-r {root}', shell=True)
imagenet_r_wnids = {'n01443537', 'n01484850', 'n01494475', 'n01498041', 'n01514859', 'n01518878', 'n01531178',
'n01534433', 'n01614925', 'n01616318', 'n01630670', 'n01632777', 'n01644373', 'n01677366',
'n01694178', 'n01748264', 'n01770393', 'n01774750', 'n01784675', 'n01806143', 'n01820546',
'n01833805', 'n01843383', 'n01847000', 'n01855672', 'n01860187', 'n01882714', 'n01910747',
'n01944390', 'n01983481', 'n01986214', 'n02007558', 'n02009912', 'n02051845', 'n02056570',
'n02066245', 'n02071294', 'n02077923', 'n02085620', 'n02086240', 'n02088094', 'n02088238',
'n02088364', 'n02088466', 'n02091032', 'n02091134', 'n02092339', 'n02094433', 'n02096585',
'n02097298', 'n02098286', 'n02099601', 'n02099712', 'n02102318', 'n02106030', 'n02106166',
'n02106550', 'n02106662', 'n02108089', 'n02108915', 'n02109525', 'n02110185', 'n02110341',
'n02110958', 'n02112018', 'n02112137', 'n02113023', 'n02113624', 'n02113799', 'n02114367',
'n02117135', 'n02119022', 'n02123045', 'n02128385', 'n02128757', 'n02129165', 'n02129604',
'n02130308', 'n02134084', 'n02138441', 'n02165456', 'n02190166', 'n02206856', 'n02219486',
'n02226429', 'n02233338', 'n02236044', 'n02268443', 'n02279972', 'n02317335', 'n02325366',
'n02346627', 'n02356798', 'n02363005', 'n02364673', 'n02391049', 'n02395406', 'n02398521',
'n02410509', 'n02423022', 'n02437616', 'n02445715', 'n02447366', 'n02480495', 'n02480855',
'n02481823', 'n02483362', 'n02486410', 'n02510455', 'n02526121', 'n02607072', 'n02655020',
'n02672831', 'n02701002', 'n02749479', 'n02769748', 'n02793495', 'n02797295', 'n02802426',
'n02808440', 'n02814860', 'n02823750', 'n02841315', 'n02843684', 'n02883205', 'n02906734',
'n02909870', 'n02939185', 'n02948072', 'n02950826', 'n02951358', 'n02966193', 'n02980441',
'n02992529', 'n03124170', 'n03272010', 'n03345487', 'n03372029', 'n03424325', 'n03452741',
'n03467068', 'n03481172', 'n03494278', 'n03495258', 'n03498962', 'n03594945', 'n03602883',
'n03630383', 'n03649909', 'n03676483', 'n03710193', 'n03773504', 'n03775071', 'n03888257',
'n03930630', 'n03947888', 'n04086273', 'n04118538', 'n04133789', 'n04141076', 'n04146614',
'n04147183', 'n04192698', 'n04254680', 'n04266014', 'n04275548', 'n04310018', 'n04325704',
'n04347754', 'n04389033', 'n04409515', 'n04465501', 'n04487394', 'n04522168', 'n04536866',
'n04552348', 'n04591713', 'n07614500', 'n07693725', 'n07695742', 'n07697313', 'n07697537',
'n07714571', 'n07714990', 'n07718472', 'n07720875', 'n07734744', 'n07742313', 'n07745940',
'n07749582', 'n07753275', 'n07753592', 'n07768694', 'n07873807', 'n07880968', 'n07920052',
'n09472597', 'n09835506', 'n10565667', 'n12267677'}
imagenet_r_mask = [wnid in imagenet_r_wnids for wnid in all_imagenet_wordnet_ids]
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_r_mask) if mask]
elif dataset_name == 'imagenet-o':
assert split == 'test', f'Only test split available for {dataset_name}'
# downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-o.tar
if not os.path.exists(root):
print('Downloading imagenet-o...')
call('wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-o.tar', shell=True)
# Untar and move to `root`
call('tar xvf imagenet-o.tar', shell=True)
call(f'mv imagenet-o {root}', shell=True)
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
imagenet_o_wnids = ['n01443537', 'n01704323', 'n01770081', 'n01784675', 'n01819313', 'n01820546', 'n01910747',
'n01917289', 'n01968897', 'n02074367', 'n02317335', 'n02319095', 'n02395406', 'n02454379',
'n02606052', 'n02655020', 'n02666196', 'n02672831', 'n02730930', 'n02777292', 'n02783161',
'n02786058', 'n02787622', 'n02791270', 'n02808304', 'n02817516', 'n02841315', 'n02865351',
'n02877765', 'n02892767', 'n02906734', 'n02910353', 'n02916936', 'n02948072', 'n02965783',
'n03000134', 'n03000684', 'n03017168', 'n03026506', 'n03032252', 'n03075370', 'n03109150',
'n03126707', 'n03134739', 'n03160309', 'n03196217', 'n03207743', 'n03218198', 'n03223299',
'n03240683', 'n03271574', 'n03291819', 'n03297495', 'n03314780', 'n03325584', 'n03344393',
'n03347037', 'n03372029', 'n03376595', 'n03388043', 'n03388183', 'n03400231', 'n03445777',
'n03457902', 'n03467068', 'n03482405', 'n03483316', 'n03494278', 'n03530642', 'n03544143',
'n03584829', 'n03590841', 'n03598930', 'n03602883', 'n03649909', 'n03661043', 'n03666591',
'n03676483', 'n03692522', 'n03706229', 'n03717622', 'n03720891', 'n03721384', 'n03724870',
'n03729826', 'n03733131', 'n03733281', 'n03742115', 'n03786901', 'n03788365', 'n03794056',
'n03804744', 'n03814639', 'n03814906', 'n03825788', 'n03840681', 'n03843555', 'n03854065',
'n03857828', 'n03868863', 'n03874293', 'n03884397', 'n03891251', 'n03908714', 'n03920288',
'n03929660', 'n03930313', 'n03937543', 'n03942813', 'n03944341', 'n03961711', 'n03970156',
'n03982430', 'n03991062', 'n03995372', 'n03998194', 'n04005630', 'n04023962', 'n04033901',
'n04040759', 'n04067472', 'n04074963', 'n04116512', 'n04118776', 'n04125021', 'n04127249',
'n04131690', 'n04141975', 'n04153751', 'n04154565', 'n04201297', 'n04204347', 'n04209133',
'n04209239', 'n04228054', 'n04235860', 'n04243546', 'n04252077', 'n04254120', 'n04258138',
'n04265275', 'n04270147', 'n04275548', 'n04330267', 'n04332243', 'n04336792', 'n04347754',
'n04371430', 'n04371774', 'n04372370', 'n04376876', 'n04409515', 'n04417672', 'n04418357',
'n04423845', 'n04429376', 'n04435653', 'n04442312', 'n04482393', 'n04501370', 'n04507155',
'n04525305', 'n04542943', 'n04554684', 'n04557648', 'n04562935', 'n04579432', 'n04591157',
'n04597913', 'n04599235', 'n06785654', 'n06874185', 'n07615774', 'n07693725', 'n07695742',
'n07697537', 'n07711569', 'n07714990', 'n07715103', 'n07716358', 'n07717410', 'n07718472',
'n07720875', 'n07742313', 'n07745940', 'n07747607', 'n07749582', 'n07753275', 'n07753592',
'n07754684', 'n07768694', 'n07836838', 'n07871810', 'n07873807', 'n07880968', 'n09229709',
'n09472597', 'n12144580', 'n12267677', 'n13052670']
imagenet_o_mask = [wnid in set(imagenet_o_wnids) for wnid in all_imagenet_wordnet_ids]
ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_o_mask) if mask]
elif dataset_name == 'objectnet':
assert split == 'test', f'Only test split available for {dataset_name}'
# downloadable from https://objectnet.dev/downloads/objectnet-1.0.zip or https://www.dropbox.com/s/raw/cxeztdtm16nzvuw/objectnet-1.0.zip
if not os.path.exists(root):
print('Downloading objectnet...')
call('wget https://objectnet.dev/downloads/objectnet-1.0.zip', shell=True)
# Untar and move to `root`
call('UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE unzip -P objectnetisatestset objectnet-1.0.zip', shell=True)
os.makedirs(root)
call(f'mv objectnet-1.0 {root}', shell=True)
call(f'cp {root}/objectnet-1.0/mappings/* {root}', shell=True)
ds = objectnet.ObjectNetDataset(root=root, transform=transform)
elif dataset_name == 'voc2007':
ds = voc2007.PASCALVoc2007Cropped(root=root, set='train' if train else 'test', transform=transform,
download=download, **kwargs)
elif dataset_name == 'voc2007_multilabel':
ds = voc2007.PASCALVoc2007(root=root, set='train' if train else 'test', transform=transform, download=download,
**kwargs)
elif dataset_name == 'mscoco_captions':
# https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations)
if split == 'train':
archive_name = 'train2014.zip'
elif split in ('val', 'test'):
archive_name = 'val2014.zip'
else:
raise ValueError(f'split should be train or val or test for `{dataset_name}`')
root_split = os.path.join(root, archive_name.replace('.zip', ''))
if not os.path.exists(root_split):
print(f'Downloading mscoco_captions {archive_name}...')
if not os.path.exists(os.path.join(root, archive_name)):
call(f'wget http://images.cocodataset.org/zips/{archive_name} --output-document={root}/{archive_name}',
shell=True)
call(f'unzip {root}/{archive_name} -d {root}', shell=True)
if not annotation_file:
annotation_file = f'{root}/coco_{split}_karpathy.json'
if language == 'cn':
annotation_file = f'{root}/coco-cn_{split}.json'
root_split = root
print(annotation_file)
if not os.path.exists(annotation_file):
call(
f'wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/coco_{split}_karpathy.json --output-document={annotation_file}',
shell=True)
ds = CocoCaptions(root=root_split, annFile=annotation_file, transform=transform,
target_transform=pre_caption, **kwargs)
elif dataset_name == 'multilingual_mscoco_captions':
from clip_benchmark.datasets import multilingual_mscoco
if (language not in multilingual_mscoco.SUPPORTED_LANGUAGES):
raise ValueError('Unsupported language for multilingual_ms_coco:', language)
def get_archive_name(target_split):
if target_split == 'train':
return 'train2014.zip'
elif target_split in ('val', 'test'):
return 'val2014.zip'
else:
raise ValueError(f'split should be train or val or test for `{dataset_name}`')
def download_mscoco_split(target_split):
archive_name = get_archive_name(target_split)
root_split = os.path.join(root, archive_name.replace('.zip', ''))
if not os.path.exists(root_split):
print(f'Downloading mscoco_captions {archive_name}...')
if not os.path.exists(os.path.join(root, archive_name)):
call(
f'wget http://images.cocodataset.org/zips/{archive_name} --output-document={root}/{archive_name}',
shell=True)
call(f'unzip {root}/{archive_name} -d {root}', shell=True)
# The multilingual MS-COCO uses images from various splits
for target_split in ['train', 'val', 'test']:
download_mscoco_split(target_split)
annotation_file = os.path.join(root, multilingual_mscoco.CAPTIONS_FILE_NAME.format(language))
# if (os.path.exists(annotation_file) == False):
multilingual_mscoco.create_annotation_file(root, language)
ds = multilingual_mscoco.Multilingual_MSCOCO(root=root, ann_file=annotation_file, transform=transform, **kwargs)
elif dataset_name == 'flickr30k':
# downloadable from https://www.kaggle.com/datasets/adityajn105/flickr30k
# https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations)
# `kaggle datasets download -d adityajn105/flickr30k`
if not os.path.exists(root):
# Automatic download
print('Downloading flickr30k...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
call('kaggle datasets download -d adityajn105/flickr30k', shell=True)
call(f'unzip flickr30k.zip', shell=True)
call(f'mv Images {root}', shell=True)
call(f'mv captions.txt {root}', shell=True)
if not annotation_file:
annotation_file = f'{root}/flickr30k_{split}_karpathy.txt'
if not os.path.exists(annotation_file):
# Download Flickr30K Karpathy test set
annotation_file = f'{root}/flickr30k_{split}_karpathy.txt'
call(
f'wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr30k_{split}_karpathy.txt --output-document={annotation_file}',
shell=True)
if language == 'cn':
annotation_file = f'{root}/flickr30k_cn_{split}.txt'
print(annotation_file)
ds = flickr.Flickr(root=f'{root}/Images', ann_file=annotation_file, transform=transform,
target_transform=pre_caption, **kwargs)
elif dataset_name == 'flickr8k':
# downloadable from https://www.kaggle.com/datasets/adityajn105/flickr8k
# `kaggle datasets download -d adityajn105/flickr8k`
# https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations)
if not os.path.exists(root):
# Automatic download
print('Downloading flickr8k...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
call('kaggle datasets download -d adityajn105/flickr8k', shell=True)
call(f'unzip flickr8k.zip', shell=True)
call(f'mv Images {root}', shell=True)
call(f'mv captions.txt {root}', shell=True)
if not annotation_file:
annotation_file = f'{root}/flickr8k_{split}_karpathy.txt'
if not os.path.exists(annotation_file):
# Download Flickr8K Karpathy test set
annotation_file = f'{root}/flickr8k_{split}_karpathy.txt'
call(
f'wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr8k_{split}_karpathy.txt --output-document={annotation_file}',
shell=True)
ds = flickr.Flickr(root=root, ann_file=annotation_file, transform=transform, **kwargs)
elif dataset_name == 'food101':
ds = Food101(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
# we use the default class names, we just replace "_" by spaces
# to delimit words
ds.classes = [cl.replace('_', ' ') for cl in ds.classes]
elif dataset_name == 'sun397':
warnings.warn(
f'split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset')
# we use the default class names, we just replace "_" and "/" by spaces
# to delimit words
ds = SUN397(root=root, transform=transform, download=download, **kwargs)
ds.classes = [cl.replace('_', ' ').replace('/', ' ') for cl in ds.classes]
elif dataset_name == 'cars':
ds = StanfordCars(root=root, split='train' if train else 'test', transform=transform, download=download,
**kwargs)
elif dataset_name == 'fgvc_aircraft':
ds = FGVCAircraft(root=root, annotation_level='variant', split='train' if train else 'test',
transform=transform, download=download, **kwargs)
elif dataset_name == 'dtd':
ds = DTD(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
elif dataset_name == 'pets':
ds = OxfordIIITPet(root=root, split='train' if train else 'test', target_types='category', transform=transform,
download=download, **kwargs)
elif dataset_name == 'caltech101':
warnings.warn(
f'split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset')
# broken download link (can't download google drive), fixed by this PR https://github.com/pytorch/vision/pull/5645
# also available in "vtab/caltech101" using VTAB splits, we advice to use VTAB version rather than this one
# since in this one (torchvision) there are no pre-defined test splits
ds = caltech101.Caltech101(root=root, target_type='category', transform=transform, download=download, **kwargs)
ds.classes = classnames['caltech101']
elif dataset_name == 'flowers':
ds = Flowers102(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
# class indices started by 1 until it was fixed in a PR (#TODO link of the PR)
# if older torchvision version, fix it using a target transform that decrements label index
# TODO figure out minimal torchvision version needed instead of decrementing
if ds[0][1] == 1:
ds.target_transform = lambda y: y - 1
ds.classes = classnames['flowers']
elif dataset_name == 'birdsnap':
ds = BirdsnapV2(root=root, split='train' if train else 'test', transform=transform, **kwargs)
# ds.classes = ds.classes
elif dataset_name == 'mnist':
ds = MNIST(root=root, train=train, transform=transform, download=download, **kwargs)
ds.classes = classnames['mnist']
elif dataset_name == 'stl10':
ds = STL10(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
elif dataset_name == 'eurosat':
warnings.warn(
f'split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset')
ds = EuroSAT(root=root, transform=transform, download=download, **kwargs)
ds.classes = classnames['eurosat']
elif dataset_name == 'gtsrb':
ds = GTSRB(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
ds.classes = classnames['gtsrb']
elif dataset_name == 'country211':
ds = Country211(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
ds.classes = classnames['country211']
elif dataset_name == 'pcam':
# Dead link. Fixed by this PR on torchvision https://github.com/pytorch/vision/pull/5645
# TODO figure out minimal torchvision version needed
ds = PCAM(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
ds.classes = classnames['pcam']
elif dataset_name == 'renderedsst2':
ds = RenderedSST2(root=root, split='train' if train else 'test', transform=transform, download=download,
**kwargs)
elif dataset_name == 'fer2013':
# Downloadable from https://www.kaggle.com/datasets/msambare/fer2013
# `kaggle datasets download -d msambare/fer2013`
if not os.path.exists(root):
# Automatic download
print('Downloading fer2013...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
call('kaggle datasets download -d msambare/fer2013', shell=True)
call(f'unzip fer2013.zip -d {root}', shell=True)
root = os.path.join(root, 'train' if train else 'test')
ds = ImageFolder(root=root, transform=transform)
ds.classes = classnames['fer2013']
elif dataset_name.startswith('tfds/'):
# TFDS datasets support using `timm` and `tensorflow_datasets`
prefix, *name_list = dataset_name.split('/')
name = '/'.join(name_list)
ds = build_tfds_dataset(name, download=download, split=split, data_dir=root, transform=transform)
elif dataset_name.startswith('vtab/'):
# VTAB datasets support using `tensorflow_datasets` and `task_adaptation`
prefix, *name_list = dataset_name.split('/')
name = '/'.join(name_list)
ds = build_vtab_dataset(name, download=download, split=split, data_dir=root, transform=transform,
classnames=classnames)
elif dataset_name.startswith('wds/'):
# WebDataset support using `webdataset` library
name = dataset_name.split('/', 1)[1]
ds = build_wds_dataset(name, transform=transform, split=split, data_dir=root, cache_dir=wds_cache_dir)
return ds
elif dataset_name == 'dummy':
ds = Dummy()
else:
raise ValueError(f'Unsupported dataset: {dataset_name}.')
if cupl:
ds.templates = templates_cupl
else:
ds.templates = templates
return ds
class Dummy():
def __init__(self):
self.classes = ['blank image', 'noisy image']
def __getitem__(self, i):
return torch.zeros(3, 224, 224), 0
def __len__(self):
return 1
def get_dataset_default_task(dataset):
if dataset in ('flickr30k', 'flickr8k', 'mscoco_captions', 'multilingual_mscoco_captions'):
return 'zeroshot_retrieval'
else:
return 'zeroshot_classification'
def get_dataset_collate_fn(dataset_name):
if dataset_name in ('mscoco_captions', 'multilingual_mscoco_captions', 'flickr30k', 'flickr8k'):
return image_captions_collate_fn
else:
return default_collate
def has_gdown():
return call('which gdown', shell=True) == 0
def has_kaggle():
return call('which kaggle', shell=True) == 0
def build_vtab_dataset(dataset_name, transform, download=True, split='test', data_dir='root', classnames=[]):
# Using VTAB splits instead of default TFDS splits
from .tfds import (VTABIterableDataset, disable_gpus_on_tensorflow,
download_tfds_dataset)
# avoid Tensorflow owning GPUs to not clash with PyTorch
disable_gpus_on_tensorflow()
# by default we take classes from TFDS (default behavior if `classes` stays None),
# except for the datasets that will override `classes` (e.g., clevr_*)
classes = None
if dataset_name == 'caltech101':
from task_adaptation.data.caltech import Caltech101
tfds_dataset = Caltech101(data_dir=data_dir)
classes = classnames['caltech101_vtab']
elif dataset_name == 'cars':
from task_adaptation.data.cars import CarsData
tfds_dataset = CarsData(data_dir=data_dir)
elif dataset_name in ('cifar10', 'cifar100'):
from task_adaptation.data.cifar import CifarData
tfds_dataset = CifarData(data_dir=data_dir, num_classes=10 if dataset_name == 'cifar10' else 100)
elif dataset_name.startswith('clevr_'):
from task_adaptation.data.clevr import CLEVRData
task = _extract_task(dataset_name)
assert task in ('count_all', 'closest_object_distance')
tfds_dataset = CLEVRData(task=task, data_dir=data_dir)
if task == 'count_all':
classes = classnames['clevr_count_all']
elif task == 'closest_object_distance':
classes = classnames['clevr_closest_object_distance']
else:
raise ValueError(f'non supported: {task}')
elif dataset_name == 'cub':
from task_adaptation.data.cub import CUB2011Data
tfds_dataset = CUB2011Data(data_dir=data_dir)
elif dataset_name == 'diabetic_retinopathy':
# Needs manual download from Kaggle
# 1) `kaggle competitions download -c diabetic-retinopathy-detection` on $ROOT/downloads/manual
# 2) extract archives on $ROOT/downloads/manual
if not os.path.exists(data_dir):
# Automatic download
print('Downloading diabetic_retinopathy...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
os.makedirs(os.path.join(data_dir, 'downloads', 'manual'))
call(f'kaggle competitions download -c diabetic-retinopathy-detection -p {data_dir}/downloads/manual',
shell=True)
call(
f'cd {data_dir}/downloads/manual;unzip diabetic-retinopathy-detection.zip;cat train.zip*>train.zip;cat test.zip*>test.zip;unzip train.zip; unzip test.zip;unzip sample.zip;unzip trainLabels.csv.zip',
shell=True)
from task_adaptation.data.diabetic_retinopathy import RetinopathyData
tfds_dataset = RetinopathyData(config='btgraham-300', data_dir=data_dir)
classes = classnames['diabetic_retinopathy']
elif dataset_name == 'dmlab':
from task_adaptation.data.dmlab import DmlabData
download_tfds_dataset('dmlab',
data_dir=data_dir) # it's not called in the original VTAB code, so we do it explictly
tfds_dataset = DmlabData(data_dir=data_dir)
classes = classnames['dmlab']
elif dataset_name.startswith('dsprites_'):
from task_adaptation.data.dsprites import DSpritesData
task = _extract_task(dataset_name)
assert task in ('label_shape', 'label_scale', 'label_orientation', 'label_x_position', 'label_y_position')
tfds_dataset = DSpritesData(task, data_dir=data_dir)
classes = tfds_dataset._dataset_builder.info.features[task].names
elif dataset_name == 'dtd':
from task_adaptation.data.dtd import DTDData
tfds_dataset = DTDData(data_dir=data_dir)
elif dataset_name == 'eurosat':
from task_adaptation.data.eurosat import EurosatData
tfds_dataset = EurosatData(subset='rgb', data_key='image', data_dir=data_dir)
classes = classnames['eurosat']
elif dataset_name == 'food101':
from task_adaptation.data.food101 import Food101Data
tfds_dataset = Food101Data(data_dir=data_dir)
elif dataset_name == 'inaturalist':
from task_adaptation.data.inaturalist import INaturalistData
tfds_dataset = INaturalistData(data_dir=data_dir, year=2017)
elif dataset_name.startswith('kitti_'):
from .kitti import KittiData
task = _extract_task(dataset_name)
assert task in (
'count_all', 'count_left', 'count_far', 'count_near',
'closest_object_distance', 'closest_object_x_location',
'count_vehicles', 'closest_vehicle_distance',
)
tfds_dataset = KittiData(task=task, data_dir=data_dir)
if task == 'closest_vehicle_distance':
classes = classnames['kitti_closest_vehicle_distance']
else:
raise ValueError(f'Unsupported task: {task}')
elif dataset_name == 'flowers':
from task_adaptation.data.oxford_flowers102 import OxfordFlowers102Data
tfds_dataset = OxfordFlowers102Data(data_dir=data_dir)
elif dataset_name == 'pets':
from task_adaptation.data.oxford_iiit_pet import OxfordIIITPetData
tfds_dataset = OxfordIIITPetData(data_dir=data_dir)
classes = classnames['pets']
elif dataset_name == 'pcam':
from task_adaptation.data.patch_camelyon import PatchCamelyonData
tfds_dataset = PatchCamelyonData(data_dir=data_dir)
classes = classnames['pcam']
elif dataset_name == 'resisc45':
# Needs download from OneDrive: https://1drv.ms/u/s!AmgKYzARBl5ca3HNaHIlzp_IXjs
# The archive needs to to be put at <DATASET_ROOT>/downloads/manual then extracted
# if not os.path.exists(data_dir):
# os.makedirs(os.path.join(data_dir, "downloads", "manual"))
# call(f"wget 'https://onedrive.live.com/download?resid=5C5E061130630A68!107&authkey=!AHHNaHIlzp_IXjs' --output-document={data_dir}/downloads/manual/resisc45.rar", shell=True)
# call(f"cd {data_dir}/downloads/manual;unrar x resisc45.rar", shell=True)
from task_adaptation.data.resisc45 import Resisc45Data
tfds_dataset = Resisc45Data(data_dir=data_dir)
elif dataset_name.startswith('smallnorb_'):
from task_adaptation.data.smallnorb import SmallNORBData
task = _extract_task(dataset_name)
assert task in ('label_category', 'label_elevation', 'label_azimuth', 'label_lighting')
tfds_dataset = SmallNORBData(predicted_attribute=task, data_dir=data_dir)
classes = tfds_dataset._dataset_builder.info.features[task].names
elif dataset_name == 'sun397':
from task_adaptation.data.sun397 import Sun397Data
# FIXME There is a problem in `sun397`, when TFDS tries download it
# there is an image that cannot be decoded. For the time being
# we will use torchvision's SUN397 instead.
tfds_dataset = Sun397Data(config='tfds', data_dir=data_dir)
elif dataset_name == 'svhn':
from task_adaptation.data.svhn import SvhnData
tfds_dataset = SvhnData(data_dir=data_dir)
classes = classnames['svhn']
else:
raise ValueError(f'Unsupported dataset: {dataset_name}')
ds = VTABIterableDataset(
tfds_dataset,
input_name='image', label_name='label',
transform=transform,
target_transform=int,
split=split,
classes=classes,
)
return ds
def build_tfds_dataset(name, transform, download=True, split='test', data_dir='root', classes=None):
from .tfds import disable_gpus_on_tensorflow
disable_gpus_on_tensorflow()
import tensorflow_datasets as tfds
import timm
builder = tfds.builder(name, data_dir=data_dir)
if download:
builder.download_and_prepare()
splits = list(builder.info.splits.keys())
assert split in splits, (split, splits)
ds = timm.data.create_dataset(f'tfds/{name}', data_dir, split=split, transform=transform, target_transform=int)
ds.classes = builder.info.features['label'].names if classes is None else classes
return ds
def build_wds_dataset(dataset_name, transform, split='test', data_dir='root', cache_dir=None):
"""
Load a dataset in WebDataset format. Either local paths or HTTP URLs can be specified.
Expected file structure is:
```
data_dir/
train/
nshards.txt
0.tar
1.tar
...
test/
nshards.txt
0.tar
1.tar
...
classnames.txt
zeroshot_classification_templates.txt
dataset_type.txt
```
Classnames and templates are required for zeroshot classification, while dataset type
(equal to "retrieval") is required for zeroshot retrieval datasets.
You can use the `clip_benchmark_export_wds` or corresponding API
(`clip_benchmark.webdataset_builder.convert_dataset`) to convert datasets to this format.
Set `cache_dir` to a path to cache the dataset, otherwise, no caching will occur.
"""
import webdataset as wds
def read_txt(fname):
if '://' in fname:
stream = os.popen("curl -L -s --fail '%s'" % fname, 'r')
value = stream.read()
if stream.close():
raise FileNotFoundError('Failed to retreive data')
else:
with open(fname, 'r') as file:
value = file.read()
return value
# Special handling for Huggingface datasets
# Git LFS files have a different file path to access the raw data than other files
if data_dir.startswith('https://huggingface.co/datasets'):
# Format: https://huggingface.co/datasets/<USERNAME>/<REPO>/tree/<BRANCH>
*split_url_head, _, url_path = data_dir.split('/', 7)
url_head = '/'.join(split_url_head)
metadata_dir = '/'.join([url_head, 'raw', url_path])
tardata_dir = '/'.join([url_head, 'resolve', url_path])
else:
metadata_dir = tardata_dir = data_dir
# Get number of shards
nshards_fname = os.path.join(metadata_dir, split, 'nshards.txt')
nshards = int(read_txt(nshards_fname)) # Do not catch FileNotFound, nshards.txt should be mandatory
# Get dataset type (classification or retrieval)
type_fname = os.path.join(metadata_dir, 'dataset_type.txt')
try:
dataset_type = read_txt(type_fname).strip().lower()
except FileNotFoundError:
# print("WARNING: dataset_type.txt not found, assuming type=classification")
dataset_type = 'classification'
#
filepattern = os.path.join(tardata_dir, split, '{0..%d}.tar' % (nshards - 1))
# Load webdataset (support WEBP, PNG, and JPG for now)
if not cache_dir or not isinstance(cache_dir, str):
cache_dir = None
dataset = wds.WebDataset(filepattern, cache_dir=cache_dir).decode(
wds.autodecode.ImageHandler('pil', extensions=['webp', 'png', 'jpg', 'jpeg']))
# Load based on classification or retrieval task
if dataset_type == 'retrieval':
dataset = (dataset
.to_tuple(['webp', 'png', 'jpg', 'jpeg'], 'txt')
.map_tuple(transform, str.splitlines)
)
dataset.classes = dataset.templates = None
else:
label_type = 'npy' if dataset_type == 'multilabel' else 'cls' # Special case for multilabel
dataset = (dataset
.to_tuple(['webp', 'png', 'jpg', 'jpeg'], label_type)
.map_tuple(transform, None)
)
# Get class names if present
classnames_fname = os.path.join(metadata_dir, 'classnames.txt')
try:
dataset.classes = [line.strip() for line in read_txt(classnames_fname).splitlines() if line.strip()]
except FileNotFoundError:
print('WARNING: classnames.txt not found')
dataset.classes = None
# Get zeroshot classification templates if present
templates_fname = os.path.join(metadata_dir, 'zeroshot_classification_templates.txt')
try:
dataset.templates = [line.strip() for line in read_txt(templates_fname).splitlines() if line.strip()]
except FileNotFoundError:
print('WARNING: zeroshot_classification_templates.txt not found')
dataset.templates = None
return dataset
def _extract_task(dataset_name):
prefix, *task_name_list = dataset_name.split('_')
task = '_'.join(task_name_list)
return task
def image_captions_collate_fn(batch):
transposed = list(zip(*batch))
imgs = default_collate(transposed[0])
texts = transposed[1]
return imgs, texts
def get_dataset_collection_from_file(path):
return [l.strip() for l in open(path).readlines()]
dataset_collection = {
'vtab': [
'vtab/caltech101',
'vtab/cifar100',
'vtab/clevr_count_all',
'vtab/clevr_closest_object_distance',
'vtab/diabetic_retinopathy',
'vtab/dmlab',
'vtab/dsprites_label_orientation',
'vtab/dsprites_label_x_position',
'vtab/dtd',
'vtab/eurosat',
'vtab/kitti_closest_vehicle_distance',
'vtab/flowers',
'vtab/pets',
'vtab/pcam',
'vtab/resisc45',
'vtab/smallnorb_label_azimuth',
'vtab/smallnorb_label_elevation',
'sun397',
'vtab/svhn',
],
'vtab+': [
'imagenet1k',
'imagenetv2',
'imagenet_sketch',
'imagenet-a',
'imagenet-r',
'objectnet',
'fer2013',
'voc2007',
'voc2007_multilabel',
'sun397',
'cars',
'fgvc_aircraft',
'mnist',
'stl10',
'gtsrb',
'country211',
'renderedsst2',
'vtab/caltech101',
'vtab/cifar10',
'vtab/cifar100',
'vtab/clevr_count_all',
'vtab/clevr_closest_object_distance',
'vtab/diabetic_retinopathy',
'vtab/dmlab',
'vtab/dsprites_label_orientation',
'vtab/dsprites_label_x_position',
'vtab/dtd',
'vtab/eurosat',
'vtab/kitti_closest_vehicle_distance',
'vtab/flowers',
'vtab/pets',
'vtab/pcam',
'vtab/resisc45',
'vtab/smallnorb_label_azimuth',
'vtab/smallnorb_label_elevation',
'vtab/svhn',
],
'retrieval': [
'mscoco_captions',
'flickr8k',
'flickr30k',
],
'imagenet_robustness': [
'imagenetv2',
'imagenet_sketch',
'imagenet-a',
'imagenet-r',
'objectnet',
],
}
# use by imagenet robustness datasets
all_imagenet_wordnet_ids = ['n01440764', 'n01443537', 'n01484850', 'n01491361', 'n01494475', 'n01496331', 'n01498041',
'n01514668', 'n01514859', 'n01518878', 'n01530575', 'n01531178', 'n01532829', 'n01534433',
'n01537544', 'n01558993', 'n01560419', 'n01580077', 'n01582220', 'n01592084', 'n01601694',
'n01608432', 'n01614925', 'n01616318', 'n01622779', 'n01629819', 'n01630670', 'n01631663',
'n01632458', 'n01632777', 'n01641577', 'n01644373', 'n01644900', 'n01664065', 'n01665541',
'n01667114', 'n01667778', 'n01669191', 'n01675722', 'n01677366', 'n01682714', 'n01685808',
'n01687978', 'n01688243', 'n01689811', 'n01692333', 'n01693334', 'n01694178', 'n01695060',
'n01697457', 'n01698640', 'n01704323', 'n01728572', 'n01728920', 'n01729322', 'n01729977',
'n01734418', 'n01735189', 'n01737021', 'n01739381', 'n01740131', 'n01742172', 'n01744401',
'n01748264', 'n01749939', 'n01751748', 'n01753488', 'n01755581', 'n01756291', 'n01768244',
'n01770081', 'n01770393', 'n01773157', 'n01773549', 'n01773797', 'n01774384', 'n01774750',
'n01775062', 'n01776313', 'n01784675', 'n01795545', 'n01796340', 'n01797886', 'n01798484',
'n01806143', 'n01806567', 'n01807496', 'n01817953', 'n01818515', 'n01819313', 'n01820546',
'n01824575', 'n01828970', 'n01829413', 'n01833805', 'n01843065', 'n01843383', 'n01847000',
'n01855032', 'n01855672', 'n01860187', 'n01871265', 'n01872401', 'n01873310', 'n01877812',
'n01882714', 'n01883070', 'n01910747', 'n01914609', 'n01917289', 'n01924916', 'n01930112',
'n01943899', 'n01944390', 'n01945685', 'n01950731', 'n01955084', 'n01968897', 'n01978287',
'n01978455', 'n01980166', 'n01981276', 'n01983481', 'n01984695', 'n01985128', 'n01986214',
'n01990800', 'n02002556', 'n02002724', 'n02006656', 'n02007558', 'n02009229', 'n02009912',
'n02011460', 'n02012849', 'n02013706', 'n02017213', 'n02018207', 'n02018795', 'n02025239',
'n02027492', 'n02028035', 'n02033041', 'n02037110', 'n02051845', 'n02056570', 'n02058221',
'n02066245', 'n02071294', 'n02074367', 'n02077923', 'n02085620', 'n02085782', 'n02085936',
'n02086079', 'n02086240', 'n02086646', 'n02086910', 'n02087046', 'n02087394', 'n02088094',
'n02088238', 'n02088364', 'n02088466', 'n02088632', 'n02089078', 'n02089867', 'n02089973',
'n02090379', 'n02090622', 'n02090721', 'n02091032', 'n02091134', 'n02091244', 'n02091467',
'n02091635', 'n02091831', 'n02092002', 'n02092339', 'n02093256', 'n02093428', 'n02093647',
'n02093754', 'n02093859', 'n02093991', 'n02094114', 'n02094258', 'n02094433', 'n02095314',
'n02095570', 'n02095889', 'n02096051', 'n02096177', 'n02096294', 'n02096437', 'n02096585',
'n02097047', 'n02097130', 'n02097209', 'n02097298', 'n02097474', 'n02097658', 'n02098105',
'n02098286', 'n02098413', 'n02099267', 'n02099429', 'n02099601', 'n02099712', 'n02099849',
'n02100236', 'n02100583', 'n02100735', 'n02100877', 'n02101006', 'n02101388', 'n02101556',
'n02102040', 'n02102177', 'n02102318', 'n02102480', 'n02102973', 'n02104029', 'n02104365',
'n02105056', 'n02105162', 'n02105251', 'n02105412', 'n02105505', 'n02105641', 'n02105855',
'n02106030', 'n02106166', 'n02106382', 'n02106550', 'n02106662', 'n02107142', 'n02107312',
'n02107574', 'n02107683', 'n02107908', 'n02108000', 'n02108089', 'n02108422', 'n02108551',
'n02108915', 'n02109047', 'n02109525', 'n02109961', 'n02110063', 'n02110185', 'n02110341',
'n02110627', 'n02110806', 'n02110958', 'n02111129', 'n02111277', 'n02111500', 'n02111889',
'n02112018', 'n02112137', 'n02112350', 'n02112706', 'n02113023', 'n02113186', 'n02113624',
'n02113712', 'n02113799', 'n02113978', 'n02114367', 'n02114548', 'n02114712', 'n02114855',
'n02115641', 'n02115913', 'n02116738', 'n02117135', 'n02119022', 'n02119789', 'n02120079',
'n02120505', 'n02123045', 'n02123159', 'n02123394', 'n02123597', 'n02124075', 'n02125311',
'n02127052', 'n02128385', 'n02128757', 'n02128925', 'n02129165', 'n02129604', 'n02130308',
'n02132136', 'n02133161', 'n02134084', 'n02134418', 'n02137549', 'n02138441', 'n02165105',
'n02165456', 'n02167151', 'n02168699', 'n02169497', 'n02172182', 'n02174001', 'n02177972',
'n02190166', 'n02206856', 'n02219486', 'n02226429', 'n02229544', 'n02231487', 'n02233338',
'n02236044', 'n02256656', 'n02259212', 'n02264363', 'n02268443', 'n02268853', 'n02276258',
'n02277742', 'n02279972', 'n02280649', 'n02281406', 'n02281787', 'n02317335', 'n02319095',
'n02321529', 'n02325366', 'n02326432', 'n02328150', 'n02342885', 'n02346627', 'n02356798',
'n02361337', 'n02363005', 'n02364673', 'n02389026', 'n02391049', 'n02395406', 'n02396427',
'n02397096', 'n02398521', 'n02403003', 'n02408429', 'n02410509', 'n02412080', 'n02415577',
'n02417914', 'n02422106', 'n02422699', 'n02423022', 'n02437312', 'n02437616', 'n02441942',
'n02442845', 'n02443114', 'n02443484', 'n02444819', 'n02445715', 'n02447366', 'n02454379',
'n02457408', 'n02480495', 'n02480855', 'n02481823', 'n02483362', 'n02483708', 'n02484975',
'n02486261', 'n02486410', 'n02487347', 'n02488291', 'n02488702', 'n02489166', 'n02490219',
'n02492035', 'n02492660', 'n02493509', 'n02493793', 'n02494079', 'n02497673', 'n02500267',
'n02504013', 'n02504458', 'n02509815', 'n02510455', 'n02514041', 'n02526121', 'n02536864',
'n02606052', 'n02607072', 'n02640242', 'n02641379', 'n02643566', 'n02655020', 'n02666196',
'n02667093', 'n02669723', 'n02672831', 'n02676566', 'n02687172', 'n02690373', 'n02692877',
'n02699494', 'n02701002', 'n02704792', 'n02708093', 'n02727426', 'n02730930', 'n02747177',
'n02749479', 'n02769748', 'n02776631', 'n02777292', 'n02782093', 'n02783161', 'n02786058',
'n02787622', 'n02788148', 'n02790996', 'n02791124', 'n02791270', 'n02793495', 'n02794156',
'n02795169', 'n02797295', 'n02799071', 'n02802426', 'n02804414', 'n02804610', 'n02807133',
'n02808304', 'n02808440', 'n02814533', 'n02814860', 'n02815834', 'n02817516', 'n02823428',
'n02823750', 'n02825657', 'n02834397', 'n02835271', 'n02837789', 'n02840245', 'n02841315',
'n02843684', 'n02859443', 'n02860847', 'n02865351', 'n02869837', 'n02870880', 'n02871525',
'n02877765', 'n02879718', 'n02883205', 'n02892201', 'n02892767', 'n02894605', 'n02895154',
'n02906734', 'n02909870', 'n02910353', 'n02916936', 'n02917067', 'n02927161', 'n02930766',
'n02939185', 'n02948072', 'n02950826', 'n02951358', 'n02951585', 'n02963159', 'n02965783',
'n02966193', 'n02966687', 'n02971356', 'n02974003', 'n02977058', 'n02978881', 'n02979186',
'n02980441', 'n02981792', 'n02988304', 'n02992211', 'n02992529', 'n02999410', 'n03000134',
'n03000247', 'n03000684', 'n03014705', 'n03016953', 'n03017168', 'n03018349', 'n03026506',
'n03028079', 'n03032252', 'n03041632', 'n03042490', 'n03045698', 'n03047690', 'n03062245',
'n03063599', 'n03063689', 'n03065424', 'n03075370', 'n03085013', 'n03089624', 'n03095699',
'n03100240', 'n03109150', 'n03110669', 'n03124043', 'n03124170', 'n03125729', 'n03126707',
'n03127747', 'n03127925', 'n03131574', 'n03133878', 'n03134739', 'n03141823', 'n03146219',
'n03160309', 'n03179701', 'n03180011', 'n03187595', 'n03188531', 'n03196217', 'n03197337',
'n03201208', 'n03207743', 'n03207941', 'n03208938', 'n03216828', 'n03218198', 'n03220513',
'n03223299', 'n03240683', 'n03249569', 'n03250847', 'n03255030', 'n03259280', 'n03271574',
'n03272010', 'n03272562', 'n03290653', 'n03291819', 'n03297495', 'n03314780', 'n03325584',
'n03337140', 'n03344393', 'n03345487', 'n03347037', 'n03355925', 'n03372029', 'n03376595',
'n03379051', 'n03384352', 'n03388043', 'n03388183', 'n03388549', 'n03393912', 'n03394916',
'n03400231', 'n03404251', 'n03417042', 'n03424325', 'n03425413', 'n03443371', 'n03444034',
'n03445777', 'n03445924', 'n03447447', 'n03447721', 'n03450230', 'n03452741', 'n03457902',
'n03459775', 'n03461385', 'n03467068', 'n03476684', 'n03476991', 'n03478589', 'n03481172',
'n03482405', 'n03483316', 'n03485407', 'n03485794', 'n03492542', 'n03494278', 'n03495258',
'n03496892', 'n03498962', 'n03527444', 'n03529860', 'n03530642', 'n03532672', 'n03534580',
'n03535780', 'n03538406', 'n03544143', 'n03584254', 'n03584829', 'n03590841', 'n03594734',
'n03594945', 'n03595614', 'n03598930', 'n03599486', 'n03602883', 'n03617480', 'n03623198',
'n03627232', 'n03630383', 'n03633091', 'n03637318', 'n03642806', 'n03649909', 'n03657121',
'n03658185', 'n03661043', 'n03662601', 'n03666591', 'n03670208', 'n03673027', 'n03676483',
'n03680355', 'n03690938', 'n03691459', 'n03692522', 'n03697007', 'n03706229', 'n03709823',
'n03710193', 'n03710637', 'n03710721', 'n03717622', 'n03720891', 'n03721384', 'n03724870',
'n03729826', 'n03733131', 'n03733281', 'n03733805', 'n03742115', 'n03743016', 'n03759954',
'n03761084', 'n03763968', 'n03764736', 'n03769881', 'n03770439', 'n03770679', 'n03773504',
'n03775071', 'n03775546', 'n03776460', 'n03777568', 'n03777754', 'n03781244', 'n03782006',
'n03785016', 'n03786901', 'n03787032', 'n03788195', 'n03788365', 'n03791053', 'n03792782',
'n03792972', 'n03793489', 'n03794056', 'n03796401', 'n03803284', 'n03804744', 'n03814639',
'n03814906', 'n03825788', 'n03832673', 'n03837869', 'n03838899', 'n03840681', 'n03841143',
'n03843555', 'n03854065', 'n03857828', 'n03866082', 'n03868242', 'n03868863', 'n03871628',
'n03873416', 'n03874293', 'n03874599', 'n03876231', 'n03877472', 'n03877845', 'n03884397',
'n03887697', 'n03888257', 'n03888605', 'n03891251', 'n03891332', 'n03895866', 'n03899768',
'n03902125', 'n03903868', 'n03908618', 'n03908714', 'n03916031', 'n03920288', 'n03924679',
'n03929660', 'n03929855', 'n03930313', 'n03930630', 'n03933933', 'n03935335', 'n03937543',
'n03938244', 'n03942813', 'n03944341', 'n03947888', 'n03950228', 'n03954731', 'n03956157',
'n03958227', 'n03961711', 'n03967562', 'n03970156', 'n03976467', 'n03976657', 'n03977966',
'n03980874', 'n03982430', 'n03983396', 'n03991062', 'n03992509', 'n03995372', 'n03998194',
'n04004767', 'n04005630', 'n04008634', 'n04009552', 'n04019541', 'n04023962', 'n04026417',
'n04033901', 'n04033995', 'n04037443', 'n04039381', 'n04040759', 'n04041544', 'n04044716',
'n04049303', 'n04065272', 'n04067472', 'n04069434', 'n04070727', 'n04074963', 'n04081281',
'n04086273', 'n04090263', 'n04099969', 'n04111531', 'n04116512', 'n04118538', 'n04118776',
'n04120489', 'n04125021', 'n04127249', 'n04131690', 'n04133789', 'n04136333', 'n04141076',
'n04141327', 'n04141975', 'n04146614', 'n04147183', 'n04149813', 'n04152593', 'n04153751',
'n04154565', 'n04162706', 'n04179913', 'n04192698', 'n04200800', 'n04201297', 'n04204238',
'n04204347', 'n04208210', 'n04209133', 'n04209239', 'n04228054', 'n04229816', 'n04235860',
'n04238763', 'n04239074', 'n04243546', 'n04251144', 'n04252077', 'n04252225', 'n04254120',
'n04254680', 'n04254777', 'n04258138', 'n04259630', 'n04263257', 'n04264628', 'n04265275',
'n04266014', 'n04270147', 'n04273569', 'n04275548', 'n04277352', 'n04285008', 'n04286575',
'n04296562', 'n04310018', 'n04311004', 'n04311174', 'n04317175', 'n04325704', 'n04326547',
'n04328186', 'n04330267', 'n04332243', 'n04335435', 'n04336792', 'n04344873', 'n04346328',
'n04347754', 'n04350905', 'n04355338', 'n04355933', 'n04356056', 'n04357314', 'n04366367',
'n04367480', 'n04370456', 'n04371430', 'n04371774', 'n04372370', 'n04376876', 'n04380533',
'n04389033', 'n04392985', 'n04398044', 'n04399382', 'n04404412', 'n04409515', 'n04417672',
'n04418357', 'n04423845', 'n04428191', 'n04429376', 'n04435653', 'n04442312', 'n04443257',
'n04447861', 'n04456115', 'n04458633', 'n04461696', 'n04462240', 'n04465501', 'n04467665',
'n04476259', 'n04479046', 'n04482393', 'n04483307', 'n04485082', 'n04486054', 'n04487081',
'n04487394', 'n04493381', 'n04501370', 'n04505470', 'n04507155', 'n04509417', 'n04515003',
'n04517823', 'n04522168', 'n04523525', 'n04525038', 'n04525305', 'n04532106', 'n04532670',
'n04536866', 'n04540053', 'n04542943', 'n04548280', 'n04548362', 'n04550184', 'n04552348',
'n04553703', 'n04554684', 'n04557648', 'n04560804', 'n04562935', 'n04579145', 'n04579432',
'n04584207', 'n04589890', 'n04590129', 'n04591157', 'n04591713', 'n04592741', 'n04596742',
'n04597913', 'n04599235', 'n04604644', 'n04606251', 'n04612504', 'n04613696', 'n06359193',
'n06596364', 'n06785654', 'n06794110', 'n06874185', 'n07248320', 'n07565083', 'n07579787',
'n07583066', 'n07584110', 'n07590611', 'n07613480', 'n07614500', 'n07615774', 'n07684084',
'n07693725', 'n07695742', 'n07697313', 'n07697537', 'n07711569', 'n07714571', 'n07714990',
'n07715103', 'n07716358', 'n07716906', 'n07717410', 'n07717556', 'n07718472', 'n07718747',
'n07720875', 'n07730033', 'n07734744', 'n07742313', 'n07745940', 'n07747607', 'n07749582',
'n07753113', 'n07753275', 'n07753592', 'n07754684', 'n07760859', 'n07768694', 'n07802026',
'n07831146', 'n07836838', 'n07860988', 'n07871810', 'n07873807', 'n07875152', 'n07880968',
'n07892512', 'n07920052', 'n07930864', 'n07932039', 'n09193705', 'n09229709', 'n09246464',
'n09256479', 'n09288635', 'n09332890', 'n09399592', 'n09421951', 'n09428293', 'n09468604',
'n09472597', 'n09835506', 'n10148035', 'n10565667', 'n11879895', 'n11939491', 'n12057211',
'n12144580', 'n12267677', 'n12620546', 'n12768682', 'n12985857', 'n12998815', 'n13037406',
'n13040303', 'n13044778', 'n13052670', 'n13054560', 'n13133613', 'n15075141']
clip_benchmark/clip_benchmark/datasets/caltech101.py 0 → 100644
View file @ 1d5a34cf
"""
Code adapted from https://github.com/pytorch/vision/blob/main/torchvision/datasets/caltech.py
Modification of caltech101 from torchvision where the background class is not removed
Thanks to the authors of torchvision
"""
import os
import os.path
from glob import glob
from typing import Any, Callable, List, Optional, Tuple, Union
from PIL import Image
from torchvision.datasets.utils import (download_and_extract_archive,
verify_str_arg)
from torchvision.datasets.vision import VisionDataset
class Caltech101(VisionDataset):
"""`Caltech 101 <http://www.vision.caltech.edu/Image_Datasets/Caltech101/>`_ Dataset.
.. warning::
This class needs `scipy <https://docs.scipy.org/doc/>`_ to load target files from `.mat` format.
Args:
root (string): Root directory of dataset where directory
``caltech101`` exists or will be saved to if download is set to True.
target_type (string or list, optional): Type of target to use, ``category`` or
``annotation``. Can also be a list to output a tuple with all specified
target types. ``category`` represents the target class, and
``annotation`` is a list of points from a hand-generated outline.
Defaults to ``category``.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
download (bool, optional): If true, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
"""
def __init__(
self,
root: str,
target_type: Union[List[str], str] = 'category',
transform: Optional[Callable] = None,
target_transform: Optional[Callable] = None,
download: bool = False,
) -> None:
super().__init__(os.path.join(root, 'caltech101'), transform=transform, target_transform=target_transform)
os.makedirs(self.root, exist_ok=True)
if isinstance(target_type, str):
target_type = [target_type]
self.target_type = [verify_str_arg(t, 'target_type', ('category', 'annotation')) for t in target_type]
if download:
self.download()
if not self._check_integrity():
raise RuntimeError('Dataset not found or corrupted. You can use download=True to download it')
self.categories = sorted(os.listdir(os.path.join(self.root, '101_ObjectCategories')))
# self.categories.remove("BACKGROUND_Google") # this is not a real class
# For some reason, the category names in "101_ObjectCategories" and
# "Annotations" do not always match. This is a manual map between the
# two. Defaults to using same name, since most names are fine.
name_map = {
'Faces': 'Faces_2',
'Faces_easy': 'Faces_3',
'Motorbikes': 'Motorbikes_16',
'airplanes': 'Airplanes_Side_2',
}
self.annotation_categories = list(map(lambda x: name_map[x] if x in name_map else x, self.categories))
self.index: List[int] = []
self.y = []
for (i, c) in enumerate(self.categories):
n = len(glob(os.path.join(self.root, '101_ObjectCategories', c, '*.jpg')))
self.index.extend(range(1, n + 1))
self.y.extend(n * [i])
def __getitem__(self, index: int) -> Tuple[Any, Any]:
"""
Args:
index (int): Index
Returns:
tuple: (image, target) where the type of target specified by target_type.
"""
import scipy.io
img = Image.open(
os.path.join(
self.root,
'101_ObjectCategories',
self.categories[self.y[index]],
f'image_{self.index[index]:04d}.jpg',
)
)
target: Any = []
for t in self.target_type:
if t == 'category':
target.append(self.y[index])
elif t == 'annotation':
data = scipy.io.loadmat(
os.path.join(
self.root,
'Annotations',
self.annotation_categories[self.y[index]],
f'annotation_{self.index[index]:04d}.mat',
)
)
target.append(data['obj_contour'])
target = tuple(target) if len(target) > 1 else target[0]
if self.transform is not None:
img = self.transform(img)
if self.target_transform is not None:
target = self.target_transform(target)
return img, target
def _check_integrity(self) -> bool:
# can be more robust and check hash of files
return os.path.exists(os.path.join(self.root, '101_ObjectCategories'))
def __len__(self) -> int:
return len(self.index)
def download(self) -> None:
if self._check_integrity():
print('Files already downloaded and verified')
return
download_and_extract_archive(
'https://drive.google.com/file/d/137RyRjvTBkBiIfeYBNZBtViDHQ6_Ewsp',
self.root,
filename='101_ObjectCategories.tar.gz',
md5='b224c7392d521a49829488ab0f1120d9',
)
download_and_extract_archive(
'https://drive.google.com/file/d/175kQy3UsZ0wUEHZjqkUDdNVssr7bgh_m',
self.root,
filename='Annotations.tar',
md5='6f83eeb1f24d99cab4eb377263132c91',
)
def extra_repr(self) -> str:
return 'Target type: {target_type}'.format(**self.__dict__)
class Caltech256(VisionDataset):
"""`Caltech 256 <http://www.vision.caltech.edu/Image_Datasets/Caltech256/>`_ Dataset.
Args:
root (string): Root directory of dataset where directory
``caltech256`` exists or will be saved to if download is set to True.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
download (bool, optional): If true, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
"""
def __init__(
self,
root: str,
transform: Optional[Callable] = None,
target_transform: Optional[Callable] = None,
download: bool = False,
) -> None:
super().__init__(os.path.join(root, 'caltech256'), transform=transform, target_transform=target_transform)
os.makedirs(self.root, exist_ok=True)
if download:
self.download()
if not self._check_integrity():
raise RuntimeError('Dataset not found or corrupted. You can use download=True to download it')
self.categories = sorted(os.listdir(os.path.join(self.root, '256_ObjectCategories')))
self.index: List[int] = []
self.y = []
for (i, c) in enumerate(self.categories):
n = len(
[
item
for item in os.listdir(os.path.join(self.root, '256_ObjectCategories', c))
if item.endswith('.jpg')
]
)
self.index.extend(range(1, n + 1))
self.y.extend(n * [i])
def __getitem__(self, index: int) -> Tuple[Any, Any]:
"""
Args:
index (int): Index
Returns:
tuple: (image, target) where target is index of the target class.
"""
img = Image.open(
os.path.join(
self.root,
'256_ObjectCategories',
self.categories[self.y[index]],
f'{self.y[index] + 1:03d}_{self.index[index]:04d}.jpg',
)
)
target = self.y[index]
if self.transform is not None:
img = self.transform(img)
if self.target_transform is not None:
target = self.target_transform(target)
return img, target
def _check_integrity(self) -> bool:
# can be more robust and check hash of files
return os.path.exists(os.path.join(self.root, '256_ObjectCategories'))
def __len__(self) -> int:
return len(self.index)
def download(self) -> None:
if self._check_integrity():
print('Files already downloaded and verified')
return
download_and_extract_archive(
'https://drive.google.com/file/d/1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK',
self.root,
filename='256_ObjectCategories.tar',
md5='67b4f42ca05d46448c6bb8ecd2220f6d',
)
clip_benchmark/clip_benchmark/datasets/cn_classnames.json 0 → 100644
View file @ 1d5a34cf
{
"imagenet1k": [
"\u4e01\u9cb7",
"\u91d1\u9c7c",
"\u5927\u767d\u9ca8",
"\u864e\u9ca8",
"\u9524\u5934\u9ca8",
"\u7535\u9cd0",
"\u9ec4\u8c82\u9c7c",
"\u516c\u9e21",
"\u6bcd\u9e21",
"\u9e35\u9e1f",
"\u71d5\u96c0",
"\u91d1\u7fc5\u96c0",
"\u5bb6\u6731\u96c0",
"\u706f\u82af\u8349\u96c0",
"\u975b\u84dd\u96c0",
"\u84dd\u9e40",
"\u591c\u83ba",
"\u677e\u9e26",
"\u559c\u9e4a",
"\u5c71\u96c0",
"\u6cb3\u9e1f",
"\u9e22\uff08\u731b\u79bd\uff09",
"\u79c3\u5934\u9e70",
"\u79c3\u9e6b",
"\u5927\u7070\u732b\u5934\u9e70",
"\u6b27\u6d32\u706b\u877e\u8788",
"\u666e\u901a\u877e\u8788",
"\u6c34\u8725",
"\u6591\u70b9\u877e\u8788",
"\u877e\u8788",
"\u725b\u86d9",
"\u6811\u86d9",
"\u5c3e\u86d9",
"\u7ea2\u6d77\u9f9f",
"\u76ae\u9769\u9f9f",
"\u6ce5\u9f9f",
"\u6de1\u6c34\u9f9f",
"\u7bb1\u9f9f",
"\u5e26\u72b6\u58c1\u864e",
"\u666e\u901a\u9b23\u8725",
"\u7f8e\u56fd\u53d8\u8272\u9f99",
"\u97ad\u5c3e\u8725\u8734",
"\u98de\u9f99\u79d1\u8725\u8734",
"\u8936\u8fb9\u8725\u8734",
"\u9cc4\u9c7c\u8725\u8734",
"\u6bd2\u8725",
"\u7eff\u8725\u8734",
"\u975e\u6d32\u53d8\u8272\u9f99",
"\u79d1\u83ab\u591a\u8725\u8734",
"\u975e\u6d32\u9cc4",
"\u7f8e\u56fd\u9cc4\u9c7c",
"\u4e09\u89d2\u9f99",
"\u96f7\u86c7",
"\u73af\u86c7",
"\u5e0c\u814a\u86c7",
"\u7eff\u86c7",
"\u56fd\u738b\u86c7",
"\u889c\u5e26\u86c7",
"\u6c34\u86c7",
"\u85e4\u86c7",
"\u591c\u86c7",
"\u5927\u87d2\u86c7",
"\u5ca9\u77f3\u87d2\u86c7",
"\u5370\u5ea6\u773c\u955c\u86c7",
"\u7eff\u66fc\u5df4",
"\u6d77\u86c7",
"\u89d2\u8179\u86c7",
"\u83f1\u7eb9\u54cd\u5c3e\u86c7",
"\u89d2\u54cd\u5c3e\u86c7",
"\u4e09\u53f6\u866b",
"\u76f2\u8718\u86db",
"\u874e\u5b50",
"\u9ed1\u91d1\u82b1\u56ed\u8718\u86db",
"\u8c37\u4ed3\u8718\u86db",
"\u82b1\u56ed\u8718\u86db",
"\u9ed1\u5be1\u5987\u8718\u86db",
"\u72fc\u86db",
"\u72fc\u8718\u86db",
"\u58c1\u8671",
"\u8708\u86a3",
"\u9ed1\u677e\u9e21",
"\u677e\u9e21",
"\u62ab\u80a9\u9e21",
"\u8349\u539f\u9e21",
"\u5b54\u96c0",
"\u9e4c\u9e51",
"\u9e67\u9e2a",
"\u975e\u6d32\u7070\u9e66\u9e49",
"\u91d1\u521a\u9e66\u9e49",
"\u786b\u51a0\u9e66\u9e49",
"\u77ed\u5c3e\u9e66\u9e49",
"\u8910\u7fc5\u9e26\u9e43",
"\u98df\u8702\u9e1f\uff1b\u8702\u864e",
"\u7280\u9e1f",
"\u8702\u9e1f",
"\u9e5f\u4d15",
"\u5de8\u5634\u9e1f\uff1b\u5927\u5634\u9e1f",
"\u91ce\u9e2d",
"\u7ea2\u80f8\u79cb\u6c99\u9e2d",
"\u9e45",
"\u9ed1\u5929\u9e45",
"\u5927\u8c61",
"\u9488\u9f39\u9f20",
"\u9e2d\u5634\u517d",
"\u6c99\u888b\u9f20",
"\u8003\u62c9",
"\u888b\u718a",
"\u6c34\u6bcd",
"\u6d77\u8475",
"\u8111\u73ca\u745a",
"\u6241\u5f62\u866b\u6241\u866b",
"\u7ebf\u866b",
"\u6d77\u87ba",
"\u8717\u725b",
"\u9f3b\u6d95\u866b",
"\u6d77\u86de\u8753\uff1b\u6d77\u53c2",
"\u77f3\u9cd6",
"\u9e66\u9e49\u87ba",
"\u73cd\u5b9d\u87f9",
"\u77f3\u87f9",
"\u62db\u6f6e\u87f9",
"\u5e1d\u738b\u87f9",
"\u7f8e\u56fd\u9f99\u867e",
"\u5927\u87af\u867e",
"\u5c0f\u9f99\u867e",
"\u5bc4\u5c45\u87f9",
"\u7b49\u8db3\u76ee\u52a8\u7269\uff08\u660e\u867e\u548c\u8783\u87f9\u8fd1\u4eb2\uff09",
"\u767d\u9e73",
"\u9ed1\u9e73",
"\u9e6d",
"\u706b\u70c8\u9e1f",
"\u5c0f\u84dd\u9e6d",
"\u7f8e\u56fd\u9e6d",
"\u9ebb\u9e26",
"\u9e64",
"\u79e7\u9e64",
"\u6b27\u6d32\u6c34\u9e21",
"\u6cbc\u6cfd\u6ce5\u6bcd\u9e21",
"\u9e28",
"\u7ea2\u7ffb\u77f3\u9e6c",
"\u7ea2\u80cc\u9e6c",
"\u7ea2\u811a\u9e6c",
"\u534a\u8e7c\u9e6c",
"\u86ce\u9e6c",
"\u9e48\u9e55",
"\u56fd\u738b\u4f01\u9e45",
"\u4fe1\u5929\u7fc1",
"\u7070\u9cb8",
"\u6740\u4eba\u9cb8",
"\u6d77\u725b",
"\u6d77\u72ee",
"\u5409\u5a03\u5a03",
"\u65e5\u672c\u72c6\u72ac",
"\u9a6c\u5c14\u6d4e\u65af\u72ac",
"\u72ee\u5b50\u72d7",
"\u897f\u65bd\u72ac",
"\u5e03\u83b1\u5c3c\u59c6\u730e\u72ac",
"\u5df4\u6bd4\u72d7",
"\u73a9\u5177\u72ac",
"\u7f57\u5f97\u897f\u4e9a\u957f\u80cc\u730e\u72d7",
"\u963f\u5bcc\u6c57\u730e\u72ac",
"\u5df4\u5409\u5ea6\u730e\u72ac",
"\u6bd4\u683c\u72ac",
"\u4fa6\u63a2\u72ac",
"\u84dd\u8272\u5feb\u72d7",
"\u9ed1\u8910\u730e\u6d63\u718a\u72ac",
"\u6c83\u514b\u730e\u72ac",
"\u82f1\u56fd\u730e\u72d0\u72ac",
"\u7f8e\u6d32\u8d64\u72d7",
"\u4fc4\u7f57\u65af\u730e\u72fc\u72ac",
"\u7231\u5c14\u5170\u730e\u72fc\u72ac",
"\u610f\u5927\u5229\u7070\u72d7",
"\u60e0\u6bd4\u7279\u72ac",
"\u4f9d\u6bd4\u6c99\u730e\u72ac",
"\u632a\u5a01\u730e\u72ac",
"\u5965\u8fbe\u730e\u72ac",
"\u6c99\u514b\u72ac",
"\u82cf\u683c\u5170\u730e\u9e7f\u72ac",
"\u5a01\u739b\u730e\u72ac",
"\u65af\u5854\u798f\u5fb7\u90e1\u6597\u725b\u72ac",
"\u7f8e\u56fd\u65af\u5854\u798f\u5fb7\u90e1\u6897",
"\u8d1d\u5fb7\u7075\u987f\u6897",
"\u8fb9\u5883\u6897",
"\u51ef\u4e3d\u84dd\u6897",
"\u7231\u5c14\u5170\u6897",
"\u8bfa\u798f\u514b\u6897",
"\u8bfa\u7ef4\u5947\u6897",
"\u7ea6\u514b\u72ac\uff1b\u7ea6\u514b\u590f\u6897\u72ac",
"\u521a\u6bdb\u730e\u72d0\u6897",
"\u83b1\u514b\u5170\u6897",
"\u9521\u5229\u54c8\u59c6\u6897",
"\u827e\u5c14\u8c37\u72ac",
"\u51ef\u6069\u6897",
"\u6fb3\u5927\u5229\u4e9a\u6897",
"\u4e39\u8fea\u4e01\u8499\u6897",
"\u6ce2\u58eb\u987f\u6897",
"\u8ff7\u4f60\u96ea\u7eb3\u745e\u72ac",
"\u5de8\u578b\u96ea\u7eb3\u745e\u72ac",
"\u6807\u51c6\u96ea\u7eb3\u745e\u72ac",
"\u82cf\u683c\u5170\u6897\u72ac",
"\u897f\u85cf\u6897",
"\u4e1d\u6bdb\u6897",
"\u7231\u5c14\u5170\u8f6f\u6bdb\u6897\u72ac",
"\u897f\u9ad8\u5730\u767d\u6897",
"\u62c9\u8428\u963f\u666e\u7d22\u72ac",
"\u5e73\u6bdb\u5bfb\u56de\u72ac",
"\u5377\u6bdb\u5bfb\u56de\u72ac",
"\u91d1\u6bdb\u730e\u72ac",
"\u62c9\u5e03\u62c9\u591a\u730e\u72ac",
"\u4e5e\u6c99\u6bd4\u514b\u730e\u72ac",
"\u5fb7\u56fd\u77ed\u6bdb\u6307\u793a\u72ac",
"\u7ef4\u5179\u62c9\u72ac",
"\u82f1\u56fd\u585e\u7279\u72ac",
"\u7231\u5c14\u5170\u96ea\u8fbe\u72ac",
"\u6208\u767b\u96ea\u8fbe\u72ac",
"\u5e03\u5217\u5854\u5c3c\u72ac\u730e\u72ac",
"\u9ec4\u6bdb",
"\u82f1\u56fd\u53f2\u5bbe\u683c\u72ac",
"\u5a01\u5c14\u58eb\u53f2\u5bbe\u683c\u72ac",
"\u53ef\u5361\u72ac",
"\u8428\u585e\u514b\u65af\u730e\u72ac",
"\u7231\u5c14\u5170\u6c34\u730e\u72ac",
"\u54e5\u5a01\u65af\u72ac",
"\u8212\u67cf\u5947\u72ac",
"\u6bd4\u5229\u65f6\u7267\u7f8a\u72ac",
"\u9a6c\u91cc\u52aa\u963f\u72ac",
"\u4f2f\u745e\u72ac",
"\u51ef\u5c14\u76ae\u72ac",
"\u5308\u7259\u5229\u7267\u7f8a\u72ac",
"\u8001\u82f1\u56fd\u7267\u7f8a\u72ac",
"\u559c\u4e50\u8482\u7267\u7f8a\u72ac",
"\u7267\u7f8a\u72ac",
"\u8fb9\u5883\u7267\u7f8a\u72ac",
"\u6cd5\u5170\u5fb7\u65af\u7267\u725b\u72d7",
"\u7f57\u7279\u97e6\u5c14\u72ac",
"\u5fb7\u56fd\u7267\u7f8a\u72ac",
"\u591a\u4f2f\u66fc\u72ac",
"\u9e7f\u72ac\uff1b\u8ff7\u4f60\u675c\u5bbe\u72ac",
"\u5927\u745e\u58eb\u5c71\u5730\u72ac",
"\u4f2f\u6069\u5c71\u72ac",
"\u963f\u7b56\u5c14\u5c71\u72ac",
"\u6069\u7279\u5c14\u5e03\u8d6b\u5c71\u72ac",
"\u62f3\u5e08\u72d7",
"\u6597\u725b\u7352",
"\u85cf\u7352",
"\u6cd5\u56fd\u6597\u725b\u72ac",
"\u5927\u4e39\u72ac",
"\u5723\u4f2f\u7eb3\u5fb7\u72d7",
"\u7231\u65af\u57fa\u6469\u72ac",
"\u963f\u62c9\u65af\u52a0\u96ea\u6a47\u72ac",
"\u54c8\u58eb\u5947",
"\u8fbe\u5c14\u9a6c\u63d0\u4e9a",
"\u72ee\u6bdb\u72d7",
"\u5df4\u8f9b\u5409\u72d7",
"\u516b\u54e5\u72ac",
"\u83b1\u6602\u8d1d\u683c\u72d7",
"\u7ebd\u82ac\u5170\u72ac",
"\u5927\u767d\u718a\u72ac",
"\u8428\u6469\u8036\u72ac",
"\u535a\u7f8e\u72ac",
"\u677e\u72ee",
"\u51ef\u65af\u72ac",
"\u5e03\u9c81\u585e\u5c14\u683c\u6797\u82ac\u72ac",
"\u5f6d\u5e03\u6d1b\u514b\u5a01\u5c14\u58eb\u79d1\u57fa\u72ac",
"\u5a01\u5c14\u58eb\u67ef\u57fa\u72ac",
"\u73a9\u5177\u8d35\u5bbe\u72ac",
"\u8ff7\u4f60\u8d35\u5bbe\u72ac",
"\u6807\u51c6\u8d35\u5bbe\u72ac",
"\u58a8\u897f\u54e5\u65e0\u6bdb\u72ac",
"\u7070\u72fc",
"\u767d\u72fc",
"\u7ea2\u592a\u72fc",
"\u72fc",
"\u6fb3\u6d32\u91ce\u72d7",
"\u8c7a",
"\u975e\u6d32\u730e\u72ac",
"\u9b23\u72d7",
"\u7ea2\u72d0\u72f8",
"\u6c99\u72d0",
"\u5317\u6781\u72d0\u72f8",
"\u7070\u72d0\u72f8",
"\u864e\u6591\u732b",
"\u5c71\u732b",
"\u6ce2\u65af\u732b",
"\u66b9\u7f57\u732b",
"\u57c3\u53ca\u732b",
"\u7f8e\u6d32\u72ee",
"\u731e\u7301",
"\u8c79\u5b50",
"\u96ea\u8c79",
"\u7f8e\u6d32\u864e",
"\u72ee\u5b50",
"\u8001\u864e",
"\u730e\u8c79",
"\u68d5\u718a",
"\u7f8e\u6d32\u9ed1\u718a",
"\u51b0\u718a",
"\u61d2\u718a",
"\u7374",
"\u732b\u9f2c",
"\u864e\u7532\u866b",
"\u74e2\u866b",
"\u571f\u9cd6\u866b",
"\u5929\u725b",
"\u9f9f\u7532\u866b",
"\u7caa\u7532\u866b",
"\u7280\u725b\u7532\u866b",
"\u8c61\u7532",
"\u82cd\u8747",
"\u871c\u8702",
"\u8682\u8681",
"\u86b1\u8722",
"\u87cb\u87c0",
"\u7af9\u8282\u866b",
"\u87d1\u8782",
"\u87b3\u8782",
"\u8749",
"\u53f6\u8749",
"\u8349\u873b\u86c9",
"\u873b\u8713",
"\u8c46\u5a18",
"\u4f18\u7ea2\u86f1\u8776",
"\u5c0f\u73af\u8774\u8776",
"\u541b\u4e3b\u8774\u8776",
"\u83dc\u7c89\u8776",
"\u767d\u8774\u8776",
"\u7070\u8776",
"\u6d77\u661f",
"\u6d77\u80c6",
"\u6d77\u9ec4\u74dc\uff1b\u6d77\u53c2",
"\u91ce\u5154",
"\u5154",
"\u5b89\u54e5\u62c9\u5154",
"\u4ed3\u9f20",
"\u523a\u732c",
"\u9ed1\u677e\u9f20",
"\u571f\u62e8\u9f20",
"\u6d77\u72f8",
"\u8c5a\u9f20",
"\u6817\u8272\u9a6c",
"\u6591\u9a6c",
"\u732a",
"\u91ce\u732a",
"\u75a3\u732a",
"\u6cb3\u9a6c",
"\u725b",
"\u6c34\u725b",
"\u91ce\u725b",
"\u516c\u7f8a",
"\u5927\u89d2\u7f8a",
"\u5c71\u7f8a",
"\u72f7\u7f9a",
"\u9ed1\u6591\u7f9a",
"\u77aa\u7f9a",
"\u963f\u62c9\u4f2f\u5355\u5cf0\u9a86\u9a7c",
"\u9a86\u9a7c",
"\u9ec4\u9f20\u72fc",
"\u6c34\u8c82",
"\u81ed\u732b",
"\u9ed1\u8db3\u9f2c",
"\u6c34\u736d",
"\u81ed\u9f2c",
"\u737e",
"\u72b0\u72f3",
"\u6811\u61d2",
"\u7329\u7329",
"\u5927\u7329\u7329",
"\u9ed1\u7329\u7329",
"\u957f\u81c2\u733f",
"\u5408\u8dbe\u733f\u957f\u81c2\u733f",
"\u957f\u5c3e\u7334",
"\u8d64\u7334",
"\u72d2\u72d2",
"\u6052\u6cb3\u7334",
"\u767d\u5934\u53f6\u7334",
"\u75a3\u7334",
"\u957f\u9f3b\u7334",
"\u72e8\uff08\u7f8e\u6d32\u4ea7\u5c0f\u578b\u957f\u5c3e\u7334\uff09",
"\u5377\u5c3e\u7334",
"\u543c\u7334",
"\u4f36\u7334",
"\u8718\u86db\u7334",
"\u677e\u9f20\u7334",
"\u9a6c\u8fbe\u52a0\u65af\u52a0\u73af\u5c3e\u72d0\u7334",
"\u5927\u72d0\u7334",
"\u5370\u5ea6\u5927\u8c61",
"\u975e\u6d32\u8c61",
"\u5c0f\u718a\u732b",
"\u5927\u718a\u732b",
"\u6756\u9c7c",
"\u9cd7\u9c7c",
"\u94f6\u9c91",
"\u4e09\u8272\u523a\u8776\u9c7c",
"\u6d77\u8475\u9c7c",
"\u9c9f\u9c7c",
"\u96c0\u9cdd",
"\u72ee\u5b50\u9c7c",
"\u6cb3\u8c5a",
"\u7b97\u76d8",
"\u957f\u888d",
"\u5b66\u4f4d\u888d",
"\u624b\u98ce\u7434",
"\u539f\u58f0\u5409\u4ed6",
"\u822a\u7a7a\u6bcd\u8230",
"\u5ba2\u673a",
"\u98de\u8247",
"\u796d\u575b",
"\u6551\u62a4\u8f66",
"\u6c34\u9646\u4e24\u7528\u8f66",
"\u6a21\u62df\u65f6\u949f",
"\u8702\u623f",
"\u56f4\u88d9",
"\u5783\u573e\u6876",
"\u653b\u51fb\u6b65\u67aa",
"\u80cc\u5305",
"\u9762\u5305\u5e97",
"\u5e73\u8861\u6728",
"\u70ed\u6c14\u7403",
"\u5706\u73e0\u7b14",
"\u521b\u53ef\u8d34",
"\u73ed\u5353\u7434",
"\u680f\u6746",
"\u6760\u94c3",
"\u7406\u53d1\u5e08\u7684\u6905\u5b50",
"\u7406\u53d1\u5e97",
"\u7272\u53e3\u68da",
"\u6674\u96e8\u8868",
"\u5706\u7b52",
"\u56ed\u5730\u5c0f\u8f66",
"\u68d2\u7403",
"\u7bee\u7403",
"\u5a74\u513f\u5e8a",
"\u5df4\u677e\u7ba1",
"\u6e38\u6cf3\u5e3d",
"\u6c90\u6d74\u6bdb\u5dfe",
"\u6d74\u7f38",
"\u6c99\u6ee9\u8f66",
"\u706f\u5854",
"\u70e7\u676f",
"\u718a\u76ae\u9ad8\u5e3d",
"\u5564\u9152\u74f6",
"\u5564\u9152\u676f",
"\u949f\u5854",
"\uff08\u5c0f\u513f\u7528\u7684\uff09\u56f4\u5634",
"\u4e32\u8054\u81ea\u884c\u8f66",
"\u6bd4\u57fa\u5c3c",
"\u88c5\u8ba2\u518c",
"\u53cc\u7b52\u671b\u8fdc\u955c",
"\u9e1f\u820d",
"\u8239\u5e93",
"\u53cc\u4eba\u96ea\u6a47",
"\u9970\u6263\u5f0f\u9886\u5e26",
"\u9614\u8fb9\u5973\u5e3d",
"\u4e66\u6a71",
"\u4e66\u5e97",
"\u74f6\u76d6",
"\u5f13\u7bad",
"\u8774\u8776\u7ed3\u9886\u7ed3",
"\u94dc\u5236\u724c\u4f4d",
"\u5976\u7f69",
"\u9632\u6ce2\u5824",
"\u94e0\u7532",
"\u626b\u5e1a",
"\u6876",
"\u6263\u73af",
"\u9632\u5f39\u80cc\u5fc3",
"\u52a8\u8f66",
"\u8089\u94fa",
"\u51fa\u79df\u8f66",
"\u5927\u9505",
"\u8721\u70db",
"\u5927\u70ae",
"\u72ec\u6728\u821f",
"\u5f00\u74f6\u5668",
"\u5f00\u886b",
"\u8f66\u955c",
"\u65cb\u8f6c\u6728\u9a6c",
"\u6728\u5320\u7684\u5de5\u5177\u5305",
"\u7eb8\u7bb1",
"\u8f66\u8f6e",
"\u53d6\u6b3e\u673a",
"\u76d2\u5f0f\u5f55\u97f3\u5e26",
"\u5361\u5e26\u64ad\u653e\u5668",
"\u57ce\u5821",
"\u53cc\u4f53\u8239",
"CD\u64ad\u653e\u5668",
"\u5927\u63d0\u7434",
"\u79fb\u52a8\u7535\u8bdd",
"\u94c1\u94fe",
"\u56f4\u680f",
"\u94fe\u7532",
"\u7535\u952f",
"\u7bb1\u5b50",
"\u68b3\u5986\u53f0",
"\u7f16\u949f",
"\u4e2d\u56fd\u6a71\u67dc",
"\u5723\u8bde\u889c",
"\u6559\u5802",
"\u7535\u5f71\u9662",
"\u5207\u8089\u5200",
"\u60ac\u5d16\u5c4b",
"\u6597\u7bf7",
"\u6728\u5c50",
"\u9e21\u5c3e\u9152\u8c03\u9152\u5668",
"\u5496\u5561\u676f",
"\u5496\u5561\u58f6",
"\u87ba\u65cb\u7ed3\u6784\uff08\u697c\u68af\uff09",
"\u7ec4\u5408\u9501",
"\u7535\u8111\u952e\u76d8",
"\u7cd6\u679c",
"\u96c6\u88c5\u7bb1\u8239",
"\u655e\u7bf7\u8f66",
"\u74f6\u585e\u94bb",
"\u77ed\u53f7",
"\u725b\u4ed4\u9774",
"\u725b\u4ed4\u5e3d",
"\u6447\u7bee",
"\u8d77\u91cd\u673a",
"\u5934\u76d4",
"\u677f\u6761\u7bb1",
"\u5c0f\u513f\u5e8a",
"\u7802\u9505",
"\u69cc\u7403",
"\u62d0\u6756",
"\u80f8\u7532",
"\u5927\u575d",
"\u4e66\u684c",
"\u53f0\u5f0f\u7535\u8111",
"\u6709\u7ebf\u7535\u8bdd",
"\u5c3f\u5e03\u6e7f",
"\u6570\u5b57\u65f6\u949f",
"\u6570\u5b57\u624b\u8868",
"\u9910\u684c\u677f",
"\u62b9\u5e03",
"\u6d17\u7897\u673a",
"\u76d8\u5f0f\u5236\u52a8\u5668",
"\u7801\u5934",
"\u72d7\u62c9\u96ea\u6a47",
"\u5706\u9876",
"\u95e8\u57ab",
"\u94bb\u4e95\u5e73\u53f0",
"\u9f13",
"\u9f13\u69cc",
"\u54d1\u94c3",
"\u8377\u5170\u70e4\u7bb1",
"\u7535\u98ce\u6247",
"\u7535\u5409\u4ed6",
"\u7535\u529b\u673a\u8f66",
"\u7ec4\u5408\u7535\u89c6\u67dc",
"\u4fe1\u5c01",
"\u6d53\u7f29\u5496\u5561\u673a",
"\u6251\u9762\u7c89",
"\u5973\u7528\u957f\u56f4\u5dfe",
"\u6587\u4ef6",
"\u6d88\u9632\u8239",
"\u6d88\u9632\u8f66",
"\u706b\u7089\u680f",
"\u65d7\u6746",
"\u957f\u7b1b",
"\u6298\u53e0\u6905",
"\u6a44\u6984\u7403\u5934\u76d4",
"\u53c9\u8f66",
"\u55b7\u6cc9",
"\u94a2\u7b14",
"\u6709\u56db\u6839\u5e37\u67f1\u7684\u5e8a",
"\u8fd0\u8d27\u8f66\u53a2",
"\u5706\u53f7",
"\u714e\u9505",
"\u88d8\u76ae\u5927\u8863",
"\u5783\u573e\u8f66",
"\u9632\u6bd2\u9762\u5177",
"\u6c7d\u6cb9\u6cf5",
"\u9ad8\u811a\u676f",
"\u5361\u4e01\u8f66",
"\u9ad8\u5c14\u592b\u7403",
"\u9ad8\u5c14\u592b\u7403\u8f66",
"\u72ed\u957f\u5c0f\u8239",
"\u9523",
"\u793c\u670d",
"\u94a2\u7434",
"\u6e29\u5ba4",
"\u6563\u70ed\u5668\u683c\u6805",
"\u6742\u8d27\u5e97",
"\u65ad\u5934\u53f0",
"\u5c0f\u53d1\u5939",
"\u5934\u53d1\u55b7\u96fe",
"\u534a\u5c65\u5e26\u88c5\u7532\u8f66",
"\u9524\u5b50",
"\u5927\u7bee\u5b50",
"\u624b\u6447\u9f13\u98ce\u673a",
"\u624b\u63d0\u7535\u8111",
"\u624b\u5e15",
"\u786c\u76d8",
"\u53e3\u7434",
"\u7ad6\u7434",
"\u6536\u5272\u673a",
"\u65a7\u5934",
"\u624b\u67aa\u76ae\u5957",
"\u5bb6\u5ead\u5f71\u9662",
"\u8702\u7a9d",
"\u94a9\u722a",
"\u886c\u88d9",
"\u5355\u6760",
"\u9a6c\u8f66",
"\u6c99\u6f0f",
"iPod",
"\u71a8\u6597",
"\u5357\u74dc\u706f\u7b3c",
"\u725b\u4ed4\u88e4",
"\u5409\u666e\u8f66",
"T\u6064\u886b",
"\u62fc\u56fe",
"\u4eba\u529b\u8f66",
"\u64cd\u7eb5\u6746",
"\u548c\u670d",
"\u62a4\u819d",
"\u8774\u8776\u7ed3",
"\u5927\u8902",
"\u957f\u67c4\u52fa",
"\u706f\u7f69",
"\u7b14\u8bb0\u672c\u7535\u8111",
"\u5272\u8349\u673a",
"\u955c\u5934\u76d6",
"\u5f00\u4fe1\u5200\uff1b\u62c6\u4fe1\u5200",
"\u56fe\u4e66\u9986",
"\u6551\u751f\u8247",
"\u70b9\u706b\u5668",
"\u8c6a\u534e\u8f7f\u8f66",
"\u8fdc\u6d0b\u73ed\u8f6e",
"\u5507\u818f",
"\u5e73\u5e95\u4fbf\u978b",
"\u6d17\u5242",
"\u626c\u58f0\u5668",
"\u653e\u5927\u955c",
"\u952f\u6728\u5382",
"\u78c1\u7f57\u76d8",
"\u90ae\u888b",
"\u4fe1\u7bb1",
"\u5973\u6e38\u6cf3\u8863",
"\u6709\u80a9\u5e26\u6d74\u8863",
"\u7aa8\u4e95\u76d6",
"\u6c99\u7403\uff08\u4e00\u79cd\u6253\u51fb\u4e50\u5668\uff09",
"\u9a6c\u6797\u5df4\u6728\u7434",
"\u9762\u819c",
"\u706b\u67f4",
"\u82b1\u67f1",
"\u8ff7\u5bab",
"\u91cf\u676f",
"\u836f\u7bb1",
"\u5de8\u77f3",
"\u9ea6\u514b\u98ce",
"\u5fae\u6ce2\u7089",
"\u519b\u88c5",
"\u5976\u6876",
"\u8ff7\u4f60\u5df4\u58eb",
"\u8ff7\u4f60\u88d9",
"\u9762\u5305\u8f66\uff1b\u5c0f\u578b\u8d27\u8f66",
"\u5bfc\u5f39",
"\u8fde\u6307\u624b\u5957",
"\u6405\u62cc\u94b5",
"\u6d3b\u52a8\u623f\u5c4b\uff08\u7531\u6c7d\u8f66\u62d6\u62c9\u7684\uff09",
"\u798f\u7279T\u578b\u8f66",
"\u8c03\u5236\u89e3\u8c03\u5668\uff1b\u5149\u732b",
"\u4fee\u9053\u9662",
"\u663e\u793a\u5668",
"\u7535\u74f6\u8f66",
"\u7802\u6d46",
"\u5b66\u58eb",
"\u6e05\u771f\u5bfa",
"\u868a\u5e10",
"\u6469\u6258\u8f66",
"\u5c71\u5730\u81ea\u884c\u8f66",
"\u767b\u5c71\u5e10",
"\u9f20\u6807",
"\u6355\u9f20\u5668",
"\u642c\u5bb6\u8d27\u8f66",
"\u52a8\u7269\u7684\u53e3\u5957",
"\u91d1\u5c5e\u9489\u5b50",
"\u9888\u6258",
"\u9879\u94fe",
"\u4e73\u5934\uff08\u74f6\uff09",
"\u5e73\u677f\u7535\u8111",
"\u65b9\u5c16\u7891",
"\u53cc\u7c27\u7ba1",
"\u5c0f\u9e45\u7b1b\uff1b\u7403\u5f62\u7b1b(\u7ba1\u8eab\u692d\u5706\u5f62)",
"\u91cc\u7a0b\u8868",
"\u6ee4\u6cb9\u5668",
"\u98ce\u7434",
"\u793a\u6ce2\u5668",
"\u7f69\u88d9",
"\u725b\u8f66",
"\u6c27\u6c14\u9762\u7f69",
"\u5305\u88c5",
"\u8239\u6868",
"\u660e\u8f6e",
"\u6302\u9501",
"\u753b\u7b14",
"\u7761\u8863",
"\u5bab\u6bbf",
"\u6392\u7bab",
"\u7eb8\u5dfe",
"\u964d\u843d\u4f1e",
"\u53cc\u6760",
"\u516c\u56ed\u957f\u6905",
"\u505c\u8f66\u6536\u8d39\u8868",
"\u5ba2\u8f66",
"\u9732\u53f0",
"\u4ed8\u8d39\u7535\u8bdd",
"\u57fa\u5ea7",
"\u94c5\u7b14\u76d2",
"\u5377\u7b14\u5200",
"\u9999\u6c34\uff08\u74f6\uff09",
"\u57f9\u517b\u76bf",
"\u590d\u5370\u673a",
"\u62e8\u5f26\u7247",
"\u5c16\u9876\u5934\u76d4",
"\u7528\u5c16\u677f\u6761\u8fde\u6210\u7684\u5c16\u6869\u7bf1\u6805",
"\u76ae\u5361",
"\u6865\u58a9",
"\u5b58\u94b1\u7f50",
"\u836f\u74f6",
"\u6795\u5934",
"\u4e52\u4e53\u7403",
"\u98ce\u8f66",
"\u6d77\u76d7\u8239",
"\u6c34\u7f50",
"\u6728\u5de5\u5228",
"\u5929\u6587\u9986",
"\u5851\u6599\u888b",
"\u677f\u67b6",
"\u7281\u578b\u94f2\u96ea\u673a",
"\u624b\u538b\u76ae\u7897\u6cf5",
"\u5b9d\u4e3d\u6765\u76f8\u673a",
"\u7535\u7ebf\u6746",
"\u8b66\u8f66",
"\u96e8\u62ab",
"\u53f0\u7403\u684c",
"\u5145\u6c14\u996e\u6599\u74f6",
"\u82b1\u76c6",
"\u9676\u5de5\u65cb\u76d8",
"\u7535\u94bb",
"\u7948\u7977\u57ab",
"\u6253\u5370\u673a",
"\u76d1\u72f1",
"\u70ae\u5f39",
"\u6295\u5f71\u4eea",
"\u51b0\u7403",
"\u6c99\u5305",
"\u5c0f\u94b1\u888b\uff1b\u624b\u888b",
"\u7fbd\u7ba1\u7b14",
"\u88ab\u5b50",
"\u8d5b\u8f66",
"\u7403\u62cd",
"\u6563\u70ed\u5668",
"\u6536\u97f3\u673a",
"\u5c04\u7535\u671b\u8fdc\u955c",
"\u96e8\u6876",
"\u4f11\u95f2\u8f66",
"\u5377\u8f74",
"\u53cd\u5c04\u5f0f\u7167\u76f8\u673a",
"\u51b0\u7bb1",
"\u9065\u63a7\u5668",
"\u9910\u5385",
"\u5de6\u8f6e\u624b\u67aa",
"\u6b65\u67aa",
"\u6447\u6905",
"\u7535\u8f6c\u70e4\u8089\u67b6",
"\u6a61\u76ae",
"\u6a44\u6984\u7403",
"\u76f4\u5c3a",
"\u8dd1\u6b65\u978b",
"\u4fdd\u9669\u67dc",
"\u5b89\u5168\u522b\u9488",
"\u76d0\u74f6\uff08\u8c03\u5473\u7528\uff09",
"\u51c9\u978b",
"\u7eb1\u7b3c",
"\u8428\u514b\u65af\u7ba1",
"\u5251\u9798",
"\u79e4",
"\u6821\u8f66",
"\u5e06\u8239",
"\u8bb0\u5206\u724c",
"\u5c4f\u5e55",
"\u87ba\u4e1d",
"\u87ba\u4e1d\u5200",
"\u5b89\u5168\u5e26",
"\u7f1d\u7eab\u673a",
"\u76fe\u724c",
"\u76ae\u978b\u5e97",
"\u969c\u5b50",
"\u8d2d\u7269\u7bee",
"\u8d2d\u7269\u8f66",
"\u94c1\u9539",
"\u6d74\u5e3d",
"\u6d74\u5e18",
"\u6ed1\u96ea\u677f",
"\u6ed1\u96ea\u9762\u7f69",
"\u7761\u888b",
"\u6ed1\u5c3a",
"\u6ed1\u52a8\u95e8",
"\u89d2\u5b50\u8001\u864e\u673a",
"\u6f5c\u6c34\u901a\u6c14\u7ba1",
"\u6469\u6258\u96ea\u6a47\uff1b\u96ea\u5730\u673a\u52a8\u8f66",
"\u626b\u96ea\u673a",
"\u7682\u6db2\u5668",
"\u8db3\u7403",
"\u889c\u5b50",
"\u789f\u5f0f\u592a\u9633\u80fd",
"\u5bbd\u8fb9\u5e3d",
"\u6c64\u7897",
"\u7a7a\u683c\u952e",
"\u7a7a\u95f4\u52a0\u70ed\u5668",
"\u822a\u5929\u98de\u673a",
"\u9505\u94f2\uff1b\u505a\u996d\u7684\u94f2\u5b50",
"\u5feb\u8247",
"\u8718\u86db\u7f51",
"\u7eba\u9524\uff1b\u624b\u7eba\u7528\u7684\u7ed5\u7ebf\u6746",
"\u8dd1\u8f66",
"\u805a\u5149\u706f",
"\u821e\u53f0",
"\u84b8\u6c7d\u673a\u8f66",
"\u94a2\u62f1\u6865",
"\u94a2\u6eda\u7b52",
"\u542c\u8bca\u5668",
"\u5973\u7528\u62ab\u80a9",
"\u77f3\u5934\u5899",
"\u79d2\u8868",
"\u706b\u7089",
"\u8fc7\u6ee4\u5668",
"\u6709\u8f68\u7535\u8f66",
"\u62c5\u67b6",
"\u6c99\u53d1\u5e8a",
"\u4f5b\u5854",
"\u6f5c\u8247",
"\u5957\u88c5",
"\u65e5\u6677",
"\u592a\u9633\u955c",
"\u592a\u9633\u955c",
"\u9632\u6652\u971c",
"\u60ac\u7d22\u6865",
"\u62d6\u628a",
"\u8fd0\u52a8\u886b",
"\u6e38\u6cf3\u88e4",
"\u79cb\u5343",
"\u5f00\u5173",
"\u6ce8\u5c04\u5668\uff1b\u5438\u7ba1",
"\u53f0\u706f",
"\u5766\u514b",
"\u5f55\u97f3\u673a",
"\u8336\u58f6",
"\u6cf0\u8fea",
"\u7535\u89c6",
"\u7f51\u7403\uff1b\u6253\u7f51\u7403\u7684\u7403",
"\u8305\u8349",
"\u5e55\u5e03",
"\u9876\u9488",
"\u6253\u8c37\u673a\uff1b\u8131\u7c92\u673a",
"\u5b9d\u5ea7",
"\u74e6\u5c4b\u9876",
"\u70e4\u9762\u5305\u673a",
"\u70df\u8349\u5e97",
"\u9a6c\u6876",
"\u706b\u70ac",
"\u56fe\u817e\u67f1",
"\u62d6\u8f66\uff1b\u7275\u5f15\u8f66",
"\u73a9\u5177\u5e97",
"\u62d6\u62c9\u673a",
"\u534a\u6302\u6c7d\u8f66",
"\u6258\u76d8",
"\u98ce\u8863",
"\u4e09\u8f6e\u8f66",
"\u4e09\u4f53\u8239",
"\u4e09\u811a\u67b6",
"\u51ef\u65cb\u95e8",
"\u65e0\u8f68\u7535\u8f66",
"\u957f\u53f7",
"\u6d74\u76c6",
"\u65cb\u8f6c\u5f0f\u6805\u95e8",
"\u6253\u5b57\u673a\u952e\u76d8",
"\u4f1e",
"\u72ec\u8f6e\u8f66",
"\u76f4\u7acb\u5f0f\u94a2\u7434",
"\u5438\u5c18\u5668",
"\u82b1\u74f6\uff1b\u88c5\u9970\u74f6",
"\u62f1\u9876",
"\u5929\u9e45\u7ed2",
"\u81ea\u52a8\u552e\u8d27\u673a",
"\u6cd5\u8863\uff1b\u796d\u8863\uff1b\u796d\u670d",
"\u9ad8\u67b6\u6865",
"\u5c0f\u63d0\u7434",
"\u6392\u7403",
"\u677e\u997c\u673a",
"\u6302\u949f",
"\u94b1\u5305\uff1b\u94b1\u5939",
"\u8863\u67dc\u8863\u6a71",
"\u519b\u7528\u98de\u673a",
"\u6d17\u8138\u76c6",
"\u6d17\u8863\u673a",
"\u6c34\u74f6",
"\u6c34\u58f6",
"\u6c34\u5854",
"\u5a01\u58eb\u5fcc\u58f6",
"\u54e8\u5b50",
"\u5047\u53d1",
"\u7eb1\u7a97",
"\u767e\u53f6\u7a97",
"\u6e29\u838e\u9886\u5e26",
"\u8461\u8404\u9152\u74f6",
"\u98de\u673a\u7fc5\u8180",
"\u7092\u83dc\u9505",
"\u6728\u52fa\u5b50\uff1b\u6728\u5934\u52fa\u5b50",
"\u6bdb\u7ec7\u54c1",
"\u539f\u6728\u6805\u680f",
"\u6c89\u8239",
"\u53cc\u6845\u8239",
"\u8499\u53e4\u5305",
"\u7f51\u7ad9\uff1b\u7f51\u9875",
"\u6f2b\u753b",
"\u7eb5\u6a2a\u5b57\u8c1c",
"\u8def\u6807",
"\u4ea4\u901a\u4fe1\u53f7\u706f",
"\u9632\u5c18\u7f69",
"\u83dc\u5355",
"\u76d8\u5b50",
"\u58a8\u897f\u54e5\u9cc4\u68a8\u9171\uff1b\u58a8\u897f\u54e5\u725b\u6cb9\u679c\u9171",
"\u6e05\u7096\u8089\u6c64",
"\u706b\u9505",
"\u4e73\u8102\u86cb\u7cd5\uff1b\u82f1\u56fd\u751c\u70b9",
"\u51b0\u6dc7\u6dcb",
"\u51b0\u68cd\uff1b\u96ea\u7cd5",
"\u6cd5\u5f0f\u9762\u5305",
"\u767e\u5409\u997c",
"\u6912\u76d0\u8106\u997c",
"\u829d\u58eb\u6c49\u5821",
"\u70ed\u72d7",
"\u571f\u8c46\u6ce5",
"\u7ed3\u7403\u7518\u84dd",
"\u897f\u5170\u82b1\uff1b\u7eff\u83dc\u82b1",
"\u83dc\u82b1\uff1b\u82b1\u6930\u83dc",
"\u897f\u846b\u82a6",
"\u91d1\u4e1d\u74dc\uff1b\u610f\u9762\u5357\u74dc\uff1b\u9762\u6761\u74dc",
"\u7eff\u8272\u5c0f\u5357\u74dc\uff1b\u9752\u5357\u74dc",
"\u5357\u74dc",
"\u9ec4\u74dc",
"\u6d0b\u84df\uff1b\u7403\u84df",
"\u751c\u6912",
"\u523a\u68d8\u84df",
"\u8611\u83c7",
"\u7eff\u82f9\u679c",
"\u8349\u8393",
"\u6a58\u5b50",
"\u67e0\u6aac",
"\u65e0\u82b1\u679c",
"\u83e0\u841d",
"\u9999\u8549",
"\u83e0\u841d\u871c",
"\u756a\u8354\u679d",
"\u77f3\u69b4",
"\u5e72\u8349",
"\u57f9\u6839\u86cb\u9171\u610f\u5927\u5229\u9762",
"\u5de7\u514b\u529b\u9171",
"\u751f\u9762\uff1b\u9762\u56e2",
"\u745e\u58eb\u8089\u5305",
"\u62ab\u8428",
"\u9985\u997c",
"\u5377\u997c",
"\u7ea2\u8461\u8404\u9152",
"\u610f\u5f0f\u6d53\u7f29\u5496\u5561",
"\u676f\u5b50",
"\u86cb\u9152",
"\u9ad8\u5c71",
"\u6ce1\u6ce1",
"\u60ac\u5d16",
"\u73ca\u745a\u7901",
"\u95f4\u6b47\u6cc9\uff1b\u95f4\u65ad\u55b7\u53d1\u7684\u6e29\u6cc9",
"\u6e56\u8fb9",
"\u5cac\u89d2\uff1b\u6df1\u5165\u6d77\u4e2d\u7684\u72ed\u957f\u9ad8\u5730",
"\u6c99\u6d32",
"\u6c99\u6ee9",
"\u5ce1\u8c37",
"\u706b\u5c71",
"\u68d2\u7403\u8fd0\u52a8\u5458",
"\u65b0\u90ce",
"\u6f5c\u6c34\u5458",
"\u6cb9\u83dc",
"\u96cf\u83ca",
"\u9ec4\u8272\u6753\u5170",
"\u7389\u7c73",
"\u6a61\u5b50",
"\u73ab\u7470\u679c",
"\u4e03\u53f6\u6811\u679c\u5b9e",
"\u73ca\u745a\u83cc",
"\u6728\u8033",
"\u9e7f\u82b1\u83cc",
"\u81ed\u89d2\u83c7",
"\u5730\u661f",
"\u591a\u53f6\u5947\u679c\u83cc",
"\u725b\u809d\u83cc",
"\u7389\u7c73\u68d2\u5b50",
"\u536b\u751f\u7eb8"
]
}
clip_benchmark/clip_benchmark/datasets/cn_zeroshot_classification_templates.json 0 → 100644
View file @ 1d5a34cf
{
"imagenet1k": [
"{c}\u7684\u7167\u7247\u3002",
"\u8d28\u91cf\u5dee\u7684{c}\u7684\u7167\u7247\u3002",
"\u8bb8\u591a{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u96d5\u5851\u3002",
"\u96be\u4ee5\u770b\u5230{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u4f4e\u5206\u8fa8\u7387\u7167\u7247\u3002",
"{c}\u7684\u6e32\u67d3\u3002",
"\u6d82\u9e26{c}\u3002",
"{c}\u7684\u7cdf\u7cd5\u7167\u7247\u3002",
"{c}\u7684\u88c1\u526a\u7167\u7247\u3002",
"{c}\u7684\u7eb9\u8eab\u3002",
"{c}\u7684\u523a\u7ee3\u7167\u7247\u3002",
"\u5f88\u96be\u770b\u5230{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u660e\u4eae\u7167\u7247\u3002",
"\u4e00\u5f20\u5e72\u51c0\u7684{c}\u7684\u7167\u7247\u3002",
"\u4e00\u5f20\u5305\u542b{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u6df1\u8272\u7167\u7247\u3002",
"{c}\u7684\u624b\u7ed8\u753b\u3002",
"\u6211\u7684{c}\u7684\u7167\u7247\u3002",
"\u4e0d\u81ea\u7136\u7684{c}\u7684\u7167\u7247\u3002",
"\u4e00\u5f20\u9177\u7684{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u7279\u5199\u7167\u7247\u3002",
"{c}\u7684\u9ed1\u767d\u7167\u7247\u3002",
"\u4e00\u5e45{c}\u7684\u753b\u3002",
"\u4e00\u5e45{c}\u7684\u7ed8\u753b\u3002",
"\u4e00\u5f20{c}\u7684\u50cf\u7d20\u7167\u7247\u3002",
"{c}\u7684\u96d5\u50cf\u3002",
"\u4e00\u5f20{c}\u7684\u660e\u4eae\u7167\u7247\u3002",
"{c}\u7684\u88c1\u526a\u7167\u7247\u3002",
"\u4eba\u9020\u7684{c}\u7684\u7167\u7247\u3002",
"\u4e00\u5f20\u5173\u4e8e{c}\u7684\u7167\u7247\u3002",
"\u635f\u574f\u7684{c}\u7684jpeg\u7167\u7247\u3002",
"{c}\u7684\u6a21\u7cca\u7167\u7247\u3002",
"{c}\u7684\u76f8\u7247\u3002",
"\u4e00\u5f20{c}\u7684\u597d\u7167\u7247\u3002",
"{c}\u7684\u6e32\u67d3\u7167\u3002",
"\u89c6\u9891\u6e38\u620f\u4e2d\u7684{c}\u3002",
"\u4e00\u5f20{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u6d82\u9e26\u3002",
"{c}\u7684\u8fd1\u8ddd\u79bb\u7167\u7247\u3002",
"{c}\u7684\u6298\u7eb8\u3002",
"{c}\u5728\u89c6\u9891\u6e38\u620f\u4e2d\u3002",
"{c}\u7684\u8349\u56fe\u3002",
"{c}\u7684\u6d82\u9e26\u7167\u3002",
"{c}\u7684\u6298\u7eb8\u5f62\u72b6\u3002",
"\u4f4e\u5206\u8fa8\u7387\u7684{c}\u7684\u7167\u7247\u3002",
"\u73a9\u5177{c}\u3002",
"{c}\u7684\u526f\u672c\u3002",
"{c}\u7684\u5e72\u51c0\u7684\u7167\u7247\u3002",
"\u4e00\u5f20\u5927{c}\u7684\u7167\u7247\u3002",
"{c}\u7684\u91cd\u73b0\u3002",
"\u4e00\u5f20\u6f02\u4eae\u7684{c}\u7684\u7167\u7247\u3002",
"\u4e00\u5f20\u5947\u602a\u7684{c}\u7684\u7167\u7247\u3002",
"\u6a21\u7cca\u7684{c}\u7684\u7167\u7247\u3002",
"\u5361\u901a{c}\u3002",
"{c}\u7684\u827a\u672f\u4f5c\u54c1\u3002",
"{c}\u7684\u7d20\u63cf\u3002",
"\u523a\u7ee3{c}\u3002",
"{c}\u7684\u50cf\u7d20\u7167\u3002",
"{c}\u7684\u62cd\u7167\u3002",
"{c}\u7684\u635f\u574f\u7684\u7167\u7247\u3002",
"\u9ad8\u8d28\u91cf\u7684{c}\u7684\u7167\u7247\u3002",
"\u6bdb\u7ed2\u73a9\u5177{c}\u3002",
"\u6f02\u4eae\u7684{c}\u7684\u7167\u7247\u3002",
"\u5c0f{c}\u7684\u7167\u7247\u3002",
"\u7167\u7247\u662f\u5947\u602a\u7684{c}\u3002",
"\u6f2b\u753b{c}\u3002",
"{c}\u7684\u827a\u672f\u7167\u3002",
"{c}\u7684\u56fe\u5f62\u3002",
"\u5927{c}\u7684\u7167\u7247\u3002",
"\u9ed1\u767d\u7684{c}\u7684\u7167\u7247\u3002",
"{c}\u6bdb\u7ed2\u73a9\u5177\u3002",
"\u4e00\u5f20{c}\u7684\u6df1\u8272\u7167\u7247\u3002",
"{c}\u7684\u6444\u5f71\u56fe\u3002",
"{c}\u7684\u6d82\u9e26\u7167\u3002",
"\u73a9\u5177\u5f62\u72b6\u7684{c}\u3002",
"\u62cd\u4e86{c}\u7684\u7167\u7247\u3002",
"\u9177\u9177\u7684{c}\u7684\u7167\u7247\u3002",
"\u7167\u7247\u91cc\u7684\u5c0f{c}\u3002",
"{c}\u7684\u523a\u9752\u3002"
]
}
clip_benchmark/clip_benchmark/datasets/cupl_prompts.json 0 → 100644
View file @ 1d5a34cf
This source diff could not be displayed because it is too large. You can view the blob instead.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment