Skip to content

GitLab

Commit 26e59280 authored by wanglch's avatar wanglch
Browse files

Initial commit

parents
Pipeline #2674 failed with stages
in 0 seconds
Hide whitespace changes
Inline Side-by-side
Showing with 4802 additions and 0 deletions
clip_benchmark/LICENSE 0 → 100644
View file @ 26e59280
MIT License
Copyright (c) 2022, Mehdi Cherti
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
clip_benchmark/MANIFEST.in 0 → 100644
View file @ 26e59280
include AUTHORS.rst
include CONTRIBUTING.rst
include HISTORY.rst
include LICENSE
include README.rst
recursive-include tests *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]
recursive-include * *.json
recursive-include docs *.rst conf.py Makefile make.bat *.jpg *.png *.gif
clip_benchmark/Makefile 0 → 100644
View file @ 26e59280
.PHONY: clean clean-build clean-pyc clean-test coverage dist docs help install lint lint/flake8
.DEFAULT_GOAL := help
define BROWSER_PYSCRIPT
import os, webbrowser, sys
from urllib.request import pathname2url
webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1])))
endef
export BROWSER_PYSCRIPT
define PRINT_HELP_PYSCRIPT
import re, sys
for line in sys.stdin:
match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line)
if match:
target, help = match.groups()
print("%-20s %s" % (target, help))
endef
export PRINT_HELP_PYSCRIPT
BROWSER := python -c "$$BROWSER_PYSCRIPT"
help:
@python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
clean: clean-build clean-pyc clean-test ## remove all build, test, coverage and Python artifacts
clean-build: ## remove build artifacts
rm -fr build/
rm -fr dist/
rm -fr .eggs/
find . -name '*.egg-info' -exec rm -fr {} +
find . -name '*.egg' -exec rm -f {} +
clean-pyc: ## remove Python file artifacts
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f {} +
find . -name '__pycache__' -exec rm -fr {} +
clean-test: ## remove test and coverage artifacts
rm -fr .tox/
rm -f .coverage
rm -fr htmlcov/
rm -fr .pytest_cache
lint/flake8: ## check style with flake8
flake8 clip_benchmark tests
lint: lint/flake8 ## check style
test-all: ## run tests on every Python version with tox
tox
coverage: ## check code coverage quickly with the default Python
coverage run --source clip_benchmark setup.py test
coverage report -m
coverage html
$(BROWSER) htmlcov/index.html
docs: ## generate Sphinx HTML documentation, including API docs
rm -f docs/clip_benchmark.rst
rm -f docs/modules.rst
sphinx-apidoc -o docs/ clip_benchmark
$(MAKE) -C docs clean
$(MAKE) -C docs html
$(BROWSER) docs/_build/html/index.html
servedocs: docs ## compile the docs watching for changes
watchmedo shell-command -p '*.rst' -c '$(MAKE) -C docs html' -R -D .
release: dist ## package and upload a release
twine upload dist/*
dist: clean ## builds source and wheel package
python setup.py sdist
python setup.py bdist_wheel
ls -l dist
install: ## [Local development] Upgrade pip, install requirements, install package.
python -m pip install -U pip
python -m pip install -e .
install-dev: ## [Local development] Install test requirements
python -m pip install -r requirements-test.txt
test: ## [Local development] Run unit tests
python -m pytest -x -s -v tests
clip_benchmark/README.md 0 → 100644
View file @ 26e59280
# InternVL for Zero-Shot Image Classification & Image-Text Retrieval
This folder contains the implementation of InternVL 1.0 for zero-shot image classification and zero-shot image-text retrieval, which corresponds to Section 4.3 of our [InternVL 1.0 paper](https://arxiv.org/pdf/2312.14238).
We mainly use [CLIP Benchmark](https://github.com/LAION-AI/CLIP_benchmark) to evaluate the performance of InternVL. Thanks for this great work.
## 🛠️ Installation
First, follow the [installation guide](../INSTALLATION.md) to perform some basic installations.
In addition, using this codebase requires executing the following steps:
- Install other requirements:
```bash
pip install -r requirements.txt
```
- Install `clip_benchmark` using development mode:
```bash
python setup.py develop
# You can also add the current directory to PYTHONPATH instead.
export PYTHONPATH="${PYTHONPATH}:$(pwd)"
```
## 📦 Data Preparation
This codebase will automatically download the required dataset. If the dataset fails to download automatically, please refer to this [code](./clip_benchmark/datasets/builder.py) for manual downloading.
## 📦 Model Preparation
| model name | type | download | size |
| ------------------------ | :---------: | ------------------------------------------------------------------------------------------ | :-----: |
| internvl_c_13b_224px.pth | pytorch | 🤗 [HF link](https://huggingface.co/OpenGVLab/InternVL/blob/main/internvl_c_13b_224px.pth) | 25.4 GB |
| InternVL-14B-224px | huggingface | 🤗 [HF link](https://huggingface.co/OpenGVLab/InternVL-14B-224px) | 27.7 GB |
Please download the above model weights and place them in the `pretrained/` folder.
You can download either the PyTorch version or the Hugging Face version based on your needs.
```sh
cd pretrained/
wget https://huggingface.co/OpenGVLab/InternVL/resolve/main/internvl_c_13b_224px.pth
# pip install -U huggingface_hub
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternVL-14B-224px --local-dir InternVL-14B-224px
```
The directory structure is:
```sh
pretrained
├── internvl_c_13b_224px.pth
└── InternVL-14B-224px/
```
## 📊 Evaluation: Zero-Shot Image Classification
### ImageNet variants and ObjectNet
| model name | IN-1K | IN-A | IN-R | IN-V2 | IN-Sketch | ObjectNet | ∆ | average |
| :--------: | :---: | :--: | :--: | :---: | :-------: | :-------: | :-: | :-----: |
| InternVL-C | 83.2 | 83.8 | 95.5 | 77.3 | 73.9 | 80.6 | 0.8 | 82.4 |
<details>
<summary>[InternVL-C] ImageNet-1K val</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" \
--task "zeroshot_classification" --dataset "imagenet1k" --dataset_root ./data/imagenet-1k/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet1k", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.83178, "acc5": 0.97322, "mean_per_class_recall": 0.83204}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-A</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" \
--task "zeroshot_classification" --dataset "imagenet-a" --dataset_root ./data/imagenet-a/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet-a", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.8377333333333333, "acc5": 0.9558666666666666, "mean_per_class_recall": 0.8183934468491632}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-R</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" \
--task "zeroshot_classification" --dataset "imagenet-r" --dataset_root ./data/imagenet-r/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet-r", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9549666666666666, "acc5": 0.9918333333333333, "mean_per_class_recall": 0.9460205918105684}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-V2</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" \
--task "zeroshot_classification" --dataset "imagenetv2" --dataset_root ./data/imagenetv2/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenetv2", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7726, "acc5": 0.9468, "mean_per_class_recall": 0.7738000000000001}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-Sketch</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" \
--task "zeroshot_classification" --dataset "imagenet_sketch" --dataset_root ./data/imagenet-sketch/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet_sketch", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7385879070133035, "acc5": 0.9199827074613374, "mean_per_class_recall": 0.7386403921568627}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] ObjectNet</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" \
--task "zeroshot_classification" --dataset "objectnet" --dataset_root ./data/objectnet-1.0/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "objectnet", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.8059114891784215, "acc5": 0.9387853989447615, "mean_per_class_recall": 0.797040815749882}, "language": "en"}
```
</details>
### Multilingual ImageNet-1K
| model name | IN-1K (EN) | IN-1K (ZH) | IN-1K (JP) | IN-1K (AR) | IN-1K (IT) | average |
| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :-----: |
| InternVL-C | 83.2 | 64.5 | 61.5 | 44.9 | 65.7 | 64.0 |
<details>
<summary>[InternVL-C] ImageNet-1K val (ZH, Chinese)</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "cn" \
--task "zeroshot_classification" --dataset "imagenet1k" --dataset_root ./data/imagenet-1k/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet1k", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.6446, "acc5": 0.87842, "mean_per_class_recall": 0.6444200000000001}, "language": "cn"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-1K val (JP, Japanese)</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "jp" \
--task "zeroshot_classification" --dataset "imagenet1k" --dataset_root ./data/imagenet-1k/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet1k", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.61488, "acc5": 0.81146, "mean_per_class_recall": 0.6140599999999999}, "language": "jp"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-1K val (AR, Arabic)</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "ar" \
--task "zeroshot_classification" --dataset "imagenet1k" --dataset_root ./data/imagenet-1k/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet1k", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.4486, "acc5": 0.66418, "mean_per_class_recall": 0.44764}, "language": "ar"}
```
</details>
<details>
<summary>[InternVL-C] ImageNet-1K val (IT, Italian)</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "it" \
--task "zeroshot_classification" --dataset "imagenet1k" --dataset_root ./data/imagenet-1k/ \
--model internvl_c_classification --pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "imagenet1k", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.65686, "acc5": 0.85254, "mean_per_class_recall": 0.6557799999999999}, "language": "it"}
```
</details>
### Other Datasets
<img width="1219" alt="image" src="https://github.com/OpenGVLab/InternVL/assets/23737120/5de18a6c-8979-432d-bcb6-eb7796b4a08f">
<details>
<summary>[InternVL-C] CIFAR-10</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "cifar10" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "cifar10", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9935, "acc5": 0.9996, "mean_per_class_recall": 0.9935}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] CIFAR-100</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "cifar100" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "cifar100", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9315, "acc5": 0.9925, "mean_per_class_recall": 0.9314}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] MNIST</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "mnist" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "mnist", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.806, "acc5": 0.9743, "mean_per_class_recall": 0.8028667364603377}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Caltech-101</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "caltech101" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "caltech101", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.8949037620297463, "acc5": 0.9847987751531059, "mean_per_class_recall": 0.9548738053818752}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] SUN397</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "sun397" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "sun397", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7600180223256157, "acc5": 0.9623370174890119, "mean_per_class_recall": 0.7641970904214413}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] FGVC Aircraft</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "fgvc_aircraft" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "fgvc_aircraft", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.5271527152715272, "acc5": 0.9426942694269427, "mean_per_class_recall": 0.5255169340463458}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Country-211</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "country211" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "country211", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.34080568720379145, "acc5": 0.6048815165876777, "mean_per_class_recall": 0.3406635071090047}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Stanford Cars</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "cars" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "cars", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9416739211540853, "acc5": 0.99950254943415, "mean_per_class_recall": 0.9416684924576828}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Birdsnap</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "birdsnap" --dataset_root ./data/birdsnap/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "birdsnap", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7203252032520325, "acc5": 0.9636856368563685, "mean_per_class_recall": 0.7027551020408164}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] DTD</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "dtd" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "dtd", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7074468085106383, "acc5": 0.9367021276595745, "mean_per_class_recall": 0.7079787234042553}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Eurosat</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "eurosat" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "eurosat", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7937407407407407, "acc5": 0.9984074074074074, "mean_per_class_recall": 0.8013766666666665}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] FER2013</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "fer2013" --dataset_root ./data/fer2013 --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "fer2013", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.561994984675397, "acc5": 0.9732516021175815, "mean_per_class_recall": 0.5305440899910082}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Flowers-102</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "vtab/flowers" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "vtab/flowers", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.8606277443486746, "acc5": 0.953651000162628, "mean_per_class_recall": 0.8563173902114554}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Food-101</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "food101" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "food101", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9526336633663366, "acc5": 0.9954851485148515, "mean_per_class_recall": 0.9527524752475246}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] GTSRB</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "gtsrb" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "gtsrb", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.6548693586698338, "acc5": 0.9089469517022961, "mean_per_class_recall": 0.5775180283147926}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Pets</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "pets" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "pets", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9604796947397111, "acc5": 0.9991823385118561, "mean_per_class_recall": 0.9602545246926443}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Rendered SST2</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "renderedsst2" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "renderedsst2", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.6792970895112576, "acc5": NaN, "mean_per_class_recall": 0.6792944097041282}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] Resisc45</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "vtab/resisc45" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "vtab/resisc45", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7422631328360577, "acc5": 0.9663545468973179, "mean_per_class_recall": 0.7481098478511045}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] STL10</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "stl10" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "stl10", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9945, "acc5": 1.0, "mean_per_class_recall": 0.9945}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] VOC2007</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_classification" \
--dataset "voc2007" --dataset_root ./data/ --model internvl_c_classification \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "voc2007", "model": "internvl_c_classification", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_classification",
"metrics": {"acc1": 0.7997462606837606, "acc5": 0.9795005341880342, "mean_per_class_recall": 0.9048832641726575}, "language": "en"}
```
</details>
## 📊 Evaluation: Zero-Shot Image-Text Retrieval
### Flickr30K & COCO
<table>
<tr align=center>
<td rowspan="3" align=center><b>model</b></td>
<td colspan="6" align=center><b>Flickr30K</b></td>
<td colspan="6" align=center><b>COCO</b></td>
<td rowspan="3" align=center><b>avg</b></td>
</tr>
<tr align=center>
<td colspan="3" align=center><b>image-to-text</b></td>
<td colspan="3" align=center><b>text-to-image</b></td>
<td colspan="3" align=center><b>image-to-text</b></td>
<td colspan="3" align=center><b>text-to-image</b></td>
</tr>
<tr>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
</tr>
<tr align=center>
<td>InternVL-C</td>
<td>94.7</td>
<td>99.6</td>
<td>99.9</td>
<td>81.7</td>
<td>96.0</td>
<td>98.2</td>
<td>70.6</td>
<td>89.0</td>
<td>93.5</td>
<td>54.1</td>
<td>77.3</td>
<td>84.6</td>
<td>86.6</td>
</tr>
<tr align=center>
<td>InternVL-G</td>
<td>95.7</td>
<td>99.7</td>
<td>99.9</td>
<td>85.0</td>
<td>97.0</td>
<td>98.6</td>
<td>74.9</td>
<td>91.3</td>
<td>95.2</td>
<td>58.6</td>
<td>81.3</td>
<td>88.0</td>
<td>88.8</td>
</tr>
</table>
<details>
<summary>[InternVL-C] Flickr30K</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_retrieval" \
--dataset "flickr30k" --dataset_root ./data/flickr30k --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "flickr30k", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.8166000247001648, "text_retrieval_recall@1": 0.9470000267028809,
"image_retrieval_recall@5": 0.9603999853134155, "text_retrieval_recall@5": 0.9959999918937683,
"image_retrieval_recall@10": 0.9819999933242798, "text_retrieval_recall@10": 0.9990000128746033}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-C] COCO</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_retrieval" \
--dataset "mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.5411835312843323, "text_retrieval_recall@1": 0.7059999704360962,
"image_retrieval_recall@5": 0.7731707096099854, "text_retrieval_recall@5": 0.8902000188827515,
"image_retrieval_recall@10": 0.8463414907455444, "text_retrieval_recall@10": 0.9354000091552734}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-G] Flickr30K</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_retrieval" \
--dataset "flickr30k" --dataset_root ./data/flickr30k --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json
```
Expected results:
```
{"dataset": "flickr30k", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.8497999906539917, "text_retrieval_recall@1": 0.9570000171661377,
"image_retrieval_recall@5": 0.9700000286102295, "text_retrieval_recall@5": 0.996999979019165,
"image_retrieval_recall@10": 0.98580002784729, "text_retrieval_recall@10": 0.9990000128746033}, "language": "en"}
```
</details>
<details>
<summary>[InternVL-G] COCO</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "en" --task "zeroshot_retrieval" \
--dataset "mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json
```
Expected results:
```
{"dataset": "mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.5858056545257568, "text_retrieval_recall@1": 0.7491999864578247,
"image_retrieval_recall@5": 0.813194751739502, "text_retrieval_recall@5": 0.9129999876022339,
"image_retrieval_recall@10": 0.8795281648635864, "text_retrieval_recall@10": 0.9521999955177307}, "language": "en"}
```
</details>
### Flickr30K-CN & COCO-CN
<table>
<tr align=center>
<td rowspan="3" align=center><b>model</b></td>
<td colspan="6" align=center><b>Flickr30K-CN</b></td>
<td colspan="6" align=center><b>COCO-CN</b></td>
<td rowspan="3" align=center><b>avg</b></td>
</tr>
<tr align=center>
<td colspan="3" align=center><b>image-to-text</b></td>
<td colspan="3" align=center><b>text-to-image</b></td>
<td colspan="3" align=center><b>image-to-text</b></td>
<td colspan="3" align=center><b>text-to-image</b></td>
</tr>
<tr>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
<td>R@1</td>
<td>R@5</td>
<td>R@10</td>
</tr>
<tr align=center>
<td>InternVL-C</td>
<td>90.3</td>
<td>98.8</td>
<td>99.7</td>
<td>75.1</td>
<td>92.9</td>
<td>96.4</td>
<td>68.8</td>
<td>92.0</td>
<td>96.7</td>
<td>68.9</td>
<td>91.9</td>
<td>96.5</td>
<td>89.0</td>
</tr>
<tr align=center>
<td>InternVL-G</td>
<td>92.9</td>
<td>99.4</td>
<td>99.8</td>
<td>77.7</td>
<td>94.8</td>
<td>97.3</td>
<td>71.4</td>
<td>93.9</td>
<td>97.7</td>
<td>73.8</td>
<td>94.4</td>
<td>98.1</td>
<td>90.9</td>
</tr>
</table>
<details>
<summary>[InternVL-C] Flickr30K-CN</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "cn" --task "zeroshot_retrieval" \
--dataset "flickr30k" --dataset_root ./data/flickr30k --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "flickr30k", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.7509999871253967, "text_retrieval_recall@1": 0.902999997138977,
"image_retrieval_recall@5": 0.9290000200271606, "text_retrieval_recall@5": 0.9879999756813049,
"image_retrieval_recall@10": 0.9638000130653381, "text_retrieval_recall@10": 0.996999979019165}, "language": "cn"}
```
</details>
<details>
<summary>[InternVL-C] COCO-CN</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "cn" --task "zeroshot_retrieval" \
--dataset "mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json
```
Expected results:
```
{"dataset": "mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.6885090470314026, "text_retrieval_recall@1": 0.6880000233650208,
"image_retrieval_recall@5": 0.9192782640457153, "text_retrieval_recall@5": 0.9200000166893005,
"image_retrieval_recall@10": 0.9648622870445251, "text_retrieval_recall@10": 0.9670000076293945}, "language": "cn"}
```
</details>
<details>
<summary>[InternVL-G] Flickr30K-CN</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "cn" --task "zeroshot_retrieval" \
--dataset "flickr30k" --dataset_root ./data/flickr30k --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json
```
Expected results:
```
{"dataset": "flickr30k", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.7767999768257141, "text_retrieval_recall@1": 0.9290000200271606,
"image_retrieval_recall@5": 0.9476000070571899, "text_retrieval_recall@5": 0.9940000176429749,
"image_retrieval_recall@10": 0.9728000164031982, "text_retrieval_recall@10": 0.9980000257492065}, "language": "cn"}
```
</details>
<details>
<summary>[InternVL-G] COCO-CN</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --language "cn" --task "zeroshot_retrieval" \
--dataset "mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json
```
Expected results:
```
{"dataset": "mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval",
"metrics": {"image_retrieval_recall@1": 0.7378917336463928, "text_retrieval_recall@1": 0.7139999866485596,
"image_retrieval_recall@5": 0.9439696073532104, "text_retrieval_recall@5": 0.9390000104904175,
"image_retrieval_recall@10": 0.9810066223144531, "text_retrieval_recall@10": 0.9769999980926514}, "language": "cn"}
```
</details>
### XTD
| model name | EN | ES | FR | ZH | IT | KO | RU | JP | average |
| :--------: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :-----: |
| InternVL-C | 97.3 | 95.7 | 95.1 | 95.6 | 96.0 | 92.2 | 93.3 | 95.5 | 95.1 |
| InternVL-G | 98.6 | 97.7 | 96.5 | 96.7 | 96.9 | 95.1 | 94.8 | 96.1 | 96.6 |
<details>
<summary>[InternVL-C] XTD</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=en
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=es
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=fr
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=zh
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=it
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=ko
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=ru
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_c_retrieval \
--pretrained ./pretrained/internvl_c_13b_224px.pth --output result.json --language=jp
```
Expected results:
```
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.7670000195503235, "text_retrieval_recall@1": 0.7480000257492065, "image_retrieval_recall@5": 0.9200000166893005, "text_retrieval_recall@5": 0.921999990940094, "image_retrieval_recall@10": 0.9670000076293945, "text_retrieval_recall@10": 0.9729999899864197}, "language": "en"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.7059999704360962, "text_retrieval_recall@1": 0.7009999752044678, "image_retrieval_recall@5": 0.9020000100135803, "text_retrieval_recall@5": 0.8960000276565552, "image_retrieval_recall@10": 0.9430000185966492, "text_retrieval_recall@10": 0.9570000171661377}, "language": "es"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6970000267028809, "text_retrieval_recall@1": 0.6899999976158142, "image_retrieval_recall@5": 0.8830000162124634, "text_retrieval_recall@5": 0.8889999985694885, "image_retrieval_recall@10": 0.9350000023841858, "text_retrieval_recall@10": 0.9509999752044678}, "language": "fr"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6480000019073486, "text_retrieval_recall@1": 0.6710000038146973, "image_retrieval_recall@5": 0.8759999871253967, "text_retrieval_recall@5": 0.8769999742507935, "image_retrieval_recall@10": 0.9419999718666077, "text_retrieval_recall@10": 0.9559999704360962}, "language": "zh"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6790000200271606, "text_retrieval_recall@1": 0.7039999961853027, "image_retrieval_recall@5": 0.8989999890327454, "text_retrieval_recall@5": 0.8999999761581421, "image_retrieval_recall@10": 0.9440000057220459, "text_retrieval_recall@10": 0.9599999785423279}, "language": "it"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.5830000042915344, "text_retrieval_recall@1": 0.5920000076293945, "image_retrieval_recall@5": 0.8399999737739563, "text_retrieval_recall@5": 0.8360000252723694, "image_retrieval_recall@10": 0.9079999923706055, "text_retrieval_recall@10": 0.921999990940094}, "language": "ko"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6430000066757202, "text_retrieval_recall@1": 0.6439999938011169, "image_retrieval_recall@5": 0.8510000109672546, "text_retrieval_recall@5": 0.8640000224113464, "image_retrieval_recall@10": 0.9169999957084656, "text_retrieval_recall@10": 0.9330000281333923}, "language": "ru"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_c_retrieval", "pretrained": "./pretrained/internvl_c_13b_224px.pth", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6330000162124634, "text_retrieval_recall@1": 0.6759999990463257, "image_retrieval_recall@5": 0.875, "text_retrieval_recall@5": 0.8989999890327454, "image_retrieval_recall@10": 0.9359999895095825, "text_retrieval_recall@10": 0.9549999833106995}, "language": "jp"}
```
</details>
<details>
<summary>[InternVL-G] XTD</summary>
```bash
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=en
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=es
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=fr
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=zh
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=it
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=ko
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=ru
CUDA_VISIBLE_DEVICES=0 python3 clip_benchmark/cli.py eval --model_type internvl --task "zeroshot_retrieval" \
--dataset "multilingual_mscoco_captions" --dataset_root ./data/mscoco_captions --model internvl_g_retrieval_hf \
--pretrained ./pretrained/InternVL-14B-224px --output result_g.json --language=jp
```
Expected results:
```
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.8119999766349792, "text_retrieval_recall@1": 0.7979999780654907, "image_retrieval_recall@5": 0.9470000267028809, "text_retrieval_recall@5": 0.9480000138282776, "image_retrieval_recall@10": 0.9829999804496765, "text_retrieval_recall@10": 0.9860000014305115}, "language": "en"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.7549999952316284, "text_retrieval_recall@1": 0.7450000047683716, "image_retrieval_recall@5": 0.9350000023841858, "text_retrieval_recall@5": 0.925000011920929, "image_retrieval_recall@10": 0.9660000205039978, "text_retrieval_recall@10": 0.9769999980926514}, "language": "es"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.7450000047683716, "text_retrieval_recall@1": 0.7279999852180481, "image_retrieval_recall@5": 0.9179999828338623, "text_retrieval_recall@5": 0.9190000295639038, "image_retrieval_recall@10": 0.9620000123977661, "text_retrieval_recall@10": 0.9649999737739563}, "language": "fr"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6980000138282776, "text_retrieval_recall@1": 0.6949999928474426, "image_retrieval_recall@5": 0.9120000004768372, "text_retrieval_recall@5": 0.9110000133514404, "image_retrieval_recall@10": 0.9620000123977661, "text_retrieval_recall@10": 0.9670000076293945}, "language": "zh"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.7329999804496765, "text_retrieval_recall@1": 0.7450000047683716, "image_retrieval_recall@5": 0.9309999942779541, "text_retrieval_recall@5": 0.9309999942779541, "image_retrieval_recall@10": 0.9639999866485596, "text_retrieval_recall@10": 0.968999981880188}, "language": "it"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6430000066757202, "text_retrieval_recall@1": 0.6470000147819519, "image_retrieval_recall@5": 0.8790000081062317, "text_retrieval_recall@5": 0.8769999742507935, "image_retrieval_recall@10": 0.9419999718666077, "text_retrieval_recall@10": 0.9509999752044678}, "language": "ko"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6850000023841858, "text_retrieval_recall@1": 0.6899999976158142, "image_retrieval_recall@5": 0.8740000128746033, "text_retrieval_recall@5": 0.8920000195503235, "image_retrieval_recall@10": 0.9390000104904175, "text_retrieval_recall@10": 0.9480000138282776}, "language": "ru"}
{"dataset": "multilingual_mscoco_captions", "model": "internvl_g_retrieval_hf", "pretrained": "./pretrained/InternVL-14B-224px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@1": 0.6850000023841858, "text_retrieval_recall@1": 0.703000009059906, "image_retrieval_recall@5": 0.9020000100135803, "text_retrieval_recall@5": 0.9100000262260437, "image_retrieval_recall@10": 0.9539999961853027, "text_retrieval_recall@10": 0.9610000252723694}, "language": "jp"}
```
</details>
## Original README of CLIP Benchmark
[![pypi](https://img.shields.io/pypi/v/clip_benchmark.svg)](https://pypi.python.org/pypi/clip_benchmark)
The goal of this repo is to evaluate CLIP-like models on a standard set
of datasets on different tasks such as zero-shot classification and zero-shot
retrieval.
Below we show the average rank (1 is the best, lower is better) of different CLIP models, evaluated
on different datasets.
![benchmark.png](benchmark.png)
The current detailed results of the benchmark can be seen [here](benchmark/README.md)
or directly in the [notebook](benchmark/results.ipynb).
### Features
- Support for zero-shot classification and zero-shot retrieval
- Support for [OpenCLIP](https://github.com/mlfoundations/open_clip) pre-trained models
- Support various datasets from [torchvision](https://pytorch.org/vision/stable/datasets.html), [tensorflow datasets](https://www.tensorflow.org/datasets), and [VTAB](https://github.com/google-research/task_adaptation).
- Support [Japanese CLIP by rinna](https://github.com/rinnakk/japanese-clip)
### How to install?
`pip install clip-benchmark`
### How to use?
To evaluate we recommend to create a models.txt like
```
ViT-B-32,openai
```
to get the list of datasets
```
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txt
```
Then to run
```
clip_benchmark eval --pretrained_model models.txt \
--dataset "webdatasets.txt" \
--dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" \
--output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Then to get the full table
```
clip_benchmark build benchmark_*.json --output benchmark.csv
```
#### Command line interface (CLI)
The easiest way to benchmark the models is using the CLI, `clip_benchmark`.
You can specify the model to use, the dataset and the task to evaluate on. Once it is done, evaluation is performed and
the results are written into a JSON file.
#### Using other models than openclip
It is possible to use other models than openclip ones. For example japanese-clip is supported
Here is an example of use
```
>>> python3 clip_benchmark/cli.py eval \
--model_type "ja_clip" \ # flag to use japanese-clip
--pretrained "rinna/japanese-cloob-vit-b-16" \ # now, we have `rinna/japanese-cloob-vit-b-16` or `rinna/japanese-clip-vit-b-16`.
--language "jp" \
--task "zeroshot_classification" \
--dataset "imagenet1k" \
--dataset_root {ROOT_PATH}
>>> cat result.json
{"dataset": "imagenet1k", "model": "ViT-B-32-quickgelu", "pretrained": "rinna/japanese-cloob-vit-b-16", "task": "zeroshot_classification", "metrics": {"acc1": 0.54636, "acc5": 0.72856, "mean_per_class_recall": 0.54522}, "language": "jp"}
```
#### How to add other CLIP models
Please follow these steps:
1. Add a identity file to load model in `clip_benchmark/models`
2. Define a loading function, that returns a tuple (model, transform, tokenizer). Please see `clip_benchmark/models/open_clip.py` as an example.
3. Add the function into `TYPE2FUNC` in `clip_benchmark/models/__init__.py`
Remarks:
- The new tokenizer/model must enable to do the following things as https://github.com/openai/CLIP#usage
- `tokenizer(texts).to(device)` ... `texts` is a list of string
- `model.encode_text(tokenized_texts)` ... `tokenized_texts` is a output from `tokenizer(texts).to(device)`
- `model.encode_image(images)` ... `images` is a image tensor by the `transform`
#### CIFAR-10 example
Here is an example for CIFAR-10 zero-shot classification using OpenCLIP's pre-trained model on LAION-400m:
`clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
By default, the dataset is downloaded into `--dataset_root`, which by default is `root`.
Here is the content of `result.json` after the evaluation is done:
```json
{
"dataset": "cifar10", "model": "ViT-B-32-quickgelu",
"pretrained": "laion400m_e32", "task": "zeroshot_classification",
"metrics": {"acc1": 0.9074, "acc5": 0.998}
}
```
#### VOC2007 example
Here is another example with VOC2007, which is a multi-label classification dataset.
`clip_benchmark eval --dataset=voc2007_multilabel --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
Here is the content of `result.json` after the evaluation is done:
```json
{"dataset": "voc2007_multilabel", "model": "ViT-B-32-quickgelu", "pretrained": "laion400m_e32", "task": "zeroshot_classification", "metrics": {"mean_average_precision": 0.7627869844436646}}
```
Here, we compute the mean average precision or mAP, more details about that metric [here](https://fangdahan.medium.com/calculate-mean-average-precision-map-for-multi-label-classification-b082679d31be) in the context of multi-label classification.
#### VTAB example
Here is an example on how to run it on [VTAB](https://github.com/google-research/task_adaptation) classification tasks.
First, you need to install VTAB's dedicated package.
`pip install task_adaptation==0.1`
Then, you can run it by providing the full dataset name.
Example with `eurosat`:
`clip_benchmark eval --dataset=vtab/eurosat --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
See [clip_benchmark/datasets/builder.py#L634](clip_benchmark/datasets/builder.py#L634) for the full list of
VTAB dataset collection.
#### TensorFlow dataset example
Here is an example on how to run it on [Tensorflow datasets](https://www.tensorflow.org/datasets).
First, you need to install `tfds-nightly` and `timm`.
`pip install timm tfds-nightly`
The name of the dataset follows the template `tfds/<DATASET_NAME>`.
Example with `cifar10`:
`clip_benchmark eval --dataset=tfds/cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
#### COCO captions example
Here is an example for COCO captions zero-shot retrieval:
`clip_benchmark eval --dataset=mscoco_captions --task=zeroshot_retrieval --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64`
Note that for using COCO, you also need to install `pycocotools` (e.g., using `pip install pycocotools`).
#### Webdataset example
Here is an example on how to run it on [webdatasets](https://github.com/webdataset/webdataset).
First, you need to install `webdataset`.
`pip install webdataset`
##### Creating a webdataset
You can either convert an already supported CLIP_benchmark dataset to webdataset format, or manually create your own with the same file structure. For already supported datasets use the CLI command `clip_benchmark_export_wds` as in this example:
```
$ clip_benchmark_export_wds --dataset cifar10 --split train --dataset_root DATA_DIR/ --output wds_cifar10/
$ clip_benchmark_export_wds --dataset cifar10 --split test --dataset_root DATA_DIR/ --output wds_cifar10/
```
which will convert the train and test splits for CIFAR-10 (downloaded to `DATA_DIR/`) and save the webdataset to `wds_cifar10/` (upload to Huggingface Hub must be done manually for now). Retrieval datasets are also supported with the `--retrieval` flag.
For other datasets, data must be stored with the following file structure:
```
root_dir/
train/
nshards.txt
0.tar
1.tar
...
test/
nshards.txt
0.tar
...
classnames.txt
zeroshot_classification_templates.txt
dataset_type.txt
```
Each split should be contained in its own folder and `nshards.txt` should contain a single integer corresponding to the number of TAR files. The TAR files should follow webdataset format, with an image file (.webp, .png, or .jpg) and a label (.cls) for each example. Classnames and templates are required for zeroshot classification evaluation, with each classname or template on its own line. Dataset type is required for distinguishing zeroshot retrieval evaluation: the file should just contain the text `retrieval`.
##### Evaluating on a webdataset
The name of the dataset follows the template `wds/<DATASET_NAME>`. Note that the dataset name currently only affects the name in the results output - classnames and templates are loaded directly from the included files. The dataset root directory can be either a local path to the `root_dir` as specified above, or an HTTP URL pointing to a Huggingface Hub dataset file tree.
Example with `vtab/cifar10`:
```
$ clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root ROOT_DIR/wds_vtab-cifar10/
$ clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root https://huggingface.co/datasets/clip-benchmark/wds_vtab-cifar10/tree/main
```
All other arguments remain the same as in the other examples. See `https://huggingface.co/clip-benchmark` for a full list of datasets that have already been uploaded to Huggingface.
### Evaluate mulitple models on multiple datasets
For the purpose of benchmarking, it is possible to run the CLI with multiple
pre-trained models on multiple datasets.
#### Pretrained models and datasets list as arguments
For models, we can provide list of pretrained model names in the form of 'model,pretrained' (so `model` and `pretrained` are comma separated). For datasets, we can provide a list of datasets. For languages, we can provide a list of languages.
Example:
```bash
clip_benchmark eval --pretrained_model ViT-B-32-quickgelu,laion400m_e32 ViT-L-14,laion400m_e32 \
--dataset cifar10 cifar100 --dataset_root "clip_benchmark_datasets/{dataset}" --language en jp \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Note that `--dataset_root` and `--output` can be now in the form of a template that depends on the dataset/model/language/task (for `--output`) and dataset name (for `--dataset_root`).
Note that If the benchmark fails at some point, it is possible to resume it by skipping already evaluated models using `--skip_existing`.
#### Pretrained models and datasets list as files
We can also provide a path to files with models (each line is in the form of 'model,pretrained' where `model` and `pretrained` are comma separated) and datasets list (one dataset per line):
```bash
clip_benchmark eval --pretrained_model benchmark/models.txt \
--dataset benchmark/datasets.txt --dataset_root "clip_benchmark_datasets/{dataset}" \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Examples are available in [benchmark/datasets.txt](benchmark/datasets.txt) and [benchmark/models.txt](benchmark/models.txt)
#### Model and dataset collections
We can also provide model collection names (`openai`, `openclip_base`, `openclip_multilingual`, `openclip_full` are supported) or dataset collection names (`vtab`, `vtab+`, `retrieval`, `imagenet_robustness` are supported):
```bash
clip_benchmark eval --pretrained_model openai openclip_base --dataset vtab+ retrieval \
--dataset_root "clip_benchmark_datasets/{dataset}" --not quiet \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
#### Development
For development, you can also do this:
```bash
git clone https://github.com/LAION-AI/CLIP_benchmark
cd CLIP_benchmark
python setup.py install
```
### Credits
- Thanks to [OpenCLIP](https://github.com/mlfoundations/open_clip) authors, zero-shot accuracy code is adapted from there and pre-trained models are used in the command line interface.
- Thanks to [SLIP](https://github.com/facebookresearch/SLIP) authors, some zero-shot templates and classnames are from there.
- Thanks to [Wise-ft](https://github.com/mlfoundations/wise-ft) authors, Imagenet robustness datasets code is adapted from there
- Thanks to [LiT](https://arxiv.org/abs/2111.07991.pdf) authors, some zero-shot templates and classnames of VTAB datasets are from there.
- This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) project template. Thanks to the author.
clip_benchmark/benchmark/README.md 0 → 100644
View file @ 26e59280
# Benchmark
the benchmark results are available in [benchmark.csv](benchmark.csv).
You can visualize the results in the [notebook](results.ipynb)
# How to reproduce the CLIP benchmark results
## Webdataset evaluation: VTAB+ and retrieval datasets (MSCOCO, Flickr8k, Flickr30k)
```bash
clip_benchmark eval --pretrained_model openai openclip_base \
--dataset "webdatasets.txt" \
--dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" \
--output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
Once the evaluation finishes, you can construct a CSV with all the results:
```bash
clip_benchmark build benchmark_*.json --output benchmark.csv
```
*Notes:* Pascal VOC 2007 multilabel is not yet included in the webdataset test suite. Multilingual support with webdataset is in progress.
## Alternative: Local download
```bash
clip_benchmark eval --pretrained_model openai openclip_base --dataset vtab+ retrieval \
--dataset_root "clip_benchmark_datasets/{dataset}" \
--output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
(Change `--dataset_root` accordingly)
## Multilingual ImageNet benchmark
To run the multilingual ImageNet benchmark, use:
```bash
clip_benchmark eval --pretrained_model openclip_multilingual openclip_base openai --dataset imagenet1k --language cn it jp en ar\
--dataset_root "clip_benchmark_datasets/{dataset}" \
--output "multilingual_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
(Change `--dataset_root` accordingly)
## Multilingual MS-COCO benchmark
To run the multilingual MS-COCO benchmark, use:
```bash
clip_benchmark eval --pretrained_model openclip_multilingual openclip_base openai --dataset multilingual_mscoco_captions --language es it ko pl ru tr zh en \
--dataset_root "clip_benchmark_datasets/{dataset}" \
--output "multilingual_{dataset}_{pretrained}_{model}_{language}_{task}.json"
```
(Change `--dataset_root` accordingly)
clip_benchmark/benchmark/benchmark.csv 0 → 100644
View file @ 26e59280
acc1,acc5,mean_per_class_recall,dataset,model,pretrained,task,mean_average_precision,image_retrieval_recall@5,text_retrieval_recall@5,model_fullname
0.0232340494791666,0.1152615017361111,0.0242046402834269,vtab/dsprites_label_orientation,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.4460852605182502,0.9469211479520758,0.3940612716631316,fer2013,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.7738666666666667,0.9362666666666668,0.7345750490593081,imagenet-a,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4457446808510638,0.7585106382978724,0.449468085106383,vtab/dtd,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7521562425290105,0.9608106368501388,0.7512820500659019,sun397,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,voc2007_multilabel,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,0.796263575553894,,,ViT-B-32 laion2b_s34b_b79k
0.7557,0.9386,0.7554000000000001,vtab/cifar100,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.665,0.89844,0.66506,imagenet1k,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1889597536837475,0.819793270288102,0.1681683759314615,vtab/dmlab,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.4955,0.8497,0.5259367109048434,mnist,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5175536361103371,0.9651713569239344,0.5055924943171644,fer2013,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6246666789267521,0.91313422954558,0.6346910684418603,sun397,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.9070591441809758,0.995366584900518,0.9070894162634328,vtab/pets,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.5609659540775931,0.8069675376088677,0.5169427663206733,gtsrb,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.0313720703125,0.1469997829861111,0.0320098582855241,vtab/dsprites_label_x_position,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.3628226797787339,0.6565765212046711,0.4033487053539641,vtab/svhn,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.4362628661916072,0.7091844813935075,0.370358797280688,gtsrb,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5535494775660725,0.9136447449293178,0.5591738787984386,vtab/svhn,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4417465274038979,0.7008721869279638,0.4272611771980346,objectnet,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7174074074074074,0.9561111111111112,0.7202760268647987,vtab/eurosat,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.501821060965954,0.7586698337292161,0.4394378810250903,gtsrb,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.9321340964840557,0.9978195693649496,0.9313974108097984,vtab/pets,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.4934283452098179,0.7430720506730008,0.4353727539920664,gtsrb,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8522353714661407,0.963346482577252,0.944284654839904,vtab/caltech101,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
,,,flickr30k,ViT-B-32-quickgelu,laion400m_e32,zeroshot_retrieval,,0.8546000123023987,0.9409999847412108,ViT-B-32-quickgelu laion400m_e32
0.1178600823045267,0.5817283950617284,0.1208963734895024,vtab/smallnorb_label_elevation,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.76664,0.9485,0.76656,imagenet1k,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.7679,0.9386,0.7581398074696393,mnist,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.2883263009845288,,0.3645688070267072,vtab/kitti_closest_vehicle_distance,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.1854185418541854,0.4452445244524452,0.1875846702317291,fgvc_aircraft,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0197618272569444,0.1133355034722222,0.0176459217575104,vtab/dsprites_label_orientation,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
,,,flickr8k,ViT-B-16-plus-240,laion400m_e32,zeroshot_retrieval,,0.873199999332428,0.9549999833106995,ViT-B-16-plus-240 laion400m_e32
0.551063829787234,0.8356382978723405,0.5499999999999999,vtab/dtd,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8391,0.9729,0.8388,vtab/cifar100,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.482037037037037,0.935,0.493913656654034,vtab/eurosat,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.6828752642706131,0.8578630671653927,0.6628139602370955,vtab/flowers,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5844813935075218,0.820744259699129,0.5442606899522975,gtsrb,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.50628662109375,,0.5062329426609509,vtab/pcam,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.6932,0.8887,0.6785851251044699,imagenet-r,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.2304230423042304,0.5295529552955296,0.2319696969696969,fgvc_aircraft,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.605010986328125,,0.605165824864527,vtab/pcam,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr8k,ViT-B-16,laion400m_e32,zeroshot_retrieval,,0.8575999736785889,0.9409999847412109,ViT-B-16 laion400m_e32
0.7835420393559929,0.9242153195641568,0.7862863094545812,vtab/flowers,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.9467,0.999,0.9466,vtab/cifar10,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.6961,0.9086,0.6957000000000001,imagenetv2,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.4345209817893903,0.707680126682502,0.400638686800972,gtsrb,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.0317789713541666,0.147705078125,0.0324921668671336,vtab/dsprites_label_x_position,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.7871,0.9505,0.7775130720394369,mnist,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.63334,0.88778,0.63284,imagenet1k,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6031914893617021,0.8867021276595745,0.6042553191489363,vtab/dtd,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.985875,0.99975,0.9864999999999998,stl10,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2636018957345971,0.5149289099526067,0.2629857819905213,country211,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5132978723404256,0.7787234042553192,0.5095744680851063,vtab/dtd,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.4342160768689946,1.0,0.2167811161586062,vtab/diabetic_retinopathy,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5512,0.8156,0.5509,imagenetv2,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7105220361034315,0.8590014636526264,0.6857234783638442,vtab/flowers,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6867793368519779,0.9372344925244128,0.6845985471139586,sun397,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.5016666666666667,0.959074074074074,0.511474800858698,vtab/eurosat,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.8468,0.9733,0.8471000000000001,vtab/cifar100,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,voc2007_multilabel,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,0.801436722278595,,,ViT-H-14 laion2b_s32b_b79k
0.9172,0.9975,0.9172,vtab/cifar10,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6981352410026298,0.9398183055335896,0.6849982147927691,sun397,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.77972,0.95216,0.77952,imagenet1k,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.029541015625,0.1558973524305555,0.0292117961574083,vtab/dsprites_label_x_position,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5,0.7502771179730799,0.4499584478173962,gtsrb,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7590333333333333,0.9128666666666668,0.7444515544684158,imagenet-r,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.4690721649484536,0.931178601281694,0.4334946917742944,fer2013,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.485626220703125,,0.4856418903784925,vtab/pcam,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.9438539111474517,0.9986372308530936,0.9434557685576204,vtab/pets,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,flickr8k,ViT-L-14,laion400m_e32,zeroshot_retrieval,,0.8984000086784363,0.9649999737739563,ViT-L-14 laion400m_e32
0.0972839506172839,0.5397530864197531,0.0973286727349915,vtab/smallnorb_label_elevation,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5159912109375,,0.5157975788193991,vtab/pcam,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.7824519230769231,0.9688835470085472,0.8629106820310023,voc2007,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0293918185763888,0.150390625,0.0306791200330755,vtab/dsprites_label_x_position,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5710666666666666,0.8348,0.5639196371233688,imagenet-a,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6408566721581549,,0.6410094956864107,renderedsst2,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1134156378600823,0.5579423868312757,0.1146512139135114,vtab/smallnorb_label_elevation,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1590499230261711,0.8403782713877281,0.1701282125753662,vtab/dmlab,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
,,,voc2007_multilabel,ViT-B-32,laion2b_e16,zeroshot_classification,0.7927550077438354,,,ViT-B-32 laion2b_e16
,,,mscoco_captions,ViT-B-16,laion400m_e32,zeroshot_retrieval,,0.6364254355430603,0.7961999773979187,ViT-B-16 laion400m_e32
0.758985200845666,0.8894129126687266,0.7455642498757189,vtab/flowers,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-32,openai,zeroshot_classification,0.7601363658905029,,,ViT-B-32 openai
0.6968,0.9081,0.6974,imagenetv2,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8893431452711911,0.9940038157536112,0.884512216368383,vtab/pets,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7021,0.9244,0.703,vtab/cifar100,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5072222222222222,0.9255555555555556,0.489609055911575,vtab/eurosat,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7067545304777595,,0.7068315384169996,renderedsst2,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.665743087897188,0.8816640138340309,0.6655023529411764,imagenet_sketch,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.5155555555555555,0.9201851851851852,0.526225901185735,vtab/eurosat,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
,,,flickr30k,ViT-B-32,openai,zeroshot_retrieval,,0.8338000178337097,0.9490000009536744,ViT-B-32 openai
0.1593333333333333,0.9299333333333332,0.1673057808855792,vtab/clevr_closest_object_distance,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7464,0.9285,0.7471,vtab/cifar100,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
,,,mscoco_captions,ViT-L-14,laion400m_e32,zeroshot_retrieval,,0.6805678009986877,0.8216000199317932,ViT-L-14 laion400m_e32
0.9307713273371492,0.9980921231943308,0.9330900923082088,vtab/pets,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6188,0.8745,0.6202000000000001,imagenetv2,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.4230383776454636,0.7030792509186661,0.4230898039215686,imagenet_sketch,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7910657051282052,0.9600026709401708,0.8052125971178338,voc2007,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1674950516824279,0.8040906091928745,0.1782001238774596,vtab/dmlab,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8310322156476002,0.9529914529914528,0.903296243135675,vtab/caltech101,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1094650205761317,0.5571193415637861,0.1098932195729559,vtab/smallnorb_label_elevation,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5716,0.8386,0.5721,imagenetv2,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1080658436213991,0.5204115226337449,0.108510287776451,vtab/smallnorb_label_elevation,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.8982,0.9963,0.8995000000000001,vtab/cifar10,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.9381302807304442,0.9986372308530936,0.9372081115891412,vtab/pets,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5819,0.8386,0.5815,imagenetv2,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.6666666666666666,0.941904761904762,0.6759805529181834,vtab/resisc45,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,0.7846916913986206,,,ViT-B-16-plus-240 laion400m_e32
0.5363831083338246,0.7926467409459804,0.53684,imagenet_sketch,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7930605646063923,0.986693197363512,0.79277195221544,cars,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.6457990115321252,,0.6459400874297956,renderedsst2,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6445715707001617,0.9434149981345604,0.6469166001999892,cars,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.503753662109375,,0.5035515136049098,vtab/pcam,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.5706822372464659,0.9138752304855562,0.5886888789913961,vtab/svhn,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.7761084401709402,0.9418402777777778,0.8508423918048074,voc2007,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1884360189573459,0.4163507109004739,0.1883412322274882,country211,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.97075,0.999375,0.971875,stl10,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1508247195953375,0.8088849791071036,0.1720986035113953,vtab/dmlab,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.2629716428404031,1.0,0.2194476579174435,vtab/diabetic_retinopathy,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.4630454824830977,0.817340196681008,0.4869931863892911,vtab/svhn,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,mscoco_captions,ViT-B-32,laion2b_e16,zeroshot_retrieval,,0.6467413306236267,0.7950000166893005,ViT-B-32 laion2b_e16
0.270042194092827,,0.3517916468296155,vtab/kitti_closest_vehicle_distance,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7082,0.9169,0.709,imagenetv2,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6352380952380953,0.9225396825396824,0.6419889996880412,vtab/resisc45,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5404255319148936,0.8398936170212766,0.5367021276595745,vtab/dtd,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.2236286919831223,,0.3717030717825018,vtab/kitti_closest_vehicle_distance,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5850793650793651,0.910952380952381,0.5919202546199539,vtab/resisc45,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7567441239316239,0.9461805555555556,0.7914514618991711,voc2007,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.3319333333333333,0.9534,0.3193231509666999,vtab/clevr_count_all,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5661724327292696,,0.5658672775172254,renderedsst2,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2686357243319268,,0.3735376764204429,vtab/kitti_closest_vehicle_distance,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.1856168902573125,0.82608313173521,0.1925396140565279,vtab/dmlab,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.8504273504273504,0.9681130834976988,0.9394300669046936,vtab/caltech101,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7768934212162666,0.983957219251337,0.777448930592225,cars,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.3048171481253841,0.7613706207744315,0.3503741918499782,vtab/svhn,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.96575,0.999375,0.966625,stl10,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.6219482421875,,0.6220625731388706,vtab/pcam,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
,,,flickr30k,ViT-L-14,laion400m_e32,zeroshot_retrieval,,0.9082000255584716,0.977999985218048,ViT-L-14 laion400m_e32
0.72734,0.9293,0.72694,imagenet1k,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7766666666666666,0.9304333333333332,0.7605432098970494,imagenet-r,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.6137,0.8644,0.6146999999999999,imagenetv2,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.2877333333333333,0.9075333333333332,0.2821869879006831,vtab/clevr_count_all,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6190740740740741,0.962037037037037,0.6309344676440423,vtab/eurosat,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0313313802083333,0.1560872395833333,0.031335128135504,vtab/dsprites_label_x_position,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1149794238683127,0.5761316872427984,0.115904554765943,vtab/smallnorb_label_elevation,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7760611481541714,0.9046999512115792,0.7813275676765017,vtab/flowers,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.8865,0.9695666666666668,0.8748158307459671,imagenet-r,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
,,,mscoco_captions,ViT-H-14,laion2b_s32b_b79k,zeroshot_retrieval,,0.734306275844574,0.8604000210762024,ViT-H-14 laion2b_s32b_b79k
0.7524,0.9418,0.7528999999999999,vtab/cifar100,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.2618666666666667,0.5721333333333334,0.2839019082932269,imagenet-a,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.9345852505907224,0.9988807362268376,0.9351484667320789,cars,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.4286709389802173,0.898300362217888,0.392124222029496,fer2013,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.68352,0.91864,0.68396,imagenet1k,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.1123456790123456,0.5467489711934156,0.1133381228564946,vtab/smallnorb_label_elevation,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4932893159621922,0.7566861207726621,0.4940521568627451,imagenet_sketch,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5009752020061299,0.932153803287824,0.449919123283142,fer2013,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7549,0.946,0.7545,imagenet1k,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.0346001519097222,0.1698269314236111,0.0339338982697889,vtab/dsprites_label_x_position,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.71777753849467,0.9007214385700442,0.7007187288769688,objectnet,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.7294,0.9415,0.7332910901261492,mnist,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0342746310763888,0.1416965060763889,0.034365141983162,vtab/dsprites_label_orientation,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.210639793766112,1.0,0.2335698910327324,vtab/diabetic_retinopathy,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8392504930966469,0.9510190664036818,0.9090841082001052,vtab/caltech101,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2841068917018284,,0.4076831334000694,vtab/kitti_closest_vehicle_distance,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6813829787234043,0.925531914893617,0.6829787234042553,vtab/dtd,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6909,0.91432,0.69156,imagenet1k,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.2424666666666666,0.7876,0.2312503165591237,vtab/clevr_count_all,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.67002,0.90424,0.67026,imagenet1k,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-B-16,laion400m_e32,zeroshot_retrieval,,0.881600022315979,0.9679999947547911,ViT-B-16 laion400m_e32
0.8787,0.9709333333333332,0.8651131734542029,imagenet-r,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.62918,0.87652,0.6289,imagenet1k,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.5469522240527183,,0.5464163192635053,renderedsst2,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.4425531914893617,0.7638297872340426,0.4430851063829787,vtab/dtd,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.5149133196941962,0.7512652094325402,0.5017005288059357,objectnet,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.4855,0.8418,0.4575381785680641,mnist,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6550016151609777,0.8503284160654678,0.6433184742666794,objectnet,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7744,0.9471,0.7737999999999998,vtab/cifar100,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7920656634746922,0.9787339883099117,0.7926165075935756,cars,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.2081575246132208,,0.1791674645508319,vtab/kitti_closest_vehicle_distance,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0260687934027777,0.1259223090277778,0.0268337475785376,vtab/dsprites_label_orientation,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6922,0.9403,0.6883700135057857,mnist,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1143209876543209,0.5373662551440329,0.1138628810502107,vtab/smallnorb_label_elevation,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6769,0.9012,0.6781,imagenetv2,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7683627136752137,0.952590811965812,0.8035754023986389,voc2007,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.8504273504273504,0.953155818540434,0.9440706929933655,vtab/caltech101,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6434889751181566,0.9243614027989776,0.6527406670624641,sun397,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.3311163895486936,0.7250989707046714,0.3196447660118034,gtsrb,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6521841655367565,0.8748649020416986,0.6524090196078433,imagenet_sketch,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1469194312796208,0.3500473933649289,0.1470142180094787,country211,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.96875,0.99975,0.9695,stl10,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6908043501669,0.8805319263486594,0.6736647184602601,objectnet,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.984375,0.999875,0.9849999999999998,stl10,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.5544309249488533,0.7926671691611931,0.5363732822578842,objectnet,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7555422008547008,0.9489182692307692,0.830992972154603,voc2007,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5293285385839769,0.78944369117098,0.5286741176470588,imagenet_sketch,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.7889957264957265,0.9573317307692308,0.8054447759101763,voc2007,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.675285506739982,0.9369310554094562,0.6824513086557495,sun397,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.111275720164609,0.5548971193415638,0.1102182875784799,vtab/smallnorb_label_elevation,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,flickr8k,ViT-B-32,laion2b_e16,zeroshot_retrieval,,0.8574000000953674,0.9319999814033508,ViT-B-32 laion2b_e16
0.5989555292344136,0.8103262625174976,0.586320244179702,objectnet,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
,,,mscoco_captions,ViT-g-14,laion2b_s12b_b42k,zeroshot_retrieval,,0.7239903807640076,0.853600025177002,ViT-g-14 laion2b_s12b_b42k
0.7101,0.9209,0.7106999999999999,vtab/cifar100,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.189873417721519,,0.2568338943834677,vtab/kitti_closest_vehicle_distance,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,mscoco_captions,ViT-B-32,laion2b_s34b_b79k,zeroshot_retrieval,,0.654218316078186,0.7982000112533569,ViT-B-32 laion2b_s34b_b79k
0.1186008230452674,0.560082304526749,0.1176434679315522,vtab/smallnorb_label_elevation,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5634266886326195,,0.5635892536622082,renderedsst2,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
,,,flickr8k,ViT-L-14,laion2b_s32b_b82k,zeroshot_retrieval,,0.9147999882698059,0.9670000076293945,ViT-L-14 laion2b_s32b_b82k
0.0237358940972222,0.1186116536458333,0.0218059467808842,vtab/dsprites_label_orientation,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1316147176001874,1.0,0.230780955782727,vtab/diabetic_retinopathy,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.8599676657132197,0.9912946150976246,0.8615494787047302,cars,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.5569148936170213,0.8361702127659575,0.5563829787234043,vtab/dtd,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
,,,flickr30k,ViT-B-32,laion2b_s34b_b79k,zeroshot_retrieval,,0.8835999965667725,0.9629999995231628,ViT-B-32 laion2b_s34b_b79k
0.0715256620576517,1.0,0.2196341124982623,vtab/diabetic_retinopathy,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5699,0.8279,0.5675693520579019,mnist,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.5940803382663847,0.917174480785972,0.5967107342155968,cars,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.98275,1.0,0.982875,stl10,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.4199524940617577,0.6838479809976247,0.393417229364651,gtsrb,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.0633231778767283,1.0,0.2107285863733837,vtab/diabetic_retinopathy,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.9497,0.9963,0.9497,vtab/cifar10,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.76548,0.95168,0.7656000000000001,imagenet1k,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.2742616033755274,,0.405770386325608,vtab/kitti_closest_vehicle_distance,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.2380126552613077,1.0,0.233320813717688,vtab/diabetic_retinopathy,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.2307582938388625,0.4708056872037914,0.2308530805687204,country211,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.2633333333333333,0.574,0.2790514196577626,imagenet-a,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,voc2007_multilabel,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,0.7621888518333435,,,ViT-B-32-quickgelu laion400m_e32
0.199,0.7285333333333334,0.1948019376127608,vtab/clevr_count_all,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5376190476190477,0.8671428571428571,0.5417275836816031,vtab/resisc45,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.0302191840277777,0.1466335720486111,0.0300817149220147,vtab/dsprites_label_x_position,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,flickr8k,ViT-g-14,laion2b_s12b_b42k,zeroshot_retrieval,,0.9175999760627747,0.9739999771118164,ViT-g-14 laion2b_s12b_b42k
0.157994281944139,0.8222564328128437,0.1661260800820199,vtab/dmlab,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1812322274881516,0.4011374407582938,0.181563981042654,country211,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.9272,0.9988,0.9272000000000002,vtab/cifar10,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.9401,0.9992,0.9405,vtab/cifar10,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.7331380360909304,1.0,0.206266837915639,vtab/diabetic_retinopathy,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.2587904360056259,,0.3397102822066769,vtab/kitti_closest_vehicle_distance,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2166666666666666,0.5310666666666667,0.2348332752565157,imagenet-a,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.99375,1.0,0.993625,stl10,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.087368174361378,1.0,0.2520359622114083,vtab/diabetic_retinopathy,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.1236939151813153,0.6480485556238476,0.1332639799678708,vtab/svhn,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.5782537067545305,,0.5785203520352036,renderedsst2,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0588477366255144,0.268395061728395,0.0601925439678357,vtab/smallnorb_label_azimuth,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6367,0.9218,0.6276012948452819,mnist,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.0313856336805555,0.1553955078125,0.0307666564982354,vtab/dsprites_label_x_position,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.9569,0.9963,0.9572,vtab/cifar10,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.4973871733966746,0.7704671417260491,0.4655936453259506,gtsrb,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5853968253968254,0.896984126984127,0.5932699170625959,vtab/resisc45,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.1419837255333186,0.8292060699362217,0.16573619863836,vtab/dmlab,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.612762451171875,,0.6128028129314551,vtab/pcam,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.1945898394545854,0.8513305476138113,0.1635130198328585,vtab/dmlab,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.7259778950659286,0.9500616069294004,0.7127784763377728,sun397,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.4661333333333333,0.7688,0.4728184257301568,imagenet-a,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.6371428571428571,0.9277777777777778,0.6463008933552572,vtab/resisc45,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.1111111111111111,,0.2722929936305732,vtab/kitti_closest_vehicle_distance,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6105248678496336,0.8503998899565721,0.6103113725490197,imagenet_sketch,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.8732624693376942,0.9926410466067048,0.8695864391489154,vtab/pets,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1486,0.9095333333333332,0.1443124682607226,vtab/clevr_closest_object_distance,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0243462456597222,0.1128879123263889,0.0249444752915047,vtab/dsprites_label_orientation,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.596238872840889,0.8439741397944546,0.5964254901960785,imagenet_sketch,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6276595744680851,0.9069148936170212,0.6324468085106383,vtab/dtd,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-16,openai,zeroshot_classification,0.788827121257782,,,ViT-B-16 openai
,,,flickr30k,ViT-L-14,openai,zeroshot_retrieval,,0.8715999722480774,0.9739999771118164,ViT-L-14 openai
0.2505250525052505,0.6012601260126013,0.2483244206773618,fgvc_aircraft,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5559259259259259,0.8868518518518519,0.5469811732579133,vtab/eurosat,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.9427636958299264,0.9983646770237122,0.9434000102313552,vtab/pets,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1746174617461746,0.45004500450045,0.1753386809269162,fgvc_aircraft,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-B-16,openai,zeroshot_retrieval,,0.855400025844574,0.9629999995231628,ViT-B-16 openai
0.8934314527119106,0.9956391387298992,0.8906060208128682,vtab/pets,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.6266666666666667,0.9611111111111112,0.6380077170682305,vtab/eurosat,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5596,0.8341,0.5602,imagenetv2,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.481439996855902,0.763229774607479,0.4822572549019607,imagenet_sketch,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.2450666666666666,0.8254,0.1666666666666666,vtab/clevr_closest_object_distance,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-H-14,laion2b_s32b_b79k,zeroshot_retrieval,,0.9409999847412108,0.9929999709129332,ViT-H-14 laion2b_s32b_b79k
,,,voc2007_multilabel,ViT-L-14,openai,zeroshot_classification,0.7903817892074585,,,ViT-L-14 openai
0.4267205349679576,0.9361939258846476,0.3989364402674789,fer2013,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.6960663515824705,0.9390183349577946,0.6804128851625355,sun397,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.3318666666666666,0.6664,0.3409181030776702,imagenet-a,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.9083,0.9944,0.9082,vtab/cifar10,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
,,,voc2007_multilabel,ViT-L-14-336,openai,zeroshot_classification,0.8035513162612915,,,ViT-L-14-336 openai
0.5922666666666667,0.8565333333333334,0.5810468077571583,imagenet-a,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6730158730158731,0.938095238095238,0.6781338184038964,vtab/resisc45,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7354112959925005,1.0,0.1999235670840696,vtab/diabetic_retinopathy,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,flickr30k,ViT-B-16-plus-240,laion400m_e32,zeroshot_retrieval,,0.8894000053405762,0.9710000157356262,ViT-B-16-plus-240 laion400m_e32
,,,flickr8k,ViT-B-32,openai,zeroshot_retrieval,,0.805400013923645,0.9139999747276306,ViT-B-32 openai
0.5897858319604613,,0.5895015488390944,renderedsst2,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.5388,0.8212,0.5362716338708938,imagenet-a,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,mscoco_captions,ViT-B-16,openai,zeroshot_retrieval,,0.5836865305900574,0.7681999802589417,ViT-B-16 openai
,,,flickr8k,ViT-B-32,laion2b_s34b_b79k,zeroshot_retrieval,,0.8629999756813049,0.9409999847412109,ViT-B-32 laion2b_s34b_b79k
0.99425,0.999875,0.9945,stl10,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5612707437000615,0.8849492931776275,0.5565312569182044,vtab/svhn,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1097119341563786,0.5409053497942387,0.1081320376350696,vtab/smallnorb_label_elevation,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.805221688034188,0.9551282051282052,0.8491537874687232,voc2007,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,flickr8k,ViT-L-14-336,openai,zeroshot_retrieval,,0.8795999884605408,0.9390000104904175,ViT-L-14-336 openai
0.1716,0.9095333333333332,0.1619443104834767,vtab/clevr_closest_object_distance,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1573333333333333,0.6964666666666667,0.1495026928557915,vtab/clevr_count_all,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,mscoco_captions,ViT-B-16-plus-240,laion400m_e32,zeroshot_retrieval,,0.6620951890945435,0.8101999759674072,ViT-B-16-plus-240 laion400m_e32
0.8336,0.9666,0.8325000000000001,vtab/cifar100,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.0984362139917695,0.5293827160493827,0.0977694426163014,vtab/smallnorb_label_elevation,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,voc2007_multilabel,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,0.8066232800483704,,,ViT-g-14 laion2b_s12b_b42k
0.6087301587301587,0.9147619047619048,0.6152225643499313,vtab/resisc45,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.0550617283950617,0.2645267489711934,0.055379474051494,vtab/smallnorb_label_azimuth,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1676666666666666,0.8819333333333333,0.1954414937508546,vtab/clevr_closest_object_distance,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0200330946180555,0.1053195529513889,0.0222572574909185,vtab/dsprites_label_orientation,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5526123046875,,0.5526478181769814,vtab/pcam,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,mscoco_captions,ViT-L-14,openai,zeroshot_retrieval,,0.6108356714248657,0.7918000221252441,ViT-L-14 openai
,,,mscoco_captions,ViT-B-32-quickgelu,laion400m_e32,zeroshot_retrieval,,0.6084766387939453,0.7675999999046326,ViT-B-32-quickgelu laion400m_e32
0.7537811026183119,0.8923402179216132,0.7256696912813558,vtab/flowers,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.1659634317862166,,0.3247233185334074,vtab/kitti_closest_vehicle_distance,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.0193684895833333,0.1180826822916666,0.0197744129948227,vtab/dsprites_label_orientation,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.9037884982284,0.9970019078768056,0.9039815014388496,vtab/pets,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.3742,0.7294,0.3706020613065869,mnist,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.6746527403897922,0.8668030580381177,0.6650572756513145,objectnet,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.765625,0.959869123931624,0.8071517771314477,voc2007,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.546031746031746,0.902063492063492,0.5542849348347576,vtab/resisc45,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.0427924262152777,0.1601019965277778,0.0430371102717507,vtab/dsprites_label_x_position,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6957142857142857,0.9571428571428572,0.706242238089474,vtab/resisc45,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.6514814814814814,0.9551851851851852,0.6638062361650154,vtab/eurosat,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8475333333333334,0.9550666666666666,0.8331685531673508,imagenet-r,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.1902,0.9085333333333332,0.1387289271305892,vtab/clevr_closest_object_distance,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.3852181929932391,0.7822295636140135,0.379296565517112,vtab/svhn,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.3182464454976303,0.5937914691943128,0.3175829383886256,country211,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6659,0.9224,0.6665327286676481,mnist,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6327693607655879,0.8639588123170037,0.6325447058823529,imagenet_sketch,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.2279620853080568,0.486303317535545,0.2282938388625592,country211,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.764,0.9231,0.7589335620721703,mnist,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0268283420138888,0.1092258029513889,0.025387675404758,vtab/dsprites_label_orientation,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.3693369336933693,0.744974497449745,0.3649286987522281,fgvc_aircraft,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.4102258758451137,0.8022818070067609,0.4216815643098484,vtab/svhn,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.743319785938908,0.9583279695459478,0.7348385903018446,sun397,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,flickr8k,ViT-H-14,laion2b_s32b_b79k,zeroshot_retrieval,,0.9277999997138977,0.9729999899864197,ViT-H-14 laion2b_s32b_b79k
0.5487,0.8411,0.5430718178404617,mnist,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.7171428571428572,0.958095238095238,0.7258469461953507,vtab/resisc45,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.3454976303317535,0.6221800947867299,0.3445971563981042,country211,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.9645,0.999375,0.965125,stl10,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.779,0.9289333333333334,0.7643246651538985,imagenet-r,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.1633333333333333,0.7125333333333334,0.1575975877364315,vtab/clevr_count_all,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.8683565004088307,0.9934587080948488,0.8661667839491306,vtab/pets,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.647962962962963,0.9805555555555556,0.6445566610022502,vtab/eurosat,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6929582045861116,0.8668076109936576,0.6668176645957112,vtab/flowers,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.6630346397788258,0.8573751829565783,0.6645264657992297,vtab/flowers,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.1772666666666666,0.7803333333333333,0.2270014558851883,vtab/clevr_closest_object_distance,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2784,0.8929333333333334,0.2563239722391655,vtab/clevr_count_all,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.8328402366863905,0.94543063773833,0.9085289082247568,vtab/caltech101,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.2786954517516902,0.7308312845728334,0.2795999671407248,vtab/svhn,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.035400390625,0.1624348958333333,0.0364155761076967,vtab/dsprites_label_x_position,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2699554722287321,1.0,0.2211405088000341,vtab/diabetic_retinopathy,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.0259874131944444,0.1214463975694444,0.0262399607702056,vtab/dsprites_label_orientation,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.819197896120973,0.9465811965811964,0.8786521640800292,vtab/caltech101,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.4896907216494845,0.9721370855391472,0.4887152232577444,fer2013,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
,,,voc2007_multilabel,ViT-L-14,laion400m_e32,zeroshot_classification,0.7847012877464294,,,ViT-L-14 laion400m_e32
0.8020816392909416,0.9253537160513904,0.7985485908748879,vtab/flowers,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.5163895486935867,0.7605700712589074,0.4472134621454642,gtsrb,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5255354200988468,,0.5258471570841294,renderedsst2,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.2242797448867385,0.8738508906971629,0.1816019549723685,vtab/dmlab,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5442040519562185,0.8052231327005837,0.5451807843137254,imagenet_sketch,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.545867919921875,,0.5459467330999297,vtab/pcam,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.697049639280715,0.877516959190266,0.6846202591899297,objectnet,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0558847736625514,0.2832098765432099,0.0522661133926831,vtab/smallnorb_label_azimuth,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.2352,0.7871333333333334,0.2199570643185355,vtab/clevr_count_all,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.8039,0.9392333333333334,0.7907448605291261,imagenet-r,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.2292545710267229,,0.3081761474014508,vtab/kitti_closest_vehicle_distance,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.04559670781893,0.274320987654321,0.0456436809466583,vtab/smallnorb_label_azimuth,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.585723876953125,,0.5857677281000415,vtab/pcam,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8932,0.9735333333333334,0.8804663010091829,imagenet-r,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.3677333333333333,0.7024,0.3814206048810433,imagenet-a,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.378037803780378,0.8028802880288028,0.3781639928698752,fgvc_aircraft,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.5430851063829787,0.8356382978723405,0.5473404255319149,vtab/dtd,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.9261285909712722,0.9976371098122124,0.9261518670531818,cars,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.6904,0.9518,0.6833555055021581,mnist,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6174603174603175,0.921904761904762,0.624279367877562,vtab/resisc45,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.537057676232934,0.9492894956812482,0.5338791359180816,fer2013,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
,,,voc2007_multilabel,ViT-B-16,laion400m_e32,zeroshot_classification,0.7843208312988281,,,ViT-B-16 laion400m_e32
0.4582056283087211,0.9531903037057676,0.4167779537443768,fer2013,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.1594666666666666,0.8705333333333334,0.1702024394180443,vtab/clevr_closest_object_distance,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.955,0.9995,0.955375,stl10,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7640666666666667,0.9159,0.7521727338627011,imagenet-r,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8328402366863905,0.952827087442472,0.9177765781998192,vtab/caltech101,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.53631591796875,,0.53614377703907,vtab/pcam,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.0517695473251028,0.2697942386831276,0.0537673960190341,vtab/smallnorb_label_azimuth,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.6453,0.888,0.6451,vtab/cifar100,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
,,,flickr30k,ViT-B-32,laion2b_e16,zeroshot_retrieval,,0.8812000155448914,0.9639999866485596,ViT-B-32 laion2b_e16
0.4393776246365888,0.684612899752342,0.4268760394558317,objectnet,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
,,,flickr30k,ViT-g-14,laion2b_s12b_b42k,zeroshot_retrieval,,0.9348000288009644,0.99099999666214,ViT-g-14 laion2b_s12b_b42k
0.381760909649662,0.7632913337430854,0.4057750407393451,vtab/svhn,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5006666666666667,0.8033333333333333,0.4832835476168289,imagenet-a,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.9705,0.9994,0.9711,vtab/cifar10,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.4800780161604904,0.9711618835330176,0.4909158276198425,fer2013,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
,,,mscoco_captions,ViT-L-14-336,openai,zeroshot_retrieval,,0.615513801574707,0.8101999759674072,ViT-L-14-336 openai
0.7068,0.9062666666666668,0.6753602288814418,imagenet-a,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.9078,0.9977,0.9083,vtab/cifar10,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.3860633066994468,0.8061232329440688,0.3685311302620919,vtab/svhn,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7160513904699951,0.8744511302650837,0.6998995164388323,vtab/flowers,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.1971197119711971,0.5022502250225023,0.197344028520499,fgvc_aircraft,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.6033343577135832,0.8973570989551322,0.5683458085959752,vtab/svhn,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.4899860019381932,0.7341983417680629,0.4820376550831692,objectnet,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7626201923076923,0.9526575854700856,0.815718996924191,voc2007,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1462728551336146,,0.1818997858588953,vtab/kitti_closest_vehicle_distance,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.6787234042553192,0.923936170212766,0.6813829787234043,vtab/dtd,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.1721574664614031,0.8585001099626127,0.1577427840925754,vtab/dmlab,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.7916734428362335,0.9193364774760124,0.7931691849985836,vtab/flowers,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.0549794238683127,0.2670781893004115,0.0559609415916413,vtab/smallnorb_label_azimuth,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.7843883547008547,0.9570646367521368,0.835061321772101,voc2007,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.6177777777777778,0.957037037037037,0.6299267597724122,vtab/eurosat,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.543701171875,,0.5435553179465629,vtab/pcam,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0315348307291666,0.1596544053819444,0.0323590170008084,vtab/dsprites_label_x_position,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.5796296296296296,0.9525925925925924,0.5888803943202612,vtab/eurosat,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.9185064050149904,0.9956391387298992,0.9161666963118203,vtab/pets,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5381177990739744,0.7715085603531818,0.5274232792623416,objectnet,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
,,,flickr30k,ViT-L-14,laion2b_s32b_b82k,zeroshot_retrieval,,0.929199993610382,0.9869999885559082,ViT-L-14 laion2b_s32b_b82k
0.8368362144011939,0.987439373212287,0.8375394945435981,cars,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr8k,ViT-B-32-quickgelu,laion400m_e32,zeroshot_retrieval,,0.8303999900817871,0.9169999957084656,ViT-B-32-quickgelu laion400m_e32
0.9742,0.9994,0.9742999999999998,vtab/cifar10,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.613015873015873,0.91,0.6150226494017114,vtab/resisc45,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1901913349461183,0.8472839234660215,0.1733383864929211,vtab/dmlab,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1979333333333333,0.8076,0.1820160989597282,vtab/clevr_count_all,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
,,,voc2007_multilabel,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,0.8199208974838257,,,ViT-L-14 laion2b_s32b_b82k
0.1674,0.8695333333333334,0.1827530663763387,vtab/clevr_closest_object_distance,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1715165876777251,0.4024170616113744,0.1707582938388625,country211,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.75202,0.94252,0.7526400000000001,imagenet1k,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8923412373943854,0.9967293540474244,0.8919759072052595,vtab/pets,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.487670937870141,0.7249919241951115,0.4750858280106649,objectnet,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.8959084690958836,0.9960203954732,0.8961635505798664,cars,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.7832532051282052,0.9692174145299144,0.864352156405908,voc2007,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5031481481481481,0.92,0.5110567650187513,vtab/eurosat,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.161,0.9128666666666668,0.1739135544326629,vtab/clevr_closest_object_distance,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.8441736102474816,0.9912946150976246,0.8456794155511405,cars,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.7372,0.9338,0.7372000000000001,vtab/cifar100,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.7136119694259229,0.8591640917222313,0.691284904068223,vtab/flowers,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.1485814822960193,0.8129316032548933,0.1482313831309264,vtab/dmlab,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.6112026359143328,,0.6113264286955011,renderedsst2,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.5370675453047776,,0.5373060332349024,renderedsst2,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
,,,mscoco_captions,ViT-B-32,openai,zeroshot_retrieval,,0.5584565997123718,0.748199999332428,ViT-B-32 openai
,,,flickr8k,ViT-B-16,openai,zeroshot_retrieval,,0.8285999894142151,0.9139999747276306,ViT-B-16 openai
0.8379355687047995,0.9518408941485864,0.9328934325841473,vtab/caltech101,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0307752821180555,0.1437852647569444,0.0304426618193977,vtab/dsprites_label_orientation,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.841715976331361,0.940828402366864,0.9341112975275198,vtab/caltech101,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0311008029513888,0.1568060980902778,0.0321637232587243,vtab/dsprites_label_x_position,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
,,,flickr30k,ViT-L-14-336,openai,zeroshot_retrieval,,0.8889999985694885,0.9810000061988832,ViT-L-14-336 openai
0.5961406197803062,0.8388060288077973,0.5956784313725489,imagenet_sketch,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.4657286152131513,0.9445528002229032,0.4812190866355833,fer2013,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.0562962962962962,0.274156378600823,0.0567608380809354,vtab/smallnorb_label_azimuth,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.1583333333333333,0.8006,0.1676244908434004,vtab/clevr_closest_object_distance,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.7326224513709867,1.0,0.2068672210509669,vtab/diabetic_retinopathy,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.8911666666666667,0.9757333333333332,0.8776131879598171,imagenet-r,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.988625,0.999875,0.988625,stl10,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.6865218750574692,0.9403608143148756,0.6921311150610732,sun397,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.5509033203125,,0.5508520491113902,vtab/pcam,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.1606666666666666,0.9079333333333334,0.1772310296867651,vtab/clevr_closest_object_distance,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.5574468085106383,0.8648936170212767,0.5622340425531915,vtab/dtd,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.2315333333333333,0.805,0.2332901934437162,vtab/clevr_count_all,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.0298394097222222,0.1414794921875,0.0308127750442299,vtab/dsprites_label_x_position,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.0623868312757201,0.2708641975308642,0.0631906837062103,vtab/smallnorb_label_azimuth,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.4774310392867094,0.9402340484814712,0.4649079545536754,fer2013,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.5230010414824422,0.7913694511584036,0.5233670588235293,imagenet_sketch,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7341666666666666,0.9035,0.7214778957504825,imagenet-r,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.8353057199211046,0.9546351084812624,0.9005042720791782,vtab/caltech101,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.3109333333333333,0.8006,0.3066391557930237,vtab/clevr_count_all,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.6985172981878089,,0.6986603265589717,renderedsst2,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.1087242798353909,0.540164609053498,0.1086253991970707,vtab/smallnorb_label_elevation,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.9351,0.998,0.936,vtab/cifar10,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
,,,flickr8k,ViT-L-14,openai,zeroshot_retrieval,,0.8633999824523926,0.9409999847412109,ViT-L-14 openai
0.65528,0.894,0.6563199999999999,imagenet1k,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.5930807248764415,,0.5925413265010712,renderedsst2,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.1650710900473933,0.3788625592417061,0.164218009478673,country211,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.168016801680168,0.4119411941194119,0.1658110516934046,fgvc_aircraft,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.0303819444444444,0.1311170789930555,0.0332405586865836,vtab/dsprites_label_orientation,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.3171317131713171,0.7827782778277828,0.3170053475935828,fgvc_aircraft,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6691,0.8925,0.6685000000000001,vtab/cifar100,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.0516872427983539,0.2725925925925926,0.0526600212836742,vtab/smallnorb_label_azimuth,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.6559,0.8854,0.6541,imagenetv2,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.5569148936170213,0.8558510638297873,0.5542553191489361,vtab/dtd,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.1663981042654028,0.3825118483412322,0.1670142180094786,country211,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.7074,0.9179,0.7075,imagenetv2,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.3137333333333333,0.6410666666666667,0.3236449813371542,imagenet-a,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.0454320987654321,0.2672427983539094,0.0450240211431134,vtab/smallnorb_label_azimuth,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.8741,0.9655666666666668,0.8599962924086103,imagenet-r,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.4275427542754275,0.8361836183618362,0.4260962566844919,fgvc_aircraft,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
,,,mscoco_captions,ViT-L-14,laion2b_s32b_b82k,zeroshot_retrieval,,0.7107957005500793,0.8399999737739563,ViT-L-14 laion2b_s32b_b82k
0.5958,0.8547,0.5955999999999999,imagenetv2,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7540228405391985,0.9617025580668296,0.7523924485563404,sun397,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2418241824182418,0.6054605460546054,0.2405525846702317,fgvc_aircraft,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.0590123456790123,0.279835390946502,0.0603763349553147,vtab/smallnorb_label_azimuth,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
0.7613,0.9289,0.7611,vtab/cifar100,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.9277453053102848,0.9983832856609874,0.9288577913034778,cars,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.0257297092013888,0.1208224826388889,0.0259727501505128,vtab/dsprites_label_orientation,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.8254437869822485,0.954963839579224,0.908884143116176,vtab/caltech101,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.980125,0.999875,0.980875,stl10,ViT-L-14,laion400m_e32,zeroshot_classification,,,,ViT-L-14 laion400m_e32
0.2463246324632463,0.5724572457245725,0.2460249554367201,fgvc_aircraft,ViT-B-32,laion2b_s34b_b79k,zeroshot_classification,,,,ViT-B-32 laion2b_s34b_b79k
0.8102964743589743,0.9655448717948718,0.8579252085748035,voc2007,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.2044,0.7866666666666666,0.2151124081246672,vtab/clevr_count_all,ViT-B-16,openai,zeroshot_classification,,,,ViT-B-16 openai
0.2872511848341232,0.542085308056872,0.2880094786729857,country211,ViT-g-14,laion2b_s12b_b42k,zeroshot_classification,,,,ViT-g-14 laion2b_s12b_b42k
0.06,0.288312757201646,0.0630119225348046,vtab/smallnorb_label_azimuth,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.9664,0.9987,0.9665,vtab/cifar10,ViT-L-14,laion2b_s32b_b82k,zeroshot_classification,,,,ViT-L-14 laion2b_s32b_b82k
0.1895333333333333,0.7248666666666667,0.1870254885776544,vtab/clevr_count_all,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.8415619947767691,0.9900509886829996,0.8435641961196442,cars,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.337965783923131,1.0,0.2591163084325771,vtab/diabetic_retinopathy,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.1582,0.8814666666666666,0.1812267071156741,vtab/clevr_closest_object_distance,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.0475720164609053,0.2691358024691358,0.0457953163680852,vtab/smallnorb_label_azimuth,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.3651623119556611,0.7007917656373713,0.3512103003164681,gtsrb,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.1592258632065097,0.8009676709918627,0.1713416703583154,vtab/dmlab,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.4948535233570863,0.7578780680918448,0.4319824074083427,gtsrb,ViT-B-16-plus-240,laion400m_e32,zeroshot_classification,,,,ViT-B-16-plus-240 laion400m_e32
0.3294329432943294,0.7836783678367837,0.3317290552584671,fgvc_aircraft,ViT-L-14-336,openai,zeroshot_classification,,,,ViT-L-14-336 openai
0.684701252367729,0.9328576420177648,0.6783238400960281,sun397,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.6885672467067816,0.8466417303626605,0.6732279033655394,vtab/flowers,ViT-B-32,laion2b_e16,zeroshot_classification,,,,ViT-B-32 laion2b_e16
0.8385930309007232,0.9539776462853384,0.9334530557615972,vtab/caltech101,ViT-L-14,openai,zeroshot_classification,,,,ViT-L-14 openai
0.6696489324530592,0.9273222134358275,0.6609448269493161,sun397,ViT-B-32-quickgelu,laion400m_e32,zeroshot_classification,,,,ViT-B-32-quickgelu laion400m_e32
0.3000947867298578,0.556872037914692,0.2994312796208531,country211,ViT-H-14,laion2b_s32b_b79k,zeroshot_classification,,,,ViT-H-14 laion2b_s32b_b79k
0.409445528002229,0.9413485650599052,0.3587300457745208,fer2013,ViT-B-32,openai,zeroshot_classification,,,,ViT-B-32 openai
0.96975,0.999875,0.96975,stl10,ViT-B-16,laion400m_e32,zeroshot_classification,,,,ViT-B-16 laion400m_e32
clip_benchmark/benchmark/dataset_type.csv 0 → 100644
View file @ 26e59280
dataset,type
imagenet1k,natural
imagenetv2,natural
imagenet-r,natural
imagenet_sketch,specialized
objectnet,natural
imagenet-a,natural
imagenet-o,natural
vtab/cifar10,natural
vtab/cifar100,natural
mnist,specialized
vtab/flowers,natural
cars,natural
vtab/svhn,natural
fer2013,natural
renderedsst2,specialized
vtab/pets,natural
vtab/caltech101,natural
voc2007_multilabel,natural
voc2007,natural
sun397,natural
fgvc_aircraft,natural
country211,natural
vtab/dtd,natural
gtsrb,natural
stl10,natural
vtab/diabetic_retinopathy,specialized
vtab/eurosat,specialized
vtab/resisc45,specialized
vtab/pcam,specialized
vtab/clevr_count_all,structured
vtab/clevr_closest_object_distance,structured
vtab/dsprites_label_orientation,structured
vtab/dsprites_label_x_position,structured
vtab/dsprites_label_y_position,structured
vtab/smallnorb_label_elevation,structured
vtab/smallnorb_label_azimuth,structured
vtab/dmlab,structured
vtab/kitti_closest_vehicle_distance,structured
mscoco_captions,retrieval
flickr8k,retrieval
flickr30k,retrieval
clip_benchmark/benchmark/datasets.txt 0 → 100644
View file @ 26e59280
mscoco_captions
flickr8k
flickr30k
imagenet1k
imagenetv2
imagenet_sketch
imagenet-a
imagenet-r
objectnet
fer2013
voc2007
voc2007_multilabel
sun397
cars
fgvc_aircraft
mnist
stl10
gtsrb
country211
renderedsst2
vtab/caltech101
vtab/cifar10
vtab/cifar100
vtab/clevr_count_all
vtab/clevr_closest_object_distance
vtab/diabetic_retinopathy
vtab/dmlab
vtab/dsprites_label_orientation
vtab/dsprites_label_x_position
vtab/dtd
vtab/eurosat
vtab/kitti_closest_vehicle_distance
vtab/flowers
vtab/pets
vtab/pcam
vtab/resisc45
vtab/smallnorb_label_azimuth
vtab/smallnorb_label_elevation
vtab/svhn
clip_benchmark/benchmark/datasets_multilingual.txt 0 → 100644
View file @ 26e59280
multilingual_mscoco_captions,es
multilingual_mscoco_captions,it
multilingual_mscoco_captions,ko
multilingual_mscoco_captions,pl
multilingual_mscoco_captions,ru
multilingual_mscoco_captions,tr
multilingual_mscoco_captions,zh
multilingual_mscoco_captions,en
imagenet1k,zh
imagenet1k,it
imagenet1k,jp
imagenet1k,en
imagenet1k,ar
clip_benchmark/benchmark/models.txt 0 → 100644
View file @ 26e59280
ViT-B-32,openai
ViT-B-16,openai
ViT-L-14,openai
ViT-L-14-336,openai
ViT-B-32-quickgelu,laion400m_e32
ViT-B-32,laion2b_e16
ViT-B-32,laion2b_s34b_b79k
ViT-B-16,laion400m_e32
ViT-B-16-plus-240,laion400m_e32
ViT-L-14,laion400m_e32
ViT-L-14,laion2b_s32b_b82k
ViT-H-14,laion2b_s32b_b79k
ViT-g-14,laion2b_s12b_b42k
clip_benchmark/benchmark/results.ipynb 0 → 100644
View file @ 26e59280
This source diff could not be displayed because it is too large. You can view the blob instead.
clip_benchmark/benchmark/webdatasets.txt 0 → 100644
View file @ 26e59280
wds/mscoco_captions
wds/flickr8k
wds/flickr30k
wds/imagenet1k
wds/imagenetv2
wds/imagenet_sketch
wds/imagenet-a
wds/imagenet-r
wds/imagenet-o
wds/objectnet
wds/fer2013
wds/voc2007
wds/voc2007_multilabel
wds/sun397
wds/cars
wds/fgvc_aircraft
wds/mnist
wds/stl10
wds/gtsrb
wds/country211
wds/renderedsst2
wds/vtab/caltech101
wds/vtab/cifar10
wds/vtab/cifar100
wds/vtab/clevr_count_all
wds/vtab/clevr_closest_object_distance
wds/vtab/diabetic_retinopathy
wds/vtab/dmlab
wds/vtab/dsprites_label_orientation
wds/vtab/dsprites_label_x_position
wds/vtab/dsprites_label_y_position
wds/vtab/dtd
wds/vtab/eurosat
wds/vtab/kitti_closest_vehicle_distance
wds/vtab/flowers
wds/vtab/pets
wds/vtab/pcam
wds/vtab/resisc45
wds/vtab/smallnorb_label_azimuth
wds/vtab/smallnorb_label_elevation
wds/vtab/svhn
clip_benchmark/clip_benchmark/__init__.py 0 → 100644
View file @ 26e59280
"""Top-level package for CLIP Benchmark."""
__author__ = """Mehdi Cherti"""
__email__ = 'mehdicherti@gmail.com'
__version__ = '0.1.0'
clip_benchmark/clip_benchmark/cli.py 0 → 100644
View file @ 26e59280
"""Console script for clip_benchmark."""
import argparse
import csv
import json
import os
import sys
from copy import copy
import torch
from clip_benchmark.datasets.builder import (build_dataset, dataset_collection,
get_dataset_collate_fn,
get_dataset_collection_from_file,
get_dataset_default_task)
from clip_benchmark.metrics import (linear_probe, mscoco_generative,
zeroshot_classification,
zeroshot_retrieval)
from clip_benchmark.model_collection import (get_model_collection_from_file,
model_collection)
from clip_benchmark.models import MODEL_TYPES, load_clip
def get_parser_args():
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_eval = subparsers.add_parser('eval', help='Evaluate')
parser_eval.add_argument('--dataset', type=str, default='cifar10', nargs='+',
help="Dataset(s) to use for the benchmark. Can be the name of a dataset, or a collection name ('vtab', 'vtab+', 'imagenet_robustness', 'retrieval') or path of a text file where each line is a dataset name")
parser_eval.add_argument('--dataset_root', default='root', type=str,
help="dataset root folder where the datasets are downloaded. Can be in the form of a template depending on dataset name, e.g., --dataset_root='datasets/{dataset}'. This is useful if you evaluate on multiple datasets.")
parser_eval.add_argument('--split', type=str, default='test', help='Dataset split to use')
parser_eval.add_argument('--model', type=str, default='ViT-B-32-quickgelu',
help='Model architecture to use from OpenCLIP')
parser_eval.add_argument('--pretrained', type=str, default='laion400m_e32',
help='Model checkpoint name to use from OpenCLIP')
parser_eval.add_argument('--pretrained_model', type=str, default='', nargs='+',
help="Pre-trained model(s) to use. Can be the full model name where `model` and `pretrained` are comma separated (e.g., --pretrained_model='ViT-B-32-quickgelu,laion400m_e32'), a model collection name ('openai' or 'openclip_base' or 'openclip_multilingual' or 'openclip_all'), or path of a text file where each line is a model fullname where model and pretrained are comma separated (e.g., ViT-B-32-quickgelu,laion400m_e32). --model and --pretrained are ignored if --pretrained_model is used.")
parser_eval.add_argument('--task', type=str, default='auto',
choices=['zeroshot_classification', 'zeroshot_retrieval', 'zeroshot_retrieval_with_itm',
'linear_probe', 'mscoco_generative', 'auto'],
help='Task to evaluate on. With --task=auto, the task is automatically inferred from the dataset.')
parser_eval.add_argument('--no_amp', action='store_false', dest='amp', default=True,
help='whether to use mixed precision')
parser_eval.add_argument('--num_workers', default=4, type=int)
parser_eval.add_argument('--recall_k', default=[1, 5, 10], type=int,
help='for retrieval, select the k for Recall@K metric. ', nargs='+', )
parser_eval.add_argument('--fewshot_k', default=-1, type=int,
help='for linear probe, how many shots. -1 = whole dataset.')
parser_eval.add_argument('--fewshot_epochs', default=10, type=int, help='for linear probe, how many epochs.')
parser_eval.add_argument('--fewshot_lr', default=0.1, type=float,
help='for linear probe, what is the learning rate.')
parser_eval.add_argument('--skip_load', action='store_true',
help='for linear probes, when everything is cached, no need to load model.')
parser_eval.add_argument('--seed', default=0, type=int, help='random seed.')
parser_eval.add_argument('--batch_size', default=64, type=int)
parser_eval.add_argument('--model_cache_dir', default=None, type=str,
help='directory to where downloaded models are cached')
parser_eval.add_argument('--feature_root', default='features', type=str,
help='feature root folder where the features are stored.')
parser_eval.add_argument('--annotation_file', default='', type=str,
help='text annotation file for retrieval datasets. Only needed for when `--task` is `zeroshot_retrieval`.')
parser_eval.add_argument('--language', default='en', type=str, nargs='+',
help='language(s) of classname and prompts to use for zeroshot classification.')
parser_eval.add_argument('--output', default='result.json', type=str,
help="output file where to dump the metrics. Can be in form of a template, e.g., --output='{dataset}_{pretrained}_{model}_{language}_{task}.json'")
parser_eval.add_argument('--quiet', dest='verbose', action='store_false', help='suppress verbose messages')
parser_eval.add_argument('--cupl', default=False, action='store_true',
help='Use natural language prompt from CuPL paper')
parser_eval.add_argument('--save_clf', default=None, type=str,
help='optionally save the classification layer output by the text tower')
parser_eval.add_argument('--load_clfs', nargs='+', default=[], type=str,
help='optionally load and average mutliple layers output by text towers.')
parser_eval.add_argument('--skip_existing', default=False, action='store_true',
help='whether to skip an evaluation if the output file exists.')
parser_eval.add_argument('--model_type', default='open_clip', type=str, choices=MODEL_TYPES, help='clip model type')
parser_eval.add_argument('--wds_cache_dir', default=None, type=str,
help='optional cache directory for webdataset only')
parser_eval.set_defaults(which='eval')
parser_build = subparsers.add_parser('build', help='Build CSV from evaluations')
parser_build.add_argument('files', type=str, nargs='+', help='path(s) of JSON result files')
parser_build.add_argument('--output', type=str, default='benchmark.csv', help='CSV output file')
parser_build.set_defaults(which='build')
args = parser.parse_args()
return args
def main():
base = get_parser_args()
if base.which == 'eval':
main_eval(base)
elif base.which == 'build':
main_build(base)
def main_build(base):
# Build a benchmark single CSV file from a set of evaluations (JSON files)
rows = []
fieldnames = set()
for path in base.files:
data = json.load(open(path))
row = {}
row.update(data['metrics'])
row.update(data)
del row['metrics']
row['model_fullname'] = row['model'] + ' ' + row['pretrained']
for field in row.keys():
fieldnames.add(field)
rows.append(row)
with open(base.output, 'a') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
for row in rows:
writer.writerow(row)
def main_eval(base):
# Get list of pre-trained models to evaluate
pretrained_model = _as_list(base.pretrained_model)
if pretrained_model:
models = []
for name in pretrained_model:
if os.path.isfile(name):
# if path, read file, each line is a pre-trained model
models.extend(get_model_collection_from_file(name))
elif name in model_collection:
# if part of `model_collection`, retrieve from it
models.extend(model_collection[name])
else:
# if not, assume it is in the form of `model,pretrained`
model, pretrained = name.split(',')
models.append((model, pretrained))
else:
models = [(base.model, base.pretrained)]
# Ge list of datasets to evaluate on
datasets = []
for name in _as_list(base.dataset):
if os.path.isfile(name):
# If path, read file, each line is a dataset name
datasets.extend(get_dataset_collection_from_file(name))
elif name in dataset_collection:
# if part of `dataset_collection`, retrieve from it
datasets.extend(dataset_collection[name])
else:
# if not, assume it is simply the name of the dataset
datasets.append(name)
# Get list of languages to evaluate on
languages = _as_list(base.language)
if base.verbose:
print(f'Models: {models}')
print(f'Datasets: {datasets}')
print(f'Languages: {languages}')
for model, pretrained in models:
for dataset in datasets:
for language in languages:
# We iterative over all possible model/dataset/languages
# TODO: possibility to parallelize evaluation here
args = copy(base)
args.model = model
args.pretrained = pretrained
args.dataset = dataset
args.language = language
run(args)
def _as_list(l):
if not l:
return []
return [l] if type(l) != list else l
def run(args):
"""Console script for clip_benchmark."""
args.device = 'cuda' if torch.cuda.is_available() else 'cpu'
# set seed.
torch.manual_seed(args.seed)
task = args.task
if args.dataset.startswith('wds/'):
dataset_name = args.dataset.replace('wds/', '', 1)
else:
dataset_name = args.dataset
if task == 'auto':
task = get_dataset_default_task(dataset_name)
pretrained_slug = os.path.basename(args.pretrained) if os.path.isfile(args.pretrained) else args.pretrained
pretrained_slug_full_path = args.pretrained.replace('/', '_') if os.path.isfile(
args.pretrained) else args.pretrained
dataset_slug = dataset_name.replace('/', '_')
output = args.output.format(
model=args.model,
pretrained=pretrained_slug,
pretrained_full_path=pretrained_slug_full_path,
task=task,
dataset=dataset_slug,
language=args.language
)
if os.path.exists(output) and args.skip_existing:
if args.verbose:
print(f'Skip {output}, exists already.')
return
if args.verbose:
print(f"Running '{task}' on '{dataset_name}' with the model '{args.pretrained}' on language '{args.language}'")
dataset_root = args.dataset_root.format(dataset=dataset_name, dataset_cleaned=dataset_name.replace('/', '-'))
if args.skip_load:
model, transform, collate_fn, dataloader = None, None, None, None
else:
model, transform, tokenizer = load_clip(
model_type=args.model_type,
model_name=args.model,
pretrained=args.pretrained,
cache_dir=args.model_cache_dir,
device=args.device
)
print(transform)
model.eval()
dataset = build_dataset(
dataset_name=args.dataset,
root=dataset_root,
transform=transform,
split=args.split,
annotation_file=args.annotation_file,
download=True,
language=args.language,
task=task,
cupl=args.cupl,
wds_cache_dir=args.wds_cache_dir,
)
collate_fn = get_dataset_collate_fn(args.dataset)
if args.verbose:
try:
print(f'Dataset size: {len(dataset)}')
except TypeError:
print('IterableDataset has no len()')
print(f'Dataset split: {args.split}')
try:
print(f'Dataset classes: {dataset.classes}')
print(f'Dataset number of classes: {len(dataset.classes)}')
except:
print('Dataset has no classes.')
if args.dataset.startswith('wds/'):
dataloader = torch.utils.data.DataLoader(
dataset.batched(args.batch_size), batch_size=None,
shuffle=False, num_workers=args.num_workers,
)
else:
dataloader = torch.utils.data.DataLoader(
dataset, batch_size=args.batch_size,
shuffle=False, num_workers=args.num_workers,
collate_fn=collate_fn
)
if task == 'zeroshot_classification':
zeroshot_templates = dataset.templates if hasattr(dataset, 'templates') else None
if args.cupl:
assert (zeroshot_templates is not None), 'Dataset does not support CuPL prompts'
# if args.verbose:
# print(f"Zero-shot templates: {zeroshot_templates}")
classnames = dataset.classes if hasattr(dataset, 'classes') else None
assert (zeroshot_templates is not None and classnames is not None), 'Dataset does not support classification'
metrics = zeroshot_classification.evaluate(
model,
dataloader,
tokenizer,
classnames, zeroshot_templates,
device=args.device,
amp=args.amp,
verbose=args.verbose,
cupl=args.cupl,
save_clf=args.save_clf,
load_clfs=args.load_clfs,
)
elif task == 'zeroshot_retrieval':
metrics = zeroshot_retrieval.evaluate(
model,
dataloader,
tokenizer,
recall_k_list=args.recall_k,
device=args.device,
amp=args.amp
)
elif task == 'zeroshot_retrieval_with_itm':
metrics = zeroshot_retrieval_with_itm.evaluate(
model,
dataloader,
tokenizer,
recall_k_list=args.recall_k,
device=args.device,
amp=args.amp
)
elif task == 'linear_probe':
# we also need the train split for linear probing.
train_dataset = build_dataset(
dataset_name=args.dataset,
root=dataset_root,
transform=transform,
split='train',
annotation_file=args.annotation_file,
download=True,
)
train_dataloader = torch.utils.data.DataLoader(
train_dataset, batch_size=args.batch_size,
shuffle=False, num_workers=args.num_workers,
collate_fn=collate_fn, pin_memory=True,
)
metrics = linear_probe.evaluate(
model,
train_dataloader,
dataloader,
args.fewshot_k,
args.batch_size,
args.num_workers,
args.fewshot_lr,
args.fewshot_epochs,
(args.model + '-' + args.pretrained + '-' + args.dataset).replace('/', '_'),
args.seed,
args.feature_root,
device=args.device,
amp=args.amp,
verbose=args.verbose,
)
elif task == 'mscoco_generative':
metrics = mscoco_generative.evaluate(
model=model,
dataloader=dataloader,
batch_size=args.batch_size,
num_workers=args.num_workers,
device=args.device,
amp=args.amp,
verbose=args.verbose,
transform=transform
)
else:
raise ValueError(
'Unsupported task: {}. task should `zeroshot_classification` or `zeroshot_retrieval`'.format(task))
dump = {
'dataset': args.dataset,
'model': args.model,
'pretrained': args.pretrained,
'task': task,
'metrics': metrics,
'language': args.language,
}
if args.verbose:
print(f'Dump results to: {output}')
with open(output, 'a') as f:
f.write(json.dumps(dump) + '\n')
# json.dump(dump, f)
return 0
if __name__ == '__main__':
sys.exit(main()) # pragma: no cover
clip_benchmark/clip_benchmark/datasets/ar_classnames.json 0 → 100644
View file @ 26e59280
{
"imagenet1k": [
"\u0633\u0645\u0643 \u0627\u0644\u062a\u0646\u0634",
"\u0627\u0644\u0633\u0645\u0643\u0629 \u0627\u0644\u0630\u0647\u0628\u064a\u0629",
"\u0627\u0644\u0642\u0631\u0634 \u0627\u0644\u0623\u0628\u064a\u0636 \u0627\u0644\u0643\u0628\u064a\u0631",
"\u0627\u0644\u0642\u0631\u0634 \u0627\u0644\u0628\u0628\u0631\u064a",
"\u0627\u0644\u0642\u0631\u0634 \u0627\u0644\u0645\u0637\u0631\u0642\u0629",
"\u0633\u0645\u0643 \u0627\u0644\u0631\u0639\u0627\u062f",
"\u0633\u0645\u0643 \u0627\u0644\u0631\u0642\u064a\u0637\u0629",
"\u062f\u064a\u0643",
"\u062f\u062c\u0627\u062c\u0629",
"\u0646\u0639\u0627\u0645\u0629",
"\u0627\u0644\u0634\u0631\u0634\u0648\u0631 \u0627\u0644\u062c\u0628\u0644\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u0633\u0648\u0646",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062a\u0641\u0627\u062d\u064a \u0627\u0644\u0627\u0648\u0631\u0648\u0628\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062c\u0646\u0643 \u062f\u0627\u0643\u0646 \u0627\u0644\u0639\u064a\u0648\u0646",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062f\u0631\u0633\u0629 \u0627\u0644\u0633\u0645\u0627\u0648\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0628\u0648 \u0627\u0644\u062d\u0646\u0627\u0621",
"\u0628\u0644\u0628\u0644",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0642\u064a\u0642",
"\u0639\u0642\u0639\u0642 \u0637\u0627\u0626\u0631 \u0627\u0644\u0630\u064a\u0644 \u0627\u0644\u0637\u0648\u064a\u0644",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0642\u0631\u0642\u0641",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u063a\u0637\u0627\u0633",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u062f\u0623\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0639\u0642\u0627\u0628 \u0627\u0644\u0631\u062e\u0645\u0629",
"\u0646\u0633\u0631",
"\u0627\u0644\u0628\u0648\u0645\u0629 \u0627\u0644\u0631\u0645\u0627\u062f\u064a\u0629",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0646\u0627\u0631\u064a",
"\u0644\u064a\u0633\u0648\u062a\u0631\u064a\u062a\u0648\u0646 \u0641\u0648\u0644\u062c\u0627\u0631\u064a\u0633",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0645\u0627\u0626\u064a",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0645\u0631\u0642\u0637",
"\u0627\u0644\u0633\u0645\u0646\u062f\u0631 \u0627\u0644\u0645\u0643\u0633\u064a\u0643\u064a",
"\u0636\u0641\u062f\u0639 \u0627\u0644\u062b\u0648\u0631 \u0627\u0644\u0627\u0645\u0631\u064a\u0643\u064a",
"\u0636\u0641\u062f\u0639 \u0627\u0644\u0634\u062c\u0631",
"\u0627\u0644\u0636\u0641\u0627\u062f\u0639 \u0630\u0627\u062a \u0627\u0644\u0630\u064a\u0644",
"\u0627\u0644\u0633\u0644\u062d\u0641\u0627\u0629 \u0627\u0644\u0628\u062d\u0631\u064a\u0629 \u0636\u062e\u0645\u0629 \u0627\u0644\u0631\u0623\u0633",
"\u0633\u0644\u062d\u0641\u0627\u0629 \u0627\u0644\u0645\u062d\u064a\u0637 \u062c\u0644\u062f\u064a\u0629 \u0627\u0644\u0638\u0647\u0631",
"\u0633\u0644\u062d\u0641\u0627\u0629 \u0627\u0644\u0637\u064a\u0646",
"\u0627\u0644\u0633\u0644\u062d\u0641\u0627\u0629 \u0630\u0627\u062a \u0638\u0647\u0631 \u0627\u0644\u0645\u0639\u064a\u0646",
"\u0633\u0644\u062d\u0641\u0627\u0629 \u0635\u0646\u062f\u0648\u0642\u064a\u0629",
"\u0627\u0644\u0648\u0632\u063a",
"\u0627\u0644\u0625\u063a\u0648\u0627\u0646\u0629",
"\u0627\u0644\u062d\u0631\u0628\u0627\u0621 \u0627\u0644\u062e\u0636\u0631\u0627\u0621",
"\u0627\u0644\u0633\u062d\u0627\u0644\u064a \u0627\u0644\u0635\u062d\u0631\u0627\u0648\u064a\u0629",
"\u0627\u0644\u0639\u064e\u0636\u0652\u0631\u064e\u0641\u064f\u0648\u0637",
" \u0633\u062d\u0644\u064a\u0629 \u0647\u062f\u0628 \u0627\u0644\u0639\u0646\u0642",
"\u0633\u062d\u0644\u064a\u0629 \u0627\u0644\u062a\u0645\u0633\u0627\u062d",
"\u0648\u062d\u0634 \u062c\u064a\u0644\u0627",
"\u0633\u062d\u0644\u064a\u0629 \u062e\u0636\u0631\u0627\u0621",
"\u062d\u0631\u0628\u0627\u0621 \u0627\u0641\u0631\u064a\u0642\u064a\u0629",
"\u062a\u0646\u064a\u0646 \u0643\u0648\u0645\u0648\u062f\u0648",
"\u0627\u0644\u062a\u0645\u0633\u0627\u062d \u0627\u0644\u0623\u0641\u0631\u064a\u0642\u064a",
"\u062a\u0645\u0633\u0627\u062d \u0627\u0644\u0642\u0627\u0637\u0648\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0627\u0644\u062f\u064a\u0646\u0627\u0635\u0648\u0631 \u062b\u064f\u0644\u0627\u062b\u064a\u064f\u0651 \u0627\u0644\u0642\u064f\u0631\u0648\u0646",
"\u0627\u0644\u062b\u0639\u0627\u0628\u064a\u0646 \u0627\u0644\u062f\u0648\u062f\u064a\u0629",
"\u0623\u0641\u0639\u0649 \u0627\u0644\u0637\u0648\u0642",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u0646\u0641\u0627\u062b\u0629 ",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u062e\u0636\u0631\u0627\u0621",
"\u0627\u0644\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0645\u0644\u0643",
"\u0623\u0641\u0639\u0649 \u0627\u0644\u0631\u0628\u0627\u0637",
"\u0627\u0641\u0639\u0649 \u0627\u0644\u0645\u0627\u0621",
"\u062b\u0639\u0628\u0627\u0646 \u0646\u0628\u0627\u062a \u0643\u0631\u0645\u0629",
"\u0627\u0644\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0644\u064a\u0644\u064a ",
"\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0623\u0635\u0644\u0629 \u0627\u0644\u0639\u0627\u0635\u0631\u0629",
"\u0627\u0644\u062b\u0639\u0628\u0627\u0646 \u0627\u0644\u0635\u062e\u0631\u064a",
"\u0643\u0648\u0628\u0631\u0627 \u0647\u0646\u062f\u064a\u0629",
"\u0645\u0627\u0645\u0628\u0627 \u062e\u0636\u0631\u0627\u0621",
"\u0627\u0644\u063a\u064a\u062f\u0642\u0627\u0648\u0627\u062a",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u0645\u0642\u0631\u0646\u0629",
"\u0627\u0644\u0623\u0641\u0639\u0649 \u0627\u0644\u062c\u0631\u0633\u064a\u0629 \u0630\u0627\u062a \u0627\u0644\u0635\u062f\u0631 \u0627\u0644\u0645\u0627\u0633\u064a",
"\u0627\u0641\u0639\u0649 \u0644\u0627\u0641\u0651\u0629 \u0627\u0644\u062c\u0646\u0628",
"\u0645\u0641\u0635\u0644\u064a\u0627\u062a \u062b\u0644\u0627\u062b\u064a\u0629 \u0627\u0644\u0641\u0635\u0648\u0635",
"\u062d\u0634\u0631\u0629 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a \u0630\u0627\u062a \u0627\u0644\u0642\u0648\u0627\u0626\u0645 \u0627\u0644\u0637\u0648\u064a\u0644\u0629",
"\u0639\u0642\u0631\u0628",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u062d\u062f\u064a\u0642\u0629 \u0630\u0627 \u0627\u0644\u0644\u0648\u0646 \u0627\u0644\u0623\u0633\u0648\u062f \u0648\u0627\u0644\u0623\u0635\u0641\u0631 ",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u062d\u0638\u064a\u0631\u0629",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u062d\u062f\u064a\u0642\u0629 \u0627\u0644\u0623\u0648\u0631\u0628\u064a",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u0623\u0631\u0645\u0644\u0629 \u0627\u0644\u0633\u0648\u062f\u0627\u0621",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0631\u062a\u064a\u0644\u0627\u0621 \u0630\u0627\u062a \u0623\u0631\u062c\u0644 \u062d\u0645\u0631\u0627\u0621",
"\u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u0630\u0626\u0628\u064a",
"\u0639\u0646\u0643\u0628\u0648\u062a \u0627\u0644\u0642\u064f\u0631\u064e\u0627\u062f",
"\u062d\u0634\u0631\u0629 \u0623\u0645 \u0623\u0631\u0628\u0639\u0629 \u0648\u0623\u0631\u0628\u0639\u064a\u0646",
"\u062f\u062c\u0627\u062c\u0629 \u0627\u0644\u0637\u0647\u064a\u0648\u062c \u0627\u0644\u0623\u0633\u0648\u062f",
"\u062f\u062c\u0627\u062c\u0629 \u062a\u0631\u0645\u062c\u0627\u0646",
"\u062f\u062c\u0627\u062c\u0629 \u0637\u064a\u0647\u0648\u062c \u0645\u0637\u0648\u0642",
"\u062f\u062c\u0627\u062c\u0629 \u0627\u0644\u0637\u0647\u0628\u0648\u062c",
"\u0627\u0644\u0637\u0627\u0648\u0648\u0633",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0633\u0645\u0627\u0646",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u062c\u0644",
"\u0627\u0644\u0628\u0628\u063a\u0627\u0621 \u0627\u0644\u0631\u0645\u0627\u062f\u064a",
"\u0628\u0628\u063a\u0627\u0621 \u0645\u0643\u0627\u0648",
"\u0628\u0628\u063a\u0627\u0621 \u0643\u0648\u0643\u0627\u062a\u0648 \u0643\u0628\u0631\u064a\u062a\u064a \u0627\u0644\u0639\u0631\u0641",
"\u0628\u0628\u063a\u0627\u0621 \u0642\u0648\u0633 \u0642\u0632\u062d",
"\u0637\u064a\u0631 \u0627\u0644\u0648\u0642\u0648\u0627\u0642 \u0643\u0648\u0643\u0627\u0644",
"\u0637\u064a\u0631 \u0648\u0631\u0648\u0627\u0631 ",
"\u0637\u064a\u0631 \u0623\u0628\u0648 \u0642\u0631\u0646",
"\u0627\u0644\u0637\u0627\u0626\u0631 \u0627\u0644\u0637\u0646\u0627\u0646",
"\u0637\u064a\u0648\u0631 \u0627\u0644\u064a\u0642\u0645\u064e\u0631",
"\u0637\u0627\u0626\u0631 \u0645\u0637\u0648\u0642",
"\u0630\u0643\u0631 \u0627\u0644\u0628\u0637",
"\u0627\u0644\u0628\u0637\u0629 \u0627\u0644\u063a\u0648\u0627\u0635\u0629 \u062d\u0645\u0631\u0627\u0621 \u0627\u0644\u0635\u062f\u0631",
"\u0625\u0648\u0632\u0629",
"\u0627\u0644\u0625\u0648\u0632\u0629 \u0633\u0648\u062f\u0627\u0621",
"\u0627\u0644\u0641\u064a\u0644",
" \u0622\u0643\u0644 \u0627\u0644\u0646\u0645\u0644 \u0627\u0644\u0634\u0648\u0643\u064a",
"\u062e\u0644\u062f \u0627\u0644\u0645\u0627\u0621",
"\u0648\u0644\u0628",
"\u0643\u0648\u0627\u0644\u0627",
"\u0627\u0644\u0648\u0645\u0628\u062a\u064a\u0627\u062a",
"\u0642\u0646\u062f\u064a\u0644 \u0628\u062d\u0631",
"\u0634\u0642\u0627\u0626\u0642 \u0646\u0639\u0645\u0627\u0646 \u0627\u0644\u0628\u062d\u0631",
"\u0645\u0631\u062c\u0627\u0646 \u0627\u0644\u062f\u0645\u0627\u063a",
"\u062f\u064a\u062f\u0627\u0646 \u0645\u0633\u0637\u062d\u0629",
"\u062f\u064a\u062f\u0627\u0646 \u0623\u0633\u0637\u0648\u0627\u0646\u064a\u0629",
"\u0645\u062d\u0627\u0631\u0629",
"\u062d\u0644\u0632\u0648\u0646",
"\u0628\u0632\u0627\u0642",
"\u062d\u0644\u0632\u0648\u0646 \u0645\u0627\u0626\u064a",
"\u0631\u062e\u0648\u064a\u0627\u062a \u0628\u062d\u0631\u064a\u0647 \u062a\u0633\u0645\u0649 \u0627\u0644\u062e\u064a\u062a\u0648\u0646",
"\u0646\u0648\u062a\u064a\u0644\u0648\u0633 \u0627\u0644\u062d\u062c\u0631\u064a",
"\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0628\u062d\u0631",
"\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0628\u062d\u0631 \u0627\u0644\u0623\u0637\u0644\u0633\u064a",
"\u0627\u0644\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0639\u0627\u0632\u0641",
"\u0645\u0644\u0643 \u0627\u0644\u0633\u0644\u0637\u0639\u0648\u0646",
"\u062c\u0631\u0627\u062f \u0627\u0644\u0628\u062d\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0643\u0631\u0643\u0646\u062f \u0634\u0627\u0626\u0643",
"\u062c\u0631\u0627\u062f \u0627\u0644\u0645\u064a\u0627\u0647 \u0627\u0644\u0639\u0630\u0628\u0629",
"\u0627\u0644\u0633\u0631\u0637\u0627\u0646 \u0627\u0644\u0646\u0627\u0633\u0643",
"\u0645\u062a\u0633\u0627\u0648\u064a\u0627\u062a \u0627\u0644\u0623\u0642\u062f\u0627\u0645",
"\u0644\u0642\u0644\u0642 \u0623\u0628\u064a\u0636",
"\u0644\u0642\u0644\u0642 \u0623\u0633\u0648\u062f",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0628\u062c\u0639 \u0627\u0644\u0645\u0633\u0645\u0649 \u0623\u0628\u0648 \u0645\u0644\u0639\u0642\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0641\u0644\u0627\u0645\u064a\u0646\u063a\u0648",
"\u0637\u0627\u0626\u0631 \u0628\u0644\u0634\u0648\u0646 \u0627\u0644\u0623\u0632\u0631\u0642 \u0627\u0644\u0635\u063a\u064a\u0631",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0628\u0644\u0634\u0648\u0646 \u0627\u0644\u0623\u0628\u064a\u0636 \u0627\u0644\u0643\u0628\u064a\u0631",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0639\u062c\u0627\u062c",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0643\u0631\u0643\u064a\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0631\u0637\u0627\u0633",
"\u062f\u062c\u0627\u062c\u0629 \u0627\u0644\u0645\u0627\u0621 \u0627\u0644\u0623\u0631\u062c\u0648\u0627\u0646\u064a\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u063a\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u062d\u0628\u0627\u0631",
"\u0637\u0627\u0626\u0631 \u0642\u0646\u0628\u0631\u0629 \u0627\u0644\u0645\u0627\u0621 \u0627\u0644\u0645\u062a\u0648\u0631\u062f",
"\u0637\u0627\u0626\u0631 \u062f\u0631\u064a\u062c\u0629 \u0623\u0644\u0628\u064a\u0629",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0637\u064a\u0637\u0648\u064a \u0623\u062d\u0645\u0631 \u0627\u0644\u0633\u0627\u0642",
"\u0639\u064e\u062f\u0651\u0627\u0621 \u0627\u0644\u0645\u0633\u062a\u0646\u0642\u0639\u0627\u062a \u0623\u0648 \u0627\u0644\u062f\u0651\u062a\u0634\u0631",
"\u0635\u0627\u0626\u062f \u0627\u0644\u0645\u062d\u0627\u0631",
"\u0628\u062c\u0639\u0629",
"\u0627\u0644\u0628\u0637\u0631\u064a\u0642 \u0627\u0644\u0645\u0644\u0643",
"\u0637\u0627\u0626\u0631 \u0627\u0644\u0642\u0637\u0631\u0633",
"\u0627\u0644\u062d\u0648\u062a \u0627\u0644\u0631\u0645\u0627\u062f\u064a \u0627\u0644\u0635\u0644\u0628",
"\u0627\u0644\u062d\u064f\u0648\u062a\u064f \u0627\u0644\u0642\u0627\u062a\u0650\u0644\u064f",
"\u0628\u0642\u0631\u0629 \u0627\u0644\u0628\u062d\u0631",
"\u0623\u0633\u062f \u0627\u0644\u0628\u062d\u0631",
"\u0643\u0644\u0628 \u0634\u064a\u0648\u0627\u0648\u0627",
"\u0643\u0644\u0628 \u0627\u0644\u0630\u0642\u0646 \u0627\u0644\u064a\u0627\u0628\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0645\u0627\u0644\u0637\u064a",
"\u0643\u0644\u0628 \u0628\u0643\u064a\u0646\u064a",
"\u0643\u0644\u0628 \u062a\u0634\u064a\u0647 \u062a\u0632\u0648 \u0627\u0644\u0635\u064a\u0646\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0644\u0643 \u062a\u0634\u0627\u0631\u0644\u0632",
"\u0643\u0644\u0628 \u0628\u0627\u0628\u064a\u0644\u0648\u0646",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0631\u064a\u0631 \u0627\u0644\u0639\u0631\u0636",
"\u0643\u0644\u0627\u0628 \u0631\u064a\u062f\u062c \u0628\u0627\u0643",
"\u0643\u0644\u0628 \u0627\u0644\u0635\u064a\u062f \u0627\u0644\u0623\u0641\u063a\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0627\u0633\u0637",
"\u0643\u0644\u0628 \u0628\u064a\u063a\u0644",
"\u0643\u0644\u0628 \u0627\u0644\u062f\u0645\u0648\u0645",
"\u0643\u0644\u0628 \u0628\u0644\u0648\u064a\u062a\u064a\u0643 \u0643\u0648\u0646\u0647\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0643\u0648\u0646\u0647\u0648\u0646\u062f \u0627\u0644\u0628\u0646\u064a \u0648\u0627\u0644\u0623\u0633\u0648\u062f",
"\u0643\u0644\u0628 \u0627\u0644\u0648\u0648\u0643\u0631",
"\u0643\u0644\u0628 \u0635\u064a\u062f \u0627\u0644\u062b\u0639\u0627\u0644\u0628 \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a",
"\u0643\u0644\u0628 \u0631\u064a\u062f\u0628\u0648\u0646 \u0643\u0648\u0646\u0647\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0628\u0648\u0631\u0632\u0648\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0627\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0633\u0644\u0648\u0642\u064a \u0627\u0644\u0627\u064a\u0637\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0648\u064a\u0628\u062a",
"\u0643\u0644\u0628 \u0627\u064a\u0628\u064a\u0632\u0627\u0646 \u0647\u0648\u0646\u062f",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0646\u0631\u0648\u064a\u062c\u064a",
"\u0643\u0644\u0628 \u0623\u0648\u062a\u064a\u0631 \u0647\u0627\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0633\u0644\u0648\u0642\u064a",
"\u0643\u0644\u0628 \u062f\u064a\u0631 \u0647\u0627\u0648\u0646\u062f \u0627\u0644\u0627\u0633\u0643\u062a\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0648\u0627\u064a\u0645\u0631\u064a",
"\u0643\u0644\u0628 \u0633\u062a\u0627\u0641\u0648\u0631\u062f\u0634\u0627\u064a\u0631 \u0628\u0648\u0644 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0633\u062a\u0627\u0641\u0648\u0631\u062f\u0634\u0627\u064a\u0631 \u0627\u0644\u0623\u0645\u0631\u064a\u0643\u064a",
"\u0643\u0644\u0628 \u0628\u064a\u062f\u0644\u064a\u0646\u062c\u062a\u0648\u0646 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u0631\u062f\u0631 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0643\u064a\u0631\u064a \u0628\u0644\u0648 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0631\u064a\u0631 \u0627\u0644\u0625\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0646\u0648\u0631\u0641\u0648\u0644\u0643 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0646\u0648\u0631\u064a\u062a\u0634 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u064a\u0648\u0631\u0643 \u0634\u0627\u064a\u0631",
"\u0643\u0644\u0628 \u0648\u064a\u0631 \u0641\u0648\u0643\u0633 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0644\u064a\u0643\u0644\u0627\u0646\u062f \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0633\u064a\u0627\u0644\u064a\u0647\u0627\u0645 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0627\u0628 \u0627\u0644\u0623\u0631\u062f\u064a\u0644",
"\u0643\u0644\u0628 \u0643\u064a\u0631\u0646 \u062a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u062a\u0631\u064a\u0631 \u0627\u0633\u062a\u0631\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u062f\u0627\u0646\u062f\u064a \u062f\u064a\u0646\u0645\u0648\u0646\u062a",
"\u0643\u0644\u0628 \u0628\u0648\u0633\u0637\u0646 \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0634\u0646\u0627\u0648\u062a\u0633\u0631 \u0645\u0646\u0645\u0646\u0645",
"\u0643\u0644\u0628 \u0634\u0646\u0627\u0648\u062a\u0633\u0631 \u0627\u0644\u0639\u0645\u0644\u0627\u0642",
"\u0643\u0644\u0628 \u0634\u0646\u0627\u0648\u062a\u0633\u0631 \u0627\u0644\u0639\u0627\u062f\u064a",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0627\u0633\u0643\u062a\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u062a\u0631\u064a\u0631 \u0627\u0644\u062a\u0628\u062a",
"\u0643\u0644\u0628 \u0633\u064a\u0644\u0643\u064a \u062a\u064a\u0631\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u064a\u0631\u064a\u0631 \u0627\u0644\u0642\u0645\u062d\u064a \u0627\u0644\u0646\u0627\u0639\u0645",
"\u0643\u0644\u0628 \u0648\u064a\u0633\u062a\u064a",
"\u0643\u0644\u0628 \u0644\u0627\u0633\u0627 \u0623\u0628\u0633\u0648",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0633\u062a\u0631\u062f \u0627\u0644\u0630\u0647\u0628\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0633\u062a\u0631\u062f \u0645\u062c\u0639\u062f \u0627\u0644\u0634\u0639\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0633\u062a\u0631\u062f \u0627\u0644\u0630\u0647\u0628\u064a",
"\u0643\u0644\u0628 \u0644\u0627\u0628\u0631\u0627\u062f\u0648\u0631 \u0631\u064a\u062a\u0631\u064a\u0641\u0631",
"\u0643\u0644\u0628 \u0634\u064a\u0633\u0628\u064a\u0643\u0627",
"\u0643\u0644\u0628 \u0628\u0648\u064a\u0646\u062a\u0631 \u0627\u0644\u0623\u0644\u0645\u0627\u0646\u064a \u0642\u0635\u064a\u0631 \u0627\u0644\u0634\u0639\u0631",
"\u0643\u0644\u0628 \u0641\u064a\u0632\u0644\u0627",
"\u0643\u0644\u0628 \u0633\u064a\u062a\u0631 \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a",
"\u0643\u0644\u0628 \u0633\u064a\u062a\u0631 \u0627\u0644\u0625\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0633\u064a\u062a\u0631 \u0627\u0644\u062c\u0648\u0631\u062f\u0648\u0646\u064a",
"\u0643\u0644\u0628 \u0628\u0631\u064a\u062a\u0627\u0646\u064a \u0633\u0628\u0627\u0646\u064a\u0644",
"\u0643\u0644\u0628 \u0642\u0644\u0645\u0628\u0631",
"\u0643\u0644\u0628 \u0633\u0628\u0631\u064a\u0646\u063a\u0631 \u0633\u0628\u0627\u0646\u064a\u0644 \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0633\u0628\u0631\u064a\u0646\u063a\u0631 \u0627\u0644\u0648\u064a\u0644\u0632\u064a",
"\u0643\u0644\u0628 \u062f\u0644\u0644 \u0627\u0644\u0630\u0644\u064a\u0644",
"\u0643\u0644\u0628 \u0627\u0644\u0633\u0627\u0643\u0633 \u0627\u0644\u0625\u0633\u0628\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0633\u0628\u0627\u064a\u0646\u0644 \u0627\u0644\u0645\u0627\u0621 \u0627\u0644\u0625\u064a\u0631\u0644\u0646\u062f\u064a",
"\u0643\u0644\u0628 \u0643\u0648\u0641\u0627\u0632",
"\u0643\u0644\u0628 \u0634\u064a\u0628\u0631\u0643",
"\u0643\u0644\u0628 \u062c\u0631\u0648\u0646\u064a\u0646\u062f\u064a\u0644",
"\u0643\u0644\u0628 \u0645\u0627\u0644\u064a\u0646\u0648",
"\u0643\u0644\u0628 \u0628\u0631\u064a\u0627\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u0644\u064a\u0628\u064a \u0627\u0644\u0625\u0633\u062a\u0631\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u0643\u0648\u0645\u0648\u0646\u062f\u0648\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0631\u0627\u0639\u064a \u0627\u0644\u0625\u0646\u062c\u0644\u064a\u0632\u064a \u0627\u0644\u0642\u062f\u064a\u0645",
"\u0643\u0644\u0628 \u0627\u0644\u0631\u0627\u0639\u064a \u0627\u0644\u0634\u062a\u0644\u0646\u062f\u0649",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u0648\u0644\u064a",
"\u0643\u0644\u0628 \u0628\u0648\u0631\u062f\u0631 \u0643\u0648\u0644\u064a",
"\u0643\u0644\u0628 \u0628\u0648\u0641\u064a \u062f\u064a \u0641\u0644\u0627\u0646\u062f\u0631",
"\u0643\u0644\u0628 \u0631\u0648\u062a \u0648\u0627\u064a\u0644\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0631\u0627\u0639\u064a \u0627\u0644\u0623\u0644\u0645\u0627\u0646\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u062f\u0648\u0628\u064a\u0631\u0645\u0627\u0646",
"\u0643\u0644\u0628 \u0628\u064a\u0646\u0634\u0631 \u0627\u0644\u0645\u0635\u063a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u062c\u0628\u0644 \u0627\u0644\u0633\u0648\u064a\u0633\u0631\u064a",
"\u0643\u0644\u0628 \u062c\u0628\u0644 \u0627\u0644\u0628\u0631\u0646\u064a\u0632",
"\u0643\u0644\u0628 \u0627\u067e\u064a\u0646\u0632\u064a\u0644\u064a\u0631 \u0633\u064a\u0646\u064a\u0646\u0647\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0627\u0646\u062a\u0644\u0628\u062a\u0634\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u0643\u0633\u0631",
"\u0643\u0644\u0628 \u0628\u0648\u0644 \u0645\u0627\u0633\u062a\u064a\u0641",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0627\u0633\u062a\u064a\u0641 \u0627\u0644\u062a\u064a\u0628\u062a\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u0644\u062f\u0648\u063a \u0627\u0644\u0641\u0631\u0646\u0633\u064a",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u062f\u0627\u0646\u0645\u0627\u0631\u0643\u064a \u0627\u0644\u0636\u062e\u0645",
"\u0643\u0644\u0628 \u0633\u0627\u0646\u062a \u0628\u0631\u0646\u0627\u0631\u062f",
"\u0643\u0644\u0628 \u0627\u0644\u0647\u0627\u0633\u0643\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0645\u0644\u0645\u0648\u062a \u0627\u0644\u0623\u0644\u0627\u0633\u0643\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0647\u0633\u0643\u064a \u0627\u0644\u0633\u064a\u0628\u064a\u0631\u064a",
"\u0643\u0644\u0628 \u062f\u0644\u0645\u0627\u0633\u064a \u0627\u0644\u0645\u0631\u0642\u0637",
"\u0643\u0644\u0628 \u0623\u0641\u064a\u0646\u0628\u064a\u0646\u0634\u0631",
"\u0643\u0644\u0628 \u0628\u0627\u0633\u0646\u062c\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u062c",
"\u0643\u0644\u0628 \u0644\u064a\u0648\u0646 \u0628\u064a\u0631\u062c\u0631",
"\u0643\u0644\u0628 \u0646\u064a\u0648\u0641\u0627\u0648\u0646\u062f\u0644\u0627\u0646\u062f",
"\u0643\u0644\u0640\u0628 \u062c\u0628\u0627\u0644 \u0627\u0644\u0628\u0631\u0627\u0646\u0633",
"\u0643\u0644\u0628 \u0633\u0627\u0645\u0648\u062f\u064a",
"\u0643\u0644\u0628 \u0628\u0648\u0645\u064a\u0631\u0627\u0646\u064a\u0627\u0646",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0634\u0627\u0648 \u062a\u0634\u0627\u0648",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u064a\u0634\u0648\u0646\u062f",
"\u0643\u0644\u0628 \u0627\u0644\u062c\u0631\u064a\u0641\u0648\u0646",
"\u0643\u0644\u0628\u0628 \u0628\u064a\u0645\u0628\u0631\u0648\u0643 \u0648\u064a\u0644\u0634 \u0643\u0648\u0631\u062c\u0649",
"\u0643\u0644\u0628 \u0627\u0644\u0643\u0627\u0631\u062f\u064a\u062c\u0627\u0646",
"\u0643\u0644\u0628 \u0627\u0644\u062a\u0648\u064a \u0627\u0644\u0628\u0648\u062f\u0644",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u062f\u0644 \u0627\u0644\u0635\u063a\u064a\u0631",
"\u0643\u0644\u0628 \u0627\u0644\u0628\u0648\u062f\u0644 \u0627\u0644\u0642\u064a\u0627\u0633\u064a",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0645\u0643\u0633\u064a\u0643\u064a \u0628\u0644\u0627 \u0634\u0639\u0631",
"\u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0631\u0645\u0627\u062f\u064a",
"\u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0623\u0628\u064a\u0636",
"\u0627\u0644\u0630\u0626\u0628 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0627\u0644\u0642\u064a\u0648\u0637",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0625\u0633\u062a\u0631\u0627\u0644\u064a",
"\u0643\u0644\u0628 \u0627\u0644\u062f\u0648\u0644",
"\u0627\u0644\u0643\u0644\u0628 \u0627\u0644\u0628\u0631\u064a \u0627\u0644\u0625\u0641\u0631\u064a\u0642\u064a",
"\u0627\u0644\u0636\u0628\u0639",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0642\u0632\u0645",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0642\u0637\u0628\u064a",
"\u0627\u0644\u062b\u0639\u0644\u0628 \u0627\u0644\u0631\u0645\u0627\u062f\u064a",
"\u0627\u0644\u0642\u0637\u0637 \u0627\u0644\u0646\u0645\u0631\u064a\u0629",
"\u0627\u0644\u0633\u0646\u0648\u0631 \u0627\u0644\u0645\u064f\u0631\u0642\u0637 \u0627\u0644\u0635\u063a\u064a\u0631",
"\u0642\u0637 \u0634\u064a\u0631\u0627\u0632\u064a",
"\u0627\u0644\u0642\u0637 \u0627\u0644\u0633\u064a\u0627\u0645\u064a",
"\u0642\u0637 \u0644\u064a\u0628\u064a",
"\u0623\u0633\u062f \u0627\u0644\u062c\u0628\u0627\u0644",
"\u0627\u0644\u0648\u064e\u0634\u064e\u0642",
"\u0646\u0645\u0631 ",
"\u0646\u0645\u0631 \u0627\u0644\u062b\u0644\u0648\u062c",
"\u064a\u063a\u0648\u0631",
"\u0623\u0633\u062f",
"\u0627\u0644\u0628\u0628\u0631",
"\u0627\u0644\u0641\u0647\u062f",
"\u062f\u0628 \u0628\u0646\u064a",
"\u062f\u0628 \u0623\u0633\u0648\u062f \u0623\u0645\u0631\u064a\u0643\u064a",
"\u062f\u0628 \u0642\u0637\u0628\u064a",
"\u0627\u0644\u062f\u0628 \u0627\u0644\u0643\u0633\u0644\u0627\u0646 ",
"\u0627\u0644\u0633\u0645\u0648\u0631\u064a\u0627\u062a",
"\u0633\u0631\u0642\u0627\u0637",
"\u062e\u0646\u0627\u0641\u0633 \u0646\u0645\u0631\u064a\u0629",
"\u062f\u0639\u0633\u0648\u0642\u0629",
"\u062e\u0646\u0641\u0633\u0627\u0621 \u0623\u0631\u0636\u064a\u0629",
"\u062e\u0646\u0627\u0641\u0633 \u0637\u0648\u064a\u0644\u0629 \u0627\u0644\u0642\u0631\u0648\u0646 ",
"\u062e\u0646\u0641\u0633\u0629 \u0627\u0644\u0623\u0648\u0631\u0627\u0642",
"\u062e\u0646\u0627\u0641\u0633 \u0627\u0644\u0631\u0648\u062b",
"\u062e\u0646\u0641\u0633\u0627\u0621 \u0648\u062d\u064a\u062f \u0627\u0644\u0642\u0631\u0646",
"\u0627\u0644\u0633\u064f\u0648\u0633\u064a\u0646\u0627\u062a",
"\u062d\u0634\u0631\u0629 \u0630\u0648\u0627\u062a \u0627\u0644\u062c\u0646\u0627\u062d\u064a\u0646",
"\u0646\u062d\u0644",
"\u0627\u0644\u0646\u0645\u0644",
"\u062c\u0646\u062f\u0628",
"\u062d\u0634\u0631\u0629 \u0627\u0644\u0643\u0631\u064a\u0643\u064a\u062a",
"\u0627\u0644\u062d\u0634\u0631\u0629 \u0627\u0644\u0639\u0635\u0648\u064a\u0629",
"\u0635\u0631\u0635\u0648\u0631",
"\u0641\u0631\u0633 \u0627\u0644\u0646\u0628\u064a",
"\u062d\u0634\u0631\u0629 \u0632\u064a\u0632\u064a\u0627\u062a",
"\u0642\u0627\u0641\u0632\u0627\u062a \u0627\u0644\u0623\u0648\u0631\u0627\u0642",
"\u0639\u0631\u0642\u064a\u0627\u062a \u0627\u0644\u0623\u062c\u0646\u062d\u0629",
"\u0627\u0644\u064a\u0639\u0633\u0648\u0628",
"\u0645\u0642\u062a\u0631\u0646\u0627\u062a \u0627\u0644\u0623\u062c\u0646\u062d\u0629",
"\u0641\u0631\u0627\u0634\u0629 \u0628\u0634\u0648\u0631\u0629 \u0627\u0644\u0635\u064a\u0641",
"\u0641\u0631\u0627\u0634\u0629 \u062d\u0644\u064a\u0642\u0629",
"\u0641\u064e\u0631\u064e\u0627\u0634\u0629 \u0627\u0644\u0645\u064e\u0644\u0643",
"\u0641\u0631\u0627\u0634\u0629 \u0627\u0644\u0628\u064a\u0636\u0627\u0621 \u0627\u0644\u0635\u063a\u064a\u0631\u0629",
"\u0641\u0631\u0627\u0634\u0629 \u0627\u0644\u0643\u0628\u0631\u064a\u062a",
"\u0627\u0644\u0641\u0631\u0627\u0634\u0629 \u0627\u0644\u0646\u062d\u0627\u0633\u064a\u0629",
"\u0646\u062c\u0645 \u0627\u0644\u0628\u062d\u0631",
"\u0642\u0646\u0641\u0630 \u0627\u0644\u0628\u062d\u0631",
"\u062e\u064a\u0627\u0631 \u0627\u0644\u0628\u062d\u0631",
"\u0623\u0631\u0627\u0646\u0628 \u0642\u0637\u0646\u064a\u0629 \u0627\u0644\u0630\u064a\u0644",
"\u0623\u0631\u0646\u0628 \u0628\u0631\u064a",
"\u0627\u0644\u0623\u0646\u062c\u0648\u0631\u0627",
"\u0623\u0642\u062f\u0627\u062f",
"\u0627\u0644\u0646\u064a\u0635",
"\u0633\u0646\u062c\u0627\u0628 \u062b\u0639\u0644\u0628\u064a",
"\u0627\u0644\u0645\u0631\u0645\u0648\u0637",
"\u0627\u0644\u0642\u0646\u062f\u0633",
"\u0643\u0627\u0628\u064a\u0627\u0621 \u062e\u0646\u0632\u064a\u0631\u064a\u0629",
"\u0627\u0644\u062d\u0635\u0627\u0646 \u0627\u0644\u062d\u0645\u064a\u0636",
"\u0627\u0644\u062d\u0645\u0627\u0631 \u0627\u0644\u0645\u062e\u0637\u0637",
"\u0627\u0644\u062e\u0646\u0632\u064a\u0631 \u0627\u0644\u0623\u0644\u064a\u0641 \u0623\u0648 \u0627\u0644\u062e\u0646\u0632\u064a\u0631 \u0627\u0644\u0645\u0633\u062a\u0623\u0646\u0633",
"\u0627\u0644\u062e\u0646\u0632\u064a\u0631 \u0627\u0644\u0628\u0631\u064a ",
"\u062e\u0646\u0627\u0632\u064a\u0631 \u062a\u0624\u0644\u0648\u0644\u064a\u0629",
"\u0641\u0631\u0633 \u0627\u0644\u0646\u0647\u0631",
"\u0627\u0644\u062b\u0648\u0631",
"\u062c\u0627\u0645\u0648\u0633 \u0627\u0644\u0645\u0627\u0621",
"\u0627\u0644\u0628\u064a\u0633\u0648\u0646",
"\u0630\u0643\u0631 \u0627\u0644\u062e\u0631\u0648\u0641",
"\u0643\u0628\u0634 \u0627\u0644\u062c\u0628\u0627\u0644 \u0627\u0644\u0635\u062e\u0631\u064a\u0629",
"\u0627\u0644\u0648\u0639\u0644",
"\u062b\u064a\u062a\u0644 \u0627\u0644\u0647\u0631\u062a\u0628\u064a\u0633",
"\u0625\u0645\u0628\u0627\u0644\u0629",
"\u063a\u0632\u0627\u0644",
"\u0627\u0644\u062c\u0645\u0644 \u0627\u0644\u0639\u0631\u0628\u064a",
"\u0627\u0644\u0644\u0627\u0651\u0645\u0629",
"\u0627\u0628\u0646 \u0639\u0631\u0633",
"\u0627\u0644\u0645\u0646\u0643",
"\u0627\u0644\u0633\u0641\u0634\u0629",
"\u0627\u0628\u0646 \u0645\u0642\u0631\u0636 \u0623\u0633\u0648\u062f \u0627\u0644\u0623\u0642\u062f\u0627\u0645",
"\u0642\u0636\u0627\u0639\u0629",
"\u0638\u0631\u0628\u0627\u0646",
"\u0627\u0644\u063a\u0631\u064a\u0631",
"\u0627\u0644\u0645\u064f\u062f\u064e\u0631\u064e\u0651\u0639 \u0623\u0648 \u0627\u0644\u0623\u0631\u0645\u0627\u062f\u064a\u0644\u0644\u0648",
"\u0643\u0633\u0644\u0627\u0646 \u062b\u0644\u0627\u062b\u064a \u0627\u0644\u0623\u0635\u0627\u0628\u0639",
"\u0642\u0631\u062f \u0627\u0644\u0627\u0648\u0631\u0627\u0646\u063a\u0648\u062a\u0627\u0646",
"\u0627\u0644\u063a\u0648\u0631\u064a\u0644\u0627 \u0623\u0648 \u0627\u0644\u0642\u064f\u0631\u062f\u0648\u062d",
"\u0627\u0644\u0634\u0645\u0628\u0627\u0646\u0632\u064a \u0627\u0644\u0634\u0627\u0626\u0639 \u0623\u0648 \u0627\u0644\u0628\u064e\u0639\u0627\u0645",
"\u0642\u0631\u062f \u0627\u0644\u062c\u0628\u0648\u0646",
"\u0642\u0631\u062f \u0627\u0644\u0633\u064a\u0627\u0645\u0646\u062c",
"\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u063a\u064a\u0646\u0648\u0646",
"\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u0628\u0627\u062a\u0627\u0633",
"\u0627\u0644\u0631\u064f\u0628\u064e\u0651\u0627\u062d",
"\u0642\u0631\u062f \u0627\u0644\u0645\u0643\u0627\u0643",
"\u0642\u0631\u062f \u0627\u0644\u0643\u0648\u0644\u0628\u0633\u0627\u0648\u0627\u062a",
"\u0642\u0631\u062f \u0627\u0644\u0643\u0648\u0644\u0628\u0633",
"\u0642\u0631\u062f \u0627\u0644\u0645\u0644\u0645\u0644\u0629",
"\u0642\u0631\u0648\u062f \u0627\u0644\u0642\u0634\u0629",
"\u0642\u0631\u062f \u0627\u0644\u0643\u0628\u0648\u0634\u0629 \u0623\u0628\u064a\u0636 \u0627\u0644\u0648\u062c\u0647",
"\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u0639\u0648\u0627\u0621",
"\u0642\u0631\u062f \u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u062a\u064a\u062a\u064a",
"\u0627\u0644\u0633\u064e\u0651\u0639\u062f\u0627\u0646 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a\u064a",
"\u0627\u0644\u0633\u0639\u062f\u0627\u0646 \u0627\u0644\u0633\u0646\u062c\u0627\u0628\u064a",
"\u0644\u064a\u0645\u0648\u0631 \u062d\u0644\u0642\u064a \u0627\u0644\u0630\u064a\u0644",
"\u062d\u064a\u0648\u0627\u0646 \u0627\u0644\u0627\u0646\u062f\u0631\u064a",
"\u0641\u064a\u0644 \u0647\u0646\u062f\u064a",
"\u0641\u064a\u0644 \u0623\u0641\u0631\u064a\u0642\u064a",
"\u0627\u0644\u0628\u0627\u0646\u062f\u0627 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0627\u0644\u0628\u0627\u0646\u062f\u0627 \u0627\u0644\u0639\u0645\u0644\u0627\u0642\u0629",
"\u062b\u064a\u0631\u0633\u064a\u062a\u064a\u0627\u062a",
"\u0633\u0645\u0643 \u0627\u0644\u0627\u0646\u0642\u0644\u064a\u0633",
"\u0633\u0645\u0643 \u0627\u0644\u0643\u0648\u0647\u0648 \u0627\u0644\u0633\u064a\u0644\u0645\u0648\u0646",
"\u0633\u0645\u0643 \u0627\u0644\u062c\u0645\u0627\u0644 \u0627\u0644\u0635\u062e\u0631\u064a",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u0645\u0647\u0631\u062c",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u062d\u0641\u0634\u064a\u0629",
"\u0633\u0645\u0643 \u0627\u0644\u0631\u0645\u062d",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u062a\u0646\u064a\u0646",
"\u0633\u0645\u0643\u0629 \u0627\u0644\u064a\u0646\u0641\u0648\u062e\u064a\u0629",
"\u0627\u0644\u0645\u0650\u0639\u0652\u062f\u064e\u0627\u062f",
"\u0627\u0644\u0639\u0628\u0627\u0621\u0629",
"\u0644\u0628\u0627\u0633 \u062a\u062e\u0631\u062c",
"\u0623\u0643\u0648\u0631\u062f\u064a\u0648\u0646",
"\u0627\u0644\u0642\u064a\u062b\u0627\u0631\u0629 \u0627\u0644\u0635\u0648\u062a\u064a\u0629",
"\u062d\u0627\u0645\u0644\u0629 \u0637\u0627\u0626\u0631\u0627\u062a",
"\u0637\u0627\u0626\u0631\u0629 \u0631\u062d\u0644\u0627\u062a",
"\u0633\u0641\u064a\u0646\u0629 \u0647\u0648\u0627\u0626\u064a\u0629",
"\u0645\u0630\u0628\u062d",
"\u0633\u064a\u0627\u0631\u0629 \u0625\u0633\u0639\u0627\u0641",
"\u0627\u0644\u0645\u0631\u0643\u0628\u0629 \u0627\u0644\u0628\u0631\u0645\u0627\u0626\u064a\u0629",
"\u0627\u0644\u0633\u0627\u0639\u0629 \u0627\u0644\u0645\u062a\u0646\u0627\u0638\u0631\u0629",
"\u0627\u0644\u0645\u0646\u062d\u0644 \u0623\u0648 \u0627\u0644\u0645\u064e\u0646\u062d\u064e\u0644\u064e\u0629",
"\u0645\u0626\u0632\u0631",
"\u062d\u0627\u0648\u064a\u0629 \u0627\u0644\u0646\u0641\u0627\u064a\u0627\u062a",
"\u0628\u0646\u062f\u0642\u064a\u0629 \u0627\u0642\u062a\u062d\u0627\u0645",
"\u062d\u0642\u064a\u0628\u0629 \u0638\u0647\u0631",
"\u0627\u0644\u0645\u062e\u0628\u0632",
"\u0639\u0627\u0631\u0636\u0629 \u0627\u0644\u062a\u0648\u0627\u0632\u0646",
"\u0627\u0644\u0628\u0627\u0644\u0648\u0646",
"\u0642\u0644\u0645 \u062d\u0628\u0631 \u062c\u0627\u0641",
"\u0636\u0645\u0627\u062f\u0629 \u0637\u0628\u064a\u0629 \u0644\u0627\u0635\u0642\u0629",
"\u0627\u0644\u0628\u0627\u0646\u062c\u0648",
"\u062f\u0631\u0627\u0628\u0632\u064a\u0646",
"\u062d\u062f\u064a\u062f\u0629 (\u0631\u0641\u0639 \u0623\u062b\u0642\u0627\u0644)",
"\u0643\u0631\u0633\u064a \u0627\u0644\u062d\u0644\u0627\u0642\u0629",
"\u0645\u062d\u0644 \u0635\u0627\u0644\u0648\u0646 \u0627\u0644\u062d\u0644\u0627\u0642\u0629",
"\u062d\u0638\u064a\u0631\u0629",
"\u0627\u0644\u0628\u0627\u0631\u0648\u0645\u062a\u0631",
"\u0627\u0644\u0628\u0631\u0645\u064a\u0644",
"\u0639\u062c\u0644\u0629 \u0627\u0644\u064a\u062f",
"\u0643\u0631\u0629 \u0627\u0644\u0642\u0627\u0639\u062f\u0629 \u0623\u0648 \u0627\u0644\u0628\u064a\u0633\u0628\u0648\u0644",
"\u0643\u0631\u0629 \u0633\u0644\u0629",
"\u0633\u0631\u064a\u0631 \u0627\u0644\u0623\u0637\u0641\u0627\u0644",
"\u0645\u0632\u0645\u0627\u0631",
"\u0642\u0628\u0639\u0629 \u0633\u0628\u0627\u062d\u0629",
"\u0645\u0646\u0634\u0641\u0629",
"\u062d\u0648\u0636 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u0633\u064a\u0627\u0629 \u0648\u0627\u063a\u0646",
"\u0627\u0644\u0645\u0646\u0627\u0631\u0629 \u0623\u0648 \u0627\u0644\u0641\u0646\u0627\u0631",
"\u0643\u0648\u0628 \u0632\u062c\u0627\u062c\u064a",
"\u0642\u0628\u0639\u0629 \u0627\u0644\u062f\u0628",
"\u0632\u062c\u0627\u062c\u0629 \u0627\u0644\u0628\u064a\u0631\u0629",
"\u0643\u0623\u0633 \u062c\u0639\u0629",
"\u0628\u0631\u062c \u0627\u0644\u0646\u0627\u0642\u0648\u0633",
"\u0645\u0631\u0648\u0644\u0629",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u062a\u0631\u0627\u062f\u0641\u064a\u0629",
"\u0628\u0643\u064a\u0646\u064a",
"\u0627\u0644\u0645\u062c\u0644\u062f\u0627\u062a \u0627\u0644\u062d\u0644\u0642\u064a\u0629",
"\u0627\u0644\u0645\u0650\u0646\u0652\u0638\u0627\u0631",
"\u0635\u0646\u062f\u0648\u0642 \u0627\u0644\u0639\u0634",
"\u0627\u0644\u0645\u0631\u0641\u0623",
"\u0627\u0644\u0632\u0644\u0627\u062c\u0629 \u0627\u0644\u062c\u0645\u0627\u0639\u064a\u0629",
"\u0631\u0628\u0637\u0629 \u0639\u0646\u0642 \u0628\u0648\u0644\u0648",
"\u0627\u0644\u0643\u0632\u0629",
"\u0631\u0641 \u0627\u0644\u0643\u062a\u0628",
"\u0645\u0643\u062a\u0628\u0629",
"\u063a\u0637\u0627\u0621 \u0642\u0627\u0631\u0648\u0631\u0629",
"\u0627\u0644\u0642\u0648\u0633",
"\u0623\u0631\u0628\u0629 \u0641\u0631\u0627\u0634\u064a\u0629",
"\u0644\u0627\u0641\u062a\u0629 \u062a\u0627\u0631\u064a\u062e\u064a\u0629 \u0645\u0646 \u0627\u0644\u0646\u062d\u0627\u0633",
"\u062d\u0645\u0627\u0644\u0629 \u0627\u0644\u0635\u062f\u0631",
"\u062d\u0627\u062c\u0632 \u0627\u0644\u0623\u0645\u0648\u0627\u062c",
"\u062f\u0631\u0639 \u0627\u0644\u0635\u062f\u0631",
"\u0627\u0644\u0645\u0643\u0646\u0633\u0629",
"\u0627\u0644\u062f\u0644\u0648",
"\u0645\u0631\u0628\u0637 \u0627\u0644\u062d\u0632\u0627\u0645",
"\u0627\u0644\u0633\u062a\u0631\u0629 \u0627\u0644\u0648\u0627\u0642\u064a\u0629 \u0645\u0646 \u0627\u0644\u0631\u0635\u0627\u0635",
"\u0642\u0637\u0627\u0631 \u0627\u0644\u0637\u0644\u0642\u0629",
"\u0627\u0644\u0645\u062c\u0632\u0631\u0629",
"\u0633\u064a\u0627\u0631\u0629 \u0623\u062c\u0631\u0629",
"\u062d\u0644\u0629 (\u0622\u0646\u064a\u0629)",
"\u0634\u0645\u0639\u0629",
"\u0645\u062f\u0641\u0639",
"\u0642\u0627\u0631\u0628 \u0627\u0644\u0643\u0627\u0646\u0648",
"\u0641\u0627\u062a\u062d\u0629 \u0639\u0644\u0628",
"\u0633\u062a\u0631\u0629 \u0645\u062d\u0628\u0648\u0643\u0629",
"\u0627\u0644\u0645\u0631\u0622\u0629 \u0627\u0644\u062c\u0627\u0646\u0628\u064a\u0629",
"\u062f\u0648\u0627\u0645\u0629 \u0627\u0644\u062e\u064a\u0644",
" \u0623\u062f\u0648\u0627\u062a \u0627\u0644\u0635\u064a\u0627\u0646\u0629",
"\u0635\u0646\u062f\u0648\u0642 \u0643\u0631\u062a\u0648\u0646",
"\u0627\u0644\u0625\u0637\u0627\u0631 \u0627\u0644\u0645\u0637\u0627\u0637",
"\u0627\u0644\u0635\u0631\u0627\u0641 \u0627\u0644\u0622\u0644\u064a",
"\u0627\u0644\u0634\u0631\u064a\u0637 \u0627\u0644\u0645\u062f\u0645\u062c",
"\u0627\u0644\u0645\u0633\u062c\u0644",
"\u0627\u0644\u0642\u064e\u0644\u0652\u0639\u064e\u0629",
"\u0627\u0644\u0642\u0637\u0645\u0631\u0627\u0646",
"\u062c\u0647\u0627\u0632 \u0627\u0644\u0642\u0631\u0635 \u0627\u0644\u0645\u0636\u063a\u0648\u0637",
"\u062a\u0634\u064a\u0644\u0648",
"\u0647\u0627\u062a\u0641 \u0645\u062d\u0645\u0648\u0644",
"\u0633\u0644\u0633\u0644\u0629",
"\u0633\u064a\u0627\u062c \u0645\u0634\u0628\u0643",
"\u0627\u0644\u0632\u0631\u062f",
"\u0645\u0646\u0634\u0627\u0631 \u062c\u0646\u0632\u064a\u0631\u064a",
" \u0635\u0646\u062f\u0648\u0642 \u0627\u0644\u062a\u062e\u0632\u064a\u0646",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0623\u062b\u0627\u062b",
"\u0627\u0644\u0622\u0644\u0629 \u0627\u0644\u0625\u064a\u0642\u0627\u0639\u064a\u0629 ",
"\u0627\u0644\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0635\u064a\u0646\u064a\u0629",
"\u062c\u0648\u0631\u0628 \u0639\u064a\u062f \u0627\u0644\u0645\u064a\u0644\u0627\u062f",
"\u0643\u0646\u064a\u0633\u0629",
"\u0645\u0633\u0631\u062d \u0623\u0641\u0644\u0627\u0645",
"\u0627\u0644\u0633\u0627\u0637\u0648\u0631",
"\u0645\u0633\u0627\u0643\u0646 \u0627\u0644\u062c\u0631\u0641",
"\u0627\u0644\u0645\u0639\u0637\u0641 \u0627\u0644\u0641\u0636\u0641\u0627\u0636",
"\u0627\u0644\u0642\u0628\u0642\u0627\u0628",
"\u062e\u0627\u0644\u0637 \u0627\u0644\u0645\u0634\u0631\u0648\u0628\u0627\u062a \u0627\u0644\u0643\u062d\u0648\u0644\u064a\u0629",
"\u0643\u0648\u0632 (\u0622\u0646\u064a\u0629)",
"\u0622\u0644\u0629 \u062a\u062d\u0636\u064a\u0631 \u0627\u0644\u0642\u0647\u0648\u0629",
"\u0627\u0644\u0634\u0643\u0644 \u0627\u0644\u062d\u0644\u0632\u0648\u0646\u064a",
"\u0627\u0644\u0642\u0641\u0644 \u0627\u0644\u0631\u0645\u0632\u064a",
"\u0644\u0648\u062d\u0629 \u0627\u0644\u0645\u0641\u0627\u062a\u064a\u062d",
"\u0645\u062a\u062c\u0631 \u0627\u0644\u062d\u0644\u0648\u064a\u0627\u062a",
"\u0633\u0641\u064a\u0646\u0629 \u062d\u0627\u0648\u064a\u0627\u062a",
"\u0633\u064a\u0627\u0631\u0629 \u0645\u0643\u0634\u0648\u0641\u0629",
"\u0628\u0631\u0627\u0645\u0629",
"\u0627\u0644\u0634\u064a\u0627\u0639",
"\u062c\u0632\u0645\u0629 \u0631\u0627\u0639\u064a \u0627\u0644\u0628\u0642\u0631",
"\u0642\u0628\u0639\u0629 \u0631\u0627\u0639\u064a \u0627\u0644\u0628\u0642\u0631",
"\u0627\u0644\u0645\u0647\u062f",
"\u0631\u0627\u0641\u0639\u0629",
"\u0627\u0644\u062e\u0648\u0630\u0629",
"\u062d\u0627\u0648\u064a\u0629 \u0634\u062d\u0646 \u0643\u0628\u064a\u0631\u0629",
"\u0633\u0631\u064a\u0631 \u0627\u0644\u0631\u0636\u064a\u0639",
"\u0642\u062f\u0631 \u0627\u0644\u0637\u0628\u062e \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0643\u0631\u0648\u0643\u064a\u062a",
"\u0627\u0644\u0639\u0643\u0627\u0632",
"\u0643\u0648\u064a\u0631\u0633",
"\u0627\u0644\u0633\u062f",
"\u0627\u0644\u0645\u0643\u062a\u0628",
"\u062d\u0627\u0633\u0648\u0628 \u0645\u0643\u062a\u0628\u064a",
"\u0627\u0644\u0647\u0627\u062a\u0641 \u0627\u0644\u062f\u0648\u0627\u0631",
"\u0627\u0644\u062d\u0641\u0627\u0638\u0629",
"\u0627\u0644\u0633\u0627\u0639\u0629 \u0627\u0644\u0631\u0642\u0645\u064a\u0629",
"\u0633\u0627\u0639\u0627\u062a \u0627\u0644\u064a\u062f \u0627\u0644\u0631\u0642\u0645\u064a\u0629",
"\u0627\u0644\u0645\u0646\u0636\u062f\u0629",
"\u0642\u0645\u0627\u0634 \u0627\u0644\u0623\u0637\u0628\u0627\u0642",
"\u063a\u0633\u0627\u0644\u0629 \u0635\u062d\u0648\u0646",
"\u0645\u0643\u0628\u062d \u0642\u0631\u0635\u064a",
"\u0645\u064a\u0646\u0627\u0621",
"\u0627\u0644\u0632\u0644\u0627\u062c\u0629 \u0627\u0644\u062a\u064a \u062a\u062c\u0631\u0647\u0627 \u0627\u0644\u0643\u0644\u0627\u0628",
"\u0642\u0628\u0629",
"\u0627\u0644\u062d\u0635\u064a\u0631\u0629",
"\u0645\u0646\u0635\u0629 \u062d\u0641\u0631",
"\u0627\u0644\u0637\u0628\u0644",
"\u0639\u0635\u0627 \u0627\u0644\u0637\u0628\u0644",
"\u062b\u0642\u0627\u0644\u0627\u062a \u062d\u062f\u064a\u062f",
"\u0641\u0631\u0646 \u0647\u0648\u0644\u0646\u062f\u064a",
"\u0645\u0631\u0648\u062d\u0629",
"\u0627\u0644\u062c\u064a\u062a\u0627\u0631 \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0627\u0644\u0642\u0627\u0637\u0631\u0629 \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a\u0629",
"\u0645\u0631\u0643\u0632 \u0627\u0644\u062a\u0631\u0641\u064a\u0647",
"\u0638\u0631\u0641 \u0628\u0631\u064a\u062f\u064a",
"\u0622\u0644\u0629 \u0627\u0644\u0625\u0633\u0628\u0631\u064a\u0633\u0648",
"\u0628\u0648\u062f\u0631\u0629 \u0627\u0644\u0648\u062c\u0647",
"\u0623\u0635\u0644\u0629 \u0631\u064a\u0634\u064a\u0629",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0645\u0644\u0641\u0627\u062a",
"\u0632\u0648\u0631\u0642 \u0627\u0644\u0625\u0637\u0641\u0627\u0621",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0625\u0637\u0641\u0627\u0621",
"\u0648\u0627\u0642\u064a \u0627\u0644\u0646\u0627\u0631",
"\u0633\u0627\u0631\u064a\u0629 \u0627\u0644\u0639\u0644\u0645",
"\u0627\u0644\u0641\u0644\u0648\u062a",
"\u0643\u0631\u0633\u064a \u0642\u0627\u0628\u0644 \u0644\u0644\u0637\u064a",
"\u062e\u0648\u0630\u0629 \u0643\u0631\u0629 \u0627\u0644\u0642\u062f\u0645",
"\u0631\u0627\u0641\u0639\u0629 \u0627\u0644\u062d\u0645\u0648\u0644\u0629",
"\u0627\u0644\u0646\u0627\u0641\u0648\u0631\u0629",
"\u0642\u0644\u0645 \u062d\u0628\u0631 \u0633\u0627\u0626\u0644",
"\u0627\u0644\u0633\u0631\u064a\u0631 \u0628\u0623\u0631\u0628\u0639\u0629 \u0623\u0639\u0645\u062f\u0629",
"\u0633\u064a\u0627\u0631\u0629 \u0634\u062d\u0646",
"\u0627\u0644\u0628\u0648\u0642 \u0627\u0644\u0641\u0631\u0646\u0633\u064a",
"\u0645\u0642\u0644\u0627\u0629",
"\u0627\u0644\u0645\u0644\u0627\u0628\u0633 \u0627\u0644\u0645\u0635\u0646\u0648\u0639\u0629 \u0645\u0646 \u0627\u0644\u0641\u0631\u0648\u0629",
"\u0634\u0627\u062d\u0646\u0629 \u0642\u0645\u0627\u0645\u0629",
"\u0642\u0646\u0627\u0639 \u0627\u0644\u063a\u0627\u0632",
"\u0645\u0636\u062e\u0629 \u0627\u0644\u0648\u0642\u0648\u062f",
"\u0643\u0623\u0633 \u0627\u0644\u0646\u0628\u064a\u0630",
"\u0633\u064a\u0627\u0631\u0629 \u062c\u0648 \u0643\u0627\u0631\u062a",
"\u0643\u0631\u0629 \u0627\u0644\u062c\u0648\u0644\u0641",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062c\u0648\u0644\u0641",
"\u0627\u0644\u0642\u0627\u0631\u0628 \u0627\u0644\u0637\u0648\u064a\u0644 \u0623\u0648 \u0627\u0644\u063a\u0646\u062f\u0648\u0644",
"\u0627\u0644\u0635\u0646\u062c\u0629",
"\u0627\u0644\u062b\u0648\u0628 \u0627\u0644\u0646\u0633\u0627\u0626\u064a \u0623\u0648 \u0627\u0644\u0641\u0633\u062a\u0627\u0646",
"\u0627\u0644\u0628\u064a\u0627\u0646\u0648 \u0627\u0644\u0643\u0628\u064a\u0631",
"\u0627\u0644\u062f\u0641\u064a\u0626\u0629 \u0627\u0644\u0632\u0631\u0627\u0639\u064a\u0629",
"\u0634\u0628\u0643 \u0627\u0644\u0633\u064a\u0627\u0631\u0629",
"\u0627\u0644\u0628\u0642\u0627\u0644\u0629",
"\u0627\u0644\u0645\u0642\u0635\u0644\u0629",
"\u0645\u0634\u0628\u0643 \u0644\u0644\u0634\u0639\u0631",
"\u0628\u062e\u0627\u062e \u0645\u062b\u0628\u062a \u0627\u0644\u0634\u0639\u0631",
"\u0627\u0644\u0639\u0631\u0628\u0629 \u0646\u0635\u0641 \u0627\u0644\u0645\u062c\u0646\u0632\u0631\u0629",
"\u0645\u0637\u0631\u0642\u0629",
"\u0627\u0644\u0633\u0644\u0629",
"\u0645\u062c\u0641\u0641 \u0627\u0644\u0634\u0639\u0631",
"\u062c\u0647\u0627\u0632 \u0645\u062d\u0645\u0648\u0644 \u0628\u0627\u0644\u064a\u062f",
"\u0627\u0644\u0645\u0646\u062f\u064a\u0644 \u0623\u0648 \u0627\u0644\u0645\u062d\u0631\u0645\u0629",
"\u0642\u0631\u0635 \u0635\u0644\u0628",
"\u0627\u0644\u0634\u064e\u0651\u0641\u064e\u0648\u0650\u064a\u064e\u0651\u0629",
"\u0627\u0644\u0642\u064a\u062b\u0627\u0631\u0629",
"\u0627\u0644\u062d\u0635\u0651\u0627\u062f\u0629",
"\u062e\u0635\u064a\u0646",
"\u062d\u0627\u0641\u0638\u0629 \u0627\u0644\u0645\u0633\u062f\u0633",
"\u0627\u0644\u0645\u0633\u0631\u062d \u0627\u0644\u0645\u0646\u0632\u0644\u064a",
"\u0642\u0631\u0635 \u0639\u0633\u0644",
"\u0627\u0644\u062e\u0637\u0627\u0641",
"\u0627\u0644\u062a\u0646\u0648\u0631\u0629 \u0627\u0644\u0645\u064f\u0637\u064e\u0648\u064e\u0651\u0642\u0629",
"\u0639\u0642\u0644\u0629 (\u062c\u0645\u0628\u0627\u0632)",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062e\u064a\u0648\u0644",
"\u0633\u0627\u0639\u0629 \u0631\u0645\u0644\u064a\u0629",
"\u0622\u064a \u0628\u0648\u062f",
"\u0627\u0644\u0645\u0643\u0648\u0627\u0629",
"\u0627\u0644\u0642\u0631\u0639\u0629 \u0627\u0644\u0645\u0636\u064a\u0626\u0629",
"\u0627\u0644\u062c\u064a\u0646\u0632",
"\u062c\u064a\u0628 (\u0633\u064a\u0627\u0631\u0629)",
"\u0642\u0645\u064a\u0635 \u0642\u0635\u064a\u0631 \u0627\u0644\u0643\u0645\u064a\u0646",
"\u0623\u062d\u062c\u064a\u0629 \u0627\u0644\u0635\u0648\u0631 \u0627\u0644\u0645\u0642\u0637\u0648\u0639\u0629",
"\u0631\u064a\u0643\u0634\u0627",
"\u0630\u0631\u0627\u0639 \u0627\u0644\u062a\u0648\u062c\u064a\u0647",
"\u0627\u0644\u0643\u064a\u0645\u0648\u0646\u0648",
"\u0648\u0633\u0627\u062f\u0627\u062a \u0627\u0644\u0631\u0643\u0628\u0629",
"\u0627\u0644\u0639\u0642\u062f\u0629",
"\u0645\u0639\u0637\u0641 \u0627\u0644\u0645\u062e\u062a\u0628\u0631",
"\u0627\u0644\u0645\u063a\u0631\u0641\u0629",
"\u0639\u0627\u0643\u0633 \u0627\u0644\u0636\u0648\u0621",
"\u062d\u0627\u0633\u0648\u0628 \u0645\u062d\u0645\u0648\u0644",
"\u062c\u0632\u0627\u0632\u0629 \u0627\u0644\u0639\u0634\u0628",
"\u063a\u0637\u0627\u0621 \u0627\u0644\u0639\u062f\u0633\u0629",
"\u0641\u062a\u0627\u062d\u0629 \u0627\u0644\u0631\u0633\u0627\u0626\u0644",
"\u0645\u0643\u062a\u0628\u0629",
"\u0642\u0627\u0631\u0628 \u0627\u0644\u0646\u062c\u0627\u0629",
"\u0627\u0644\u0642\u064e\u062f\u064e\u0651\u0627\u062d\u064e\u0629",
"\u0627\u0644\u0644\u064a\u0645\u0648\u0632\u064a\u0646",
"\u0639\u0627\u0628\u0631\u0629 \u0645\u062d\u064a\u0637 \u0645\u0646\u062a\u0638\u0645\u0629",
"\u0623\u062d\u0645\u0631 \u0634\u0641\u0627\u0647",
"\u0627\u0644\u062d\u0630\u0627\u0621 \u0633\u0647\u0644 \u0627\u0644\u0627\u0631\u062a\u062f\u0627\u0621",
"\u063a\u0633\u0648\u0644",
"\u0645\u0643\u0628\u0631 \u0627\u0644\u0635\u0648\u062a",
"\u0639\u062f\u0633\u0629",
"\u0627\u0644\u0645\u0646\u0634\u0631\u0629",
"\u0627\u0644\u0628\u0648\u0635\u0644\u0629",
"\u062d\u0642\u064a\u0628\u0629 \u0633\u0627\u0639\u064a \u0627\u0644\u0628\u0631\u064a\u062f",
"\u0635\u0646\u062f\u0648\u0642 \u0628\u0631\u064a\u062f",
"\u0627\u0644\u062c\u0648\u0627\u0631\u0628 \u0627\u0644\u0637\u0648\u064a\u0644\u0629",
"\u0627\u0644\u0645\u0627\u064a\u0648\u0647",
"\u063a\u0637\u0627\u0621 \u0627\u0644\u0645\u0637\u0628\u0642",
"\u0622\u0644\u0629 \u0645\u0627\u0631\u0627\u0643\u0633",
"\u0627\u0644\u0645\u0627\u0631\u064a\u0645\u0628\u0627",
"\u0642\u0646\u0627\u0639",
"\u0623\u0639\u0648\u0627\u062f \u0627\u0644\u062b\u0642\u0627\u0628",
"\u0633\u0627\u0631\u064a\u0629 \u0645\u0627\u064a\u0648",
"\u0627\u0644\u0645\u062a\u0627\u0647\u0629",
"\u0643\u0648\u0628 \u0627\u0644\u0642\u064a\u0627\u0633",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0623\u062f\u0648\u064a\u0629",
"\u0627\u0644\u0622\u062b\u0627\u0631 \u0627\u0644\u0635\u062e\u0631\u064a\u0629",
"\u0627\u0644\u0644\u0627\u0642\u0637 \u0627\u0644\u0635\u0648\u062a\u064a",
"\u0641\u0631\u0646 \u0627\u0644\u0645\u064a\u0643\u0631\u0648\u064a\u0641",
"\u0627\u0644\u0632\u064a \u0627\u0644\u0639\u0633\u0643\u0631\u064a",
"\u0645\u062f\u0644\u062c\u0629",
"\u0627\u0644\u062d\u0627\u0641\u0644\u0629 \u0627\u0644\u0635\u063a\u064a\u0631\u0629",
"\u0627\u0644\u062a\u0646\u0648\u0631\u0629 \u0627\u0644\u0642\u0635\u064a\u0631\u0629",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0645\u064a\u0646\u064a \u0641\u0627\u0646 \u0627\u0644\u0639\u0627\u0626\u0644\u064a\u0629",
"\u0627\u0644\u0642\u0630\u064a\u0641\u0629 \u0627\u0644\u0645\u0648\u062c\u0647\u0629",
"\u0627\u0644\u0642\u0641\u0627\u0632 \u0645\u0644\u062a\u0635\u0642 \u0627\u0644\u0623\u0635\u0627\u0628\u0639",
"\u0637\u0628\u0642 \u062e\u0644\u0637",
"\u0627\u0644\u0645\u0646\u0632\u0644 \u0627\u0644\u0645\u062a\u0646\u0642\u0644",
"\u0641\u0648\u0631\u062f \u0645\u0648\u062f\u064a\u0644 \u062a\u064a",
"\u0627\u0644\u0645\u0648\u062f\u0645",
"\u0627\u0644\u062f\u064a\u0631",
"\u0634\u0627\u0634\u0629 \u062d\u0627\u0633\u0648\u0628",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u0646\u0627\u0631\u064a\u0629 \u0627\u0644\u0635\u063a\u064a\u0631\u0629",
"\u0627\u0644\u0647\u0627\u0648\u0646 \u0648\u0627\u0644\u0645\u062f\u0642\u0629",
"\u0627\u0644\u0642\u0628\u0639\u0629 \u0627\u0644\u062c\u0627\u0645\u0639\u064a\u0629 \u0627\u0644\u0645\u0631\u0628\u0639\u0629",
"\u0645\u0633\u062c\u062f",
"\u0627\u0644\u0646\u0627\u0645\u0648\u0633\u064a\u0651\u0629",
"\u0627\u0644\u062f\u0639\u0631\u0648\u0645\u0629",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u0647\u0648\u0627\u0626\u064a\u0629 \u0627\u0644\u062c\u0628\u0644\u064a\u0629",
"\u062e\u064a\u0645\u0629",
"\u0627\u0644\u0641\u0623\u0631\u0629",
"\u0645\u0635\u064a\u062f\u0629 \u0627\u0644\u0641\u0626\u0631\u0627\u0646",
"\u0633\u064a\u0627\u0631\u0627\u062a \u0634\u0631\u0643\u0629 \u0627\u0644\u0646\u0642\u0644",
"\u0643\u0645\u0627\u0645 \u0627\u0644\u0641\u0645",
"\u0645\u0633\u0645\u0627\u0631",
"\u0637\u0648\u0642 \u0627\u0644\u0639\u0646\u0642",
"\u0627\u0644\u0642\u0644\u0627\u062f\u0629",
"\u0627\u0644\u0631\u0636\u0627\u0639\u0629",
"\u062d\u0627\u0633\u0628 \u0627\u0644\u0645\u0641\u0643\u0631\u0629",
"\u0627\u0644\u0645\u0633\u0644\u0629",
"\u0622\u0644\u0629 \u0627\u0644\u0623\u0648\u0628\u0648\u0627",
"\u0623\u0643\u0631\u064a\u0646\u0629",
"\u0639\u062f\u0627\u062f \u0627\u0644\u0645\u0633\u0627\u0641\u0627\u062a",
"\u0641\u0644\u062a\u0631 \u0627\u0644\u0632\u064a\u062a",
"\u0627\u0644\u0623\u0631\u063a\u0646 \u0630\u0648 \u0627\u0644\u0623\u0646\u0627\u0628\u064a\u0628",
"\u062c\u0647\u0627\u0632 \u0631\u0627\u0633\u0645 \u0627\u0644\u0625\u0634\u0627\u0631\u0629",
"\u0627\u0644\u062a\u0646\u0648\u0631\u0629 \u0627\u0644\u062e\u0627\u0631\u062c\u064a\u0629",
"\u0639\u0631\u0628\u0629 \u064a\u062c\u0631\u0647\u0627 \u0627\u0644\u062b\u0648\u0631",
"\u0642\u0646\u0627\u0639 \u0623\u0643\u0633\u062c\u064a\u0646",
"\u0627\u0644\u062a\u063a\u0644\u064a\u0641",
"\u0627\u0644\u0645\u0650\u063a\u0652\u062f\u0627\u0641",
"\u0639\u064e\u062c\u064e\u0644\u0629 \u0627\u0644\u062a\u064e\u063a\u062f\u064a\u0641",
"\u0642\u0641\u0644 \u062d\u0644\u0642\u064a",
"\u0641\u0631\u0634\u0627\u0629 \u0627\u0644\u0631\u0633\u0645",
"\u0627\u0644\u0645\u0650\u0646\u064e\u0627\u0645\u064e\u0629\u064f",
"\u0642\u0635\u0631",
"\u0627\u0644\u0645\u0650\u0635\u0641\u0627\u0631",
"\u0645\u0646\u0634\u0641\u0629 \u0648\u0631\u0642\u064a\u0629",
"\u0645\u0650\u0638\u064e\u0644\u064e\u0651\u0629 \u0627\u0644\u0647\u0628\u064f\u0648\u0637",
"\u062c\u0647\u0627\u0632 \u0627\u0644\u0639\u0642\u0644\u0629",
"\u0645\u0642\u0639\u062f \u0639\u0627\u0645",
"\u0639\u062f\u0627\u062f \u0627\u0646\u062a\u0638\u0627\u0631 \u0627\u0644\u0633\u064a\u0627\u0631\u0627\u062a",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u0642\u0637\u0627\u0631",
"\u0627\u0644\u0641\u0646\u0627\u0621",
"\u0627\u0644\u0647\u0627\u062a\u0641 \u0627\u0644\u0639\u0645\u0648\u0645\u064a",
"\u0627\u0644\u0631\u0643\u064a\u0632\u0629",
"\u0627\u0644\u0645\u064e\u0642\u0652\u0644\u064e\u0645\u064e\u0629",
"\u0627\u0644\u0645\u0628\u0631\u0627\u0629",
"\u0627\u0644\u0639\u0637\u0631",
"\u0637\u0628\u0642 \u0628\u062a\u0631\u064a",
"\u0627\u0644\u0622\u0644\u0629 \u0627\u0644\u0646\u0627\u0633\u062e\u0629",
"\u0627\u0644\u0631\u064a\u0634\u0629 \u0627\u0644\u0645\u0648\u0633\u064a\u0642\u064a\u0629",
"\u062e\u0648\u0630\u0629 \u0628\u064a\u0643\u0644\u0647\u0627\u0648\u0628\u0647",
"\u0627\u0644\u0633\u064a\u0627\u062c \u0627\u0644\u0648\u062a\u062f\u064a",
"\u0627\u0644\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0646\u0635\u0641-\u0646\u0642\u0644",
"\u0627\u0644\u0631\u0635\u064a\u0641 \u0627\u0644\u0628\u062d\u0631\u064a",
"\u062d\u0635\u0627\u0644\u0629",
"\u0627\u0644\u062a\u063a\u0644\u064a\u0641 \u0627\u0644\u0635\u064a\u062f\u0644\u0627\u0646\u064a",
"\u0648\u0633\u0627\u062f\u0629",
"\u0643\u0631\u0629 \u0627\u0644\u0637\u0627\u0648\u0644\u0629",
"\u0644\u0639\u0628\u0629 \u0637\u0627\u062d\u0648\u0646\u0629 \u0647\u0648\u0627\u0621",
"\u0633\u0641\u064a\u0646\u0629 \u0627\u0644\u0642\u0631\u0627\u0635\u0646\u0629",
"\u0627\u0644\u0625\u0628\u0631\u064a\u0642",
"\u0627\u0644\u0645\u0650\u0633\u0652\u062d\u064e\u062c",
"\u0627\u0644\u0642\u0628\u0629 \u0627\u0644\u0641\u0644\u0643\u064a\u0629",
"\u0643\u064a\u0633 \u0646\u0627\u064a\u0644\u0648\u0646",
"\u0631\u0641 \u062a\u0646\u0634\u064a\u0641 \u0627\u0644\u0623\u0637\u0628\u0627\u0642",
"\u0627\u0644\u0645\u062d\u0627\u0631\u064a\u062b \u0627\u0644\u062d\u0641\u0627\u0631\u0629",
"\u0627\u0644\u0645\u0643\u0628\u0633 \u0627\u0644\u063a\u0637\u064e\u0651\u0627\u0633",
"\u0627\u0644\u0643\u0627\u0645\u064a\u0631\u0627 \u0627\u0644\u0641\u0648\u0631\u064a\u0629",
"\u0627\u0644\u0639\u0645\u0648\u062f",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u0634\u0631\u0637\u0629",
"\u0644\u0628\u0627\u0633 \u0627\u0644\u0628\u0646\u0634",
"\u0637\u0627\u0648\u0644\u0629 \u0627\u0644\u0628\u0644\u064a\u0627\u0631\u062f\u0648",
"\u0639\u0628\u0648\u0629 \u0627\u0644\u0645\u0634\u0631\u0648\u0628\u0627\u062a",
"\u0627\u0644\u0623\u0635\u064a\u0635",
"\u0639\u062c\u0644\u0629 \u0641\u062e\u0627\u0631",
"\u0627\u0644\u0645\u0650\u062b\u0652\u0642\u064e\u0628",
"\u0633\u062c\u0627\u062f\u0629 \u0627\u0644\u0635\u0644\u0627\u0629",
"\u0627\u0644\u0637\u0627\u0628\u0639\u0629 \u0627\u0644\u062d\u0627\u0633\u0648\u0628\u064a\u0629",
"\u0627\u0644\u0633\u062c\u0646",
"\u0627\u0644\u0642\u0630\u064a\u0641\u0629",
"\u0627\u0644\u0628\u0631\u0648\u062c\u0643\u062a\u0631",
"\u0642\u0631\u0635 \u0627\u0644\u0647\u0648\u0643\u064a",
"\u0627\u0644\u0645\u0644\u0643\u0645\u0629",
"\u062d\u0642\u064a\u0628\u0629 \u064a\u062f",
"\u0627\u0644\u0631\u064a\u0634\u0629 ",
"\u0627\u0644\u0644\u0650\u062d\u064e\u0627\u0641",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0633\u0628\u0627\u0642",
"\u0645\u0636\u0631\u0628 \u0627\u0644\u062a\u0646\u0633",
"\u0645\u0634\u0639\u0627\u0639",
"\u0627\u0644\u0645\u0630\u064a\u0627\u0639",
"\u0627\u0644\u0645\u0642\u0631\u0627\u0628 \u0627\u0644\u0627\u0630\u0627\u0639\u064a",
"\u062e\u0632\u0627\u0646 \u0645\u064a\u0627\u0647 \u0627\u0644\u0623\u0645\u0637\u0627\u0631",
"\u0627\u0644\u0645\u0631\u0643\u0628\u0627\u062a \u0627\u0644\u062a\u0631\u0641\u064a\u0647\u064a\u0629",
"\u0628\u0643\u0631\u0629 \u0635\u064a\u062f",
"\u0627\u0644\u0643\u0627\u0645\u064a\u0631\u0627 \u0627\u0644\u0627\u0646\u0639\u0643\u0627\u0633\u064a\u0629",
"\u0627\u0644\u062b\u0644\u0627\u064e\u0651\u062c\u0629",
"\u062c\u0647\u0627\u0632 \u062a\u062d\u0643\u0645 \u0639\u0646 \u0628\u0639\u062f",
"\u0645\u0637\u0639\u0645",
"\u0627\u0644\u0645\u0633\u062f\u0633 \u0627\u0644\u062f\u0648\u0627\u0631",
"\u0628\u0646\u062f\u0642\u064a\u0629",
"\u0627\u0644\u0643\u0631\u0633\u064a \u0627\u0644\u0647\u0632\u0627\u0632",
"\u0627\u0644\u0645\u0634\u0648\u0627\u0629",
"\u0627\u0644\u0645\u0645\u062d\u0627\u0629",
"\u0643\u0631\u0629 \u0627\u0644\u0631\u063a\u0628\u064a",
"\u0627\u0644\u0645\u0633\u0637\u0631\u0629",
"\u0627\u0644\u062d\u0630\u0627\u0621 \u0627\u0644\u0631\u064a\u0627\u0636\u064a",
"\u0627\u0644\u062e\u0632\u0627\u0646\u0629",
"\u062f\u0628\u0648\u0633 \u0645\u0634\u0628\u0643",
"\u0639\u0644\u0628 \u0627\u0644\u0645\u0644\u062d \u0648\u0627\u0644\u0641\u0644\u0641\u0644",
"\u0627\u0644\u0635\u0646\u062f\u0644",
"\u0627\u0644\u0633\u0627\u0631\u0648\u0646\u062c",
"\u0633\u0627\u0643\u0633\u0641\u0648\u0646 \u0627\u0644\u0629 \u0646\u0641\u062e \u0645\u0648\u0633\u064a\u0642\u064a\u0629",
"\u0627\u0644\u063a\u0650\u0645\u0652\u062f \u0623\u0648 \u063a\u0650\u0645\u0652\u062f \u0627\u0644\u0633\u064a\u0641 \u0623\u0648 \u063a\u0650\u0645\u0652\u062f \u0627\u0644\u062e\u0646\u062c\u0631",
"\u0627\u0644\u0645\u064a\u0632\u0627\u0646",
"\u062d\u0627\u0641\u0644\u0629 \u0645\u062f\u0631\u0633\u064a\u0629",
"\u0627\u0644\u0645\u0631\u0643\u0628 \u0627\u0644\u0634\u0631\u0627\u0639\u064a ",
"\u0644\u0648\u062d\u0629 \u0627\u0644\u0646\u062a\u0627\u0626\u062c",
"\u0634\u0627\u0634\u0629 \u0627\u0644\u0633\u064a \u0623\u0631 \u062a\u064a",
"\u0628\u0631\u063a\u064a",
"\u0645\u0641\u0643 \u0627\u0644\u0628\u0631\u0627\u063a\u064a",
"\u062d\u0632\u0627\u0645 \u0627\u0644\u0623\u0645\u0627\u0646",
"\u0622\u0644\u0629 \u0627\u0644\u062e\u064a\u0627\u0637\u0629",
"\u0627\u0644\u062f\u0631\u0639",
"\u0645\u062a\u062c\u0631 \u0627\u0644\u0623\u062d\u0630\u064a\u0629",
"\u0645\u0642\u0633\u0645 \u0627\u0644\u063a\u0631\u0641\u0629",
"\u0633\u0644\u0629 \u0627\u0644\u062a\u0633\u0648\u0642",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062a\u0633\u0648\u0642",
"\u0627\u0644\u0645\u0650\u062c\u0652\u0631\u064e\u0641\u064e\u0629",
"\u0642\u0628\u0639\u0629 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u0633\u062a\u0627\u0626\u0631 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u062a\u0632\u062d\u0644\u0642 \u0639\u0644\u0649 \u0627\u0644\u062b\u0644\u062c",
"\u0642\u0646\u0627\u0639 \u0627\u0644\u062a\u0632\u0644\u062c",
"\u0643\u064a\u0633 \u0627\u0644\u0646\u0648\u0645",
"\u0627\u0644\u0645\u0633\u0637\u0631\u0629 \u0627\u0644\u062d\u0627\u0633\u0628\u0629",
"\u0627\u0644\u0628\u0627\u0628 \u0627\u0644\u0645\u0646\u0632\u0644\u0642",
"\u0645\u0627\u0643\u064a\u0646\u0629 \u0627\u0644\u062d\u0638",
"\u0627\u0644\u063a\u0637\u0633 \u062a\u062d\u062a \u0627\u0644\u0645\u0627\u0621",
"\u0639\u0631\u0628\u0629 \u0627\u0644\u062c\u0644\u064a\u062f \u0627\u0644\u0622\u0644\u064a\u0629",
"\u0643\u0627\u0633\u062d\u0629 \u062b\u0644\u0648\u062c",
"\u0645\u0648\u0632\u0639 \u0627\u0644\u0635\u0627\u0628\u0648\u0646",
"\u0643\u0631\u0629 (\u0643\u0631\u0629 \u0627\u0644\u0642\u062f\u0645)",
"\u062c\u0648\u0631\u0628",
"\u0645\u062c\u0645\u0639 \u0627\u0644\u0637\u0627\u0642\u0629 \u0627\u0644\u0634\u0645\u0633\u064a\u0629 \u0627\u0644\u062d\u0631\u0627\u0631\u064a\u0629",
"\u0627\u0644\u0635\u064e\u0645\u0652\u0628\u0631\u0650\u064a\u0631\u0629",
"\u0648\u0639\u0627\u0621 \u0627\u0644\u0634\u0648\u0631\u0628\u0629",
"\u0645\u0641\u062a\u0627\u062d \u0627\u0644\u0645\u0633\u0627\u0641\u0629",
"\u0627\u0644\u0645\u062f\u0641\u0623\u0629",
"\u0645\u0643\u0648\u0643 \u0627\u0644\u0641\u0636\u0627\u0621",
"\u0627\u0644\u0645\u0650\u0644\u0648\u064e\u0642",
"\u0627\u0644\u0642\u0627\u0631\u0628 \u0627\u0644\u0633\u0631\u064a\u0639",
"\u0634\u0628\u0643\u0629 \u0627\u0644\u0639\u0646\u0643\u0628\u0648\u062a",
"\u062e\u0634\u0628\u0629 \u0627\u0644\u0645\u063a\u0632\u0644",
"\u0627\u0644\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u0631\u064a\u0627\u0636\u064a\u0629",
"\u0628\u0642\u0639\u0629 \u0636\u0648\u0621",
"\u0633\u0637\u062d \u0627\u0644\u0645\u0633\u0631\u062d",
"\u0627\u0644\u0642\u0627\u0637\u0631\u0629 \u0627\u0644\u0628\u062e\u0627\u0631\u064a\u0629",
"\u062c\u0633\u0631 \u0645\u0642\u0648\u0633 \u0646\u0641\u0642\u064a",
"\u0627\u0644\u0637\u0628\u0644 \u0627\u0644\u0646\u062d\u0627\u0633\u064a",
"\u0627\u0644\u0633\u0645\u0627\u0639\u0629 \u0627\u0644\u0637\u0628\u064a\u0629",
"\u0627\u0644\u0644\u0650\u0641\u0627\u0639",
"\u0627\u0644\u062c\u062f\u0627\u0631 \u0627\u0644\u062c\u0627\u0641",
"\u0645\u0624\u0642\u062a",
"\u0627\u0644\u0645\u0648\u0642\u062f",
"\u0627\u0644\u063a\u0631\u0628\u0627\u0644",
"\u0627\u0644\u062a\u0631\u0627\u0645",
"\u0627\u0644\u0646\u0642\u0627\u0644\u0629",
"\u0627\u0644\u0623\u0631\u064a\u0643\u0629",
"\u0645\u0628\u0646\u0649 \u0633\u062a\u0648\u064a\u0627",
"\u0627\u0644\u063a\u0648\u0627\u0635\u0629",
"\u0627\u0644\u0628\u0630\u0644\u0629",
"\u0627\u0644\u0645\u0632\u0648\u0644\u0629",
"\u0627\u0644\u0646\u0638\u0627\u0631\u0629 \u0627\u0644\u0634\u0645\u0633\u064a\u0629",
"\u0627\u0644\u0646\u0638\u0627\u0631\u0629 \u0627\u0644\u0634\u0645\u0633\u064a\u0629",
"\u0627\u0644\u0648\u0627\u0642\u064a \u0627\u0644\u0634\u0645\u0633\u064a",
"\u0627\u0644\u062c\u0633\u0631 \u0627\u0644\u0645\u0639\u0644\u0642",
"\u0627\u0644\u0645\u0645\u0633\u062d\u0629",
"\u0627\u0644\u0642\u0645\u064a\u0635 \u0627\u0644\u062b\u0642\u064a\u0644",
"\u0627\u0644\u062a\u064f\u0628\u0651\u0627\u0646 \u0623\u0648 \u0627\u0644\u0628\u0646\u0637\u0627\u0644 \u0627\u0644\u0642\u0635\u064a\u0631\\",
"\u0627\u0644\u0623\u0631\u062c\u0648\u062d\u0629",
"\u0627\u0644\u0645\u0641\u062a\u0627\u062d \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0645\u062d\u0642\u0646\u0629 \u0623\u0648 \u0627\u0644\u0625\u0628\u0631\u0629",
"\u0627\u0644\u0623\u0628\u0627\u062c\u0648\u0631\u0629",
"\u0627\u0644\u062f\u0628\u0627\u0628\u0629",
"\u0645\u0633\u062c\u0644 \u0627\u0644\u0634\u0631\u064a\u0637 \u0627\u0644\u0635\u0648\u062a\u064a",
"\u0625\u0628\u0631\u064a\u0642 \u0627\u0644\u0634\u0627\u064a",
"\u0627\u0644\u062f\u0628\u062f\u0648\u0628",
"\u0627\u0644\u0631\u0627\u0626\u064a",
"\u0643\u0631\u0629 \u062a\u0646\u0633",
"\u0627\u0644\u062a\u0633\u0642\u064a\u0641 \u0628\u0627\u0644\u0642\u0634",
"\u0627\u0644\u0633\u062a\u0627\u0631\u0629 \u0627\u0644\u0645\u0633\u0631\u062d\u064a\u0629",
"\u0627\u0644\u0643\u064f\u0634\u0652\u062a\u0650\u0628\u064e\u0627\u0646",
"\u0627\u0644\u062f\u0631\u064e\u0651\u0627\u0633\u0629",
"\u0639\u0631\u0634",
"\u0628\u0644\u0627\u0637 \u0627\u0644\u0633\u0642\u0641",
"\u0622\u0644\u0629 \u062a\u062d\u0645\u064a\u0635 \u0627\u0644\u062e\u0628\u0632",
"\u0645\u062d\u0644\u0627\u062a \u0628\u064a\u0639 \u0644\u0648\u0627\u0632\u0645 \u0627\u0644\u062a\u062f\u062e\u064a\u0646",
"\u0645\u0642\u0639\u062f \u0627\u0644\u0645\u0631\u062d\u0627\u0636",
"\u0645\u0634\u0639\u0644\u0629",
"\u0627\u0644\u0623\u064e\u0639\u0652\u0645\u0650\u062f\u064e\u0629\u064f \u0627\u0644\u0637\u064e\u0651\u0648\u0652\u0637\u064e\u0645\u0650\u064a\u064e\u0651\u0629\u0650",
"\u0634\u0627\u062d\u0646\u0629 \u0627\u0644\u0642\u0637\u0631",
"\u0645\u062a\u062c\u0631 \u0627\u0644\u0623\u0644\u0639\u0627\u0628",
"\u0633\u064a\u0627\u0631\u0629 \u0627\u0644\u062c\u0631\u0627\u0631",
"\u0634\u0627\u062d\u0646\u0629 \u0646\u0635\u0641 \u0645\u0642\u0637\u0648\u0631\u0629",
"\u0627\u0644\u0635\u064a\u0646\u064a\u0629",
"\u0645\u0639\u0637\u0641 \u0627\u0644\u062e\u0646\u062f\u0642",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u062b\u0644\u0627\u062b\u064a\u0629 \u0627\u0644\u0639\u062c\u0644\u0627\u062a",
"\u0642\u0627\u0631\u0628 \u0627\u0644\u062f\u0639\u0627\u0645\u0629 \u0627\u0644\u0645\u0632\u062f\u0648\u062c\u0629",
"\u062d\u0627\u0645\u0644 \u062b\u0644\u0627\u062b\u064a",
"\u0642\u0648\u0633 \u0627\u0644\u0646\u0635\u0631",
"\u0627\u0644\u062d\u0627\u0641\u0644\u0629 \u0633\u0637\u062d\u064a\u0629 \u0627\u0644\u062a\u0645\u062f\u064a\u062f \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a",
"\u0627\u0644\u062a\u0631\u0648\u0645\u0628\u0648\u0646",
"\u062d\u0648\u0636 \u0627\u0644\u0627\u0633\u062a\u062d\u0645\u0627\u0645",
"\u0627\u0644\u0628\u0648\u0627\u0628\u0629 \u0627\u0644\u062f\u0648\u0627\u0631\u0629",
"\u0627\u0644\u0622\u0644\u0629 \u0627\u0644\u0643\u0627\u062a\u0628\u0629",
"\u0627\u0644\u0645\u0638\u0644\u0629",
"\u0627\u0644\u062f\u0631\u0627\u062c\u0629 \u0627\u0644\u0623\u062d\u0627\u062f\u064a\u0629",
"\u0627\u0644\u0628\u064a\u0627\u0646\u0648 \u0627\u0644\u0642\u0627\u0626\u0645",
"\u0627\u0644\u0645\u0643\u0646\u0633\u0629 \u0627\u0644\u0643\u0647\u0631\u0628\u0627\u0626\u064a\u0629",
"\u0627\u0644\u0645\u0632\u0647\u0631\u064a\u0629",
"\u0627\u0644\u0642\u0646\u0637\u0631\u0629",
"\u0627\u0644\u0642\u0645\u0627\u0634 \u0627\u0644\u0645\u062e\u0645\u0644\u064a",
"\u0622\u0644\u0629 \u0627\u0644\u0628\u064a\u0639 \u0627\u0644\u0630\u0627\u062a\u064a",
"\u0627\u0644\u0635\u062f\u0627\u0631\u064a",
"\u0642\u0646\u0637\u0631\u0629 \u0645\u062a\u0639\u062f\u062f\u0629 \u0627\u0644\u0631\u0643\u0627\u0626\u0632",
"\u0627\u0644\u0643\u0645\u0627\u0646",
"\u0627\u0644\u0643\u0631\u0629 \u0627\u0644\u0637\u0627\u0626\u0631\u0629",
"\u0635\u0627\u0646\u0639\u0629 \u0627\u0644\u0648\u0627\u0641\u0644",
"\u0633\u0627\u0639\u0629 \u0627\u0644\u062d\u0627\u0626\u0637",
"\u0645\u062d\u0641\u0638\u0629",
"\u062e\u0632\u0627\u0646\u0629 \u0627\u0644\u0635\u0648\u0627\u0646",
"\u0637\u0627\u0626\u0631\u0629 \u0639\u0633\u0643\u0631\u064a\u0629",
"\u0627\u0644\u0645\u062c\u0644\u0649",
"\u0627\u0644\u063a\u0633\u0627\u0644\u0629",
"\u0642\u0627\u0631\u0648\u0631\u0629 \u0645\u0627\u0621",
"\u0625\u0628\u0631\u064a\u0642 \u0627\u0644\u0645\u0627\u0621",
"\u0628\u0631\u062c \u0627\u0644\u0645\u064a\u0627\u0647",
"\u0625\u0628\u0631\u064a\u0642 \u0627\u0644\u0643\u062d\u0648\u0644\u064a\u0627\u062a",
"\u0627\u0644\u0635\u0627\u0641\u0631\u0629",
"\u0634\u0639\u0631 \u0645\u0633\u062a\u0639\u0627\u0631",
"\u0627\u0644\u0646\u0627\u0641\u0630\u0629 \u0627\u0644\u0648\u0627\u0642\u064a\u0629",
"\u0627\u0644\u0633\u062a\u0627\u0631 \u0627\u0644\u0644\u0641\u064e\u0651\u0627\u0641",
"\u0631\u0628\u0637\u0629 \u0639\u0646\u0642 \u0648\u0646\u062f\u0633\u0648\u0631 ",
"\u0632\u062c\u0627\u062c\u0629 \u0627\u0644\u0646\u0628\u064a\u0630",
"\u062c\u0646\u0627\u062d \u0627\u0644\u0637\u0627\u0626\u0631\u0629",
"\u0645\u0642\u0644\u0627\u0629 \u0635\u064a\u0646\u064a\u0629",
"\u0627\u0644\u0645\u0644\u0639\u0642\u0629 \u0627\u0644\u062e\u0634\u0628\u064a\u0629",
"\u0627\u0644\u0635\u0648\u0641",
"\u0627\u0644\u0633\u064a\u0627\u062c \u0627\u0644\u0645\u0646\u0642\u0633\u0645",
"\u062d\u0637\u0627\u0645 \u0627\u0644\u0633\u0641\u064a\u0646\u0629",
"\u0627\u0644\u0632\u0648\u0631\u0642 \u0627\u0644\u0634\u0631\u0627\u0639\u064a",
"\u0645\u0646\u0632\u0644 \u0627\u0644\u064a\u0648\u0631\u062a",
"\u0645\u0648\u0627\u0642\u0639 \u0627\u0644\u0648\u064a\u0628",
"\u0643\u062a\u0627\u0628 \u0631\u0633\u0648\u0645 \u0647\u0632\u0644\u064a\u0629",
"\u0627\u0644\u0643\u0644\u0645\u0627\u062a \u0627\u0644\u0645\u062a\u0642\u0627\u0637\u0639\u0629",
"\u0644\u0627\u0641\u062a\u0629 \u0645\u0631\u0648\u0631\u064a\u0629",
"\u0625\u0634\u0627\u0631\u0629 \u0627\u0644\u0645\u0631\u0648\u0631 \u0627\u0644\u0636\u0648\u0626\u064a\u0629",
"\u063a\u0644\u0627\u0641 \u0627\u0644\u062d\u0645\u0627\u064a\u0629 \u0644\u0644\u0643\u062a\u0627\u0628",
"\u0642\u0627\u0626\u0645\u0629 \u0637\u0639\u0627\u0645",
"\u0635\u062d\u0646",
"\u0642\u0646\u0628\u064a\u0637 \u0623\u062e\u0636\u0631",
"\u062d\u0633\u0627\u0621 \u0627\u0644\u0643\u0648\u0646\u0633\u0648\u0645\u064a\u0629",
"\u0627\u0644\u0648\u0639\u0627\u0621 \u0627\u0644\u0633\u0627\u062e\u0646",
"\u062a\u0631\u0627\u064a\u0641\u0644",
"\u0627\u0644\u0645\u062b\u0644\u062c\u0627\u062a",
"\u0627\u0644\u0645\u0635\u0627\u0635\u0629",
"\u0627\u0644\u062e\u0628\u0632 \u0627\u0644\u0641\u0631\u0646\u0633\u064a",
"\u062e\u0628\u0632 \u0627\u0644\u0628\u064a\u063a\u0644",
"\u0627\u0644\u0645\u062e\u0628\u0648\u0632\u0627\u062a \u0627\u0644\u0639\u064f\u0642\u0652\u062f\u0650\u064a\u064e\u0651\u0629",
"\u062a\u0634\u064a\u0632 \u0628\u0631\u062c\u0631",
"\u0627\u0644\u0646\u0642\u0627\u0646\u0642",
"\u0627\u0644\u0628\u0637\u0627\u0637\u0627 \u0627\u0644\u0645\u0647\u0631\u0648\u0633\u0629",
"\u0645\u0644\u0641\u0648\u0641",
"\u0627\u0644\u0642\u0631\u0646\u0628\u064a\u0637 \u0627\u0644\u0623\u062e\u0636\u0631",
"\u0627\u0644\u0642\u0631\u0646\u0628\u064a\u0637",
"\u0627\u0644\u0643\u0648\u0633\u0627",
"\u0645\u0639\u0643\u0631\u0648\u0646\u0629 \u0627\u0644\u0627\u0633\u0643\u0648\u0627\u0634",
"\u0642\u0631\u0639 \u0627\u0644\u0628\u0644\u0648\u0637",
"\u0642\u0631\u0639 \u0627\u0644\u062c\u0648\u0632",
"\u062e\u064a\u0627\u0631",
"\u0627\u0644\u062e\u0631\u0634\u0648\u0641 \u0627\u0644\u0634\u0648\u0643\u064a",
"\u0641\u0644\u0641\u0644 \u062d\u0644\u0648",
"\u0627\u0644\u062e\u0631\u0634\u0648\u0641 \u0627\u0644\u0633\u0643\u0648\u0644\u064a\u0645\u064a",
"\u0639\u064a\u0634 \u0627\u0644\u063a\u0631\u0627\u0628",
"\u062a\u0641\u0627\u062d \u0623\u062e\u0636\u0631",
"\u0627\u0644\u0641\u0631\u0627\u0648\u0644\u0629",
"\u0627\u0644\u0628\u0631\u062a\u0642\u0627\u0644",
"\u0627\u0644\u0644\u064a\u0645\u0648\u0646",
"\u0627\u0644\u062a\u064a\u0646",
"\u0627\u0644\u0623\u0646\u0627\u0646\u0627\u0633",
"\u0627\u0644\u0645\u0648\u0632",
"\u062c\u0627\u0643 \u0641\u0631\u0648\u062a",
"\u0642\u0634\u0637\u0629 \u0634\u0631\u064a\u0645\u0648\u0644\u064a\u0627",
"\u0627\u0644\u0631\u0645\u0627\u0646",
"\u0627\u0644\u062f\u0631\u064a\u0633",
"\u0643\u0631\u0628\u0646\u0627\u0631\u0629",
"\u0635\u0644\u0635\u0629 \u0627\u0644\u0634\u0648\u0643\u0648\u0644\u0627",
"\u0639\u062c\u064a\u0646\u0629 \u0627\u0644\u062e\u0628\u0632",
"\u0631\u063a\u064a\u0641 \u0627\u0644\u0644\u062d\u0645",
"\u0628\u064a\u062a\u0632\u0627",
"\u0641\u0637\u064a\u0631\u0629 \u0627\u0644\u0642\u062f\u0631",
"\u0628\u0648\u0631\u064a\u062a\u0648",
"\u0627\u0644\u0646\u0628\u064a\u0630 \u0627\u0644\u0623\u062d\u0645\u0631",
"\u0642\u0647\u0648\u0629 \u0625\u0633\u0628\u0631\u064a\u0633\u0648",
"\u0641\u0646\u062c\u0627\u0646 \u0627\u0644\u0634\u0627\u064a",
"\u062d\u0644\u064a\u0628 \u0627\u0644\u0628\u064a\u0636",
"\u0627\u0644\u062c\u0628\u0644",
"\u0641\u0642\u0627\u0639\u0629",
"\u0627\u0644\u062c\u0631\u0641",
"\u0627\u0644\u0634\u0639\u0627\u0628 \u0627\u0644\u0645\u0631\u062c\u0627\u0646\u064a\u0629",
"\u0627\u0644\u0641\u0648\u0627\u0631\u0629 \u0627\u0644\u062d\u0627\u0631\u0629",
"\u0627\u0644\u0636\u0641\u0629",
"\u0627\u0644\u0634\u0646\u062e\u0629",
"\u0627\u0644\u0645\u064a\u0627\u0647 \u0627\u0644\u0636\u062d\u0644\u0629",
"\u0627\u0644\u0634\u0627\u0637\u0626",
"\u0627\u0644\u0648\u0627\u062f\u064a",
"\u0628\u0631\u0643\u0627\u0646",
"\u0644\u0627\u0639\u0628 \u0643\u0631\u0629 \u0627\u0644\u0642\u0627\u0639\u062f\u0629",
"\u0627\u0644\u0639\u0631\u064a\u0633",
"\u0627\u0644\u063a\u0648\u0635 \u0628\u062c\u0647\u0627\u0632 \u0627\u0644\u062a\u0646\u0641\u0633 ",
"\u0627\u0644\u0633\u0644\u062c\u0645",
"\u0632\u0647\u0631\u0629 \u0627\u0644\u0644\u0624\u0644\u0624 ",
"\u062e\u0641 \u0627\u0644\u0633\u064a\u062f\u0629 \u0627\u0644\u0623\u0635\u0641\u0631",
"\u0627\u0644\u0630\u0631\u0629",
"\u0634\u062c\u0631\u0629 \u062b\u0645\u0631\u0629 \u0627\u0644\u0628\u0644\u0648\u0637",
"\u062b\u0645\u0631 \u0627\u0644\u0648\u0631\u062f \u0627\u0644\u0628\u0631\u064a",
"\u0628\u0630\u0648\u0631 \u0643\u0633\u062a\u0646\u0627\u0621 \u0627\u0644\u062d\u0635\u0627\u0646",
"\u0627\u0644\u0641\u0637\u0631\u064a\u0627\u062a \u0627\u0644\u0645\u0631\u062c\u0627\u0646\u064a\u0629",
"\u0641\u0637\u0631 \u063a\u0627\u0631\u064a\u0642\u0648\u0646",
"\u0641\u0637\u0631 \u062c\u0627\u0631\u0648\u0645\u064a\u062a\u0631\u0627 \u0627\u064a\u0633\u0643\u0644\u0646\u062a\u0627",
"\u0627\u0644\u0642\u0631\u0646 \u0627\u0644\u0646\u062a\u0646",
"\u0641\u0637\u0631 \u0646\u062c\u0645 \u0627\u0644\u0623\u0631\u0636",
"\u0641\u0637\u0631 \u0631\u0641 \u0627\u0644\u0643\u0628\u0631\u064a\u062a",
"\u0641\u0637\u0631 \u0627\u0644\u0628\u0648\u0644\u064a\u0637",
"\u0627\u0644\u0639\u0631\u0646\u0627\u0633",
"\u0648\u0631\u0642 \u0627\u0644\u0645\u0631\u062d\u0627\u0636"
]
}
clip_benchmark/clip_benchmark/datasets/ar_zeroshot_classification_templates.json 0 → 100644
View file @ 26e59280
{
"imagenet1k": [
"{c}",
"\u0635\u0648\u0631\u0629 \u0633\u064a\u0626\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0633\u064a\u0626\u0629 \u062a\u062d\u062a\u0648\u064a \u0639\u0644\u0649 {c}",
"\u0646\u062d\u062a \u0644\u0634\u0643\u0644 {c}",
"\u0646\u062d\u062a \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0630\u0627\u062a \u062c\u0648\u0648\u062f\u0629 \u0645\u0646\u062e\u0641\u0636\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0630\u0627\u062a \u062c\u0648\u0648\u062f\u0629 \u0645\u0646\u062e\u0641\u0636\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0631\u0633\u0648\u0645\u0627\u062a \u062c\u062f\u0627\u0631\u064a\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0631\u0633\u0648\u0645\u0627\u062a \u062c\u062f\u0627\u0631\u064a\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0642\u062a\u0637\u0639\u0629 \u062a\u062d\u062a\u0648\u064a \u0639\u0644\u0649 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0642\u062a\u0637\u0639\u0629 \u0644\u0640 {c}",
"\u062a\u0637\u0631\u064a\u0632 {c} ",
" \u0635\u0648\u0631\u0629 \u064a\u0635\u0639\u0628 \u0641\u064a\u0647\u0627 \u0631\u0624\u064a\u0629 {c} ",
"\u0635\u0648\u0631\u0629 \u0633\u0627\u0637\u0639\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0648\u0627\u0636\u062d\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u062a\u0633\u062e\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0638\u0644\u0645\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0623\u0628\u064a\u0636 \u0648\u0623\u0633\u0648\u062f {c}",
"{c} \u0641\u064a \u0644\u0642\u0637\u0629 \u0642\u0631\u064a\u0628\u0629",
"\u0635\u0648\u0631\u0629 \u0631\u0627\u0626\u0639\u0629 \u0644\u0640 {c}",
"\u0644\u0642\u0637\u0629 \u0642\u0631\u064a\u0628\u0629 \u0644\u0640 {c}",
"\u0631\u0633\u0645 \u062d\u0627\u0633\u0648\u0628\u064a \u064a\u062d\u062a\u0648\u064a {c}",
"\u0635\u0648\u0631\u0629 \u0645\u0631\u0633\u0648\u0645\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0631\u0633\u0645\u0629 \u0644\u0640 {c}",
"\u0631\u0633\u0645\u0629 {c}",
"\u0631\u0633\u0645 \u064a\u062d\u062a\u0648\u064a {c} ",
"\u0635\u0648\u0631\u0629 \u0628\u0646\u0645\u0637 \u0627\u0644\u0628\u0643\u0633\u0644 \u0644\u0640 {c}",
" \u0635\u0648\u0631\u0629 \u0633\u0627\u0637\u0639\u0629 {c}",
"\u0648\u0634\u0645 {c}",
"{c} \u0641\u064a \u0627\u0644\u0635\u0648\u0631\u0629",
"\u0635\u0648\u0631\u0629 \u0645\u062a\u0633\u062e\u0629 \u062a\u062d\u062a\u0648\u064a {c}",
"\u0635\u0648\u0631\u0629 \u062a\u0627\u0644\u0641\u0629 {c}",
"\u0635\u0648\u0631\u0629 \u0636\u0628\u0627\u0628\u064a\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 {c}",
"\u0635\u0648\u0631\u0629 \u062c\u064a\u062f\u0629 \u0644\u0640 {c}",
"\u0635\u0648\u0631\u0629 \u0644\u0640 {c}",
"\u062a\u0635\u064a\u064a\u0631 \u0644\u0640 {c}",
"{c} \u0639\u0644\u0649 \u0634\u0643\u0644 \u0631\u0633\u0645 \u062d\u0627\u0633\u0648\u0628\u064a \u062b\u0646\u0627\u0626\u064a \u0623\u0648 \u062b\u0644\u0627\u062b\u064a \u0627\u0644\u0623\u0628\u0639\u0627\u062f",
"\u064a\u0648\u062c\u062f {c} \u0648\u0627\u062d\u062f \u0641\u064a \u0627\u0644\u0635\u0648\u0631\u0629",
"\u0631\u0633\u0645 \u062d\u0627\u0633\u0648\u0628\u064a \u0644\u0640 {c}",
"\u0627\u0648\u0631\u064a\u063a\u0627\u0645\u064a \u0644\u0640 {c}",
"{c} \u0645\u0635\u0646\u0648\u0639 \u0639\u0646 \u0637\u0631\u064a\u0642 \u0641\u0646 \u0637\u064a \u0627\u0644\u0648\u0631\u0642",
"{c} \u0641\u064a \u0644\u0639\u0628\u0629 \u0641\u064a\u062f\u064a\u0648",
"{c} \u0645\u0648\u062c\u0648\u062f \u0641\u064a \u0644\u0639\u0628\u0629 \u0627\u0644\u0641\u064a\u062f\u064a\u0648",
"\u0631\u0633\u0645 \u062a\u0642\u0631\u064a\u0628\u064a \u0644\u0640 {c}",
"{c} \u0645\u0631\u0633\u0648\u0645 \u0628\u0627\u0644\u062e\u0631\u0627\u0628\u064a\u0634",
"\u0635\u0648\u0631\u0629 \u0628\u0641\u0646 \u0627\u0644\u062e\u0631\u0627\u0628\u064a\u0634 \u0644\u0640 {c}",
"\u0644\u0639\u0628\u0629 {c}",
"\u0635\u0648\u0631\u0629 \u064a\u0648\u062c\u062f \u0641\u064a\u0647\u0627 {c}",
"\u0631\u0633\u0648\u0645 \u0645\u062a\u062d\u0631\u0643\u0629 \u0644\u0640 {c} ",
"\u0635\u0648\u0631\u0629 \u0644\u0639\u062f\u062f \u0645\u0646 {c}",
"\u0635\u0648\u0631\u0629 \u064a\u0638\u0647\u0631 \u0641\u064a\u0647\u0627 {c}",
"\u0635\u0648\u0631\u0629 {c} \u0635\u063a\u064a\u0631 ",
"\u0635\u0648\u0631\u0629 {c} \u0643\u0628\u064a\u0631",
"{c} \u064a\u0638\u0647\u0631 \u0641\u064a \u0627\u0644\u0635\u0648\u0631\u0629"
]
}
clip_benchmark/clip_benchmark/datasets/birdsnap.py 0 → 100644
View file @ 26e59280
import concurrent.futures
import csv
import hashlib
import os
from pathlib import Path
import torch
from PIL import Image as Image
from torchvision.datasets.utils import download_and_extract_archive
Image.LOAD_TRUNCATED_IMAGES = True
class Birdsnap(torch.utils.data.Dataset):
"""This is the BirdSnap dataset presented in
- Berg et al., "Birdsnap: Large-scale Fine-grained Visual Categorization of Birds"
It contains a lot of classes of birds and can be used as a replacement for ImageNet validation images
with similar image fidelity but less of the baggage, given that all subjects are in fact birds.
This is too small to train on though and hence not even partitioned into train/test.
Several images are missing from flickr (in 2021), these will be discarded automatically.
"""
METADATA_URL = 'http://thomasberg.org/datasets/birdsnap/1.1/birdsnap.tgz'
METADATA_ARCHIVE = 'birdsnap.tgz'
META_MD5 = '1788158175f6ae794aebf27bcd7a3f5d'
BASE_FOLDER = 'birdsnap'
def __init__(self, root, split='train', transform=None, target_transform=None, download=True, crop_to_bbx=False):
"""Init with split, transform, target_transform."""
self.root = os.path.expanduser(root)
self.transform = transform
self.target_transform = target_transform
self.crop_to_bbx = crop_to_bbx # Crop to dataset default bounding boxes
if download:
self.download()
if not self.check_integrity():
raise ValueError('Dataset Birdsnap not downloaded completely or possibly corrupted.')
# self._purge_missing_data()
def _check_integrity_of_metadata(self, chunk_size=8192):
"""This only checks if all files are there."""
try:
with open(os.path.join(self.root, self.METADATA_ARCHIVE), 'rb') as f:
archive_hash = hashlib.md5()
while chunk := f.read(chunk_size):
archive_hash.update(chunk)
return self.META_MD5 == archive_hash.hexdigest()
except FileNotFoundError:
return False
def check_integrity(self):
"""Full integrity check."""
if not self._check_integrity_of_metadata():
return False
else:
self._parse_metadata()
missing_images = 0
for idx, file in enumerate(self.meta):
if not self._verify_image(idx):
missing_images += 1
if missing_images > 0:
print(f'{missing_images} images could not be downloaded.')
return True
def download(self):
# Metadata:
if self._check_integrity_of_metadata():
print('Metadata already downloaded and verified')
else:
download_and_extract_archive(self.METADATA_URL, self.root, filename=self.METADATA_ARCHIVE)
# Actual files:
self._parse_metadata()
missing_ids = []
for idx, file in enumerate(self.meta):
if not self._verify_image(idx):
missing_ids.append(idx)
if len(missing_ids) > 0:
print(f'Downloading {len(missing_ids)} missing files now...')
self.scrape_images(missing_ids)
def __len__(self):
"""Return length via metainfo."""
return len(self.meta)
def __getitem__(self, index):
"""Return image, label."""
img = Image.open(self.paths[index])
if self.crop_to_bbx:
img = img.crop(
(
self.meta[index]['bb_x1'],
self.meta[index]['bb_y1'],
self.meta[index]['bb_x2'],
self.meta[index]['bb_y2'],
)
)
img = img.convert('RGB')
label = self.labels[index]
img = self.transform(img) if self.transform else img
label = self.target_transform(label) if self.target_transform else label
return img, label
def _parse_metadata(self):
"""Metadata keys are
dict_keys(['url', 'md5', 'path', 'species_id', 'bb_x1', 'bb_y1', 'bb_x2', 'bb_y2', 'back_x', 'back_y', 'beak_x',
'beak_y', 'belly_x', 'belly_y', 'breast_x', 'breast_y', 'crown_x', 'crown_y', 'forehead_x', 'forehead_y',
'left_cheek_x', 'left_cheek_y', 'left_eye_x', 'left_eye_y', 'left_leg_x', 'left_leg_y', 'left_wing_x',
'left_wing_y', 'nape_x', 'nape_y', 'right_cheek_x', 'right_cheek_y', 'right_eye_x', 'right_eye_y',
'right_leg_x', 'right_leg_y', 'right_wing_x', 'right_wing_y', 'tail_x', 'tail_y', 'throat_x', 'throat_y']
"""
with open(os.path.join(self.root, self.BASE_FOLDER, 'images.txt'), 'r') as f:
reader = csv.DictReader(f, delimiter='\t')
self.meta = list(reader) # List of dictionaries.
self.labels = [int(entry['species_id']) for entry in self.meta]
self.paths = [os.path.join(self.root, self.BASE_FOLDER, entry['path']) for entry in self.meta]
with open(os.path.join(self.root, self.BASE_FOLDER, 'species.txt'), 'r') as f:
reader = csv.DictReader(f, delimiter='\t')
self.classes_metadata = list(reader)
self.classes = [str(entry['common']) for entry in self.classes_metadata]
def _verify_image(self, idx):
try:
# Do this if you want to check in detail:
with open(os.path.join(self.root, self.BASE_FOLDER, self.meta[idx]['path']), 'rb') as fin:
return (hashlib.md5(fin.read()).hexdigest() == self.meta[idx]['md5'])
# In the mean time, just check if everything is there:
# return os.path.exists(os.path.join(self.root, self.BASE_FOLDER, self.meta[idx]["path"]))
except FileNotFoundError:
return False
def scrape_images(self, missing_ids, chunk_size=8196):
"""Scrape images using the python default ThreadPool example."""
import requests
def _load_url_and_save_image(idx, timeout):
full_path = os.path.join(self.root, self.BASE_FOLDER, self.meta[idx]['path'])
os.makedirs(os.path.split(full_path)[0], exist_ok=True)
r = requests.get(self.meta[idx]['url'], stream=True)
with open(full_path, 'wb') as write_file:
for chunk in r.iter_content(chunk_size=chunk_size):
write_file.write(chunk)
return True
# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=None) as executor: # Choose max_workers dynamically
# Start the load operations and mark each future with its URL
future_to_url = {
executor.submit(_load_url_and_save_image, idx, 600): self.meta[idx]['url'] for idx in missing_ids
}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
data = future.result()
except Exception as exc:
print(f'{url} generated exception: {exc}')
else:
print(f'{url} downloaded successfully.')
def _purge_missing_data(self):
"""Iterate over all data and throw out missing images."""
JPG = b'\xff\xd8\xff'
clean_meta = []
invalid_files = 0
for entry in self.meta:
full_path = os.path.join(self.root, self.BASE_FOLDER, entry['path'])
with open(full_path, 'rb') as file_handle:
if file_handle.read(3) == JPG:
clean_meta.append(entry)
else:
invalid_files += 1
print(f'Discarded {invalid_files} invalid files.')
self.meta = clean_meta
self.labels = [int(entry['species_id']) for entry in self.meta]
self.paths = [os.path.join(self.root, self.BASE_FOLDER, entry['path']) for entry in self.meta]
class BirdsnapV2(torch.utils.data.Dataset):
def __init__(self, root, split='test', transform=None, target_transform=None):
assert split == 'test', print('Only test split is supported for now.')
self.root_dir = Path(root)
self.image_dir = self.root_dir
self.test_elements = self.root_dir / 'test_images_valid.txt'
self.classes_file = self.root_dir / 'species.txt'
self.target_transform = target_transform
self.transform = transform
with open(self.test_elements, 'r') as f:
self.imgs = [line.rstrip('\n') for line in f][1:]
with open(self.classes_file, 'r') as f:
self.classes = [line.rstrip('\n').split('\t')[1] for line in f][1:]
with open(self.classes_file, 'r') as f:
self.dirs = [line.rstrip('\n').split('\t')[-1] for line in f][1:]
dir2id = {dir: idx for idx, dir in enumerate(self.dirs)}
self.labels = []
for image in self.imgs:
dir = image.split('/')[0].lower()
label = dir2id[dir]
self.labels.append(label)
self.imgs = [os.path.join(self.root_dir, item) for item in self.imgs]
def __len__(self):
"""Return length via metainfo."""
return len(self.imgs)
def __getitem__(self, index):
"""Return image, label."""
img = Image.open(self.imgs[index])
img = img.convert('RGB')
label = self.labels[index]
img = self.transform(img) if self.transform else img
label = self.target_transform(label) if self.target_transform else label
return img, label
if __name__ == '__main__':
from tqdm import tqdm
dataset = BirdsnapV2(root='/mnt/petrelfs/wangwenhai/workspace/InternVL2/benchmark/imagenet/birdsnap', split='test')
print(len(dataset.imgs))
for i in tqdm(range(len(dataset))):
try:
dataset.__getitem__(i)
print(dataset.imgs[i])
except:
print(dataset.imgs[i])
print(dataset.classes)
clip_benchmark/clip_benchmark/datasets/builder.py 0 → 100644
View file @ 26e59280
import json
import os
import sys
import warnings
from subprocess import call
import torch
from torch.utils.data import default_collate
from torchvision.datasets import (CIFAR10, CIFAR100, DTD, GTSRB, MNIST, PCAM,
STL10, SUN397, CocoCaptions, Country211,
EuroSAT, FGVCAircraft, Flowers102, Food101,
ImageFolder, ImageNet, OxfordIIITPet,
RenderedSST2, StanfordCars)
from . import caltech101, flickr, imagenetv2, objectnet, voc2007
from .birdsnap import BirdsnapV2
from .tools import pre_caption
def _load_classnames_and_classification_templates(dataset_name, current_folder, language):
with open(os.path.join(current_folder, language + '_classnames.json'), 'r') as f:
classnames = json.load(f)
# Zero-shot classification templates, collected from a bunch of sources
# - CLIP paper (https://github.com/openai/CLIP/blob/main/data/prompts.md)
# - Lit Paper (https://arxiv.org/pdf/2111.07991.pdf)
# - SLIP paper (https://github.com/facebookresearch/SLIP/blob/main/templates.json)
# Some are fixed mnaually
with open(os.path.join(current_folder, language + '_zeroshot_classification_templates.json'), 'r') as f:
zeroshot_classification_templates = json.load(f)
# default template to use when the dataset name does not belong to `zeroshot_classification_templates`
DEFAULT_ZEROSHOT_CLASSIFICATION_TEMPLATES = zeroshot_classification_templates['imagenet1k']
if dataset_name.startswith('tfds/') or dataset_name.startswith('vtab/') or dataset_name.startswith('wds/'):
name = dataset_name.split('/')[-1]
else:
name = dataset_name
templates = zeroshot_classification_templates.get(name, DEFAULT_ZEROSHOT_CLASSIFICATION_TEMPLATES)
return classnames, templates
def build_dataset(dataset_name, root='root', transform=None, split='test', download=True, annotation_file=None,
language='en', task='zeroshot_classification', cupl=False, wds_cache_dir=None, **kwargs):
"""
Main function to use in order to build a dataset instance,
dataset_name: str
name of the dataset
root: str
root folder where the dataset is downloaded and stored. can be shared among datasets.
transform: torchvision transform applied to images
split: str
split to use, depending on the dataset can have different options.
In general, `train` and `test` are available.
For specific splits, please look at the corresponding dataset.
annotation_file: str or None
only for datasets with captions (used for retrieval) such as COCO
and Flickr.
"""
current_folder = os.path.dirname(__file__)
if task in ('zeroshot_classification', 'linear_probe'): # Only load templates and classnames if we have to
classnames, templates = _load_classnames_and_classification_templates(dataset_name, current_folder, language)
else:
classnames, templates = None, None
with open(os.path.join(current_folder, 'cupl_prompts.json'), 'r') as f:
cupl_prompts = json.load(f)
templates_cupl = None
train = (split == 'train')
if dataset_name == 'cifar10':
ds = CIFAR10(root=root, train=train, transform=transform, download=download, **kwargs)
elif dataset_name == 'cifar100':
ds = CIFAR100(root=root, train=train, transform=transform, download=download, **kwargs)
elif dataset_name == 'imagenet1k':
if not os.path.exists(root):
os.makedirs(root, exist_ok=True)
call(
f'wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_devkit_t12.tar.gz --output-document={root}/ILSVRC2012_devkit_t12.tar.gz',
shell=True)
call(
f'wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar --output-document={root}/ILSVRC2012_img_val.tar',
shell=True)
ds = ImageNet(root=root, split='train' if train else 'val', transform=transform, **kwargs)
# use classnames from OpenAI
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenet1k-unverified':
split = 'train' if train else 'val'
ds = ImageFolder(root=os.path.join(root, split), transform=transform, **kwargs)
# use classnames from OpenAI
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenetv2':
assert split == 'test', f'Only test split available for {dataset_name}'
os.makedirs(root, exist_ok=True)
ds = imagenetv2.ImageNetV2Dataset(variant='matched-frequency', transform=transform, location=root)
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenet_sketch':
assert split == 'test', f'Only test split available for {dataset_name}'
# Downloadable from https://drive.google.com/open?id=1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA
if not os.path.exists(root):
# Automatic download
print('Downloading imagenet_sketch...')
if not has_gdown():
print('GDown is needed to download the dataset. Please install it via `pip install gdown`')
sys.exit(1)
# Download ImageNet-Sketch.zip
call('gdown --id 1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA', shell=True)
assert os.path.exists('ImageNet-Sketch.zip')
# Unzip and move to `root`
call('unzip ImageNet-Sketch.zip', shell=True)
call(f'mv sketch {root}', shell=True)
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
templates_cupl = cupl_prompts['imagenet1k']
elif dataset_name == 'imagenet-a':
assert split == 'test', f'Only test split available for {dataset_name}'
# Downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar
if not os.path.exists(root):
print('Downloading imagenet-a...')
call('wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar', shell=True)
# Untar and move to `root`
call('tar xvf imagenet-a.tar', shell=True)
call(f'mv imagenet-a {root}', shell=True)
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
imagenet_a_wnids = ['n01498041', 'n01531178', 'n01534433', 'n01558993', 'n01580077', 'n01614925', 'n01616318',
'n01631663', 'n01641577', 'n01669191', 'n01677366', 'n01687978', 'n01694178', 'n01698640',
'n01735189', 'n01770081', 'n01770393', 'n01774750', 'n01784675', 'n01819313', 'n01820546',
'n01833805', 'n01843383', 'n01847000', 'n01855672', 'n01882714', 'n01910747', 'n01914609',
'n01924916', 'n01944390', 'n01985128', 'n01986214', 'n02007558', 'n02009912', 'n02037110',
'n02051845', 'n02077923', 'n02085620', 'n02099601', 'n02106550', 'n02106662', 'n02110958',
'n02119022', 'n02123394', 'n02127052', 'n02129165', 'n02133161', 'n02137549', 'n02165456',
'n02174001', 'n02177972', 'n02190166', 'n02206856', 'n02219486', 'n02226429', 'n02231487',
'n02233338', 'n02236044', 'n02259212', 'n02268443', 'n02279972', 'n02280649', 'n02281787',
'n02317335', 'n02325366', 'n02346627', 'n02356798', 'n02361337', 'n02410509', 'n02445715',
'n02454379', 'n02486410', 'n02492035', 'n02504458', 'n02655020', 'n02669723', 'n02672831',
'n02676566', 'n02690373', 'n02701002', 'n02730930', 'n02777292', 'n02782093', 'n02787622',
'n02793495', 'n02797295', 'n02802426', 'n02814860', 'n02815834', 'n02837789', 'n02879718',
'n02883205', 'n02895154', 'n02906734', 'n02948072', 'n02951358', 'n02980441', 'n02992211',
'n02999410', 'n03014705', 'n03026506', 'n03124043', 'n03125729', 'n03187595', 'n03196217',
'n03223299', 'n03250847', 'n03255030', 'n03291819', 'n03325584', 'n03355925', 'n03384352',
'n03388043', 'n03417042', 'n03443371', 'n03444034', 'n03445924', 'n03452741', 'n03483316',
'n03584829', 'n03590841', 'n03594945', 'n03617480', 'n03666591', 'n03670208', 'n03717622',
'n03720891', 'n03721384', 'n03724870', 'n03775071', 'n03788195', 'n03804744', 'n03837869',
'n03840681', 'n03854065', 'n03888257', 'n03891332', 'n03935335', 'n03982430', 'n04019541',
'n04033901', 'n04039381', 'n04067472', 'n04086273', 'n04099969', 'n04118538', 'n04131690',
'n04133789', 'n04141076', 'n04146614', 'n04147183', 'n04179913', 'n04208210', 'n04235860',
'n04252077', 'n04252225', 'n04254120', 'n04270147', 'n04275548', 'n04310018', 'n04317175',
'n04344873', 'n04347754', 'n04355338', 'n04366367', 'n04376876', 'n04389033', 'n04399382',
'n04442312', 'n04456115', 'n04482393', 'n04507155', 'n04509417', 'n04532670', 'n04540053',
'n04554684', 'n04562935', 'n04591713', 'n04606251', 'n07583066', 'n07695742', 'n07697313',
'n07697537', 'n07714990', 'n07718472', 'n07720875', 'n07734744', 'n07749582', 'n07753592',
'n07760859', 'n07768694', 'n07831146', 'n09229709', 'n09246464', 'n09472597', 'n09835506',
'n11879895', 'n12057211', 'n12144580', 'n12267677']
imagenet_a_mask = [wnid in set(imagenet_a_wnids) for wnid in all_imagenet_wordnet_ids]
ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_a_mask) if mask]
elif dataset_name == 'imagenet-r':
assert split == 'test', f'Only test split available for {dataset_name}'
# downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar
if not os.path.exists(root):
print('Downloading imagenet-r...')
call('wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar', shell=True)
# Untar and move to `root`
call('tar xvf imagenet-r.tar', shell=True)
call(f'mv imagenet-r {root}', shell=True)
imagenet_r_wnids = {'n01443537', 'n01484850', 'n01494475', 'n01498041', 'n01514859', 'n01518878', 'n01531178',
'n01534433', 'n01614925', 'n01616318', 'n01630670', 'n01632777', 'n01644373', 'n01677366',
'n01694178', 'n01748264', 'n01770393', 'n01774750', 'n01784675', 'n01806143', 'n01820546',
'n01833805', 'n01843383', 'n01847000', 'n01855672', 'n01860187', 'n01882714', 'n01910747',
'n01944390', 'n01983481', 'n01986214', 'n02007558', 'n02009912', 'n02051845', 'n02056570',
'n02066245', 'n02071294', 'n02077923', 'n02085620', 'n02086240', 'n02088094', 'n02088238',
'n02088364', 'n02088466', 'n02091032', 'n02091134', 'n02092339', 'n02094433', 'n02096585',
'n02097298', 'n02098286', 'n02099601', 'n02099712', 'n02102318', 'n02106030', 'n02106166',
'n02106550', 'n02106662', 'n02108089', 'n02108915', 'n02109525', 'n02110185', 'n02110341',
'n02110958', 'n02112018', 'n02112137', 'n02113023', 'n02113624', 'n02113799', 'n02114367',
'n02117135', 'n02119022', 'n02123045', 'n02128385', 'n02128757', 'n02129165', 'n02129604',
'n02130308', 'n02134084', 'n02138441', 'n02165456', 'n02190166', 'n02206856', 'n02219486',
'n02226429', 'n02233338', 'n02236044', 'n02268443', 'n02279972', 'n02317335', 'n02325366',
'n02346627', 'n02356798', 'n02363005', 'n02364673', 'n02391049', 'n02395406', 'n02398521',
'n02410509', 'n02423022', 'n02437616', 'n02445715', 'n02447366', 'n02480495', 'n02480855',
'n02481823', 'n02483362', 'n02486410', 'n02510455', 'n02526121', 'n02607072', 'n02655020',
'n02672831', 'n02701002', 'n02749479', 'n02769748', 'n02793495', 'n02797295', 'n02802426',
'n02808440', 'n02814860', 'n02823750', 'n02841315', 'n02843684', 'n02883205', 'n02906734',
'n02909870', 'n02939185', 'n02948072', 'n02950826', 'n02951358', 'n02966193', 'n02980441',
'n02992529', 'n03124170', 'n03272010', 'n03345487', 'n03372029', 'n03424325', 'n03452741',
'n03467068', 'n03481172', 'n03494278', 'n03495258', 'n03498962', 'n03594945', 'n03602883',
'n03630383', 'n03649909', 'n03676483', 'n03710193', 'n03773504', 'n03775071', 'n03888257',
'n03930630', 'n03947888', 'n04086273', 'n04118538', 'n04133789', 'n04141076', 'n04146614',
'n04147183', 'n04192698', 'n04254680', 'n04266014', 'n04275548', 'n04310018', 'n04325704',
'n04347754', 'n04389033', 'n04409515', 'n04465501', 'n04487394', 'n04522168', 'n04536866',
'n04552348', 'n04591713', 'n07614500', 'n07693725', 'n07695742', 'n07697313', 'n07697537',
'n07714571', 'n07714990', 'n07718472', 'n07720875', 'n07734744', 'n07742313', 'n07745940',
'n07749582', 'n07753275', 'n07753592', 'n07768694', 'n07873807', 'n07880968', 'n07920052',
'n09472597', 'n09835506', 'n10565667', 'n12267677'}
imagenet_r_mask = [wnid in imagenet_r_wnids for wnid in all_imagenet_wordnet_ids]
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_r_mask) if mask]
elif dataset_name == 'imagenet-o':
assert split == 'test', f'Only test split available for {dataset_name}'
# downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-o.tar
if not os.path.exists(root):
print('Downloading imagenet-o...')
call('wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-o.tar', shell=True)
# Untar and move to `root`
call('tar xvf imagenet-o.tar', shell=True)
call(f'mv imagenet-o {root}', shell=True)
ds = ImageFolder(root=root, transform=transform, **kwargs)
ds.classes = classnames['imagenet1k']
imagenet_o_wnids = ['n01443537', 'n01704323', 'n01770081', 'n01784675', 'n01819313', 'n01820546', 'n01910747',
'n01917289', 'n01968897', 'n02074367', 'n02317335', 'n02319095', 'n02395406', 'n02454379',
'n02606052', 'n02655020', 'n02666196', 'n02672831', 'n02730930', 'n02777292', 'n02783161',
'n02786058', 'n02787622', 'n02791270', 'n02808304', 'n02817516', 'n02841315', 'n02865351',
'n02877765', 'n02892767', 'n02906734', 'n02910353', 'n02916936', 'n02948072', 'n02965783',
'n03000134', 'n03000684', 'n03017168', 'n03026506', 'n03032252', 'n03075370', 'n03109150',
'n03126707', 'n03134739', 'n03160309', 'n03196217', 'n03207743', 'n03218198', 'n03223299',
'n03240683', 'n03271574', 'n03291819', 'n03297495', 'n03314780', 'n03325584', 'n03344393',
'n03347037', 'n03372029', 'n03376595', 'n03388043', 'n03388183', 'n03400231', 'n03445777',
'n03457902', 'n03467068', 'n03482405', 'n03483316', 'n03494278', 'n03530642', 'n03544143',
'n03584829', 'n03590841', 'n03598930', 'n03602883', 'n03649909', 'n03661043', 'n03666591',
'n03676483', 'n03692522', 'n03706229', 'n03717622', 'n03720891', 'n03721384', 'n03724870',
'n03729826', 'n03733131', 'n03733281', 'n03742115', 'n03786901', 'n03788365', 'n03794056',
'n03804744', 'n03814639', 'n03814906', 'n03825788', 'n03840681', 'n03843555', 'n03854065',
'n03857828', 'n03868863', 'n03874293', 'n03884397', 'n03891251', 'n03908714', 'n03920288',
'n03929660', 'n03930313', 'n03937543', 'n03942813', 'n03944341', 'n03961711', 'n03970156',
'n03982430', 'n03991062', 'n03995372', 'n03998194', 'n04005630', 'n04023962', 'n04033901',
'n04040759', 'n04067472', 'n04074963', 'n04116512', 'n04118776', 'n04125021', 'n04127249',
'n04131690', 'n04141975', 'n04153751', 'n04154565', 'n04201297', 'n04204347', 'n04209133',
'n04209239', 'n04228054', 'n04235860', 'n04243546', 'n04252077', 'n04254120', 'n04258138',
'n04265275', 'n04270147', 'n04275548', 'n04330267', 'n04332243', 'n04336792', 'n04347754',
'n04371430', 'n04371774', 'n04372370', 'n04376876', 'n04409515', 'n04417672', 'n04418357',
'n04423845', 'n04429376', 'n04435653', 'n04442312', 'n04482393', 'n04501370', 'n04507155',
'n04525305', 'n04542943', 'n04554684', 'n04557648', 'n04562935', 'n04579432', 'n04591157',
'n04597913', 'n04599235', 'n06785654', 'n06874185', 'n07615774', 'n07693725', 'n07695742',
'n07697537', 'n07711569', 'n07714990', 'n07715103', 'n07716358', 'n07717410', 'n07718472',
'n07720875', 'n07742313', 'n07745940', 'n07747607', 'n07749582', 'n07753275', 'n07753592',
'n07754684', 'n07768694', 'n07836838', 'n07871810', 'n07873807', 'n07880968', 'n09229709',
'n09472597', 'n12144580', 'n12267677', 'n13052670']
imagenet_o_mask = [wnid in set(imagenet_o_wnids) for wnid in all_imagenet_wordnet_ids]
ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_o_mask) if mask]
elif dataset_name == 'objectnet':
assert split == 'test', f'Only test split available for {dataset_name}'
# downloadable from https://objectnet.dev/downloads/objectnet-1.0.zip or https://www.dropbox.com/s/raw/cxeztdtm16nzvuw/objectnet-1.0.zip
if not os.path.exists(root):
print('Downloading objectnet...')
call('wget https://objectnet.dev/downloads/objectnet-1.0.zip', shell=True)
# Untar and move to `root`
call('UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE unzip -P objectnetisatestset objectnet-1.0.zip', shell=True)
os.makedirs(root)
call(f'mv objectnet-1.0 {root}', shell=True)
call(f'cp {root}/objectnet-1.0/mappings/* {root}', shell=True)
ds = objectnet.ObjectNetDataset(root=root, transform=transform)
elif dataset_name == 'voc2007':
ds = voc2007.PASCALVoc2007Cropped(root=root, set='train' if train else 'test', transform=transform,
download=download, **kwargs)
elif dataset_name == 'voc2007_multilabel':
ds = voc2007.PASCALVoc2007(root=root, set='train' if train else 'test', transform=transform, download=download,
**kwargs)
elif dataset_name == 'mscoco_captions':
# https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations)
if split == 'train':
archive_name = 'train2014.zip'
elif split in ('val', 'test'):
archive_name = 'val2014.zip'
else:
raise ValueError(f'split should be train or val or test for `{dataset_name}`')
root_split = os.path.join(root, archive_name.replace('.zip', ''))
if not os.path.exists(root_split):
print(f'Downloading mscoco_captions {archive_name}...')
if not os.path.exists(os.path.join(root, archive_name)):
call(f'wget http://images.cocodataset.org/zips/{archive_name} --output-document={root}/{archive_name}',
shell=True)
call(f'unzip {root}/{archive_name} -d {root}', shell=True)
if not annotation_file:
annotation_file = f'{root}/coco_{split}_karpathy.json'
if language == 'cn':
annotation_file = f'{root}/coco-cn_{split}.json'
root_split = root
print(annotation_file)
if not os.path.exists(annotation_file):
call(
f'wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/coco_{split}_karpathy.json --output-document={annotation_file}',
shell=True)
ds = CocoCaptions(root=root_split, annFile=annotation_file, transform=transform,
target_transform=pre_caption, **kwargs)
elif dataset_name == 'multilingual_mscoco_captions':
from clip_benchmark.datasets import multilingual_mscoco
if (language not in multilingual_mscoco.SUPPORTED_LANGUAGES):
raise ValueError('Unsupported language for multilingual_ms_coco:', language)
def get_archive_name(target_split):
if target_split == 'train':
return 'train2014.zip'
elif target_split in ('val', 'test'):
return 'val2014.zip'
else:
raise ValueError(f'split should be train or val or test for `{dataset_name}`')
def download_mscoco_split(target_split):
archive_name = get_archive_name(target_split)
root_split = os.path.join(root, archive_name.replace('.zip', ''))
if not os.path.exists(root_split):
print(f'Downloading mscoco_captions {archive_name}...')
if not os.path.exists(os.path.join(root, archive_name)):
call(
f'wget http://images.cocodataset.org/zips/{archive_name} --output-document={root}/{archive_name}',
shell=True)
call(f'unzip {root}/{archive_name} -d {root}', shell=True)
# The multilingual MS-COCO uses images from various splits
for target_split in ['train', 'val', 'test']:
download_mscoco_split(target_split)
annotation_file = os.path.join(root, multilingual_mscoco.CAPTIONS_FILE_NAME.format(language))
# if (os.path.exists(annotation_file) == False):
multilingual_mscoco.create_annotation_file(root, language)
ds = multilingual_mscoco.Multilingual_MSCOCO(root=root, ann_file=annotation_file, transform=transform, **kwargs)
elif dataset_name == 'flickr30k':
# downloadable from https://www.kaggle.com/datasets/adityajn105/flickr30k
# https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations)
# `kaggle datasets download -d adityajn105/flickr30k`
if not os.path.exists(root):
# Automatic download
print('Downloading flickr30k...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
call('kaggle datasets download -d adityajn105/flickr30k', shell=True)
call(f'unzip flickr30k.zip', shell=True)
call(f'mv Images {root}', shell=True)
call(f'mv captions.txt {root}', shell=True)
if not annotation_file:
annotation_file = f'{root}/flickr30k_{split}_karpathy.txt'
if not os.path.exists(annotation_file):
# Download Flickr30K Karpathy test set
annotation_file = f'{root}/flickr30k_{split}_karpathy.txt'
call(
f'wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr30k_{split}_karpathy.txt --output-document={annotation_file}',
shell=True)
if language == 'cn':
annotation_file = f'{root}/flickr30k_cn_{split}.txt'
print(annotation_file)
ds = flickr.Flickr(root=f'{root}/Images', ann_file=annotation_file, transform=transform,
target_transform=pre_caption, **kwargs)
elif dataset_name == 'flickr8k':
# downloadable from https://www.kaggle.com/datasets/adityajn105/flickr8k
# `kaggle datasets download -d adityajn105/flickr8k`
# https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations)
if not os.path.exists(root):
# Automatic download
print('Downloading flickr8k...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
call('kaggle datasets download -d adityajn105/flickr8k', shell=True)
call(f'unzip flickr8k.zip', shell=True)
call(f'mv Images {root}', shell=True)
call(f'mv captions.txt {root}', shell=True)
if not annotation_file:
annotation_file = f'{root}/flickr8k_{split}_karpathy.txt'
if not os.path.exists(annotation_file):
# Download Flickr8K Karpathy test set
annotation_file = f'{root}/flickr8k_{split}_karpathy.txt'
call(
f'wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr8k_{split}_karpathy.txt --output-document={annotation_file}',
shell=True)
ds = flickr.Flickr(root=root, ann_file=annotation_file, transform=transform, **kwargs)
elif dataset_name == 'food101':
ds = Food101(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
# we use the default class names, we just replace "_" by spaces
# to delimit words
ds.classes = [cl.replace('_', ' ') for cl in ds.classes]
elif dataset_name == 'sun397':
warnings.warn(
f'split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset')
# we use the default class names, we just replace "_" and "/" by spaces
# to delimit words
ds = SUN397(root=root, transform=transform, download=download, **kwargs)
ds.classes = [cl.replace('_', ' ').replace('/', ' ') for cl in ds.classes]
elif dataset_name == 'cars':
ds = StanfordCars(root=root, split='train' if train else 'test', transform=transform, download=download,
**kwargs)
elif dataset_name == 'fgvc_aircraft':
ds = FGVCAircraft(root=root, annotation_level='variant', split='train' if train else 'test',
transform=transform, download=download, **kwargs)
elif dataset_name == 'dtd':
ds = DTD(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
elif dataset_name == 'pets':
ds = OxfordIIITPet(root=root, split='train' if train else 'test', target_types='category', transform=transform,
download=download, **kwargs)
elif dataset_name == 'caltech101':
warnings.warn(
f'split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset')
# broken download link (can't download google drive), fixed by this PR https://github.com/pytorch/vision/pull/5645
# also available in "vtab/caltech101" using VTAB splits, we advice to use VTAB version rather than this one
# since in this one (torchvision) there are no pre-defined test splits
ds = caltech101.Caltech101(root=root, target_type='category', transform=transform, download=download, **kwargs)
ds.classes = classnames['caltech101']
elif dataset_name == 'flowers':
ds = Flowers102(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
# class indices started by 1 until it was fixed in a PR (#TODO link of the PR)
# if older torchvision version, fix it using a target transform that decrements label index
# TODO figure out minimal torchvision version needed instead of decrementing
if ds[0][1] == 1:
ds.target_transform = lambda y: y - 1
ds.classes = classnames['flowers']
elif dataset_name == 'birdsnap':
ds = BirdsnapV2(root=root, split='train' if train else 'test', transform=transform, **kwargs)
# ds.classes = ds.classes
elif dataset_name == 'mnist':
ds = MNIST(root=root, train=train, transform=transform, download=download, **kwargs)
ds.classes = classnames['mnist']
elif dataset_name == 'stl10':
ds = STL10(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
elif dataset_name == 'eurosat':
warnings.warn(
f'split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset')
ds = EuroSAT(root=root, transform=transform, download=download, **kwargs)
ds.classes = classnames['eurosat']
elif dataset_name == 'gtsrb':
ds = GTSRB(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
ds.classes = classnames['gtsrb']
elif dataset_name == 'country211':
ds = Country211(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
ds.classes = classnames['country211']
elif dataset_name == 'pcam':
# Dead link. Fixed by this PR on torchvision https://github.com/pytorch/vision/pull/5645
# TODO figure out minimal torchvision version needed
ds = PCAM(root=root, split='train' if train else 'test', transform=transform, download=download, **kwargs)
ds.classes = classnames['pcam']
elif dataset_name == 'renderedsst2':
ds = RenderedSST2(root=root, split='train' if train else 'test', transform=transform, download=download,
**kwargs)
elif dataset_name == 'fer2013':
# Downloadable from https://www.kaggle.com/datasets/msambare/fer2013
# `kaggle datasets download -d msambare/fer2013`
if not os.path.exists(root):
# Automatic download
print('Downloading fer2013...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
call('kaggle datasets download -d msambare/fer2013', shell=True)
call(f'unzip fer2013.zip -d {root}', shell=True)
root = os.path.join(root, 'train' if train else 'test')
ds = ImageFolder(root=root, transform=transform)
ds.classes = classnames['fer2013']
elif dataset_name.startswith('tfds/'):
# TFDS datasets support using `timm` and `tensorflow_datasets`
prefix, *name_list = dataset_name.split('/')
name = '/'.join(name_list)
ds = build_tfds_dataset(name, download=download, split=split, data_dir=root, transform=transform)
elif dataset_name.startswith('vtab/'):
# VTAB datasets support using `tensorflow_datasets` and `task_adaptation`
prefix, *name_list = dataset_name.split('/')
name = '/'.join(name_list)
ds = build_vtab_dataset(name, download=download, split=split, data_dir=root, transform=transform,
classnames=classnames)
elif dataset_name.startswith('wds/'):
# WebDataset support using `webdataset` library
name = dataset_name.split('/', 1)[1]
ds = build_wds_dataset(name, transform=transform, split=split, data_dir=root, cache_dir=wds_cache_dir)
return ds
elif dataset_name == 'dummy':
ds = Dummy()
else:
raise ValueError(f'Unsupported dataset: {dataset_name}.')
if cupl:
ds.templates = templates_cupl
else:
ds.templates = templates
return ds
class Dummy():
def __init__(self):
self.classes = ['blank image', 'noisy image']
def __getitem__(self, i):
return torch.zeros(3, 224, 224), 0
def __len__(self):
return 1
def get_dataset_default_task(dataset):
if dataset in ('flickr30k', 'flickr8k', 'mscoco_captions', 'multilingual_mscoco_captions'):
return 'zeroshot_retrieval'
else:
return 'zeroshot_classification'
def get_dataset_collate_fn(dataset_name):
if dataset_name in ('mscoco_captions', 'multilingual_mscoco_captions', 'flickr30k', 'flickr8k'):
return image_captions_collate_fn
else:
return default_collate
def has_gdown():
return call('which gdown', shell=True) == 0
def has_kaggle():
return call('which kaggle', shell=True) == 0
def build_vtab_dataset(dataset_name, transform, download=True, split='test', data_dir='root', classnames=[]):
# Using VTAB splits instead of default TFDS splits
from .tfds import (VTABIterableDataset, disable_gpus_on_tensorflow,
download_tfds_dataset)
# avoid Tensorflow owning GPUs to not clash with PyTorch
disable_gpus_on_tensorflow()
# by default we take classes from TFDS (default behavior if `classes` stays None),
# except for the datasets that will override `classes` (e.g., clevr_*)
classes = None
if dataset_name == 'caltech101':
from task_adaptation.data.caltech import Caltech101
tfds_dataset = Caltech101(data_dir=data_dir)
classes = classnames['caltech101_vtab']
elif dataset_name == 'cars':
from task_adaptation.data.cars import CarsData
tfds_dataset = CarsData(data_dir=data_dir)
elif dataset_name in ('cifar10', 'cifar100'):
from task_adaptation.data.cifar import CifarData
tfds_dataset = CifarData(data_dir=data_dir, num_classes=10 if dataset_name == 'cifar10' else 100)
elif dataset_name.startswith('clevr_'):
from task_adaptation.data.clevr import CLEVRData
task = _extract_task(dataset_name)
assert task in ('count_all', 'closest_object_distance')
tfds_dataset = CLEVRData(task=task, data_dir=data_dir)
if task == 'count_all':
classes = classnames['clevr_count_all']
elif task == 'closest_object_distance':
classes = classnames['clevr_closest_object_distance']
else:
raise ValueError(f'non supported: {task}')
elif dataset_name == 'cub':
from task_adaptation.data.cub import CUB2011Data
tfds_dataset = CUB2011Data(data_dir=data_dir)
elif dataset_name == 'diabetic_retinopathy':
# Needs manual download from Kaggle
# 1) `kaggle competitions download -c diabetic-retinopathy-detection` on $ROOT/downloads/manual
# 2) extract archives on $ROOT/downloads/manual
if not os.path.exists(data_dir):
# Automatic download
print('Downloading diabetic_retinopathy...')
if not has_kaggle():
print('Kaggle is needed to download the dataset. Please install it via `pip install kaggle`')
sys.exit(1)
os.makedirs(os.path.join(data_dir, 'downloads', 'manual'))
call(f'kaggle competitions download -c diabetic-retinopathy-detection -p {data_dir}/downloads/manual',
shell=True)
call(
f'cd {data_dir}/downloads/manual;unzip diabetic-retinopathy-detection.zip;cat train.zip*>train.zip;cat test.zip*>test.zip;unzip train.zip; unzip test.zip;unzip sample.zip;unzip trainLabels.csv.zip',
shell=True)
from task_adaptation.data.diabetic_retinopathy import RetinopathyData
tfds_dataset = RetinopathyData(config='btgraham-300', data_dir=data_dir)
classes = classnames['diabetic_retinopathy']
elif dataset_name == 'dmlab':
from task_adaptation.data.dmlab import DmlabData
download_tfds_dataset('dmlab',
data_dir=data_dir) # it's not called in the original VTAB code, so we do it explictly
tfds_dataset = DmlabData(data_dir=data_dir)
classes = classnames['dmlab']
elif dataset_name.startswith('dsprites_'):
from task_adaptation.data.dsprites import DSpritesData
task = _extract_task(dataset_name)
assert task in ('label_shape', 'label_scale', 'label_orientation', 'label_x_position', 'label_y_position')
tfds_dataset = DSpritesData(task, data_dir=data_dir)
classes = tfds_dataset._dataset_builder.info.features[task].names
elif dataset_name == 'dtd':
from task_adaptation.data.dtd import DTDData
tfds_dataset = DTDData(data_dir=data_dir)
elif dataset_name == 'eurosat':
from task_adaptation.data.eurosat import EurosatData
tfds_dataset = EurosatData(subset='rgb', data_key='image', data_dir=data_dir)
classes = classnames['eurosat']
elif dataset_name == 'food101':
from task_adaptation.data.food101 import Food101Data
tfds_dataset = Food101Data(data_dir=data_dir)
elif dataset_name == 'inaturalist':
from task_adaptation.data.inaturalist import INaturalistData
tfds_dataset = INaturalistData(data_dir=data_dir, year=2017)
elif dataset_name.startswith('kitti_'):
from .kitti import KittiData
task = _extract_task(dataset_name)
assert task in (
'count_all', 'count_left', 'count_far', 'count_near',
'closest_object_distance', 'closest_object_x_location',
'count_vehicles', 'closest_vehicle_distance',
)
tfds_dataset = KittiData(task=task, data_dir=data_dir)
if task == 'closest_vehicle_distance':
classes = classnames['kitti_closest_vehicle_distance']
else:
raise ValueError(f'Unsupported task: {task}')
elif dataset_name == 'flowers':
from task_adaptation.data.oxford_flowers102 import OxfordFlowers102Data
tfds_dataset = OxfordFlowers102Data(data_dir=data_dir)
elif dataset_name == 'pets':
from task_adaptation.data.oxford_iiit_pet import OxfordIIITPetData
tfds_dataset = OxfordIIITPetData(data_dir=data_dir)
classes = classnames['pets']
elif dataset_name == 'pcam':
from task_adaptation.data.patch_camelyon import PatchCamelyonData
tfds_dataset = PatchCamelyonData(data_dir=data_dir)
classes = classnames['pcam']
elif dataset_name == 'resisc45':
# Needs download from OneDrive: https://1drv.ms/u/s!AmgKYzARBl5ca3HNaHIlzp_IXjs
# The archive needs to to be put at <DATASET_ROOT>/downloads/manual then extracted
# if not os.path.exists(data_dir):
# os.makedirs(os.path.join(data_dir, "downloads", "manual"))
# call(f"wget 'https://onedrive.live.com/download?resid=5C5E061130630A68!107&authkey=!AHHNaHIlzp_IXjs' --output-document={data_dir}/downloads/manual/resisc45.rar", shell=True)
# call(f"cd {data_dir}/downloads/manual;unrar x resisc45.rar", shell=True)
from task_adaptation.data.resisc45 import Resisc45Data
tfds_dataset = Resisc45Data(data_dir=data_dir)
elif dataset_name.startswith('smallnorb_'):
from task_adaptation.data.smallnorb import SmallNORBData
task = _extract_task(dataset_name)
assert task in ('label_category', 'label_elevation', 'label_azimuth', 'label_lighting')
tfds_dataset = SmallNORBData(predicted_attribute=task, data_dir=data_dir)
classes = tfds_dataset._dataset_builder.info.features[task].names
elif dataset_name == 'sun397':
from task_adaptation.data.sun397 import Sun397Data
# FIXME There is a problem in `sun397`, when TFDS tries download it
# there is an image that cannot be decoded. For the time being
# we will use torchvision's SUN397 instead.
tfds_dataset = Sun397Data(config='tfds', data_dir=data_dir)
elif dataset_name == 'svhn':
from task_adaptation.data.svhn import SvhnData
tfds_dataset = SvhnData(data_dir=data_dir)
classes = classnames['svhn']
else:
raise ValueError(f'Unsupported dataset: {dataset_name}')
ds = VTABIterableDataset(
tfds_dataset,
input_name='image', label_name='label',
transform=transform,
target_transform=int,
split=split,
classes=classes,
)
return ds
def build_tfds_dataset(name, transform, download=True, split='test', data_dir='root', classes=None):
from .tfds import disable_gpus_on_tensorflow
disable_gpus_on_tensorflow()
import tensorflow_datasets as tfds
import timm
builder = tfds.builder(name, data_dir=data_dir)
if download:
builder.download_and_prepare()
splits = list(builder.info.splits.keys())
assert split in splits, (split, splits)
ds = timm.data.create_dataset(f'tfds/{name}', data_dir, split=split, transform=transform, target_transform=int)
ds.classes = builder.info.features['label'].names if classes is None else classes
return ds
def build_wds_dataset(dataset_name, transform, split='test', data_dir='root', cache_dir=None):
"""
Load a dataset in WebDataset format. Either local paths or HTTP URLs can be specified.
Expected file structure is:
```
data_dir/
train/
nshards.txt
0.tar
1.tar
...
test/
nshards.txt
0.tar
1.tar
...
classnames.txt
zeroshot_classification_templates.txt
dataset_type.txt
```
Classnames and templates are required for zeroshot classification, while dataset type
(equal to "retrieval") is required for zeroshot retrieval datasets.
You can use the `clip_benchmark_export_wds` or corresponding API
(`clip_benchmark.webdataset_builder.convert_dataset`) to convert datasets to this format.
Set `cache_dir` to a path to cache the dataset, otherwise, no caching will occur.
"""
import webdataset as wds
def read_txt(fname):
if '://' in fname:
stream = os.popen("curl -L -s --fail '%s'" % fname, 'r')
value = stream.read()
if stream.close():
raise FileNotFoundError('Failed to retreive data')
else:
with open(fname, 'r') as file:
value = file.read()
return value
# Special handling for Huggingface datasets
# Git LFS files have a different file path to access the raw data than other files
if data_dir.startswith('https://huggingface.co/datasets'):
# Format: https://huggingface.co/datasets/<USERNAME>/<REPO>/tree/<BRANCH>
*split_url_head, _, url_path = data_dir.split('/', 7)
url_head = '/'.join(split_url_head)
metadata_dir = '/'.join([url_head, 'raw', url_path])
tardata_dir = '/'.join([url_head, 'resolve', url_path])
else:
metadata_dir = tardata_dir = data_dir
# Get number of shards
nshards_fname = os.path.join(metadata_dir, split, 'nshards.txt')
nshards = int(read_txt(nshards_fname)) # Do not catch FileNotFound, nshards.txt should be mandatory
# Get dataset type (classification or retrieval)
type_fname = os.path.join(metadata_dir, 'dataset_type.txt')
try:
dataset_type = read_txt(type_fname).strip().lower()
except FileNotFoundError:
# print("WARNING: dataset_type.txt not found, assuming type=classification")
dataset_type = 'classification'
#
filepattern = os.path.join(tardata_dir, split, '{0..%d}.tar' % (nshards - 1))
# Load webdataset (support WEBP, PNG, and JPG for now)
if not cache_dir or not isinstance(cache_dir, str):
cache_dir = None
dataset = wds.WebDataset(filepattern, cache_dir=cache_dir).decode(
wds.autodecode.ImageHandler('pil', extensions=['webp', 'png', 'jpg', 'jpeg']))
# Load based on classification or retrieval task
if dataset_type == 'retrieval':
dataset = (dataset
.to_tuple(['webp', 'png', 'jpg', 'jpeg'], 'txt')
.map_tuple(transform, str.splitlines)
)
dataset.classes = dataset.templates = None
else:
label_type = 'npy' if dataset_type == 'multilabel' else 'cls' # Special case for multilabel
dataset = (dataset
.to_tuple(['webp', 'png', 'jpg', 'jpeg'], label_type)
.map_tuple(transform, None)
)
# Get class names if present
classnames_fname = os.path.join(metadata_dir, 'classnames.txt')
try:
dataset.classes = [line.strip() for line in read_txt(classnames_fname).splitlines() if line.strip()]
except FileNotFoundError:
print('WARNING: classnames.txt not found')
dataset.classes = None
# Get zeroshot classification templates if present
templates_fname = os.path.join(metadata_dir, 'zeroshot_classification_templates.txt')
try:
dataset.templates = [line.strip() for line in read_txt(templates_fname).splitlines() if line.strip()]
except FileNotFoundError:
print('WARNING: zeroshot_classification_templates.txt not found')
dataset.templates = None
return dataset
def _extract_task(dataset_name):
prefix, *task_name_list = dataset_name.split('_')
task = '_'.join(task_name_list)
return task
def image_captions_collate_fn(batch):
transposed = list(zip(*batch))
imgs = default_collate(transposed[0])
texts = transposed[1]
return imgs, texts
def get_dataset_collection_from_file(path):
return [l.strip() for l in open(path).readlines()]
dataset_collection = {
'vtab': [
'vtab/caltech101',
'vtab/cifar100',
'vtab/clevr_count_all',
'vtab/clevr_closest_object_distance',
'vtab/diabetic_retinopathy',
'vtab/dmlab',
'vtab/dsprites_label_orientation',
'vtab/dsprites_label_x_position',
'vtab/dtd',
'vtab/eurosat',
'vtab/kitti_closest_vehicle_distance',
'vtab/flowers',
'vtab/pets',
'vtab/pcam',
'vtab/resisc45',
'vtab/smallnorb_label_azimuth',
'vtab/smallnorb_label_elevation',
'sun397',
'vtab/svhn',
],
'vtab+': [
'imagenet1k',
'imagenetv2',
'imagenet_sketch',
'imagenet-a',
'imagenet-r',
'objectnet',
'fer2013',
'voc2007',
'voc2007_multilabel',
'sun397',
'cars',
'fgvc_aircraft',
'mnist',
'stl10',
'gtsrb',
'country211',
'renderedsst2',
'vtab/caltech101',
'vtab/cifar10',
'vtab/cifar100',
'vtab/clevr_count_all',
'vtab/clevr_closest_object_distance',
'vtab/diabetic_retinopathy',
'vtab/dmlab',
'vtab/dsprites_label_orientation',
'vtab/dsprites_label_x_position',
'vtab/dtd',
'vtab/eurosat',
'vtab/kitti_closest_vehicle_distance',
'vtab/flowers',
'vtab/pets',
'vtab/pcam',
'vtab/resisc45',
'vtab/smallnorb_label_azimuth',
'vtab/smallnorb_label_elevation',
'vtab/svhn',
],
'retrieval': [
'mscoco_captions',
'flickr8k',
'flickr30k',
],
'imagenet_robustness': [
'imagenetv2',
'imagenet_sketch',
'imagenet-a',
'imagenet-r',
'objectnet',
],
}
# use by imagenet robustness datasets
all_imagenet_wordnet_ids = ['n01440764', 'n01443537', 'n01484850', 'n01491361', 'n01494475', 'n01496331', 'n01498041',
'n01514668', 'n01514859', 'n01518878', 'n01530575', 'n01531178', 'n01532829', 'n01534433',
'n01537544', 'n01558993', 'n01560419', 'n01580077', 'n01582220', 'n01592084', 'n01601694',
'n01608432', 'n01614925', 'n01616318', 'n01622779', 'n01629819', 'n01630670', 'n01631663',
'n01632458', 'n01632777', 'n01641577', 'n01644373', 'n01644900', 'n01664065', 'n01665541',
'n01667114', 'n01667778', 'n01669191', 'n01675722', 'n01677366', 'n01682714', 'n01685808',
'n01687978', 'n01688243', 'n01689811', 'n01692333', 'n01693334', 'n01694178', 'n01695060',
'n01697457', 'n01698640', 'n01704323', 'n01728572', 'n01728920', 'n01729322', 'n01729977',
'n01734418', 'n01735189', 'n01737021', 'n01739381', 'n01740131', 'n01742172', 'n01744401',
'n01748264', 'n01749939', 'n01751748', 'n01753488', 'n01755581', 'n01756291', 'n01768244',
'n01770081', 'n01770393', 'n01773157', 'n01773549', 'n01773797', 'n01774384', 'n01774750',
'n01775062', 'n01776313', 'n01784675', 'n01795545', 'n01796340', 'n01797886', 'n01798484',
'n01806143', 'n01806567', 'n01807496', 'n01817953', 'n01818515', 'n01819313', 'n01820546',
'n01824575', 'n01828970', 'n01829413', 'n01833805', 'n01843065', 'n01843383', 'n01847000',
'n01855032', 'n01855672', 'n01860187', 'n01871265', 'n01872401', 'n01873310', 'n01877812',
'n01882714', 'n01883070', 'n01910747', 'n01914609', 'n01917289', 'n01924916', 'n01930112',
'n01943899', 'n01944390', 'n01945685', 'n01950731', 'n01955084', 'n01968897', 'n01978287',
'n01978455', 'n01980166', 'n01981276', 'n01983481', 'n01984695', 'n01985128', 'n01986214',
'n01990800', 'n02002556', 'n02002724', 'n02006656', 'n02007558', 'n02009229', 'n02009912',
'n02011460', 'n02012849', 'n02013706', 'n02017213', 'n02018207', 'n02018795', 'n02025239',
'n02027492', 'n02028035', 'n02033041', 'n02037110', 'n02051845', 'n02056570', 'n02058221',
'n02066245', 'n02071294', 'n02074367', 'n02077923', 'n02085620', 'n02085782', 'n02085936',
'n02086079', 'n02086240', 'n02086646', 'n02086910', 'n02087046', 'n02087394', 'n02088094',
'n02088238', 'n02088364', 'n02088466', 'n02088632', 'n02089078', 'n02089867', 'n02089973',
'n02090379', 'n02090622', 'n02090721', 'n02091032', 'n02091134', 'n02091244', 'n02091467',
'n02091635', 'n02091831', 'n02092002', 'n02092339', 'n02093256', 'n02093428', 'n02093647',
'n02093754', 'n02093859', 'n02093991', 'n02094114', 'n02094258', 'n02094433', 'n02095314',
'n02095570', 'n02095889', 'n02096051', 'n02096177', 'n02096294', 'n02096437', 'n02096585',
'n02097047', 'n02097130', 'n02097209', 'n02097298', 'n02097474', 'n02097658', 'n02098105',
'n02098286', 'n02098413', 'n02099267', 'n02099429', 'n02099601', 'n02099712', 'n02099849',
'n02100236', 'n02100583', 'n02100735', 'n02100877', 'n02101006', 'n02101388', 'n02101556',
'n02102040', 'n02102177', 'n02102318', 'n02102480', 'n02102973', 'n02104029', 'n02104365',
'n02105056', 'n02105162', 'n02105251', 'n02105412', 'n02105505', 'n02105641', 'n02105855',
'n02106030', 'n02106166', 'n02106382', 'n02106550', 'n02106662', 'n02107142', 'n02107312',
'n02107574', 'n02107683', 'n02107908', 'n02108000', 'n02108089', 'n02108422', 'n02108551',
'n02108915', 'n02109047', 'n02109525', 'n02109961', 'n02110063', 'n02110185', 'n02110341',
'n02110627', 'n02110806', 'n02110958', 'n02111129', 'n02111277', 'n02111500', 'n02111889',
'n02112018', 'n02112137', 'n02112350', 'n02112706', 'n02113023', 'n02113186', 'n02113624',
'n02113712', 'n02113799', 'n02113978', 'n02114367', 'n02114548', 'n02114712', 'n02114855',
'n02115641', 'n02115913', 'n02116738', 'n02117135', 'n02119022', 'n02119789', 'n02120079',
'n02120505', 'n02123045', 'n02123159', 'n02123394', 'n02123597', 'n02124075', 'n02125311',
'n02127052', 'n02128385', 'n02128757', 'n02128925', 'n02129165', 'n02129604', 'n02130308',
'n02132136', 'n02133161', 'n02134084', 'n02134418', 'n02137549', 'n02138441', 'n02165105',
'n02165456', 'n02167151', 'n02168699', 'n02169497', 'n02172182', 'n02174001', 'n02177972',
'n02190166', 'n02206856', 'n02219486', 'n02226429', 'n02229544', 'n02231487', 'n02233338',
'n02236044', 'n02256656', 'n02259212', 'n02264363', 'n02268443', 'n02268853', 'n02276258',
'n02277742', 'n02279972', 'n02280649', 'n02281406', 'n02281787', 'n02317335', 'n02319095',
'n02321529', 'n02325366', 'n02326432', 'n02328150', 'n02342885', 'n02346627', 'n02356798',
'n02361337', 'n02363005', 'n02364673', 'n02389026', 'n02391049', 'n02395406', 'n02396427',
'n02397096', 'n02398521', 'n02403003', 'n02408429', 'n02410509', 'n02412080', 'n02415577',
'n02417914', 'n02422106', 'n02422699', 'n02423022', 'n02437312', 'n02437616', 'n02441942',
'n02442845', 'n02443114', 'n02443484', 'n02444819', 'n02445715', 'n02447366', 'n02454379',
'n02457408', 'n02480495', 'n02480855', 'n02481823', 'n02483362', 'n02483708', 'n02484975',
'n02486261', 'n02486410', 'n02487347', 'n02488291', 'n02488702', 'n02489166', 'n02490219',
'n02492035', 'n02492660', 'n02493509', 'n02493793', 'n02494079', 'n02497673', 'n02500267',
'n02504013', 'n02504458', 'n02509815', 'n02510455', 'n02514041', 'n02526121', 'n02536864',
'n02606052', 'n02607072', 'n02640242', 'n02641379', 'n02643566', 'n02655020', 'n02666196',
'n02667093', 'n02669723', 'n02672831', 'n02676566', 'n02687172', 'n02690373', 'n02692877',
'n02699494', 'n02701002', 'n02704792', 'n02708093', 'n02727426', 'n02730930', 'n02747177',
'n02749479', 'n02769748', 'n02776631', 'n02777292', 'n02782093', 'n02783161', 'n02786058',
'n02787622', 'n02788148', 'n02790996', 'n02791124', 'n02791270', 'n02793495', 'n02794156',
'n02795169', 'n02797295', 'n02799071', 'n02802426', 'n02804414', 'n02804610', 'n02807133',
'n02808304', 'n02808440', 'n02814533', 'n02814860', 'n02815834', 'n02817516', 'n02823428',
'n02823750', 'n02825657', 'n02834397', 'n02835271', 'n02837789', 'n02840245', 'n02841315',
'n02843684', 'n02859443', 'n02860847', 'n02865351', 'n02869837', 'n02870880', 'n02871525',
'n02877765', 'n02879718', 'n02883205', 'n02892201', 'n02892767', 'n02894605', 'n02895154',
'n02906734', 'n02909870', 'n02910353', 'n02916936', 'n02917067', 'n02927161', 'n02930766',
'n02939185', 'n02948072', 'n02950826', 'n02951358', 'n02951585', 'n02963159', 'n02965783',
'n02966193', 'n02966687', 'n02971356', 'n02974003', 'n02977058', 'n02978881', 'n02979186',
'n02980441', 'n02981792', 'n02988304', 'n02992211', 'n02992529', 'n02999410', 'n03000134',
'n03000247', 'n03000684', 'n03014705', 'n03016953', 'n03017168', 'n03018349', 'n03026506',
'n03028079', 'n03032252', 'n03041632', 'n03042490', 'n03045698', 'n03047690', 'n03062245',
'n03063599', 'n03063689', 'n03065424', 'n03075370', 'n03085013', 'n03089624', 'n03095699',
'n03100240', 'n03109150', 'n03110669', 'n03124043', 'n03124170', 'n03125729', 'n03126707',
'n03127747', 'n03127925', 'n03131574', 'n03133878', 'n03134739', 'n03141823', 'n03146219',
'n03160309', 'n03179701', 'n03180011', 'n03187595', 'n03188531', 'n03196217', 'n03197337',
'n03201208', 'n03207743', 'n03207941', 'n03208938', 'n03216828', 'n03218198', 'n03220513',
'n03223299', 'n03240683', 'n03249569', 'n03250847', 'n03255030', 'n03259280', 'n03271574',
'n03272010', 'n03272562', 'n03290653', 'n03291819', 'n03297495', 'n03314780', 'n03325584',
'n03337140', 'n03344393', 'n03345487', 'n03347037', 'n03355925', 'n03372029', 'n03376595',
'n03379051', 'n03384352', 'n03388043', 'n03388183', 'n03388549', 'n03393912', 'n03394916',
'n03400231', 'n03404251', 'n03417042', 'n03424325', 'n03425413', 'n03443371', 'n03444034',
'n03445777', 'n03445924', 'n03447447', 'n03447721', 'n03450230', 'n03452741', 'n03457902',
'n03459775', 'n03461385', 'n03467068', 'n03476684', 'n03476991', 'n03478589', 'n03481172',
'n03482405', 'n03483316', 'n03485407', 'n03485794', 'n03492542', 'n03494278', 'n03495258',
'n03496892', 'n03498962', 'n03527444', 'n03529860', 'n03530642', 'n03532672', 'n03534580',
'n03535780', 'n03538406', 'n03544143', 'n03584254', 'n03584829', 'n03590841', 'n03594734',
'n03594945', 'n03595614', 'n03598930', 'n03599486', 'n03602883', 'n03617480', 'n03623198',
'n03627232', 'n03630383', 'n03633091', 'n03637318', 'n03642806', 'n03649909', 'n03657121',
'n03658185', 'n03661043', 'n03662601', 'n03666591', 'n03670208', 'n03673027', 'n03676483',
'n03680355', 'n03690938', 'n03691459', 'n03692522', 'n03697007', 'n03706229', 'n03709823',
'n03710193', 'n03710637', 'n03710721', 'n03717622', 'n03720891', 'n03721384', 'n03724870',
'n03729826', 'n03733131', 'n03733281', 'n03733805', 'n03742115', 'n03743016', 'n03759954',
'n03761084', 'n03763968', 'n03764736', 'n03769881', 'n03770439', 'n03770679', 'n03773504',
'n03775071', 'n03775546', 'n03776460', 'n03777568', 'n03777754', 'n03781244', 'n03782006',
'n03785016', 'n03786901', 'n03787032', 'n03788195', 'n03788365', 'n03791053', 'n03792782',
'n03792972', 'n03793489', 'n03794056', 'n03796401', 'n03803284', 'n03804744', 'n03814639',
'n03814906', 'n03825788', 'n03832673', 'n03837869', 'n03838899', 'n03840681', 'n03841143',
'n03843555', 'n03854065', 'n03857828', 'n03866082', 'n03868242', 'n03868863', 'n03871628',
'n03873416', 'n03874293', 'n03874599', 'n03876231', 'n03877472', 'n03877845', 'n03884397',
'n03887697', 'n03888257', 'n03888605', 'n03891251', 'n03891332', 'n03895866', 'n03899768',
'n03902125', 'n03903868', 'n03908618', 'n03908714', 'n03916031', 'n03920288', 'n03924679',
'n03929660', 'n03929855', 'n03930313', 'n03930630', 'n03933933', 'n03935335', 'n03937543',
'n03938244', 'n03942813', 'n03944341', 'n03947888', 'n03950228', 'n03954731', 'n03956157',
'n03958227', 'n03961711', 'n03967562', 'n03970156', 'n03976467', 'n03976657', 'n03977966',
'n03980874', 'n03982430', 'n03983396', 'n03991062', 'n03992509', 'n03995372', 'n03998194',
'n04004767', 'n04005630', 'n04008634', 'n04009552', 'n04019541', 'n04023962', 'n04026417',
'n04033901', 'n04033995', 'n04037443', 'n04039381', 'n04040759', 'n04041544', 'n04044716',
'n04049303', 'n04065272', 'n04067472', 'n04069434', 'n04070727', 'n04074963', 'n04081281',
'n04086273', 'n04090263', 'n04099969', 'n04111531', 'n04116512', 'n04118538', 'n04118776',
'n04120489', 'n04125021', 'n04127249', 'n04131690', 'n04133789', 'n04136333', 'n04141076',
'n04141327', 'n04141975', 'n04146614', 'n04147183', 'n04149813', 'n04152593', 'n04153751',
'n04154565', 'n04162706', 'n04179913', 'n04192698', 'n04200800', 'n04201297', 'n04204238',
'n04204347', 'n04208210', 'n04209133', 'n04209239', 'n04228054', 'n04229816', 'n04235860',
'n04238763', 'n04239074', 'n04243546', 'n04251144', 'n04252077', 'n04252225', 'n04254120',
'n04254680', 'n04254777', 'n04258138', 'n04259630', 'n04263257', 'n04264628', 'n04265275',
'n04266014', 'n04270147', 'n04273569', 'n04275548', 'n04277352', 'n04285008', 'n04286575',
'n04296562', 'n04310018', 'n04311004', 'n04311174', 'n04317175', 'n04325704', 'n04326547',
'n04328186', 'n04330267', 'n04332243', 'n04335435', 'n04336792', 'n04344873', 'n04346328',
'n04347754', 'n04350905', 'n04355338', 'n04355933', 'n04356056', 'n04357314', 'n04366367',
'n04367480', 'n04370456', 'n04371430', 'n04371774', 'n04372370', 'n04376876', 'n04380533',
'n04389033', 'n04392985', 'n04398044', 'n04399382', 'n04404412', 'n04409515', 'n04417672',
'n04418357', 'n04423845', 'n04428191', 'n04429376', 'n04435653', 'n04442312', 'n04443257',
'n04447861', 'n04456115', 'n04458633', 'n04461696', 'n04462240', 'n04465501', 'n04467665',
'n04476259', 'n04479046', 'n04482393', 'n04483307', 'n04485082', 'n04486054', 'n04487081',
'n04487394', 'n04493381', 'n04501370', 'n04505470', 'n04507155', 'n04509417', 'n04515003',
'n04517823', 'n04522168', 'n04523525', 'n04525038', 'n04525305', 'n04532106', 'n04532670',
'n04536866', 'n04540053', 'n04542943', 'n04548280', 'n04548362', 'n04550184', 'n04552348',
'n04553703', 'n04554684', 'n04557648', 'n04560804', 'n04562935', 'n04579145', 'n04579432',
'n04584207', 'n04589890', 'n04590129', 'n04591157', 'n04591713', 'n04592741', 'n04596742',
'n04597913', 'n04599235', 'n04604644', 'n04606251', 'n04612504', 'n04613696', 'n06359193',
'n06596364', 'n06785654', 'n06794110', 'n06874185', 'n07248320', 'n07565083', 'n07579787',
'n07583066', 'n07584110', 'n07590611', 'n07613480', 'n07614500', 'n07615774', 'n07684084',
'n07693725', 'n07695742', 'n07697313', 'n07697537', 'n07711569', 'n07714571', 'n07714990',
'n07715103', 'n07716358', 'n07716906', 'n07717410', 'n07717556', 'n07718472', 'n07718747',
'n07720875', 'n07730033', 'n07734744', 'n07742313', 'n07745940', 'n07747607', 'n07749582',
'n07753113', 'n07753275', 'n07753592', 'n07754684', 'n07760859', 'n07768694', 'n07802026',
'n07831146', 'n07836838', 'n07860988', 'n07871810', 'n07873807', 'n07875152', 'n07880968',
'n07892512', 'n07920052', 'n07930864', 'n07932039', 'n09193705', 'n09229709', 'n09246464',
'n09256479', 'n09288635', 'n09332890', 'n09399592', 'n09421951', 'n09428293', 'n09468604',
'n09472597', 'n09835506', 'n10148035', 'n10565667', 'n11879895', 'n11939491', 'n12057211',
'n12144580', 'n12267677', 'n12620546', 'n12768682', 'n12985857', 'n12998815', 'n13037406',
'n13040303', 'n13044778', 'n13052670', 'n13054560', 'n13133613', 'n15075141']
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment