# LiBai Model Zoo
To date, LiBai has implemented the following models:
- [Vision Transformer](https://arxiv.org/abs/2010.11929)
- [Swin Transformer](https://arxiv.org/abs/2103.14030)
- [ResMLP](https://arxiv.org/abs/2105.03404)
- [BERT](https://arxiv.org/abs/1810.04805)
- [T5](https://arxiv.org/abs/1910.10683)
- [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
## Parallelism Mode in LiBai
A collection of parallel training strategies is supported in LiBai:
- **Data Parallel Training**
- **Tensor Parallel Training**
- **Pipeline Parallel Training**
You can refer to OneFlow official [tutorial](https://docs.oneflow.org/en/master/parallelism/01_introduction.html) to better understand the basic conception of parallelization techniques.
## Supported Models in LiBai
For more details about the supported parallelism training on different models, please refer to the following table:
| Model |
Data Parallel |
Tensor Parallel |
Pipeline Parallel |
| Vision Transformer |
✔ |
✔ |
✔ |
| Swin Transformer |
✔ |
- |
- |
| ResMLP |
✔ |
✔ |
✔ |
| BERT |
✔ |
✔ |
✔ |
| T5 |
✔ |
✔ |
✔ |
| GPT-2 |
✔ |
✔ |
✔ |
**Additions:**
✔ means you can train this model under specific parallelism techniques or combine two or three of them with ✔ for 2D or 3D paralleism training.
## Baselines
Here is the collection of baselines trained with LiBai. Due to our resource constraints, we will gradually release the training results in the future.
### Main Results on ImageNet with Pretrained Models
**ImageNet-1K Pretrained Models**
| Model |
Pretrain |
Resolution |
Acc@1 |
Acc@5 |
Download |
| ViT-Tiny w/o EMA |
ImageNet-1K |
224x224 |
72.7 |
91.0 |
Config | Checkpoint |
| ViT-Small w/o EMA |
ImageNet-1K |
224x224 |
79.3 |
94.5 |
Config | Checkpoint |
**Notes:** `w/o EMA` denotes to models pretrained without **Exponential Moving Average** (EMA).