README.md

# Masked Autoencoders Are Scalable Vision Learners (MAE)

TF2 implementation of [MAE](https://arxiv.org/abs/2111.06377).

## Imagenet pretrain

Model | reolution | pathch size | batch size | epochs | target pixel norm | val MSE
------------ | ------: | ------: | -----:| -----:| -----: | --------: 
(a) ViT-L14 | 224x224 | 14 | 4096 | 800 | no | 0.2456
(b) ViT-L14 | 224x224 | 14 | 4096 | 800 | yes | 0.3630
(c) ViT-L16 | 224x224 | 16 | 4096 | 800 | yes | 0.3866


## ImageNet linear probing

Model     | resolution | pathch size | base learning rate | batch size | init checkpoint | epochs | top1 Acc | dashboard
------------ | :--------: | -----:| -----:| -----:| -----:| -----:| -----: | --------:
ViT-L14 | 224x224 | 14 | 0.1 | 16384 | (b) | 90 | 72.8 | -
ViT-L16 | 224x224 | 16 | 0.1 | 16384 | (c) | 90 | 73.0 | -
ViT-L16 | 224x224 | 16 | 0.1 | 16384 | norm | 90 | 73.9 | Table 1 (d)

## ImageNet finetune

Model     | resolution | pathch size | base learning rate | batch size | init checkpoint | epochs | top1 Acc | dashboard
------------ | :--------: | -----:| -----:| -----:| -----:| -----:| -----: | -----:
ViT-L14 | 224x224 | 14 | 0.001 | 1024 | (a) | 50 | 84.4 | -
ViT-L14 | 224x224 | 14 | 0.001 | 1024 | (b) | 50 | 85.3 | -
ViT-L14 | 224x224 | 14 | 0.00075 | 1024 |(b) | 50 | 85.4 | -
ViT-L14 | 224x224 | 14 | 0.0001 | 4096| scratch | 200 | 82.4 | -
ViT-L16 | 224x224 | 16 | 0.001 | 1024 | (c)| 50 | 84.9 | -
ViT-L16 | 224x224 | 16 | 0.001 | 1024| no-norm | 50 | 84.9 | Table 1(d)
ViT-L16 | 224x224 | 16 | 0.001 | 1024| norm    | 50 | 85.4 | paper section 4.
ViT-L16 | 224x224 | 16 | 0.0001 | 4096| scratch | 200 | 82.5 | paper section 4.


## Known discrepancy with the paper:

*   ~-0.9 linear probing top1 acc (w/ norm) compared to paper results with patch
    size 16.

*   ~-0.5 finetune top1 acc (w/ norm) compared to paper results with patch
    size 16.