Crime and Punishment text was taken from `gs://trax-ml/reformer/crime-and-punishment-2554.txt`.
Crime and Punishment text was taken from `gs://trax-ml/reformer/crime-and-punishment-2554.txt`.
Model was trained in flax using colab notebook proposed by authors: https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb
Model was trained in flax using colab notebook proposed by authors: https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb
Weights were converted to Hugging Face PyTorch `ReformerModelWithLMHead`.
Weights were converted to Hugging Face PyTorch `ReformerModelWithLMHead`.
Model is used as a proof of concept that the forward pass works for a `ReformerModelWithLMHead`.
Given that the model was trained only for 30mins on a ~0.5M tokens dataset and has only 320 tokens,