GitLab

Very often splitting large files to smaller files can prevent tokenizer going out of memory in environment like Colab that does not have swap memory