• RafaelWO's avatar
    Added feature to move added tokens in vocabulary for Transformer-XL (#4953) · b99ad457
    RafaelWO authored
    
    
    * Fixed resize_token_embeddings for transfo_xl model
    
    * Fixed resize_token_embeddings for transfo_xl.
    
    Added custom methods to TransfoXLPreTrainedModel for resizing layers of
    the AdaptiveEmbedding.
    
    * Updated docstring
    
    * Fixed resizinhg cutoffs; added check for new size of embedding layer.
    
    * Added test for resize_token_embeddings
    
    * Fixed code quality
    
    * Fixed unchanged cutoffs in model.config
    
    * Added feature to move added tokens in tokenizer.
    
    * Fixed code quality
    
    * Added feature to move added tokens in tokenizer.
    
    * Fixed code quality
    
    * Fixed docstring, renamed sym to 	oken.
    Co-authored-by: default avatarRafael Weingartner <rweingartner.its-b2015@fh-salzburg.ac.at>
    b99ad457
test_tokenization_transfo_xl.py 3.2 KB