• NielsRogge's avatar
    Add LayoutXLMProcessor (and LayoutXLMTokenizer, LayoutXLMTokenizerFast) (#14115) · 5f789a68
    NielsRogge authored
    
    
    * Add LayoutXLMTokenizer and LayoutXLMTokenizerFast
    
    * Fix styling issues
    
    * Fix more styling issues
    
    * Fix more styling issues
    
    * Fix docstring
    
    * Fix unit tests
    
    * Fix docs
    
    * Fix unit tests
    
    * Fix typos and styling issues
    
    * Fix styling issues
    
    * Fix docstring
    
    * Make all tests of test_tokenization_layoutxlm pass
    
    * Add LayoutXLMProcessor
    
    * Make fixup
    
    * Make all LayoutXLMProcessor tests pass
    
    * Minor fixes
    
    * Leave LayoutLMv2Processor tests unchanged
    
    * Fix code quality
    
    * Move LayoutXLM tokenizers and processor to separate folder
    
    * Fix code quality
    
    * Apply suggestions from code review
    
    * Replace assertions by value errors
    
    * Remove methods from fast tokenizer
    Co-authored-by: default avatarKing Yiu Suen <kingyiusuen@gmail.com>
    5f789a68
test_tokenization_layoutxlm.py 90.1 KB