• Arthur's avatar
    [`CodeLlama`] Add support for `CodeLlama` (#25740) · 015f8e11
    Arthur authored
    
    
    * add all
    
    * Revert "Delete .github directory"
    
    This reverts commit 9b0ff7b052e2b20b629a26fb13606b78a42944d1.
    
    * make conversion script backward compatible
    
    * fixup
    
    * more styling
    
    * copy to llama changes
    
    * fix repo consistency
    
    * nits
    
    * document correct classes
    
    * updates
    
    * more fixes
    
    * nits
    
    * update auto mappings
    
    * add readmes
    
    * smallupdates
    
    * llama-code replace with llama_code
    
    * make fixup
    
    * updates to the testsing suite
    
    * fix fast nits
    
    * more small fixes
    
    * fix decode
    
    * fix template processing
    
    * properly reset the normalizer
    
    * nits processor
    
    * tokenization tests pass
    
    * styling
    
    * last tests
    
    * additional nits
    
    * one test is left
    
    * nits
    
    Co-authored-by faabian <faabian@users.noreply.github.com>
    
    * update failing test
    
    * fixup
    
    * remove decode infilling users should handle it on their onw after generation, padding can be a problem
    
    * update
    
    * make test slow and more meaningfull
    
    * fixup
    
    * doc update
    
    * fixup
    
    * Apply suggestions from code review
    
    * add kwargs doc
    
    * tokenizer requires `requires_backend`
    
    * type requires_backends
    
    * CodeLlama instead of LlamaCode
    
    * more name cahnges
    
    * nits
    
    * make doctests happy
    
    * small pipeline nits
    
    * last nit
    
    * Apply suggestions from code review
    Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
    
    * update
    
    * add codellama to toctree
    
    ---------
    Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
    015f8e11
README_ja.md 99.4 KB