• Pi Esposito's avatar
    add in layer gpt2 tokenizer (#20421) · fb2b45e5
    Pi Esposito authored
    * add minimal working gpt2 tokenizer
    
    * graph mode and output equivalence tests working
    
    * not today tensorflow. serialization test passing!
    
    * fix style, documentation, docstrings and all that jazz
    
    * passing consistency checks
    
    * move keras nlp to tf dependencies
    
    * fix tf modeling utils and gpt2 attention to enable compiling
    
    * fix (I hope) keras nlp dependencies
    
    * rever changes on generation
    
    * remove debug prints
    
    * remove redundant tf dummy objects
    
    * add from config, get config and max length settings to address review
    
    * let flake ignore the error on distillation you are welcome
    
    * test from config
    
    * add padding test
    
    * address sgugger review
    fb2b45e5
gpt2.mdx 7.12 KB