• Quentin Lhoest's avatar
    Add DPR model (#5279) · fbd87921
    Quentin Lhoest authored
    
    
    * beginning of dpr modeling
    
    * wip
    
    * implement forward
    
    * remove biencoder + better init weights
    
    * export dpr model to embed model for nlp lib
    
    * add new api
    
    * remove old code
    
    * make style
    
    * fix dumb typo
    
    * don't load bert weights
    
    * docs
    
    * docs
    
    * style
    
    * move the `k` parameter
    
    * fix init_weights
    
    * add pretrained configs
    
    * minor
    
    * update config names
    
    * style
    
    * better config
    
    * style
    
    * clean code based on PR comments
    
    * change Dpr to DPR
    
    * fix config
    
    * switch encoder config to a dict
    
    * style
    
    * inheritance -> composition
    
    * add messages in assert startements
    
    * add dpr reader tokenizer
    
    * one tokenizer per model
    
    * fix base_model_prefix
    
    * fix imports
    
    * typo
    
    * add convert script
    
    * docs
    
    * change tokenizers conf names
    
    * style
    
    * change tokenizers conf names
    
    * minor
    
    * minor
    
    * fix wrong names
    
    * minor
    
    * remove unused convert functions
    
    * rename convert script
    
    * use return_tensors in tokenizers
    
    * remove n_questions dim
    
    * move generate logic to tokenizer
    
    * style
    
    * add docs
    
    * docs
    
    * quality
    
    * docs
    
    * add tests
    
    * style
    
    * add tokenization tests
    
    * DPR full tests
    
    * Stay true to the attention mask building
    
    * update docs
    
    * missing param in bert input docs
    
    * docs
    
    * style
    Co-authored-by: default avatarLysandre <lysandre.debut@reseau.eseo.fr>
    fbd87921
test_tokenization_dpr.py 3.48 KB