• Quentin Lhoest's avatar
    Allow Custom Dataset in RAG Retriever (#7763) · 033f29c6
    Quentin Lhoest authored
    * add CustomHFIndex
    
    * typo in config
    
    * update tests
    
    * add custom dataset example
    
    * clean script
    
    * update test data
    
    * minor in test
    
    * docs
    
    * docs
    
    * style
    
    * fix imports
    
    * allow to pass the indexed dataset directly
    
    * update tests
    
    * use multiset DPR
    
    * address thom and patrick's comments
    
    * style
    
    * update dpr tokenizer
    
    * add output_dir flag in use_own_knowledge_dataset.py
    
    * allow custom datasets in examples/rag/finetune.py
    
    * add test for custom dataset in distributed rag retriever
    033f29c6
tokenization_dpr_fast.py 19.9 KB