2. The `examples/pretrain_ict.sh` script runs single GPU 217M parameter biencoder model for ICT retriever training. Single GPU training is primarily intended for debugging purposes, as the code is developed for distributed training. The script uses pretrained BERT model with batch size of 4096 (hence need data parallel world size of 32).
2. The `examples/pretrain_ict.sh` script runs single GPU 217M parameter biencoder model for ICT retriever training. Single GPU training is primarily intended for debugging purposes, as the code is developed for distributed training. The script uses pretrained BERT model with batch size of 4096 (hence need data parallel world size of 32).
<pre>
3. Evaluate the pretrained ICT model using `examples/evaluate_retriever_nq.sh` for natural question answering dataset.
PRETRAINED_BERT_PATH="Specify path of pretrained BERT model"
### Supervised finetuning
TEXT_DATA_PATH="Specify path and file prefix of the text data"
TITLE_DATA_PATH="Specify path and file prefix od the titles"
1. We use the above pretrained ICT model to finetune using [Google's natural question answering dataset](https://ai.google.com/research/NaturalQuestions/). We use the script `examples/finetune_retriever_distributed.sh` for this purpose. Our finetuning consists of score scaling, longer training (80 epochs), and hard negative examples.
CHECKPOINT_PATH="Specify path"
2. We evaluate the finetuned model using the same evaluation script as mentioned above for the unsupervised model.
python pretrain_ict.py \
--num-layers 12 \
--hidden-size 768 \
More details on the retriever are available in [our paper](https://arxiv.org/abs/2101.00408).
