Commit 258ed2ea authored by thomwolf's avatar thomwolf
Browse files

adding details in readme

parent 50ee5957
...@@ -766,27 +766,36 @@ Here is an example on evaluating a model using adversarial evaluation of natural ...@@ -766,27 +766,36 @@ Here is an example on evaluating a model using adversarial evaluation of natural
The HANS dataset can be downloaded from [this location](https://github.com/tommccoy1/hans). The HANS dataset can be downloaded from [this location](https://github.com/tommccoy1/hans).
This is an example of using test_hans.py:
```bash ```bash
export HANS_DIR=/path/to/HANS export HANS_DIR=path-to-hans
export MODEL_TYPE=type-of-the-model-e.g.-bert-roberta-xlnet-etc
export MODEL_PATH=path-to-the-model-directory-that-is-trained-on-NLI-e.g.-by-using-run_glue.py
python ./hans/test_hans.py \ python examples/test_hans.py \
--model_type bert \ --task_name hans \
--model_name_or_path bert-base-multilingual-cased \ --model_type $MODEL_TYPE \
--language de \ --do_eval \
--train_language en \ --do_lower_case \
--do_train \ --data_dir $HANS_DIR \
--do_eval \ --model_name_or_path $MODEL_PATH \
--data_dir $XNLI_DIR \ --max_seq_length 128 \
--per_gpu_train_batch_size 32 \ -output_dir $MODEL_PATH \
--learning_rate 5e-5 \
--num_train_epochs 2.0 \
--max_seq_length 128 \
--output_dir /tmp/debug_xnli/ \
--save_steps -1
``` ```
Evaluating with the previously defined hyper-parameters yields the following results: This will create the hans_predictions.txt file in MODEL_PATH, which can then be evaluated using hans/evaluate_heur_output.py from the HANS dataset.
The results of the BERT-base model that is trained on MNLI using batch size 8 and the random seed 42 on the HANS dataset is as follows:
```bash ```bash
acc = 0.7093812375249501 Heuristic entailed results:
lexical_overlap: 0.9702
subsequence: 0.9942
constituent: 0.9962
Heuristic non-entailed results:
lexical_overlap: 0.199
subsequence: 0.0396
constituent: 0.118
``` ```
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment