Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
258ed2ea
Commit
258ed2ea
authored
Jan 16, 2020
by
thomwolf
Browse files
adding details in readme
parent
50ee5957
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
26 additions
and
17 deletions
+26
-17
examples/README.md
examples/README.md
+26
-17
No files found.
examples/README.md
View file @
258ed2ea
...
@@ -766,27 +766,36 @@ Here is an example on evaluating a model using adversarial evaluation of natural
...
@@ -766,27 +766,36 @@ Here is an example on evaluating a model using adversarial evaluation of natural
The HANS dataset can be downloaded from
[
this location
](
https://github.com/tommccoy1/hans
)
.
The HANS dataset can be downloaded from
[
this location
](
https://github.com/tommccoy1/hans
)
.
This is an example of using test_hans.py:
```
bash
```
bash
export
HANS_DIR
=
/path/to/HANS
export
HANS_DIR
=
path-to-hans
export
MODEL_TYPE
=
type-of-the-model-e.g.-bert-roberta-xlnet-etc
export
MODEL_PATH
=
path-to-the-model-directory-that-is-trained-on-NLI-e.g.-by-using-run_glue.py
python ./hans/test_hans.py
\
python examples/test_hans.py
\
--model_type
bert
\
--task_name
hans
\
--model_name_or_path
bert-base-multilingual-cased
\
--model_type
$MODEL_TYPE
\
--language
de
\
--do_eval
\
--train_language
en
\
--do_lower_case
\
--do_train
\
--data_dir
$HANS_DIR
\
--do_eval
\
--model_name_or_path
$MODEL_PATH
\
--data_dir
$XNLI_DIR
\
--max_seq_length
128
\
--per_gpu_train_batch_size
32
\
-output_dir
$MODEL_PATH
\
--learning_rate
5e-5
\
--num_train_epochs
2.0
\
--max_seq_length
128
\
--output_dir
/tmp/debug_xnli/
\
--save_steps
-1
```
```
Evaluating with the previously defined hyper-parameters yields the following results:
This will create the hans_predictions.txt file in MODEL_PATH, which can then be evaluated using hans/evaluate_heur_output.py from the HANS dataset.
The results of the BERT-base model that is trained on MNLI using batch size 8 and the random seed 42 on the HANS dataset is as follows:
```
bash
```
bash
acc
=
0.7093812375249501
Heuristic entailed results:
lexical_overlap: 0.9702
subsequence: 0.9942
constituent: 0.9962
Heuristic non-entailed results:
lexical_overlap: 0.199
subsequence: 0.0396
constituent: 0.118
```
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment