Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
258ed2ea
Commit
258ed2ea
authored
Jan 16, 2020
by
thomwolf
Browse files
adding details in readme
parent
50ee5957
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
26 additions
and
17 deletions
+26
-17
examples/README.md
examples/README.md
+26
-17
No files found.
examples/README.md
View file @
258ed2ea
...
@@ -766,27 +766,36 @@ Here is an example on evaluating a model using adversarial evaluation of natural
...
@@ -766,27 +766,36 @@ Here is an example on evaluating a model using adversarial evaluation of natural
The HANS dataset can be downloaded from
[
this location
](
https://github.com/tommccoy1/hans
)
.
The HANS dataset can be downloaded from
[
this location
](
https://github.com/tommccoy1/hans
)
.
This is an example of using test_hans.py:
```
bash
```
bash
export
HANS_DIR
=
/path/to/HANS
export
HANS_DIR
=
path-to-hans
export
MODEL_TYPE
=
type-of-the-model-e.g.-bert-roberta-xlnet-etc
export
MODEL_PATH
=
path-to-the-model-directory-that-is-trained-on-NLI-e.g.-by-using-run_glue.py
python ./hans/test_hans.py
\
python examples/test_hans.py
\
--model_type
bert
\
--task_name
hans
\
--model_name_or_path
bert-base-multilingual-cased
\
--model_type
$MODEL_TYPE
\
--language
de
\
--do_eval
\
--train_language
en
\
--do_lower_case
\
--do_train
\
--data_dir
$HANS_DIR
\
--do_eval
\
--model_name_or_path
$MODEL_PATH
\
--data_dir
$XNLI_DIR
\
--max_seq_length
128
\
--per_gpu_train_batch_size
32
\
-output_dir
$MODEL_PATH
\
--learning_rate
5e-5
\
--num_train_epochs
2.0
\
--max_seq_length
128
\
--output_dir
/tmp/debug_xnli/
\
--save_steps
-1
```
```
Evaluating with the previously defined hyper-parameters yields the following results:
This will create the hans_predictions.txt file in MODEL_PATH, which can then be evaluated using hans/evaluate_heur_output.py from the HANS dataset.
The results of the BERT-base model that is trained on MNLI using batch size 8 and the random seed 42 on the HANS dataset is as follows:
```
bash
```
bash
acc
=
0.7093812375249501
Heuristic entailed results:
lexical_overlap: 0.9702
subsequence: 0.9942
constituent: 0.9962
Heuristic non-entailed results:
lexical_overlap: 0.199
subsequence: 0.0396
constituent: 0.118
```
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment