Skip to content

GitLab

Commit 60f6d6cb authored by Frederick Liu's avatar Frederick Liu Committed by A. Unique TensorFlower
Browse files

Internal change 

PiperOrigin-RevId: 418817477
parent 5fea53a7
Hide whitespace changes
Inline Side-by-side
Showing with 48 additions and 32 deletions
official/nlp/MODEL_GARDEN.md
View file @ 60f6d6cb
...@@ -2,53 +2,69 @@ ...@@ -2,53 +2,69 @@
## Introduction ## Introduction
This TF-NLP library provides a collection of scripts for the training and The TF-NLP library provides a collection of scripts for training and
evaluation of transformer-based models, on various tasks such as sentence evaluating transformer-based models, on various tasks such as sentence
classification, question answering, and translation. Additionally, we provide classification, question answering, and translation. Additionally, we provide
checkpoints of pretrained models which can be finetuned on downstream tasks. checkpoints of pretrained models which can be finetuned on downstream tasks.
### How to Train Models ### How to Train Models
Model Garden can be easily installed using PIP Model Garden can be easily installed with
(`pip install tf-models-nightly`). After installation, check out `pip install tf-models-nightly`. After installation, check out
[this instruction](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md) [this instruction](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md)
on how to train models with this codebase. on how to train models with this codebase.
## Available Tasks
There are two available model configs (we will add more) under By default, the experiment runs on GPUs. To run on TPUs, one should overwrite
`configs/experiments/`: `runtime.distribution_strategy` and set the tpu address. See [RuntimeConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) for details.
In general, the experiments can run with the folloing command by setting the
corresponding `${TASK}`, `${TASK_CONFIG}`, `${MODEL_CONFIG}`.
```
EXPERIMENT=???
TASK_CONFIG=???
MODEL_CONFIG=???
EXRTRA_PARAMS=???
MODEL_DIR=??? # a-folder-to-hold-checkpoints-and-logs
python3 train.py \
--experiment=${EXPERIMENT} \
--mode=train_and_eval \
--model_dir=${MODEL_DIR} \
--config_file=${TASK_CONFIG} \
--config_file=${MODEL_CONFIG} \
--params_override=${EXRTRA_PARAMS}
```
* `EXPERIMENT` can be found under `configs/`
* `TASK_CONFIG` can be found under `configs/experiments/`
* `MODEL_CONFIG` can be found under `configs/models/`
#### Order of params override:
1. `train.py` looks up the registered `ExperimentConfig` with `${EXPERIMENT}`
2. Overrides params in `TaskConfig` in `${TASK_CONFIG}`
3. Overrides params `model` in `TaskConfig` with `${MODEL_CONFIG}`
4. Overrides any params in `ExperimentConfig` with `${EXTRA_PARAMS}`
Note that
1. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` can be optional when EXPERIMENT default is enough.
2. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` are only guaranteed to be compatible to it's `${EXPERIMENT}` that defines it.
## Experiments
| NAME | EXPERIMENT | TASK_CONFIG | MODEL_CONFIG | EXRTRA_PARAMS |
| ----------------- | ------------------------ | ------- | -------- | ----------- |
| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | <details> <summary>data and bert-base hub init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 </details> |
| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | <details> <summary>data and bert-base ckpt init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.init_checkpoint=gs://tf_model_garden/nlp/bert/uncased_L-12_H-768_A-12/bert_model.ckpt </details> |
| BERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | <details> <summary>data and bert-base hub init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 </details> |
|ALBERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml)| <details> <summary>data and albert-base hub init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3 </details>|
| Transformer-large WMT14/en-de scratch |[wmt_transformer/large](https://github.com/tensorflow/models/blob/master/official/nlp/configs/wmt_transformer_experiments.py)| | | <details> <summary>ende-32k sentencepiece</summary>task.sentencepiece_model_path='gs://tf_model_garden/nlp/transformer_wmt/ende_bpe_32k.model'</details> |
| Dataset | Task | Config | Example command |
| ----------------- | ------------------------ | ------- | ---- |
| GLUE/MNLI-matched | bert/sentence_prediction | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | <details> <summary>finetune BERT-base on this task</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4<br/><br/>python3 train.py \\<br/> --experiment=bert/sentence_prediction \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/bert_en_uncased_base.yaml \\<br/> --config_file=configs/experiments/glue_mnli_matched.yaml \\<br/> --params_override=${PARAMS}</details> |
| SQuAD v1.1 | bert/squad | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | <details> <summary>finetune BERT-base on this task</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4<br/><br/>python3 train.py \\<br/> --experiment=bert/squad \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/bert_en_uncased_base.yaml \\<br/> --config_file=configs/experiments/squad_v1.yaml \\<br/> --params_override=${PARAMS}</details> |
One example on how to use the config file: if you want to work on the SQuAD
question answering task, set
`--config_file=configs/experiments/squad_v1.yaml` and
`--experiment=bert/squad`
as arguments to `train.py`.
## Available Model Configs
There are two available model configs (we will add more) under
`configs/models/`:
| Model | Config | Pretrained checkpoint & Vocabulary | TF-HUB SavedModel | Example command |
| ------------ | ------- | ---------------------------------- | ----------------- | --------------- |
| BERT-base | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | [uncased_L-12_H-768_A-12](https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/uncased_L-12_H-768_A-12.tar.gz) | [uncased_L-12_H-768_A-12](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/) | <details> <summary>finetune on SQuAD v1.1</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4<br/><br/>python3 train.py \\<br/> --experiment=bert/squad \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/bert_en_uncased_base.yaml \\<br/> --config_file=configs/experiments/squad_v1.yaml \\<br/> --params_override=${PARAMS}</details> |
| ALBERT-base | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml) | [albert_en_base](https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_base.tar.gz) | [albert_en_base](https://tfhub.dev/tensorflow/albert_en_base/3) | <details> <summary>finetune on SQuAD v1.1</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3<br/><br/>python3 train.py \\<br/> --experiment=bert/squad \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/albert_base.yaml \\<br/> --config_file=configs/experiments/squad_v1.yaml \\<br/> --params_override=${PARAMS}</details> |
One example on how to use the config file: if you want to train an ALBERT-base
model, set `--config_file=configs/models/albert_base.yaml` as an argument to
`train.py`.
## Useful links ## Useful links
[How to Train Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md) [How to Train Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md)
[List of Pretrained Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md) [List of Pretrained Models for finetuning](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md)
[How to Publish Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/tfhub.md) [How to Publish Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/tfhub.md)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment