Commit 71e05d36 authored by A. Unique TensorFlower's avatar A. Unique TensorFlower
Browse files

Updated the text and added some formatting to render better.

PiperOrigin-RevId: 438209905
parent 0c6367ee
This directory contains binaries and utils required for input preprocessing, This directory contains binaries and utils required for input preprocessing,
tokenization, etc that can be used with model building blocks available in tokenization, etc that can be used with model building blocks available in
NLP modeling library [nlp/modelling] NLP modeling library [nlp/modelling](https://github.com/tensorflow/models/tree/master/official/nlp/modeling)
(https://github.com/tensorflow/models/tree/master/official/nlp/modeling) to to train custom models and validate new research ideas.
train custom models and validate new research ideas.
This directory contain guides to help users to train NLP models.
1. [Training guide](train.md) explain the steps to follow for training NLP
models on GPU and TPU.
2. [Pretrained_models guide](pretrained_models.md) explain how to load
pre-trained NLP models (baselines and checkpoints) that can be finetuned
further depending on application.
3. [TF-Hub guide](tfhub.md) explain how to use TF-NLP's
[export_tfhub](https://github.com/tensorflow/models/blob/master/official/nlp/tools/export_tfhub.py)
tool to export pre-trained Transformer encoders to SavedModels format that are
suitable for publication on TF Hub.
# Model Garden NLP Common Training Driver # Model Garden NLP Common Training Driver
[train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py) is the common training driver that supports multiple [train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py)
is the common training driver that supports multiple
NLP tasks (e.g., pre-training, GLUE and SQuAD fine-tuning etc) and multiple NLP tasks (e.g., pre-training, GLUE and SQuAD fine-tuning etc) and multiple
models (e.g., BERT, ALBERT, MobileBERT etc). models (e.g., BERT, ALBERT, MobileBERT etc).
## Experiment Configuration ## Experiment Configuration
[train.py] is driven by configs defined by the [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) [train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py)
is driven by configs defined by the [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py)
including configurations for `task`, `trainer` and `runtime`. The pre-defined including configurations for `task`, `trainer` and `runtime`. The pre-defined
NLP related [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) can be found in NLP related [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) can be found in
[configs/experiment_configs.py](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiment_configs.py). [configs/experiment_configs.py](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiment_configs.py).
...@@ -78,7 +80,9 @@ setting `task.validation_data.input_path` in `PARAMS`. ...@@ -78,7 +80,9 @@ setting `task.validation_data.input_path` in `PARAMS`.
## Run on Cloud TPUs ## Run on Cloud TPUs
Next, we will describe how to run the [train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py) on Cloud TPUs. Next, we will describe how to run
the [train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py)
on Cloud TPUs.
### Setup ### Setup
First, you need to create a `tf-nightly` TPU with First, you need to create a `tf-nightly` TPU with
...@@ -99,6 +103,8 @@ pip3 install --user -r official/requirements.txt ...@@ -99,6 +103,8 @@ pip3 install --user -r official/requirements.txt
### Fine-tuning Sentence Classification with BERT from TF-Hub ### Fine-tuning Sentence Classification with BERT from TF-Hub
<details>
This example fine-tunes BERT-base from TF-Hub on the the Multi-Genre Natural This example fine-tunes BERT-base from TF-Hub on the the Multi-Genre Natural
Language Inference (MultiNLI) corpus using TPUs. Language Inference (MultiNLI) corpus using TPUs.
...@@ -163,8 +169,12 @@ python3 train.py \ ...@@ -163,8 +169,12 @@ python3 train.py \
You can monitor the training progress in the console and find the output You can monitor the training progress in the console and find the output
models in `$OUTPUT_DIR`. models in `$OUTPUT_DIR`.
</details>
### Fine-tuning SQuAD with a pre-trained BERT checkpoint ### Fine-tuning SQuAD with a pre-trained BERT checkpoint
<details>
This example fine-tunes a pre-trained BERT checkpoint on the This example fine-tunes a pre-trained BERT checkpoint on the
Stanford Question Answering Dataset (SQuAD) using TPUs. Stanford Question Answering Dataset (SQuAD) using TPUs.
The [SQuAD website](https://rajpurkar.github.io/SQuAD-explorer/) contains The [SQuAD website](https://rajpurkar.github.io/SQuAD-explorer/) contains
...@@ -219,4 +229,6 @@ python3 train.py \ ...@@ -219,4 +229,6 @@ python3 train.py \
``` ```
</details>
Note: More examples about pre-training will come soon. Note: More examples about pre-training will come soon.
...@@ -20,8 +20,7 @@ examples. ...@@ -20,8 +20,7 @@ examples.
* [`losses`](losses) contains common loss computation used in NLP tasks. * [`losses`](losses) contains common loss computation used in NLP tasks.
Please see the colab Please see the colab
[nlp_modeling_library_intro.ipynb] [nlp_modeling_library_intro.ipynb](https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/nlp_modeling_library_intro.ipynb)
(https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/nlp_modeling_library_intro.ipynb)
for how to build transformer-based NLP models using above primitives. for how to build transformer-based NLP models using above primitives.
Besides the pre-defined primitives, it also provides scaffold classes to allow Besides the pre-defined primitives, it also provides scaffold classes to allow
...@@ -44,8 +43,7 @@ custom hidden layer (which will replace the Transformer instantiation in the ...@@ -44,8 +43,7 @@ custom hidden layer (which will replace the Transformer instantiation in the
encoder). encoder).
Please see the colab Please see the colab
[customize_encoder.ipynb] [customize_encoder.ipynb](https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/customize_encoder.ipynb)
(https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/customize_encoder.ipynb)
for how to use scaffold classes to build noval achitectures. for how to use scaffold classes to build noval achitectures.
BERT and ALBERT models in this repo are implemented using this library. BERT and ALBERT models in this repo are implemented using this library.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment