"doc/git@developer.sourcefind.cn:ox696c/ktransformers.git" did not exist on "107e4be41791c3f888051bb64e9ff91f0dab77f0"
Commit 71e05d36 authored by A. Unique TensorFlower's avatar A. Unique TensorFlower
Browse files

Updated the text and added some formatting to render better.

PiperOrigin-RevId: 438209905
parent 0c6367ee
This directory contains binaries and utils required for input preprocessing,
tokenization, etc that can be used with model building blocks available in
NLP modeling library [nlp/modelling]
(https://github.com/tensorflow/models/tree/master/official/nlp/modeling) to
train custom models and validate new research ideas.
NLP modeling library [nlp/modelling](https://github.com/tensorflow/models/tree/master/official/nlp/modeling)
to train custom models and validate new research ideas.
This directory contain guides to help users to train NLP models.
1. [Training guide](train.md) explain the steps to follow for training NLP
models on GPU and TPU.
2. [Pretrained_models guide](pretrained_models.md) explain how to load
pre-trained NLP models (baselines and checkpoints) that can be finetuned
further depending on application.
3. [TF-Hub guide](tfhub.md) explain how to use TF-NLP's
[export_tfhub](https://github.com/tensorflow/models/blob/master/official/nlp/tools/export_tfhub.py)
tool to export pre-trained Transformer encoders to SavedModels format that are
suitable for publication on TF Hub.
# Model Garden NLP Common Training Driver
[train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py) is the common training driver that supports multiple
[train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py)
is the common training driver that supports multiple
NLP tasks (e.g., pre-training, GLUE and SQuAD fine-tuning etc) and multiple
models (e.g., BERT, ALBERT, MobileBERT etc).
## Experiment Configuration
[train.py] is driven by configs defined by the [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py)
[train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py)
is driven by configs defined by the [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py)
including configurations for `task`, `trainer` and `runtime`. The pre-defined
NLP related [ExperimentConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) can be found in
[configs/experiment_configs.py](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiment_configs.py).
......@@ -78,7 +80,9 @@ setting `task.validation_data.input_path` in `PARAMS`.
## Run on Cloud TPUs
Next, we will describe how to run the [train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py) on Cloud TPUs.
Next, we will describe how to run
the [train.py](https://github.com/tensorflow/models/blob/master/official/nlp/train.py)
on Cloud TPUs.
### Setup
First, you need to create a `tf-nightly` TPU with
......@@ -99,6 +103,8 @@ pip3 install --user -r official/requirements.txt
### Fine-tuning Sentence Classification with BERT from TF-Hub
<details>
This example fine-tunes BERT-base from TF-Hub on the the Multi-Genre Natural
Language Inference (MultiNLI) corpus using TPUs.
......@@ -163,8 +169,12 @@ python3 train.py \
You can monitor the training progress in the console and find the output
models in `$OUTPUT_DIR`.
</details>
### Fine-tuning SQuAD with a pre-trained BERT checkpoint
<details>
This example fine-tunes a pre-trained BERT checkpoint on the
Stanford Question Answering Dataset (SQuAD) using TPUs.
The [SQuAD website](https://rajpurkar.github.io/SQuAD-explorer/) contains
......@@ -219,4 +229,6 @@ python3 train.py \
```
</details>
Note: More examples about pre-training will come soon.
......@@ -20,8 +20,7 @@ examples.
* [`losses`](losses) contains common loss computation used in NLP tasks.
Please see the colab
[nlp_modeling_library_intro.ipynb]
(https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/nlp_modeling_library_intro.ipynb)
[nlp_modeling_library_intro.ipynb](https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/nlp_modeling_library_intro.ipynb)
for how to build transformer-based NLP models using above primitives.
Besides the pre-defined primitives, it also provides scaffold classes to allow
......@@ -44,8 +43,7 @@ custom hidden layer (which will replace the Transformer instantiation in the
encoder).
Please see the colab
[customize_encoder.ipynb]
(https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/customize_encoder.ipynb)
[customize_encoder.ipynb](https://colab.sandbox.google.com/github/tensorflow/models/blob/master/official/colab/nlp/customize_encoder.ipynb)
for how to use scaffold classes to build noval achitectures.
BERT and ALBERT models in this repo are implemented using this library.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment