Unverified Commit cfb777f2 authored by Joao Gante's avatar Joao Gante Committed by GitHub
Browse files

Docs - Guide to add a new TensorFlow model (#19256)


Co-authored-by: default avataramyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: default avatarMatt <Rocketknight1@users.noreply.github.com>
parent 6a08162a
...@@ -31,7 +31,7 @@ ...@@ -31,7 +31,7 @@
- local: sagemaker - local: sagemaker
title: Run training on Amazon SageMaker title: Run training on Amazon SageMaker
- local: converting_tensorflow_models - local: converting_tensorflow_models
title: Converting TensorFlow Checkpoints title: Converting from TensorFlow checkpoints
- local: serialization - local: serialization
title: Export 🤗 Transformers models title: Export 🤗 Transformers models
- local: troubleshooting - local: troubleshooting
...@@ -109,6 +109,8 @@ ...@@ -109,6 +109,8 @@
title: How to contribute to transformers? title: How to contribute to transformers?
- local: add_new_model - local: add_new_model
title: How to add a model to 🤗 Transformers? title: How to add a model to 🤗 Transformers?
- local: add_tensorflow_model
title: How to convert a 🤗 Transformers model to TensorFlow?
- local: add_new_pipeline - local: add_new_pipeline
title: How to add a pipeline to 🤗 Transformers? title: How to add a pipeline to 🤗 Transformers?
- local: testing - local: testing
......
...@@ -683,10 +683,11 @@ work left to be done should be a cakewalk 😊. ...@@ -683,10 +683,11 @@ work left to be done should be a cakewalk 😊.
At this point, you have successfully added a new model. However, it is very much possible that the model does not yet At this point, you have successfully added a new model. However, it is very much possible that the model does not yet
fully comply with the required design. To make sure, the implementation is fully compatible with 🤗 Transformers, all fully comply with the required design. To make sure, the implementation is fully compatible with 🤗 Transformers, all
common tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under common tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under
the same `tests/test_modeling_brand_new_bert.py`. Run this test file to verify that all common tests pass: the same `tests/models/brand_new_bert/test_modeling_brand_new_bert.py`. Run this test file to verify that all common
tests pass:
```bash ```bash
pytest tests/test_modeling_brand_new_bert.py pytest tests/models/brand_new_bert/test_modeling_brand_new_bert.py
``` ```
Having fixed all common tests, it is now crucial to ensure that all the nice work you have done is well tested, so that Having fixed all common tests, it is now crucial to ensure that all the nice work you have done is well tested, so that
...@@ -700,7 +701,7 @@ Cookiecutter, called `BrandNewBertModelIntegrationTests` and only has to be fill ...@@ -700,7 +701,7 @@ Cookiecutter, called `BrandNewBertModelIntegrationTests` and only has to be fill
tests are passing, run tests are passing, run
```bash ```bash
RUN_SLOW=1 pytest -sv tests/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests RUN_SLOW=1 pytest -sv tests/models/brand_new_bert/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests
``` ```
<Tip> <Tip>
...@@ -758,7 +759,8 @@ contain a couple of hard-coded integration tests. ...@@ -758,7 +759,8 @@ contain a couple of hard-coded integration tests.
**10. Run End-to-end integration tests** **10. Run End-to-end integration tests**
Having added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the Having added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the
tokenizer to `tests/test_modeling_brand_new_bert.py` in 🤗 Transformers. Such a test should show on a meaningful tokenizer to `tests/models/brand_new_bert/test_modeling_brand_new_bert.py` in 🤗 Transformers.
Such a test should show on a meaningful
text-to-text sample that the 🤗 Transformers implementation works as expected. A meaningful text-to-text sample can text-to-text sample that the 🤗 Transformers implementation works as expected. A meaningful text-to-text sample can
include *e.g.* a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc… If none include *e.g.* a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc… If none
of the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a of the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a
......
This diff is collapsed.
...@@ -10,7 +10,7 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o ...@@ -10,7 +10,7 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o
specific language governing permissions and limitations under the License. specific language governing permissions and limitations under the License.
--> -->
# Converting Tensorflow Checkpoints # Converting From Tensorflow Checkpoints
A command-line interface is provided to convert original Bert/GPT/GPT-2/Transformer-XL/XLNet/XLM checkpoints to models A command-line interface is provided to convert original Bert/GPT/GPT-2/Transformer-XL/XLNet/XLM checkpoints to models
that can be loaded using the `from_pretrained` methods of the library. that can be loaded using the `from_pretrained` methods of the library.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment