Skip to content

GitLab

"...git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "5760a8fcf61472c63a87c1079aa6c53ec8ebe7d7"
Unverified Commit 5964f820 authored by Maria Khalusova's avatar Maria Khalusova Committed by GitHub
Browse files

[Docs] Model_doc structure/clarity improvements (#26876) 

* first batch of structure improvements for model_docs

* second batch of structure improvements for model_docs

* more structure improvements for model_docs

* more structure improvements for model_docs

* structure improvements for cv model_docs

* more structural refactoring

* addressed feedback about image processors
parent ad8ff962
Hide whitespace changes
Inline Side-by-side
Showing with 217 additions and 123 deletions
docs/source/en/model_doc/albert.md
View file @ 5964f820
...@@ -45,7 +45,10 @@ self-supervised loss that focuses on modeling inter-sentence coherence, and show ...@@ -45,7 +45,10 @@ self-supervised loss that focuses on modeling inter-sentence coherence, and show
with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and
SQuAD benchmarks while having fewer parameters compared to BERT-large.* SQuAD benchmarks while having fewer parameters compared to BERT-large.*
Tips: This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).
## Usage tips
- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather - ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather
than the left. than the left.
...@@ -56,11 +59,7 @@ Tips: ...@@ -56,11 +59,7 @@ Tips:
- Layers are split in groups that share parameters (to save memory). - Layers are split in groups that share parameters (to save memory).
Next sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have been swapped or not. Next sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have been swapped or not.
## Resources
This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).
## Documentation resources
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification) - [Token classification task guide](../tasks/token_classification)
...@@ -90,6 +89,9 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This ...@@ -90,6 +89,9 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This
[[autodoc]] models.albert.modeling_tf_albert.TFAlbertForPreTrainingOutput [[autodoc]] models.albert.modeling_tf_albert.TFAlbertForPreTrainingOutput
<frameworkcontent>
<pt>
## AlbertModel ## AlbertModel
[[autodoc]] AlbertModel [[autodoc]] AlbertModel
...@@ -124,6 +126,10 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This ...@@ -124,6 +126,10 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This
[[autodoc]] AlbertForQuestionAnswering [[autodoc]] AlbertForQuestionAnswering
- forward - forward
</pt>
<tf>
## TFAlbertModel ## TFAlbertModel
[[autodoc]] TFAlbertModel [[autodoc]] TFAlbertModel
...@@ -159,6 +165,9 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This ...@@ -159,6 +165,9 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This
[[autodoc]] TFAlbertForQuestionAnswering [[autodoc]] TFAlbertForQuestionAnswering
- call - call
</tf>
<jax>
## FlaxAlbertModel ## FlaxAlbertModel
[[autodoc]] FlaxAlbertModel [[autodoc]] FlaxAlbertModel
...@@ -193,3 +202,8 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This ...@@ -193,3 +202,8 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This
[[autodoc]] FlaxAlbertForQuestionAnswering [[autodoc]] FlaxAlbertForQuestionAnswering
- __call__ - __call__
</jax>
</frameworkcontent>
docs/source/en/model_doc/align.md
View file @ 5964f820
...@@ -24,7 +24,10 @@ The abstract from the paper is the following: ...@@ -24,7 +24,10 @@ The abstract from the paper is the following:
*Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.* *Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.*
## Usage This model was contributed by [Alara Dirik](https://huggingface.co/adirik).
The original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.
## Usage example
ALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score. ALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score.
...@@ -56,9 +59,6 @@ probs = logits_per_image.softmax(dim=1) ...@@ -56,9 +59,6 @@ probs = logits_per_image.softmax(dim=1)
print(probs) print(probs)
``` ```
This model was contributed by [Alara Dirik](https://huggingface.co/adirik).
The original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ALIGN. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ALIGN.
...@@ -69,7 +69,6 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -69,7 +69,6 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource.
## AlignConfig ## AlignConfig
[[autodoc]] AlignConfig [[autodoc]] AlignConfig
......
docs/source/en/model_doc/altclip.md
View file @ 5964f820
...@@ -31,7 +31,9 @@ teacher learning and contrastive learning. We validate our method through evalua ...@@ -31,7 +31,9 @@ teacher learning and contrastive learning. We validate our method through evalua
performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with
CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.* CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*
## Usage This model was contributed by [jongjyh](https://huggingface.co/jongjyh).
## Usage tips and example
The usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention The usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention
and we take the [CLS] token in XLM-R to represent text embedding. and we take the [CLS] token in XLM-R to represent text embedding.
...@@ -50,7 +52,6 @@ The [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokeni ...@@ -50,7 +52,6 @@ The [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokeni
encode the text and prepare the images. The following example shows how to get the image-text similarity scores using encode the text and prepare the images. The following example shows how to get the image-text similarity scores using
[`AltCLIPProcessor`] and [`AltCLIPModel`]. [`AltCLIPProcessor`] and [`AltCLIPModel`].
```python ```python
>>> from PIL import Image >>> from PIL import Image
>>> import requests >>> import requests
...@@ -70,11 +71,11 @@ encode the text and prepare the images. The following example shows how to get t ...@@ -70,11 +71,11 @@ encode the text and prepare the images. The following example shows how to get t
>>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
``` ```
Tips: <Tip>
This model is build on `CLIPModel`, so use it like a original CLIP. This model is based on `CLIPModel`, use it like you would use the original [CLIP](clip).
This model was contributed by [jongjyh](https://huggingface.co/jongjyh). </Tip>
## AltCLIPConfig ## AltCLIPConfig
......
docs/source/en/model_doc/audio-spectrogram-transformer.md
View file @ 5964f820
...@@ -26,15 +26,6 @@ The abstract from the paper is the following: ...@@ -26,15 +26,6 @@ The abstract from the paper is the following:
*In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.* *In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.*
Tips:
- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make
sure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet
mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how
the authors compute the stats for a downstream dataset.
- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the
[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png" <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png"
alt="drawing" width="600"/> alt="drawing" width="600"/>
...@@ -43,6 +34,15 @@ alt="drawing" width="600"/> ...@@ -43,6 +34,15 @@ alt="drawing" width="600"/>
This model was contributed by [nielsr](https://huggingface.co/nielsr). This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/YuanGongND/ast). The original code can be found [here](https://github.com/YuanGongND/ast).
## Usage tips
- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make
sure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet
mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how
the authors compute the stats for a downstream dataset.
- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the
[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with the Audio Spectrogram Transformer. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with the Audio Spectrogram Transformer.
......
docs/source/en/model_doc/autoformer.md
View file @ 5964f820
...@@ -39,13 +39,11 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -39,13 +39,11 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
[[autodoc]] AutoformerConfig [[autodoc]] AutoformerConfig
## AutoformerModel ## AutoformerModel
[[autodoc]] AutoformerModel [[autodoc]] AutoformerModel
- forward - forward
## AutoformerForPrediction ## AutoformerForPrediction
[[autodoc]] AutoformerForPrediction [[autodoc]] AutoformerForPrediction
......
docs/source/en/model_doc/bark.md
View file @ 5964f820
...@@ -14,8 +14,7 @@ specific language governing permissions and limitations under the License. ...@@ -14,8 +14,7 @@ specific language governing permissions and limitations under the License.
## Overview ## Overview
Bark is a transformer-based text-to-speech model proposed by Suno AI in [suno-ai/bark](https://github.com/suno-ai/bark). Bark is a transformer-based text-to-speech model proposed by Suno AI in [suno-ai/bark](https://github.com/suno-ai/bark).
Bark is made of 4 main models: Bark is made of 4 main models:
...@@ -26,6 +25,9 @@ Bark is made of 4 main models: ...@@ -26,6 +25,9 @@ Bark is made of 4 main models:
It should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice. It should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice.
This model was contributed by [Yoach Lacombe (ylacombe)](https://huggingface.co/ylacombe) and [Sanchit Gandhi (sanchit-gandhi)](https://github.com/sanchit-gandhi).
The original code can be found [here](https://github.com/suno-ai/bark).
### Optimizing Bark ### Optimizing Bark
Bark can be optimized with just a few extra lines of code, which **significantly reduces its memory footprint** and **accelerates inference**. Bark can be optimized with just a few extra lines of code, which **significantly reduces its memory footprint** and **accelerates inference**.
...@@ -86,7 +88,7 @@ model.enable_cpu_offload() ...@@ -86,7 +88,7 @@ model.enable_cpu_offload()
Find out more on inference optimization techniques [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one). Find out more on inference optimization techniques [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one).
### Tips ### Usage tips
Suno offers a library of voice presets in a number of languages [here](https://suno-ai.notion.site/8b8e8749ed514b0cbf3f699013548683?v=bc67cff786b04b50b3ceb756fd05f68c). Suno offers a library of voice presets in a number of languages [here](https://suno-ai.notion.site/8b8e8749ed514b0cbf3f699013548683?v=bc67cff786b04b50b3ceb756fd05f68c).
These presets are also uploaded in the hub [here](https://huggingface.co/suno/bark-small/tree/main/speaker_embeddings) or [here](https://huggingface.co/suno/bark/tree/main/speaker_embeddings). These presets are also uploaded in the hub [here](https://huggingface.co/suno/bark-small/tree/main/speaker_embeddings) or [here](https://huggingface.co/suno/bark/tree/main/speaker_embeddings).
...@@ -142,11 +144,6 @@ To save the audio, simply take the sample rate from the model config and some sc ...@@ -142,11 +144,6 @@ To save the audio, simply take the sample rate from the model config and some sc
>>> write_wav("bark_generation.wav", sample_rate, audio_array) >>> write_wav("bark_generation.wav", sample_rate, audio_array)
``` ```
This model was contributed by [Yoach Lacombe (ylacombe)](https://huggingface.co/ylacombe) and [Sanchit Gandhi (sanchit-gandhi)](https://github.com/sanchit-gandhi).
The original code can be found [here](https://github.com/suno-ai/bark).
## BarkConfig ## BarkConfig
[[autodoc]] BarkConfig [[autodoc]] BarkConfig
......
docs/source/en/model_doc/bart.md
View file @ 5964f820
...@@ -25,9 +25,6 @@ rendered properly in your Markdown viewer. ...@@ -25,9 +25,6 @@ rendered properly in your Markdown viewer.
</a> </a>
</div> </div>
**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign
@patrickvonplaten
## Overview ## Overview
The Bart model was proposed in [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, The Bart model was proposed in [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation,
...@@ -45,7 +42,9 @@ According to the abstract, ...@@ -45,7 +42,9 @@ According to the abstract,
state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains
of up to 6 ROUGE. of up to 6 ROUGE.
Tips: This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/bart).
## Usage tips:
- BART is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than - BART is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left. the left.
...@@ -57,18 +56,6 @@ Tips: ...@@ -57,18 +56,6 @@ Tips:
* permute sentences * permute sentences
* rotate the document to make it start at a specific token * rotate the document to make it start at a specific token
This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/bart).
### Examples
- Examples and scripts for fine-tuning BART and other models for sequence to sequence tasks can be found in
[examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).
- An example of how to train [`BartForConditionalGeneration`] with a Hugging Face `datasets`
object can be found in this [forum discussion](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904).
- [Distilled checkpoints](https://huggingface.co/models?search=distilbart) are described in this [paper](https://arxiv.org/abs/2010.13002).
## Implementation Notes ## Implementation Notes
- Bart doesn't use `token_type_ids` for sequence classification. Use [`BartTokenizer`] or - Bart doesn't use `token_type_ids` for sequence classification. Use [`BartTokenizer`] or
...@@ -112,6 +99,7 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -112,6 +99,7 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb). - [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).
- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb). - [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).
- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization). - [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).
- An example of how to train [`BartForConditionalGeneration`] with a Hugging Face `datasets` object can be found in this [forum discussion](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904)
- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the 🤗 Hugging Face course. - [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the 🤗 Hugging Face course.
- [Summarization task guide](../tasks/summarization) - [Summarization task guide](../tasks/summarization)
...@@ -134,6 +122,7 @@ See also: ...@@ -134,6 +122,7 @@ See also:
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering) - [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling) - [Causal language modeling task guide](../tasks/language_modeling)
- [Distilled checkpoints](https://huggingface.co/models?search=distilbart) are described in this [paper](https://arxiv.org/abs/2010.13002).
## BartConfig ## BartConfig
...@@ -150,6 +139,10 @@ See also: ...@@ -150,6 +139,10 @@ See also:
[[autodoc]] BartTokenizerFast [[autodoc]] BartTokenizerFast
- all - all
<frameworkcontent>
<pt>
## BartModel ## BartModel
[[autodoc]] BartModel [[autodoc]] BartModel
...@@ -175,6 +168,9 @@ See also: ...@@ -175,6 +168,9 @@ See also:
[[autodoc]] BartForCausalLM [[autodoc]] BartForCausalLM
- forward - forward
</pt>
<tf>
## TFBartModel ## TFBartModel
[[autodoc]] TFBartModel [[autodoc]] TFBartModel
...@@ -190,6 +186,9 @@ See also: ...@@ -190,6 +186,9 @@ See also:
[[autodoc]] TFBartForSequenceClassification [[autodoc]] TFBartForSequenceClassification
- call - call
</tf>
<jax>
## FlaxBartModel ## FlaxBartModel
[[autodoc]] FlaxBartModel [[autodoc]] FlaxBartModel
...@@ -222,3 +221,8 @@ See also: ...@@ -222,3 +221,8 @@ See also:
[[autodoc]] FlaxBartForCausalLM [[autodoc]] FlaxBartForCausalLM
- __call__ - __call__
</jax>
</frameworkcontent>
docs/source/en/model_doc/barthez.md
View file @ 5964f820
...@@ -38,8 +38,14 @@ provides a significant boost over vanilla BARThez, and is on par with or outperf ...@@ -38,8 +38,14 @@ provides a significant boost over vanilla BARThez, and is on par with or outperf
This model was contributed by [moussakam](https://huggingface.co/moussakam). The Authors' code can be found [here](https://github.com/moussaKam/BARThez). This model was contributed by [moussakam](https://huggingface.co/moussakam). The Authors' code can be found [here](https://github.com/moussaKam/BARThez).
<Tip>
### Examples BARThez implementation is the same as BART, except for tokenization. Refer to [BART documentation](bart) for information on
configuration classes and their parameters. BARThez-specific tokenizers are documented below.
</Tip>
## Resources
- BARThez can be fine-tuned on sequence-to-sequence tasks in a similar way as BART, check: - BARThez can be fine-tuned on sequence-to-sequence tasks in a similar way as BART, check:
[examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md). [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).
......
docs/source/en/model_doc/bartpho.md
View file @ 5964f820
...@@ -29,7 +29,9 @@ on a downstream task of Vietnamese text summarization show that in both automati ...@@ -29,7 +29,9 @@ on a downstream task of Vietnamese text summarization show that in both automati
outperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future outperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future
research and applications of generative Vietnamese NLP tasks.* research and applications of generative Vietnamese NLP tasks.*
Example of use: This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BARTpho).
## Usage example
```python ```python
>>> import torch >>> import torch
...@@ -54,7 +56,7 @@ Example of use: ...@@ -54,7 +56,7 @@ Example of use:
>>> features = bartpho(**input_ids) >>> features = bartpho(**input_ids)
``` ```
Tips: ## Usage tips
- Following mBART, BARTpho uses the "large" architecture of BART with an additional layer-normalization layer on top of - Following mBART, BARTpho uses the "large" architecture of BART with an additional layer-normalization layer on top of
both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use
...@@ -79,8 +81,6 @@ Tips: ...@@ -79,8 +81,6 @@ Tips:
Other languages, if employing this pre-trained multilingual SentencePiece model "vocab_file" for subword Other languages, if employing this pre-trained multilingual SentencePiece model "vocab_file" for subword
segmentation, can reuse BartphoTokenizer with their own language-specialized "monolingual_vocab_file". segmentation, can reuse BartphoTokenizer with their own language-specialized "monolingual_vocab_file".
This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BARTpho).
## BartphoTokenizer ## BartphoTokenizer
[[autodoc]] BartphoTokenizer [[autodoc]] BartphoTokenizer
docs/source/en/model_doc/beit.md
View file @ 5964f820
...@@ -39,7 +39,10 @@ with previous pre-training methods. For example, base-size BEiT achieves 83.2% t ...@@ -39,7 +39,10 @@ with previous pre-training methods. For example, base-size BEiT achieves 83.2% t
significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains
86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).* 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*
Tips: This model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was
contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).
## Usage tips
- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They - BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They
outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as
...@@ -68,9 +71,6 @@ alt="drawing" width="600"/> ...@@ -68,9 +71,6 @@ alt="drawing" width="600"/>
<small> BEiT pre-training. Taken from the <a href="https://arxiv.org/abs/2106.08254">original paper.</a> </small> <small> BEiT pre-training. Taken from the <a href="https://arxiv.org/abs/2106.08254">original paper.</a> </small>
This model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was
contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BEiT. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BEiT.
...@@ -107,6 +107,9 @@ If you're interested in submitting a resource to be included here, please feel f ...@@ -107,6 +107,9 @@ If you're interested in submitting a resource to be included here, please feel f
- preprocess - preprocess
- post_process_semantic_segmentation - post_process_semantic_segmentation
<frameworkcontent>
<pt>
## BeitModel ## BeitModel
[[autodoc]] BeitModel [[autodoc]] BeitModel
...@@ -127,6 +130,9 @@ If you're interested in submitting a resource to be included here, please feel f ...@@ -127,6 +130,9 @@ If you're interested in submitting a resource to be included here, please feel f
[[autodoc]] BeitForSemanticSegmentation [[autodoc]] BeitForSemanticSegmentation
- forward - forward
</pt>
<jax>
## FlaxBeitModel ## FlaxBeitModel
[[autodoc]] FlaxBeitModel [[autodoc]] FlaxBeitModel
...@@ -141,3 +147,6 @@ If you're interested in submitting a resource to be included here, please feel f ...@@ -141,3 +147,6 @@ If you're interested in submitting a resource to be included here, please feel f
[[autodoc]] FlaxBeitForImageClassification [[autodoc]] FlaxBeitForImageClassification
- __call__ - __call__
</jax>
</frameworkcontent>
\ No newline at end of file
docs/source/en/model_doc/bert-generation.md
View file @ 5964f820
...@@ -33,10 +33,13 @@ GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the ...@@ -33,10 +33,13 @@ GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the
encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation,
Text Summarization, Sentence Splitting, and Sentence Fusion.* Text Summarization, Sentence Splitting, and Sentence Fusion.*
Usage: This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder).
- The model can be used in combination with the [`EncoderDecoderModel`] to leverage two pretrained ## Usage examples and tips
BERT checkpoints for subsequent fine-tuning.
The model can be used in combination with the [`EncoderDecoderModel`] to leverage two pretrained BERT checkpoints for
subsequent fine-tuning:
```python ```python
>>> # leverage checkpoints for Bert2Bert model... >>> # leverage checkpoints for Bert2Bert model...
...@@ -61,8 +64,7 @@ Usage: ...@@ -61,8 +64,7 @@ Usage:
>>> loss.backward() >>> loss.backward()
``` ```
- Pretrained [`EncoderDecoderModel`] are also directly available in the model hub, e.g., Pretrained [`EncoderDecoderModel`] are also directly available in the model hub, e.g.:
```python ```python
>>> # instantiate sentence fusion model >>> # instantiate sentence fusion model
...@@ -85,9 +87,6 @@ Tips: ...@@ -85,9 +87,6 @@ Tips:
- For summarization, sentence splitting, sentence fusion and translation, no special tokens are required for the input. - For summarization, sentence splitting, sentence fusion and translation, no special tokens are required for the input.
Therefore, no EOS token should be added to the end of the input. Therefore, no EOS token should be added to the end of the input.
This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder).
## BertGenerationConfig ## BertGenerationConfig
[[autodoc]] BertGenerationConfig [[autodoc]] BertGenerationConfig
......
docs/source/en/model_doc/bert-japanese.md
View file @ 5964f820
...@@ -67,11 +67,15 @@ Example of using a model with Character tokenization: ...@@ -67,11 +67,15 @@ Example of using a model with Character tokenization:
>>> outputs = bertjapanese(**inputs) >>> outputs = bertjapanese(**inputs)
``` ```
Tips: This model was contributed by [cl-tohoku](https://huggingface.co/cl-tohoku).
- This implementation is the same as BERT, except for tokenization method. Refer to the [documentation of BERT](bert) for more usage examples. <Tip>
This implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for
API reference information.
</Tip>
This model was contributed by [cl-tohoku](https://huggingface.co/cl-tohoku).
## BertJapaneseTokenizer ## BertJapaneseTokenizer
......
docs/source/en/model_doc/bert.md
View file @ 5964f820
...@@ -45,7 +45,9 @@ language processing tasks, including pushing the GLUE score to 80.5% (7.7% point ...@@ -45,7 +45,9 @@ language processing tasks, including pushing the GLUE score to 80.5% (7.7% point
accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute
improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).* improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).*
Tips: This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).
## Usage tips
- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than - BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left. the left.
...@@ -59,10 +61,6 @@ Tips: ...@@ -59,10 +61,6 @@ Tips:
- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not. - The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.
This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
...@@ -137,14 +135,23 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -137,14 +135,23 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
- create_token_type_ids_from_sequences - create_token_type_ids_from_sequences
- save_vocabulary - save_vocabulary
<frameworkcontent>
<pt>
## BertTokenizerFast ## BertTokenizerFast
[[autodoc]] BertTokenizerFast [[autodoc]] BertTokenizerFast
</pt>
<tf>
## TFBertTokenizer ## TFBertTokenizer
[[autodoc]] TFBertTokenizer [[autodoc]] TFBertTokenizer
</tf>
</frameworkcontent>
## Bert specific outputs ## Bert specific outputs
[[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput [[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput
...@@ -153,6 +160,10 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -153,6 +160,10 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
[[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput [[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput
<frameworkcontent>
<pt>
## BertModel ## BertModel
[[autodoc]] BertModel [[autodoc]] BertModel
...@@ -198,6 +209,9 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -198,6 +209,9 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
[[autodoc]] BertForQuestionAnswering [[autodoc]] BertForQuestionAnswering
- forward - forward
</pt>
<tf>
## TFBertModel ## TFBertModel
[[autodoc]] TFBertModel [[autodoc]] TFBertModel
...@@ -243,6 +257,9 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -243,6 +257,9 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
[[autodoc]] TFBertForQuestionAnswering [[autodoc]] TFBertForQuestionAnswering
- call - call
</tf>
<jax>
## FlaxBertModel ## FlaxBertModel
[[autodoc]] FlaxBertModel [[autodoc]] FlaxBertModel
...@@ -287,3 +304,8 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h ...@@ -287,3 +304,8 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
[[autodoc]] FlaxBertForQuestionAnswering [[autodoc]] FlaxBertForQuestionAnswering
- __call__ - __call__
</jax>
</frameworkcontent>
docs/source/en/model_doc/bertweet.md
View file @ 5964f820
...@@ -28,7 +28,9 @@ al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa- ...@@ -28,7 +28,9 @@ al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-
2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks:
Part-of-speech tagging, Named-entity recognition and text classification.* Part-of-speech tagging, Named-entity recognition and text classification.*
Example of use: This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BERTweet).
## Usage example
```python ```python
>>> import torch >>> import torch
...@@ -55,7 +57,12 @@ Example of use: ...@@ -55,7 +57,12 @@ Example of use:
>>> # bertweet = TFAutoModel.from_pretrained("vinai/bertweet-base") >>> # bertweet = TFAutoModel.from_pretrained("vinai/bertweet-base")
``` ```
This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BERTweet). <Tip>
This implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for
API reference information.
</Tip>
## BertweetTokenizer ## BertweetTokenizer
......
docs/source/en/model_doc/big_bird.md
View file @ 5964f820
...@@ -41,7 +41,10 @@ sequence as part of the sparse attention mechanism. The proposed sparse attentio ...@@ -41,7 +41,10 @@ sequence as part of the sparse attention mechanism. The proposed sparse attentio
BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also
propose novel applications to genomics data.* propose novel applications to genomics data.*
Tips: This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta). The original code can be found
[here](https://github.com/google-research/bigbird).
## Usage tips
- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird). - For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).
- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using - BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using
...@@ -53,10 +56,8 @@ Tips: ...@@ -53,10 +56,8 @@ Tips:
- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than - BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left. the left.
This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta). The original code can be found
[here](https://github.com/google-research/bigbird).
## Documentation resources ## Resources
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification) - [Token classification task guide](../tasks/token_classification)
...@@ -85,6 +86,9 @@ This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta ...@@ -85,6 +86,9 @@ This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta
[[autodoc]] models.big_bird.modeling_big_bird.BigBirdForPreTrainingOutput [[autodoc]] models.big_bird.modeling_big_bird.BigBirdForPreTrainingOutput
<frameworkcontent>
<pt>
## BigBirdModel ## BigBirdModel
[[autodoc]] BigBirdModel [[autodoc]] BigBirdModel
...@@ -125,6 +129,9 @@ This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta ...@@ -125,6 +129,9 @@ This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta
[[autodoc]] BigBirdForQuestionAnswering [[autodoc]] BigBirdForQuestionAnswering
- forward - forward
</pt>
<jax>
## FlaxBigBirdModel ## FlaxBigBirdModel
[[autodoc]] FlaxBigBirdModel [[autodoc]] FlaxBigBirdModel
...@@ -164,3 +171,8 @@ This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta ...@@ -164,3 +171,8 @@ This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta
[[autodoc]] FlaxBigBirdForQuestionAnswering [[autodoc]] FlaxBigBirdForQuestionAnswering
- __call__ - __call__
</jax>
</frameworkcontent>
docs/source/en/model_doc/bigbird_pegasus.md
View file @ 5964f820
...@@ -41,7 +41,9 @@ sequence as part of the sparse attention mechanism. The proposed sparse attentio ...@@ -41,7 +41,9 @@ sequence as part of the sparse attention mechanism. The proposed sparse attentio
BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also
propose novel applications to genomics data.* propose novel applications to genomics data.*
Tips: The original code can be found [here](https://github.com/google-research/bigbird).
## Usage tips
- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird). - For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).
- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using - BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using
...@@ -54,9 +56,7 @@ Tips: ...@@ -54,9 +56,7 @@ Tips:
- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than - BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left. the left.
The original code can be found [here](https://github.com/google-research/bigbird). ## Resources
## Documentation resources
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering) - [Question answering task guide](../tasks/question_answering)
......
docs/source/en/model_doc/biogpt.md
View file @ 5964f820
...@@ -25,15 +25,15 @@ The abstract from the paper is the following: ...@@ -25,15 +25,15 @@ The abstract from the paper is the following:
*Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.* *Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.*
Tips: This model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/BioGPT).
## Usage tips
- BioGPT is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right rather than the left. - BioGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left.
- BioGPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows BioGPT to generate syntactically coherent text as it can be observed in the run_generation.py example script. - BioGPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows BioGPT to generate syntactically coherent text as it can be observed in the run_generation.py example script.
- The model can take the `past_key_values` (for PyTorch) as input, which is the previously computed key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the BioGptForCausalLM.forward() method for more information on its usage. - The model can take the `past_key_values` (for PyTorch) as input, which is the previously computed key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the BioGptForCausalLM.forward() method for more information on its usage.
This model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/BioGPT). ## Resources
## Documentation resources
- [Causal language modeling task guide](../tasks/language_modeling) - [Causal language modeling task guide](../tasks/language_modeling)
......
docs/source/en/model_doc/bit.md
View file @ 5964f820
...@@ -25,15 +25,15 @@ The abstract from the paper is the following: ...@@ -25,15 +25,15 @@ The abstract from the paper is the following:
*Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.* *Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.*
Tips: This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/big_transfer).
## Usage tips
- BiT models are equivalent to ResNetv2 in terms of architecture, except that: 1) all batch normalization layers are replaced by [group normalization](https://arxiv.org/abs/1803.08494), - BiT models are equivalent to ResNetv2 in terms of architecture, except that: 1) all batch normalization layers are replaced by [group normalization](https://arxiv.org/abs/1803.08494),
2) [weight standardization](https://arxiv.org/abs/1903.10520) is used for convolutional layers. The authors show that the combination of both is useful for training with large batch sizes, and has a significant 2) [weight standardization](https://arxiv.org/abs/1903.10520) is used for convolutional layers. The authors show that the combination of both is useful for training with large batch sizes, and has a significant
impact on transfer learning. impact on transfer learning.
This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/big_transfer).
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BiT. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BiT.
...@@ -62,5 +62,4 @@ If you're interested in submitting a resource to be included here, please feel f ...@@ -62,5 +62,4 @@ If you're interested in submitting a resource to be included here, please feel f
## BitForImageClassification ## BitForImageClassification
[[autodoc]] BitForImageClassification [[autodoc]] BitForImageClassification
- forward - forward
\ No newline at end of file
docs/source/en/model_doc/blenderbot-small.md
View file @ 5964f820
...@@ -40,15 +40,16 @@ and code publicly available. Human evaluations show our best models are superior ...@@ -40,15 +40,16 @@ and code publicly available. Human evaluations show our best models are superior
dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing
failure cases of our models.* failure cases of our models.*
Tips:
- Blenderbot Small is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left.
This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The authors' code can be This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The authors' code can be
found [here](https://github.com/facebookresearch/ParlAI). found [here](https://github.com/facebookresearch/ParlAI).
## Documentation resources ## Usage tips
Blenderbot Small is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left.
## Resources
- [Causal language modeling task guide](../tasks/language_modeling) - [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation) - [Translation task guide](../tasks/translation)
...@@ -70,6 +71,9 @@ found [here](https://github.com/facebookresearch/ParlAI). ...@@ -70,6 +71,9 @@ found [here](https://github.com/facebookresearch/ParlAI).
[[autodoc]] BlenderbotSmallTokenizerFast [[autodoc]] BlenderbotSmallTokenizerFast
<frameworkcontent>
<pt>
## BlenderbotSmallModel ## BlenderbotSmallModel
[[autodoc]] BlenderbotSmallModel [[autodoc]] BlenderbotSmallModel
...@@ -85,6 +89,9 @@ found [here](https://github.com/facebookresearch/ParlAI). ...@@ -85,6 +89,9 @@ found [here](https://github.com/facebookresearch/ParlAI).
[[autodoc]] BlenderbotSmallForCausalLM [[autodoc]] BlenderbotSmallForCausalLM
- forward - forward
</pt>
<tf>
## TFBlenderbotSmallModel ## TFBlenderbotSmallModel
[[autodoc]] TFBlenderbotSmallModel [[autodoc]] TFBlenderbotSmallModel
...@@ -95,6 +102,9 @@ found [here](https://github.com/facebookresearch/ParlAI). ...@@ -95,6 +102,9 @@ found [here](https://github.com/facebookresearch/ParlAI).
[[autodoc]] TFBlenderbotSmallForConditionalGeneration [[autodoc]] TFBlenderbotSmallForConditionalGeneration
- call - call
</tf>
<jax>
## FlaxBlenderbotSmallModel ## FlaxBlenderbotSmallModel
[[autodoc]] FlaxBlenderbotSmallModel [[autodoc]] FlaxBlenderbotSmallModel
...@@ -108,3 +118,6 @@ found [here](https://github.com/facebookresearch/ParlAI). ...@@ -108,3 +118,6 @@ found [here](https://github.com/facebookresearch/ParlAI).
- __call__ - __call__
- encode - encode
- decode - decode
</jax>
</frameworkcontent>
docs/source/en/model_doc/blenderbot.md
View file @ 5964f820
...@@ -16,8 +16,6 @@ rendered properly in your Markdown viewer. ...@@ -16,8 +16,6 @@ rendered properly in your Markdown viewer.
# Blenderbot # Blenderbot
**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) .
## Overview ## Overview
The Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, The Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,
...@@ -36,26 +34,14 @@ and code publicly available. Human evaluations show our best models are superior ...@@ -36,26 +34,14 @@ and code publicly available. Human evaluations show our best models are superior
dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing
failure cases of our models.* failure cases of our models.*
Tips:
- Blenderbot is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
the left.
This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/facebookresearch/ParlAI) . This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/facebookresearch/ParlAI) .
## Usage tips and example
## Implementation Notes Blenderbot is a model with absolute position embeddings so it's usually advised to pad the inputs on the right
rather than the left.
- Blenderbot uses a standard [seq2seq model transformer](https://arxiv.org/pdf/1706.03762.pdf) based architecture.
- Available checkpoints can be found in the [model hub](https://huggingface.co/models?search=blenderbot).
- This is the *default* Blenderbot model class. However, some smaller checkpoints, such as
`facebook/blenderbot_small_90M`, have a different architecture and consequently should be used with
[BlenderbotSmall](blenderbot-small).
## Usage
Here is an example of model usage: An example:
```python ```python
>>> from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration >>> from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration
...@@ -70,7 +56,16 @@ Here is an example of model usage: ...@@ -70,7 +56,16 @@ Here is an example of model usage:
["<s> That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?</s>"] ["<s> That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?</s>"]
``` ```
## Documentation resources ## Implementation Notes
- Blenderbot uses a standard [seq2seq model transformer](https://arxiv.org/pdf/1706.03762.pdf) based architecture.
- Available checkpoints can be found in the [model hub](https://huggingface.co/models?search=blenderbot).
- This is the *default* Blenderbot model class. However, some smaller checkpoints, such as
`facebook/blenderbot_small_90M`, have a different architecture and consequently should be used with
[BlenderbotSmall](blenderbot-small).
## Resources
- [Causal language modeling task guide](../tasks/language_modeling) - [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation) - [Translation task guide](../tasks/translation)
...@@ -90,9 +85,13 @@ Here is an example of model usage: ...@@ -90,9 +85,13 @@ Here is an example of model usage:
[[autodoc]] BlenderbotTokenizerFast [[autodoc]] BlenderbotTokenizerFast
- build_inputs_with_special_tokens - build_inputs_with_special_tokens
<frameworkcontent>
<pt>
## BlenderbotModel ## BlenderbotModel
See `transformers.BartModel` for arguments to *forward* and *generate* See [`~transformers.BartModel`] for arguments to *forward* and *generate*
[[autodoc]] BlenderbotModel [[autodoc]] BlenderbotModel
- forward - forward
...@@ -109,6 +108,9 @@ See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* an ...@@ -109,6 +108,9 @@ See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* an
[[autodoc]] BlenderbotForCausalLM [[autodoc]] BlenderbotForCausalLM
- forward - forward
</pt>
<tf>
## TFBlenderbotModel ## TFBlenderbotModel
[[autodoc]] TFBlenderbotModel [[autodoc]] TFBlenderbotModel
...@@ -119,6 +121,9 @@ See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* an ...@@ -119,6 +121,9 @@ See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* an
[[autodoc]] TFBlenderbotForConditionalGeneration [[autodoc]] TFBlenderbotForConditionalGeneration
- call - call
</tf>
<jax>
## FlaxBlenderbotModel ## FlaxBlenderbotModel
[[autodoc]] FlaxBlenderbotModel [[autodoc]] FlaxBlenderbotModel
...@@ -132,3 +137,8 @@ See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* an ...@@ -132,3 +137,8 @@ See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* an
- __call__ - __call__
- encode - encode
- decode - decode
</jax>
</frameworkcontent>
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment