Unverified Commit 5964f820 authored by Maria Khalusova's avatar Maria Khalusova Committed by GitHub
Browse files

[Docs] Model_doc structure/clarity improvements (#26876)

* first batch of structure improvements for model_docs

* second batch of structure improvements for model_docs

* more structure improvements for model_docs

* more structure improvements for model_docs

* structure improvements for cv model_docs

* more structural refactoring

* addressed feedback about image processors
parent ad8ff962
...@@ -27,11 +27,6 @@ The abstract from the paper is the following: ...@@ -27,11 +27,6 @@ The abstract from the paper is the following:
*The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.* *The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*
Tips:
- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.
- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg" <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg"
alt="drawing" width="600"/> alt="drawing" width="600"/>
...@@ -40,6 +35,11 @@ alt="drawing" width="600"/> ...@@ -40,6 +35,11 @@ alt="drawing" width="600"/>
This model was contributed by [nielsr](https://huggingface.co/nielsr). This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207). The original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).
## Usage tips
- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.
- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BLIP-2. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BLIP-2.
......
...@@ -20,7 +20,7 @@ rendered properly in your Markdown viewer. ...@@ -20,7 +20,7 @@ rendered properly in your Markdown viewer.
The BLIP model was proposed in [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086) by Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi. The BLIP model was proposed in [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086) by Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi.
BLIP is a model that is able to perform various multi-modal tasks including BLIP is a model that is able to perform various multi-modal tasks including:
- Visual Question Answering - Visual Question Answering
- Image-Text retrieval (Image-text matching) - Image-Text retrieval (Image-text matching)
- Image Captioning - Image Captioning
...@@ -39,7 +39,6 @@ The original code can be found [here](https://github.com/salesforce/BLIP). ...@@ -39,7 +39,6 @@ The original code can be found [here](https://github.com/salesforce/BLIP).
- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset - [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset
## BlipConfig ## BlipConfig
[[autodoc]] BlipConfig [[autodoc]] BlipConfig
...@@ -57,12 +56,14 @@ The original code can be found [here](https://github.com/salesforce/BLIP). ...@@ -57,12 +56,14 @@ The original code can be found [here](https://github.com/salesforce/BLIP).
[[autodoc]] BlipProcessor [[autodoc]] BlipProcessor
## BlipImageProcessor ## BlipImageProcessor
[[autodoc]] BlipImageProcessor [[autodoc]] BlipImageProcessor
- preprocess - preprocess
<frameworkcontent>
<pt>
## BlipModel ## BlipModel
[[autodoc]] BlipModel [[autodoc]] BlipModel
...@@ -75,30 +76,29 @@ The original code can be found [here](https://github.com/salesforce/BLIP). ...@@ -75,30 +76,29 @@ The original code can be found [here](https://github.com/salesforce/BLIP).
[[autodoc]] BlipTextModel [[autodoc]] BlipTextModel
- forward - forward
## BlipVisionModel ## BlipVisionModel
[[autodoc]] BlipVisionModel [[autodoc]] BlipVisionModel
- forward - forward
## BlipForConditionalGeneration ## BlipForConditionalGeneration
[[autodoc]] BlipForConditionalGeneration [[autodoc]] BlipForConditionalGeneration
- forward - forward
## BlipForImageTextRetrieval ## BlipForImageTextRetrieval
[[autodoc]] BlipForImageTextRetrieval [[autodoc]] BlipForImageTextRetrieval
- forward - forward
## BlipForQuestionAnswering ## BlipForQuestionAnswering
[[autodoc]] BlipForQuestionAnswering [[autodoc]] BlipForQuestionAnswering
- forward - forward
</pt>
<tf>
## TFBlipModel ## TFBlipModel
[[autodoc]] TFBlipModel [[autodoc]] TFBlipModel
...@@ -111,26 +111,24 @@ The original code can be found [here](https://github.com/salesforce/BLIP). ...@@ -111,26 +111,24 @@ The original code can be found [here](https://github.com/salesforce/BLIP).
[[autodoc]] TFBlipTextModel [[autodoc]] TFBlipTextModel
- call - call
## TFBlipVisionModel ## TFBlipVisionModel
[[autodoc]] TFBlipVisionModel [[autodoc]] TFBlipVisionModel
- call - call
## TFBlipForConditionalGeneration ## TFBlipForConditionalGeneration
[[autodoc]] TFBlipForConditionalGeneration [[autodoc]] TFBlipForConditionalGeneration
- call - call
## TFBlipForImageTextRetrieval ## TFBlipForImageTextRetrieval
[[autodoc]] TFBlipForImageTextRetrieval [[autodoc]] TFBlipForImageTextRetrieval
- call - call
## TFBlipForQuestionAnswering ## TFBlipForQuestionAnswering
[[autodoc]] TFBlipForQuestionAnswering [[autodoc]] TFBlipForQuestionAnswering
- call - call
\ No newline at end of file </tf>
</frameworkcontent>
...@@ -56,16 +56,20 @@ See also: ...@@ -56,16 +56,20 @@ See also:
[[autodoc]] BloomConfig [[autodoc]] BloomConfig
- all - all
## BloomModel
[[autodoc]] BloomModel
- forward
## BloomTokenizerFast ## BloomTokenizerFast
[[autodoc]] BloomTokenizerFast [[autodoc]] BloomTokenizerFast
- all - all
<frameworkcontent>
<pt>
## BloomModel
[[autodoc]] BloomModel
- forward
## BloomForCausalLM ## BloomForCausalLM
[[autodoc]] BloomForCausalLM [[autodoc]] BloomForCausalLM
...@@ -86,6 +90,9 @@ See also: ...@@ -86,6 +90,9 @@ See also:
[[autodoc]] BloomForQuestionAnswering [[autodoc]] BloomForQuestionAnswering
- forward - forward
</pt>
<jax>
## FlaxBloomModel ## FlaxBloomModel
[[autodoc]] FlaxBloomModel [[autodoc]] FlaxBloomModel
...@@ -95,3 +102,8 @@ See also: ...@@ -95,3 +102,8 @@ See also:
[[autodoc]] FlaxBloomForCausalLM [[autodoc]] FlaxBloomForCausalLM
- __call__ - __call__
</jax>
</frameworkcontent>
...@@ -18,7 +18,7 @@ rendered properly in your Markdown viewer. ...@@ -18,7 +18,7 @@ rendered properly in your Markdown viewer.
<Tip warning={true}> <Tip warning={true}>
This model is in maintenance mode only, so we won't accept any new PRs changing its code. This model is in maintenance mode only, we do not accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0. If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.
You can do so by running the following command: `pip install -U transformers==4.30.0`. You can do so by running the following command: `pip install -U transformers==4.30.0`.
...@@ -43,13 +43,15 @@ hardware. It is also 7.9x faster on a CPU, as well as being better performing th ...@@ -43,13 +43,15 @@ hardware. It is also 7.9x faster on a CPU, as well as being better performing th
architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%, architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%,
absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.* absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.*
Tips: This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/alexa/bort/).
## Usage tips
- BORT's model architecture is based on BERT, so one can refer to [BERT's documentation page](bert) for the - BORT's model architecture is based on BERT, refer to [BERT's documentation page](bert) for the
model's API as well as usage examples. model's API reference as well as usage examples.
- BORT uses the RoBERTa tokenizer instead of the BERT tokenizer, so one can refer to [RoBERTa's documentation page](roberta) for the tokenizer's API as well as usage examples. - BORT uses the RoBERTa tokenizer instead of the BERT tokenizer, refer to [RoBERTa's documentation page](roberta) for the tokenizer's API reference as well as usage examples.
- BORT requires a specific fine-tuning algorithm, called [Agora](https://adewynter.github.io/notes/bort_algorithms_and_applications.html#fine-tuning-with-algebraic-topology) , - BORT requires a specific fine-tuning algorithm, called [Agora](https://adewynter.github.io/notes/bort_algorithms_and_applications.html#fine-tuning-with-algebraic-topology) ,
that is sadly not open-sourced yet. It would be very useful for the community, if someone tries to implement the that is sadly not open-sourced yet. It would be very useful for the community, if someone tries to implement the
algorithm to make BORT fine-tuning work. algorithm to make BORT fine-tuning work.
This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/alexa/bort/).
...@@ -37,7 +37,9 @@ alt="drawing" width="600"/> ...@@ -37,7 +37,9 @@ alt="drawing" width="600"/>
<small> BridgeTower architecture. Taken from the <a href="https://arxiv.org/abs/2206.08657">original paper.</a> </small> <small> BridgeTower architecture. Taken from the <a href="https://arxiv.org/abs/2206.08657">original paper.</a> </small>
## Usage This model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).
## Usage tips and examples
BridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers. BridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.
The goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder. The goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.
...@@ -116,9 +118,6 @@ The following example shows how to run masked language modeling using [`BridgeTo ...@@ -116,9 +118,6 @@ The following example shows how to run masked language modeling using [`BridgeTo
.a cat looking out of the window. .a cat looking out of the window.
``` ```
This model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).
Tips: Tips:
- This implementation of BridgeTower uses [`RobertaTokenizer`] to generate text embeddings and OpenAI's CLIP/ViT model to compute visual embeddings. - This implementation of BridgeTower uses [`RobertaTokenizer`] to generate text embeddings and OpenAI's CLIP/ViT model to compute visual embeddings.
......
...@@ -31,12 +31,13 @@ AMLM is a 2D version of TMLM. It randomly masks text tokens and predicts with th ...@@ -31,12 +31,13 @@ AMLM is a 2D version of TMLM. It randomly masks text tokens and predicts with th
BROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features. BROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.
The abstract from the paper is the following: The abstract from the paper is the following:
*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.* *Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*
Tips: This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).
## Usage tips and examples
- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height. - [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.
...@@ -78,9 +79,9 @@ def make_box_first_token_mask(bboxes, words, tokenizer, max_seq_length=512): ...@@ -78,9 +79,9 @@ def make_box_first_token_mask(bboxes, words, tokenizer, max_seq_length=512):
``` ```
- Demo scripts can be found [here](https://github.com/clovaai/bros). ## Resources
This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros). - Demo scripts can be found [here](https://github.com/clovaai/bros).
## BrosConfig ## BrosConfig
...@@ -102,13 +103,11 @@ This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The ...@@ -102,13 +103,11 @@ This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The
[[autodoc]] BrosForTokenClassification [[autodoc]] BrosForTokenClassification
- forward - forward
## BrosSpadeEEForTokenClassification ## BrosSpadeEEForTokenClassification
[[autodoc]] BrosSpadeEEForTokenClassification [[autodoc]] BrosSpadeEEForTokenClassification
- forward - forward
## BrosSpadeELForTokenClassification ## BrosSpadeELForTokenClassification
[[autodoc]] BrosSpadeELForTokenClassification [[autodoc]] BrosSpadeELForTokenClassification
......
...@@ -40,14 +40,18 @@ experiments.* ...@@ -40,14 +40,18 @@ experiments.*
This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://github.com/google-research/byt5). found [here](https://github.com/google-research/byt5).
ByT5's architecture is based on the T5v1.1 model, so one can refer to [T5v1.1's documentation page](t5v1.1). They <Tip>
ByT5's architecture is based on the T5v1.1 model, refer to [T5v1.1's documentation page](t5v1.1) for the API reference. They
only differ in how inputs should be prepared for the model, see the code examples below. only differ in how inputs should be prepared for the model, see the code examples below.
</Tip>
Since ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task Since ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task
fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix. fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.
### Example ## Usage example
ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer: ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:
......
...@@ -34,14 +34,16 @@ dependency parsing, named-entity recognition, and natural language inference. Ca ...@@ -34,14 +34,16 @@ dependency parsing, named-entity recognition, and natural language inference. Ca
for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and
downstream applications for French NLP.* downstream applications for French NLP.*
Tips: This model was contributed by [camembert](https://huggingface.co/camembert). The original code can be found [here](https://camembert-model.fr/).
- This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples <Tip>
as well as the information relative to the inputs and outputs.
This model was contributed by [camembert](https://huggingface.co/camembert). The original code can be found [here](https://camembert-model.fr/). This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well
as the information relative to the inputs and outputs.
## Documentation resources </Tip>
## Resources
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification) - [Token classification task guide](../tasks/token_classification)
...@@ -66,6 +68,9 @@ This model was contributed by [camembert](https://huggingface.co/camembert). The ...@@ -66,6 +68,9 @@ This model was contributed by [camembert](https://huggingface.co/camembert). The
[[autodoc]] CamembertTokenizerFast [[autodoc]] CamembertTokenizerFast
<frameworkcontent>
<pt>
## CamembertModel ## CamembertModel
[[autodoc]] CamembertModel [[autodoc]] CamembertModel
...@@ -94,6 +99,9 @@ This model was contributed by [camembert](https://huggingface.co/camembert). The ...@@ -94,6 +99,9 @@ This model was contributed by [camembert](https://huggingface.co/camembert). The
[[autodoc]] CamembertForQuestionAnswering [[autodoc]] CamembertForQuestionAnswering
</pt>
<tf>
## TFCamembertModel ## TFCamembertModel
[[autodoc]] TFCamembertModel [[autodoc]] TFCamembertModel
...@@ -121,3 +129,7 @@ This model was contributed by [camembert](https://huggingface.co/camembert). The ...@@ -121,3 +129,7 @@ This model was contributed by [camembert](https://huggingface.co/camembert). The
## TFCamembertForQuestionAnswering ## TFCamembertForQuestionAnswering
[[autodoc]] TFCamembertForQuestionAnswering [[autodoc]] TFCamembertForQuestionAnswering
</tf>
</frameworkcontent>
...@@ -37,7 +37,9 @@ To use its finer-grained input effectively and efficiently, CANINE combines down ...@@ -37,7 +37,9 @@ To use its finer-grained input effectively and efficiently, CANINE combines down
sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by
2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.* 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.*
Tips: This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/language/tree/master/language/canine).
## Usage tips
- CANINE uses no less than 3 Transformer encoders internally: 2 "shallow" encoders (which only consist of a single - CANINE uses no less than 3 Transformer encoders internally: 2 "shallow" encoders (which only consist of a single
layer) and 1 "deep" encoder (which is a regular BERT encoder). First, a "shallow" encoder is used to contextualize layer) and 1 "deep" encoder (which is a regular BERT encoder). First, a "shallow" encoder is used to contextualize
...@@ -50,19 +52,18 @@ Tips: ...@@ -50,19 +52,18 @@ Tips:
(which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of (which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of
tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The
details for this can be found in the paper. details for this can be found in the paper.
- Models:
Model checkpoints:
- [google/canine-c](https://huggingface.co/google/canine-c): Pre-trained with autoregressive character loss, - [google/canine-c](https://huggingface.co/google/canine-c): Pre-trained with autoregressive character loss,
12-layer, 768-hidden, 12-heads, 121M parameters (size ~500 MB). 12-layer, 768-hidden, 12-heads, 121M parameters (size ~500 MB).
- [google/canine-s](https://huggingface.co/google/canine-s): Pre-trained with subword loss, 12-layer, - [google/canine-s](https://huggingface.co/google/canine-s): Pre-trained with subword loss, 12-layer,
768-hidden, 12-heads, 121M parameters (size ~500 MB). 768-hidden, 12-heads, 121M parameters (size ~500 MB).
This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/language/tree/master/language/canine).
## Usage example
### Example CANINE works on raw characters, so it can be used **without a tokenizer**:
CANINE works on raw characters, so it can be used without a tokenizer:
```python ```python
>>> from transformers import CanineModel >>> from transformers import CanineModel
...@@ -96,17 +97,13 @@ sequences to the same length): ...@@ -96,17 +97,13 @@ sequences to the same length):
>>> sequence_output = outputs.last_hidden_state >>> sequence_output = outputs.last_hidden_state
``` ```
## Documentation resources ## Resources
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification) - [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering) - [Question answering task guide](../tasks/question_answering)
- [Multiple choice task guide](../tasks/multiple_choice) - [Multiple choice task guide](../tasks/multiple_choice)
## CANINE specific outputs
[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling
## CanineConfig ## CanineConfig
[[autodoc]] CanineConfig [[autodoc]] CanineConfig
...@@ -118,6 +115,10 @@ sequences to the same length): ...@@ -118,6 +115,10 @@ sequences to the same length):
- get_special_tokens_mask - get_special_tokens_mask
- create_token_type_ids_from_sequences - create_token_type_ids_from_sequences
## CANINE specific outputs
[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling
## CanineModel ## CanineModel
[[autodoc]] CanineModel [[autodoc]] CanineModel
......
...@@ -25,7 +25,9 @@ The abstract from the paper is the following: ...@@ -25,7 +25,9 @@ The abstract from the paper is the following:
*The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). Our codes, pretrained models, and demos have been released.* *The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). Our codes, pretrained models, and demos have been released.*
## Usage The Chinese-CLIP model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys).
## Usage example
The code snippet below shows how to compute image & text features and similarities: The code snippet below shows how to compute image & text features and similarities:
...@@ -59,15 +61,13 @@ The code snippet below shows how to compute image & text features and similariti ...@@ -59,15 +61,13 @@ The code snippet below shows how to compute image & text features and similariti
>>> probs = logits_per_image.softmax(dim=1) # probs: [[1.2686e-03, 5.4499e-02, 6.7968e-04, 9.4355e-01]] >>> probs = logits_per_image.softmax(dim=1) # probs: [[1.2686e-03, 5.4499e-02, 6.7968e-04, 9.4355e-01]]
``` ```
Currently, we release the following scales of pretrained Chinese-CLIP models at HF Model Hub: Currently, following scales of pretrained Chinese-CLIP models are available on 🤗 Hub:
- [OFA-Sys/chinese-clip-vit-base-patch16](https://huggingface.co/OFA-Sys/chinese-clip-vit-base-patch16) - [OFA-Sys/chinese-clip-vit-base-patch16](https://huggingface.co/OFA-Sys/chinese-clip-vit-base-patch16)
- [OFA-Sys/chinese-clip-vit-large-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14) - [OFA-Sys/chinese-clip-vit-large-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14)
- [OFA-Sys/chinese-clip-vit-large-patch14-336px](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14-336px) - [OFA-Sys/chinese-clip-vit-large-patch14-336px](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14-336px)
- [OFA-Sys/chinese-clip-vit-huge-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-huge-patch14) - [OFA-Sys/chinese-clip-vit-huge-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-huge-patch14)
The Chinese-CLIP model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys).
## ChineseCLIPConfig ## ChineseCLIPConfig
[[autodoc]] ChineseCLIPConfig [[autodoc]] ChineseCLIPConfig
......
...@@ -30,7 +30,6 @@ The abstract from the paper is the following: ...@@ -30,7 +30,6 @@ The abstract from the paper is the following:
This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArtZucker) . This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArtZucker) .
The original code can be found [here](https://github.com/LAION-AI/Clap). The original code can be found [here](https://github.com/LAION-AI/Clap).
## ClapConfig ## ClapConfig
[[autodoc]] ClapConfig [[autodoc]] ClapConfig
...@@ -78,4 +77,3 @@ The original code can be found [here](https://github.com/LAION-AI/Clap). ...@@ -78,4 +77,3 @@ The original code can be found [here](https://github.com/LAION-AI/Clap).
[[autodoc]] ClapAudioModelWithProjection [[autodoc]] ClapAudioModelWithProjection
- forward - forward
...@@ -40,7 +40,9 @@ for any dataset specific training. For instance, we match the accuracy of the or ...@@ -40,7 +40,9 @@ for any dataset specific training. For instance, we match the accuracy of the or
without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained
model weights at this https URL.* model weights at this https URL.*
## Usage This model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/openai/CLIP).
## Usage tips and example
CLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image CLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image
classification. CLIP uses a ViT like transformer to get visual features and a causal language model to get the text classification. CLIP uses a ViT like transformer to get visual features and a causal language model to get the text
...@@ -77,8 +79,6 @@ encode the text and prepare the images. The following example shows how to get t ...@@ -77,8 +79,6 @@ encode the text and prepare the images. The following example shows how to get t
>>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
``` ```
This model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/openai/CLIP).
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CLIP. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CLIP.
...@@ -142,6 +142,9 @@ The resource should ideally demonstrate something new instead of duplicating an ...@@ -142,6 +142,9 @@ The resource should ideally demonstrate something new instead of duplicating an
[[autodoc]] CLIPProcessor [[autodoc]] CLIPProcessor
<frameworkcontent>
<pt>
## CLIPModel ## CLIPModel
[[autodoc]] CLIPModel [[autodoc]] CLIPModel
...@@ -164,12 +167,14 @@ The resource should ideally demonstrate something new instead of duplicating an ...@@ -164,12 +167,14 @@ The resource should ideally demonstrate something new instead of duplicating an
[[autodoc]] CLIPVisionModelWithProjection [[autodoc]] CLIPVisionModelWithProjection
- forward - forward
## CLIPVisionModel ## CLIPVisionModel
[[autodoc]] CLIPVisionModel [[autodoc]] CLIPVisionModel
- forward - forward
</pt>
<tf>
## TFCLIPModel ## TFCLIPModel
[[autodoc]] TFCLIPModel [[autodoc]] TFCLIPModel
...@@ -187,6 +192,9 @@ The resource should ideally demonstrate something new instead of duplicating an ...@@ -187,6 +192,9 @@ The resource should ideally demonstrate something new instead of duplicating an
[[autodoc]] TFCLIPVisionModel [[autodoc]] TFCLIPVisionModel
- call - call
</tf>
<jax>
## FlaxCLIPModel ## FlaxCLIPModel
[[autodoc]] FlaxCLIPModel [[autodoc]] FlaxCLIPModel
...@@ -208,3 +216,6 @@ The resource should ideally demonstrate something new instead of duplicating an ...@@ -208,3 +216,6 @@ The resource should ideally demonstrate something new instead of duplicating an
[[autodoc]] FlaxCLIPVisionModel [[autodoc]] FlaxCLIPVisionModel
- __call__ - __call__
</jax>
</frameworkcontent>
...@@ -41,13 +41,6 @@ to any binary segmentation task where a text or image query ...@@ -41,13 +41,6 @@ to any binary segmentation task where a text or image query
can be formulated. Finally, we find our system to adapt well can be formulated. Finally, we find our system to adapt well
to generalized queries involving affordances or properties* to generalized queries involving affordances or properties*
Tips:
- [`CLIPSegForImageSegmentation`] adds a decoder on top of [`CLIPSegModel`]. The latter is identical to [`CLIPModel`].
- [`CLIPSegForImageSegmentation`] can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text
(provided to the model as `input_ids`) or an image (provided to the model as `conditional_pixel_values`). One can also provide custom
conditional embeddings (provided to the model as `conditional_embeddings`).
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png" <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png"
alt="drawing" width="600"/> alt="drawing" width="600"/>
...@@ -56,6 +49,13 @@ alt="drawing" width="600"/> ...@@ -56,6 +49,13 @@ alt="drawing" width="600"/>
This model was contributed by [nielsr](https://huggingface.co/nielsr). This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/timojl/clipseg). The original code can be found [here](https://github.com/timojl/clipseg).
## Usage tips
- [`CLIPSegForImageSegmentation`] adds a decoder on top of [`CLIPSegModel`]. The latter is identical to [`CLIPModel`].
- [`CLIPSegForImageSegmentation`] can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text
(provided to the model as `input_ids`) or an image (provided to the model as `conditional_pixel_values`). One can also provide custom
conditional embeddings (provided to the model as `conditional_embeddings`).
## Resources ## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CLIPSeg. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource. A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CLIPSeg. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
......
...@@ -24,7 +24,11 @@ The abstract from the paper is the following: ...@@ -24,7 +24,11 @@ The abstract from the paper is the following:
*We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.* *We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*
Check out all Code Llama models [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [codellama org](https://huggingface.co/codellama). Check out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [codellama org](https://huggingface.co/codellama).
This model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).
## Usage tips and examples
<Tip warning={true}> <Tip warning={true}>
...@@ -38,21 +42,22 @@ As mentioned above, the `dtype` of the storage weights is mostly irrelevant unle ...@@ -38,21 +42,22 @@ As mentioned above, the `dtype` of the storage weights is mostly irrelevant unle
</Tip> </Tip>
Tips:
- These models have the same architecture as the `Llama2` models Tips:
- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled. - The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.
- The model conversion script is the same as for the `Llama2` family: - The model conversion script is the same as for the `Llama2` family:
Here is a sample usage Here is a sample usage:
```bash ```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \ python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
``` ```
Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).
- After conversion, the model and tokenizer can be loaded via: After conversion, the model and tokenizer can be loaded via:
```python ```python
>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer >>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer
...@@ -95,9 +100,13 @@ If you only want the infilled part: ...@@ -95,9 +100,13 @@ If you only want the infilled part:
Under the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug. To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value. Under the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug. To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value.
- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string. The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string.
This model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama). <Tip>
Code Llama has the same architecture as the `Llama2` models, refer to [Llama2's documentation page](llama2) for the API reference.
Find Code Llama tokenizer reference below.
</Tip>
## CodeLlamaTokenizer ## CodeLlamaTokenizer
......
...@@ -40,7 +40,7 @@ The original code can be found [here](https://github.com/salesforce/codegen). ...@@ -40,7 +40,7 @@ The original code can be found [here](https://github.com/salesforce/codegen).
* `mono`: Initialized with `multi`, then further pre-trained on Python data * `mono`: Initialized with `multi`, then further pre-trained on Python data
* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python. * For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python.
## How to use ## Usage example
```python ```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer >>> from transformers import AutoModelForCausalLM, AutoTokenizer
...@@ -60,7 +60,7 @@ def hello_world(): ...@@ -60,7 +60,7 @@ def hello_world():
hello_world() hello_world()
``` ```
## Documentation resources ## Resources
- [Causal language modeling task guide](../tasks/language_modeling) - [Causal language modeling task guide](../tasks/language_modeling)
......
...@@ -31,7 +31,7 @@ alt="drawing" width="600"/> ...@@ -31,7 +31,7 @@ alt="drawing" width="600"/>
This model was contributed by [DepuMeng](https://huggingface.co/DepuMeng). The original code can be found [here](https://github.com/Atten4Vis/ConditionalDETR). This model was contributed by [DepuMeng](https://huggingface.co/DepuMeng). The original code can be found [here](https://github.com/Atten4Vis/ConditionalDETR).
## Documentation resources ## Resources
- [Object detection task guide](../tasks/object_detection) - [Object detection task guide](../tasks/object_detection)
......
...@@ -44,12 +44,14 @@ ConvBERT significantly outperforms BERT and its variants in various downstream t ...@@ -44,12 +44,14 @@ ConvBERT significantly outperforms BERT and its variants in various downstream t
fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while
using less than 1/4 training cost. Code and pre-trained models will be released.* using less than 1/4 training cost. Code and pre-trained models will be released.*
ConvBERT training tips are similar to those of BERT.
This model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found This model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found
here: https://github.com/yitu-opensource/ConvBert here: https://github.com/yitu-opensource/ConvBert
## Documentation resources ## Usage tips
ConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert).
## Resources
- [Text classification task guide](../tasks/sequence_classification) - [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification) - [Token classification task guide](../tasks/token_classification)
...@@ -73,6 +75,9 @@ here: https://github.com/yitu-opensource/ConvBert ...@@ -73,6 +75,9 @@ here: https://github.com/yitu-opensource/ConvBert
[[autodoc]] ConvBertTokenizerFast [[autodoc]] ConvBertTokenizerFast
<frameworkcontent>
<pt>
## ConvBertModel ## ConvBertModel
[[autodoc]] ConvBertModel [[autodoc]] ConvBertModel
...@@ -103,6 +108,9 @@ here: https://github.com/yitu-opensource/ConvBert ...@@ -103,6 +108,9 @@ here: https://github.com/yitu-opensource/ConvBert
[[autodoc]] ConvBertForQuestionAnswering [[autodoc]] ConvBertForQuestionAnswering
- forward - forward
</pt>
<tf>
## TFConvBertModel ## TFConvBertModel
[[autodoc]] TFConvBertModel [[autodoc]] TFConvBertModel
...@@ -132,3 +140,6 @@ here: https://github.com/yitu-opensource/ConvBert ...@@ -132,3 +140,6 @@ here: https://github.com/yitu-opensource/ConvBert
[[autodoc]] TFConvBertForQuestionAnswering [[autodoc]] TFConvBertForQuestionAnswering
- call - call
</tf>
</frameworkcontent>
...@@ -32,10 +32,6 @@ of a vision Transformer, and discover several key components that contribute to ...@@ -32,10 +32,6 @@ of a vision Transformer, and discover several key components that contribute to
dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy
and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.* and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*
Tips:
- See the code examples below each model regarding usage.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg" <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg"
alt="drawing" width="600"/> alt="drawing" width="600"/>
...@@ -68,6 +64,9 @@ If you're interested in submitting a resource to be included here, please feel f ...@@ -68,6 +64,9 @@ If you're interested in submitting a resource to be included here, please feel f
[[autodoc]] ConvNextImageProcessor [[autodoc]] ConvNextImageProcessor
- preprocess - preprocess
<frameworkcontent>
<pt>
## ConvNextModel ## ConvNextModel
[[autodoc]] ConvNextModel [[autodoc]] ConvNextModel
...@@ -78,14 +77,18 @@ If you're interested in submitting a resource to be included here, please feel f ...@@ -78,14 +77,18 @@ If you're interested in submitting a resource to be included here, please feel f
[[autodoc]] ConvNextForImageClassification [[autodoc]] ConvNextForImageClassification
- forward - forward
</pt>
<tf>
## TFConvNextModel ## TFConvNextModel
[[autodoc]] TFConvNextModel [[autodoc]] TFConvNextModel
- call - call
## TFConvNextForImageClassification ## TFConvNextForImageClassification
[[autodoc]] TFConvNextForImageClassification [[autodoc]] TFConvNextForImageClassification
- call - call
</tf>
</frameworkcontent>
\ No newline at end of file
...@@ -25,10 +25,6 @@ The abstract from the paper is the following: ...@@ -25,10 +25,6 @@ The abstract from the paper is the following:
*Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.* *Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.*
Tips:
- See the code examples below each model regarding usage.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnextv2_architecture.png" <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnextv2_architecture.png"
alt="drawing" width="600"/> alt="drawing" width="600"/>
......
...@@ -37,7 +37,14 @@ NLP tasks in the settings of few-shot (even zero-shot) learning.* ...@@ -37,7 +37,14 @@ NLP tasks in the settings of few-shot (even zero-shot) learning.*
This model was contributed by [canwenxu](https://huggingface.co/canwenxu). The original implementation can be found This model was contributed by [canwenxu](https://huggingface.co/canwenxu). The original implementation can be found
here: https://github.com/TsinghuaAI/CPM-Generate here: https://github.com/TsinghuaAI/CPM-Generate
Note: We only have a tokenizer here, since the model architecture is the same as GPT-2.
<Tip>
CPM's architecture is the same as GPT-2, except for tokenization method. Refer to [GPT-2 documentation](gpt2) for
API reference information.
</Tip>
## CpmTokenizer ## CpmTokenizer
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment