Get started docs (#15098)

* clean commit of changes

* apply review feedback, make edits

* fix backticks, minor formatting

* 🖍 make fixup and minor edits

* 🖍 fix # in header

* 📝 update code sample without from_pt

* 📝 final review

docs/source/index.mdx

@@ -12,25 +12,18 @@ specific language governing permissions and limitations under the License.
 
 # 🤗 Transformers
 
-State-of-the-art Machine Learning for Jax, Pytorch and TensorFlow
+State-of-the-art Machine Learning for PyTorch, TensorFlow and JAX.
 
-🤗 Transformers (formerly known as _pytorch-transformers_ and _pytorch-pretrained-bert_) provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.
+🤗 Transformers provides APIs to easily download and train state-of-the-art pretrained models. Using pretrained models can reduce your compute costs, carbon footprint, and save you time from training a model from scratch. The models can be used across different modalities such as:
 
-These models can applied on:
+* 📝 Text: text classification, information extraction, question answering, summarization, translation, and text generation in over 100 languages.
+* 🖼️ Images: image classification, object detection, and segmentation.
+* 🗣️ Audio: speech recognition and audio classification.
+* 🐙 Multimodal: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.
 
-* 📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.
-* 🖼️ Images, for tasks like image classification, object detection, and segmentation.
-* 🗣️ Audio, for tasks like speech recognition and audio classification.
+Our library supports seamless integration between three of the most popular deep learning libraries: [PyTorch](https://pytorch.org/), [TensorFlow](https://www.tensorflow.org/) and [JAX](https://jax.readthedocs.io/en/latest/). Train your model in three lines of code in one framework, and load it for inference with another.
 
-Transformer models can also perform tasks on **several modalities combined**, such as table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.
-
-🤗 Transformers provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets and then share them with the community on our [model hub](https://huggingface.co/models). At the same time, each python module defining an architecture is fully standalone and can be modified to enable quick research experiments.
-
-🤗 Transformers is backed by the three most popular deep learning libraries — [Jax](https://jax.readthedocs.io/en/latest/), [PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/) — with a seamless integration between them. It's straightforward to train your models with one before loading them for inference with the other.
-
-This is the documentation of our repository [transformers](https://github.com/huggingface/transformers). You can
-also follow our [online course](https://huggingface.co/course) that teaches how to use this library, as well as the
-other libraries developed by Hugging Face and the Hub.
+Each 🤗 Transformers architecture is defined in a standalone Python module so they can be easily customized for research and experiments.
 
 ## If you are looking for custom support from the Hugging Face team
 
@@ -38,35 +31,6 @@ other libraries developed by Hugging Face and the Hub.
 <img alt="HuggingFace Expert Acceleration Program" src="https://huggingface.co/front/thumbnails/support.png" style="max-width: 600px; border: 1px solid #eee; border-radius: 4px; box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.05);">
 </a><br>
 
-## Features
-
-1. Easy-to-use state-of-the-art models:
-    - High performance on natural language understanding & generation, computer vision, and audio tasks.
-    - Low barrier to entry for educators and practitioners.
-    - Few user-facing abstractions with just three classes to learn.
-    - A unified API for using all our pretrained models.
-
-1. Lower compute costs, smaller carbon footprint:
-    - Researchers can share trained models instead of always retraining.
-    - Practitioners can reduce compute time and production costs.
-    - Dozens of architectures with over 20,000 pretrained models, some in more than 100 languages.
-
-1. Choose the right framework for every part of a model's lifetime:
-    - Train state-of-the-art models in 3 lines of code.
-    - Move a single model between TF2.0/PyTorch/JAX frameworks at will.
-    - Seamlessly pick the right framework for training, evaluation and production.
-
-1. Easily customize a model or an example to your needs:
-    - We provide examples for each architecture to reproduce the results published by its original authors.
-    - Model internals are exposed as consistently as possible.
-    - Model files can be used independently of the library for quick experiments.
-
-[All the model checkpoints](https://huggingface.co/models) are seamlessly integrated from the huggingface.co [model
-hub](https://huggingface.co) where they are uploaded directly by [users](https://huggingface.co/users) and
-[organizations](https://huggingface.co/organizations).
-
-Current number of checkpoints: <img src="https://img.shields.io/endpoint?url=https://huggingface.co/api/shields/models&color=brightgreen">
-
 ## Contents
 
 The documentation is organized in five parts:

docs/source/installation.mdx

 <!---
-Copyright 2020 The HuggingFace Team. All rights reserved.
+Copyright 2022 The HuggingFace Team. All rights reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -16,183 +16,220 @@ limitations under the License.
 
 # Installation
 
-🤗 Transformers is tested on Python 3.6+, and PyTorch 1.1.0+ or TensorFlow 2.0+.
+Install 🤗 Transformers for whichever deep learning library you're working with, setup your cache, and optionally configure 🤗 Transformers to run offline.
 
-You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're
-unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). Create a virtual environment with the version of Python you're going
-to use and activate it.
+🤗 Transformers is tested on Python 3.6+, PyTorch 1.1.0+, TensorFlow 2.0+, and Flax. Follow the installation instructions below for the deep learning library you are using:
 
-Now, if you want to use 🤗 Transformers, you can install it with pip. If you'd like to play with the examples, you
-must install it from source.
+* [PyTorch](https://pytorch.org/get-started/locally/) installation instructions.
+* [TensorFlow 2.0](https://www.tensorflow.org/install/pip) installation instructions.
+* [Flax](https://flax.readthedocs.io/en/latest/) installation instructions.
 
-## Installation with pip
+## Install with pip
 
-First you need to install one of, or both, TensorFlow 2.0 and PyTorch.
-Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available),
-[PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) and/or
-[Flax installation page](https://github.com/google/flax#quick-install)
-regarding the specific install command for your platform.
+You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, take a look at this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.
 
-When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:
+Start by creating a virtual environment in your project directory:
+
+```bash
+python -m venv .env
+```
+
+Activate the virtual environment:
+
+```bash
+source .env/bin/activate
+```
+
+Now you're ready to install 🤗 Transformers with the following command:
 
 ```bash
 pip install transformers
 ```
 
-Alternatively, for CPU-support only, you can install 🤗 Transformers and PyTorch in one line with:
+For CPU-support only, you can conveniently install 🤗 Transformers and a deep learning library in one line. For example, install 🤗 Transformers and PyTorch with:
 
 ```bash
 pip install transformers[torch]
 ```
 
-or 🤗 Transformers and TensorFlow 2.0 in one line with:
+🤗 Transformers and TensorFlow 2.0:
 
 ```bash
 pip install transformers[tf-cpu]
 ```
 
-or 🤗 Transformers and Flax in one line with:
+🤗 Transformers and Flax:
 
 ```bash
 pip install transformers[flax]
 ```
 
-To check 🤗 Transformers is properly installed, run the following command:
+Finally, check if 🤗 Transformers has been properly installed by running the following command. It will download a pretrained model:
 
 ```bash
 python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))"
 ```
 
-It should download a pretrained model then print something like
+Then print out the label and score:
 
 ```bash
 [{'label': 'POSITIVE', 'score': 0.9998704791069031}]
 ```
 
-(Note that TensorFlow will print additional stuff before that last statement.)
-
-## Installing from source
+## Install from source
 
-Here is how to quickly install `transformers` from source:
+Install 🤗 Transformers from source with the following command:
 
 ```bash
 pip install git+https://github.com/huggingface/transformers
 ```
 
-Note that this will install not the latest released version, but the bleeding edge `master` version, which you may want to use in case a bug has been fixed since the last official release and a new release hasn't been yet rolled out.
+This command installs the bleeding edge `master` version rather than the latest `stable` version. The `master` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `master` version may not always be stable. We strive to keep the `master` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!
 
-While we strive to keep `master` operational at all times, if you notice some issues, they usually get fixed within a few hours or a day and you're more than welcome to help us detect any problems by opening an [Issue](https://github.com/huggingface/transformers/issues) and this way, things will get fixed even sooner.
-
-Again, you can run:
+Check if 🤗 Transformers has been properly installed by running the following command:
 
 ```bash
-python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I hate you'))"
+python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))"
 ```
 
-to check 🤗 Transformers is properly installed.
-
 ## Editable install
 
-If you want to constantly use the bleeding edge `master` version of the source code, or if you want to contribute to the library and need to test the changes in the code you're making, you will need an editable install. This is done by cloning the repository and installing with the following commands:
+You will need an editable install if you'd like to:
+
+* Use the `master` version of the source code.
+* Contribute to 🤗 Transformers and need to test changes in the code.
 
-``` bash
+Clone the repository and install 🤗 Transformers with the following commands:
+
+```bash
 git clone https://github.com/huggingface/transformers.git
 cd transformers
 pip install -e .
 ```
 
-This command performs a magical link between the folder you cloned the repository to and your python library paths, and it'll look inside this folder in addition to the normal library-wide paths. So if normally your python packages get installed into:
-```
-~/anaconda3/envs/main/lib/python3.7/site-packages/
-```
-now this editable install will reside where you clone the folder to, e.g. `~/transformers/` and python will search it too.
+These commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.
 
-Do note that you have to keep that `transformers` folder around and not delete it to continue using the  `transformers` library.
+<Tip warning={true}>
 
-Now, let's get to the real benefit of this installation approach. Say, you saw some new feature has been just committed into `master`. If you have already performed all the steps above, to update your transformers to include all the latest commits, all you need to do is to `cd` into that cloned repository folder and update the clone to the latest version:
+You must keep the `transformers` folder if you want to keep using the library.
 
-```
+</Tip>
+
+Now you can easily update your clone to the latest version of 🤗 Transformers with the following command:
+
+```bash
 cd ~/transformers/
 git pull
 ```
 
-There is nothing else to do. Your python environment will find the bleeding edge version of `transformers` on the next run.
-
+Your Python environment will find the `master` version of 🤗 Transformers on the next run.
 
-## With conda
+## Install with conda
 
-Since Transformers version v4.0.0, we now have a conda channel: `huggingface`.
+Install from the conda channel `huggingface`:
 
-🤗 Transformers can be installed using conda as follows:
-
-```
+```bash
 conda install -c huggingface transformers
 ```
 
-Follow the installation pages of TensorFlow, PyTorch or Flax to see how to install them with conda.
+## Cache setup
 
-## Caching models
+Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/transformers/`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\transformers`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:
 
-This library provides pretrained models that will be downloaded and cached locally. Unless you specify a location with
-`cache_dir=...` when you use methods like `from_pretrained`, these models will automatically be downloaded in the
-folder given by the shell environment variable ``TRANSFORMERS_CACHE``. The default value for it will be the Hugging
-Face cache home followed by ``/transformers/``. This is (by order of priority):
+1. Shell environment variable (default): `TRANSFORMERS_CACHE`.
+2. Shell environment variable: `HF_HOME` + `transformers/`.
+3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface/transformers`.
 
-  * shell environment variable ``HF_HOME``
-  * shell environment variable ``XDG_CACHE_HOME`` + ``/huggingface/``
-  * default: ``~/.cache/huggingface/``
+<Tip>
 
-So if you don't have any specific environment variable set, the cache directory will be at
-``~/.cache/huggingface/transformers/``.
+🤗 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.
 
-**Note:** If you have set a shell environment variable for one of the predecessors of this library
-(``PYTORCH_TRANSFORMERS_CACHE`` or ``PYTORCH_PRETRAINED_BERT_CACHE``), those will be used if there is no shell
-environment variable for ``TRANSFORMERS_CACHE``.
+</Tip>
 
-### Offline mode
+## Offline mode
 
-It's possible to run 🤗 Transformers in a firewalled or a no-network environment.
+🤗 Transformers is able to run in a firewalled or offline environment by only using local files. Set the environment variable `TRANSFORMERS_OFFLINE=1` to enable this behavior.
 
-Setting environment variable `TRANSFORMERS_OFFLINE=1` will tell 🤗 Transformers to use local files only and will not try to look things up.
+<Tip>
 
-Most likely you may want to couple this with `HF_DATASETS_OFFLINE=1` that performs the same for 🤗 Datasets if you're using the latter.
+Add [🤗 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow by setting the environment variable `HF_DATASETS_OFFLINE=1`.
 
-Here is an example of how this can be used on a filesystem that is shared between a normally networked and a firewalled to the external world instances.
+</Tip>
 
-On the instance with the normal network run your program which will download and cache models (and optionally datasets if you use 🤗 Datasets). For example:
+For example, you would typically run a program on a normal network firewalled to external instances with the following command:
 
-```
+```bash
 python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --dataset_name wmt16 --dataset_config ro-en ...
 ```
 
-and then with the same filesystem you can now run the same program on a firewalled instance:
-```
+Run this same program in an offline instance with:
+
+```bash
 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
 python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --dataset_name wmt16 --dataset_config ro-en ...
 ```
-and it should succeed without any hanging waiting to timeout.
 
-#### Fetching models and tokenizers to use offline
+The script should now run without hanging or waiting to timeout because it knows it should only look for local files.
+
+### Fetch models and tokenizers to use offline
+
+Another option for using 🤗 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:
+
+* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the ↓ icon.
+
+    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)
+
+* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:
+
+    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:
 
-When running a script the first time like mentioned above, the downloaded files will be cached for future reuse. 
-However, it is also possible to download files and point to their local path instead.
+    ```py
+    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
 
-Downloading files can be done through the Web Interface by clicking on the "Download" button, but it can also be handled
-programmatically using the `huggingface_hub` library that is a dependency to `transformers`:
+    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
+    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
+    ```
 
-- Using `snapshot_download` to download an entire repository
-- Using `hf_hub_download` to download a specific file
+    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:
 
-See the reference for these methods in the huggingface_hub
-[documentation](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub).
+    ```py
+    >>> tokenizer.save_pretrained("./your/path/bigscience_t0")
+    >>> model.save_pretrained("./your/path/bigscience_t0")
+    ```
 
-## Do you want to run a Transformer model on a mobile device?
+    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:
+
+    ```py
+    >>> tokenizer = AutoTokenizer.from_pretrained("./your/path/bigscience_t0")
+    >>> model = AutoModel.from_pretrained("./your/path/bigscience_t0")
+    ```
+
+* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:
+
+    1. Install the `huggingface_hub` library in your virtual environment:
+
+    ```bash
+    python -m pip install huggingface_hub
+    ```
+
+    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path. For example, the following command downloads the `config.json` file from the [T0](https://huggingface.co/bigscience/T0_3B) model to your desired path:
+
+    ```py
+    >>> from huggingface_hub import hf_hub_download
+
+    >>> hf_hub_download(repo_id="bigscience/T0_3B", filename="config.json", cache_dir="./your/path/bigscience_t0")
+    ```
+
+Once your file is downloaded and locally cached, specify it's local path to load and use it:
+
+```py
+>>> from transformers import AutoConfig
+
+>>> config = AutoConfig.from_pretrained("./your/path/bigscience_t0/config.json")
+```
 
-You should check out our [swift-coreml-transformers](https://github.com/huggingface/swift-coreml-transformers) repo.
+<Tip>
 
-It contains a set of tools to convert PyTorch or TensorFlow 2.0 trained Transformer models (currently contains `GPT-2`,
-`DistilGPT-2`, `BERT`, and `DistilBERT`) to CoreML models that run on iOS devices.
+See the [How to download files from the Hub](https://huggingface.co/docs/hub/how-to-downstream) section for more details on downloading files stored on the Hub.
 
-At some point in the future, you'll be able to seamlessly move from pretraining or fine-tuning models in PyTorch or
-TensorFlow 2.0 to productizing them in CoreML, or prototype a model or an app in CoreML then research its
-hyperparameters or architecture from PyTorch or TensorFlow 2.0. Super exciting!
+</Tip>
\ No newline at end of file