Other architectures are supported on a best effort basis using:
Other architectures are supported on a best effort basis using:
...
@@ -86,7 +87,7 @@ The easiest way of getting started is using the official Docker container:
...
@@ -86,7 +87,7 @@ The easiest way of getting started is using the official Docker container:
model=tiiuae/falcon-7b-instruct
model=tiiuae/falcon-7b-instruct
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all --shm-size 1g -p 8080:80 -v$volume:/data ghcr.io/huggingface/text-generation-inference:1.0.2--model-id$model
docker run --gpus all --shm-size 1g -p 8080:80 -v$volume:/data ghcr.io/huggingface/text-generation-inference:1.0.3--model-id$model
```
```
**Note:** To use GPUs, you need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). We also recommend using NVIDIA drivers with CUDA version 11.8 or higher. For running the Docker container on a machine with no GPUs or CUDA support, it is enough to remove the `--gpus all` flag and add `--disable-custom-kernels`, please note CPU is not the intended platform for this project, so performance might be subpar.
**Note:** To use GPUs, you need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). We also recommend using NVIDIA drivers with CUDA version 11.8 or higher. For running the Docker container on a machine with no GPUs or CUDA support, it is enough to remove the `--gpus all` flag and add `--disable-custom-kernels`, please note CPU is not the intended platform for this project, so performance might be subpar.
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
token=<your cli READ token>
token=<your cli READ token>
docker run --gpus all --shm-size 1g -eHUGGING_FACE_HUB_TOKEN=$token-p 8080:80 -v$volume:/data ghcr.io/huggingface/text-generation-inference:1.0.2--model-id$model
docker run --gpus all --shm-size 1g -eHUGGING_FACE_HUB_TOKEN=$token-p 8080:80 -v$volume:/data ghcr.io/huggingface/text-generation-inference:1.0.3--model-id$model
@@ -8,7 +8,7 @@ Let's say you want to deploy [Falcon-7B Instruct](https://huggingface.co/tiiuae/
...
@@ -8,7 +8,7 @@ Let's say you want to deploy [Falcon-7B Instruct](https://huggingface.co/tiiuae/
model=tiiuae/falcon-7b-instruct
model=tiiuae/falcon-7b-instruct
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all --shm-size 1g -p 8080:80 -v$volume:/data ghcr.io/huggingface/text-generation-inference:1.0.2--model-id$model
docker run --gpus all --shm-size 1g -p 8080:80 -v$volume:/data ghcr.io/huggingface/text-generation-inference:1.0.3--model-id$model
```
```
<Tipwarning={true}>
<Tipwarning={true}>
...
@@ -85,7 +85,7 @@ curl 127.0.0.1:8080/generate \
...
@@ -85,7 +85,7 @@ curl 127.0.0.1:8080/generate \
To see all possible deploy flags and options, you can use the `--help` flag. It's possible to configure the number of shards, quantization, generation parameters, and more.
To see all possible deploy flags and options, you can use the `--help` flag. It's possible to configure the number of shards, quantization, generation parameters, and more.
```bash
```bash
docker run ghcr.io/huggingface/text-generation-inference:1.0.2--help
docker run ghcr.io/huggingface/text-generation-inference:1.0.3--help