Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
norm
vllm
Commits
37c1e3c2
Unverified
Commit
37c1e3c2
authored
Nov 19, 2023
by
Simon Mo
Committed by
GitHub
Nov 19, 2023
Browse files
Documentation about official docker image (#1709)
parent
06e9ebeb
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
16 additions
and
2 deletions
+16
-2
docs/source/serving/deploying_with_docker.rst
docs/source/serving/deploying_with_docker.rst
+16
-2
No files found.
docs/source/serving/deploying_with_docker.rst
View file @
37c1e3c2
...
...
@@ -3,11 +3,25 @@
Deploying with Docker
============================
vLLM offers official docker image for deployment.
The image can be used to run OpenAI compatible server.
The image is available on Docker Hub as `vllm/vllm-openai <https://hub.docker.com/r/vllm/vllm-openai/tags>`_.
... code-block:: console
$ docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1
You can build and run vLLM from source via the provided dockerfile. To build vLLM:
.. code-block:: console
$ DOCKER_BUILDKIT=1 docker build . --target vllm --tag vllm --build-arg max_jobs=8
$ DOCKER_BUILDKIT=1 docker build . --target vllm
-openai
--tag vllm
/vllm-openai
--build-arg max_jobs=8
To run vLLM:
...
...
@@ -17,5 +31,5 @@ To run vLLM:
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
vllm <args...>
vllm
/vllm-openai
<args...>
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment