Unverified Commit 36f53035 authored by Charles Frye's avatar Charles Frye Committed by GitHub
Browse files

[Docs] Add Modal to deployment frameworks (#11907)

parent 9a228348
...@@ -2,6 +2,6 @@ ...@@ -2,6 +2,6 @@
# BentoML # BentoML
[BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes. [BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-compliant image and deploy it on Kubernetes.
For details, see the tutorial [vLLM inference in the BentoML documentation](https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html). For details, see the tutorial [vLLM inference in the BentoML documentation](https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html).
...@@ -8,6 +8,7 @@ cerebrium ...@@ -8,6 +8,7 @@ cerebrium
dstack dstack
helm helm
lws lws
modal
skypilot skypilot
triton triton
``` ```
(deployment-modal)=
# Modal
vLLM can be run on cloud GPUs with [Modal](https://modal.com), a serverless computing platform designed for fast auto-scaling.
For details on how to deploy vLLM on Modal, see [this tutorial in the Modal documentation](https://modal.com/docs/examples/vllm_inference).
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment