Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
22b39e11
Unverified
Commit
22b39e11
authored
Aug 15, 2024
by
Kameshwara Pavan Kumar Mantha
Committed by
GitHub
Aug 14, 2024
Browse files
llama_index serving integration documentation (#6973)
Co-authored-by:
pavanmantha
<
pavan.mantha@thevaslabs.io
>
parent
f55a9aea
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
28 additions
and
0 deletions
+28
-0
docs/source/serving/integrations.rst
docs/source/serving/integrations.rst
+1
-0
docs/source/serving/serving_with_llamaindex.rst
docs/source/serving/serving_with_llamaindex.rst
+27
-0
No files found.
docs/source/serving/integrations.rst
View file @
22b39e11
...
@@ -12,3 +12,4 @@ Integrations
...
@@ -12,3 +12,4 @@ Integrations
deploying_with_lws
deploying_with_lws
deploying_with_dstack
deploying_with_dstack
serving_with_langchain
serving_with_langchain
serving_with_llamaindex
docs/source/serving/serving_with_llamaindex.rst
0 → 100644
View file @
22b39e11
.. _run_on_llamaindex:
Serving with llama_index
============================
vLLM is also available via `llama_index <https://github.com/run-llama/llama_index>`_ .
To install llamaindex, run
.. code-block:: console
$ pip install llama-index-llms-vllm -q
To run inference on a single or multiple GPUs, use ``Vllm`` class from ``llamaindex``.
.. code-block:: python
from llama_index.llms.vllm import Vllm
llm = Vllm(
model="microsoft/Orca-2-7b",
tensor_parallel_size=4,
max_new_tokens=100,
vllm_kwargs={"swap_space": 1, "gpu_memory_utilization": 0.5},
)
Please refer to this `Tutorial <https://docs.llamaindex.ai/en/latest/examples/llm/vllm/>`_ for more details.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment