Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
kecinstone
2024pra-vllm
Commits
096827c2
You need to sign in or sign up before continuing.
Unverified
Commit
096827c2
authored
Dec 13, 2023
by
Woosuk Kwon
Committed by
GitHub
Dec 13, 2023
Browse files
[Docs] Add notes on ROCm-supported models (#2087)
parent
6565d9e3
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
10 additions
and
3 deletions
+10
-3
docs/source/models/supported_models.rst
docs/source/models/supported_models.rst
+10
-3
No files found.
docs/source/models/supported_models.rst
View file @
096827c2
...
@@ -73,6 +73,10 @@ If your model uses one of the above model architectures, you can seamlessly run
...
@@ -73,6 +73,10 @@ If your model uses one of the above model architectures, you can seamlessly run
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
.. note::
Currently, the ROCm version of vLLM does not support Mixtral.
Additionally, it only supports Mistral for context lengths up to 4096.
.. tip::
.. tip::
The easiest way to check if your model is supported is to run the program below:
The easiest way to check if your model is supported is to run the program below:
...
@@ -84,12 +88,17 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
...
@@ -84,12 +88,17 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
output = llm.generate("Hello, my name is")
output = llm.generate("Hello, my name is")
print(output)
print(output)
To use model from www.modelscope.cn
If vLLM successfully generates text, it indicates that your model is supported.
.. tip::
To use models from `ModelScope <www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
.. code-block:: shell
.. code-block:: shell
$ export VLLM_USE_MODELSCOPE=True
$ export VLLM_USE_MODELSCOPE=True
And use with :code:`trust_remote_code=True`.
.. code-block:: python
.. code-block:: python
from vllm import LLM
from vllm import LLM
...
@@ -97,5 +106,3 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
...
@@ -97,5 +106,3 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
output = llm.generate("Hello, my name is")
output = llm.generate("Hello, my name is")
print(output)
print(output)
If vLLM successfully generates text, it indicates that your model is supported.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment