Unverified Commit 60a98b2d authored by Harry Mellor's avatar Harry Mellor Committed by GitHub
Browse files

[Docs] Mention `model_impl` arg when explaining Transformers fallback (#14552)


Signed-off-by: default avatarHarry Mellor <19981378+hmellor@users.noreply.github.com>
parent 460f553a
...@@ -59,6 +59,10 @@ llm.apply_model(lambda model: print(type(model))) ...@@ -59,6 +59,10 @@ llm.apply_model(lambda model: print(type(model)))
If it is `TransformersModel` then it means it's based on Transformers! If it is `TransformersModel` then it means it's based on Transformers!
:::{tip}
You can force the use of `TransformersModel` by setting `model_impl="transformers"` for <project:#offline-inference> or `--model-impl transformers` for the <project:#openai-compatible-server>.
:::
:::{note} :::{note}
vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM. vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM.
::: :::
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment