Skip to content

GitLab

"wrappers/python/vscode:/vscode.git/clone" did not exist on "6c82c11ba6feb50953e33131072a5fe0e925422c"
Unverified Commit 6edbfa92 authored by Harry Mellor's avatar Harry Mellor Committed by GitHub
Browse files

Mention `extra_body` as a way top pass vLLM only parameters using the OpenAI client (#15240) 


Signed-off-by: default avatarHarry Mellor <19981378+hmellor@users.noreply.github.com>
parent 1e508343
Show whitespace changes
Inline Side-by-side
Showing with 5 additions and 0 deletions
docs/source/serving/openai_compatible_server.md
View file @ 6edbfa92
...@@ -29,6 +29,11 @@ completion = client.chat.completions.create( ...@@ -29,6 +29,11 @@ completion = client.chat.completions.create(
print(completion.choices[0].message) print(completion.choices[0].message)
``` ```
:::{tip}
vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
:::
## Supported APIs ## Supported APIs
We currently support the following OpenAI APIs: We currently support the following OpenAI APIs:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment