Skip to content

GitLab

"csrc/vscode:/vscode.git/clone" did not exist on "b0746fae3d576528598d43da129ec5655197560b"
Unverified Commit 6edbfa92 authored by Harry Mellor's avatar Harry Mellor Committed by GitHub
Browse files

Mention `extra_body` as a way top pass vLLM only parameters using the OpenAI client (#15240) 


Signed-off-by: default avatarHarry Mellor <19981378+hmellor@users.noreply.github.com>
parent 1e508343
Hide whitespace changes
Inline Side-by-side
Showing with 5 additions and 0 deletions
docs/source/serving/openai_compatible_server.md
View file @ 6edbfa92
......@@ -29,6 +29,11 @@ completion = client.chat.completions.create(
print(completion.choices[0].message)
```
:::{tip}
vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
:::
## Supported APIs
We currently support the following OpenAI APIs:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment