Skip to content

GitLab

Unverified Commit 3daca38e authored by Nicolò Lucchesi's avatar Nicolò Lucchesi Committed by GitHub
Browse files

[Misc] `toy_proxy_server` handle min_tokens (#39706) 


Signed-off-by: default avatarNickLucche <nlucches@redhat.com>
parent a302a8fd
Hide whitespace changes
Inline Side-by-side
Showing with 7 additions and 0 deletions
tests/v1/kv_connector/nixl_integration/toy_proxy_server.py
View file @ 3daca38e
...@@ -173,6 +173,9 @@ async def send_request_to_service( ...@@ -173,6 +173,9 @@ async def send_request_to_service(
req_data["max_completion_tokens"] = 1 req_data["max_completion_tokens"] = 1
if "stream_options" in req_data: if "stream_options" in req_data:
del req_data["stream_options"] del req_data["stream_options"]
# These args are not supported for P
min_tokens = req_data.pop("min_tokens", None)
min_completion_tokens = req_data.pop("min_completion_tokens", None)
headers = { headers = {
"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
"X-Request-Id": request_id, "X-Request-Id": request_id,
...@@ -187,6 +190,10 @@ async def send_request_to_service( ...@@ -187,6 +190,10 @@ async def send_request_to_service(
# otherwise, it would http.ReadError # otherwise, it would http.ReadError
await response.aread() await response.aread()
# Add back the min_tokens and min_completion_tokens so D can use them
req_data["min_tokens"] = min_tokens
req_data["min_completion_tokens"] = min_completion_tokens
return response return response
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment