Commit 3c60fe2a authored by ptarasiewiczNV's avatar ptarasiewiczNV Committed by GitHub
Browse files

feat: Use round robin for disagg routing (#40)

Co-authored-by: ptarasiewicz@nvidia.com <Piotr Tarasiewicz>
parent bc42616e
......@@ -79,7 +79,7 @@ class RequestHandler:
def get_remote_prefill_request_callback(self):
async def callback(request: RemotePrefillRequest):
json_request = msgspec.json.encode(request).decode("utf-8")
self.prefill_client.generate(json_request)
self.prefill_client.round_robin(json_request)
return callback
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment