Skip to content

GitLab

"lib/bindings/python/vscode:/vscode.git/clone" did not exist on "abd0ba5d95dcccfbeb11d4b55b12a063461c81fa"
Unverified Commit e3f15ee1 authored by jthomson04's avatar jthomson04 Committed by GitHub
Browse files

fix: Little kv routing fix (#1677)

parent 8392e7a1
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 3 deletions
examples/llm/components/kv_router.py
View file @ e3f15ee1
...@@ -100,8 +100,7 @@ def parse_args(service_name, prefix) -> Namespace: ...@@ -100,8 +100,7 @@ def parse_args(service_name, prefix) -> Namespace:
) )
parser.add_argument( parser.add_argument(
"--softmax-sample", "--softmax-sample",
type=bool, action="store_true",
default=False,
help="Whether to do softmax sampling based on worker logits (default is to pick smallest)", help="Whether to do softmax sampling based on worker logits (default is to pick smallest)",
) )
config = ServiceConfig.get_instance() config = ServiceConfig.get_instance()
......
examples/llm/configs/agg_router.yaml
View file @ e3f15ee1
...@@ -29,7 +29,7 @@ Processor: ...@@ -29,7 +29,7 @@ Processor:
Router: Router:
min-workers: 1 min-workers: 1
softmax_sample: true softmax-sample: true
common-configs: [model, block-size, router] common-configs: [model, block-size, router]
VllmWorker: VllmWorker:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment