Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
8eb0a1d9
Unverified
Commit
8eb0a1d9
authored
Sep 29, 2025
by
Zhuohan Li
Committed by
GitHub
Sep 29, 2025
Browse files
[Doc] Polish example for torchrun dp (#25899)
parent
fea3e476
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
10 additions
and
6 deletions
+10
-6
examples/offline_inference/torchrun_dp_example.py
examples/offline_inference/torchrun_dp_example.py
+10
-6
No files found.
examples/offline_inference/torchrun_dp_example.py
View file @
8eb0a1d9
...
@@ -4,6 +4,11 @@
...
@@ -4,6 +4,11 @@
experimental support for data-parallel inference with torchrun
experimental support for data-parallel inference with torchrun
Note the data load balancing and distribution is done out of the vllm engine,
Note the data load balancing and distribution is done out of the vllm engine,
no internal lb supported in external_launcher mode.
no internal lb supported in external_launcher mode.
To run this example:
```bash
$ torchrun --nproc-per-node=2 examples/offline_inference/torchrun_dp_example.py
```
"""
"""
from
vllm
import
LLM
,
SamplingParams
from
vllm
import
LLM
,
SamplingParams
...
@@ -14,7 +19,7 @@ prompts = [
...
@@ -14,7 +19,7 @@ prompts = [
"The president of the United States is"
,
"The president of the United States is"
,
"The capital of France is"
,
"The capital of France is"
,
"The future of AI is"
,
"The future of AI is"
,
]
*
50
]
# Create sampling parameters, the same across all ranks
# Create sampling parameters, the same across all ranks
sampling_params
=
SamplingParams
(
temperature
=
0.8
,
top_p
=
0.95
)
sampling_params
=
SamplingParams
(
temperature
=
0.8
,
top_p
=
0.95
)
...
@@ -45,14 +50,13 @@ prompts = [
...
@@ -45,14 +50,13 @@ prompts = [
outputs
=
llm
.
generate
(
prompts
,
sampling_params
)
outputs
=
llm
.
generate
(
prompts
,
sampling_params
)
# all ranks will have the same outputs
print
(
"-"
*
50
)
for
output
in
outputs
:
for
output
in
outputs
:
prompt
=
output
.
prompt
prompt
=
output
.
prompt
generated_text
=
output
.
outputs
[
0
].
text
generated_text
=
output
.
outputs
[
0
].
text
print
(
f
"Prompt:
{
prompt
!
r
}
\n
Generated text:
{
generated_text
!
r
}
\n
"
)
print
(
print
(
"-"
*
50
)
f
"DP Rank:
{
dp_rank
}
Prompt:
{
prompt
!
r
}
\n
Generated text:
{
generated_text
!
r
}
\n
"
)
"""
"""
Further tips:
Further tips:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment