Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
change
sglang
Commits
b085e06b
Unverified
Commit
b085e06b
authored
Dec 30, 2024
by
Yineng Zhang
Committed by
GitHub
Dec 30, 2024
Browse files
docs: add development guide using docker (#2645)
parent
763dd55d
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
45 additions
and
0 deletions
+45
-0
docs/developer/development_guide_using_docker.md
docs/developer/development_guide_using_docker.md
+45
-0
No files found.
docs/developer/development_guide_using_docker.md
0 → 100644
View file @
b085e06b
# Development Guide Using Docker
## Setup VSCode
Download
`code`
from
`Https://code.visualstudio.com/docs/?dv=linux64cli`
```
bash
wget https://vscode.download.prss.microsoft.com/dbazure/download/stable/fabdb6a30b49f79a7aba0f2ad9df9b399473380f/vscode_cli_alpine_x64_cli.tar.gz
tar
xf vscode_cli_alpine_x64_cli.tar.gz
# https://code.visualstudio.com/docs/remote/tunnels
./code tunnel
```
## Setup Docker Container
### H100
```
bash
# Change the name to yours
docker run
-itd
--shm-size
32g
--gpus
all
-v
/opt/dlami/nvme/.cache:/root/.cache
--ipc
=
host
--name
sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
docker
exec
-it
sglang_zhyncs /bin/zsh
```
### H200
```
bash
docker run
-itd
--shm-size
32g
--gpus
all
-v
/mnt/co-research/shared-models:/root/.cache/huggingface
--ipc
=
host
--name
sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
docker
exec
-it
sglang_zhyncs /bin/zsh
```
## Profile
```
bash
# Change batch size, input, output and add `disable-cuda-graph` (for easier analysis)
# e.g. DeepSeek V3
nsys profile
-o
deepseek_v3 python3
-m
sglang.bench_one_batch
--batch-size
1
--input
128
--output
256
--model
deepseek-ai/DeepSeek-V3
--trust-remote-code
--tp
8
--disable-cuda-graph
```
## Evaluation
```
bash
# e.g. gsm8k 8 shot
python3 benchmark/gsm8k/bench_sglang.py
--num-questions
2000
--parallel
2000
--num-shots
8
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment