Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
change
sglang
Commits
7802df1e
Commit
7802df1e
authored
Jul 25, 2024
by
Ying Sheng
Browse files
Update readme
parent
1a491d00
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
4 additions
and
5 deletions
+4
-5
README.md
README.md
+3
-3
docker/Dockerfile
docker/Dockerfile
+1
-2
No files found.
README.md
View file @
7802df1e
...
...
@@ -4,7 +4,7 @@
--------------------------------------------------------------------------------
|
[
**Blog**
](
https://lmsys.org/blog/2024-0
1-17
-sglang/
)
|
[
**Paper**
](
https://arxiv.org/abs/2312.07104
)
|
|
[
**Blog**
](
https://lmsys.org/blog/2024-0
7-25
-sglang
-llama3
/
)
|
[
**Paper**
](
https://arxiv.org/abs/2312.07104
)
|
SGLang is a fast serving framework for large language models and vision language models.
It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language.
...
...
@@ -57,7 +57,7 @@ pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
```
### Method 3: Using docker
The docker images are available on Docker Hub as
[
lmsysorg/sglang
](
https://hub.docker.com/r/lmsysorg/sglang/tags
)
.
The docker images are available on Docker Hub as
[
lmsysorg/sglang
](
https://hub.docker.com/r/lmsysorg/sglang/tags
)
, built from
[
Dockerfile
](
docker
)
.
```
bash
docker run
--gpus
all
\
...
...
@@ -66,7 +66,7 @@ docker run --gpus all \
--env
"HUGGING_FACE_HUB_TOKEN=<secret>"
\
--ipc
=
host
\
lmsysorg/sglang:latest
\
python3
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3-8B
--host
0.0.0.0
--port
30000
python3
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3-8B
-Instruct
--host
0.0.0.0
--port
30000
```
### Common Notes
...
...
docker/Dockerfile
View file @
7802df1e
ARG
CUDA_VERSION=12.
4
.1
ARG
CUDA_VERSION=12.
1
.1
FROM
nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04
ARG
CUDA_VERSION=12.4.1
ARG
PYTHON_VERSION=3
ENV
DEBIAN_FRONTEND=noninteractive
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment