"vscode:/vscode.git/clone" did not exist on "afffd3cc8a99ce1cf0f6f1687852e5519d725a3b"
Unverified Commit 887e900b authored by rongfu.leng's avatar rongfu.leng Committed by GitHub
Browse files

[Docs] Add profiler user docs for http request (#31370)


Signed-off-by: default avatarrongfu.leng <rongfu.leng@daocloud.io>
parent 48e74497
......@@ -54,6 +54,29 @@ vllm bench serve \
--num-prompts 2
```
Or use http request:
```shell
# We need first call /start_profile api to start profile.
$ curl -X POST http://localhost:8000/start_profile
# Call model generate.
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{
"role": "user",
"content": "San Francisco is a"
}
]
}'
# After need call /stop_profile api to stop profile.
$ curl -X POST http://localhost:8000/stop_profile
```
## Profile with NVIDIA Nsight Systems
Nsight systems is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment