Commits · 8048c28c11b7b377d769bfc38fd8b8c87fb187de · change / sglang · GitLab

22 Nov, 2024 2 commits
- Fix #2037 - Context length check does not take into out pad tokens for visual models (#2106) · 8048c28c
  Jake Poznanski authored Nov 21, 2024
  
  8048c28c
- [router] add base_gpu_id server args & merged radix tree python reference (#2115) · 30af7dfb
  Byron Hsu authored Nov 21, 2024
  
  30af7dfb
21 Nov, 2024 4 commits
- Add support for Qwen2-VL-based embedding models (#2055) · f6f71379
  James Xu authored Nov 21, 2024
  
  f6f71379
- ROCm: Fix MoE padding for none FP8 cases (#2111) · f35cb46c
  HAI authored Nov 21, 2024
  
  f35cb46c
- Turn off autotune for scaled mm for fp8 dynamic quant in torchao (#2116) · 7f8fcd39
  Jerry Zhang authored Nov 21, 2024
  
  7f8fcd39
- Error out when torchao-config option is not recognized (#2107) · 5c6a41fa
  Jerry Zhang authored Nov 20, 2024
  
  5c6a41fa
20 Nov, 2024 5 commits
- Enable overlap scheduler by default for the triton attention backend (#2105) · 722530fa
  Lianmin Zheng authored Nov 20, 2024
  
  722530fa
- Allow skipping warmup in bench_offline_throughput.py (#2103) · 3295cd8a
  Lianmin Zheng authored Nov 20, 2024
  
  3295cd8a
- [feat] Add session control (#2073) · 5942dfc0
  Ying Sheng authored Nov 20, 2024
  
  5942dfc0
- Enable overlap by default (#2067) · 7d671e4a
  Lianmin Zheng authored Nov 19, 2024
  
  7d671e4a
- Set schedule policy more conservative for DP attention (#2096) · 699384cb
  Ke Bao authored Nov 20, 2024
  
  699384cb
19 Nov, 2024 6 commits
- Make constrained decoding work for overlap scheduler (#2095) · ffd20fcd
  Lianmin Zheng authored Nov 19, 2024
  
  ffd20fcd
- minor: add dataset dump and questions shuffle (#2093) · 55bd97f3
  Yineng Zhang authored Nov 20, 2024
  
  55bd97f3
- Use native fp8 format on MI300X (#2094) · e57c3e12
  HAI authored Nov 19, 2024
  
  e57c3e12
- Fix: incorrect top_logprobs in chat completion (#2088) · 929c7621
  Alexander Waitz authored Nov 19, 2024
  
  929c7621
- Use cuda event wait and synchronization instead of busy waiting (#2089) · b7a065ea
  Lianmin Zheng authored Nov 19, 2024
  
  b7a065ea
- Simplify logits penalizer (#2086) · b1104538
  Lianmin Zheng authored Nov 18, 2024
  
  b1104538
18 Nov, 2024 14 commits
- Allow passing extra request body to bench_offline_throughput.py (#2085) · 3b44bbee
  Lianmin Zheng authored Nov 18, 2024
  
  3b44bbee
- Fix chunked prefill with output logprob (#2083) · 80e2c4a8
  Lianmin Zheng authored Nov 18, 2024
  
  80e2c4a8
- Rename layer_idx to layer_id for consistency (#2078) · 66318ffe
  Jani Monoses authored Nov 18, 2024
  
  66318ffe
- feat: update torch 2.5.1 (#2069) · 76619261
  Yineng Zhang authored Nov 18, 2024
  
  76619261
- support set role as 'tool' (#2075) · 2a3992b6
  yukavio authored Nov 18, 2024
```
Co-authored-by: kavioyu <kavioyu@tencent.com>
```
  2a3992b6
- Simplify flashinfer indices update for prefill (#2074) · 4af3f889
  Lianmin Zheng authored Nov 18, 2024
```
Co-authored-by: kavioyu <kavioyu@tencent.com>
Co-authored-by: kavioyu <kavioyu@gmail.com>
```
  4af3f889
- Crash the CI jobs on model import errors (#2072) · df7fe452
  Lianmin Zheng authored Nov 17, 2024
  
  df7fe452
- Fix cuda illegal memory access in overlap mode (#2070) · 11668533
  Lianmin Zheng authored Nov 17, 2024
  
  11668533
- [Minor] Fix styles for overlap mode (#2068) · a9e90b4b
  Lianmin Zheng authored Nov 17, 2024
  
  a9e90b4b
- add phi-3 small support (#2062) · 8c280cee
  Tanjiro authored Nov 17, 2024
```
Co-authored-by: Tushar Goel <114812108+AI-Tushar@users.noreply.github.com>
```
  8c280cee
- [Performance] Update xgrammar-related constrained decoding (#2056) · 9c745d07
  DarkSharpness authored Nov 18, 2024
  
  9c745d07
- Rename arguments `--disable-nan-detection` to `--enable-nan-detection` (#2066) · ebaa2f31
  Lianmin Zheng authored Nov 17, 2024
  
  ebaa2f31
- Support cuda graph for DP attention (#2061) · 62832bb2
  Ke Bao authored Nov 18, 2024
  
  62832bb2
- Deprecate --disable-flashinfer and --disable-flashinfer-sampling (#2065) · 11f881d1
  Lianmin Zheng authored Nov 17, 2024
  
  11f881d1
17 Nov, 2024 5 commits
- Remove monkey_patch_vllm_dummy_weight_loader (#2064) · 38625e21
  Lianmin Zheng authored Nov 17, 2024
  
  38625e21
- Revert "chore: update torch v2.5.1" (#2063) · c1f401fc
  Lianmin Zheng authored Nov 17, 2024
  
  c1f401fc
- chore: update torch v2.5.1 (#1849) · 3b878863
  Yineng Zhang authored Nov 18, 2024
  
  3b878863
- Launch dp ranks in parallel (#2053) · f719d9ae
  Lianmin Zheng authored Nov 16, 2024
```
Co-authored-by: Haotian Liu <6631389+haotian-liu@users.noreply.github.com>
```
  f719d9ae
- Fix illegal memory access in overlap mode & Use more fused triton kernels for... · edad3731
  Lianmin Zheng authored Nov 16, 2024
```
Fix illegal memory access in overlap mode & Use more fused triton kernels for building meta data (#2051)
```
  edad3731
16 Nov, 2024 4 commits
- Support DP MLA (#1970) · 976bc302
  Ke Bao authored Nov 16, 2024
  
  976bc302
- Fix weight update for data parallelism (#2050) · 2f2e0743
  Lianmin Zheng authored Nov 16, 2024
  
  2f2e0743
- Add get_amdgpu_memory_capacity() (#2049) · 2ffe0a73
  HAI authored Nov 15, 2024
  
  2ffe0a73
- Add Tensor Parallel to torch_native_llama (#1876) · cf248976
  Ke Wen authored Nov 15, 2024
  
  cf248976