Commits · 406524821457fb52123d7b3e433e016b4a2a1d2f · change / sglang · GitLab

09 Apr, 2025 14 commits
- Support Llama4 fp8 inference (#5194) · 40652482
  HandH1998 authored Apr 09, 2025
```
Co-authored-by: laixinn <xielx@shanghaitech.edu.cn>
Co-authored-by: sleepcoo <sleepcoo@gmail.com>
Co-authored-by: zhyncs <me@zhyncs.com>
```
  40652482
- Optimize topk operation in llama4 (#5128) · 86a876d8
  fzyzcjy authored Apr 09, 2025
  
  86a876d8
- Fix ci test "test_eval_fp8_accuracy" failed (#5185) · 92823069
  kk authored Apr 09, 2025
```
Co-authored-by: wunhuang <wunhuang@amd.com>
```
  92823069
- [Misc] clean up vllm in sgl-kernel test (#5189) · d2e507df
  yinfan98 authored Apr 09, 2025
  
  d2e507df
- Let `bench_one_batch` support `enable_dp_attention` (#4058) · 61970b08
  fzyzcjy authored Apr 09, 2025
  
  61970b08
- [DeepEP] fix: import buffer error (#5179) · 76c48a09
  Cheng Wan authored Apr 09, 2025
  
  76c48a09
- fix: use DeepEPDispatcher on CUDA (#5180) · 90caf06c
  Yineng Zhang authored Apr 08, 2025
  
  90caf06c
- feat: add DeepGEMM build warning (#5176) · 6669d127
  Yineng Zhang authored Apr 08, 2025
```
Co-authored-by: grimoire <streetyao@live.com>
```
  6669d127
- docs: remove the use of Downward API for LWS_WORKER_INDEX (#5110) · f2b70afd
  Kay Yan authored Apr 09, 2025
```
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
```
  f2b70afd
- [Fix] DeepEP Compatibility with Low Latency (#5068) · bc3f6db2
  Jinyan Chen authored Apr 09, 2025
```
Co-authored-by: ch-wan <cwan39@gatech.edu>
```
  bc3f6db2
- [Bugfix] Fix index out of bounds in local attention with large sequences (#5173) · aac531c5
  Chang Su authored Apr 08, 2025
  
  aac531c5
- Tiny disable model that does not work (#5175) · 39efad4f
  fzyzcjy authored Apr 09, 2025
  
  39efad4f
- Fix multimodal hashing error (#5174) · 466899e6
  fzyzcjy authored Apr 09, 2025
  
  466899e6
- FP4 weight loading and inference (2/2) (#3972) · 11d760d5
  Trevor Morris authored Apr 08, 2025
  
  11d760d5
08 Apr, 2025 20 commits
- Support 2x8xH100 for Llama 4 (#5159) · 5039d547
  fzyzcjy authored Apr 09, 2025
  
  5039d547
- [feat&refactor] Enhance multimodal input support with refactor io_struct (#4938) · d09a51f1
  XinyuanTong authored Apr 08, 2025
```
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
```
  d09a51f1
- Small improvement of native api docs (#5139) · f8194b26
  simveit authored Apr 08, 2025
```
Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
```
  f8194b26
- [PD] Simplify mini LB (#4911) · 6d3b35fa
  Byron Hsu authored Apr 08, 2025
```
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
```
  6d3b35fa
- Add optimized native kernels in sgl-kernel (#5150) · a73c4df4
  Ma Mingfei authored Apr 09, 2025
```
Co-authored-by: Chunyuan WU <chunyuan.wu@intel.com>
Co-authored-by: YanbingJiang <yanbing.jiang@intel.com>
Co-authored-by: blzheng <beilei.zheng@intel.com>
```
  a73c4df4
- [PD] Fix unclosed prefill connection warning of mini_lb (#5155) · 89a55418
  shangmingc authored Apr 09, 2025
```
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
```
  89a55418
- Fix loading KV quantization scale; Enable modelopt kv cache (#4686) · 2695ab05
  Yun Dai authored Apr 08, 2025
```
Co-authored-by: qingquansong <ustcsqq@gmail.com>
```
  2695ab05
- Fix torch compile errors (#5158) · 88d6fd9a
  kk authored Apr 08, 2025
  
  88d6fd9a
- fix empty_cache error in pt_weights_iterator (#5151) · cc88d98a
  DangKai authored Apr 08, 2025
```
Co-authored-by: dangkai.dk <dangkai.dk@alibaba-inc.com>
```
  cc88d98a
- Add dummy grok test to amd CI. (#5115) · 3033c11a
  saienduri authored Apr 08, 2025
  
  3033c11a
- Use public model for FA3 speculative decode testing (#5152) · fd5a55cf
  Yubo Wang authored Apr 08, 2025
  
  fd5a55cf
- Add unit test on page_size > 1 and mla and integration test for Flash Attention 3 (#4760) · 804d9f2e
  Yubo Wang authored Apr 07, 2025
  
  804d9f2e
- [FA3 Feature] Support multi modal Llama-3.2-11B-Vision-Instruct (#5103) · a7c3f74b
  Chunan Zeng authored Apr 07, 2025
  
  a7c3f74b
- Fix run time error in ROCm platform (#5147) · 5a144a8a
  kk authored Apr 08, 2025
```
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: root <root@dell300x-pla-t10-17.pla.dcgpu>
```
  5a144a8a
- fix multimodal hash feature (#5083) · 27f8e6b9
  huangtingwei authored Apr 08, 2025
  
  27f8e6b9
- [AMD] Fix missing per_token_group_quant_fp8 for ROCm (#5140) · afb752bc
  Hubert Lu authored Apr 07, 2025
  
  afb752bc
- [modelopt] automatically inspect if model is ModelOpt quantized and set quantization method (#5145) · 9731eca7
  Yun Dai authored Apr 07, 2025
  
  9731eca7
- feat: disable grammar restrictions within reasoning sections (#4984) · 7c5658c1
  mlmz authored Apr 08, 2025
```
Co-authored-by: tianhaoyu <thy@mail.ecust.edu.cn>
Co-authored-by: DarkSharpness <2040703891@qq.com>
```
  7c5658c1
- [Misc] Use pytest.mark.skipif in sgl-kernel test (#5137) · 9798e72b
  yinfan98 authored Apr 08, 2025
  
  9798e72b
- Add Llama4 user guide (#5133) · ade714a6
  Ke Bao authored Apr 08, 2025
```
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
```
  ade714a6
07 Apr, 2025 5 commits
- Refactor and Optimize FA3 Code (#5090) · 93470a14
  Stefan He authored Apr 07, 2025
```
Co-authored-by: Qingquan Song <ustcsqq@gmail.com>
```
  93470a14
- [ci] fix llama4 ci error (#5126) · db452760
  Xiaoyu Zhang authored Apr 07, 2025
  
  db452760
- bump v0.4.5 (#5117) · 57f99608
  Yineng Zhang authored Apr 07, 2025
  
  57f99608
- Fix refactor error - fp8.py (#5106) · 81992474
  HAI authored Apr 07, 2025
```
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
```
  81992474
- Add Llama4 support (#5092) · f04c80dc
  Chang Su authored Apr 07, 2025
```
Co-authored-by: Cheng Wan <cwan39@gatech.edu>
Co-authored-by: fzyzcjy <ch271828n@outlook.com>
Co-authored-by: ispobock <ispobaoke@163.com>
```
  f04c80dc
06 Apr, 2025 1 commit
- Fix: Reduce the number of document ci attempts to avoid long ci running (#5097) · d1bb1711
  mlmz authored Apr 06, 2025
```
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
```
  d1bb1711