"git@developer.sourcefind.cn:renzhc/diffusers_dcu.git" did not exist on "8705af0914c5957057f0faf9cde05b528ab90d97"
Unverified Commit 3cde5eb6 authored by vchzls's avatar vchzls Committed by GitHub
Browse files

docs: Improve instructions for supporting new models (#2363)


Co-authored-by: default avatarzhaohoulong <zhaohoulong@xiaomi.com>
parent f5b2a3aa
...@@ -80,3 +80,30 @@ To port a model from vLLM to SGLang, you can compare these two files [SGLang Lla ...@@ -80,3 +80,30 @@ To port a model from vLLM to SGLang, you can compare these two files [SGLang Lla
- Remove `Sample`. - Remove `Sample`.
- Change `forward()` functions, and add `forward_batch`. - Change `forward()` functions, and add `forward_batch`.
- Add `EntryClass` at the end. - Add `EntryClass` at the end.
### Registering an external model implementation
In addition to the methods described above, you can also register your new model with the `ModelRegistry` before launching the server. This approach is useful if you want to integrate your model without needing to modify the source code.
Here is how you can do it:
```python
from sglang.srt.models.registry import ModelRegistry
from sglang.srt.server import launch_server
# for a single model, you can add it to the registry
ModelRegistry.models[model_name] = model_class
# for multiple models, you can imitate the import_model_classes() function in sglang/srt/models/registry.py
from functools import lru_cache
@lru_cache()
def import_new_model_classes():
model_arch_name_to_cls = {}
...
return model_arch_name_to_cls
ModelRegistry.models.update(import_new_model_classes())
launch_server(server_args)
```
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment