@@ -218,6 +218,10 @@ for out in state.text_iter():
...
@@ -218,6 +218,10 @@ for out in state.text_iter():
print(out,end="",flush=True)
print(out,end="",flush=True)
```
```
### Tips and Implementation Details
- The `choices` argument in `sgl.gen` is implemented by computing the normalized log probabilities of all choices and selecting the one with the highest probability.
- The `regex` argument in `sgl.gen` is implemented through autoregressive decoding with logit bias masking, according to the constraints set by the regex.
## Backend: SGLang Runtime (SRT)
## Backend: SGLang Runtime (SRT)
The SGLang Runtime (SRT) is designed to work best with the SGLang frontend.
The SGLang Runtime (SRT) is designed to work best with the SGLang frontend.
However, it can also be used as a standalone API server.
However, it can also be used as a standalone API server.
