-
drbh authored
This PR resolves an issue loading in tokenizer_configs where the eos or bos token is null as in: [Qwen/Qwen1.5-72B-Chat](https://huggingface.co/Qwen/Qwen1.5-72B-Chat/blob/main/tokenizer_config.json) resolves: https://github.com/huggingface/text-generation-inference/issues/1545 and related to https://github.com/QwenLM/Qwen1.5/issues/162
9be1db31