# TinyChat: Efficient and Minimal Chatbot with AWQ
We introduce TinyChat, a cutting-edge chatbot interface designed for minimal resource consumption and fast inference speed on GPU platforms. It allows for seamless deployment on consumer-level GPUs such as 3090/4090 and low-power edge devices like the NVIDIA Jetson Orin, empowering users with a responsive conversational experience like never before.
We introduce TinyChat, a fast GPU inference library for LLMs quantized by AWQ (W4A16). It allows real-time LLM deployment on consumer-level GPUs such as 3090/4090 and low-power edge devices like the NVIDIA Jetson Orin, empowering users with a responsive conversational experience on the edge. It runs LLaMA2-7B at 8.7ms/token on 4090 and 75.1ms/token on Jetson Orin.
