from Hacker News

TinyChat: Large Language Model on the Edge

by enduku on 12/8/23, 7:50 PM with 1 comments

• by enduku on 12/8/23, 7:50 PM

  TinyChat is an efficient, lightweight, Python-native serving framework for 4-bit LLMs by AWQ. It delivers 2.3x generation speed up on RTX4090.

  Code: https://github.com/mit-han-lab/llm-awq/tree/main/tinychat