from Hacker News

Tabby v0.1.1: Metal inference and StarCoder supports

by wsxiaoys on 9/20/23, 4:15 PM with 1 comments

  • by wsxiaoys on 9/20/23, 4:23 PM

    Previously, Tabby ran exclusively on CUDA devices, posing a significant barrier for developers looking to effectively utilize LLMs in their day-to-day coding.

    The Tabby team made a significant contribution by enhancing support for the StarCoder series models (1B/3B/7B) in llama.cpp. This enhancement allows these models to run on Metal, providing comparable performance to that of an NVIDIA GPU.