from Hacker News

Truffle-1 is an AI inference engine designed to run open source models at home

by firloop on 3/12/24, 5:38 PM with 22 comments

  • by simonw on 3/13/24, 5:34 PM

    The documentation says you start with:

        brew install truffle
    But the Truffle formula currently on Homebrew - https://formulae.brew.sh/formula/truffle - is for something else, its for an Ethereum testing environment of some sort: https://archive.trufflesuite.com/

  • by smoldesu on 3/13/24, 5:41 PM

    As a friendly heads-up to the people interested - you can buy the board inside this off Ebay for like 300-400 dollars cheaper. "Jetson AGX Orin 64GB" specifically, then you wouldn't have to deal with whatever SaaS middleware that the Truffle ships with.

  • by root_axis on 3/13/24, 5:46 PM

    The 22 tokens per second claim for Mixtral conveniently fails to mention what type of quantization is going on with that benchmark.

  • by flakiness on 3/13/24, 6:11 PM

    Looks cute!

    > NVIDIA Orin Module

    So it has some NVIDIA chip inside it looks like?

    https://www.nvidia.com/en-us/autonomous-machines/embedded-sy...

    The 64GB Orin module is sold at about $2K on Amazon. https://www.amazon.com/NVIDIA-Jetson-Orin-64GB-Developer/dp/...

  • by swyx on 3/13/24, 5:28 PM

    im not 100% sure yet what this is for. is this basically stand in for an always on laptop that is running mixtral, that has an api endpoint? and effectively no different than self hosting mixtral on some cloud somewhere?

  • by raj_khare on 3/12/24, 9:12 PM

  • by parentheses on 3/13/24, 6:15 PM

    I want a DIY guide that basically spells out from hardware purchases -> usably running models. I haven't seen one yet.

  • by toisanji on 3/13/24, 5:55 PM

    can it be used for whisper and asr as well? I would buy it if its cheaper.