from Hacker News

DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

by thyrox on 1/29/25, 3:13 AM with 29 comments

  • by mythz on 1/29/25, 3:48 AM

    Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1].

    Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released.

    [1] https://x.com/carrigmat/status/1884244369907278106

    [2] https://www.nvidia.com/en-us/project-digits/

  • by danans on 1/29/25, 5:41 AM

    Check out the power draw metrics. Following the CPU+GPU power consumption, it seems like it averaged 22W for about a minute. Unless I'm missing something, the inference for this example consumed at most .0004 kWh.

    That's almost nothing. If these models are capable/functional enough for most day-to-day uses, then useful LLM-based GenAI is already at the "too cheap to meter" stage.

  • by teruakohatu on 1/29/25, 3:47 AM

    I am amazed mlx-lm/mlx.distributed works that well on prosumer hardware.

    I don't think they specified what they were using for networking, but it was probably Thunderbolt/USB4 networking which can reach 40Gbps.

  • by shihab on 1/29/25, 3:30 AM

    Please note that it’s using pretty aggressive quantization (around 4 bits per weight)

  • by rashidae on 1/29/25, 3:46 AM

    This is amazing!! What kind of applications are you considering for this? A part from saving variable costs, fine tuning extensively and security… I’m curious to evaluate this in a financial perspective, as variable costs can be daunting, but not too much “yet”.

    I’m hoping NVIDIA comes up with their new consumer computer soon!

  • by iFred on 1/29/25, 3:45 AM

    Complete aside, but I think this is the first time I’ve seen Apple’s internal DNS outside of Apple.

  • by creativenolo on 1/30/25, 11:46 PM

    How is this split between two computers?

  • by DrNosferatu on 1/29/25, 9:46 AM

    Heavily quantized…

    Still interesting though.

  • by mrcwinn on 1/29/25, 3:34 AM

    Fascinating to read the thinking process of a flush vs a straight in poker. It's circular nonsense that is not at all grounded in reason — it's grounded in the factual memory of the rules of Poker, repeated over and over as it continues to doubt itself and double-check. What nonsense!

    How many additional nuclear power plants will need to be built because even these incredibly technical achievements are, under the hood, morons? XD