from Hacker News

Running DeepSeek V3 671B on M4 Mac Mini Cluster

by chadash on 12/27/24, 3:55 PM with 3 comments

• by jauntywundrkind on 12/27/24, 8:39 PM

  Shout out to the video team, for this super cute tower of minis next to the Christmas tree.

  https://x.com/exolabs/status/1872444906851229814

  Only just considering now that Strix Halo could help fill this gap that Mac chips with their huge memory bandwidth enjoy. 256GB systems shouldn't be hard to build!!

  MI300a APU seems not popular but for consumers, this mix of big CPU and GPU seems perhaps quite compelling!

• by talldayo on 12/27/24, 4:02 PM

  > The M4 Max has 546GB/s of memory bandwidth and ~34TFLOPS (fp16) = ~68 GB/s, a ratio of ~8.02. Whereas NVIDIA RTX 4090 has 1008GB/s memory bandwidth and ~330TFLOPS (fp16) = ~660GB/s, a ratio of ~1.52.

  Why are we comparing FP16 performance when you're inferencing INT4 quantized models? Seems like a misleading figure to compare with when it's not really even the performance you're measuring.