from Hacker News

Building a deep learning rig

by dvcoolarun on 2/23/24, 1:52 PM with 106 comments

  • by Yenrabbit on 2/24/24, 6:21 PM

    Note that they shared part two recently: https://samsja.github.io/blogs/rig/part_2/

    For those talking about breakeven points and cheap cloud compute, you need to factor in the mental difference it makes running a test locally (which feels free) vs setting up a server and knowing you're paying per hour it's running. Even if the cost is low, I do different kinds of experiments knowing I'm not 'wasting money' every minute the GPU sits idle. Once something is working, then sure scaling up on cheap cloud compute makes sense. But it's really, really nice having local compute to get to that state.

  • by abra0 on 2/24/24, 2:42 PM

    I was thinking of doing something similar, but I am a bit sceptical about how the economics on this works out. On vast.ai renting a 3x3090 rig is $0.6/hour. The electricity price of operating this in e.g. Germany is somewhere about $0.05/hour. If the OP paid 1700 EUR for the cards, the breakeven point would be around (haha) 3090 hours in, or ~128 days, assuming non-stop usage. It's probably cool to do that if you have a specific goal in mind, but to tinker around with LLMs and for unfocused exploration I'd advise folks to just rent.

  • by infogulch on 2/24/24, 1:51 PM

  • by bick_nyers on 2/24/24, 8:02 PM

    Somewhat tangential question, but I'm wondering if anyone knows of a solution (or Google search terms for this):

    I have a 3U supermicro server chassis that I put an AM4 motherboard into, but I'm looking at upgrading the Mobo so that I can run ~6 3090s in it. I don't have enough physical PCIE slots/brackets in the chassis (7 expansion slots), so I either need to try to do some complicated liquid cooling setup to make the cards single slot (I don't want to do this), or I need to get a bunch of riser cables and mount the GPU above the chassis. Is there like a JBOD equivalent enclosure for PCIE cards? I don't really think I can run the risers out the back of the case, so I'll likely need to take off/modify the top panel somehow. What I'm picturing in my head is basically a 3U to 6U case conversion, but I'm trying to minimize cost (let's say $200 for the chassis/mount component) as well as not have to cut metal.

  • by kaycebasques on 2/24/24, 3:24 PM

    I really enjoy and am inspired by the idea that people like Dettmer (and probably this Samsja person) are the spiritual successors to homebrew hackers in the 70s and 80s. They have pretty intimate knowledge of many parts of the whole goddamn stack, from what's going on in each hardware component, to how to assemble all the components into a rig, up to all the software stuff: algorithms, data, orchestration, etc.

    Am also inspired by embedded developers for the same reason

  • by neilv on 2/24/24, 5:10 PM

    For large VRAM models, what about selling one of the 3090s, and putting the money towards an NVLink and a motherboard with two x16 PCIe slots (and preferably spaced so you don't need riser cables)?

  • by Uehreka on 2/24/24, 5:00 PM

    > I just got my hands on a mining rig with 3 rtx 3090 founder edition for the modest sum of 1.7k euros.

    I would prefer a tutorial on how to do this.

  • by whoisthemachine on 2/24/24, 4:52 PM

    I've been slowly expanding my HTPC/media server into a gaming server and box for running LLMs (and possibly diffusion models?) locally for playing around with. I think it's becoming clear that the future of LLM's will be local!

    My box has a Gigabyte B450M, Ryzen 2700X, 32GB RAM, Radeon 6700XT (for gaming/streaming to steam link on Linux), and an "old" Geforce GTX 1650 with a paltry 6GB of RAM for running models on. Currently it works nicely with smaller models on ollama :) and it's been fun to get it set up. Obviously, now that the software is running I could easily swap in a more modern NVidia card with little hassle!

    I've also been eyeing the b450 steel legend as a more capable board for expansion than the Gigabyte board, this article gives me some confidence that it is a solid board.

  • by smokeydoe on 2/24/24, 5:45 PM

    Does anyone have any good recommendations for an epyc server grade motherboard that can use 3x3090? My current motherboard (strix trx40-xe) has memory issues now. 2 slots cause boot errors no matter what memory is inserted. I plan to sell the threadripper. Other option is to just swap out the current motherboard with a trx zenith extreme but I feel server grade would be better at this point after experiencing issues. Is supermicro worth it?

  • by 0x20cowboy on 2/24/24, 8:01 PM

    If you would like to put Kubernetes on top of this kind of setup this repo is helpful https://github.com/robrohan/skoupidia

    The main benefit is you can shut off nodes entirely when not using them, and then when you turn them back on they just rejoin the cluster.

    It also helps managing different types of devices and workloads (tpu vs gpu vs cpu)

  • by akasakahakada on 2/25/24, 1:55 AM

    Just sharing.

    2 x RTX4090 workstation guide

    You can put two aircooled 4090 in the same ATX case if you do enough research.

    https://github.com/eul94458/Memo/blob/main/dual_rtx4090works...

  • by cyanydeez on 2/24/24, 2:48 PM

    just ordered a 15k thread ripper platform because it's the only way to cheaply maximize the pcie16x bottleneck. the mining rigs are neat because the space you need for consumer GPU is a big issue.

    those rigs need pcie riser slots that are also limited.

    looks like the primary value is the rig and the cards. they'll need another 1-2k for a thread ripper and then the riser slots.

  • by nirav72 on 2/24/24, 4:52 PM

    This is nice. I would’ve used one of those ETH mining cases that support multiple GPUs. Ebay has them $100-150 these days.

  • by whimsicalism on 2/24/24, 5:25 PM

    I strongly, strongly suspect most people doing this are significantly short of the breakeven prices for transitioning from cloud 3090s.

    inb4 there are no cloud 3090s: yes there are, just not in formal datacenters

  • by jeffybefffy519 on 2/24/24, 8:33 PM

    Are m1/m2/m3 max mac's any good for this?

  • by gigatexal on 2/24/24, 5:03 PM

    I thought this looked like a cryptocurrency miner. Seems the crypto to AI pivot is legit happening. And good. Would rather we boiled the oceans for something marginally more valuable than in-game tokens we traded for fiat funds in this video game we call life.