from Hacker News

Building an AI server on a budget

by mful on 6/6/25, 2:33 AM with 106 comments

  • by mythz on 6/9/25, 5:02 AM

    Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for?

    It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc.

    My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB RAM, 32/64 core Xeon 1U rack server. Nothing amazing but wanted the most GPU VRAM before paying the premium Nvidia tax on their larger GPUs.

    This works well for Ollama/llama-server which can make use of all GPU VRAM unfortunately ComfyUI can't make use of all GPU VRAM to run larger models, so on the lookout for a lot more RAM in my next GPU Server.

    Really hoping Intel can deliver with its upcoming Arc Pro B60 Dual GPU for a great value 48GB option which can be run 4x in an affordable 192GB VRAM workstation [1]. If it runs Ollama and ComfyUI efficiently I'm sold.

    [1] https://www.servethehome.com/maxsun-intel-arc-pro-b60-dual-g...

  • by 7speter on 6/6/25, 4:46 AM

    I dunno everyone, but I think Intel has something big on their hands with their announced workstation gpus. The b50 is a low profile card that doesn’t have a powersupply hookup because it only uses something like 60 watts, and comes with 16gb vram at a msrp of 300 dollars.

    I imagine companies will have first dibs via the likes of agreements with suppliers like CDW, etc, but if Intel had enough of these battlemage dies accumulated, it could also drastically change the local ai enthusiast/hobbyist landscape; for starters this could drive down the price of workstation cards that are ideal for inference, at the very least. I’m cautiously excited.

    On the AMD front (really, a sort of open compute front), Vulkan Kompute is picking up steam and it would be really cool to have a standard that mostly(?) ships with Linux, and older ports available for Freebsd, so that we can actually run free as in freedom inference locally.

  • by JKCalhoun on 6/8/25, 8:59 PM

    Someone posted that they had used a "mining rig" [0] from AliExpress for less than $100. It even has RAM and a CPU. He picked up a 2000W (!) DELL server PS for cheap off eBay. The GPUs were NVIDIA TESLAs (M40 for example) since they often have a lot of RAM and are less expensive.

    I followed in those footsteps to create my own [1] (photo [2]).

    I picked up a 24GB M40 for around $300 off eBay. I 3D printed a "cowl" for the GPU that I found online and picked up two small fans from Amazon that got int he cowl. Attached the cowl + fans keep the GPU cool. (These TESLA server GPUs have no fan since they're expected to live in one of those wind-tunnels called a server rack).

    I bought the same cheap DELL server PS that the original person had used and I also had to get a break-out board (and power-supply cables and adapters) for the GPU.

    Thanks to LLMs, I was able to successfully install Rocky Linux as well as CUDA and NVIDIA drivers. I SSH into it and run ollama commands.

    My own hurdle at this point is: I have a 2nd 24 GB M40 TESLA but when installed on the motherboard, Linux will not boot. LLMs are helping me try to set up BIOS correctly or otherwise determine what the issue is. (We'll see.) I would love to get to 48 GB.

    [0] https://www.aliexpress.us/item/3256806580127486.html

    [1] https://bsky.app/profile/engineersneedart.com/post/3lmg4kiz4...

    [2] https://cdn.bsky.app/img/feed_fullsize/plain/did:plc:oxjqlam...

  • by Uehreka on 6/6/25, 4:55 AM

    Love the attention to detail, I can tell this was a lot of work to put together and I hope it helps people new to PC building.

    I will note though, 12GB of VRAM and 32GB of system RAM is a ceiling you’re going to hit pretty quickly if you’re into messing with LLMs. There’s basically no way to do a better job at the budget you’re working with though.

    One thing I hear about a lot is people using things like RunPod to briefly get access to powerful GPUs/servers when they need one. If you spend $2/hr you can get access to an H100. If you have a budget of $1300 that could get you about 600 hours of compute time, which (unless you’re doing training runs) should last you several months.

    In several months time the specs required to run good models will be different again in ways that are hard to predict, so this approach can help save on the heartbreak of buying an RTX 5090 only to find that even that doesn’t help much with LLM inference and we’re all gonna need the cheaper-but-more-VRAM Intel Arc B60s.

  • by DogRunner on 6/6/25, 10:27 AM

    I used a similar budget and build something like this:

    7x RTX 3060 - 12 GB which results in 84GB Vram AMD Ryzen 5 - 5500GT with 32GB Ram

    All in a 19-inch rack with a nice cooling solution and a beefy power supply.

    My costs? 1300 Euro, but yeah, I sourced my parts on ebay / second hand.

    (Added some 3d printed parts into the mix: https://www.printables.com/model/1142963-inter-tech-and-gene... https://www.printables.com/model/1142973-120mm-5mm-rised-noc... https://www.printables.com/model/1142962-cable-management-fu... if you think about building something similar)

    My power consumption is below 500 Watt at the wall, when using LLLMs,since I did some optimizations:

    * Worked on power optimizations and after many weeks of benchmarking, the sweet spot on the RTX3060 12GB cards is a 105 Watt limit

    * Created Patches for Ollama ( https://github.com/ollama/ollama/pull/10678) to group models to exactly memory allocation instead of spreading over all available GPUs (This also reduces the VRAM overhead)

    * ensured that ASPM is used on all relevant PCI components (Powertop is your friend)

    It's not all shiny:

    * I still use PCIe3 X1 for most of the cards, which limits their capability, but all I found so far (PCIe Gen4 x4 extender and bifurcation/special PCIE routers) are just too expensive to be used on such low powered cards

    * Due to the slow PCIe bandwidth, the performance drops significantly

    * Max VRAM per GPU is king. If you split up a model over several cards, the RAM allocation overhead is huge! (See Examples in my ollama patch about). I would rather use 3x 48GB instead of 7x 12G.

    * Some RTX 3060 12GB Cards do idle at 11-15 Watt, which is unacceptable. Good BIOSes like the one from Gigabyte (Windforce xxx) do idle at 3 Watt, which is a huge difference when you use 7 or more cards. These BIOSes can be patched, but this can be risky

    All in all, this server idles at 90-100Watt currently, which is perfect as a central service for my tinkerings and my family usage.

  • by teleforce on 6/9/25, 7:53 AM

    >DECISION: Nvidia RTX 4070

    I'm curiuos why OP didn't go for the more recent Nvidia RTX 4060 Ti with 16 GB VRAM that cost cheaper (~USD500) brand new and lesser power consumption at 165W [1].

    [1] RTX 5060 Ti 16GB sucks for gaming, but seems like a diamond in the rough for AI:

    https://news.ycombinator.com/item?id=44196991

  • by politelemon on 6/6/25, 7:10 AM

    If the author is reading this I'll point out that the cuda toolkit you find in the repositories is generally older. You can find the latest versions straight from Nvidia: https://developer.nvidia.com/cuda-downloads?target_os=Linux&...

    The caveat is that sometimes a library might be expecting an older version of cuda.

    The vram on the GPU does make a difference, so it would at some point be worth looking at another GPU or increasing your system ram if you start running into limits.

    However I wouldn't worry too much right away, it's more important to get started and get an understanding of how these local LLMs operate and take advantage of the optimisations that the community is making to make it more accessible. Not everyone has a 5090, and if LLMs remain in the realms of high end hardware, it's not worth the time.

  • by Havoc on 6/9/25, 12:08 AM

    > You pay a lot upfront for the hardware, but if your usage of the GPU is heavy, then you save a lot of money in the long run.

    Last I saw data on this wasn’t true. A like for like comparison (same model and quant) API is cheaper than elec so you never make back hardware cost. That was a year ago and api costs have plummeted so I’d imagine it’s even worse now.

    Datacenters have cheaper elec, can do batch inference at scale and more efficient cards. And that’s before we consider the huge free allowances by Google etc

    Own AI gear is cool…but not due to economics

  • by vunderba on 6/6/25, 4:19 AM

    The RTX market is particularly irritating right now, even second-hard 4090s are still going for MSRP if you can find them at all.

    Most of the recommendations for this budget AI system are on point - the only thing I'd recommend is more RAM. 32GB is not a lot - particularly if you start to load larger models through formats such as GGUF and want to take advantage of system ram to split the layers at the cost of inference speed. I'd recommend at least 2 x 32GB or even 4 x 32GB if you can swing it budget-wise.

    Author mentioned using Claude for recommendations, but another great resource for building machines is PC Part Picker. They'll even show warnings if you try pairing incompatible parts or try to use a PSU that won't supply the minimum recommended power.

    https://pcpartpicker.com

  • by golly_ned on 6/6/25, 4:46 AM

    Whenever I get to a section that was clearly autogenerated by an LLM I lose interest in the entire article. Suddenly the entire thing is suspect and I feel like I’m wasting my time, since I’m lo lingering encountering the mind of another person, just interacting with a system.

  • by Jedd on 6/6/25, 5:15 AM

    In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time.

    (They probably still are, or at least pretty close to it.)

    That informed my decision shortly after, when I built something similar - that video card model was widely panned by gamers (or more accurately, gamer 'influencers'), but it was an excellent choice if you wanted 16GB of VRAM with relatively low power draw (150W peak).

    TFA doesn't say where they are, or what currency they're using (which implies the hubris of a North American) - at which point that pricing for a second hand, smaller-capacity, higher-power-drawing 4070 just seems weird.

    Appreciate the 'on a budget' aspect, it just seems like an objectively worse path, as upgrades are going to require replacement, rather than augment.

    As per other comments here, 32 / 12 is going to be really limiting. Yes - lower parameter / smaller-quant models are becoming more capable, but at the same time we're seeing increasing interest in larger context for these at home use cases, and that chews up memory real fast.

  • by T-A on 6/8/25, 10:49 PM

    I would consider adding $400 for something like this instead:

    https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto...

  • by danielhep on 6/9/25, 2:53 AM

    What are the practical uses of a self hosted LLM? Is it actually possible to approach the likes of Claude or one of the other big ones on your own hardware for a reasonable budget? I don’t know if this is something that’s actually worth it or if people are just building these rigs for fun or niche use cases that don’t require the intelligence of a hosted LLM.

  • by noufalibrahim on 6/9/25, 2:58 AM

    This is interesting. We recently built a similar machine to implement a product that we're building on a customer site.

    I didn't buy second hand parts since i wasn't sure of the quality so it was a little pricey but we have the entire thing working now and over the last week, we added the llm server to the mix. Haven't released it yet though.

    I wrote about some "fun" we had getting it together here but it's not as technically detailed as the original article.

    https://blog.hpcinfra.com/when-linkedin-met-reality-our-bang...

  • by PeterStuer on 6/9/25, 6:45 AM

    For image generation the article's setup might be viable, but do not expect to run LLM's with satisfactory quality and speed on 12GB vram.

  • by djhworld on 6/8/25, 8:20 PM

    With system builds like this I always feel the VRAM is the limiting factor when it comes to what models you can run, and consumer grade stuff tends to max out at 16GB or (somemtimes) 24GB for more expensive models.

    It does make me wonder whether we'll start to see more and more computers with unified memory architecture (like the Mac) - I know nvidia have the Digits thing which has been renamed to something else

  • by pshirshov on 6/8/25, 10:31 PM

    3090 for ~1000 is much more solid choice. Also these old mining mobos play very well for multi-gpu ollama.

  • by rcarmo on 6/6/25, 6:57 AM

    The trouble with these things is that “on a budget” doesn’t deliver much when most interesting and truly useful models are creeping beyond the 16GB VRAM limit and/or require a lot of wattage. Even a Mac mini with enough RAM is starting to look like an expensive proposition, and the AMD Stryx Halo APUs (the SKUs that matter, like the Framework Desktop at 128GB) are around $2K.

    As someone who built a period-equivalent rig (with a 12GB 3060 and 128GB RAM) a few years ago, I am not overly optimistic that local models will keep being a cheap alternative (never mind the geopolitics). And yeah, there are vey cheap ways to run inference, but hey become pointless - I can run Qwen and Phi4 locally on an ARM chip like the RK3588, but it is still dog slow.

  • by AJRF on 6/9/25, 12:36 AM

    Why a 4070 over a 3090? A 4070 has half the VRAM. In the UK you can get a 3090 for like 600GBP.

  • by uniposterz on 6/6/25, 4:25 AM

    I had a similar setup for a local LLM, 32GB was not enough. I recommend going for 64GB.

  • by jacekm on 6/8/25, 9:35 PM

    For $100 more you could get a used 3090 with twice as much VRAM. You could also get 4060 Ti which is cheaper than 4070 and it has 16 GB VRAM (although it's less powerfull too, so I guess depends on the use case)

  • by numpad0 on 6/9/25, 1:44 AM

    Couple best vram for buck && borderline space heater GPUs off top of my head: Tesla K80(12GBx2), M40(24GB), Radeon Instinct MI(25|50|60|100)(8-32GB?), Radeon Pro V340(16GBx2), bunch of other Radeon Vega 8GB cards e.g. Vega 56, NVIDIA P102/P104(~16GB), Intel A770(16GB). Note: some of these are truly just space heaters.

    I'm not sure if right now is the best timing for building an LLM rig, as Intel Arc B60(24GBx2) is about to go on sale. Or maybe it is to secure multiples of 16GB cards hastily offloaded before its launch?

  • by usercvapp on 6/8/25, 10:45 PM

    I have a server at home sitting IDLE for the last 2 years with 2 TB of RAM and 4 CPUs.

    I am gonna push it this week and launch some LLM models to see how they perform!

    How much electric bill efficient are they running locally?

  • by incomingpain on 6/6/25, 12:28 PM

    I've been dreaming on pcpartpicker.

    I think Radeon RX 7900 XT - 20 GB has been the best bang for your buck. Enables full gpu 32B?

    Looking at what other people have been doing lately, they arent doing this.

    They are getting 64+ core cpus and 512GB of ram. Keeping it on cpu and enabling massive models. This setup lets you do deepseek 671B.

    It makes me wonder, how much better is 671B vs 32B?

  • by lazylizard on 6/9/25, 5:37 AM

  • by zlies on 6/9/25, 7:25 AM

    Did you not use any thermal paste at all, or did you just forget to mention it in your post?

  • by v5v3 on 6/6/25, 7:06 AM

    I thought prevailing wisdom was that a used 3090 with it's larger vram was the best budget gpu choice?

    And in general, if on a budget then why not buy used and not new? And more so as the author himself talks about the resale value for when he sells it on.

  • by msp26 on 6/8/25, 9:45 PM

    > 12GB vram

    waste of effort, why would you go through the trouble of building + blogging for this?

  • by alganet on 6/9/25, 8:28 AM

    Let me try to put this in the scale of coffee:

    --

    Using LLM via api: Starbucks.

    Inference at home: Nespresso capsules.

    Fine-tune a small model at home: Owning a grinder and an italian espresso machine.

    Pre-training a model: Owning a moderate coffee plantation.

  • by eachro on 6/9/25, 10:11 PM

    A lot of people are saying 12gb is too small to do anything interesting with. What's the most useful thing people __have__ gotten to work?

  • by v3ss0n on 6/9/25, 1:22 PM

    12GB GPU can't do a thing that is useful. Minium should be 32GB VRam where you can run actual models (Mistral-Small , Qwen3-32B , etc).

  • by atentaten on 6/8/25, 8:27 PM

    Enjoyed the article as I am interested in the same. I would like to have seen more about the specific use cases and how they performed on the rig.

  • by ww520 on 6/8/25, 8:43 PM

    I use a 10-year old laptop to run a local LLM. The time between prompts are 10-30 seconds. Not for speedy interactive usage.

  • by iJohnDoe on 6/8/25, 9:15 PM

    Details about the ML software or AI software?

  • by ntlm1686 on 6/9/25, 8:34 PM

    Building a PC that can play video games and run some LLMs.

  • by whalesalad on 6/8/25, 11:32 PM

    I would rather spend $1,300 on openai/anthropic credits. The performance from that 4070 cannot be worth the squeeze.

  • by burnt-resistor on 6/6/25, 9:55 AM

    Reminds me of https://cr.yp.to/hardware/build-20090123.html

    I'll be that guy™ that says if you're going to do any computing half-way reliably, only use ECC RAM. Silent bit flips suck.