from Hacker News

LegoGPT: Generating Physically Stable and Buildable Lego

by nkko on 5/9/25, 4:55 AM with 136 comments

• by haberman on 5/9/25, 6:59 AM

  > To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints.

  I'm far from an AI expert, but I've long felt that this is one of the most interesting ways to use AI: to generate and optimize possibilities within a set of domain-specific constraints that are programmed manually.

  For example, imagine an AI that is designed to optimize traffic light patterns. You want a hard constraint that no intersection gives a combination of green lights that could cause collisions. But within that set of constraints, which you could manually specify, the AI could go wild trying whatever ideas it can come up with.

  At that point, the interesting work is deciding how to design the problem space and the set of constraints. In this case it's a set of lego bricks and how they can be built (and be stable).

• by sschueller on 5/9/25, 6:40 AM

  This is probably going to get a letter from LEGO's lawyers.

  If you want to be safe do not use the word LEGO. Use Bricks or in German "Klemmbausteine".

  Many people have had to deal with LEGO's lawyers and it ain't pretty.

• by stevage on 5/9/25, 8:42 AM

  This does not seem like a very impressive result. It's using such a small set of bricks and the results don't really look much like the intended thing.

  It feels like a hand-crafted algorithm would get a much better result.

• by jader201 on 5/9/25, 6:06 AM

  There’s a bug on the page (on iPhone, at least) once you scroll to the gifs that it starts to auto load them without doing anything, making it really hard to navigate anywhere at that point.

• by RaSoJo on 5/9/25, 9:26 AM

  I don’t need automation to build LEGO sets — that’s the fun part, and I want to do it myself. What I need is automation after the build: to clean up, sort the bricks by color and shape, and store them properly.

  I just wish scientists would start by solving problems that actually exist in the real world. There’s real value — and real money — in that.

• by yathaid on 5/9/25, 6:26 AM

  This is super cool! The GIFs showing the object being built are just yummy; I have no other way to describe it.

  If anyone else was searching for the dataset, it is at https://huggingface.co/datasets/AvaLovelace/StableText2Lego

  It contains " contains 47,000+ different LEGO structures, covering 28,000+ unique 3D objects from 21 common object categories of the ShapeNetCore dataset".

  Local inference instructions are over at their github page - https://github.com/AvaLovelace1/LegoGPT/?tab=readme-ov-file

• by psiops on 5/9/25, 7:47 AM

  I noticed that "a basic sofa" involves some placing some floating bricks if built in the order of the animation. It hints at the way this model generates the designs. The automated assembly of generated LEGO structures using robots would have serious trouble creating these designs I reckon.

• by gilgoomesh on 5/9/25, 6:15 AM

  It's hilarious watching $50,000 worth of robots take so long to assemble a couple dollars worth of Lego. It's like peering into the old folks home for robots.

• by kilimounjaro on 5/9/25, 6:56 AM

  Doesn’t seem to add much to just converting a 3d model into voxels and therefore bricks.

  Using bricks other than 2x2 and 2x4 blocks creatively to make interesting things is really important, i’m not sure what type if algorithm would best auto generate beautiful MOCs however? Was thinking of doing a $50000 kaggle comp for this, what do others think?

• by carstenhag on 5/9/25, 6:34 AM

  Have the authors never heard of Lego being one of the companies that are super strict about their trademark? They file takedown notices etc on every project they see. Even if the stone design has the little thingies on top/bottom...

• by W0lfEagle on 5/9/25, 6:35 AM

  Great. Please do cabinets next. Constrain to some specified material such as 2.5m by 1.25m 18mm ply. Iterate designs by text and output the model, cutlist and assembly instructions. Simple right?

• by dwighttk on 5/9/25, 11:05 AM

  Quit trying to read the article after the 15th video went to full screen and had to be dismissed hitting the tiny x in the upper left… 3 more interfered with me trying to go back to this page

• by 9dev on 5/9/25, 6:35 AM

  When I was a kid, I proudly exclaimed I wanted to become a professional lego builder. Not in my wildest dreams would I have assumed how close to that career path I could have come.

• by soared on 5/9/25, 1:14 PM

  The high backed chair gif example is interesting - the way it’s animated it would completely fall apart and be unstable. But if you built it in reverse, it would work fine.

  But it also shows the weirdness of the solution - in places where larger bricks make sense, multiple smaller bricks are used instead. In a section where a 2x6 should be repeated, in on instance of the repetition it uses tow 1x6s. It’s weird.

  Cool idea.

• by 6stringmerc on 5/9/25, 4:44 PM

  Is this use of the trademark approved by the LEGO owners by any chance? I skimmed it and there didn’t seem to be an indication this has been endorsed by the company.

  It kind of makes me suspicious of the integrity of Carnegie Mellon if they will allow trademark infringement of this type because, well, it does make me feel like I can shit in a bag and call it a Carnegie Mellon Socking Stuffer without consequence.

• by oaiey on 5/9/25, 8:12 AM

  Real challenge are not LEGO System pieces but LEGO Technic pieces where you do not have to build layers bottom-up but everything inside out

• by z3t4 on 5/9/25, 10:41 AM

  The results are a bit underwhelming, considering what I'm used to see in image generation, world generation in games, etc.

• by yunusabd on 5/9/25, 6:57 AM

  Cool project, but judging from the videos, it looks like some of them can't actually be built using those instructions. E.g. "A backless bench with armrest" would require some bricks to float in the air with no support while you're assembling the rest.

• by vladde on 5/9/25, 7:36 AM

  Looks amazing, with the arms building the piece!

  However, the model "A high-backed chair" has some floating pieces in the middle of the seat, that are fastened from above. Can these robots handle building these?

• by necovek on 5/9/25, 12:05 PM

  I love this: it'd be great if it also worked to give age-appropriate designs as well (eg. 5 yo has less patience and ability than an 8 yo).

• by andyjohnson0 on 5/9/25, 6:27 PM

  Its been a long day and and I can think of nothing to add other than that those blue robot arms are both calming and visually pleasant.

• by josefx on 5/9/25, 7:55 AM

  Does it have a sense of scale? Can it make a matching bookcase, table and chair or are they all going to end up with a random size?

• by belter on 5/9/25, 11:32 AM

  Can I have IKEA-GPT ? This weekend if possible?

• by Traubenfuchs on 5/9/25, 7:01 AM

  Is this aligned, safeguarded, censored?

  Can it produce an ample bossom made of lego? And indecent protrusion? Weapons?

• by sargstuff on 5/10/25, 6:46 PM

  ?? 'type'-less / block structured language/design ??

• by londons_explore on 5/9/25, 8:40 AM

  > voxelizing it onto a grid and applying legolization

  I guess I learned a word today...

• by m3kw9 on 5/9/25, 3:03 PM

  These legos doesn't look very intricate even by 200 pc box standards.

• by xeyownt on 5/9/25, 7:19 AM

  So, besides training a LLM to generate build instructions for lego model, they have robots to assemble these models, and they applied 3D texture on 3D generated model (what for?).

  Sometimes the amount of money and energy that are spent in "recreation" projects just amazes me.

• by benob on 5/9/25, 6:20 AM

  It looks like it is an extension of 3d model generation techniques.

• by serial_dev on 5/9/25, 8:14 PM

  It would make sense for a car to have proper LEGO wheels.

• by nurettin on 5/9/25, 6:47 AM

  This could prove useful for minecraft bot scene