from Hacker News

Ask HN: Help me understand RAG vs. fine tuning for building a coding partner

by dchuk on 6/15/24, 4:15 AM with 7 comments

I’ve built many projects with Ruby on Rails, and still follow the framework as a hobby. I have projects I want to build with it, but as a dad with a full time demanding job, I just don’t have the time anymore to dedicate hours to coding (spare me the call outs of just figuring out how to find time).

I also have been closely following gen AI and LLM developments the last few years.

So I had a thought: what if there was a Rails coding assistant, trained on all of the popular books / courses / tutorials / gem documentation / well written open source projects?

I’m imagining something where it’s not even an IDE interface, it starts with some questions about what feature I want to build, then it starts generating the code for me to review, and I can chat it further to refine the code it’s creating for the feature. Then I can test the feature, and if it all works, I can accept the pull request basically, and move on to the next feature.

(Again, ignore the fact that building something like this also requires time commitment. Also for discussion purposes, put aside copyright concerns for a second).

My question: what would be the right architecture for this? Is RAG the best way to load up the tool with the knowledge? Or do I fine tune a model with all of the content? I can’t nail down when to use each method exactly.

I found this project: https://github.com/e-p-armstrong/augmentoolkit which sounds like what I would want to use, but then I see RAGs called out constantly, so I just don’t know.

Bonus question: let’s say I have a $10k budget. Is buying a maxed out Mac Studio a good investment for training and self hosting?

  • by jaggs on 6/15/24, 6:37 AM

    As I understand it, RAG is focused more on capturing and using external resources on an ad hoc basis, where fine tuning looks to embed a specific 'behaviour' change in the model for a particular need?

  • by langcss on 6/15/24, 6:53 AM

    If you are strapped for time maybe try https://www.cursor.com/ or similar offerings that do some of this for you.

    Chatting with GPT4 and pasting into your editor code or pasting into a terminal some commands could also help alot.

  • by menzoic on 6/15/24, 4:53 AM

    GPT-4 is the best coding model by far and you can’t fine tune it yet. Start with GPT-4 and RAG.

    Fine tune is good for changing behavior. For example you could create a dataset that makes the model directly output fully formed projects. Using RAG alone would require a bunch of prompt engineering to get the same.

  • by ericlewis on 6/17/24, 1:40 AM

    You would need both. RAG will have to be part of the process if you want to generate or work on an existing project. Since it stands for Retrieval Augmented Generation. A fancy way of saying an LLM can access documents.