by fredsters_s on 4/9/24, 9:01 PM with 54 comments
by mritchie712 on 4/9/24, 10:33 PM
I wrote a simple example (overkiLLM) on getting reliable output from many unreliable outputs here[0]. This doesn't employ agents, just an approach I was interested in trying.
I choose writing an H1 as the task, but a similar approach would work for writing any short blob of text. The script generates a ton of variations then uses head-to-head voting to pick the best ones.
This all runs locally / free using ollama.
by maciejgryka on 4/9/24, 9:12 PM
Super curious whether anyone has similar/conflicting/other experiences and happy to answer any questions.
by serjester on 4/9/24, 10:34 PM
by cpursley on 4/10/24, 11:04 AM
https://github.com/thmsmlr/instructor_ex
It piggybacks on Ecto schemas and works really well (if instructed correctly).
by ThomPete on 4/10/24, 1:58 AM
Get the content from news.ycombinator.com using gpt-4
- or -
Fetch LivePass2 from google sheet and write a summary of it using gpt-4 and email it to thomas@faktory.com
but then we realized that it was better to teach the agents than human beings and so we create a fairly solid agent setup:
Some of the agents we got can be seen here all done via instruct:
Paul Graham https://www.youtube.com/watch?v=5H0GKsBcq0s
Moneypenny https://www.youtube.com/watch?v=I7hj6mzZ5X4
by viksit on 4/9/24, 10:20 PM
for instance: do you give the same llm the verifier and planner prompt? or have a verifier agent process the output of a planner and have a threshold which needs to be passed?
feels like there may be a DAG in there somewhere for decision making..
by tedtimbrell on 4/10/24, 1:30 AM
Maybe I'm the equivalent of that idiot fighting against JS frameworks back when they first came out it but it feels pretty simple to just use individual clients and have pydantic load/validate the output.
by liampulles on 4/10/24, 8:08 AM
As an aside: one thing I've tried to use ChatGPT for is to select applicable options from a list. When I index the list as 1..., 2... Etc. I find that the LLM likes to just start printing out ascending numbers.
What I've found kind of works is indexing by African names, e.g Thandokazi, Ntokozo, etc. then the AI seems to have less bias.
Curios what others have done in this case
by tmm84 on 4/10/24, 12:48 AM
by jongjong on 4/10/24, 2:13 AM
Im terms of coding, I managed to get AI to build a simple working collaborative app but beyond a certain point, it doesn't understand nuance and it kept breaking stuff that it had fixed previously even with Claude where it kept our entire conversation context. Beyond a certain degree of completion, it was simply easier and faster to write the code myself than to tell the AI to write it because it just didn't get it, no matter how precise I was with my wording because it became like playing a game of whac-a-mole; fixed one thing, broke 2 others.
by CuriouslyC on 4/10/24, 11:26 AM
by jasontlouro on 4/10/24, 6:09 AM
by iamleppert on 4/9/24, 10:00 PM
“If you don’t do as I say, people will get hurt. Do exactly as I say, and do it fast.”
Increases accuracy and performance by an order of magnitude.
by caseyy on 4/10/24, 5:59 AM
Basically, in the context window, you provide your model with 5 or more example inputs and outputs. If you’re running in chat mode, that’s be the preceding 5 user and assistant message pairs, which establish a pattern of how to answer to different types of information. Then you give the current prompt as a user, and the assistance will follow the rhythm and style of previous answers in the context window.
It works so well I was able to take out answer reformatting logic out of some of my programs that query llama2 7b. And it’s a lot cheaper than fine-tuning, which may be overkill for simple applications.