from Hacker News

A guidance language for controlling LLMs

by evanmays on 5/16/23, 4:14 PM with 190 comments

by ubj on 5/16/23, 6:43 PM
I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees.
I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

by simonw on 5/16/23, 5:59 PM

This is pretty fascinating, but I'm not sure I understand the benefit of using a Handlebars-like DSL here.

For example, given this code from https://github.com/microsoft/guidance/blob/main/notebooks/ch...

    create_plan = guidance('''{{#system~}}
    You are a helpful assistant.
    {{~/system}}
    {{#block hidden=True}}
    {{#user~}}
    I want to {{goal}}.
    {{~! generate potential options ~}}
    Can you please generate one option for how to accomplish this?
    Please make the option very short, at most one line.
    {{~/user}}
    {{#assistant~}}
    {{gen 'options' n=5 temperature=1.0 max_tokens=500}}
    {{~/assistant}}
    {{/block}}
    {{~! generate pros and cons and select the best option ~}}
    {{#block hidden=True}}
    {{#user~}}
    I want to {{goal}}.
    ''')

How about something like this instead?

    create_plan = guidance([
        system("You are a helpful assistant."),
        hidden([
            user("I want to {{goal}}."),
            comment("generate potential options"),
            user([
                "Can you please generate one option for how to accomplish this?",
                "Please make the option very short, at most one line."
            ]),
            assistant(gen('options', n=5, temperature=1.0, max_tokens=500)),
        ]),
        comment("generate pros and cons and select the best option"),
        hidden(
            user("I want to {{goal}}"),
        )
    ])

by ryanklee on 5/16/23, 5:29 PM
I'm personally starting with learning Guidance and LMQL rather than LangChain just in order to get a better grasp of the behaviors that I've gathered LangChain papers over. Even after that, I'm likely to look at Haystack before LangChain.
Just getting the feeling that LangChain is going to end up being considered a kitchen sink solution full of anti patterns so might as well spend time a little lower level while I see which way the winds end up blowing.
by ntonozzi on 5/16/23, 4:54 PM
How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040, but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
by bjackman on 5/16/23, 7:22 PM
Wow I think there are details here I'm not fully understanding but this feels like a bit of a quantum leap* in terms of leveraging the strengths while avoiding the weaknesses of LLMs.
It seems like anything that provides access to the fuzzy "intelligence" in these systems while minimizing the cost to predictability and efficiency is really valuable.
I can't quite put it into words but it seems like we are gonna be moving into a more hybrid model for lots of computing tasks in the next 3 years or so and I wonder if this is a huge peek at the kind of paradigms we'll be seeing?
I feel so ignorant in such an exciting way at the moment! That tidbit about the problem solved by "token healing" is fascinating.
*I'm sure this isn't as novel to people in the AI space but I haven't seen anything like it before myself.
by Der_Einzige on 5/16/23, 6:33 PM
There has been a huge explosion of awesome tooling which utilizes constrained text generation.
Awhile ago, I tried my own hand at constraining the output of LLMs. I'm actively working on this to make it better, especially with the lessons learned from repos like this and from guidance
https://github.com/hellisotherpeople/constrained-text-genera...
by ftxbro on 5/16/23, 5:33 PM
Will it still be all like "As an AI language model I cannot ..." or can this fix it? I mean asking to sexy roleplay as Yoda isn't the same level as asking how to discreetly manufacture methamphetamine at industrial scale there are levels people
by m3kw9 on 5/16/23, 7:10 PM
I’m not understanding how Guidence Accelerating works. It says “ This cuts this prompt's runtime in half vs. a standard generation approach.” and it gives an example of it asking LLM to generate json. I don’t see anywhere how it accelerates anything because it’s a simple json completion call. How can you accelerate that?
by EddieEngineers on 5/16/23, 7:59 PM
What's with all these weird-looking projects with similar names using Guidance?
https://github.com/microsoft/guidance/network/dependents
They don't even appear to be using Guidance anywhere anyway
https://github.com/IFIF3526/aws-memo-server/blob/master/requ...
by indus on 5/16/23, 6:58 PM
This reminds me of the time when I wrote a cgi script.
Basically instructing the templating engine (a very crude regex) to replace session variables, database lookups to the merge fields:
Hello {{firstname}}!
1996 and 2023 smells alike.
by BeefySwain on 5/16/23, 11:32 PM
Using Mustache instead of Jinja for a Python package is a choice
by ahnick on 5/16/23, 5:21 PM
This strikes me as being very similar to Jargon (https://github.com/jbrukh/gpt-jargon), but maybe more formal in its specification?
by iamflimflam1 on 5/17/23, 6:41 AM
Very Interesting. One of the big challenges with LLMs is getting well formed JSON output. GPT4 is much better at this. But is very expensive. So anything that can help is good. Looking forward to trying this out locally with LLAMA.
by alexb_ on 5/16/23, 5:34 PM
I hope this becomes extremely popular, so that anyone who wants to can completely decouple this from the base model and actually use LLMs to their full potential.
by sharemywin on 5/16/23, 4:51 PM
It does look like it makes easier to code against a model. But, is this supposed to work along side lang-chain or hugging face agents or as an alternative to?
by sheepscreek on 5/17/23, 4:01 AM
It is in the same spirit as Maven AI, but takes a slightly different approach. Great to see the progress in this space!
by m3kw9 on 5/16/23, 7:16 PM
There should be a standard template/language to structurally prompt LLMs. Once that is good, all good LLMs should use the doc to fine tune it to take in that standard. Right now each model has their own little way to best prompt it and you end up needing programs like this to sit in between and handle it for you
by Spivak on 5/16/23, 7:03 PM
I think it's cool that a company like Microsoft is willing to base a real-boy product on pybars3 which is its author's side-project instead of something like Jinja2. If this catches on I can imagine MS essentially adopting the pybars3 project and turning it into a mature thing.
by amkkma on 5/16/23, 7:37 PM
How does this compare with lmql?
by jhoffbauer on 5/19/23, 9:58 AM
How does this compare to LMQL (https://lmql.ai/)?
by candiddevmike on 5/16/23, 5:31 PM
Will there be a tool to convert natural language into Guidance?
by jxy on 5/16/23, 6:26 PM
They must hate lisp so much that they opt to use {{}} instead.
by imgi456 on 5/17/23, 12:35 AM
Desperate approach from microsoft to gain market share of langchain.
by nico on 5/16/23, 6:36 PM
It’s so amazing to see how we are essentially trying to solve “programming human beings”
Although on the other hand, that’s what social media and smartphones have already done
Maybe AI already took over, doesn’t seem to be wiping out all of humanity
by wahnfrieden on 5/17/23, 1:11 PM
Could this language be used outside of notebooks?
by rain1 on 5/16/23, 6:36 PM
Does this do one query per {{}} thing?
by obiefernandez on 5/17/23, 6:24 AM
Can this be used with OpenAI APIs?
by marcopicentini on 5/16/23, 8:11 PM
What’s the best practice to let an existing Ruby on Rails application use this python framework?
by Animats on 5/16/23, 6:45 PM
Is this a "language", or just a Python library?
by wprl on 5/17/23, 11:22 AM
The hubris of mutating spiritual texts as a marketing gimmick is contemptible.