from Hacker News

Language models as compilers: Simulating pseudocode execution

by milliondreams on 4/4/24, 7:46 PM with 52 comments

• by pkoird on 4/4/24, 11:39 PM

  Any sufficiently advanced LLM is indistinguishable from Prolog.

  I half-jest but I envision the direction of LLM research to head towards a parser-oriented setup where LLMs merely extract the entities and relations and the actual logic is done by a logical engine such as Prolog.

• by Voultapher on 4/5/24, 6:30 AM

  Non deterministic compilers, yay! Where do I sign up?

  In more seriousness, miscompilations or in general unexpected behavior caused by layers below you are expensive to find and fix. I think LLMs have a long way to go before such use cases seem appealing to me.

• by imranq on 4/5/24, 12:18 PM

  Reading the paper, the connection to compilers is more of an analogy rather than a direct technical link.

  The authors propose using an LLM to reframe the task as high level psuedocode, and then reason on that code on the specific details of the task

  No compilers were used or compiled - no real code was generated or executed. Its just the idea that a programming language syntax has good structure to process details, and a way to interpret some of the results. Many of the other comments here seem like they didn't read the paper at all and are reacting to the headline

• by novideogame on 4/5/24, 10:32 AM

  I think the title is a little misleading. The main difference between this paper and CoC (Chain of Code) is that the LLM is instructed to make a plan to solve all the given instances and then code that plan in pseudocode, while in CoC the plan is to solve the single instance given.

  From the paper: The main difference between THINK-AND-EXECUTE and CoC is that we use pseu- docodes which are generated to express logic shared among the tasks instances, while CoC incorporates pseudocode as part of the intermediate reasoning steps towards the solution of a given instance. Hence, the results indicate the advantages of applying pseudocode for the generation of task-level instruction over solely using them as a part of rationales.

  I find the phrase "as a part of rationales" a little strange, but English is not my native language.

• by Mathnerd314 on 4/4/24, 9:01 PM

  The phase 2 prompt is complete, but the phase 3 prompt's initial part ends in "When constructing the main function, ...", and no mention of random seeds, so I guess this paper is not reproducible at all.

• by m3kw9 on 4/5/24, 1:49 AM

  If you train a LLM to compile, you probably also want to set the randomness to zero, if that is the case you’ve just “brute forced” an actual compiler

• by spxneo on 4/4/24, 8:36 PM

  This seems quite promising. Using pseudo-code as an intermediary step isn't new but seems like this takes it a bit further. Will need to see some code and test it out.

• by jumploops on 4/5/24, 2:03 AM

  English is terribly imprecise, so it makes sense to use pseudo instructions to improve the bounds/outcome of a language model’s execution.

  I do wonder how long hacks like this will be necessary; as it stands, many of these prompting techniques are essentially artificially expanding the input to enhance reasoning ability (increasing tokens, thus increasing chance of success).

• by ingigauti on 4/5/24, 10:36 AM

  Couple of weeks ago I published a new programming language called Plang (as in pseudo language) that uses LLM to translate user intent into executable code, basically LLM as a compiler.

  It saves you incredible amount of work, cutting code writing down by 90%+. The built code is deterministic(it will never change after build) and as a programmer you can validate the code that will be executed. It compiles to C#, so it handles GC, encoding, etc. that languages need to solve, so I can focus on other areas.

  Plang also has some features that other language don't have, e.g. events on variables, built in identity and interesting(I think) approach to privacy.

  I have not been advertising to much since it is still early development and I create still to many breaking changes, but help is welcome(and needed) so if it something that is interesting to you the repo is at https://github.com/plangHQ

• by inciampati on 4/4/24, 10:39 PM

  It's going to be really fascinating to see this applied instead of chain of thought and other kinds of reasoning approaches, because it's generic. It should in principle work on every kind of LLM.

• by eeue56 on 4/5/24, 10:20 AM

  I wrote a toy language along these lines a while back[0]. Basically, types and function signatures, with comments in English, produce a valid program. You write a type and a comment, and the compiler goes through GPT to run the code. Fun novel idea.

  [0] - https://github.com/eeue56/neuro-lingo

• by lionkor on 4/5/24, 7:47 AM

  How is it better than a compiler written by people?

• by emmender2 on 4/5/24, 2:54 PM

  Researchers are trying their damndest to build a "reasoning" layer using LLMs as the foundation. But, they need to go back to the drawing-board and understand from first principles what it means to reason. For this in my view, they need to go back to epistemology (and refer to Peirce and logicians like him).

• by skeledrew on 4/5/24, 12:42 AM

  Seeing this makes me want to reactivate an old project[0]. Been thinking more and more that LLMs could give it superpowers.

  [0] https://pypi.org/project/neulang/

• by 29athrowaway on 4/5/24, 6:09 PM

  Up next: A LLM that can tell me if a program stops