from Hacker News

Autodoc: Toolkit for auto-generating codebase documentation using LLMs

by funfunfunction on 3/25/23, 12:44 AM with 86 comments

  • by SilverBirch on 3/25/23, 10:10 AM

    I have a conceptual problem with this. Documentation is meant to describe stuff thats's not in the code. Sure, there's the odd occasion where you've done some super weird optimization where you want to say "Over here I've translated all my numbers into base 5, bit reversed them and then added them, mathematically this is the same as just adding them in base 9, but fits our custom logic cell better". But that's the exception, the general purpose of documentation is to describe why it's doing hwat it's doing, not how. Tell me that this module does X this way because that helps this other module do Y". Tell me why you've divided the problem this way. You're giving information about why certain design decisions were made, not just describe what they are.

    It doesn't matter how good your LLM is, the information simply isn't there for it to know the information it needs to document. You're never going to get a comment out of this that says "This interface is meant to be backwards compatible with the interface Bob once wrote on a napkin in the pub on a particularly quiet friday afternoon when he decided to reinvent Kafka".

  • by divan on 3/25/23, 8:10 AM

    Just tested it on a side-project codebase.

    Main impression - it does hallucinate like crazy. I asked "How does authorization of HTTP request work?" and it started spitting explanation of how user bcrypt hash is stored in SQlite database and token is stored in Redis cache. There are no signs of SQLite or Redis whatsover on this project.

    In other query it started confidently explaining how `getTeam` and `createTeam` functions work. There are no such entity or a word "team" in the entire codebase. To add to the insult, it said that this whole team handling logic is stored in `/assets/sbadmin2/scss/_mixins.scss`.

    Other time it offered extremely detailed explanation of some business-logic related question, linking to a lot of existing files from the project, but that was completely off.

    Sometimes it offered meaningful explanations, but was ignoring the question. Like I ask to explain relation between two entities and it started showing how to display that entity in a HTML template.

    But I guess it's just a question of time when tools like this become a daily assistant. Seems invaluable for the newcomers to the codebase.

  • by ch33zer on 3/25/23, 2:38 AM

    Most interesting part to me, the prompts:

    https://github.com/context-labs/autodoc/blob/83f03a3cee62d6e...

    > You are acting as a code documentation expert for a project called ${projectName}. Below is the code from a file located at \`${filePath}\`. Write a detailed technical explanation of what this code does. Focus on the high-level purpose of the code and how it may be used in the larger project. Include code examples where appropriate. Keep you response between 100 and 300 words. DO NOT RETURN MORE THAN 300 WORDS. Output should be in markdown format. Do not say "this file is a part of the ${projectName} project". Do not just list the methods and classes in this file. Code: ${fileContents} Response:

  • by hedora on 3/25/23, 2:27 AM

    This has far surpassed my dystopian predictions of how people would misuse LLMs.

    Self spamming your own code base with comments that are either obvious, misleading or wrong was previously unfathomable to me.

    Most people think I’m unrealistically pessimistic.

    Well done.

  • by andrewmcwatters on 3/25/23, 5:22 AM

    I'm far, far more interested in having an LLM tell me where particular functionality is in a codebase, and how it works from a high level.

    Autogenerating function documentation seems like such a low bar by comparison. It's like taking limited creativity and applying it with high powered tools.

    Literally like asking for a faster horse.

    Tell me how WebKit generates tiles for rasterizing a document tree. Show me specifically where it takes virtualized rendering commands and translates them into port specific graphics calls.

    Show me the specific binary format and where it is written for Unreal Engine 5 .umaps so that I can understand the embedded information for working between different types of software or porting to different engines.

    Some codebases are so large that it literally doesn't matter if individual functions are documented when you have to build a mental model of several layers of abstraction to understand how something works.

  • by golem14 on 3/25/23, 7:19 AM

    Today, LLMs learn from a codebase that has mostly insightful comments from well-meaning humans.

    In the future, the training sets will contain more and more automatically generated stuff I believe will not be curated well, leading to a spiral of ever declining quality.

  • by ch33zer on 3/25/23, 4:29 AM

    One thing I find worse than no docs is wrong docs.

    It would be really cool if we could take code + docs, feed it into an LLM and get a determination of whether the code matches what's in the docs. It could also be a good way to evaluate the correctness of the generated docs from the linked tool (assuming it works).

  • by userbinator on 3/25/23, 3:11 AM

    If docs can be generated, they're not worth reading.

  • by rkagerer on 3/25/23, 3:54 AM

    Come talk to me when it can reverse engineer as good as Ken Shirriff, develop a complete understanding of the whole codebase, and generate authoritatively accurate and useful output. Oh, and uncover bugs while it's at it.

  • by zx8080 on 3/25/23, 2:27 AM

    How to verify the meaning of docs? How to deal with the model hallucinations?

    It would be hell to lose trust to api docs due to those risks.

  • by petesergeant on 3/25/23, 3:38 AM

  • by verdverm on 3/25/23, 2:31 AM

    Please run this on the BuildKit code base which has almost no comments but a huge usage footprint. It's also (obviously) a non-trivial test case

  • by smrtinsert on 3/25/23, 3:26 PM

    I'm more interested for LLMs to get the point when it can look at the several hundred codebases in your company and tell you who sets what value in your local data model, and why they set it. There's always slack instead of poorly generated documentation.

  • by splatzone on 3/25/23, 2:40 AM

    This is extremely interesting. I have a few monstrous repositories I’d like to try it on.

    The thing I’m wondering about is the cost. How much would it cost to run this on the entire WordPress source, for example?

  • by whiplash451 on 3/25/23, 11:10 AM

    Very cool stuff.

    I think people who dismiss this kind of tool because it can hallucinate stuff are off topic.

    The AI will get better and better, but more importantly we will evolve and learn to work with this kind of tool.

  • by ImageDeeply on 3/25/23, 3:01 AM

    Granting that some hallucination is likely, still seems like a step in the right direction.