from Hacker News

There were some interesting challenges we encountered using an LLM to fix breaking changes:

[1] Finding the breaking changes → know how is a dependency used through the codebase

[2] Knowing how to fix the a breaking change → know how the dependency has changed from one version to another

[3] Understanding how the fix has modified existing behavior → know how the function the dependency is used in is used through the codebase

We addressed challenge [1] and [2] by choosing to support Typescript first. The strong typing helps us identify the breaking changes with a version bump pretty well. We can then pass the type errors into the LLM to increase its fix accuracy.

Challenge [3] is definitely the trickiest. We built an AST of the codebase that we then use to create a “plan graph” (https://huggingface.co/papers/2309.12499) of the type error, the function it’s used in, and the functions calling it. This plan graph should tell us how a fix would need to be perpetuated throughout the codebase.

At a high level bumpgen’s core loop is something like this:

- build an AST to understand your code’s relationships

- bump version and get type errors

- combine the above to create a plan graph (https://huggingface.co/papers/2309.12499) of how to fix the error and apply it to the rest of the codebase

- prompt the LLM to step through the plan graph and fix each breaking change

- validate the fixes with a rebuild

We also built our own benchmark suite (https://github.com/xeol-io/swe-bump-bench) to test bumpgen’s accuracy. It is a set of repos with human commits for version bumps. We would run bumpgen on the prior commit then compare bumpgen’s PR to that of the human commit to determine success. Our latest benchmark sits around ~50% accuracy.

Up next we want to better address challenge [2] by building embeddings for different dependency versions to give the LLM more context on the changes across versions. After that we will be releasing a GitHub app with some qol features such as configuring update cadences, etc before moving onto C# and Golang support.

We covered a lot of our architecture and thought process for bumpgen, we would love to hear your thoughts and feedback in the comments!