from Hacker News

Kolmogorov-Arnold Networks

by sumo43 on 5/1/24, 3:30 AM with 142 comments

  • by GistNoesis on 5/1/24, 12:25 PM

    I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer :

    https://github.com/GistNoesis/FourierKAN/

    The core is really just a few lines.

    In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of individual coordinates.

    It should give an idea of Kolmogorov-Arnold networks representation power, it should probably converge easier than their spline version but spline version have less operations.

    Of course, if my code doesn't work, it doesn't mean theirs doesn't.

    Feel free to experiment and publish paper if you want.

  • by krasin on 5/1/24, 6:40 AM

    I've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]).

    It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays around 3e-1. Getting to larger network sizes does not help either.

    I would really like to see a bigger example with many more parameters and more data complexity and if it could be trained at all. MNIST would be a good start.

    Update: I increased the training dataset size 100x, and that helps with the overfitting, but now I can't get training loss below 1e-2. Still iterating on it; a GPU acceleration would really help - right now, my progress is limited by the speed of my CPU.

    1. https://github.com/KindXiaoming/pykan/blob/master/tutorials/...

  • by esafak on 5/2/24, 1:09 AM

    There exists a Kolmogorov-Arnold inspired model in classical statistics called GAMs (https://en.wikipedia.org/wiki/Generalized_additive_model), developed by Hastie and Tibshirani as an extension of GLMs (https://en.wikipedia.org/wiki/Generalized_linear_model).

    GLMs in turn generalize logistic-, linear and other popular regression models.

    Neural GAMs with learned basis functions have already been proposed, so I'm a bit surprised that the prior art is not mentioned in this new paper. Previous applications focused more on interpretability.

  • by montebicyclelo on 5/1/24, 8:04 AM

    The success we're seeing with neural networks is tightly coupled with the ability to scale - the algorithm itself works at scale (more layers), but it also scales well with hardware, (neural nets mostly consist of matrix multiplications, and GPUs have specialised matrix multiplication acceleration) - one of the most impactful neural network papers, AlexNet, was impactful because it showed that NNs could be put on the GPU, scaled and accelerated, to great effect.

    It's not clear from the paper how well this algorithm will scale, both in terms of the algorithm itself (does it still train well with more layers?), and ability to make use of hardware acceleration, (e.g. it's not clear to me that the structure, with its per-weight activation functions, can make use of fast matmul acceleration).

    It's an interesting idea, that seems to work well and have nice properties on a smaller scale; but whether it's a good architecture for imagenet, LLMs, etc. is not clear at this stage.

  • by cs702 on 5/1/24, 1:58 PM

    It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transformers, so we may well need new alternatives.[a]

    The best thing about this new work is that it's not an either/or proposition. The proposed "learnable spline interpolations as activation functions" can be used in conventional DNNs, to improve their expressivity. Now we just have to test the stuff to see if it really works better.

    Very nice. Thank you for sharing this work here!

    ---

    [a] https://news.ycombinator.com/item?id=40179232

  • by mxwsn on 5/1/24, 6:32 AM

    From the preprint - 100 input dimensions is considered "high", and most problems considered have 5 or fewer input dimensions. This is typical of physics-inspired settings I've seen considered in ML. The next step would be demonstrating them on MNIST, which, at 784 dimensions is tiny by modern standards.

  • by ubj on 5/1/24, 1:09 PM

    Very interesting! Kolmogorov neutral networks can represent discontinuous functions [1], but I've wondered about how practically applicable they are. This repo seems to show that they have some use after all.

    [1]: https://arxiv.org/abs/2311.00049

  • by reynoldss on 5/1/24, 1:20 PM

    Perhaps a hasty comment but linear combinations of B-splines are yet another (higher-degree) B-spline. Isn't this simply fitting high degree B-splines to functions?

  • by Lichtso on 5/1/24, 6:21 PM

    1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago:

    1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...

    1958: https://en.wikipedia.org/wiki/Multilayer_perceptron

    2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has three classes of parameters (weights, biases, and the globally uniform activation function).

    3. Everybody is talking transformers. I want to see diffusion models with this approach.

  • by cbsmith on 5/1/24, 5:42 AM

    Feels like someone stuffed splines into decision trees.

  • by adityang5 on 5/4/24, 4:49 AM

    Very cool stuff! Exciting to see so many people sharing their works on KANs. Seeing as the authors claim that KANs are able to reduce the issues of catastrophic forgetting that we see in MLPs, I thought "Wouldn't it be nice if there was an LLM that substituted MLPs with KANs?". I looked around and didn't find one, so I built one!

    - PyTorch Module of the KAN GPT

    - Deployed to PyPi

    - MIT Licence

    - Test Cases to ensure forward-backward passes work as expected

    - Training script

    I am currently working on training it on the WebText dataset to compare it to the original gpt2. Facing a few out-of-memory issues at the moment. Perhaps the vocab size (50257) is too large?

    I'm open to contributions and would love to hear your thoughts!

    https://github.com/AdityaNG/kan-gpt

    https://pypi.org/project/kan-gpt/

  • by cloudhan on 5/2/24, 12:59 AM

    This reminds me of Weight Agnostic Neural Networks https://weightagnostic.github.io/

  • by apolar on 5/5/24, 4:38 PM

  • by yobbo on 5/1/24, 9:41 AM

    https://kindxiaoming.github.io/pykan/intro.html

    At the end of this example, they recover the symbolic formula that generated their training set: exp(x₂² + sin(3.14x₁)).

    It's like a computation graph with a library of "activation functions" that is optimised, and then pruned. You can recover good symbolic formulas from the pruned graph.

    Maybe not meaningful for MNIST.

  • by diwank on 5/1/24, 4:53 AM

    It’d be really cool to see a transformer with the MLP layers swapped for KANs and then compare its scaling properties with vanilla transformers

  • by SpaceManNabs on 5/1/24, 3:14 PM

    How does back propagation work now? Do these suffer from vanishing or exploding gradients?

  • by phpkar on 5/5/24, 2:21 PM

  • by ALittleLight on 5/1/24, 5:48 AM

    I can't assess this, but I do worry that overnight some algorithmic advance will enhance LLMs by orders of magnitude and the next big model to get trained is suddenly 10,000x better than GPT-4 and nobody's ready for it.

  • by erwincoumans on 5/2/24, 6:42 PM

    If you like this, you may also like this 2019 research paper: "Deep networks and the Kolmogorov–Arnold theorem" https://hadrien-montanelli.github.io/2019-06-25.html https://arxiv.org/pdf/1906.11945

  • by mipt98 on 5/3/24, 6:07 PM

    A more elaborate implementation of this was published years ago, and it wasn't the very first one https://www.science.org/doi/10.1126/science.1165893

  • by kevmo314 on 5/1/24, 8:33 PM

    This seems very similar in concept to the finite element method. Nice to see patterns across fields like that.

  • by syassami on 5/3/24, 4:04 PM

    Nice implementation I've been playing with, https://github.com/Blealtan/efficient-kan alongside @GistNoesis's.

  • by Maro on 5/1/24, 4:37 AM

    Interesting!

    Would this approach (with non-linear learning) still be able to utilize GPUs to speed up training?

  • by coderenegade on 5/2/24, 2:28 AM

    I was under the impression that graph neural nets already trained learnable functions on graph edges rather than nodes, albeit typically on a fully connected graph. Is there any comparison to just a basic GNN here?

  • by renonce on 5/5/24, 1:27 PM

    So a new type of neural network that has been proven to work well on regression tasks common in physics? And tested in practice to fit well on elementary algebra and compositions of complex functions. But no evidence at all if it works on the most basic machine learning tasks like MNIST, not to mention language models.

    I mean it's great but at the current state it seems better suited for tasks where an explicit formula exists (though not known) and the goal is to predict it on unknown points (and be able to interpret the formula as a side effect). Deep learning tasks are more of a statistical nature (think models with a cross entropy loss - it's statistically predicting the frequency of different choices of the class/next token), it requires a specialized training procedure and it is designed to fit 100% rather than somewhat close (think linear algebra - it won't be good at it). It would very likely take a radically different idea to apply it to deep learning tasks. The recently updated "Author's note" also mentions this: "KANs are designed for applications where one cares about high accuracy and/or interpretability."

    It's great but let's be patient before we see this improve LLM accuracy or be used elsewhere.

  • by nico on 5/1/24, 4:36 AM

    Looks super interesting

    I wonder how many more new architectures are going to be found in the next few years

  • by ComplexSystems on 5/1/24, 7:26 PM

    Very interesting! Could existing MLP-style neural networks be put into this form?

  • by nu91 on 5/2/24, 5:59 AM

    I am curious to know if this type of network can help with causal inference.

  • by brrrrrm on 5/1/24, 7:21 PM

    doesn't KA representation require continuous univariate functions? do B-splines actually cover the space of all continuous functions? wouldn't... MLPs be better for the learnable activation functions?

  • by arianvanp on 5/1/24, 8:03 AM

    This really reminds me of petrinets but an analog version? But instead of places and discrete tokens we have activation functions and signals. You can only trigger a transition if an activation function (place) has the right signal (tokens).

  • by keynesyoudigit on 5/1/24, 2:51 PM

    Eli5: why aren't these more popular and broadly used?

  • by yza on 5/2/24, 12:21 PM

    Bayesian KANs, KAN Transformers and KAN VAE's in 3.2...

  • by WithinReason on 5/1/24, 8:09 AM

    Looks very interesting, but my guess would be that this would run into the problem of exploding/vanishing gradients at larger depths, just like TanH or sigmoid networks do.