from Hacker News

What Is Differentiable Programming?

by one-more-minute on 2/11/19, 11:20 AM with 41 comments

• by throwawaymath on 2/13/19, 12:22 AM

  I have to be honest, I don't think this is a good explanation. I don't know what differential programming is, but I'm fairly sure I have the mathematical background to understand it. But I didn't come away from this article with any confidence that I'm following along.

  On a superficial level it seems like it:

  1. Generalizes deep learning to an optimization function on decomposable input, and

  2. Reduces the number of parameters required to learn the input by exploiting the structure of the input, thereby making learning more efficient.

  Is that correct? Is it completely off? What am I missing? Is there any more meat to the article than this?

  Could someone who has upvoted this (and ideally understands the topic well) provide a different explanation of the concept? It would be great if I could see a real world example (even a relatively trivial one) represented in both the traditional matrix computation form and the sexy new differentiable form.

• by damip on 2/13/19, 1:27 AM

  Differentiable does not mean easy to optimize. One could imagine implementing sha-256 using differentiable operators, and yet the system as a whole would not be optimizable at all. It would be interesting to have compilers that optimize the "optimizability" of differentiable programs tho...

  Also, here are two interesting examples of differentiation through physical systems for classification:

  https://arxiv.org/pdf/1808.08412.pdf

  https://innovate.ee.ucla.edu/wp-content/uploads/2018/07/2018...

• by ricksharp on 2/13/19, 2:45 AM

  Could someone list some practical examples where Differential Programming would be useful?

  I am familiar where Nueral Networks and Convolutional Networks have done well especially around image processing etc.

  But I can’t imagine where having differential code would help unless it is just tying multiple neural networks together in a continuous chain of differentiation.

  For most programming tasks, I can’t imagine how differentiation would be possible or beneficial.

  Is there a possibility that one could start with a series of unit tests and partial results and through gradient descent actually arrive at additional passing test cases? Most of the time in my experience, passing additional test cases like this requires significantly more complex structures that would not be found via differentiation.

• by ktpsns on 2/13/19, 8:59 AM

  Something I don't understand about Automatic Differentiation is: Why not use a Computer Algebra System instead for generating derivatives of given functions?

• by ricksharp on 2/13/19, 2:03 AM

  I found this paper that helps answer the question: https://arxiv.org/pdf/1803.10228.pdf

• by xitrium on 2/14/19, 7:10 PM

  Be cautious using this article to try to learn anything. Differentiable programming is not actually related to deep learning; it's another word for automatic differentiation, a technique that is very important in deep learning implementations but useful for a variety of other tasks where having gradients available for arbitrary functions is useful.

  The article is correct that "Differentiable Programming" seems to be a rebranding effort that I believe just helped automatic differentiation work from the machine learning world get published in Programming Languages journals. I wouldn't read too much into it.

• by tanilama on 2/13/19, 2:01 AM

  Is Differentiable Programming just a rebranding of tape-based auto differentiation?

• by hnuser355 on 2/13/19, 12:27 AM

  What the hell is that supposed to mean and how it’s different from automatic differentiation