from Hacker News

Matrix Calculus: Calculate derivatives of matrices

by mrv_asura on 4/14/23, 11:38 AM with 46 comments

• by bmc7505 on 4/15/23, 3:56 AM

  Sören Laue, creator and first author of this work [1] gave a nice talk [2,3] about it at our reading group not long ago. He also has written about how automatic and symbolic differentiation are essentially equivalent. [4]

  [1]: https://papers.nips.cc/paper/2018/file/0a1bf96b7165e962e90cb...

  [2]: https://www.youtube.com/watch?v=IbTRRlPZwgc

  [3]: https://compcalc.github.io/public/laue/tensor_derivatives.pd...

  [4]: https://arxiv.org/pdf/1904.02990.pdf

• by FabHK on 4/15/23, 5:59 AM

  Example where it comes in handy: OLS (ordinary least squares).

  You want to solve Ax = b approximately. So, minimise the two-norm |Ax-b|, or equivalently, |Ax-b|^2, or equivalently (Ax-b)ᵀ(Ax-b) = xᵀAᵀAx - 2xᵀAᵀb + bᵀb.

  How to minimise it? Easy, take the derivative wrt the vector x and set to zero (the zero vector):

  2AᵀAx - 2Aᵀb = 0, so x = (AᵀA)⁻¹ Aᵀb.

  (Note: that's the mathematical formulation of the solution, not how you'd actually compute it.)

• by eachro on 4/15/23, 1:31 AM

  Knowing matrix derivatives well is one of those skills that were essential in machine learning 10 years ago. Not so much anymore with the dominance of massive neural networks.

• by hgibbs on 4/15/23, 5:03 AM

  The term "matrix derivative" is a bit loaded - you can either mean the derivative of functions with matrix arguments, or functions with vector arguments that have some matrix multiplication terms. Either way, I don't really understand what the confusion is about - if you slightly modify the definition of a derivative to be directional (e.g. lim h->0 (f(X + hA) - f(X))/h) then all of this stuff looks the same (vector derivatives, matrix derivatives and so forth). Taking this perspective was very useful during my PhD where I had to work with analytic operator valued functions.

• by TinkersW on 4/16/23, 8:21 AM

  That is useful but sure would be even more useful if it let you define functions and call them.

  Also the output is pretty gross, wish it had an option for a statically typed language.

  Also wtf? It doesn't have sqrt? Have to write it in power form..sigh

  Also seems to also not grok values with decimal points as the power, so you have to write it as a fraction..sigh..why math people..why?

  Also why can you only select the output for 1 value at a time? For instance if we have a 3d position, we want gradient for x/y/z not just 1 of them..

• by Aardwolf on 4/15/23, 12:13 PM

  How do you enter the matrix transpose in this tool?

  In an example they use X' for transpose and it works.

  But if I try A' * A, it either becomes just A * A, or it shows nothing and says " This 4th order tensor cannot be displayed as a matrix. See the documentation section for more details."

  The documentation also doesn't show how to enter the transpose operator.

• by keithalewis on 4/15/23, 8:40 AM

  https://en.m.wikipedia.org/wiki/Fréchet_derivative

  Doesn't this cover all examples presented?

• by tpoacher on 4/15/23, 12:17 PM

  I have an annoyance reflex when people talk about things like "the derivative of a matrix". A matrix is a notational concept, not an object in itself. It makes as much sense to me as saying the derivative of a set, or the derivative of an array (i.e. as opposed to a vector).

  It should be derivatives "with" matrices, not "of", in my mind.

  Not that it matters in practice, but ... if there's one field where precision of language matters, it should have been mathematics. So it bothers me.

• by bobsmooth on 4/15/23, 1:55 AM

  I didn't know matrices had derivatives.

• by pkoird on 4/15/23, 2:50 AM

  Any good Textbooks for this topic?

• by amelius on 4/15/23, 12:02 PM

  How about computing integrals?