from Hacker News

Deep Learning Is Easy – Learn Something Harder

by hlynurd on 9/6/23, 6:39 PM with 23 comments

• by jph00 on 9/6/23, 10:25 PM

  IMO this 2016 article really hasn't aged well. It turned out that architectural improvements really did matter, and there was still loads of low-hanging fruit. His prediction that bayesian approaches (which were the topic of his PhD) would turn out to be fundamental, has not turned out to be true so far (although they do have their place).

  (I think in general when people say that their special area of study is particularly important, it should be taken with a grain of salt!)

• by light_hue_1 on 9/6/23, 10:05 PM

  This is terrible and frankly self serving advice.

  "Don't build on top of deep learning. Build on top of MCMC-like methods"

  I used to do research into such methods. That game is over. It's a massive waste of time at the moment. The whole idea is how do we import what was good about those methods into the modern deep learning toolkit. How do I sample from distribution with dl? How do I get uncertainty estimates? How do I compose models, get disentangled representations, get few shot learning, etc.

  The idea that people should go back to tools like MCMC today is pretty absurd. That entire research program was a failure and never scaled to anything. I say this of my many dozens of papers in the area too.

  I would never give this advice to my PhD students.

  Maybe in a decade or two someone will rescue MCMC like methods. In the meantime your PhD students will suffer by being irrelevant and having skills that no one needs.

• by dbmikus on 9/6/23, 9:53 PM

  My general belief, is that the best way to learn at the frontier of something is to pick a problem or a goal and try to solve it. Then you will learn what is in the way of getting that done.

  Unless you already have deep expertise, I think it's a bad idea to pick a research area and just go and research that. You won't have intuition about why it's a good thing to research. However, you can have intuition about real world problems and the solutions you want to see, and then work backwards to what you need to research.

• by blind666 on 9/6/23, 10:02 PM

  It is interesting to look back and evaluate the preferences / intuitions prominent researchers had in the field (most of whom started their careers experimenting with MNIST-scale of data, at best)

  With access to unfathomable amounts of data, especially over the last couple of years, the game changed entirely and is not seeming to cool down anytime soon.

  The field, certainly, values engineering a lot more than it used to, and it is exciting to see how major advances together with open-source contributions are going to take us

• by danielmarkbruce on 9/6/23, 10:24 PM

  Maybe if you stick to purely theoretical stuff, it's easy. Actually building real systems that work and add value using deep learning isn't easy. There are so many gotchas.

• by hedgehog0 on 9/7/23, 9:57 AM

  https://www.inference.vc/we-may-be-surprised-again/

  The author wrote this article one month ago, and he mentioned where he thought wrong about DL

• by jacobn on 9/6/23, 9:51 PM

  [2016]

• by sullyj3 on 9/7/23, 9:06 AM

  Pretty straightforward case of the curse of knowledge (https://en.wikipedia.org/wiki/Curse_of_knowledge), in my opinion.