from Hacker News

Tips for Better Deep Learning Models

by lauradhamilton on 7/18/14, 5:58 PM with 15 comments

  • by gamegoblin on 7/18/14, 7:33 PM

    A note on dropout:

    If your layer size is relatively small (not hundreds or thousands of nodes), dropout is usually detrimental and a more traditional regularization method such as weight-decay is superior.

    For the size networks Hinton et al are playing with nowadays (with thousands of nodes in a layer), dropout is good, though.

  • by vundervul on 7/18/14, 10:00 PM

    Who is Arno Candel and why should we pay attention to his tips on training neural networks? Anyone who suggests grid search for metaparameter tuning is out of touch with the consensus among experts in deep learning. A lot of people are coming out of the woodwork and presenting themselves as experts in this exciting area because it has had so much success recently, but most of them seem to be beginners. Having lots of beginners learning is fine and healthy, but a lot of these people act as if they are experts.

  • by agibsonccc on 7/18/14, 7:09 PM

    I would just like to link to my comments from before for people who maybe curious:

    https://news.ycombinator.com/item?id=7803101

    I will also add that looking in to hessian free for training over conjugate gradient/LBFGS/SGD for feed forward nets has proven to be amazing[1].

    Recursive nets I'm still playing with yet, but based on the work by socher, they used LBFGS just fine.

    [1]: http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf

    [2]: http://socher.org/

  • by prajit on 7/18/14, 7:41 PM

    A question about the actual slides: why don't they use unsupervised pretraining (i.e. Sparse Autoencoder) for predicting MNIST? Is it just to show that they don't need pretraining to achieve good results or is there something deeper?

  • by TrainedMonkey on 7/18/14, 6:47 PM

  • by ivan_ah on 7/18/14, 6:38 PM

    direct link to slides anyone?