from Hacker News

A Primer in BERTology: What We Know About How Bert Works

by whym on 11/10/20, 6:23 AM with 16 comments

• by mobilio on 11/10/20, 11:08 AM

  " BERT is a method of pre-training language representations, meaning that we train a general-purpose "language understanding" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP."

  https://github.com/google-research/bert

• by niea_11 on 11/10/20, 11:59 AM

  Can anyone please explain (in layman terms if it's possible) how did the researchers come up with the method in the first place if the process how the method finds the answers is not understood?

• by taneq on 11/10/20, 8:08 AM

  For anyone who, like me, isn't a BERTologist, BERT is a neural network architecture.

• by hallqv on 11/10/20, 12:59 PM

  Any new information in the paper since the first version came out in Mars? Otherwise a 6 month old meta-study seems kind of dated given rate of progress in NLP atm.