from Hacker News

KnowledgeNet: A Benchmark for Knowledge Base Population

by miket on 1/17/20, 12:27 AM with 7 comments

• by miket on 1/17/20, 1:18 AM

  When people think about using computers for Natural Language Processing, they often think about end-tasks like classification, translation, question answering, and models like BERT that model the statistical regularities in text. However, these tasks only measure indirectly how much the system has understood the meaning of the text, are largely unexplainable black boxes, and require reams of training data.

  NLP is good enough that we can now explicitly measure how well a system reads text in terms of what knowledge is extracted from it. This task is called Knowledge Base Population, and we've released the first reproducible dataset called KnowledgeNet that measures this task, along with an open source state-of-the-art baseline.

  Direct link to the Github repo: https://github.com/diffbot/knowledge-net EMNLP paper: https://www.aclweb.org/anthology/D19-1069.pdf

• by g82918 on 1/17/20, 2:51 AM

  Mostly an article pushing their benchmark and article: https://www.aclweb.org/anthology/D19-1069.pdf. In the article they compare existing benchmarks against a criteria they create to show their benchmark is the only one that features the things they say are important. All the others are somehow deficient by the totally objective metric they create.

• by bhl on 1/17/20, 9:35 AM

  Reminds of a submission from a year ago on autogenerating knowledge base from articles from the web [1]. I think it'd be neat if Q&A Nets and other techniques sufficed to the point where we would prefer using "knowledge engines" over search engines, like a generalized Wolfram Alpha.

  [1] https://primer.ai/blog/quicksilver

• by nl on 1/17/20, 3:01 AM

  State-of-the-art models (using BERT) are far from achieving human performance (0.504 vs 0.822).

  This is moderately surprising.

  In question answering (QA) style tasks (SQUAD, SQUAD 2) we see state of the art models approach human performance. QA is similar to KBC in the sense that the answers are usually extracted from text in a similar way.

  I'd imaging there is potential for fairly rapid improvement in this (Knowledge Base Population) task.

• by sdan on 1/17/20, 1:28 AM

  Amazing! Love using Diffbot and although I'm not too deep into the NLP space yet, finding the relations of the text itself is a pretty important task.