from Hacker News

I added context data to the TruthfulQA dataset

by roh26it on 8/10/24, 3:31 AM with 1 comments

• by roh26it on 8/10/24, 3:31 AM

  Being one of the most downloaded datasets on Huggingface, I was a little bit surprised by how dirty this dataset was. Plus it had very limited information and some incorrect classifications as well.

  For an internal experiment on building a "Truthful Evaluator", we picked up this dataset and tried fine-tuning a model on these 8000 odd examples.

  Realised that it needed: 1. Cleaning up 2. Some reclassification

  But, most importantly - it lacked context data. It only had a link pointing to the source which was also absent for a few rows.

  We scraped the internet for the link in the dataset, matched it to the question and narrowed down on a small context to be added to the main dataset.

  Releasing it publicly so that someone else may avoid the 2-3 days of pain of wrangling with this data.