from Hacker News

Tell HN: Hacker News BigQuery dataset is updated daily again

by wietsevenema on 6/11/24, 10:26 AM with 11 comments

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

by zX41ZdbW on 6/25/24, 12:22 AM
Daily is too infrequently.
I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...
I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...
by dang on 6/11/24, 8:31 PM
Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.
by throwaway211 on 6/15/24, 12:31 PM
How big is it?