from Hacker News

How to process 100gb tsv and XML files?

by anindha on 8/28/24, 4:12 PM with 6 comments

• by FlyingAvatar on 8/28/24, 10:05 PM

  It really depends on what you need to do with the data, but in most cases Python could do this pretty easily with csv.reader (with a \t delimiter for TSV) or xml.etree.ElementTree.iterparse (for XML) in streaming fashion such that you're not loading the whole file at once.

• by pradeepchhetri on 8/30/24, 3:14 AM

  You can leverage ClickHouse to process your music data. ClickHouse supports both TSV[0] and XML[1] data formats.

  [0] https://clickhouse.com/docs/en/interfaces/formats#tabseparat...

  [1] https://clickhouse.com/docs/en/interfaces/formats#xml

• by mobilio on 8/29/24, 7:50 PM

  Great solution is DuckDB: https://duckdb.org/docs/data/csv/overview.html

• by datadrivenangel on 8/28/24, 8:42 PM

  What kind of single music data file is 100gb?

  Also how is it structured? If it's actually a tab separated value file, consider using something like polars or DuckDB?

• by anindha on 8/28/24, 4:30 PM

  I found this https://klogg.filimonov.dev

• by abdusco on 8/29/24, 12:04 PM

  For TSV, you might wanna consider importing it into a Sqlite database, then querying it however you please.

  https://stackoverflow.com/a/35454070/5298150

  You can also use datasette & sqlite utils for it

  https://sqlite-utils.datasette.io/en/stable/cli.html#inserti...