from Hacker News

How to fix CSV: make it even more U+1F4A9 PILE OF POO

by paulfitz on 3/29/24, 6:30 PM with 27 comments

  • by akira2501 on 3/29/24, 7:01 PM

    What if my data contains a new line? People focus on the comma then forget the newline is just as significant. That still needs to be escaped and we're right back where we started.

    Meanwhile, RFC4180 takes less time to read than this entire article.

  • by kristopolous on 3/29/24, 8:50 PM

    If we are willing to throw away the comma, use the ASCII RS, record separator symbol. It's exactly what you want, even has a visual ␞ these days.

    It's a problem solved decades ago with solutions we've failed to adopt. Weird, buggy, poorly parsable CSV is still somehow the norm.

    Not saying you should, but if you want to change, the answer is already there. Change has to start somewhere...

  • by ogoffart on 3/29/24, 7:02 PM

    Not long ago there was also a post about "Unicode Separated Value" https://github.com/sixarm/usv https://news.ycombinator.com/item?id=39679378

  • by verandaguy on 3/29/24, 6:47 PM

    Technically editorialized title (since the original article just uses the emoji verbatim), but I think this is a net improvement.

  • by dbt00 on 3/29/24, 7:01 PM

    0x1d and 0x1e in the ascii standard exist for exactly this reason and don’t need more than one byte unlike this goofy thing.

  • by gwbas1c on 3/29/24, 9:04 PM

    I don't think CSV can ever be "fixed." Its popular because there is always someone naive enough to think that it works, and ignorant of specs that handle corner cases.

  • by gwbas1c on 3/29/24, 7:27 PM

    I think we should name the files ".cso"

    CSO is a stormwater industry term for "Combined Sewer Overflow." They happen in older cities where storm runoff and raw sewage (poop) go into the same sewer system. When there is a lot of rain, the wastewater treatment plants overflow, and then raw sewage runs into waterways.

    https://en.wikipedia.org/wiki/Combined_sewer#Combined_sewer_...

  • by dwheeler on 3/29/24, 7:16 PM

    No! The poop symbol is used in data, and thus is a terrible separator. If you have to quote it anyway, use commas, as that is already in use. Or use "Unicode" separated values.

  • by ghusto on 3/29/24, 7:07 PM

    Perhaps naive, but we escape with \ everywhere else, so why not here?

    If you're typing in CSV manually, escape with \

    If you're exporting to CSV, the program already know which part is data and which part is the next cell, so again the program can escape with \

  • by refulgentis on 3/29/24, 6:56 PM

    https://webcache.googleusercontent.com/search?q=cache:https:...

    n.b. not worth your time. tl;dr: lets replace the comma with the poop emoji because commas occur in data.

    There's already a solution to that (obviously). Best argument a contrarian could make is you "learn about unicode", by which they'd mean, the words "basic multilingual plane" are included at one point.