from Hacker News

Ask HN: Best way to store large amount of binary files?

by kluck on 2/16/17, 9:42 AM with 7 comments

What is the best technology to store a large amount of binary files?

I thought about this problem and explored the solution space but still fail to get a final answer. The problem with flat file storage is that the size of the directories grows very large making backups and browsing a pain. And the dbms solution has the disadvantage over beeing not browseable (using file system tools) and beeing a "black box" of sort.

Also I prefer solutions that are slick and do their job fast. That any alpha/beta projects are out of the question for a durable data store, is obvious IMHO.

  • by stephenr on 2/16/17, 9:49 AM

    On disk, using nested directories - (00-zz)/(00-zz)/file.ext

    The two levels above can either be based on the initial letters of either the filename of a hash of the file content, or use incremental names until each one is "full".

    Increase depth to increase maximum scale.