from Hacker News

Quantcast File System on S3

by lumens on 11/29/16, 6:33 PM with 4 comments

• by lumens on 11/29/16, 6:44 PM

  For some context:

  Quantcast moved off Hadoop in favor of a home-rolled solution (QFS) because of Hadoop wasn't scaling well to our needs (40PB+ data processed/day).

  This release of QFS (https://github.com/quantcast/qfs) includes support for S3, among other improvements.

• by mck35 on 11/29/16, 6:33 PM

  Hi, we recently announced Quantcast File System version 1.2. Besides many bug fixes and improvements, it comes with the ability to use Amazon S3 as a backend store. You can check it out on QFS Github. I personally contributed to this release, and I'd be happy to answer any questions. Thanks.