from Hacker News

Petabytes on a budget: How to build cheap cloud storage (2010)

by Oculus on 10/26/14, 3:03 PM with 124 comments

• by KaiserPro on 10/26/14, 4:38 PM

  I look after about 15PB of tier1 storage, and I'd recommend not doing it the backblaze way.

  Its grand that its worked out for them, but there are a few big drawbacks that backblasze have software'd their way around.

  Nowadays its cheaper to use an engenio based array like the MD3260 (also sold/made by netapps/LSI)

  First you can hot swap the disks. Second you don't need to engineer your own storage manager. Thirdly you get much much better performance(2gigabytes a second sustained, which is enough to saturate to 10 gig nics). Fourthly you can get 4 hour 24/7 response. Finally the air flow is a bit suspect.

  we use a 1u two socket server with SAS to server the data.

  If you're brave, you can skip the raid controller and the JBOD enclosure instead and ZFS over the top. However ZFS fragments like a bitch, so watch out if you're running at 75% plus

• by 2close4comfort on 10/26/14, 3:10 PM

  https://www.backblaze.com/blog/backblaze-storage-pod-4/ here is the latest version. I have loved being able to follow the iterations of the storage pod. They are very thoughtful about HW choices but leave it open which I think is the best part!

• by jpalomaki on 10/26/14, 6:17 PM

  I've always been very suspicious about backup vendors offering unlimited space for fixed price. These storage pod posts by Backblaze was primary reason why I decided to give their service a try. Knowing the technology behind the system made it much more credible for me.

• by andyidsinga on 10/26/14, 4:02 PM

  > A Backblaze Storage Pod is a Building Block

  > But the intelligence of where to store data and how to encrypt it, deduplicate it, and index it is all at a higher level (outside the scope of this blog post).

  I'm curious about their software that works outside the nodes too. I've been working on storage clusters over this past 9 months using the Ceph ( http://ceph.com/ ) open source storage software. Its pretty amazing -- and I suspect it could be deployed to a set of backblaze pods too.

  It seems to be that for production environment where you wanted to maintain availability you would to build at least 3 of those pods for any deployment - enabling replication across pods/storage nodes.

• by jdub on 10/26/14, 5:34 PM

  But in 2014, 1PB in S3 – with 11 nines of data durability – costs ~USD$30,000.

• by bkruse on 10/26/14, 7:31 PM

  This has always interested me. I need to do decently big storage for genomic data. It doesn't have to be fast, but it needs to be able to survive one data center blowing up. If I have 3 data centers, need to store 2-3 petabytes and need storage to survive in the case of a data center failure - the solutions really narrow down when you have to get under the $200/tb range.

  Playing with Swift now - but it has really opened my eyes to how much more difficult 2-3 petabytes of storage is (disk failures, number of disks in your infrastructure, the time to redeploy a datacenter on a 1gpbs connection). All the little problems become much bigger!

• by harel on 10/26/14, 8:40 PM

  Tech aside, I'm quite curious about the economics of storage here. By the price tags and 'Debian 4' I'm guessing this is an older post. But still, $7867 per 67TB and $5 per month, means they need 131 users pay for one year to recoup the cost of one pod, assuming those 131 do not generate over 67TB worth of storage in that period of time. I've not factored in data centre costs, salaries etc. Just a pod. I'm guessing they have enough users as they have been around for many years now, but still, $5 seems a bit on the cheap side to me (not that I'm complaining)

• by corv on 10/26/14, 3:41 PM

  I wonder how they can guarantee data integrity.

  Are they checksumming on a higher level and is that cheaper than using ZFS with necessarily more expensive hardware?

• by immortalx on 10/26/14, 7:29 PM

  I decided to try it. You can only select entire hard drives and work around this by excluding folders. That's odd but what i don't understand is why you cannot exclude your c:/ (or Main Drive). Why should anyone be forced to backup something?

  Seems to me like the design is backwards and doesn't make any sense.

• by ciupicri on 10/26/14, 3:38 PM

  If you submit an old article even if newer versions of it exist, at least mention the year in parenthesis.

• by hendzen on 10/26/14, 3:25 PM

  Honest question, why JFS vs say, ext4?

• by fredsted on 10/26/14, 5:48 PM

  >In the future, we will dedicate an entire blog post to vibration.

  In the meantime, does anyone have a link?

• by aliakhtar on 10/26/14, 6:48 PM

  Do they only do backups or also cloud storage? If they have an API for uploading / deleting / viewing files, I'd use them over S3 given how lower their costs are. But, I can't find any info on that on their website.

• by mschuster91 on 10/26/14, 3:18 PM

  I'd love it if either Backblaze or a 3rd party makes a business of selling these pods!

  edit: just spotted it, their boot drive is PATA?! Why is this, given that PATA drives are slower and more expensive than SATA ones?

• by ksec on 10/26/14, 3:49 PM

  Need the Add the Year at the title. This post is old.

• by tkinom on 10/26/14, 3:38 PM

  Great article!

  Love to see more write up on software selection process, tradeoff, failure recovery process/methods and benchmark data.

• by sidcool on 10/26/14, 4:01 PM

  Quite detailed post. Loved reading it.

• by iflyun on 10/26/14, 7:33 PM

  why such an old and expensive cpu?

• by codeonfire on 10/26/14, 4:22 PM

  What do you do when a drive goes bad? Do you move ~50TB of data, pull the entire pod out of the rack, and then try to determine which of 45 drives is bad?