from Hacker News

Ask HN: cloud data storage cheaper than AWS S3?

by dpapathanasiou on 2/17/20, 8:45 PM with 34 comments

I'm working on a new neural network based project, which means I'll need to access and store more data than I can keep on my hard drives.

While I'm familiar with AWS S3 from my day job, I was wondering what other alternatives are out there that may be cheaper (or even free)?

I'm also aware of dropbox, google docs, etc., but ideally I'd like to have programmatic access via an API.

  • by okneil on 2/17/20, 9:06 PM

  • by pentium10 on 2/17/20, 9:21 PM

    BigQuery with it's $20/TB for active, and half of the price for cold storage, is the best structured (eg: columns or JSON) solution where import and export is free. You even get SQL as the product is a data warehouse.

  • by whalesalad on 2/17/20, 9:33 PM

    If you need that much storage, I would posit that cloud storage is going to be a problem not due to cost but due to network performance. A bunch of cheap storage is great but if it takes you days to transfer data there it might be a dealbreaker.

    Have you considered a few external hard drives? You can get a couple of terabytes for pretty cheap these days.

  • by nicoburns on 2/17/20, 9:05 PM

  • by stu2010 on 2/17/20, 9:09 PM

    If you can tolerate risk of data loss, VPS or even physical hosts are likely to be far cheaper than any cloud. Look at OVH, Hetzner, Linode, DigitalOcean.

  • by celerrimus on 2/18/20, 2:54 PM

    I assume that everything depends on: - how much storage you need, - how long you need it to store this, - if this data easy reproducible.

    Generally storage isn't cheap, and cloud storage is quite expensive in the long run. If you need storage for more that a year, I would invest in own local HDDs - put to your PC or buy used NAS server or PC. You will benefit with much better performance and this would be the most cost effective solution.

    Keep in mind that often transfer to cheap cloud storage is slow, I tried to keep my backup in few different providers, it could take literally months to upload 6TB of data. Also keep in mind that you may be charged for data transfer separately, for every data access, so cloud cost may be much higher than expected.

    If you plan use this in shorter periods, I would go with OVH offer - they probably have best quality/cost ratio. Depending on your needs I would suggest buying dedicated storage server, or use their Data Storage (3x replicated $0.0112/month/GB, plus outgoing transfer - $0.011/GB). They also have cold storage for about $0.0023/month/GB.

  • by crobibero on 2/17/20, 9:17 PM

    Wasabi is decently inexpensive at $.0059 per GB/month https://wasabi.com/

    I currently use it as my personal cloud backup

  • by miohtama on 2/17/20, 11:25 PM

    Storj Tardigrade

    https://tardigrade.io/

  • by rmeertens on 2/17/20, 9:18 PM

    Keep in mind that if you use this data often as training data you want to store it close to your GPU. No point in saving money on the data storage and have your expensive GPU idle because you are waiting for data to download from Dropbox...

  • by zknz on 2/17/20, 9:26 PM

    What is your tolerance for losing data?

  • by wajidansari on 2/19/20, 11:56 AM

    Google gets ready for its entry in cloud services market

    https://www.headlinesoftoday.com/technology/tech-reviews/goo...

  • by tyingq on 2/18/20, 1:04 PM

    You don't mention how much data. At some size point, a cheap storage server and minio (open source s3 compatible) might be a better value.

    At the low end, OVH has a 4×4TB HDD SATA + 1×500GB SSD NVMe server for ~$90/month.

    Of course, you have to configure and administer it, so not for everyone.

  • by heartbeats on 2/17/20, 8:49 PM

    Depending on your needs, you can rent a dedicated server with hard drives. For example, Hetzner has an offering of 10x 10TB HDDs for $200/month or so.

    Disclaimer: I have never used Hetzner's services nor can I vouch for them.

  • by bunya017 on 2/18/20, 6:45 AM

    Scaleway Object Storage

    S3 compatible and comes with free bandwidth

    https://www.scaleway.com/en/object-storage/

  • by Havoc on 2/17/20, 9:26 PM

    Too big for my hard drives covers a lot of space.

  • by haidrali on 2/17/20, 10:48 PM

  • by invisiblerobot on 2/17/20, 9:19 PM

    s3 may be cheaper than you think.

    If you're willing to tolerate hr+ delays in accessing your data aws glacier deep archive is 70cents per terabyte month.

    that's pretty awesome in my book.

    If you need to access the data in under an hr it comes out around $2 per TB/mo

  • by dvasdekis on 2/18/20, 4:00 AM

    Why don't you try Usenet? Unlimited storage for $3.14/month[1].

    There are a couple of Python libraries out there for posting and fetching, but it'll definitely be shabbier than a purpose-built service. However, for seriously large storage requirements, you can't beat that price.

    I have a full PC backup I did 5 years ago drifting around on Usenet somewhere. It was still there when I checked a year ago.

    [1]https://newsgroupdirect.com/member/billing/?planid=189&deal= but I've had problems with their service, and personally use FrugalUsenet for a little more.