from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

by elitan on 9/26/22, 6:15 PM with 145 comments

• by nunopato on 9/26/22, 9:50 PM

  (Nhost)

  Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company.

  First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses. A project/backend is comprised of a Postgres Database and the aforementioned services, none of it is shared. You get your own GraphQL engine, your own auth service, etc. We also provide the means to interface with the backend through our official SDKs.

  Some points I see mentioned below that are worth exploring:

  - One RDS instance per tenant is prohibited from a cost perspective, obviously. RDS is expensive and we have a very generous free tier.

  - We run the infrastructure for thousands of projects/backends which we have absolutely no control over what they are used for. Users might be building a simple job board, or the next Facebook (please don't). This means we have no idea what the workloads and access patterns will look like.

  - RDS is mature and a great product, AWS is a billion dolar company, etc - that is all true. But is it also true that we do not control if a user's project is missing an index and the fact that RDS does not provide any means to limit CPU/memory usage per database/tenant.

  - We had a couple of discussions with folks at AWS and for the reasons already mentioned, there was no obvious solution to our problem. Let me reiterate this, the folks that own the service didn't have a solution to our problem given our constraints.

  - Yes, this is a DIY scenario, but this is part of our core business.

  I hope this clarifies some of the doubts. And I expect to have a more detailed and technical blog post about our experience soon.

  By the way, we are hiring. If you think what we're doing is interesting and you have experience operating Postgres at scale, please write me an email at nuno@nhost.io. And don't forget to star us at https://github.com/nhost/nhost.

• by MBCook on 9/26/22, 7:41 PM

  So they switch from one giant RDS instance with all tenants per AZ to per-tenant PG in Kubernetes.

  So really we don’t know how much RDS was a problem compared to the the tenant distribution.

  For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options.

  But I understand why they just moved forward. They said they consulted experts, it would also be nice to discuss some of what they looked or asked about.

• by radimm on 9/26/22, 7:18 PM

  Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question.

  > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched

  This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be already stretching it). Always wonder about use cases around 3 - 5k req/sec as minimum.

  [edit] PS: not really ditching neither k8s pg nor AWS RDS or similar solutions. Just being curious.

• by qeternity on 9/26/22, 8:29 PM

  These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves.

  Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-your-own spectrum.

• by jmarbach on 9/26/22, 7:30 PM

  $0.50 per extra GB seems high, especially for a storage-intensive app. Given the cost of cloud Object Storage services it doesn't seem to make much sense.

  Examples of alternatives for managed Postgres:

  * Supabase is $0.125 per GB

  * DigitalOcean managed Postgres is ~$0.35 per GB

• by neilv on 9/26/22, 8:34 PM

  I didn't see "backups" mentioned in that, though I'm sure they have them. Depending on your needs, it's a big thing to keep in mind while weighing options.

  For a small startup or operation, a managed service having credible snapshots, PITR backups, failover, etc. is going to save a business a lot of ops cost, compared to DIY designing, implementing, testing, and drilling, to the same level of credibility.

  One recent early startup, I looked at the amount of work for me or a contractor/consultant/hire to upgrade our Postgres recovery capability (including testing and drills) with confidence. I soon decided to move from self-hosted Postgres to RDS Postgres.

  RDS was a significant chunk of our modest AWS bill (otherwise, almost entirely plain EC2, S3, and traffic), but easy to justify to the founders, just by mentioning the costs it saved us for business existential protection we needed.

• by xwowsersx on 9/26/22, 7:10 PM

  I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but I'm not clear on which instance type best serves our needs — I think it may very well be the case that the t3 family isn't fit for our purposes. I'm also unclear on whether we ought to switch to provisioned IOPS SSD. Does anyone have any general pointers here? I know the question is pretty open-ended, but would be great if anyone has general advice from personal experience?

• by ransom1538 on 9/27/22, 4:57 AM

  I operate a large fleet of mysql db instances. We cannot use Cloudsql (RDS competitor) due mainly to cost. BUT, one thing left out, was the ability to have complex topologies. EG. MasterA <- SlaveA[1..n] <- MasterB <- SlaveB[1..n]. With extremely high writes, being able to cut and shard where you want if very powerful. In this example you could write to MasterB with different data. If i need to filter a table in replication: done. We don't need to beg AWS RDS team for the option to change a db variable (I have done this). Warning: Doing this stuff at scale with massive bills is very stressful. It took about a year to get everything ironed out [snapshots, autoscaling, sharding, custom monitoring, etc].

• by qubit23 on 9/26/22, 7:47 PM

  I was hoping to see a bit more of an explanation of how this was implemented.

• by KaiserPro on 9/26/22, 8:11 PM

  In this instance I can see the point, being able to give raw access to customer's own psql instance is a good feature.

  but. It sounds bloody expensive to develop and maintain a reliable psql service on k8s

• by geggam on 9/26/22, 7:26 PM

  I would love to see the monitoring on this.

  Network IOPs and NAT nastiness or disk IO the bigger issue ?

• by HL33tibCe7 on 9/26/22, 7:42 PM

  Couldn’t you just spin up an RDS instance for each project (so, single-tenant RDS instances) to avoid the noisy neighbour problem? Or is that too expensive?

• by techn00 on 9/26/22, 7:45 PM

  So what solution did you end up using? Crunchy operator?

• by xyzzy_plugh on 9/27/22, 6:50 AM

  If the cost of operating a postgres database is eating into your margins so much (and you can't simply adjust your prices to eat the difference) then I would suspect the wrong technology is in place.

  Sure, RDS is expensive, but it's also quite well done. Almost every cloud platform service is more expensive than doing it yourself. No surprise here.

  In the past I've deployed SQLite over Postgres for cost cutting reasons. It's not too difficult to swap out unless you're heavily bought into database features.

• by mp3tricord on 9/26/22, 9:50 PM

  In a production data base why are people executing long running queries on the primary. They should be using a DB replica.

• by e-clinton on 9/27/22, 1:12 AM

  Congrats on the launch. Curious to see what else is in store for this week.

  Do I have to manually upgrade my old instances?

• by stunt on 9/26/22, 9:01 PM

  What's the benefit of running Postgres in Kubernetes vs VMs (with replication obviously)?

• by maxyurk on 9/27/22, 10:13 AM

  did you consider https://www.pgbouncer.org/ ?

• by 0xbadcafebee on 9/26/22, 7:21 PM

  Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work.

  Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.