from Hacker News

Firebase outages and misleading status reporting

by sauldcosta on 10/13/18, 1:14 AM with 49 comments

  • by mattbillenstein on 10/13/18, 1:23 AM

    AppEngine had the same problems - seemingly every week some component of the service would be down for some non-negligible amount of time (laughably it was often search -- we're talking about Google here).

    I've generally found AWS more reliable than GCP - even when GCP isn't having downtime, you'll occasionally get 503's from their APIs, so you need to wrap all your calls to them in retries.

    AWS has had multiple instances of cascading EBS backplane failures, but outside of that I've found their core services pretty reliable -- 400+ days of uptime on a lot of VMs in systems I've worked on -- I avoid EBS when I can.

    My advice is to keep your stuff simple - PaaS might seem attractive, but you have so little control as you mention when something goes down. Embrace multi-cloud by using the lowest common denominator of tech available - virtual machines, dns, networking, and instance storage if that suits your needs. Treat vms as disposable - and make sure you have system, service, and data redundancy at that level to survive the failure of an entire availability zone across your application.

  • by EZ-E on 10/13/18, 12:07 PM

    We had the same exact problem with Firebase Realtime Database. Our product uses it heavily and is dependent on its latency so we notice anytime an issue appear.

    The unacceptable thing is : not only outages are fairly common, many smaller, briefer outages and disruptions are not even reported. For example the day after the 2 hour outage mentioned in the article, there was an issue where while writing to the database seemingly successful, but the clients listening to the changes would NOT receive the notification that the data their are observing was updated, for more than 30 minutes. It wasn't reported in Firebase's status dashboard.

    Google bought Firebase back then, and to replace Firebase Realtime Database, Google developed Firebase Firestore (now in beta). I suspect that Firebase Realtime Database isn't receiving much attention these days and that the service will be closed after some time.

  • by xrd on 10/13/18, 4:44 PM

    Have to say, having worked in a huge organization with multiple clients accessing services, I much prefer the firebase solution. You still have downtime in any polyglot solution and the problem is pretty clear here (it's firebase database, not one of dozens of legacy layers...). When you own the entire stack it is amazing how much of the organizational effort goes into obscuring who is responsible. And the stack is much more opaque.

    It really is possible to design a system around firebase with a much smaller team. You give up control but control is a myth anyway. And, Firestore is actually designed to support offline mode, so wonder if they neglected to design for that feature which might help here.

    The unfortunate reality is that we are in a moment where Firestore is beta and Firebase Database is not supported as it should be. Google should do a better job of helping people to migrate and explaining the roadmao. I imagine the writer of this article just doesn't have as much company clout to get that level of involvement from Google. This was probably an attempt to get that attention that other higher paying clients can get.

  • by pg_bot on 10/13/18, 5:43 PM

    If you need to build a product that relies heavily on real time updates, I would look into using Elixir and Phoenix.[0] They nailed the channel abstraction which is the main entry point for realtime communication over websockets. It takes me hours to make scalable realtime applications in what would normally take me days using other systems. The language may take some time to get used to, and the ecosystem isn't as mature as other languages, but what is there is incredibly impressive.

    [0]: https://phoenixframework.org/

  • by crystaln on 10/13/18, 6:34 AM

    Firebase is really awesome. However there kinds of reliability issues and the lack of integrity and communication with which Google handles such things are major reasons I would avoid committing to it. On top of that, Google's history of overlapping products (Firebase or Firestore?) and discontinuing or foot dragging support make decisions confusing and commitment harrowing.

    Amazon on the other hand has a history of committing to clear product direction which makes committing to their platforms much easier. Amplify and AppSync for instance feel like safer choices.

  • by dotmanish on 10/13/18, 2:27 PM

    I stopped using the realtime database once firestore was released in beta. So haven't experienced the downtime you have demonstrated in the status graphs, but Firebase's SLA [1] for realtime database apparently guarantees service credit for monthly uptime less than 99.95%. To corroborate your observations, check if you received this credit:

    Less than 99.95% but equal to or greater than 99.0%: 10% credit

    Less than 99.0%: 30% credit

    [1] https://firebase.google.com/terms/service-level-agreement/

  • by nslog on 10/13/18, 2:59 AM

    Check out the AWS offerings (Amplify + AppSync) if you're rolling off Firebase: https://aws-amplify.github.io https://docs.aws.amazon.com/appsync/latest/devguide/welcome....

  • by sampl on 10/13/18, 1:20 PM

    Not really convinced firebase is “covering it up”.

    The official status page breaks down availability by-service with descriptions of each outage and updates with timestamps.

    https://status.firebase.google.com

  • by joeblau on 10/13/18, 11:42 AM

    I just build a service/website last weekend using App Engine and Firebase. After reading these comments and this blog post, I think I might migrate it over to AWS. I didn't realize that Firebase was so unreliable.

  • by romed on 10/13/18, 5:28 PM

    Firebase RTDB is basically the legacy product that barely works. Firestore is the post-acquisition product built on Google tech. It’s a rotten situation. I noticed the outage mentioned in this post because it took down Ford GoBike (and Citibike and all the other Motivate/Lyft bike share systems).

  • by burtonator on 10/13/18, 4:16 PM

    I've been thinking about implementing Firebase as part of Polar: https://getpolarized.io/

    The idea is that you update your documents (PDF, HTML, etc) into Polar, tag them, and then we sync them to the cloud. Then when you go to another machine like work or home your documents are always synchronized.

    At first I fell in love with Firebase and was very very excited to start implementing it.

    They've spent a ton of time working on the initial implementation experience.

    Their Firebase Auth support was amazingly simple to setup. Same with Firebase hosting. It's top notch. You can be up and running with a CDN hosting with SSL in like 2 minutes and the firebase tools are exceptional.

    Cloud Firestore seems really interesting and easy to setup. It's basically designed for 'apps'. IE user-facing apps and works pretty well if all the data is private to the user.

    I do struggle with these issues of reliability though. At Datastreamer (http://www.datastreamer.io/) we use Hetzner and have about a half petabyte stored there.

    It's a blog content search engine which we license to other startups so high availability is critical.

    Their infra is amazingly reliable. Very very happy here.

    The problem of course is that you then have to manage your own software stack which of course requires extra effort on your part.

  • by ramkalari on 10/13/18, 2:32 PM

    How does firestore fare in terms of reliability? I heard it is a cleaner and more scalable version of Firebase.

  • by jondubois on 10/13/18, 9:55 AM

    I think that PaaS and BaaS where you don't have access to the back end is a dead end. It's going to go the way of Windows Server. Open source solutions will always win in the end when it comes to developers.

  • by novaleaf on 10/13/18, 5:14 PM

    I think that now, Firebase is build on Google Cloud Datastore. I have used Datastore in production since 2015, and have had no outages, but if I had to do it again I think I'd go normal RDB, just because query support is extremely limited (no full text search) and "schema change == data rebuild" issues.

  • by ankit219 on 10/13/18, 3:59 PM

    yeah, we have suffered too. Initially we were using firebase Real Time DB for authentication as well as delivering messages. Messages suffered outages every now and then (and we suffered more cos our backend is in Python Django and Pyrebase comes with its own set of issues on top of Firebase). When we found out messages arent being delivered, we switched to pusher as a backup first and then to websocket. Now we use Firebase only for authentication (via real time database) and Notification sending, and still have a backend/app trigger every time there is an error on firebase.

    I have always wondered what a reliable backup to the realtime db could be. Havent found much till date.

  • by iamleppert on 10/13/18, 5:19 AM

    Serves you right for using a “real-time database” (whatever that is). I’m sure your chat product feature could have been designed using a flat file as a datastore and a simple web socket server.