from Hacker News

Zen and the Art of Reliability

by perch56 on 3/8/22, 7:37 AM with 7 comments

  • by amdelamar on 3/10/22, 3:25 AM

    > 4. Fail small, reducing blast radius

    One more thing not mentioned here, is that using a microservice architecture can naturally help isolate outages to small parts of your app/website. Rather than take it down entirely.

    My team supports a large microservice system, and while there are definite drawbacks to the architecture, one of the major benefits is that its never 100% down at any given time. Usually a prod incident will make one particular button flakey or one view/page fail to load. Some users won't even notice theres an outage. Oncall is paged and can quickly rollback the squeaky microservice to a previously deployed version, and let an engineer investigate the root cause in a test environment later.

  • by kaycebasques on 3/9/22, 11:53 PM

    > In other words, we decided to measure only the systems we controlled. In retrospect that was naive. Our customers don’t care if we run the service that fails, or a vendor we use runs the service that fails. They care that they can’t use Zendesk to do their job.

    Yes!

  • by blakesterz on 3/9/22, 11:15 PM

    Interesting to see how they do things. Of all the many things that we use at work, ZenDesk is my favorite. It never gets in my way, does things exactly the way I want them, it's just great. GitHub is probably a close second. Slack and Basecamp somewhere in them middle. With anything from Atlassian always being my least favorite.

  • by ram_rar on 3/10/22, 7:46 AM

    Cloud services have come a long way. Not trying to diss the article, but scaling CRUD app for 250k/sec is not as difficult as it used to be. It mainly comes down to how you manage state in your architecture.

    Back when I was @ yahoo, serving 10k concurrent request from single server used to be such a big deal. Now, hardly anyone thinks about. Most of the reliability/fault tolerance/auto scaling features comes from underlying AWS/GCP services. We just need to write decent microservice to glue these things together and voila!