from Hacker News

Incident with GitHub Actions

by dryja on 3/29/22, 10:40 AM with 47 comments

  • by phoe-krk on 3/29/22, 11:08 AM

  • by somehnacct3757 on 3/29/22, 11:39 AM

    As the person who gets pinged when our CI isn't working, I'm hoping my next employer doesn't have github in their stack.

    We've reached the stage where architectural decisions are being questioned. Nobody should need to ask me "well why are we using github?" but here we are.

    They needed to fix this two weeks ago. The next time I'm in charge of stack decisions I will be evaluating competitors. This is exactly how Slack beat Hipchat.

    Don't let weekends dilute your view of the situation. There's only 23 weekdays in March and Github has not been reliable for 5 of them.

  • by bob1029 on 3/29/22, 1:18 PM

    This is getting out of hand. I had a call with github enterprise sales yesterday about this. Best they could offer is a blog post link and taking more of our money.

    Hoping to migrate away our little chunk of github to a private island by end of next week.

    Hackernews talks a lot of shit about how big boys can run their computers better than us lowly startup peasants. This is a fortunate situation where that stale argument starts to fall apart really quickly. I have many other systems with far higher availability than what github offers in their public cloud or even as part of the contractual 99.5% enterprise SLA.

    Sure, github is really complicated and hard. Yes, it's an incredible tool. No, it's not OK to rest on your laurels and let COTS database technology kill your product when you are a billion dollar technology company with the ability to write your own systems from scratch several times per year using parallel teams and other investor money furnaces.

  • by rydgel on 3/29/22, 11:18 AM

    I'm glad we have a self-hosted Gitlab. Sure you need to do a bit of setup and configuration, but it's worth in the long run.

  • by 0x6c6f6c on 3/29/22, 11:09 AM

    Not even sure how to talk about this anymore with how frequently this has gone down.

    Our company uses GitHub actions and other features for deployments so every one of these outages stops us from putting any work on production.

  • by hypeatei on 3/29/22, 11:32 AM

    I think it's easy to pile-on and say "GitHub is down again! Should've self hosted lol!".

    When, in reality, it's one service having issues and not the whole site. These incidents also seem to be resolved quickly.

    Downtime is not the end of the world.

  • by throwmeariver1 on 3/29/22, 11:44 AM

    At this point in time it really is a gamble if Actions work or not. Even if there is no “Incident” the success rate is probably at 70-80% in the last three weeks.

  • by RapperWhoMadeIt on 3/29/22, 11:32 AM

    I would maybe start migrating repos away from GitHub, since they have proven to be quite unreliable. Nonetheless, I must say I do appreciate GitHub's UI and even their CLI is quite nice. Is there a service out there that basically provides the same good user experience, CI/CD and has no costs for public repos? Apart from the unreliability I can't complain about GitHub.

  • by ricardobayes on 3/29/22, 11:35 AM

    Oh it's just the daily GitHub incident.

  • by jasoneckert on 3/29/22, 11:24 AM

    I had a lot of issues with GitHub actions failing yesterday (the infamous "Request failed with status code 502"), but there were no service issues listed on their status page. It just started working late yesterday and seems to be fine today though.

  • by rvz on 3/29/22, 11:23 AM

    Again? Last time this happened was 4 days ago as I also said:

    >>> Hopefully GitHub won't have another outage in a week's (or even a months) time. [0]

    Yet another incident less than a week / 4 days later with GitHub Actions if one was to go 'all in' on GitHub. Would have to reset the counter once again.

    You can see in the whole comment chain [0] and in [1] why I was totally right in the 'long term' of not 'centralizing everything' on GitHub since 2020. [1]

    [0] https://news.ycombinator.com/item?id=30790842

    [1] https://news.ycombinator.com/item?id=22867803

  • by eurasiantiger on 3/29/22, 11:08 AM

    Attacks or incompetence?

    Hello Microsoft executives, we would like some transparency.

  • by anotherhue on 3/29/22, 1:27 PM

    Like Okta, I pay them to not do this.

    Anyone skimmed their enterprise SLA docs to see how much we are due back?

    Developers ain't free.