from Hacker News

GitHub Actions Down

by kakamiokatsu on 5/16/21, 8:47 AM with 52 comments

And I'm here waiting for my scheduled build to start...

  • by rvz on 5/16/21, 9:31 AM

    So now I am expecting a major GitHub Actions incident (worst case, the whole of GitHub) to go down every single month at least once. Last time this went down was last month. [0]

    I now doubt if they can consistently manage more than a month without a major incident like this one.

    [0] https://news.ycombinator.com/item?id=26666843

  • by zomglings on 5/16/21, 1:32 PM

    In the last week, there have been at least two ~1 hour periods where actions were stuck in the queue. Even posted about this on the GitHub community, but no response. [0]

    As unreliable as GitHub Actions are, their convenience factor and price are right.

    We take a very simple measure so we don't get fucked by these kinds of incidents: we don't use any actions from the marketplace.

    All our GitHub Actions workflows are bash scripts that we wrote (and which often live in our repos at `deploy/deploy.bash`). The secrets necessary to run these scripts are available to the infrastructure team on 1Password.

    This makes it easy for us to deploy manually and retroactively reflect that release on GitHub (e.g. through a tag or a release).

    [0] https://github.community/t/github-action-stuck-on-starting-w...

  • by andreareina on 5/16/21, 9:23 AM

  • by limoce on 5/16/21, 10:14 AM

    My workflows have been queued for at least four hours. It's a hotfix commit. GitHub Actions helped me a lot in synchronizing updates with other repositories automatically, but this time I have to manually apply it. Because I'm the first time expecting GitHub Actions incident, I was wondering that I ran out of my free quota for this month...

    By the way, should GitHub send an email to the owner if any workflow has been delayed for an unreasonable time?

  • by sebmellen on 5/16/21, 10:08 AM

    We build an Electron app for MacOS regularly. We run these builds on Github’s MacOS VMs.

    The internet fails to connect when running yarn install about 3 out of every 5 times.

    We’ve gotten multiple refunds for this issue, but it’s still a complete mess. Unfortunately we’ve built much of our process around GH Actions... if not I’m sure we would be on CircleCI or TravisCI already. We’ve also considered switching to self-hosted runners.

  • by bob1029 on 5/16/21, 1:54 PM

    We only use GH Actions for check builds right now, which are easily disabled as a temporary measure.

    Having the ability to build & deploy your software outside the confines of a cloud vendor is essential to survival. When the automation works, its great. When it doesn't, have a manual process that you can follow on a local workstation.

    At the end of the day, you can always email the customer a zip file and walk them through installing the update in production. That is, as long as you didn't make your architecture and CI/CD one in the same thing, in which case you probably need to hit the reset button and try again.

  • by 19h on 5/16/21, 9:09 PM

    Tip: you can run and host your own GitHub Actions runners.

    https://docs.github.com/en/actions/hosting-your-own-runners/...

  • by OzzyB on 5/16/21, 1:44 PM

    Since I have a Job that runs every hour, I can see that it's been down for ~4hours.

  • by afrcnc on 5/16/21, 2:36 PM