from Hacker News

GitHub was down

by benbruscella on 7/13/20, 6:47 AM with 128 comments

• by ivalm on 7/13/20, 7:32 AM

  This is a nice reminder that you can have multiple remote repos and push/pull to all of them at the same time. For my side projects I usually use both github and google cloud source (I use gcp). If one is down the other is still available and then just resync when service is recovered.

• by a012 on 7/13/20, 7:15 AM

  I know Github is down because I'm trying to update a kops cluster, but Github breaks it

      > kops update cluster
      error reading channel "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": unexpected response code "500 Internal Server Error" for "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": 500: Internal Server Error

• by lprd on 7/13/20, 7:02 AM

  This is slowly becoming a weekly occurrence ever since Microsoft entered the picture...

  https://www.githubstatus.com/

• by FBISurveillance on 7/13/20, 9:14 AM

  From what I understood, they are having capacity issues with one of their MySQL masters or something. I've read that they are in process of sharding/resolving that but it takes time.

  To GitHub SRE/oncall people: hang in there, you're awesome.

• by luiseduardo on 7/13/20, 6:53 AM

  Yeah, our builds are failing for a couple of hours now, and we just use Github because one of the dependencies in NPM downloads a binary from a public repository. We're already forking what we can to our gitlab self-hosted server, but even a simple git clone or even browsing the website can lead to a HTTP 500 right now.

  Lots of incidents lately, but it's becoming increasingly hard to get away from Github.

• by rvz on 7/13/20, 7:27 AM

  I said this before many times and I'll say it again, consider self-hosting your projects on a solution like GitLab or Gitea to avoid this sort of situation. [0]

  GNOME, Xfce, Redox, Wireguard, KDE and Haiku all have self-hosted on either cgit, Gitlab, Phabricator or Gitea.

  [0] https://news.ycombinator.com/item?id=23676072

• by jonny383 on 7/13/20, 6:57 AM

  Seriously, again!? I am growing seriously impatient with this. It's been downhill since Microsoft took over.

  GitLab is starting to look good (or even Gitea self-hosted).

• by yuppiepuppie on 7/13/20, 8:13 AM

  ITT: Lots of Github hate.

  Dont forget that change in software is inherently risky and will result in bugs, etc. Id rather have a platform that is always looking to make things better and risking a bit of downtime, than a stale platform that we all know we depend on.

• by tdonovic on 7/13/20, 6:48 AM

  Getting 500s all over the api, gl whoever is on call

• by sairamkunala on 7/13/20, 12:11 PM

  (copied over from other thread - https://news.ycombinator.com/item?id=23817794)

  Github started doing availability reports. Last month's details in the blog post below with summary of the issue.

  Stay tuned till next month for the current outage.

  https://github.blog/2020-07-08-introducing-the-github-availa...

• by ashishb on 7/13/20, 9:47 AM

  What if we had a smart failover. Use GitHub and GitLab simultaneously. All issues, all comments, all PRs duplicated. If one goes down you use the other one in the time being with no interference at all. One can probably then build a frontend which magically does this failover for ci/CD etc. Isn't that's how much redundant this should be?

• by bamboozled on 7/13/20, 8:06 AM

  I guess it's time Github realizes that they're not longer just relied upon for git, but for so much more.

• by chvid on 7/13/20, 7:53 AM

  Cool thing about git is that it is distributed and there is no single point of failure ...

• by mro_name on 7/13/20, 10:28 AM

  Who would really rely on a single, external, free, no-guarantees service and not have redundancy to tolerate some hours of downtime?

  Make github a mirror (at least source-wise) and you can benefit from it's outreach without being held hostage. Am happy with that e.g. https://notabug.org/mro/ShaarliOS/src/master/doap.rdf Inspired by https://indieweb.org/POSSE

• by m0xte on 7/13/20, 7:43 AM

  Yay this crashed our Jenkins instance as well.

• by holler on 7/13/20, 7:55 AM

  Hopefully it's just them reverting their latest UI changes

• by INTPenis on 7/13/20, 7:35 AM

  I just had to access my Github stars to find an old app I bookmarked. No dice. Otherwise I've moved all my current projects to Gitlab so Stars and contributing to other repos are my two most used features atm.

• by Evidlo on 7/13/20, 8:38 AM

  Everything seems up now. Does anyone know if Github pages went down too?

• by mxschumacher on 7/13/20, 8:35 AM

  in my build pipeline, I query several different package hosts (npm, pypi, docker-hub etc) and Github/Gitlab. If any of them is unavailable, the build fails.

  What's the best way to keep my own copy of the packages my software needs (and their dependencies), so that my build process is less fragile? Ideally, I'd only have to rely on those 3rd party platforms to download new versions or have them as a backup.

  When relying on my own copy of required packages - can I expect much faster builds?

• by quyleanh on 7/13/20, 7:30 AM

  I still don't understand people who always mentions to Microsoft's acquisition. Until the official statement, it isn't Microsoft failure. Don't blame them.

• by jschulenklopper on 7/13/20, 8:10 AM

  This can be merged with https://news.ycombinator.com/item?id=23817794

• by mindfreeze on 7/13/20, 7:49 AM

  Another Discussion: https://news.ycombinator.com/item?id=23817794

• by bezmenov on 7/13/20, 7:59 AM

  They’re minutes away from dropping below two nines looking at Issues, Pull Requests, Projects. Other services look comparably unreliable.

• by benbruscella on 7/13/20, 6:50 AM

  "We have identified the source of elevated errors and are working on recovery."

  A day wrecker!

• by hhas01 on 7/13/20, 8:33 AM

  Human sacrifice, dogs and cats living together… mass hysteria!

• by Beldin on 7/13/20, 7:59 AM

  It is ironic that a version control system engineered to be distributed is now typically used in such a centralised way.

• by maxbaines on 7/13/20, 7:58 AM

  Savage Unicorn, Github who knew.

• by nicc on 7/13/20, 7:09 AM

  People are advocating hosting their own Git repos, but wouldn't those go down, too, and wreck the day even more?

  Or, are you guys all devops geniuses better than those who work at GitHub?

• by quantummkv on 7/13/20, 7:03 AM

  Surely it can't be a coincidence that Github is down every other week after the Microsoft acquisition? Is Microsoft interfering too much? Or did the core technical expertise leave for other greener pastures in Microsoft or outside?