by jcims on 1/14/25, 1:00 AM
Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise.
https://www.federalregister.gov/documents/2023/06/09/2023-12...
On the surface a product like managed git repos would seem to be relatively straightforward to deal with, but these same regulated firms are also under tremendous scrutiny for access management, change management, SDLC, etc etc. They also have a huge software footprint which just increases the impact.
Self-hosting is obviously the traditional answer, but that's not exactly a simple one either.
Just an interesting problem.
by crest on 1/14/25, 12:02 AM
It's almost as if only Git is distributed, but people sold out to Github for convenience. Too bad Git lacks a distributed bug tracker and wiki system like Fossil. Guess Github has to fail a lot more for things to change.
by freewizard on 1/14/25, 12:40 AM
It was down for ~2 hours. The status website claims "degraded performance", but in reality we get
git@github.com: Permission denied (publickey).
Either GitHub didn't know how to communicate, or they were not sure about the real impact.
This is bad.
by itsjustjordan on 1/14/25, 12:13 AM
I was going insane doubting my SSH knowledge, stopped short of creating new keys thankfully!
by vinayan3 on 1/13/25, 11:59 PM
Experiencing this as well. I was a bit concerned something was wrong with my key but alas it is not my key but an outage.
Developer "snow day".
by tom1337 on 1/14/25, 12:31 AM
Not sure if I am just noticing GitHub's issues more often as I am using their tools pretty much every day but their availability is kinda not great. Be it Actions failing or something "not core business" (read: git operations) but I can't remember a month in the past where I was not annoyed by any outage on GitHub
by cdme on 1/14/25, 12:14 AM
It's just a hub at the moment.
by jxi on 1/13/25, 11:56 PM
GitHub Actions are also failing with 500 (gateway timeout) errors when trying to fetch the repository.
by hunkins on 1/14/25, 12:16 AM
Update: looks like the problem has been identified.
"We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."
by j45 on 1/14/25, 12:09 AM
I guess it's a good time as any to setup a backup or check out alternatives that could mirror or standalone. Gitlab, Gitea, any others worth checking out?
by remram on 1/13/25, 11:59 PM
I hope you didn't wait until now to push your day's work!
by clintonb on 1/14/25, 12:18 AM
by localghost3000 on 1/13/25, 11:50 PM
Can't pull anything from my repos which is not great.
by james33 on 1/14/25, 12:30 AM
All checkmarks just turned green and my push went through. Snow day over.
by arnath on 1/14/25, 12:17 AM
It's odd to me that there's nothing indicating an issue is ongoing on the github.com homepage
by burgerrito on 1/14/25, 12:48 AM
Thankfully I use Forgejo! No reason to stick with GitHub when Git itself is distributed by nature
by eru on 1/14/25, 12:48 AM
They were also down a few days ago, but HN flagged my submission as a dupe. :)
by lukeh on 1/13/25, 11:58 PM
Yup, SSH rejecting my key, thought it was something at my end but no. Can't run any Yocto builds... I guess a good opportunity to take a walk or do some life admin.
by rasengan on 1/14/25, 12:13 AM
Just the info I was expecting to see on HN. Unfortunately (or Fortunately?), a better source for these kinds of things than anywhere else.
by impure on 1/14/25, 12:23 AM
I thought it was my internet. Oh well, time to finish that game of Civ 6. Edit: and it literally came back up the moment I posted this.
by __turbobrew__ on 1/13/25, 11:56 PM
Yep, 100% of git operations are failing. CI/CD and dev operations are at a halt.
by the_svd_doctor on 1/14/25, 1:03 AM
Ha so that's why my git pull were so slow. I should have checked HN faster.
by 486sx33 on 1/14/25, 12:26 AM
Damn you azure! Or is Microsoft not even running their precious GitHub in azure
by hunkins on 1/13/25, 11:47 PM
GitHub git operations are down. Any git operation returns a 500 error response.
by __d on 1/13/25, 11:54 PM
SSH access appears to be rejecting my key too.
by ashishb on 1/14/25, 12:51 AM
Why does GitHub go down almost once a month?
by noplacelikehome on 1/14/25, 12:54 AM
I've just ended my Pro plan; enough is enough. I've been debating moving my repositories to a self-hosted forge for a while; it'll be an interesting opportunity to move more CI operations out of Actions and into Nix.
by qqqult on 1/14/25, 12:29 AM
lol I changed my ssh key 2 times until I figured out pubkey permission denied was an issue on their side
by souenzzo on 1/14/25, 4:55 AM
windows window operations are down
by 0xbadcafebee on 1/14/25, 12:53 AM
> all Git operations were unavailable due to a configuration change
eats popcorn waiting for explanation of why they didn't catch it in non-prod
by xyst on 1/14/25, 2:29 AM
Yet another reason to self host your VCS. Only thing hosted to “the cloud” these days are my backups - split between S3 and GCS.
~$25/mo
by atulvi on 1/14/25, 12:09 AM
I just reset my ssh key lol.
by Rexogamer on 1/14/25, 12:34 AM
the incident has been marked as resolved; I am now able to push
by JSTrading on 1/14/25, 12:08 AM
Panic for a minute SSH rejected a key!
by unwoven on 1/14/25, 12:01 AM
lmao i deleted and readded a new key thinking my key was fucked