from Hacker News

GitHub issue - resolved

by mactavish88 on 3/27/23, 12:19 PM with 160 comments

Seems like GitHub's struggling at the moment: https://downdetector.com/status/github/

I'm getting unicorns.

  • by larsnystrom on 3/27/23, 12:38 PM

    HN should really have a “Down HN” category, as this is the most reliable place to get this information.

  • by davemp on 3/27/23, 1:22 PM

    Now's a good opportunity to ask about alternative Git hosts. What other services do HNers use?

    I've been unwilling to host any personal projects on GH after Copilot launched and it because clear that GH/MS doesn't really respect the authors of the code they host. Honestly open source in general has gotten a little less compelling to me after Copilot. The recent security issue at GH has also turned me off even more on Git hosting services.

    For closed source projects maybe it's just best to store encrypted backups off site and spin up a self hosted option whenever collaboration is needed. Seems pretty inefficient though.

  • by Phemist on 3/27/23, 12:32 PM

    This was done out of an abundance of caution, I'm sure.

  • by mabbo on 3/27/23, 12:48 PM

    I feel like HN could just have a traffic monitor that adds a little icon to the main page. Like "I dunno what's up, but there's a LOT of y'all here right now, so something probably is".

  • by can16358p on 3/27/23, 12:22 PM

    Yup. Tried pushing and got "remote: fatal error in commit_refs". Was trying to understand what I was doing wrong.

  • by MattIPv4 on 3/27/23, 12:26 PM

  • by hakanderyal on 3/27/23, 12:36 PM

    Because this needs to happen as one prepares to push to day's last deploy. Back to backups.

    Have backups for critical systems, people. In my case, it's building docker containers locally and luckily deploying to one server via ssh.

  • by belter on 3/27/23, 1:28 PM

    Git Docs... - https://git-scm.com/book/en/v2/Getting-Started-About-Version...

    "...The next major issue that people encounter is that they need to collaborate with developers on other systems. To deal with this problem, Centralized Version Control Systems (CVCSs) were developed. These systems (such as CVS, Subversion, and Perforce) have a single server that contains all the versioned files, and a number of clients that check out files from that central place. For many years, this has been the standard for version control."

    "...However, this setup also has some serious downsides. The most obvious is the single point of failure that the centralized server represents. If that server goes down for an hour, then during that hour nobody can collaborate at all or save versioned changes to anything they’re working on. If the hard disk the central database is on becomes corrupted, and proper backups haven’t been kept, you lose absolutely everything..."

    "...This is where Distributed Version Control Systems (DVCSs) step in. In a DVCS (such as Git, Mercurial, Bazaar or Darcs), clients don’t just check out the latest snapshot of the files; rather, they fully mirror the repository, including its full history. Thus, if any server dies, and these systems were collaborating via that server, any of the client repositories can be copied back up to the server to restore it. Every clone is really a full backup of all the data...."

  • by gitgud on 3/27/23, 12:56 PM

    Hmm this is quite an incident, I haven't seen Github reject all pushes to all repos like this in quite some time.... hopefully not SSH key related

  • by TheLocehiliosan on 3/27/23, 12:29 PM

    Also known as Monday.

  • by rvz on 3/27/23, 12:34 PM

    It goes down every month like I said before [0]. The last time this happened was 2 days ago [1] then weeks ago [2] and it is evident that it is falling apart in front of us.

    First, it was the RSA key leak in [1][3], then the site's key expired causing down time again [2] and now this.

    I don't think anyone can tell me with a straight face that GitHub was any more reliable or better when Microsoft acquired it. It is now worse off.

    Nothing has changed except for more outages and downtime.

    So so reliable. /s

    [0] https://news.ycombinator.com/item?id=35004629

    [1] https://news.ycombinator.com/item?id=35295216

    [2] https://news.ycombinator.com/item?id=35003741

    [3] https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k...

  • by Fabricio20 on 3/27/23, 1:38 PM

    Looks like it's back and the status page reports it as green, however GitHub Actions builds are not triggering on push as usual when this happens, so for me it doesn't seem to be fully recovered yet.

  • by Bostonian on 3/27/23, 12:48 PM

    Suppose you start your day by downloading some code from GitHub to work on. This morning you would be stuck. Do you save code on your hard drive or company servers or GitLab to handle this risk?

  • by drewbug01 on 3/27/23, 1:28 PM

    One wonders if MSFT leadership will ever connect the dots between “hollowing out talent over the years,” “hiring freezes,” and “layoffs” to this outcome.

    One wonders sometimes if that’s the goal.

  • by tiffanyh on 3/27/23, 1:36 PM

    Only code with Github is replicated locally, not everything else like wiki, issues, etc.

    That's something I love about Fossil.

    Everything with Fossil (wiki, issues, code) is replicated as well.

  • by tomaszcekalo on 3/27/23, 1:40 PM

    I wonder if it has anything to do with them updating their RSA SSH host key (https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k...) 3 days ago.

  • by vidluther on 3/27/23, 1:06 PM

    I feel like events like these are the new "compiling".. https://xkcd.com/303/

  • by MattIPv4 on 3/27/23, 12:20 PM

    Also seeing unicorns, mostly for "No server is currently available to service your request.". If I don't get a unicorn, the page takes a long old time to load.

  • by willejs on 3/27/23, 1:06 PM

    I used to get my github status updates in slack via an RSS feed, and just searched for the feed again, but its gone? Is there an alternative for this?

  • by robinhoodexe on 3/27/23, 1:34 PM

    Is anyone self-hosting the enterprise version of github as a backup? Or using it as the primary source and then using the cloud version as backup?

  • by mnsh on 3/27/23, 1:29 PM

    > We’ve identified an infrastructure change that has been rolled back and we are monitoring recovery.

    From github status

  • by CipherThrowaway on 3/27/23, 12:22 PM

    Back up for me

  • by powerage on 3/27/23, 12:34 PM

    I have been getting error when I try git push. I taught the problem is me but I saw this thread.

  • by bouncycastle on 3/27/23, 12:40 PM

    Ever since Microsoft took over...

  • by tibanne on 3/27/23, 1:30 PM

    One has to ask oneself... would this have happened without the Microsoft acquisition?

  • by henriquenunez on 3/27/23, 12:38 PM

    Seems it's fixed now?

  • by new_user_final on 3/27/23, 12:47 PM

    Why GitHub is down so often? Why is it not possible to keep it up 100% of time (without counting physical failure)?. I haven't seen any down time for my system (it has hundreds of thousands of users online) in months since I have completed the setup.

  • by tarruda on 3/27/23, 1:17 PM

    Maybe they are experimenting with GPT automating devops

  • by oneeyedpigeon on 3/27/23, 12:43 PM

    Related to the Twitter source code leak, perchance?

  • by Madmallard on 3/27/23, 12:56 PM

    ok this makes sense i was just trying to push

  • by 5mv2 on 3/27/23, 2:25 PM

    Are there downtime safe GitHub alternatives?

  • by siva7 on 3/27/23, 1:07 PM

    Codespaces is not connecting anymore for me

  • by dainiusse on 3/27/23, 3:05 PM

    Commited code straight from copilot :)

  • by dolmen on 3/27/23, 1:38 PM

    Looks OK from here (Paris, France).

  • by lesquivemeau on 3/27/23, 1:36 PM

    It seems like you can push again !

  • by 0xcb0 on 3/27/23, 12:52 PM

    I cant push anymore! Anyone else ?

  • by yla92 on 3/27/23, 1:30 PM

    Everything is green now!

  • by tomconte on 3/27/23, 12:36 PM

    Time for another coffee

  • by kukimonsta on 3/27/23, 1:09 PM

    I am getting unicorns!

  • by xwdv on 3/27/23, 1:40 PM

    It’s up now.

  • by dark-star on 3/27/23, 1:33 PM

    seems to work fine for me...?

  • by rurban on 3/27/23, 12:39 PM

    read-only

  • by expotesticle on 3/27/23, 12:35 PM

    Thanks for ruining my day