from Hacker News

Facebook, Instagram go down around the world in an apparent outage

by ajiang on 3/13/19, 6:00 PM with 331 comments

  • by wybiral on 3/13/19, 6:26 PM

    Can you imagine if Twitter and Google went down at the same time?

    People would be reactivating their Facebook accounts and having to sift through conspiracy theory posts about Hillary Clinton still just to figure out what was going on.

    Edit: The points on this post keep going up and down every time I check these comments. Yes, it was sarcasm, I was joking, but I was trying to point out that most people rely on a small set of services. "Cloud" has centralized things a lot.

  • by linsomniac on 3/13/19, 7:11 PM

    Could this be related to the storm?

    I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate UPS and generator. Last status update I had says that they are running off generators, but they've been shockingly tight-lipped about it.

    Don't get me wrong, it was hi-LAR-ious to call into their NOC and have them pretend that I was the only one having problems. "Can you tell me if there is a major data center outage going on?" "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." "... Why are you making a bunch of client calls if you aren't having an outage?"

  • by 40acres on 3/13/19, 9:31 PM

    I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.

  • by snazz on 3/13/19, 9:32 PM

    I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my workplace.

    This leaves me wondering what software all these places have in common. The application layers are all different, the databases are all different, the containerization and provisioning systems are different, but I imagine that all these systems rely on two things: the global Internet backbone, and maybe the Linux kernel.

    Have there been major security vulnerabilities patched lately in the Linux kernel that could have had unintended consequences?

  • by Implicated on 3/13/19, 7:31 PM

    Facebook's own status dashboard (https://developers.facebook.com/status/dashboard/) showed no issues or outtage just 30 min ago.

    I run a messenger bot platform - the webhooks stopped being delivered _hours_ ago... nothing on their status page until it had been down for hours.

    Their current issue...

    "We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution."

    What? lmao

  • by cronix on 3/13/19, 7:54 PM

    It looks like something much larger is going on. If you look at the front page of https://downdetector.com/ you'll see most major sites/backbones are having issues (Verizon/ATT/Sprint/CenturyLink/TMobile/Comcast/Level3/etc).

  • by jedberg on 3/13/19, 5:50 PM

    So yesterday Google had a major (and out of character) outage across its apps, and today Facebook has a major (and also out of character) outage across its apps.

    I can't wait to see the RCA for both of these and if they're related.

  • by CodeWriter23 on 3/13/19, 10:18 PM

  • by FabHK on 3/13/19, 10:31 PM

    Let's see whether we have a spike in the birth rate in 9 months.

    (Oh, turns out the Great Blackout Baby Boom was a myth:

    https://www.snopes.com/fact-check/from-here-to-maternity/ )

  • by kartan on 3/13/19, 7:33 PM

    "This usually means we're making an improvement to the database your account is stored on. While this process won't affect your account, you temporarily won't be able to access the site." https://www.facebook.com/help/134401680031995

    I guess that this is all that I will get. Facebook is never down, it is just making improvements (like restarting the services to make them work again).

  • by earless1 on 3/13/19, 6:46 PM

    What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.

  • by agosnell on 3/13/19, 6:27 PM

    If you use their API and haven't seen it yet, their issue is listed here on their status page:

    https://developers.facebook.com/status/issues/55989644784543...

  • by casper345 on 3/13/19, 7:59 PM

    The real storm is realizing through Facebook OAuth you cannot access your affiliate accounts. Caution to move your accounts away from Facebook

    Edit: Or have other methods than just relying on Facebook authentication

  • by nabla9 on 3/13/19, 7:18 PM

    Serious question: Was any value lost? (this may appear sarcastic)

    Facebook obviously loses some ad revenue and Facebook customers may lose sales. But do Facebook/Instagram users suffer? But how does losing social media for several hours affect the quality of life of users?

  • by btown on 3/13/19, 6:39 PM

    I've also seen issues uploading images to Whatsapp in the past half hour. I wonder if there's anything to do with the Google Cloud Storage outage that took down Gmail yesterday?

  • by dstola on 3/13/19, 8:53 PM

    The only things that I can think of that would cause this scale of being down is either a T1 center outage or (conspiracy hat on) a major hack and everyone is rush patching

    Would be interesting to read the post mortem if there is any regardless

  • by BucketSort on 3/13/19, 6:25 PM

    Google and FB having successive outages? Is this just a coincidence or is there some shared infrastructure that would explain this?

  • by mizeandmen on 3/13/19, 7:03 PM

    Bunch fb employees near pacific catch was talking about how fb was hacked

  • by JakeWesorick on 3/13/19, 5:18 PM

    https://www.facebook.com/platform/api-status/ still returns "Facebook Platform is Healthy", but you can't even load https://developers.facebook.com/status/dashboard/. Why have status pages if they are so susceptible to going down themselves?

  • by tinyhouse on 3/14/19, 1:33 AM

    BTW, many apps are affected by this. I cannot log in to any app that uses Facebook for authentication.

  • by jedberg on 3/13/19, 5:48 PM

    So yesterday Google had a major (and out of character) outage, and today Facebook has a major (and also out of character) outage.

    I can't wait to see the RCA for both of these and if they're related.

  • by Dangeranger on 3/13/19, 8:34 PM

    Ironically as I've been checking this post HN has been experiencing errors loading new comments.

  • by llamataboot on 3/13/19, 7:13 PM

    Instagram seems to load the feed here fine (EU), but doesn't allow you to log in from any device or post anything new. FB is totally fine if you are logged in for reading, but also can't log in if logged out.

    VPN to US, insta can login, but still not post.

    Distributed services are weird man!

  • by markstos on 3/13/19, 9:42 PM

    I see they've made some progress putting Instagram on the same infrastructure as Facebook.

  • by subcosmos on 3/13/19, 9:17 PM

    :/

    Something fun happening in Germany? https://www.akamai.com/us/en/resources/visualizing-akamai/re...

    And Level3 traffic going to Argentina? https://twitter.com/bgpstream/status/1105819050968580096

    And GreatBritain going to cambodia? https://bgpstream.com/event/197968

  • by goblin89 on 3/14/19, 6:42 AM

    Coincidentally, just watched The Social Network, the plot of which includes that quote by Mark:

    > Let me tell you difference between Facebook and everybody else. We don't crash ever! If the serves are down for even a day, our entire reputation is irreversibly destroyed. <…>

    > Even a few people leaving would reverberate through the entire use base. The users are interconnected. That is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go.

  • by hnruss on 3/13/19, 10:03 PM

    Did they move too fast and break too many things?

  • by evolvedcleaning on 3/14/19, 4:48 AM

    Doesn’t their world class team make such a long outage to be quite unlikely? How hard would it be to devote ample resources to a cover story for the “incident report”? Is the timing relative to the plethora of indictments relevant at all? Reasonable that this may be related to shredding of data and/or code, or even a cooperation to turn over data to government in secret deal?

  • by minimaxir on 3/13/19, 7:11 PM

    Minor update: https://twitter.com/facebook/status/1105907126424109056?s=21

    > We're focused on working to resolve the issue as soon as possible, but can confirm that the issue is not related to a DDoS attack.

  • by wybiral on 3/13/19, 7:13 PM

    Look at how many service providers have increased incidents reported here: https://downdetector.com/

    My bet is that people are having problems with FB/Insta and immediately assuming that the whole internet is messed up.

  • by johnchristopher on 3/13/19, 8:16 PM

    > The team at Jefferies remains reasonably positive, and in the firm's top growth stock calls for the week we found four tech stocks that are offering more aggressive accounts good entry points. Carl Court / Getty Images

    What's that weird tagline about ?

  • by akulbe on 3/13/19, 9:22 PM

    In other news, productivity everywhere skyrocketed!

  • by alien2003 on 3/14/19, 7:03 AM

    It's hard to believe that people simply can not live without fresh instagraphies

  • by revskill on 3/14/19, 1:26 PM

    What's the cause of outage then ? Disk, memory, CPU, network bandwidth,... ?

  • by benatkin on 3/13/19, 8:58 PM

    I got my github two factor auth SMS two hours late. Fortunately it was just my old laptop. I wonder if it was related. Good reminder to set up an authenticator app on my new phone so I don’t have to rely on SMS!

  • by armortech on 3/13/19, 10:09 PM

    Whatever happens right now at Facebook is less important than the fact they will never say what affected them. Of course nobody would tell 'hey, outage right now due to 0day / mistake' but...

  • by aviral on 3/13/19, 5:19 PM

  • by entwife on 3/14/19, 1:40 AM

    It's an experiment to see whether productivity improves when people aren't able to access FB and Instagram to slack off.

  • by nodesocket on 3/13/19, 7:02 PM

    And yet Facebook's stock is still up on the day (+.74%)?

    You'd think being down for hours would be negative news and revenue impacting.

  • by boshomi on 3/13/19, 6:16 PM

    the issue is also in EU: Is Facebook down? Messenger, site, app and Instagram hit by issues[1]

    [1]https://www.manchestereveningnews.co.uk/news/uk-news/faceboo...

  • by indigochill on 3/14/19, 12:19 PM

    First thought when I heard the news was BGP hijacking (ignoring whether accidental or deliberate). Doesn't the symptom fit other known cases like the Telegram incident in Iran last year, just at a larger scale?

    Admittedly networking is not my strength, so perfectly happy for someone to shoot down this hypothesis.

  • by JDiculous on 3/13/19, 5:45 PM

    Is Facebook actually working for anyone?

  • by js2 on 3/13/19, 10:20 PM

    I haven’t been able to post anything on Facebook, neither a new post to my wall nor add a comment on a friend’s post, since mid-morning US/Eastern and this is still the case. In addition I can’t login to the site - I am able to access the site only where I’m already logged-in.

  • by bluedino on 3/13/19, 9:37 PM

    Instagram works for me on my iPhone, but the comments are all missing. I kind of like it that way.

  • by letientai299 on 3/13/19, 5:02 PM

    This is the first time I experience this. Also note that current session on messenger.com still work, we can still send/receive message, but can't upload any image or send sticker. Looking forward for a post mortem analysis on this.

  • by aboutruby on 3/13/19, 7:45 PM

    Hacker News might just have a "Major Internet Services" status board.

  • by rdtsc on 3/13/19, 6:57 PM

    Google then Facebook and Instagram?

    My hunch is that it's the end of Q1 and people are trying to release code changes so they can pad their Q1 performance reviews "designed and delivered feature X on time in Q1".

  • by bradynapier on 3/14/19, 2:19 AM

    Perhaps relevant that npm has been having issues although they only recently caught and fixed them. Scoped Private npm packages were getting cloudflare 503 errors

  • by Endy on 3/14/19, 1:43 AM

    Since it went down for PC and not mobile, I was concerned if it was just an idea of audience testing, in the process of moving to an app-only platform.

  • by winrid on 3/13/19, 9:47 PM

  • by boshomi on 3/13/19, 6:10 PM

    see also: Facebook, Instagram down: Social media sites not working for many, FB doing 'required maintenance'

    [1] https://www.abc15.com/news/national/facebook-down-social-med...

  • by llamataboot on 3/13/19, 7:23 PM

    Whatsapp now down across much of Europe it looks like. Cannot send/receive messages.

  • by evesprini on 3/13/19, 8:35 PM

    Argentina: Whatsapp works for text, but any type of media takes very long to send.

  • by red_admiral on 3/13/19, 7:20 PM

    FB has been down for me for around an hour, but has just come back up again.

  • by tinyhouse on 3/13/19, 6:52 PM

    From Google I got an error clicking on instagram and Quora links today.

  • by peepX on 3/14/19, 4:47 AM

    I'm surprised there wasn't a national crisis alert sent out

  • by yeutterg on 3/13/19, 5:44 PM

    Can confirm. Also, API integrations, such as Buffer, are not working.

  • by nathan929 on 3/13/19, 6:28 PM

    ...terrifying API developers everywhere...

  • by allannienhuis on 3/13/19, 5:07 PM

    seems to be more than just messenger login; All of facebook is super flakey this morning.

  • by deca6cda37d0 on 3/13/19, 9:24 PM

    Unfortunately the AI software used to censor fake news from facebook has decided, again, to censor facebook :D

  • by abbiya on 3/14/19, 6:39 AM

    somebody please wipe off everything from all these big corps

  • by pragmaticalien8 on 3/13/19, 7:26 PM

    must be a super 0 day.

  • by nilskidoo on 3/14/19, 12:52 AM

    Affecting more lives than any terrorist act.

  • by jak92 on 3/13/19, 6:38 PM

      ... and nothing of value was lost.

  • by shrthnd on 3/13/19, 7:41 PM

    I didn't notice.

  • by njn on 3/13/19, 6:38 PM

    Good! Keep them down.

  • by arisAlexis on 3/13/19, 6:25 PM

    yesterday Google and today Facebook. My conspirator says it's the Chinese government showcasing.

  • by zomg on 3/13/19, 6:56 PM

    All my funny cat videos and memes are loading fine. Did you try rebooting your router 62 times?

    All joking aside, is this news? :/

  • by miloignis on 3/13/19, 11:50 PM

    My fiance's uncle sent something today that because of a school shooting in Brazil, they were blocking all images and video shared to social networks like "WhatsApp, Instagram, Facebook and other social networks". I haven't been able to verify this myself or from any other sources, but I wonder if either people are misinterpreting the FB outage or if Brazil is blocking content it's having weird ripple effects.