from Hacker News

Ask HN: What's the actual fallout been from the Cloudflare bug?

by anon456 on 3/5/17, 5:52 PM with 75 comments

• by perlgeek on 3/5/17, 8:17 PM

  The trouble with this sort of bug is that we'll likely never know.

  Some people's accounts will be compromised, and nobody will know if it's been due to fishing, insecure passwords, or an information leak such as the Cloudflare bug, or an undisclosed or undiscovered breach somewhere.

  The more responsible Cloudflare customers have invalidated existing sessions; that's much less hassle than forcing a password reset, and since session tokens are transmitted in every request, a leaked token is much more likely than a leaked password.

• by tgragnato on 3/5/17, 8:37 PM

  An information leak is an information leak : we still fail to realise that it's something that's happening daily. There's no drama in it.

  Criminals are taking advantage of opportunities like this every day, still no one cares too much about it (HN bubble & friends excluded).

  Things like this may have a strong impact or not in the press/popularity circus, but in this particular case it seems they promptly monitored the situation (thanks to their competent staff).

  What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION .

  No one (no matter how able you are) can write absolutely bug-free algorithms : even when dealing with formal verified software you can still attack the assumptions.

  Security by correctness is a laudable effort, but computing customers data with a single process is not sane. I'm aware they're doing this for performance reasons, but a well implemented isolation layer would have prevented this (even while dealing with a bug like that).

  Their architecture is vulnerable.

• by ScottBurson on 3/5/17, 8:27 PM

  As I understand it -- somebody please correct me if I have this wrong -- the thing about Cloudbleed is that there isn't necessarily any relationship between the site whose page is cached and the site whose credentials appear in that cached page. So the only way to know that a particular site didn't have credentials leaked is to search all the caches of all the search engines on the Internet.

  So, as perlgeek says, we'll probably never know specifically what the impact was.

• by i336_ on 3/5/17, 10:06 PM

  I think Google freaked out to the Nth degree because it's quite likely that cached data is probably stored in a system that doesn't have [m]any security restrictions attached to it, and... well, there are >72k people. You're going to find well-meaning "what happens if I... OOPS" types (for any definition of "OOPS"), along with (ostensibly equally well-meaning) "hey, an OAuth token that actually works! Let's see just how far we can take this..." people... and then some that aren't just interested in fun engineering challenges, if you get what I mean.

  I have no idea what Google employees have access to. I've always wondered whether they can hand-code their own MapReduce syntax over Google's actual Web index (I could find SO MANY THINGS if that were possible!). I wouldn't be surprised if the cache data <-> index were accessible to everyone who's been around for >6 months, so they can tinker with it.

  But I guess the only reason I'm able to type this is that I haven't signed The Large Book Of NDAs (I presume it's large).

• by bostik on 3/5/17, 8:23 PM

  For a remarkably level-headed take on the fallout, I recommend listening to the latest episode of Risky Business [0]. The interview with Troy Hunt gives a calm, informed and above all well reasoned baseline for response.

  0: https://risky.biz/RB445

• by ufmace on 3/6/17, 4:51 AM

  I think part of the disconnect is that this issue is a big deal for tech professionals, but barely noticeable for everyone else.

  By the nature of the bug, the likelihood of any particular individual having any meaningful exploitable information exposed to somebody in a position to exploit it is astronomically low. So most ordinary people are ignoring it, and justifiably so.

  If you're responsible for security for a site that sends traffic through CloudFare, then it's a very big deal for you. You'd better be quick on the trigger to see and react to this stuff, and you'll have to mass-reset sessions at the very least, and possibly reconsider whether you really want to be terminating SSL at CloudFare. Exactly because, while not much has probably been exposed, you will never be able to be sure what was exposed to anyone from random hackers to the whole world, via search engine caches. So a broad reaction is justified.

  And of course people who like tech but aren't actually responsible for any sites being served through CloudFare tend to react the most. Even though it's not a big deal if you're already doing all of the standard security precautions, like different passwords everywhere and 2-factor authentication on anything important.

• by smilesnd on 3/6/17, 12:40 AM

  Old saying don't believe everything you hear, and only believe half what you read. Most news and blog outlets are horrible for information. Either they redigest someone else information, spin it to be more interesting, or just jump on the hype train. Their are few blogs and news outlets that actually have experts worth listening to. When the news/blog outlets heard whatsapp had a technique flaw by a security expert they jump on it like a fresh piece of meat. Whatsapp didn't have any security flaws or implementation flaws, but that didn't matter. The news/blogs didn't even understand it they were just jacked for some revenue from ads for this fresh piece of meat. It did so much harm to the people that use it for security reasons and to the company. But that didn't matter it is all about getting you to download a piece of javascript to tell some company you may have notice there ad that was place on someones website. News is dead. Bloggers only care about traffic. Experts are either paid to be used and abused or don't have a big enough audience to be heard.

• by rini17 on 3/5/17, 10:25 PM

  I would welcome if this incident shone light on possible Cloudflare alternatives. For example, it should be technically doable for DDoS protection service to only initially verify user is not a bot, and then merely tunnel unchanged SSL traffic directly between server and client. Does anyone do this?

• by jacquesm on 3/6/17, 4:26 AM

  Cloudflare should count its blessings. If not for Google the fall-out would have been a lot larger.

  So even if the sky didn't fall that's no reason to pretend this wasn't a big deal.

• by dedalus on 3/5/17, 10:26 PM

  The way you know it's real is when you call up cloudflare's top customers and ask if they would switch to the competition the answer was a resounding yes. That's how I know it's not based on a HN bubble

• by _pmf_ on 3/6/17, 1:26 PM

  > "this is the end" and "this is so bad, we're f*cked"

  End of what? It will just give rise to slightly more secure, improved services (maybe be the same providers, maybe by competitors, but definitely financed and implemented by the same people).

  > And I haven't seen any huge leaks or items in the press about some terrible hack or theft that has brought someone or a corporate "down".

  Look at the Sony/PSN breach; there has been zero accountability, and it has not hurt the PS4 launch at all. Consumers just don't give a shit.

• by cookiecaper on 3/5/17, 8:28 PM

  I think you're misinterpreting the comments about the scale of the leak. The risk that a concrete compromise would occur as a result was always pretty small.

  The bigger thing was the grandiose scale, the impact on administrators in having to rotate a significant number of credentials, and the hit to CloudFlare's reputation. A bug where you randomly dump random data without regard to its sensitivity or origin (i.e., data from completely unrelated sites could've been included in the dump), and have no way to tell what actually leaked, is the worst kind of privacy bug there is, precisely because it's impossible to triage. No one can ever know everything that actually got out.

  CloudFlare is now a major piece of internet infrastructure. It's impossible to know that anything sent through a CloudFlare server between Sept 2016 and Feb 2017 wasn't accidentally publicly leaked, and worse, non-trivial quantities of this data were being accidentally saved permanently in search indexes. Surely some bad actors have saved such results in their own private indexes as well.

  When CloudFlare says "your site was probably unaffected", they're making a guess, because they have no way to actually tell. They're just assuming that based on the volume of requests your CloudFlare endpoint receives and the volume of requests made to endpoints that exhibited this bug, content from your site probably didn't get out. But there's no way to know.

  If we take that seriously, it requires us to consider everything that went through a CloudFlare server as potentially publicized and preserved in the public record (including usually-transparent unique identifiers like session cookies/tokens). We then have to assume that an adversary obtained any and all such data, and respond as best as we can to preclude the possibility of that adversary exploiting the leaked secrets to harm our and/or our company's interests.

  Of course, the flip side of the sheer scale of this, and the fact that the bug was relatively rare and that there was no way to control what content it dumped, is that it's very unlikely any of your data specifically actually got leaked.

  If you and/or your company are OK with crossing your fingers and hoping this won't affect you, there is probably a 99.something-something-something% chance you'd be right. Most people have responded by resetting tokens/passwords for anything that uses CloudFlare, since that's relatively low-impact and most people were probably overdue for a credential recycle anyway, and have left it at that.

  This does clearly illustrate that the internet has a few de-facto junction points, which would be very high-value for an attacker. That's worth keeping in mind.

• by m348e912 on 3/6/17, 1:25 PM

  Aside from being a black eye on Cloudflare, I don't see this issue being of much consequence. I have yet to see one real-world example of a screenshot or link to a cache of data of leaked data (sensitive or not). If anyone has an example, please share. As others have mentioned, the real fear is of what could have leaked, not what did leak.

• by JumpCrisscross on 3/5/17, 9:09 PM

  I deleted my subscription and account with 23andme. I have a few friends and colleagues who acted similarly with other sites.

• by tgsovlerkhgsel on 3/6/17, 5:47 AM

  My guess: For criminals, the cost of finding the needle in the haystack is just not worth it - it's easier to phish fresh credentials than to hope that you'll find some in some hard-to-crawl archived data set. So we won't see anything there.

  Realistically, this will probably only be exploited by intelligence agencies who have the means of collecting all the data and motivation to do so, and maybe not even them (because they have better ways too). If they do exploit it, the nature of intelligence agencies, of course, means that you typically won't notice any direct impact.

  The reason why this caused such a big panic is that while the likelihood of your password being compromised is small, it could have hit anything, and by conventional wisdom, any password/key that _may_ have been exposed, even if the likelihood is small, needs to be considered compromised. Hence, "OMG everything is compromised".

  Another reason was probably that it was a really scary wake-up call demonstrating the risks of centralized services. Cloudflare is a Single Point of Failure for a lot of security, but that is easy to push aside until you see it failing.

  Realistically (and I'm going to get a lot of flak for saying this) the correct way to handle it is to rotate extremely high-value credentials (think Bitcoin exchangs, administrative access to major services, ...), reset sessions if you're hosting your website on Cloudflare (since session tokens are much more likely to leak than passwords, and the cost of forcing users to re-auth is small especially if your sessions expire regularly anyways), and then call it a day.

  In particular, keep in mind that for high-value services, you're hopefully already using 2FA, so even if an attacker did get your password through this, they probably don't have your 2FA token (although Kraken, a Bitcoin exchange, pointed out to their customers that they should re-setup 2FA if originally set up during the vulnerable timeframe, since the key used to derive the 2FA could be compromised).

• by mrmondo on 3/6/17, 1:18 AM

  On the non technical side - A LOT of multiple people's time inside our small NFP org inspecting logs, rolling passwords, keys etc, expiring sessions and communicating with clients.

• by lmm on 3/6/17, 12:36 PM

  Sometimes those of us who live in the pure, mathematical world of software forget that the real world is more resillient than that.

  People's passwords, identities, and bank and credit card details will have been leaked. Identity theft and other fraud will happen as a result of this. But we have systems in place for dealing with it, and ultimately life will go on. I've had fraudulent charges on my bank account; it was a serious inconvenience at the time, but it wasn't life-changingly bad.

• by nodesocket on 3/6/17, 3:06 AM

  This won't necessarily be a popular opinion, but I remember when everyday there were negative blog posts and stories about Apple and the new MacBook Pros. It seemed like every developer got on their Medium and wrote a blistering post. It you just read the internet and HN you'd think the world at Apple was crumbling down.

  Yet since $AAPL released the new MacBook Pro (Oct 27th '16), their stock is up 24%, with a breakout record Q1. Let's not forget that the entire market has been in an epic bull run since Trump took office, so perhaps that is a factor.

  Source ($AAPL vs Dow Jones and S&P since Oct 27th): https://www.google.com/finance?chdnp=0&chdd=0&chds=1&chdv=0&...

  Don't believe what you see on HN all the time. People here are incredibly intelligent for the most part, but there is frankly lots of disconnect from reality. In my opinion lots of conspiracy theorists, purest, and some social justice warriors pushing agendas.

  My opinion... But I think we can bundle GitLab, CloudFlare, and Uber into categories of will be just fine.

• by blibble on 3/6/17, 3:20 AM

  namecheap use cloudflare, and didn't email their customers (including me) telling them that they may want to change their password...

  I have now transferred every single one of my domains away from namecheap

  I also installed the following extension, and now watch what I put into cloudflare pages: https://chrome.google.com/webstore/detail/claire/fgbpcgddpmj...

• by davind3r on 3/5/17, 8:58 PM

  Netki sent me an email that they think they might be affected, so they strongly recommend to change password.

• by overcast on 3/6/17, 12:15 AM

  Well for me, the fallout has been the pain in the ass task of changing all my passwords.

• by simplehuman on 3/5/17, 9:13 PM

  Does anyone here (startup) use ddos protection ?

• by quirkafleeg on 3/5/17, 8:27 PM

  Hopefully the fallout is that Cloudflare gets its act together.

  Even if your friends know 100% that they can't possibly have been negatively affected by tons of private information being dumped all over the internet, I'm not sure how such anecdotal evidence is any more instructive than a HN "bubble".

  Even if nobody at all ended up negatively affected in any serious way, I don't see why people shouldn't remark on the potential effects of such a fiasco when it happens. Was anyone really predicting "the end"?

• by aaron695 on 3/6/17, 2:13 AM

  > Should we always take news like this with a grain of salt?

  Yes.

  Except this fear is part of our income source like the TSA, except they are more like 100% IT is a bit less.